The goal of this thesis is to design practical algorithms for nonlinear optimization in the case where the objective function is deterministic or stochastic. Problems of this nature arise in many applications including machine learning and image processing. The thesis is divided into four main chapters. Chapters \ref{chap:Inexact}, \ref{chap:Adasample} and...