Lagrange multiplier
Introduction:
Lagrange multipliers are used in multivariable calculus to find maxima and minima of a function subject to constraints (like "find the highest elevation along the given path" or "minimize the cost of materials for a box enclosing a given volume").
where the λ term may be either added or subtracted. If f(x0, y0) is a maximum of f(x, y) for the original constrained problem, then there exists λ0 such that (x0, y0, λ0) is a stationary point for the Lagrange function (stationary points are those points where the partial derivatives of
are zero).
History:
Lagrange was one of the creators of the calculus of variations, deriving the Euler–Lagrange equations for extrema of functionals. He also extended the method to take into account possible constraints, arriving at the method of Lagrange multipliers. The Lagrange function (or Lagrangian) is defined by:
Lagrange presented his new conception in a memoir (Lagrange 1764), written to answer a question advanced in 1762 by the Academy of Paris, concerning the libration of the moon. More than fifteen years later (Lagrange 1780), he outlined a more elaborate and general version of the same program. He finally realised this program in the Méchanique analitique (Lagrange 1788).
Lagrange's idea has two clear sources. The first is the Bernoullian principle of virtual velocities for the equilibrium of any system of bodies animated by central forces. The second source is d'Alembert's principle for dynamics.
The following text is essentially from P. Bussottis paper: On the Genesis of the Lagrange Multipliers
The Lagrange Multiplier were introduced in the framework of statics
He introduced this mathematical approach in the framework of statics in order to determine the general equations of equilibrium for problems with constraints. At the beginning of his Mécanique Analytique, Lagrange tackles statics and poses three principles as the foundations for the subject: (i) the principle of the lever (ii) the principle of the composition of forces and (iii) the principle of virtual velocities. It was this third principle virtual velocities where the Lagrange Multiplier occurred the first time.
Lagrange writes in (Ref $1$, pages $17$-$18$):
By virtual velocity, it has to be meant the one that a body in an equilibrium condition would receive if the equilibrium was interrupted; namely, the velocity that the body would really assume in the first instant of movement; the principle consists in this: the forces are in equilibrium if they are in the inverse proportion to their virtual velocities. . .
Constrained optimization
In mathematical optimization, constrained optimization (in some contexts called constraint optimization) is the process of optimizing an objective function with respect to some variables in the presence of constraints on those variables. The objective function is either a cost function or energy function which is to be minimized, or a reward functionor utility function, which is to be maximized. Constraints can be either hard constraints which set conditions for the variables that are required to be satisfied, or soft constraintswhich have some variable values that are penalized in the objective function if, and based on the extent that, the conditions on the variables are not satisfied.
A general constrained minimization problem may be written as follows:
where 
and 
are constraints that are required to be satisfied; these are called hard constraints.
and are constraints that are required to be satisfied; these are called hard constraints.History:
The history of constrained optimization spans nearly three centuries. The principal warhorse, Lagrange multipliers, was discovered by Lagrange in the Statics section of his famous book on Mechanics from 1788, by applying the idea of virtual velocities to problems in statics with constraints. The idea of virtual velocities, in turn, goes back to a letter of Johann Bernoulli from 1715 to Varignon, in which he announced a very simple rule for solving hundreds of Varignon’s problems in the blink of an eye. Varignon then explains this rule in his book published in 1725. Half a century later, Bernoulli’s rule was chosen by Lagrange as the general principle for the foundation of his mechanics, with the multipliers as the main tool for treating mechanical constraints.
In the second edition of his mechanics, published in 1811, Lagrange stressed the importance of his multipliers also for constrained optimization. In particular, they provide spectacular simplifications of entire chapters of Euler’s treatise on Variational Calculus from 1744.
Lagrange multipliers is however a much farther reaching concept; we show how one can discover the important primal and dual equations in optimal control and the famous maximum principle of Pontryagin using only Lagrange multipliers. Pontryagin and his group, however, did not discover the maximum principle this way, since they were coming from a completely different area of mathematics. We finally give the complete formulation of PDE constrained optimization based on duality introduced by Lions, and conclude with an outlook on more recent applications.
