Non-linear Constrained Optimization

min $f(x)$
s.t. $h(x)=0$
$g(x)\le 0$

$x\in {\mathbb{R}}^n,f:{\mathbb{R}}^n\to \mathbb{R},h:{\mathbb{R}}^n\to {\mathbb{R}}^m,g:{\mathbb{R}}^n\to {\mathbb{R}}^p$

注：非线性优化问题和线性优化问题的最大区别在于目标函数是否是线性函数。

Case 1

min $f(x)=0$
s.t. $h(x)=0$

$h:{\mathbb{R}}^n\to {\mathbb{R}}^m,h\in C^1$ (continuously differential)

Definition

Def: Let $x^{\ast }$ be with $h_1(x^{\ast })=0,\cdots ,h_m(x^{\ast })=0$. $x^{\ast }$ is a regular point, if $\nabla h_1(x^{\ast }),\cdots ,\nabla h_m(x^{\ast })$ are linearly independent.

Jacobian: $Dh(x^{\ast })={\left[\begin{array}{c}D{h}_1(x^{\ast })\\ D{h}_2(x^{\ast })\\ \cdots \\ D{h}_m(x^{\ast })\end{array}\right]}^T$

Def: Surface: S = { x ∈ R n : h 1 ( x ) = 0 , ⋯ , h m ( x ) = 0 } S=\{x\in\mathbb{R}^n:h_1(x)=0,\cdots,h_m(x)=0\} S={x∈Rn:h1​(x)=0,⋯,hm​(x)=0}

Example 1

$n=3,m=1,h(x)=x_2-x_3^2$
$Dh(x)=[0,1,-2x_3]$
$\forall x\in {\mathbb{R}}^3,Dh(x)\ne 0$
S = { x : x 2 − x 3 2 = 0 } S=\{x:x_2-x_3^2=0\} S={x:x2​−x32​=0}

Example 2

$h_1(x)=x_1,h_2(x)=x_2-x_3^2$

$Dh(x^{\ast })=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& -2x_3\end{array}\right]$
S = { x : x 1 = 0 , x 2 − x 3 2 = 0 } S=\{x:x_1=0,x_2-x_3^2=0\} S={x:x1​=0,x2​−x32​=0}

Necessary/Sufficient Conditions

FONC: $x^{\ast }$ local minimizer $\Rightarrow \nabla f(x^{\ast })=0$
SONC: $x^{\ast }$ local minimizer $\Rightarrow \nabla f(x^{\ast })=0,\forall y:y^{T}F(x^{\ast })y\ge 0$
SOSC: (1)$\nabla f(x^{\ast })=0$ (2)$\forall y:y^{T}F(x^{\ast })y\ge 0\Rightarrow x^{\ast }$ strictly local minimizer

Definition

Def: A curve $C$ on a surface $S$ is a set of points { x ( t ) ∈ S : t ∈ ( a , b ) } , x ( t ) : R → R n \{x(t)\in S:t\in(a,b)\},x(t):\mathbb{R}\rightarrow \mathbb{R}^n {x(t)∈S:t∈(a,b)},x(t):R→Rn is a continuous function.

Curve differentiable: $\dot{x}(t)=\frac{dx}{dt}(t)=\left[\begin{array}{c}{\dot{x}}_1(t)\\ {\dot{x}}_2(t)\\ \cdots \\ {\dot{x}}_n(t)\end{array}\right]$ exists for all $t\in (a,b)$
$\ddot{x}(t)=\frac{d^{2}x}{d{t}^2}(t)=\left[\begin{array}{c}{\ddot{x}}_1(t)\\ {\ddot{x}}_2(t)\\ \cdots \\ {\ddot{x}}_n(t)\end{array}\right]$ exists for all $t\in (a,b)$

Def: tangent space at x ∗ ∈ S = { x ∈ R n : h ( x ) = 0 } x^*\in S=\{x\in\mathbb{R}^n:h(x)=0\} x∗∈S={x∈Rn:h(x)=0} is the set T ( x ∗ ) = { y : D h ( x ∗ ) y = 0 } T(x^*)=\{y:Dh(x^*)y=0\} T(x∗)={y:Dh(x∗)y=0}

Example

S = { x ∈ R 3 : h 1 ( x ) = x 1 = 0 , h 2 ( x ) = x 1 − x 2 = 0 } S=\{x\in \mathbb{R}^3: h_1(x)=x_1=0,h_2(x)=x_1-x_2=0\} S={x∈R3:h1​(x)=x1​=0,h2​(x)=x1​−x2​=0}

$Dh(x^{\ast })=\left[\begin{array}{ccc}1& 0& 0\\ 1& -1& 0\end{array}\right]$
$S$ regular points
T ( x ) = { y : ∇ h 1 ( x ) T y = 0 , ∇ h 2 ( x ) T y = 0 } = { [ 0 , 0 , α ] : α ∈ R } ⇒ x 3 T(x)=\{y:\nabla h_1(x)^Ty=0,\nabla h_2(x)^Ty=0\}=\{[0,0,\alpha]:\alpha\in\mathbb{R}\}\Rightarrow x_3 T(x)={y:∇h1​(x)Ty=0,∇h2​(x)Ty=0}={[0,0,α]:α∈R}⇒x3​-axis

Theorem

Thm: Let $x^{\ast }$ be a regular point. $T(x^{\ast })$: tangent space at $x^{\ast }$. Then: $y\in T(x^{\ast })\iff \exists$ differentiable curve on $S$ passing through $x^{\ast }$ with derivative $y$ at $x^{\ast }$.

FONC(Lagrange’s Condition)

2-Dimensional

$h:{\mathbb{R}}^3\to \mathbb{R}$
Let $x^{\ast }=[x_1^{\ast },x_2^{\ast }{]}^T,h(x^{\ast })=0$
Assume $\nabla h(x^{\ast })\ne 0$
Let $x(t):\mathbb{R}\to {\mathbb{R}}^2,x(t)$ continuously differentiable.
$x(t)=\left[\begin{array}{c}{x}_1(t)\\ x_2(t)\end{array}\right],t\in (a,b),x^{\ast }=x(t^{\ast })$
$\because \forall t\in (a,b):h(x(t))=0$
$\therefore \forall t:\frac{d}{dt}h(x(t))=0$
$\therefore \nabla h(x^{\ast })$ orthogonal to $x(t^{\ast })$

Assume $x^{\ast }=x(t^{\ast })$ minimizer of $f(x)$ on S = { x : h ( x ) = 0 } S=\{x:h(x)=0\} S={x:h(x)=0}

Define $\varphi (t)=f(x(t))\stackrel{FONC}{\Rightarrow }\frac{d\varphi }{dt}(t^{\ast })=0$
$0=\frac{d}{dt}\varphi (t^{\ast })=\nabla f(x(t^{\ast }){)}^T\dot{x}(t^{\ast })=\nabla f(x^{\ast }{)}^T\dot{x}(t^{\ast })$
$\Rightarrow \nabla f(x^{\ast })$ is orthogonal to $\dot{x}(t^{\ast })$
$\nabla f(x^{\ast })=\lambda \nabla h(x^{\ast })$

Summary:

$x^{\ast }$ is a minimizer of $f:{\mathbb{R}}^2\to \mathbb{R}$ with $h(x)=0,h:{\mathbb{R}}^2\to \mathbb{R}$. Then, $\nabla h(x^{\ast })$ and $\nabla f(x^{\ast })$ are parallel.
$\Rightarrow$ If $\nabla h(x^{\ast })\ne 0$, then $\exists {\lambda }^{\ast }$ s.t. $\nabla f(x^{\ast })+{\lambda }^{\ast }\nabla h(x^{\ast })=0$

Lagrange’s Theorem[FONC]

$x^{\ast }$ is a local minimizer of $f:{\mathbb{R}}^n\to \mathbb{R}$, subject to $h(x)=0,h:{\mathbb{R}}^n\to {\mathbb{R}}^m,m\le n$. Assume $x^{\ast }$ is regular. Then $\exists x^{\ast }\in {\mathbb{R}}^m$ s.t. $Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })=0$

Lagrange’s Function

Lagrange’s function: $l:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$
$l(x,\lambda )=f(x)+{\lambda }^{T}h(x)$

min $l(x,\lambda )\Leftarrow$ FONC
$Dl(x^{\ast },{\lambda }^{\ast })=0\Rightarrow \{\begin{array}{l}{D}_{x}l(x^{\ast },{\lambda }^{\ast })=0\\ D_{\lambda }l(x^{\ast },{\lambda }^{\ast })=0\end{array}$

Example 1

已知长方体的表面积为$A$，求体积的最大值。
max $x_1{x}_2{x}_3$
s.t. $x_1{x}_2+x_2{x}_3+x_1{x}_3=\frac{A}{2}(A>0)$
$f(x)=-x_1{x}_2{x}_3,h(x)=x_1{x}_2+x_2{x}_3+x_1{x}_3-\frac{A}{2}$
$\nabla f(x)=[-x_2{x}_3,-x_1{x}_3,-x_1{x}_2{]}^T$
$\nabla h(x)=[x_2+x_3,x_1+x_3,x_1+x_2{]}^T$
All feasible solutions are regular.
$\lambda \in \mathbb{R}$
$\{\begin{array}{l}\nabla f(x)+\lambda \nabla h(x)=0\\ h(x)=0\end{array}\Rightarrow \{\begin{array}{l}{x}_2{x}_3-\lambda (x_2+x_3)=0\\ x_1{x}_3-\lambda (x_1+x_3)=0\\ x_1{x}_2-\lambda (x_1+x_2)=0\\ x_1{x}_2+x_2{x}_3+x_1{x}_3-\frac{A}{2}=0\end{array}$

当$x_1=x_2=x_3=\sqrt{\frac{A}{6}}$时，取到最值

Example 2

$f(x)=x_1^2+x_2^2,h(x)=x_1^2+2x_2^2-1$
$\nabla f(x)=\left[\begin{array}{c}2x_1\\ 2x_2\end{array}\right],\nabla h(x)=\left[\begin{array}{c}2x_1\\ 4x_2\end{array}\right]$
All feasible solutions are regular.
$\{\begin{array}{l}\nabla f(x)+\lambda \nabla h(x)=0\\ h(x)=0\end{array}\Rightarrow \{\begin{array}{l}2x_1+2\lambda x_1=0\\ 2x_2+4\lambda x_2=0\\ x_1^2+2x_2^2=1\end{array}$

either $x_1=0$ or $\lambda =-1$

$\lambda =-1\Rightarrow \{\begin{array}{l}{x}_1=\pm 1\\ x_2=0\end{array}$

$x_1=0\Rightarrow \{\begin{array}{l}\lambda =-\frac{1}{2}\\ x_2=\pm \frac{1}{\sqrt{2}}\end{array}$

$f(\left[\begin{array}{c}1\\ 0\end{array}\right])=f(\left[\begin{array}{c}-1\\ 0\end{array}\right])=1$

$f(\left[\begin{array}{c}0\\ \frac{1}{\sqrt{2}}\end{array}\right])=f(\left[\begin{array}{c}0\\ -\frac{1}{\sqrt{2}}\end{array}\right])=\frac{1}{2}$

当$x_1=0,x_2=\pm \frac{1}{\sqrt{2}}$时，取到最小值$\frac{1}{2}$

Example 3

min $-x^{T}Qx$
s.t. $x^{T}Px=1$
$P,Q>0,P^T=P,Q^T=Q$

$f(x)=-x^{T}Qx,h(x)=x^{T}Px-1$
$l(x,\lambda )=x^{T}Qx+\lambda (1-x^{T}Px)$
$D_{x}l(x,\lambda )=2x^{T}Q-2\lambda x^{T}P=0\Rightarrow (\lambda P-Q)x=0\Rightarrow P^{-1}Qx=\lambda x\Rightarrow \lambda ,x$ are $P^{-1}Q$’s eigenvalue and eigenvector
$D_{\lambda }l(x,\lambda )=1-x^{T}Px=0$

$Qx=P\lambda x$
$\Rightarrow x^{T}Qx=\lambda x^{T}Px$
$\Rightarrow x^{T}Qx=\lambda$
$\Rightarrow {\lambda }^{\ast }:$ maximal eigenvalue of $P^{-1}Q$

SONC

Assume $f:{\mathbb{R}}^n\to \mathbb{R},h:{\mathbb{R}}^n\to {\mathbb{R}}^m$ twice continuously differentiable.
$l(x,\lambda )=f(x)+{\lambda }^{T}h(x)=f(x)+{\lambda }_1{h}_1(x)+\cdots +{\lambda }_m{h}_m(x)$
$L(x,\lambda )=F(x)+{\lambda }_1{H}_1(x)+\cdots +{\lambda }_m{H}_m(x)$

Thm(SONC): $x^{\ast }$ a local minimizer of $f:{\mathbb{R}}^n\to \mathbb{R}$ with $h(x)=0,h:{\mathbb{R}}^n\to {\mathbb{R}}^m,m\le n,f,h\in C^2$. Then, $\exists {\lambda }^{\ast }\in {\mathbb{R}}^m$, s.t. { D f ( x ∗ ) + λ ∗ T D h ( x ∗ ) = 0 ∀ y ∈ T ( x ∗ ) = { y : D h ( x ∗ ) y = 0 } : y T L ( x ∗ , λ ∗ ) y ≥ 0 \begin{cases} Df(x^*)+{\lambda^*}^TDh(x^*)=0\\ \forall y\in T(x^*)=\{y:Dh(x^*)y=0\}:y^TL(x^*,\lambda^*)y\geq 0 \end{cases} {Df(x∗)+λ∗TDh(x∗)=0∀y∈T(x∗)={y:Dh(x∗)y=0}:yTL(x∗,λ∗)y≥0​

SOSC

$f,h\in C^2$, If $\exists x^{\ast }\in {\mathbb{R}}^n,{\lambda }^{\ast }\in {\mathbb{R}}^m$, s.t.

1. $Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })=0$
2. $\forall y\in T(x^{\ast }):y^{T}L(x^{\ast },{\lambda }^{\ast })y>0$

then $x^{\ast }$ is a strict local minimizer of $f(x)$ w.r.t. $h(x)=0$

Example 1

max $x^{T}Qx$
s.t. $x^{T}Px=1$

$Q=\left[\begin{array}{cc}4& 0\\ 0& 1\end{array}\right],P=\left[\begin{array}{cc}2& 0\\ 0& 1\end{array}\right]$

$P^{-1}Q=\left[\begin{array}{cc}2& 0\\ 0& 1\end{array}\right]$
$\Rightarrow {\lambda }_1=2,{\lambda }_2=1$
$\Rightarrow {\lambda }^{\ast }=2$
$\Rightarrow x^{\ast }=[\frac{1}{\sqrt{2}},0{]}^T$ or $x^{\ast }=[-\frac{1}{\sqrt{2}},0{]}^T$

Example 2

Consider min $\frac{1}{2}{x}^{T}Qx$
s.t. $Ax=b$

$Q>0,Q=Q^T,A\in {\mathbb{R}}^{m\times n},m\le n,b\in {\mathbb{R}}^m,rankA=m$

$l(x,\lambda )=\frac{1}{2}{x}^{T}Qx+{\lambda }^T(b-Ax)$
$D_{x}l(x,\lambda )=x^{T}Q-{\lambda }^{T}A=0$
$\Rightarrow x=Q^{-1}{A}^T\lambda$
$\Rightarrow Ax=A{Q}^{-1}{A}^T\lambda$
$\Rightarrow \lambda =(A{Q}^{-1}{A}^T{)}^{-1}b$
$\Rightarrow x=Q^{-1}{A}^T(A{Q}^{-1}{A}^T{)}^{-1}b$

$L(x,\lambda )=Q>0$

Case 2

min $f(x)$
s.t. $h(x)=0$
$g(x)\le 0$

$f:{\mathbb{R}}^n\to \mathbb{R}$
$h:{\mathbb{R}}^n\to {\mathbb{R}}^m,m\le n$
$g:{\mathbb{R}}^n\to {\mathbb{R}}^p$

Definition

Def: An inequality constraint $g_j(x)\le 0$ is called active at $x^{\ast }$, if $g_j(x^{\ast })=0$; otherwise, inactive.

Def: Let $x^{\ast }$ satisfy $h(x^{\ast })=0$ and $g(x^{\ast })\le 0$. Let J ( x ∗ ) = { j : g j ( x ∗ ) = 0 } , x ∗ J(x^*)=\{j: g_j(x^*)=0\},x^* J(x∗)={j:gj​(x∗)=0},x∗ is called regular, if $\nabla h_i(x^{\ast })$ for all $1\le i\le m$ and $\nabla g_i(x^{\ast })$ for all $j\in J(x^{\ast })$ are linear independent.

KKT-Theorem(FONC)

Let $f,h,g\in C^1,x^{\ast }$ be a regular point and a local minimizer of $f(x)$ w.r.t. $h(x^{\ast })=0$ and $g(x^{\ast })\le 0$. Then, there exist ${\lambda }^{\ast }\in {\mathbb{R}}^m$ and ${\mu }^{\ast }\in {\mathbb{R}}^p$ s.t.

1. ${\mu }^{\ast }\ge 0$
2. $Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })+{{\mu }^{\ast }}^{T}Dg(x^{\ast })=0$
3. ${{\mu }^{\ast }}^{T}g(x^{\ast })=0$

Example 1

min $-\frac{400R}{(10+R{)}^2}$
s.t. $-R\le 0$

$\nabla f(R)=-\frac{400(10-R)}{(10+R{)}^3}$

$\{\begin{array}{l}\mu \ge 0\\ Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })+{{\mu }^{\ast }}^{T}Dg(x^{\ast })=0\\ {\mu }^{T}g(x)=0\\ g(x)\le 0\\ h(x)=0\end{array}$

$\Rightarrow \{\begin{array}{l}\mu \ge 0\\ -\frac{400(10-R)}{(10+R{)}^3}-\mu =0\\ \mu R=0\\ R\ge 0\end{array}$

If $\mu >0$, then $R=0,\mu =-4$（✕）
If $\mu =0\Rightarrow R=10$（✓ ）

Example 2

min $-\frac{4000}{(10+R{)}^2}$
s.t. $-R<0$

$\nabla f(R)=\frac{8000}{(10+R{)}^3}$

KKT: $\{\begin{array}{l}\mu \ge 0\\ \frac{8000}{(10+R{)}^3}-\mu =0\\ \mu R=0\\ R\ge 0\end{array}$

$\mu =0\Rightarrow$ no solution（✕）
$\mu >0\Rightarrow R=0,\mu =8$（✓ ）

总结

这节课主要介绍了非线性约束优化问题。按照不同的约束条件，把问题分为了两种情形。第一种情形是只有等式约束，第二种情形既有等式约束又有不等式约束。在第一种情形下，重点介绍了拉格朗日条件，并在二维情况下推导出了拉格朗日条件。由于拉格朗日条件是一阶必要条件（FONC），又进一步介绍了用拉格朗日条件来求最值的拉格朗日乘数法。然后简要地介绍了二阶必要条件（SONC）和二阶充分条件（SOSC）。最后考虑了第二种情形，并给出了KKT条件。