Non-linear Constrained Optimization

KKT-Theorem(FONC)

$f,h,g\in C^1,x$: regular point & local minimizer. Then, exist ${\lambda }^{\ast }\in {\mathbb{R}}^m,\mu \in {\mathbb{R}}^p$:
①${\mu }^{\ast }\ge 0$
②$Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })+{{\mu }^{\ast }}^{T}Dg(x^{\ast })=0$
③${{\mu }^{\ast }}^{T}g(x^{\ast })=0$

SONC

Thm: $f,h,g\in C^2,x^{\ast }$:regular point & local minimizer.Then, exist ${\lambda }^{\ast }\in {\mathbb{R}}^m,\mu \in {\mathbb{R}}^p$:
①${\mu }^{\ast }\ge 0,Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })+{{\mu }^{\ast }}^{T}Dg(x^{\ast })=0,{{\mu }^{\ast }}^{T}g(x^{\ast })=0$
②$\forall y\in T(x^{\ast }):y^{T}L(x,\lambda ,\mu )y\ge 0$

注： T ( x ∗ ) = { y : D h ( x ∗ ) y = 0 , D g j ( x ∗ ) y = 0 , ∀ j ∈ J ( x ∗ ) } T(x^*)=\{y:Dh(x^*)y=0,Dg_j(x^*)y=0,\forall j\in J(x^*)\} T(x∗)={y:Dh(x∗)y=0,Dgj​(x∗)y=0,∀j∈J(x∗)}

Definition

Def: T ~ ( x ∗ , μ ) = { y : D h ( x ∗ ) y = 0 , D g j ( x ∗ ) y = 0 , f o r j ∈ J ~ ( x ∗ , μ ∗ ) } \tilde{T}(x^*,\mu)=\{y:Dh(x^*)y=0,Dg_j(x^*)y=0, for j\in \tilde{J}(x^*,\mu^*)\} T~(x∗,μ)={y:Dh(x∗)y=0,Dgj​(x∗)y=0,forj∈J~(x∗,μ∗)}

Remark: $\because \tilde{J}(x^{\ast },{\mu }^{\ast })\subseteq J(x^{\ast })\therefore T(x^{\ast })\subseteq \tilde{T}(x^{\ast },{\mu }^{\ast })$

SOSC

Thm: $f,h,g\in C^2$. If exists $x^{\ast }\in {\mathbb{R}}^n$ and ${\lambda }^{\ast }\in {\mathbb{R}}^m,\mu \in {\mathbb{R}}^p$ s.t.
①${\mu }^{\ast }\ge 0,Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })+{{\mu }^{\ast }}^{T}Dg(x^{\ast })=0,{{\mu }^{\ast }}^{T}g(x^{\ast })=0$
②$\forall y\in \tilde{T}(x^{\ast })$ with $y\ne 0:y^{T}L(x^{\ast },{\lambda }^{\ast },{\mu }^{\ast })y>0$
then $x^{\ast }$ is a strict local minimizer.

Example 1

min $x_1{x}_2$
s.t. $x_1+x_2\ge 2$
$x_2\ge x_1$

$f(x)=x_1{x}_2$
$g_1(x)=2-x_1-x_2$
$g_2(x)=x_1-x_2$

$\nabla f(x)=\left[\begin{array}{c}{x}_2\\ x_1\end{array}\right],\nabla g_1(x)=\left[\begin{array}{c}-1\\ -1\end{array}\right],\nabla g_2(x)=\left[\begin{array}{c}1\\ -1\end{array}\right]$

KKT-conditions:$\{\begin{array}{l}{\mu }_1,{\mu }_2\ge 0\\ x_2-{\mu }_1+{\mu }_2=0\\ x_1-{\mu }_1-{\mu }_2=0\\ {\mu }_1(2-x_1-x_2)+{\mu }_2(x_1-x_2)=0\\ 2-x_1-x_2\le 0\\ x_1-x_2\le 0\end{array}$

$x^{\ast }=\left[\begin{array}{c}1\\ 1\end{array}\right],{\mu }^{\ast }=\left[\begin{array}{c}1\\ 0\end{array}\right]$

$D{g}_1(x^{\ast })=[-1,-1],D{g}_2(x^{\ast })=[1,-1],Df(x)=[1,1]$
$\Rightarrow x^{\ast },D{g}_j(x^{\ast })\forall j\in J(x^{\ast })$ linearly independent
$\Rightarrow x^{\ast }$ regular point

T ( x ∗ ) = { y : [ − 1 , − 1 ] y = 0 , [ 1 , − 1 ] y = 0 } = { 0 } T(x^*)=\{y:[-1,-1]y=0,[1,-1]y=0\}=\{0\} T(x∗)={y:[−1,−1]y=0,[1,−1]y=0}={0}
SONC is satisfied by $x^{\ast },{\mu }^{\ast }$

$L(x,\lambda ,\mu )=F(x)+\lambda H(x)+\mu G(x)=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]+[1,0]\left[\begin{array}{cc}0& 0\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]$
T ~ ( x ∗ , μ ∗ ) = { y : [ − 1 , − 1 ] y = 0 } = { y : − y 1 = y 2 } \tilde{T}(x^*,\mu^*)=\{y:[-1,-1]y=0\}=\{y:-y_1=y_2\} T~(x∗,μ∗)={y:[−1,−1]y=0}={y:−y1​=y2​}

$[1,-1]\in \tilde{T}(x^{\ast },{\mu }^{\ast })$
$[1,-1]\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]\left[\begin{array}{c}1\\ -1\end{array}\right]=[-1,1]\left[\begin{array}{c}1\\ -1\end{array}\right]=-2<0$
SOSC fails.
no local min.

Example 2

min $f(x)=(x_1-1{)}^2+x_2-2$
s.t. $h(x)=x_2-x_1-1=0$
$g(x)=x_1+x_2-2\le 0$

$Df(x)=[2x_1-2,1],Dh(x)=[-1,1],Dg(x)=[1,1]$
KKT-conditions:$\{\begin{array}{l}\mu \ge 0\\ 2x_1-2-\lambda +\mu =0\\ 1+\lambda +\mu =0\\ \mu (x_1+x_2-2)=0\\ x_2-x_1-1=0\\ x_1+x_2-2\le 0\end{array}$

$\Rightarrow {\mu }^{\ast }=0,x_1^{\ast }=\frac{1}{2},x_2^{\ast }=\frac{3}{2},{\lambda }^{\ast }=-1$
$x$ regular

$L(x^{\ast },{\lambda }^{\ast },{\mu }^{\ast })=F(x^{\ast })+{{\lambda }^{\ast }}^{T}H(x^{\ast })+{{\mu }^{\ast }}^{T}G(x^{\ast })=\left[\begin{array}{cc}2& 0\\ 0& 0\end{array}\right]$

T ( x ∗ ) = { y : [ − 1 , 1 ] y = 0 , [ 1 , 1 ] y = 0 } = { 0 } T(x^*)=\{y:[-1,1]y=0,[1,1]y=0\}=\{0\} T(x∗)={y:[−1,1]y=0,[1,1]y=0}={0},SONC satisfied
T ~ ( x ∗ , μ ∗ ) = { y : [ − 1 , 1 ] y = 0 } = { y : y 1 = y 2 } \tilde{T}(x^*,\mu^*)=\{y:[-1,1]y=0\}=\{y:y_1=y_2\} T~(x∗,μ∗)={y:[−1,1]y=0}={y:y1​=y2​}
$y^T\left[\begin{array}{cc}2& 0\\ 0& 0\end{array}\right]y=[a,a]\left[\begin{array}{cc}2& 0\\ 0& 0\end{array}\right]\left[\begin{array}{c}a\\ a\end{array}\right]=2a^2$

$\forall y\ne 0:2a^2>0\Rightarrow x^{\ast }=\left[\begin{array}{c}\frac{1}{2}\\ \frac{3}{2}\end{array}\right]$ strict local minimizer

Convex Optimization Problems

min $f(x)$
s.t. $x\in \Omega$

$\Omega :$ a convex set
$f:$ a convex function

Definition

Def: $\Omega :$ convex set, if $\forall x,y\in \Omega ,\forall \alpha \in (0,1):\alpha x+(1-\alpha )y\in \Omega$.

Def: The graph of $f:\Omega \to \mathbb{R}$ is a set of points in $\Omega \times \mathbb{R}\subseteq {\mathbb{R}}^{n+1}$ by $\{\left[\begin{array}{c}x\\ f(x)\end{array}\right]:x\in \Omega \}$

Def: The epigraph of $f$, denoted by $epi(f)$ is a set of points: $epi(f)=\{\left[\begin{array}{c}x\\ \beta \end{array}\right]:x\in \Omega ,\beta \in \mathbb{R},f(x)\le \beta \}$

Def: A function $f:\Omega \to \mathbb{R},\Omega \subseteq {\mathbb{R}}^n$ is convex on $\Omega$, if its epigraph is convex.

Lemma

Lem: If a function $f:\Omega \to \mathbb{R}$ is a convex on $\Omega$, then $\Omega$ is a convex set.

Theorem

Thm: A function $f:\Omega \to \mathbb{R}$ is convex, if and only if $\forall x,y\in \Omega ,\alpha \in (0,1):f(\alpha x+(1-\alpha )y)\le \alpha f(x)+(1-\alpha )f(y)$.

注：若把上式的小于等于号改成大于等于号，则$f$是凹函数( concave function）。

Lemma

Lem: Suppose $f,f_1,f_2$ are convex. Then, $\beta f$ for $\beta \ge 0$ is convex and so is $f_1+f_2$.

Example

f ( x ) = x 1 x 2 , Ω = { x : x 1 ≥ 0 , x 2 ≥ 0 } f(x)=x_1x_2,\Omega=\{x:x_1\geq 0,x_2\geq0\} f(x)=x1​x2​,Ω={x:x1​≥0,x2​≥0}

$x=\left[\begin{array}{c}1\\ 2\end{array}\right],y=\left[\begin{array}{c}2\\ 1\end{array}\right]$

$\alpha x+(1-\alpha )y=\left[\begin{array}{c}\alpha +2(1-\alpha )\\ 2\alpha +(1-\alpha )\end{array}\right]=\left[\begin{array}{c}2-\alpha \\ 1+\alpha \end{array}\right]$

$f(\alpha x+(1-\alpha )y)=(2-\alpha )(1+\alpha )=2+\alpha -{\alpha }^2$

$\alpha f(x)+(1-\alpha )f(y)=2\alpha +2(1-\alpha )=2$

$\because \forall \alpha \in (0,1),2+\alpha -{\alpha }^2>2$
$\therefore f$ is not a convex function

Theorem

Thm: Let $f:\Omega \to \mathbb{R}$ and $f\in C^1$. $\Omega$ is an open convex set. Then, $f$ is convex $\iff \forall x,y\in \Omega :f(y)\ge f(x)+Df(x)(y-x)$.

Theorem

$f\in C^2,\Omega :$ an open convex set.
$f$ convex $\iff \forall x\in \Omega :F(x)$ of $f$ at $x$ is positive semidefinite.

Example

1. $f(x)=-8x^2\Rightarrow F(x)--16<0$ (✕)
2. $f(x)=4x_1^2+3x_2^2+5x_3^2+6x_1{x}_2+x_1{x}_3-3x_1-3x_2+15$
   $F(x)=\left[\begin{array}{ccc}8& 6& 1\\ 6& 6& 0\\ 1& 0& 10\end{array}\right]$
   ${\Delta }_1=|8|>0$
   ${\Delta }_2=\left[\begin{array}{cc}8& 6\\ 6& 6\end{array}\right]>0$
   ${\Delta }_3=\left[\begin{array}{ccc}8& 6& 1\\ 6& 6& 0\\ 1& 0& 10\end{array}\right]=114>0$
   $\Rightarrow F(x)$ positive definite
3. $f(x)=2x_1{x}_2-x_1^2-x_2^2$
   $F(x)=\left[\begin{array}{cc}-2& 2\\ 2& -2\end{array}\right]$ (✕)

Definition

Def: strictly convex: $f(\alpha x+(1-\alpha )y)<\alpha f(x)+(1-\alpha )f(y)$

Def: (strictly) concave $\iff -f$ (strictly) convex

Theorem

Thm: convex optimization:
$x^{\ast }$ is global minimizer $\iff x^{\ast }$ is a local minimizer.

Lemma

Lem: $f:$ convex function on $\Omega$. Then, for all c ∈ R , Γ c = { x ∈ Ω : f ( x ) ∈ c } c\in\mathbb{R}, \Gamma_c=\{x\in\Omega:f(x)\in c\} c∈R,Γc​={x∈Ω:f(x)∈c} is convex.

Corollary

$f:$ convex function on $\Omega$. The set of all global minimizer of $f$ is convex.

Lemma

Lem: $f:$ convex function on $\Omega$. $f\in C^1$. If $x^{\ast }\in \Omega$ satisfies $\forall x\in \Omega ,x\ne x^{\ast }:Df(x^{\ast })(x-x^{\ast })\ge 0$, then $x^{\ast }$ is a global minimizer.

Theorem

Thm: $f:$ convex. $f\in C^1$
If $x^{\ast }\in \Omega$ satisfies $\forall d\in {\mathbb{R}}^n:d^T\nabla f(x^{\ast })\ge 0$, then $x^{\ast }$ is a global minimizer.

Corollary

If $x^{\ast }$ satisfies $\nabla f(x^{\ast })=0$, then $x^{\ast }$ global minimizer.

Theorem

Consider
min $f(x)$
s.t. $h(x)=0$

$h\in C^1$
Assume Ω = { x : h ( x ) = 0 } \Omega=\{x:h(x)=0\} Ω={x:h(x)=0} is convex, for example $Ax=b$.
$f:$ convex function on Ω = { x : h ( x ) = 0 } \Omega=\{x:h(x)=0\} Ω={x:h(x)=0}. If $x^{\ast }\in \Omega$ and ${\lambda }^{\ast }\in {\mathbb{R}}^m$ satisfy $Df(x^{\ast })+{\lambda }^{\ast }Dh(x^{\ast })=0$, then $x^{\ast }$ is a global minimizer.

Theorem

Consider
min $f(x)$
s.t. $h(x)=0$
$g(x)\le 0$

Assume: Ω = { x : h ( x ) = 0 , g ( x ) ≤ 0 } \Omega=\{x:h(x)=0,g(x)\leq 0\} Ω={x:h(x)=0,g(x)≤0}

Thm: If $x^{\ast }\in \Omega ,{\lambda }^{\ast }\in {\mathbb{R}}^m$ and ${\mu }^{\ast }\in {\mathbb{R}}^p$ satisfy
KKT$\{\begin{array}{l}{\mu }^{\ast }\ge 0\\ Df(x^{\ast })+{{\lambda }^{\ast }}^{T}Dh(x^{\ast })+{{\mu }^{\ast }}^{T}Dg(x^{\ast })=0\\ {{\mu }^{\ast }}^{T}g(x^{\ast })=0\end{array}$
then $x^{\ast }$ is a global minimizer.

Example

存钱问题：$x_k$表示第$k$月存入银行的钱，银行月利率为$r$，初始银行账户为0，存入的钱总共不超过D，求怎样存钱使得$n$月后账户余额最多。

max $y_n=(1+r{)}^n{x}_1+(1+r{)}^{n-1}{x}_2+(1+r)x_n$
s.t. ${\sum }_{i=1}^n{x}_i\le D$
$x\ge 0$

$\{\begin{array}{l}{\mu }_1(e^T-D)=0,{\mu }_1\ge 0\\ {\mu }_{2}x=0\\ e^{T}x\le D\\ x\ge 0\\ [(1+r{)}^n,(1+r{)}^{n-1},\cdots ,1]+{\mu }_{1}e-{\mu }_2=0\end{array}$

${\mu }_1=(1+r{)}^n,{\mu }_2=(1+r{)}^{n}e-c$
$x_1=D,x_2=\cdots =x_n=0\Rightarrow$ global opt.

注：该问题从目标函数到限制条件全是线性函数，线性函数显然也是凸函数。于是该问题虽然是线性规划问题，但也可以用凸优化的方法来解。

总结

本文先讨论了非线性优化问题，再讨论了凸优化问题。在非线性优化问题中，介绍了FONC，SONC和SOSC。并给出了两个例子，介绍了求解非线性优化问题的思路，先通过KKT条件（FONC），求出可能的极值点，再用SONC和SOSC来验证，以此严格说明求出的点是或不是极值点。在凸优化问题部分，先介绍了凸的概念。从图的角度引入了凸函数的概念，并给出了其等价定义。最后给出了一系列定理，证明了KKT条件不仅是凸优化问题的必要条件，也是充分条件。于是，求解凸优化问题，只需要用KKT条件求解即可。