Linear Programming Learning Notes (5) Duality Theory

All the resources come from Linear Programming: Foundations and Extensions by Professor Robert J. Vanderbei.
Explore the link below for further information:
LP Book Resources

Part 1

Basic Theory: The Simplex Method and Duality

Chapter 5 Duality Theory

Associated with every linear program is another called its dual. The dual of this dual program is the original linear program.
Hence, linear programs come in primal/dual pairs. It turns out that every feasible solution for one of these two linear programs gives a bound on the optimal objective function value for the other. Thes ideas form a subject called duality theory.

Movitation: Finding Upper Bounds

Let’s begin with an example:
maximize        4x1+x2+3x3
s.t.                   x1+4x2≤1
                        3x1−x2+x3≤3
                                 x1,x2,x3≥0
Every feasible solution provides a lower bound on the optimal objective function value ζ∗. Let’s multiply the two constraints by nonnegative numbers, y1 and y2, respectively, to get:
y1(x1+4x2)≤y1
+y2(3x1−x2+x3)≤3y2
(y1+3y2)x1+(4y1−y2)x2+(y2)x3≤y1+3y2¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
If we stipulate that each of the coefficients of the xi’s be at least as large as the corresponding coefficient in the objective funcion,
y1+3y2≥4
4y1−y2≥1
y2≥3
so that :
ζ≤y1+3y2
Now we have the upper bound, which we should minimize in our effort to obtain the best possible upper bound. Therefore, we are naturally led to the following optimization problem:
minimize        y1+3y2
s.t.                    y1+3y2≥4
                         4y1−y2≥1
                              4y2≥3
                                   y1,y2≥0
This problem is called the dual linear programming problem associated with the given one.

The Dual Problem

Given a linear programming problem in standard form,
maximize        ∑nj=1cjxj
s.t.                   ∑nj=1aijxj≤bi,i=1,2,...,m
                                        xj≥0,j=1,2,...,m
the associated dual linear program is given by
minimize        ∑mi=1biyi
s.t.                   ∑mi=1aijyi≥cj,j=1,2,...,n
                                        yi≥0,i=1,2,...,n
and the standard form of the dual is
-maximize        ∑mi=1(−bi)yi
s.t.                   ∑mi=1(−aij)yi≤(−cj),j=1,2,...,n
                                        yi≥0,i=1,2,...,n

The Weak Duality Theorem

If (x1,x2,...,xn) is feasible for the primal and (y1my2,...,ym) is feasible for the dual, then
∑jcjxj≤∑ibiyi

The Strong Duality Theorem

If the primal problem has an optimal solusion,
x∗=(x∗1,x∗2,...,x∗n)
then the dual also has an optimal solution,
y∗=(y∗1,y∗2,...,y∗m)
such that
∑jcjx∗j=∑ibiy∗i
The main idea is that, as the simplex method solves the primal problem, it also implicitly solves the dual problem, and it does so in a way that the theorem holds.

We should notice here that, the dual dictionary is the negative transpose of the primal dictionary. For example,
ζ=4x1+x2+3x3−−−−−−−−−−−−−−− …………………(P)
w1=1−x1−4x2
w2=3−3x1+x2−x3
.
−ζ=−y1−3y2−−−−−−−−−−−−− ……………………….(D)
z1=−4+y1+3y2
z2=−1+4y1−y2
z3=−3+y2
It can be written as:
⎡⎣⎢0134−1−31−4130−1⎤⎦⎥ −→−−−−−−−neg.−transpo. ⎡⎣⎢⎢⎢0−4−1−3−1140−33−11⎤⎦⎥⎥⎥
Each primal dictionary generated by the simplex method implicited defines a corresponding dual dictionary as follows: first write down the negative transpose and then replace each xj with a zj and each wi with a yi. As long as the primal dictionary is not optimal, the implicitly defined dual dictionary will be infeasible. But once an optimal primal dictionary is found, the corresponding dual dictionary will be feasible.

Complementary Slackness

Sometimes it is necessary to recover an optimal dual solution when only an optimal primal solution is known. The following theorem, known as the Complementary Slackness Theorem, can help in this regard.
THEOREM Suppose that x=(x1,x2,...,xn) is primal feasible and that y=(y1,y2,...,ym) is dual feasible. Let(w1,w2,...,wm) denote the corresponding primal slack variables, and let(z1,z2,...,zn) denote the corresponding dual slack variables. Then x and y are optimal for their respective problems if and only if:
xjzj=0, for j=1,2,...,n
wiyi=0, for i=1,2,...,m
And below is how we recover the dual solution. Suppose that we have a nondegenerate primal basic optimal solution
x∗=(x∗1,x∗2,...,x∗n)
and we wish to find a corresponding optimal solution for the dual. Let
w∗=(w∗1,w∗2,...,w∗m)
denote the corresponding slack variables, which can be obtained as:
w∗i=bi−∑jaijx∗j
The dual constraints are
∑iyiaij−zj=cj,j=1,2,...,n
These constraints form n equations in m+n unknowns. Since the primal solution is assumed to be nondegenerate, it follows that the m basic variables will be strictly positive. The complementary slackness theorem then tells us that the corresponding dual variables must vanish. Hence we are then left with just n equations in n unknowns. If there is a unique solution, all the components should be nonnegative, or it would stand in contradiction to the assumed optimality of x∗.

The Dual Simplex Method

We use this method when the primal dictionary is not feasible while the dual is feasible. When we just focus on the primal, we choose the entering and leaving variables following the rules below:
1. Select the basic variable whose constant value is the most negative.
2. Pick the entering variable by scanning the row of this basic variable, looking for the largest negated ratio.
For example:
ζ=−x1−x2−−−−−−−−−−−
w1=4+2x1+x2
w2=−8+2x1−4x2
w3=−7+x1−3x2
First we select w2 as the leaving variable, since -8 is the most negative. Then we pick x1 as the entering variable, since 2/(−1)=−2 is more negative than (−4)/(−1)=4.
Finally we can get an optimized dictionary as:
ζ=−7−w3−4x2−−−−−−−−−−−−−−−
w1=18+2w3+7x2
x1=7+w3+3x2
w2=6+2w3+2x2

A Dual-Based Phase I Algorithm

At least it is more elegant than the Phase I before. Consider the following example:
ζ=−x1+4x2−−−−−−−−−−−−
w1=4+2x1+x2
w2=−8+2x1−4x2
w3=−7+x1−3x2
Now, neither the primal and the dual is feasible. Let us temporarily change the primal objective function to
η=−x1−x2
Then the corresponding initial dual dictionary is feasible. In fact, it is the same dictionary as that above. The optimal is
ζ=−7−w3−4x2−−−−−−−−−−−−−−−
w1=18+2w3+7x2
x1=7+w3+3x2
w2=6+2w3+2x2
Now we simply reinstate the intended objective function and continue with Phase II.
ζ=−x1+4x2=−(4+w3+3x2)+4x2=−7−w3+x2
Hence, the starting dictionary for Phase II is
ζ=−7−w3+x2−−−−−−−−−−−−−−
w1=18+2w3+7x2
x1=7+w3+3x2
w2=6+2w3+2x2

Rules of Forming the Dual

Primal Dual Equality constraint Free variable Inequality constraint Nonnegative variable Free variable Equality constraint Nonegative variable Inequality constraint

Lagrangian Duality

π(x1,...,xn,y1,...,ym)=∑jcjxj−∑i∑jyiaijxj+∑iyibi
maxx≥0miny≥0π(x,y)=miny≥0maxx≥0π(x,y)
The max-min is the primal, and the min-max is the dual.