1. Chapter 13 Gaussian Elimination (III) Bunch-Parlett diagonal pivoting
Speaker: Lung-Sheng Chien
Reference: James R. Bunch and Linda Kaufman, Some Stable Methods for Calculating Inertia and Solving Symmetric Linear Systems, Mathematics of Computation, volume 31, number 137, January 1977, page

2. OutLine
Preliminary - 1x1, 2x2 pivoting in Gaussian Elimination - real symmetric indefinite matrix
Symmetric permutation
LDL’ decomposition (diagonal pivoting)
Example of complete diagonal pivoting
Algorithm of complete diagonal pivoting

3. Pivoting in Traditional Gaussian Elimination
consider
then one step LU-factorization leads to
where
and
It is easy to show
from Gaussian Elimination
We use principal submatrix
as pivoting, called
pivoting since
Question: can we use
or higher
pivoting, i.e.

4. 2x2 pivoting in block Gaussian Elimination
If principal sub-matrix
is non-singular,
then one-step GE leads to
Example 1:
pivoting
where
and

5. 3x3 pivoting in block Gaussian Elimination
Example 2:
pivoting
where
and
Question: what is criterion to choose 1x1, 2x2 or 3x3 pivoting?
stability
performance
Question: what is the cost to estimate the criterion ?

6. Real symmetry indefinite matrix A
is symmetric implies
and
From linear algebra, if A is real symmetric, then A has real spectrum decomposition (譜分解)
and
or write in matrix form
and
without loss of generality, we can rearrange eigen-values to be increasing
Definition: suppose matrix A is real symmetric, then we say “A is indefinite” if there exists an eigen-value of A less than zero. If A is nonsingular, then
Thereafter we focus on LU-decomposition of symmetry indefinite matrix A

7. LU-factorization for real symmetry indefinite matrix A
where
and
Question: why not LU-decomposition?
A is real symmetric, we only store lower triangle part of A, say

8. Pointer array to store lower triangular part of A
row-major based
col-major based
Question: what is the cost to fetch one element of A ? That is, operation count for
Question: can you find another representation for lower part of matrix A ?
Question: if one uses row-major to store lower part of matrix A, then how to fetch a column of A efficiently?

9. OutLine
Preliminary
Symmetric permutation - change diagonal element to (1,1) position - change off-diagonal element to (2,1) position - implementation of symmetric permutation
LDL’ decomposition (diagonal pivoting)
Example of complete diagonal pivoting
Algorithm of complete diagonal pivoting

10. How to do permutation on real symmetry indefinite matrix A
Let
Define permutation matrix
interchange row 2, 3
interchange column 2, 3
Symmetric permutation: 1 2 3 1 2 3 1 3 2
interchange
row 2, 3
interchange
column 2, 3 2 4 5 3 5 6 3 6 5 3 5 6 2 4 5 2 5 4

11. Change diagonal element to (1,1) position [1]
suppose we want to change
to position (1,1) , then consider permutation
and do symmetric permutation on A, say 8 6 3 9 1 2 3 4 3 2 1 4 6 5 2 7 2 5 6 7
interchange
col 1, 3 6 5 2 7
interchange
row 1, 3 3 2 1 4 3 6 8 9 8 6 3 9 9 7 4 10 4 7 9 10 9 7 4 10
Question: we only store lower triangle part of matrix A, above permutation does not work, how to modify it?
Observation 1: 1 2 3 4 8 2 3 4 2 5 6 7 2 5 6 7 3 6 8 9 3 6 1 9 4 7 9 10 4 7 9 10

12. Change diagonal element to (1,1) position [2]
Observation 2: we only need to update lower triangle part of A (diagonal term is excluded ). 8 2 3 4 2 5 6 7 2 3 6 1 9 6 4 7 9 10 4 7 9
diagonal term
does not changed
we have changed
does not changed

13. Change diagonal element to (1,1) position [3]6 2 2 6 2 2 6
interchange
col 1, 3 6 2
interchange
row 1, 3 2 6 6 9 7 4 4 7 9 9 7 4
Keep lower triangle part 8 8 6 5
Fill-in A(3,1) 6 5
Fill-in
diagonal term 6 3 2 1 2 1 2 9 7 4 10 9 7 4 10 9 7 4
Question: any compact form to represent above interchange?

14. Change diagonal element to (1,1) position [4]2 6 2 6 6 2 4 7 9 9 7 4 4 7 9
case 1: 4 7 9
case 2: 2 6

15. Change diagonal element to (1,1) position [5]
For general real symmetric matrix A with dimension n, suppose we want to change
interchanges rows or columns
Step 1: interchange position (1,1) and (k,k), i.e.
keep position (k,1) and (1,k), i.e.
does not changed

16. Change diagonal element to (1,1) position [6]
Step 2: interchange column 1 and k below row k+1, i.e
Therefore

17. Change diagonal element to (1,1) position [7]
Step 3: interchange column 1 (from row-2 to row-(k-1)) and row k (from col-2 to col-(k-1))
Therefore

18. Implementation of symmetric permutation: swapping overhead [1]
Suppose we use col-major based pointer array to store real symmetric matrix A
Step 2: interchanging column 1 and k is O.K
since memory block is contiguous.
Contiguous memory block

19. Implementation of symmetric permutation: swapping overhead [2]
Step 3: interchanging column 1 and row k, i.e.
is NOT efficient since
is NOT contiguous.
NOT contiguous, swapping is very slow.
Observation: the situation is the same even we choose row-major based pointer array as storage strategy.
Solution: in order to avoid high swapping overhead, we should avoid permutation.

20. Change off-diagonal element to (2,1) position [1]
suppose we want to change
to position (2,1), then consider first permutation
which change first index from 4 to 2 1 4 3 2 1 2 3 4 1 4 3 2 4 10 9 7 2 5 6 7
interchange
col 2, 4 2 7 6 5
interchange
row 2, 4 3 9 8 6 3 6 8 9 3 9 8 6 2 7 6 5 4 7 9 10 4 10 9 7
and second permutation
which change second index from 3 to 1 1 4 3 2 3 4 1 2 8 9 3 6 4 10 9 7
interchange
col 1, 3 9 10 4 7
interchange
row 1, 3 9 10 4 7 3 9 8 6 8 9 3 6 3 4 1 2 2 7 6 5 6 7 2 5 6 7 2 5

21. Change off-diagonal element to (2,1) position [2]3 4 1 4 3 2 8 9 3 6 2 5 6 7
interchange
row and col 2, 4 4 10 9 7
interchange
row and col 1, 3 9 10 4 7 3 6 8 9 3 9 8 6 3 4 1 2 4 7 9 10 2 7 6 5 6 7 2 5
Observation: two permutation matrices can be computed easily since 2  4 and  3 are independent.

22. Change off-diagonal element to (2,1) position [3]
HOW to transform
to position (2,1), under only lower triangle part of A is stored?
Observation 1:
and
does not changed 1 2 3 4 1 4 3 2 2 5 6 7
interchange
row/col 2, 4 4 10 9 7 3 6 8 9 3 9 8 6 4 7 9 10 2 7 6 5
Step 1: interchange position (4,4) and (2,2), 1 2 3 4 1 2 3 4 2 5 6 7 2 10 6 7 3 6 8 9 3 6 8 9 4 7 9 10 4 7 9 5

23. Change off-diagonal element to (2,1) position [4]
Observation 2: we only need to update lower triangle part of A (diagonal term is excluded ). 1 2 3 4 2 10 6 7 2 3 6 8 9 3 6 4 7 9 5 4 9
Step 2: interchange row 2 and 4 below col 2, i.e 2
interchange
row 2, 4 4 9
interchange
col 2, 4 4 9 3 6 3 6 3 9 6 4 9 2 6 2 6
NOTE: Interchanging column 2, 4 does not change position (2,1) and (4,1), it suffices to interchange position (2,1) and (4,1) first.

24. Change off-diagonal element to (2,1) position [5]4 3 6 3 6 4 9 2 9
Step 3: interchange column 2 (from row-3 to row-3) and row 4 (from col-3 to col-3) 4 4 3 6 3 9 2 9 2 6

25. Change off-diagonal element to (2,1) position [6]
For general real symmetric matrix A with dimension n, suppose we want to change
interchanges rows or columns
Step 1: interchange position (2,2) and (k,k), i.e.
keep position (k,2) and (2,k), i.e.
does not changed

26. Change off-diagonal element to (2,1) position [7]
Step 2: interchange column 2 and k below row k+1, i.e
Therefore

27. Change off-diagonal element to (2,1) position [8]
Step 3: interchange row 2 (column 1) and row k (column1), i.e
Therefore

28. Change off-diagonal element to (2,1) position [9]
Step 4: interchange column 2 (from row-3 to row-(k-1)) and row k (from col-3 to col-(k-1))
Therefore

29. OutLine Preliminary Symmetric permutation
LDL’ decomposition (diagonal pivoting) - growth rate of 1x1, 2x2 pivoting - criterion for pivot strategy - comparison to complete pivoting in GE
Example of complete diagonal pivoting
Algorithm of complete diagonal pivoting

30. LDL’ decomposition: 1x1, 2x2 pivoting
diagonal pivoting method with complete pivoting: Bunch-Parlett, “Direct methods fro solving symmetric indefinite systems of linear equations,” SIAM J. Numer. Anal., v. 8, 1971, pp
diagonal pivoting method with partial pivoting: Bunch-Kaufman, “Some Stable Methods for Calculating Inertia and Solving Symmetric Linear Systems,” Mathematics of Computation, volume 31, number 137, January 1977, page

31. Growth rate in LDL’ decomposition [1]
is of order
maximal element of matrix A
Define
and constant
maximal diagonal element of matrix A
Case 1: s = 1
where
with
with
and
with

32. Growth rate in LDL’ decomposition [2]
Therefore
Observation: in PA=LU, we choose
such that
implies
Therefore
Observation: large element growth will not occur for a
pivot if
is large relative to

33. Growth rate in LDL’ decomposition [3]
Case 2: s = 2
where
and
with
and
with
and
with
implies

34. Growth rate in LDL’ decomposition [4]
for
Therefore

35. Growth rate in LDL’ decomposition [5]
Therefore

36. Criterion for pivot strategy [1]
Fix a number
( later on, we will determine optimal value of )
maximal element of matrix A
maximal diagonal element of matrix A
Case 1:
suppose
we interchange 1st and k-th rows and columns by introducing
permutation matrix
and do symmetric permutation
Then do on
with
pivot, written as

37. Criterion for pivot strategy [2]
Recall for
pivot,
, we have growth rate
and
Now for
and
Case 2:
suppose
, we interchange q-th and 1st rows and columns,
and then r-th and 2nd rows and columns by introducing permutation matrix
and do symmetric permutation
Question: we must change q1 first, then change r 2, why?

38. Criterion for pivot strategy [3]
transforms 1 2 3
(boundary case)
but

39. Criterion for pivot strategy [4]
Then do on
with
pivot, written as
Recall for
pivot,
, we have growth rate
and

40. Criterion for pivot strategy [5]
Now for
To sum up
Case 1:
and
Case 2:
and
Question: How to choose value of

41. Criterion for pivot strategy [6]
worst case analysis : choose
such that
growth rate of
pivot +
pivot
growth rate of
pivot
or equivalently
growth rate of
pivot
growth rate of
pivot
Define
where
satisfies
Exercise: verify
and

42. Diagonal pivoting versus complete pivoting of Gaussian Elimination [1]
We must search for the largest element in each reduced matrix, this is a complete pivoting strategy analogous to Gaussian Elimination with complete pivoting 1
maximal element of matrix A
maximal diagonal element of matrix A
complete pivoting of GE (Gaussian Elimination)
(1) find a
such that
(2) swap row
and row
(2) swap column
and column
then
with

43. Diagonal pivoting versus complete pivoting of Gaussian Elimination [2]
growth rate 2
diagonal pivoting
complete pivoting in GE
Remark: Bunch [1] proves that the element growth in the diagonal pivoting method
with complete pivoting is bounded by
in comparison with
for Gaussian Elimination with complete pivoting, where
[1] J.R. Bunch, “Analysis of the diagonal pivoting method”, SIAM J. Numer. Anal., v. 8, 1971, pp

44. OutLine Preliminary Symmetric permutation
LDL’ decomposition (diagonal pivoting)
Example of complete diagonal pivoting
Algorithm of complete diagonal pivoting

45. Example (complete pivoting) [1]6 12 3 -6 12 -8
-13 4
we choose
pivot 3
-13 -7 1
swap row/column
and
by permutation matrix -6 4 1 6 6 -8 -8
-13 12 4 12 -8 12 6
-13 -7 3 1 3
-13 -7
-13 3 -7 12 3 6 -6 -6 4 1 6 4 -6 1 6 4 1 -6 6

46. Example (complete pivoting) [2]-8
-13 12 4 1 -8
-13
-13 -7 3 1 1
-13 -7 12 3 6 -6
0.3982 1
4.7257 4 1 -6 6
0.1327 1
5.8584
Recursively do the same procedure for
we choose
pivot
swap row/column
by permutation matrix
such that

47. Example (complete pivoting) [3]
Do symmetry permutation for
and -8
-13 -8
-13
-13 -7
-13 -7
4.7257
5.8584
5.8584
4.7257 1 1 1 1
0.3982 1
0.1327 1
0.1327 1
0.3982 1
5.8584
we do
pivot on
4.7257

48. Example (complete pivoting) [4]
5.8584 1
5.8584 1
4.7257 1 1
Or write in original matrix form -8
-13 1 -8
-13
-13 -7 1
-13 -7
5.8584 1
5.8584
4.7257 1

49. Example (complete pivoting) [5]1 -8
-13 1
-13 -7
0.1327 1
5.8584
0.3982 1
In practice, we have two key issues
col-major based 1
we only store lower part of matrix 6 6 -8 -7 6 12 -8 12
-13 1 3
-13 -7 3 4 -6 4 1 6 -6

50. Example (complete pivoting) [6]2
we store decomposition
and
into original 1 -8 1
-13 -7
0.1327 1
5.8584
0.3982 1 -8
-13 -7
0.1327
5.8584
0.3982
Question: How can you judge correct decomposition
from

51. Example (complete pivoting) [7]
Case 1: four 1x1 pivot 1 -8 13 1 -7
0.1327 1
5.8584
0.3982 1
Case 2: two 2x2 pivot 1 -8 1
-13 -7
0.1327 1
5.8584
0.3982 1

52. Example (complete pivoting) [8]
Case 3: 1x1 pivot + 2x2 pivot + 1x1 pivot -8 1 -7
-13 1
5.8584
0.1327 1
0.3982 1
Case 4: 2x2 pivot + 1x1 pivot + 1x1 pivot 1 -8 1
-13 -7
0.1327 1
5.8584
0.3982 1
Solution: we need an array to record pivot sequence
1x1 pivot
pivot 2 1 1
2x2 pivot

53. OutLine Preliminary Symmetric permutation
LDL’ decomposition (diagonal pivoting)
Example of complete diagonal pivoting
Algorithm of complete diagonal pivoting

54. Algorithm ( PAP’ = LDL’ ) [1]
Given symmetric indefinite matrix
, construct initial lower triangle matrix
use permutation vector
to record permutation matrix
let ,
and
while
we have compute
update original matrix
, where
combines all lower triangle matrix and store in

55. Algorithm ( PAP’ = LDL’ ) [2]1
compute
and
Case 1:
define permutation
to do symmetric permutation 2
To compute
, we only update lower triangle of 1 1 3 2 3 3
then 2

56. Algorithm ( PAP’ = LDL’ ) [3]
To compute 4
We only update lower triangle matrix
then 5
do 1x1 pivot :
then 6
and

57. Algorithm ( PAP’ = LDL’ ) [4]
Case 2:
define permutation
to interchange q-th and k-th rows and columns,
and then r-th and (k+1)-th rows and columns 7
and then
To compute
, we only update lower triangle of
(1) do interchange row/col 1 1 8 2 3 3
then 2

58. Algorithm ( PAP’ = LDL’ ) [5]9
(2) do interchange row/col 1 2 3 4 1 3 4
then
where 2

59. Algorithm ( PAP’ = LDL’ ) [6]
To compute 10
(1) do interchange row 11
(2) do interchange row
then

60. Algorithm ( PAP’ = LDL’ ) [7]12
do 2x2 pivot :
then 13
and
means
is 2x2 pivot
endwhile

61. Question: recursion implementation
Initialization - check algorithm holds for k=1
Recursion formula - check algorithm holds for k or k+1, if k-1 is true
Termination condition - check algorithm holds for k=n-1
Breakdown of algorithm - check which condition PAP’=LDL’ fails
No extension: algorithm works only for square, symmetric indefinite matrix.
normal
abnormal
Extension of algorithm

62. MATLAB implementation [1]
Given a (full) symmetric indefinite matrix
, compute factorization
return four quantities
Remark: try to neglect upper triangle part of
in MATLAB implementation
Example : 6 12 3 -6 12 -8
-13 4 2 3 4 1
return : 3
-13 -7 1 2 1 1 -6 4 1 6 1 -8 1
-13 -7
0.1327 1
5.8584
0.3982 1

63. MATLAB implementation [2]
When factorization is complete,
, you need to provide linear solver 1
define
, then
by forward substitution 2
define
, then
by diagonal block inversion
Example:
, then 3
define
, then
by backward substitution
However you cannot transpose
explicitly in MATLAB 4
, you must scan
once to determine