Data Intensive and Cloud Computing Matrices and Arrays Lecture 9

Matrices and Arrays in Python: NumPy We’ve already seen lists – single-dimensional data with a single type More generically, we may want to have matrices (as in PageRank) or multidimensional arrays This leads to numpy – support for numeric operations in Python A building block for Pandas And for SciPy, which is scientific programming support

What Do We Use Matrices / Multi-Dimensional Arrays for? Graph adjacency matrices, and variants thereof Rows of readings of sets of numeric values (e.g., a suite of sensors) Machine learning features (later in the semester!) Images, volumes, numeric state values in a multidimensional space

5 Important Concepts from Linear Algebra Help! My matrix is too big to analyze by inspection. I should compute: Frobenius Norm Rank Determinant Singular Vectors / Singular Values Eigenvectors / Eigenvalues

Basic concepts Vector in Rn is an ordered set of n real numbers. e.g. v = (1,6,3,4) is in R4 A column vector: A row vector: m-by-n matrix is an object in Rmxn with m rows and n columns, each entry filled with a (typically) real number:

Basic concepts Vector norms: A norm of a vector ||x|| is informally a measure of the “length” of the vector. Common norms: L1, L2 (Euclidean) Linfinity

Basic concepts Use lower case letters for vectors. The elements are referred by xi. Vector dot (inner) product: Vector outer product: If u•v=0, ||u||2 != 0, ||v||2 != 0  u and v are orthogonal If u•v=0, ||u||2 = 1, ||v||2 = 1  u and v are orthonormal

Basic concepts e.g. Matrix product: Use upper case letters for matrices. The elements are referred by Ai,j. Matrix product: e.g.

Special matrices diagonal upper-triangular lower-triangular tri-diagonal I (identity matrix)

Basic concepts e.g. Transpose: You can think of it as “flipping” the rows and columns OR “reflecting” vector/matrix on line e.g.

Linear independence e.g. A set of vectors is linearly independent if none of them can be written as a linear combination of the others. Vectors v1,…,vk are linearly independent if c1v1+…+ckvk = 0 implies c1=…=ck=0 e.g. (u,v)=(0,0), i.e. the columns are linearly independent. x3 = −2x1 + x2

Span of a vector space e.g. If all vectors in a vector space may be expressed as linear combinations of a set of vectors v1,…,vk, then v1,…,vk spans the space. The cardinality of this set is the dimension of the vector space. A basis is a maximal set of linearly independent vectors and a minimal set of spanning vectors of a vector space (0,0,1) (0,1,0) (1,0,0) e.g.

Determinants A determinant of a matrix A is denoted by |A|. The determinant of a 22 matrix: The determinant of a 33 matrix:

The determinant of an nn matrix: The minor of aij, denoted by Aij, is the matrix after removing row i and column j. The determinant of an nn matrix: The general expression for the determinant of an nn matrix:

Example: Matrix Determinant with the first row and their minors:

with the second column and their minors: Since |A|=0, A is a singular matrix; that is the inverse of A does not exist.

If a matrix A has a zero determinant, then A is a singular matrix; that is, the inverse of A does not exist.

Rank of a Matrix rank(A) (the rank of a m-by-n matrix A) is The maximal number of linearly independent columns =The maximal number of linearly independent rows =The dimension of col(A) =The dimension of row(A) If A is n by m, then rank(A)<= min(m,n) If n=rank(A), then A has full row rank If m=rank(A), then A has full column rank

Rank– Many Equivalent Definitions A matrix of r rows and c columns is said to be of order r by c. If it is a square matrix, r by r, then the matrix is of order r. The rank of a matrix equals the order of highest-order nonsingular submatrix.

Example 1: Rank of Matrix 3 square submatrices: Each of these has a determinant of 0, so the rank is less than 2. Thus the rank of R is 1.

Example 2: Rank of Matrix Since |A|=0, the rank is not 3. The following submatrix has a nonzero determinant: Thus, the rank of A is 2.

Compute Rank of Large Matrices Theorem 1 Row-Equivalent Matrices Row-equivalent matrices have the same rank. We call a matrix A1 row-equivalent to a matrix A2 if A1 can be obtained from A2 by (finitely many!) elementary row operations.

EXAMPLE: Computing Rank

EXAMPLE Computing Rank (continued) The last matrix is in row-echelon form and has two nonzero rows. Hence rank A = 2.

Inverse of a matrix Inverse of a square matrix A, denoted by A-1 is the unique matrix s.t. AA-1 =A-1A=I (identity matrix) If A-1 and B-1 exist, then (AB)-1 = B-1A-1, (AT)-1 = (A-1)T For orthonormal matrices For diagonal matrices

Python Support for Matrices

Frobenius Norm A metric for overall “value” of the matrix (sum of squared values). numpy.linalg.norm

Rank The number of nonzero rows after a row reduction. Yet another definition numpy.linalg.matrix_rank

Determinant A metric describing the overall “scaling factor” of the matrix. How much would multiplying by this matrix stretch the unit box? numpy.linalg.det

Singular vectors / Singular Values Start with the vector that gets stretched most by this matrix (and how much). Then, find independent others in descending order. numpy.linalg.svd

Eigenvectors / Eigenvalues The special vectors that this matrix can only scale (and by how much). numpy.linalg.eig

NumPy - Details

What is NumPy? NumPy is the fundamental package needed for scientific computing with Python. It contains: a powerful N-dimensional array object basic linear algebra functions basic Fourier transforms sophisticated random number capabilities tools for integrating Fortran code tools for integrating C/C++ code

NumPy documentation Official documentation The NumPy book Example list http://docs.scipy.org/doc/ The NumPy book http://www.tramy.us/numpybook.pdf Example list http://www.scipy.org/Numpy_Example_List_With_Doc

The ndarray data structure NumPy adds a new data structure to Python – the ndarray An N-dimensional array is a homogeneous collection of “items” indexed using N integers Defined by: the shape of the array, and the kind of item the array is composed of

Numpy – ndarray attributes ndarray.ndim the number of axes (dimensions) of the array i.e. the rank. ndarray.shape the dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the rank, or number of dimensions, ndim. ndarray.size the total number of elements of the array, equal to the product of the elements of shape. ndarray.dtype an object describing the type of the elements in the array. One can create or specify dtype's using standard Python types. NumPy provides many, for example bool_, character, int_, int8, int16, int32, int64, float_, float8, float16, float32, float64, complex_, complex64, object_. ndarray.itemsize the size in bytes of each element of the array. E.g. for elements of type float64, itemsize is 8 (=64/8), while complex32 has itemsize 4 (=32/8) (equivalent to ndarray.dtype.itemsize). ndarray.data the buffer containing the actual elements of the array. Normally, we won't need to use this attribute because we will access the elements in an array using indexing facilities.

Array shape ndarrays are rectangular The shape of the array is a tuple of N integers (one for each dimension)

Array item types Every ndarray is a homogeneous collection of exactly the same data-type every item takes up the same size block of memory each block of memory in the array is interpreted in exactly the same way

Arrays and Matrices in NumPy 3 wide x 2 high

Simple Creation (Borrowed from Matlab)

Uninitialized Matrices – Fast But Full of Garbage

Numpy – arrays, matrices For two dimensional arrays NumPy defined a special matrix class in module matrix. Objects are created either with matrix() or mat() or converted from an array with method asmatrix(). >>> import numpy >>> m = numpy.mat([[1,2],[3,4]]) or >>> a = numpy.array([[1,2],[3,4]]) >>> m = numpy.mat(a) >>> m = numpy.asmatrix(a) Note that the statement m = mat(a) creates a copy of array 'a'. Changing values in 'a' will not affect 'm'. On the other hand, method m = asmatrix(a) returns a new reference to the same data. Changing values in 'a' will affect matrix 'm'. 42

An Example of Using a Graph Adjacency Matrix

Graph Adjacency Matrices in Numpy 1 Graph G 2 3 1

Recall the PageRank Linear Algebra formulation Create an m x m matrix M to capture links: M(i, j) = 1 / nj if page i is pointed to by page j and page j has nj outgoing links = 0 otherwise Initialize all PageRanks to 1, multiply by M repeatedly until all values converge: Officially computes principal eigenvector via power iteration

PageRank Example in Python 0.5 1 Google g y a 0.15 = 0.85 * + Amazon Yahoo

PageRank in Spark? Spark has a Matrix construct built over RDDs The challenge: how distribute/shard the matrix? RowMatrix – arbitrarily distributed rows. e.g., machine learning features, document vectors, … IndexedRowMatrix – rows have a numeric (long) key CoordinateMatrix – cells are (row, column, value) BlockMatrix – matrix broken into “tiles” with coordinates In general: these don’t offer strong advantages over using DataFrames with edge lists... so you are better off doing PageRank using joins!

A Sketch of PageRank with Joins Given dataframe edge(from, to) Compute a dataframe transfer_weight(node, ratio) for each node Initialize pagerank(node,score) for each node for n iterations: Compute weight_from_edge(from, to, weight) for each node, given existing pagerank(from, score) and transfer_weight(from, ratio) Sum up weight_from_edge(from, to, weight) for each to – this is the new PageRank for the next iteration

Broader Matrix Manipulation University of Pennsylvania

Matrices Have “Bulk Operations” too Addition and multiplication with scalars Simple arithmetic over arrays / matrices (add, multiply) Linear algebra operations – multiply, transpose, … “Slicing” Many of these are key building blocks for scientific computing

Array Math Basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions:

Array Math – Cont’d

Numpy – matrices Operator *, dot(), and multiply(): For array, '*' means element-wise multiplication, and the dot() function is used for matrix multiplication. For matrix, '*'means matrix multiplication, and the multiply() function is used for element-wise multiplication. Handling of higher-rank arrays (rank > 2) array objects can have rank > 2. matrix objects always have exactly rank 2. Convenience attributes array has a .T attribute, which returns the transpose of the data. matrix also has .H, .I, and .A attributes, which return the conjugate transpose, inverse, and asarray() of the matrix, respectively. Convenience constructor The array constructor takes (nested) Python sequences as initializers. As in array([[1,2,3],[4,5,6]]). The matrix constructor additionally takes a convenient string initializer. As in matrix("[1 2 3; 4 5 6]") 53

Multiplication

Computations on Arrays

Aggregation over Arrays Standard “averaging” measures – arr.mean(), median(), mode() Distributional info – std() (standard deviation) and var() (variance) min(), max() argmin(), argmax()

Printing Arrays When you print an array, NumPy displays it in a similar way to nested lists, but with the following layout: the last axis is printed from left to right, the second-to-last is printed from top to bottom, the rest are also printed from top to bottom, with each slice separated from the next by an empty line.

Shape Manipulation Changing the shape of an array An array has a shape given by the number of elements along each axis:

Changing Shape The shape of an array can be changed with various commands. The following commands return a modified array, but do not change the original array:

Reshape versus Resize The reshape function returns its argument with a modified shape, whereas the resize method modifies the array itself:

Slicing Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify a slice for each dimension of the array:

Slicing – cont’d When fewer indices are provided than the number of axes, the missing indices are considered complete slices:

An Example Application: Image Processing How does an image get represented?

Array “Slicing” – Projection or Cropping

Beware: Numpy Slices Are NOT Copies By default, a “slice” points to the data in the original array! More efficient, but it means you need to be careful about changing the slice i.e., if I change values in crop_face, I also change face If you’re going to modify the values, call array.copy()

Selecting Parts of an Array in Numpy

Recall How Rows Were Selected in Pandas

Selecting by a Boolean List

A Test on a DataFrame Produces a Boolean Series (List)

Row Selection in an Array (And Populating a Random Array)

“Fancy Indexing” (Selecting by Index) Sometimes I want to choose a list of array components in a particular order… University of Pennsylvania

Arithmetic on a Numpy Array Exploits All RGB Values 0  255

Common Operations: Unary and Binary Functions on Elements abs sqrt square exp log ceil floor … add subtract multiply power mod greater dot

Transposition – Swapping Dimensions

Putting It Together

Randomly populating values Create array with random values from distributions: np.random.randn() – normal distribution with mean 0, stdev 1 binomial normal / Gaussian beta chisquare gamma

Linear Algebra Numpy treats arrays as matrices and supports: np.dot(A, B) – matrix multiply, also in some versions of Python, A @ B Determinant (det), eigenvalues + eigenvectors (eig) Singular value decomposition (svd) And many more!

A few examples

Broadcasting Work with arrays of different shapes when performing arithmetic operations Example: add a constant vector to each row of a matrix

Broadcasting – cont’d Using broadcasting

Broadcasting Rules If the arrays do not have the same rank, prepend the shape of the lower rank array with 1’s until both shapes have the same length. The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension. The arrays can be broadcast together if they are compatible in all dimensions. After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays. In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension

Recap and Take-aways NumPy (and SciPy) provide powerful support for arrays and matrices Bulk operators across the various elements, different kinds of iteration, … For some kinds of array computations, there is a natural mapping to Spark- style distributed computation ... but for others we don’t get much speedup Thus, we can do general-purpose matrix computation on a single machine, and specialized matrix computation in Spark Sometimes it makes as much sense to encode the matrix data as a DataFrame as a Spark matrix...