Transforming Data (Python®) Computer Science and Software Engineering © 2014 Project Lead The Way, Inc.

Transforming Data: Why? Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Transforming Data: Why? Examples: Change representation ['$1', '$2', '$3', …]  [1, 2, 3, …] Arithmetic to convert units or calculate a result [1, 2, 3, …]  [1.05, 2.10, 3.15, …] What if you want to increase all your data by 5% to account for inflation? How would you do some arithmetic or other operation to each data value? In Python the for loop comes to mind. A column of formulas in Excel® sounds appropriate too. This presentation will show three other ways to transform a list of data using Python.

Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Solution 1: for Loop old_data = [1, 2, 3, 4, 5] new_data = [ ] for element in old_data: new_data.append(element*1.05) In [ ]:new_data Out[ ]:[1.05, 2.10, 3.15, 4.20, 5.25] To do this in Python with a for loop, change the elements in place or use an aggregator to create a new list for the results, as shown here.

Solution 2: Array Operations Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Solution 2: Array Operations old_data = [1, 2, 3, 4, 5] data = numpy.array(old_data) new_data = 1.05 * data In [ ]:new_data Out[ ]:array([1.05, 2.10, 3.15, 4.20, 5.25]) The numpy library contains the "array" data type and defines addition and multiplication for arrays like this. We'll say more about arrays in numpy, and arrays in general, in a few slides.

Solution 3: map(function, list) Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Solution 3: map(function, list) map() applies a function to each element in list old_data = [1, 2, 3, 4, 5] def inflate(x): return x * 1.05 new_data = map(inflate, old_data) In [ ]:new_data Out[ ]:[1.05, 2.10, 3.15, 4.20, 5.25] The Python built-in function map() will do the trick here, too. Here we've defined a new function: inflate. We use map to apply inflate() to each element.

[ <expression> for <element> in <list> ] Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Solution 4: Python Generator Expression [ <expression> for <element> in <list> ] [f(x) for x in list] old_data = [1, 2, 3, 4, 5] new_data = [x*1.05 for x in old_data] In [ ]:new_data Out[ ]:[1.05, 2.10, 3.15, 4.20, 5.25] For loops and arrays are common across nearly all programming languages. The solution shown here, however, uses a syntax unique to Python. The "lazy generator expression" looks like a for loop inside of a list and has lazy properties that conserve memory, processing power, and storage access times – details not relevant to us here but one of the more powerful aspects of Python.

Transforming Data Examples x  int(x)  min(0,(int(x)) Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Transforming Data Examples x  int(x)  min(0,(int(x)) How do these calculations connect to the picture that tells a thousand words, the histogram? A normal distribution of sample data is shown in the left plot. These thousand values are floats. Taking the int() function on each value in the data set transforms the data. The int() function sends all values toward 0. In the third plot, all negative values have been changed to 0.

A Bit More About Arrays in Python Presentation Name Course Name Unit # – Lesson #.# – Lesson Name A Bit More About Arrays in Python list + list appends instead of transforms In [ ]: a = [1, 2, 3] In [ ]: b = [4, 5, 6] In [ ]: a + b Out[ ]: [1, 2, 3, 4, 5, 6] But numpy arrays allow + and * array + array In [ ]: np.array(a) + np.array(b) Out[ ]: np.array([5,7,9]) scalar + array and scalar * array In [ ]: 2*np.array(a) + 100 Out[ ]: np.array([102, 104, 106]) The earlier example with numpy arrays multiplied all elements by 1.05. A number like 1.05 is called a scalar. If you multiply an array by a scalar, you get another array. Add an array to a scalar, and you still get an array. So the [1, 2, 3] in array a have been doubled and then increased by 100. You can also add two arrays together, item by item. The 5 is the sum of the first elements of a and b, 1 and 4. You can't add lists that way; Python will just concatenate them together.

A Lot More About Arrays – All Languages Presentation Name Course Name Unit # – Lesson #.# – Lesson Name A Lot More About Arrays – All Languages Arrays are faster than lists An array has elements of one data type Binding table Additional memory Name Starting Address Address Increment Class Element Type foo 0x32F2 0x0008 array int The last two slides here are more advanced, but they might help you understand what is going on in the computer at a lower level. Arrays are different than lists. Arrays are faster than lists because they are stored differently. In an array, each element takes up the same number of bits in memory. The address of the 10th element can easily be calculated. The computer can access any element quickly with simple arithmetic to calculate the address of the element. Address 0x32F2 0x32FA 0x3302 Contents foo[0] foo[1] foo[2]

Lists – All Languages Lists store an address and type for each element Presentation Name Course Name Unit # – Lesson #.# – Lesson Name Lists – All Languages Lists store an address and type for each element Binding table Additional memory Name Starting Address Class foo 0x32F2 list The references are each the same size, e.g., 64 bit (0x0040) Lists are more flexible since any data type can be stored in any element, but they're slower. Good data skills include thinking about how to write code that will work in reasonable time even when scaled up to terabyte data. Address 0x32F2 0x3332 0x3372 Contents:Starting Address Contents:Element Type 0x8C32 int 0xE333 float 0x9A12 tuple Address 0x8C32 Contents foo[0] Address 0xE333 Contents foo[1] Address 0x9A12 Contents foo[2]