1. Useful Python Techniques: A brief introduction to List Comprehensions, Functional Programming, and Generators October 4, 2011 Joe Cross

2. Contents 1. Looping and List Comprehensions 2. Filter, Map, Reduce (and lambda!) 3. Functional Programming Basics + Decorators 4. Generators

3. 1. Looping and List Comprehensions for(int i = 0; i < 5; i++){ cout << witty_banter(i) << endl; }

4. Looping compared to Java/C++ Looping for i in range(0, 10, 1): a /= 2.0 for i in xrange(10): a /= 2.0 for(int i = 0; i < 10; i++){ a /= 2.0; } Python Java/C++

5. Looping Range + Iter Looping aList = [0, 1, 2, 'hello', 2**-4] for i in range(len(aList)): print aList[i] for item in aList: print item

6. Looping still not as fun as… Looping To double the values in a list and assign to a new variable: winning_lottery_numbers = [0, 4, 3, 2, 3, 1] fake_lottery_numbers = [] for i in range(len(winning_lottery_numbers)): fake_lottery_numbers.append(2 * winning_lottery_numbers[i]) fake_lottery_numbers = [] for number in winning_lottery_numbers: fake_lottery_numbers.append(2 * number) Even though it’s an improvement over the tedium of c++ et. al, we can still do better.

7. List Comprehensions Woohooo! List Comprehensions winning_lottery_numbers = [0, 4, 3, 2, 3, 1] fake_lottery_numbers = [2*n for n in winning_lottery_numbers] List Comprehensions allow us to do all sorts of things: Single-function single-line code Apply a function to each item of an iterable Filter using conditionals Cleanly nest loops Syntax: [ for in if ]

8. List Comprehensions Don’t nest too many! List Comprehensions Multi-variable functions in a single line using zip: vec1 = [3, 10, 2] vec2 = [-20, 5, 1] dot_mul = [u*v for u, v in zip(vec1, vec2)] dot_prod = sum(dot_mul) Filtering: readings = [-1.2, 0.5, 12, 1.8, -9.0, 5.3] good_readings = [r for r in readings if r > 0] Bad: orig = [15, 30, 78, 91, 25] finals = [min(s, 100) for s in [f+5 for f in orig]]

9. 2. Filter, Map, Reduce Life = map(evolution, irreducible complexity) assert(sum(Life) == 42)

10. Filter Filter, Map, Reduce def isPos(number, lim = 1E-16): return number > lim >>> a = [-1,2,-3,4,-5,6,-7,8,-9,10] >>> filter(isPos, a) [2, 4, 6, 8, 10] >>> filter(not isPos, a) Traceback (most recent call last): File " ", line 1 filter(not isZero, a) TypeError: 'bool' object is not callable Syntax: result = filter(aFunction, aSequence)

11. Filter + Lambda Filter, Map, Reduce def isPos(number, lim = 1E-16): return number > lim >>> filter(lambda n: not isPos(n), a) [-1, -3, -5, -7, -9] [fnName] = lambda [args]: expression Syntax:

12. Lambda vs. def Filter, Map, Reduce def add(x, y): return x + y Ladd = lambda x, y: x+y def printWords(): print "Words" LprintWords = lambda: print "Words"

13. So… why use lambda? Filter, Map, Reduce When using verbose function declaration it is often the case that the function’s verbose declaration can be verbose, even for functions that don’t require such verbosity. def ispos(n): return n > 0 b = filter(ispos, aList) Also, there are some valid concerns about namespace clutter and the like. Verbose verbose verbose. b = filter(lambda n: n > 0, aList) Vs. b = [] for a in aList: if a > 0: b.append(a) Verbose Not Verbose

14. Map Filter, Map, Reduce Compare to list comprehension: winning_lottery_numbers = [0, 4, 3, 2, 3, 1] fake_lottery_numbers = [2*n for n in winning_lottery_numbers] fake_lottery_numbers = map(lambda n: 2*n, winning_lottery_numbers) 1. 2. Syntax: result = map(aFunction, aSequence)

15. Reduce Filter, Map, Reduce Syntax: result = reduce(aFunction, aSequence, [initial]) lambda factorial n: reduce(operator.mul, xrange(1, n)) NOTE: results get accumulated on the left, and new values applied to the right. so reduce(add, [1,2,3,4]) is processed as (((1+2)+3)4)

16. 3. Functional Programming + Decorators This isn’t your dad’s Procedural (imperative) programming

17. A Simple Example def add(x, y): return x + y def sub(x, y): return x - y def mul(x, y): return x * y def div(x, y): return x / y def mod(x, y): return x % y def op(fn, x, y): return fn(x, y) Functional Programming

18. Nested Functions Speed vs. Obfuscation Functional Programming def randNoGap(min_, max_): #random.random() -> [0,1) v = random.random() return (max_ - min_) * v - min_ def randWithGap(min_, max_): s = random.random() v = randNoGap(min_, max_) if s < 0.5: return v else: return -v #Same conditional using Python’s #Ternary operator #return v if s < 0.5 else -v def rand(min_, max_, hasGap = False): if hasGap: return randWithGap(min_, max_) else: return randNoGap(min_, max_)

19. Nested Functions Speed vs. Obfuscation (Continued) Functional Programming def randomExplosion(minv, maxv, n): particles = [] for _ in xrange(n): vx = rand(minv, maxv, True) vy = rand(minv, maxv, True) vz = rand(minv, maxv, True) vx2 = rand(minv, maxv, True) vy2 = rand(minv, maxv, True) vz2 = rand(minv, maxv, True) r = rand(0,255,False) g = rand(0,255,False) b = rand(0,255,False) mainParticle = [vx,vy,vz,r,g,b] secondParticle = [vx2,vy2,vz2,r,g,b] particleGroup = (mainParticle, secondParticle) particles.append(particleGroup) return particles NO

20. Nested Functions Speed vs. Obfuscation (Continued) Functional Programming What we’d like to do: velocities = [rndV() for _ in xrange(6)] What it actually looks like: velocities = [rand(minv,maxv,True) for i in xrange(6)] With a functional wrapper, we re-map: rndV -> make_rand_fnc(minv, maxv, True) rndV() -> make_rand_fnc(minv, maxv, True)()

21. Nested Functions Speed vs. Obfuscation (Continued) Functional Programming def mkRand(min_, max_, hasGap = False): def wrapper(): return rand(min_, max_, hasGap) return wrapper rand(minv, maxv, True) rand(minv, maxv, True) rand(minv, maxv, True) def rand(min_, max_, hasGap = False): def randomExplosion(minv, maxv, n): rVel = mkRand(minv, maxv, True) rCol = mkRnd(0,255,False) for _ in xrange(n): vx = rVel() vy = rVel() vz = rVel() vx2 = rVel() vy2 = rVel() vz2 = rVel() r = rCol() g = rCol() b = rCol()

22. Nested Functions Speed vs. Obfuscation (Continued) Functional Programming def randomExplosion(minv, maxv, n): particles = [] rndV = mkRand(minv, maxv, True) rndC = mkRnd(0,255,False) for _ in xrange(n): velocities = [rndV() for i in xrange(6)] r,g,b = [rndC() for i in xrange(3)] mainParticle = velocities[:3] + [r,g,b] secondParticle = velocities[3:] + [r,g,b] particleGroup = (mainParticle, secondParticle) particles.append(particleGroup) return particles

23. Decorators Quickly apply common tasks to methods Decorators Common pre + post function call tasks, such as: Caching Timing Counting function calls Access rights @decorator def myFunc(arg1): print “arg1: “, arg1 myFunc = decorator(myFunc) @f1(arg) @f2 def func(): pass def func(): pass func = f1(arg)(f2(func))

24. Decorators Quickly apply common tasks to methods Decorators def decorator(f): print "This line is run once during func = decorator(func)" def wrapper(*args, **kwargs): print "This line is executed just before the function is called" #Call the function ret = f(*args, **kwargs) print "This line is executed just after the function is called" #Return the function's return return ret return wrapper @decorator def foo(bar): print bar On running, we get this output: >>> ================================ RESTART ================================ >>> This line is run once during func = decorator(func) >>> foo(1) This line is executed just before the function is called 1 This line is executed just after the function is called

25. Decorators Quickly apply common tasks to methods Decorators Decorators using classes class decorator(object): def __init__(self, f): print "This line is run once during func = decorator(func)" self.f = f def __call__(self, *args, **kwargs): print "This line is executed just before the function is called" #Call the function ret = self.f(*args) print "This line is executed just after the function is called" #Return the function's return return ret

26. Decorators Quickly apply common tasks to methods Decorators (Rough) Timing import time class TIMED(object): def __init__(self, f): self.f = f def __call__(self, *args): start = time.clock() ret = self.f(*args) stop = time.clock() print "{0}: {1} ms.". format(self. f. func_name, 1000*(stop-start)) return ret

27. Decorators Quickly apply common tasks to methods Decorators @TIMED def euler(f, t0, y0, h): """ Euler's Method """ yn = y0 + h*f(t0,y0) return yn @TIMED def RK2(f, t0, y0, h): """ Heun's Method """ y_hat = y0 + h*f(t0,y0) yn = y0 + h/2.0*(f(t0,y0)+f(t0+h, y_hat)) return yn @TIMED def RK4(f, t0, y0, h): """ Standard RK4 """ k1 = f(t0, y0) k2 = f(t0+h/2.0,y0 + h*k1/2.0) k3 = f(t0+h/2.0,y0 + h*k2/2.0) k4 = f(t0+h/2.0,y0 + h*k3) yn = y0 + 1.0/6.0*h*(k1 + 2.0*k2 + 2.0*k3 + k4) return yn

28. fns = [euler, RK2, RK3, RK4, jRK4] t0 = scipy.linspace(-1,1) y0 = scipy.ones(50) h = 0.025 args = (f, t0, y0, h) for fn in fns: print fn(*args) print Decorators Quickly apply common tasks to methods Decorators >>> euler: 0.0181114469778 ms. [... ] RK2: 0.041656328049 ms. [... ] RK3: 0.0606733473757 ms. [... ] RK4: 0.0745587900587 ms. [... ] jRKN: 0.00150928724815 ms. jRK4: 1.57358288492 ms. [... ]

29. 4. Generators Memory-conscious patterns are kind of a big deal in scientific computing

30. Binary Tree from Array (simplified interface) class T(object): def __init__(self, values = None, index = 0): self.left = None self.right = None self.v = None if values is not None: self.loadValues(values, index) def loadValues(self, values, index): self.v = values[index] n = len(values) if index * 2 + 1 < n: self.left = T(values, index * 2 + 1) if index * 2 + 2 < n: self.right = T(values, index * 2 + 2) Generators

31. Guessing Game def makeT(val, delta, levels, level = 0): if level < levels: t = T() t.v = val t.left = makeT(val-delta, delta/2, levels, level+1) t.right = makeT(val+delta, delta/2, levels, level+1) return t Generators

32. Clean Code def inorder(t): if t: for v in inorder(t.left): yield v yield t.v for v in inorder(t.right): yield v Generators

33. Using our Generator for v in inorder(a_tree): print v a = [] for v in inorder(a_tree): a.append(v) b = [v for v in inorder(a_tree)] Generators

34. Questions?  “Yes, the slide with the code. What did that one do?”