1. PYTHON FOR HIGH PERFORMANCE COMPUTING

2. OUTLINE  Compiling for performance  Native ways for performance  Generator  Examples

3. IMPORTANCE OF COMPILED CODE  When a code (e.g. a for loop) is written in C, the compiler has the opportunity to optimize its use of memory and floating point units.  The Python interpreter, on the other hand, doesn’t have nearly the same ability.  Import a Python module whose implementation is in C or Fortran

4. NUMPY EXAMPLE  Using NumPy for mathematical operations on arrays and vectors can be hundreds of times faster.  NumPy libraries have been tested, and integrating C or Fortran subroutines you might have written becomes easier.  Tutorial: http://wiki.scipy.org/Tentative_NumPy_Tutorialhttp://wiki.scipy.org/Tentative_NumPy_Tutorial

5. CYTHON  An Open-Source project http://cython.org  Almost a Python compiler (almost)  An extended Python language  writing fast Python extension modules  interfacing Python with C libraries Credit: Behnal http://www.behnel.de/cython200910/talk.html

6. NATIVE WAYS TO GAIN PERFORMANCE  Python is not as strong in memory management  Using the del keyword to delete a variable  Using the feature called generators: greatly reduces memory usage and simplify programming when used in a particular design pattern

7. GENERATOR  A generator is a function that produces a sequence of results instead of a single value  Calling a generator function creates an generator object  When the generator returns, iteration stops  A generator function is a more convenient way of writing an iterator  A generator is a one-time operation – different than a list, need to call again for another iteration

8. GENERATOR EXPRESSIONS  General Syntax: Expression for i in s if condition for i in s: if condition: yield expression E.g. generated version of a list comprehension

9. EXAMPLE: PROCESSING DATA FILES  Summing up the last column of data in a log file  Non-generator solution  Generator solution

10. GENERATORS AS A PIPELINE  Each step is defined by iteration/generation  Instead of focusing on the problem at a line-by-line level, we just break it down into big operations that operate on the whole file.  The iteration that occurs in each step holds the pipeline together.

11. EXAMPLE FURTHER DEVELOPED  Developed from the last program to read hundreds of logs across various directories.  Using the function “os.walk” for searching the file system.

12.  Open a sequence of filenames  Concatenate items from one or more source into a single sequence of items

13.  Generate a sequence of lines that contain a given regular expression

14.  Call Generator Functions

15. REFERENCES  Behnel, S. Using the Cython Compiler to write fast Python code http://www.behnel.de/cython200910/talk.html http://www.behnel.de/cython200910/talk.html  Generator Tricks for Systems Programmers by David M. Beazley http://www.dabeaz.com/generators/http://www.dabeaz.com/generators/