1. Highlights from EPC 2006 Vincenzo Innocente On behalf of the Local Organizing Committee

2. VI @ EPC062 EuroPython at CERN EuroPython conference organized by SFT this year! Three days –Parallel sessions in Bld 40 –Keynotes and “Lightning” in main auditorium –Dinner in the Globe 280 participants 100 presentations (w/o lightning) –5 by “CERN”

3. VI @ EPC063 Schedule 4 parallel sessions (in bld 40) –All synchronized –5 minutes pause between talks –Easy for people to move from one session to another Plenary Lightning & key notes (in Main Amphi)

4. VI @ EPC064 Scientific Program 7 tracks –Python in Science –Python Language & Libraries –Agile Development –Web Frameworks –Business and Applications –Teaching –Games and Entertainment

5. VI @ EPC065 Community Who –Wide age spectrum Many in post-doc age-range –All 5 continents –Very few women (1-2%, all managers?) Where –Mostly Companies developing Software Solutions Revenue from Selling custom products or services Find business advantages –In using open source software (contribute to its development) –Develop components reusable beyond a specific project –Some Research Labs Domain specific applications Reuse in the community (adapting to pre-existing “habits”)

6. VI @ EPC066 Community What: –Core language development –Web framework, web applications –Software development tools (web based) –Scientific data processing, visualization –No sys-admin, net-admin, embedded-software, office automation Why: –Hear news about Language, Libraries, key products (Zope,…) Discuss, propose, complain –Present their products In many cases just a spin-off component –Work (in Sprint sessions)

7. VI @ EPC067 Messages Python: –A language for rapid-prototyping, extreme- programming, just-in-time deployment –THE integration framework –THE Business Domain Language –THE embedded scripting language Python is faster than Assembler

8. VI @ EPC068 Outline What I will not cover –Latest greatest features of Python –Python 3000 –SciPy, PyTables, PyPy, Zope, Plone, Gjango,… –Python in HEP –Google…. I will focus on –Python: a framework for scientific application –Building and sharing components –Python: from fast-prototyping to engineered code –Dispersed development

9. Scientific Frameworks

10. VI @ EPC0610 MGL Tools Independent and re-usable component for structural bioinformatics

11. VI @ EPC0611 MGL Tools Independent and re-usable component for structural bioinformatics

12. VI @ EPC0612

13. VI @ EPC0613 AutoDock tools

14. VI @ EPC0614

15. VI @ EPC0615

16. VI @ EPC0616 Python Molecular Viewer

17. VI @ EPC0617

18. VI @ EPC0618 Pyphant

19. VI @ EPC0619 Pyphant application

20. VI @ EPC0620 Pyphant architecture

21. VI @ EPC0621 Worker Code

22. VI @ EPC0622 SciPy & ETS

23. Building & Sharing Components

24. VI @ EPC0624

25. VI @ EPC0625 Builds upon SciPy (data representation) And HDF5 (I/0 layer)

26. VI @ EPC0626

27. VI @ EPC0627 The Company

28. VI @ EPC0628 The Customer

29. VI @ EPC0629 The “new” Components For this customer they had to two additional requirements to fulfill: –Avoid to blow the CMS with binary files –Count the number of accesses They developed two lightweight products –Plug in the deployed solution Reuse the existing infrastructure –Reusable outside this project and company –Extendable to other architecture/framework –Contribution to open source software

30. VI @ EPC0630 Tramline Tramline plugs between Apache and Plone/ZOPE On Upload: –extract data to disk –Assign id –Store id in ZOPE On download –Replace id with file content

31. VI @ EPC0631 Linktally Scan logs Count request Store in the DB as Metadata Rank content in CMS

32. VI @ EPC0632 LinkTally status & prospects Now Solution for one customer Limited spin-off Evolution Contribution from community Spin-in: use it in other projects!

33. From a prototype to a product

34. VI @ EPC0634 The Indico Technology Main programming language: Python Runs on Apache using the Python module mod_python Persistence based in ZODB (Zope Object Database) Transparency: no need for explicit read/writes of the objects Fits very well with Indico complex object model Proven performance and scalability Timetable generation: libXML, libXSLt + python bindings Portable technologies: runs on Windows, linux Export gateways: –iCalendar ; XML ; PDF outputs –OAI (Open Archive Initiatives) for ensuring integration with other services Standard protocol for information exchange between digital libraries Allows to expose conference data Allows other systems to fetch conference data and build services over it Simple mechanism  XML over HTTP

35. VI @ EPC0635 Main programming language: Python Runs on Apache using the Python module mod_python Uses MySQL RDBMS –Take advantage of fully featured query language Invenio home made Indexes Internal representation with XML-MARC Export gateways: –Multiple output formats: HTML, XML, MARC, OAI, DC, etc. Some modules: –Still in PHP (slowly moved to Python) –Some in Common Lisp (BibCheck) The Invenio Technology

36. VI @ EPC0636 Index Space Design (II) Two important speed factors to consider: –speed of set intersections (Web App Server) –speed of set marshalling (Web App DB Server) Data structures tested: –sorted (lists, Patricia trees) –unsorted (hashed sets, binary vectors) fast prototyping: (Python) –throw-away coding, organic-growth software development model –typical search time gain: 4.0 sec  0.2 sec –typical indexing time loss: 7 hours  4 days –binary vectors found the best compromise (for all types of sets)

37. VI @ EPC0637 Performance Benchmarks (2002) Testing marshalling/intersection/union/unmarshalling Bytecode interpreted language study: (Python, Java) –Python faster than Java (mainly due to marshalling) Machine code compiled language study: (ML, Lisp) –OCaml, CMU CL: 3+ times faster than Python C libs –CMU CL best scalable: intersecting 6M records in 0.01 sec, 30M records in 0.04 sec Data structure study: –OCaml, 3,000,000 records: bit vectors 0.43 sec, hashed sets 1.71 sec, lists 3.76 sec, Patricia trees do not scale well for dense sets Python fast enough for production (1M records) –fast C modules: Numeric (byte/bit), Marshal, Psyco

38. VI @ EPC0638 The + of Python Clean aesthetical language Easy to learn, important for many internship students and temporary members working on the project Very good for rapid prototyping & organic-growth development Plenty of ready-to-be-used modules Bytecode-compiled only, speed okay for our needs

39. VI @ EPC0639 Use Python?

40. VI @ EPC0640

41. Dispersed Teams

42. VI @ EPC0642 Dispersed teams

43. VI @ EPC0643

44. VI @ EPC0644

45. VI @ EPC0645

46. VI @ EPC0646

47. VI @ EPC0647

48. At Last

49. VI @ EPC0649 What I Learned Python is not just a language for scripting and glue code –Fully fledged, highly engineered frameworks can be written in Python Frameworks and component architectures are established practices –Frameworks tend to be domain specific –All very similar to each other and share many design patters Many concepts common to modern HEP-framework architectures B usiness D omain L anguage s are essential: –Python has the expressive power to implement them

50. VI @ EPC0650 What I learned What can be reused? –Experience, patterns, Provided one has a common “culture” –Low level components –Plugin components Provided that the interface is NOT business-domain specific LHC is not anymore at the frontier of distributed collaboration There are Individuals/Labs/Companies which value –Sharing information –Building reusable software components –Cooperating in developing the basic building blocks –Become a community around such a common ground

51. VI @ EPC0651 More? Visit –http://vanrees.org/weblog/topics/europythonhttp://vanrees.org/weblog/topics/europython –http://indico.cern.ch/conferenceDisplay.py?confId=44http://indico.cern.ch/conferenceDisplay.py?confId=44 –http://www.europython.org/http://www.europython.org/ –http://www.google.com/search?q=europythonhttp://www.google.com/search?q=europython