PyTimber & CO M. Betz, R. De Maria, M. Fitterer, C. Hernalsteens, T. Levens Install: $ pip install pytimber Sources: https://github.com/rdemaria/pytimber

PyTimber: design criteria Package goal: Provide a simple interface to CALS API for interactive data analysis in Python: get data with few lines of code works well in the typical scientific Python ecosystem: numpy (array objects), matplotlib (plotting library), ipython/jupyter (interactive environment), etc… use native python objects for input and output: string, lists, dictionary, floating point numbers, array keep python API predictable, expose simple building blocks, provide higher level functionality top of the building blocks keep performance under control use easily from in lxplus, technical network, swan, users machines reduce prerequisites to the minimum

PyTimber: example

PyTimber: prerequisites Python 2.7 or Python 3.x jpype: critical package to connect to a JVM enabling the whole concept of leveraging on Java APIs. Not in the Python standard library, community supported, complex package built on top of Java native interface, not straightforward to install, not easy to use in concurrent processes cmnbuild_dep_manager: get jars, manage JVM, resolve class location use from CERN network

PyTimber: methods __init__: choose appid, clientid, source search: from string pattern get: variables, time window, fundamental (optional) Variables: string, pattern, list of strings Time window: strings (localtime), unix time, datetime objects, ‘last’, ‘next’ tree: explore variable hierarchies getAligned: return align data getLHCFillData: return beam modes for a fill getLHCFillesByTime: return a list of fills getIntervalsByLHCModes: return a list fill numbers and intervals getMetaData: get all metadata defined by pattern or list

PyTimber: status and wish list API stable, still few methods could be added Performance regressions from first releases to be investigated Better time zone management (choose UTC time as input, time zones explicitly) Documentation is still sparse and not complete: tutorial, examples in the source distribution, swan galleries, GitHub wiki, CO wiki Decide if pytimber should evolve towards a large framework: provide generic methods + application specific classes or keep as it is and use e.g. pytimbertools to provide the additional functionalities

Pytimber Tools: pagestore Provide data persistence of pytimber time series. store everything pytimber can emit subset of pytimber API: search and get from unix timestamps fast bulk reading performance: data stored contiguously in binary files large dataset management ~TB separate data store (e.g. in EOS) path and data indexes (e.g. local) slow writing speed, brittle concurrent writing

Pytimber Tools: pagestore example

Pytimber Tools: dataquery Encapsulate query in a object: use pytimber or pagestore as data source provide shortcuts to data array for interactive manipulations provide generic plots (lines, images for 2D arrays) method to extend datasets by appending/inserting data or adding variables Interpolate and align all variables

Pytimber Tools: dataquery example

Pytimber Tools: BSRT Class to process LHC BSRT data.

Outlook Pytimber proved to be an easy way to query CALS data. Pytimber API simple and generic enough that can be used by other data service. Applications being built on top of Pytimber API. Actions and proposals: Formalize Python API and use it for future products related to machine data. Use pytimber package to provide additional tools to give a more complete experience, collect best practices, avoid reinventing wheels, reduce development time. CO being more involved in jpype maintenance. Profile pytimber and see if opportunities of speed-up emerges. Start using py.test for testing and improve documentation (Sphinx?)

Proposals for sub packages