1. Ganga: LHCb analysis on the grid (*)‏ A. Sarti (*)For dummies.... like me! I am assuming a basic knowledge of LHCb Gaudi applications. However, even if you do not know them, you'll be able to blindly try some exercises.... :) !

2. 28/11/07 A. Sarti Scuola per utenti INFN della grid Outline  Ganga  The Grid and user access  Introduction to Python programming  Using (debugging? :)!) Ganga  LHCb specific use cases  MC production : Dirac script example  Documentation  Tutorials

3. 28/11/07 A. Sarti Scuola per utenti INFN della grid Ganga  An interface that provides – Easy switch between batch and Grid resources – Hiding of the complexity in configuring jobs – Bookkeeping  Everything I'll say for lxplus/CERN should be valid for ui01-lcg/CNAF! – If not: file a bug report! :)‏

4. 28/11/07 A. Sarti Scuola per utenti INFN della grid Client Ganga.Core GPI GUI CLIP Job Repository File Workspace IN/OUT SANDBOX AtlasPR DIAL DIRAC LCG2 gLite localhost LSF Athena Gaudi Plugin Modules Monitoring Ganga Architecture Scripts j = Job(backend='LSF')‏ j.submit()‏

5. 28/11/07 A. Sarti Scuola per utenti INFN della grid How to access distributed resources  Grid certificate – This is what gives you a unique identification on the Grid. subject : /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti/CN=proxy – At the basic level just 2 files in your ~/.globus directory.  Grid proxy – By sending a Grid proxy along with a job you allow computers for a limited time to act on your behalf. Running jobs at Tier centres Reading/writing files to remote storage – You obtain a proxy for a finite time When a proxy expires jobs will fail, write operations terminate etc. Make sure your proxy is valid for longer than your job will last.

6. 28/11/07 A. Sarti Scuola per utenti INFN della grid How to access distributed resources  Getting a Grid proxy – You can only do this from a machine which is a “LCG user interface”. – You turn the lxplus machines at CERN into an LCG UI by typing [asarti@ui01-lcg]~% GridEnv – By default you get a proxy which is valid for 12 hours. You can change this by adding an option. [asarti@ui01-lcg]~% grid-proxy-init – Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti – Enter GRID pass phrase for this identity: – Your proxy is valid until: Thu Jan 5 23:48:46 2006 [asarti@ui01-lcg]~% grid-proxy-init -valid 24:00 – Your proxy is valid until: Fri Jan 6 11:48:59 2006 – In Ganga this is not needed! No need of GridEnv Proxy automatically created when starting up (you can change validity as an option as well)

7. 28/11/07 A. Sarti Scuola per utenti INFN della grid How to access distributed resources  Information about a Grid proxy – From command line [asarti@ui01-lcg]$grid-proxy-info subject : /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti/CN=proxy issuer : /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti identity : /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti type : full legacy globus proxy strength : 512 bits path : /tmp/x509up_u13005 timeleft : 299:49:24 (12.5 days)‏ – While in ganga (see later!) you can In [3]:gridProxy.info()‏ Out[3]: subject : /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti/CN=proxy issuer : /C=IT/O=INFN/OU=Personal Certificate/L=LNF/CN=Alessio Sarti etc etc etc

8. 28/11/07 A. Sarti Scuola per utenti INFN della grid Running Ganga  LHCb account on lxplus/ui01-lcg – Avoid GridEnv : conflicting paths! – % GangaEnv At the list pick the release 4.4.2 – % ganga – *** Welcome to Ganga *** – Version: Ganga-4-1-0-beta4 – Documentation and support: http:/cern.ch/ganga – Type help() or help('index') for online help.  You will be asked few questions. Answer “yes” by pressing ENTER.  GRID certificate: type your password or press Control-D several times.  To quit press Control-D at the interactive prompt – In[1]: ^D – Do you really want to exit ([y]/n)?

9. 28/11/07 A. Sarti Scuola per utenti INFN della grid Ganga prompt  Ganga is based on python 2.2 and has an enhanced python prompt (Ipython):  Python programming/scripting – myvariable = 5 – print myvariable*10  Easy access to the shell commands – !less ~/.gangarc # personal config file – !pwd  History  TAB completion – works on keywords, variables, objects, try: my  Many more features

10. 28/11/07 A. Sarti Scuola per utenti INFN della grid.gangarc file  File kept in your $home  Should be used to override the configuration defaults. Ex: – [FileWorkspace] topdir = /opt/exp_software/lhcb_user/asarti/scratch0/gangadir – [GridProxy_Properties] # Proxy validity at creation (hh:mm): #validityAtCreation = 300:00 # Minimum proxy validity (hh:mm), below which new proxy needs to be created: #minValidity = 75:00 # Number of password attempts allowed when creating proxy: #maxTry = 1 – [ROOT] # configuration of where to find ROOT on the local system and which version to use by default. #location = /afs/cern.ch/sw/lcg/external/root/ #version = 5.12.00 #arch = slc3_ia32_gcc323

11. 28/11/07 A. Sarti Scuola per utenti INFN della grid Python: Statements, Variables x = 2 print x*3 # This is a comment! if x==2: print "yes, x==2" # NOTE INITIAL SPACES! Alist = [1,2,3] for y in alist: print y print len(alist) # built-in function len help(len)‏ I hope you have your favourite editor set up with proper configuration to help you with python editing / spacing!

12. 28/11/07 A. Sarti Scuola per utenti INFN della grid Python: Functions print range(11)‏ help(range)‏ def square(v): return v*v print square(x)‏ alist = [1,2,3] for i in range(len(alist)): alist[i] = square(alist[i])‏ blist = [square(x) for x in alist]

13. 28/11/07 A. Sarti Scuola per utenti INFN della grid import sys print sys.argv import os print os.environ['HOME'] from os import environ from os import * import math, cmath print math.cos(math.pi) math.log(-1) # exception Python: Modules

14. 28/11/07 A. Sarti Scuola per utenti INFN della grid Python: Modules  Very large number of Python modules available.  Check if somebody else has written it before you go ahead.  Python documentation is excellent.  http://www.python.orgus e version 2.2 of documentation http://www.python.org

15. 28/11/07 A. Sarti Scuola per utenti INFN della grid # user defined classes a = MyClass(2,3)‏ a.my_method(1) #built-in classes: for example strings s = "Hello Tutorial" # double quotes s.split()‏ s.count('a')‏ s.upper() dir(s)‏ if 'Tut' in s: print 'Tut is in ',s print '-'*20 # single quotes Python: Classes and Objects

16. 28/11/07 A. Sarti Scuola per utenti INFN della grid # file objects f = file('/etc/password')‏ lines = f.readlines()‏ print len(lines) # writing files f2 = file('testfile','w')‏ print >> f2, “This is a test”,20*'-' f2.close() dir(f)‏ Python: Files

17. 28/11/07 A. Sarti Scuola per utenti INFN della grid The components of Ganga  Applications – These are objects that know about the type of application you run Simple executable, Gauss job,...  Backends – Objects that know about the places you can execute your job, data transfer, requirements etc  Job – This is the basic object you interact with. – You create a “Job” by combining an application with a backend: “A DaVinci job to run on the LSF batch system”  Most of them can be accessed by typing 'plugins' in the CLIP @ login...

18. 28/11/07 A. Sarti Scuola per utenti INFN della grid Ganga Scripting Job Objects and Job Registry j = Job()‏ print j print j.application print j.backend # job registry: a list of all your jobs jobs jobs(1)‏

19. 28/11/07 A. Sarti Scuola per utenti INFN della grid Jobs In [26]:J = Job()‏ In [27]:print J Job ( status = 'new', name = '', inputdir = '/opt/exp_software/lhcb_user/asarti/scratch0/gangadir/Local/102/input/', outputdir ='/opt/exp_software/lhcb_user/asarti/scratch0/gangadir/Local/102/output/', outputsandbox = [], id = 102, info = JobInfo ( submit_counter = 0 ), inputdata = None, merger = None, inputsandbox = [ ], application = Executable ( exe = 'echo', env = {}, args = ['Hello World'] ), outputdata = None, splitter = None, subjobs = [ ], backend = Local (actualCE = None, workdir = None, id = None, exitcode = None )‏ )‏

20. 28/11/07 A. Sarti Scuola per utenti INFN della grid [In:] j = Job()‏ [In:] j.application.exe = '/bin/hostname' [In:] j.name = "MyTest" [In:] print j [In:] j.submit() # wait until job is completed and look # at the output directory [In:] print j.status [In:] print j.outputdir [In:] !ls $j.outputdir [In:] !cat $j.outputdir/stdout # the ! and $ syntax is IPython – you cannot do it # in normal python it is useful for interactive work # instead of mouse copy-paste Job creation and submission

21. 28/11/07 A. Sarti Scuola per utenti INFN della grid # once a job is submitted you cannot modify # it. if you want to submit a new job you # should create a new job object [In:] j2 = j.copy()‏ [In:] j2.backend = LSF()‏ [In:] j2.submit()‏ # if you have GRID certificate you can try [In:] j3 = j.copy()‏ [In:] j3.backend = LCG()‏ [In:] j3.submit() # try jobs to see all your jobs More on job submission

22. 28/11/07 A. Sarti Scuola per utenti INFN della grid More on job submission (II)‏ # see predefined plugins ‏plugins("applications")‏ plugins("backends") help('index') # submission in a loop for i in range(5): j = Job() j.application.exe="/bin/echo" j.application.args=["hello ",str(i)])‏ print j.id, j.inputdir print j.submit()‏

23. 28/11/07 A. Sarti Scuola per utenti INFN della grid Getting organised with many jobs  Ganga has logical folders organised in the jobtree – Very similar to symbolic links in a file system. A job can be in several places e.g: – folder ‘FailedJobs’, folder ‘GaussJobs’, folder ‘TutorialJobs’ Example: – Create some Jobs (j = Job())‏ – Crate several jobtree directories : jobtree,mkdir(“a”); jobtree2,mkdir(“b”); etc etc – Add the various jobs to different subdirs jobtree.cd(“a”) and jobtree.add(job)‏ # Available methods for dir(jobtree)‏ Out[85]: ['add', 'cd', 'cleanlinks', 'copy', 'exists', 'find', 'getjobs', 'isdir', 'listdirs', 'listjobs', 'ls', 'mkdir', 'name', 'printtree', 'pwd', 'rm']

24. 28/11/07 A. Sarti Scuola per utenti INFN della grid # For frequently used jobs – use templates! t = JobTemplate(j3)‏ t.name = ‘MyTemplate’ # Persisted across sessions print templates j = Job(templates[1]) # By ID j2 = Job(templates[‘MyTemplate’][0]) # By name # Other useful methods j.kill()‏ j.remove() # Removes the output as well! for j in jobs: j.kill()‏ jobs.clean() # Faster! Deletes ALL jobs. Templates

25. 28/11/07 A. Sarti Scuola per utenti INFN della grid Exporting and loading jobs  You can export jobs (and any other Ganga object) to simple text files. # Export a job export(j,'job.txt')‏ !cat job.txt #Ganga# File created by Ganga - Fri Jan 6 17:06:00 2006 #Ganga# Job object (category: jobs)‏ Job ( name = 'Tag',.... You can then keep, edit, distribute these files. – Not a replacement for the repository!  Jobs are trivial to load again. – Exporting a job and loading it again afterwards is equivalent to making a copy of the job.  Load a job: load('job.txt')‏. To check the newly loaded job use jobs() command inside ganga

26. 28/11/07 A. Sarti Scuola per utenti INFN della grid ganga & LHCb jobs  For LHCb the main use of Ganga is for running Gaudi jobs.  This includes: – Configuring Gaudi jobs Forget setenv CMTPATH, source setup.sh, etc. – Specify the datasets – Run the jobs locally, on batch systems and on the Grid via Dirac – Managing the output data, Ntuples and histogram files.  ROOT jobs are currently supported as well! – Specify root version – Specify input macros, libraries,....  Before creating/submitting jobs for a given Gaudi Project and AFTER having obtained the ganga environment setup remember from in(out)side ganga to obtain the proper configuration for your gaudi application (setenvXXX and source setup.csh...)‏

27. 28/11/07 A. Sarti Scuola per utenti INFN della grid Intresting backends for LHCb  There are 3 backends of direct interest: – Local : Jobs runs in the background on the client – LSF : Jobs are submitted to the LSF batch system (if available)‏ – Dirac : Jobs are submitted to the Grid via Dirac  LCG of course is available BUT not supported!  Ex: Dirac backend – # Define a Dirac backend object – d = Dirac()‏ – print d Out[34]: Dirac (status = None, actualCE = None, id = None, CPUTime = 86400 )‏

28. 28/11/07 A. Sarti Scuola per utenti INFN della grid Putting together a Gaudi job  To put together and submit a job is simply matter of combining the three main different parts: options, application, backend.  Example: – # Put together an LHCb job and submit – j = Job(name='MyJob',application=g, backend=d)‏ – print j Out[38]: Job (status = 'new', name = 'MyJob', application = Gauss (...)‏, backend = Dirac (...)‏...)‏ – j.submit()‏

29. 28/11/07 A. Sarti Scuola per utenti INFN della grid The Gaudi application handler There is a specific application handler for each Gaudi application: – ['Boole', 'Brunel', 'DaVinci', 'Gauss'] – Ex: Gauss job # Define a Gauss application object g = Gauss()‏ – Check with print g the default g.extraopts='ApplicationMgr.EvtMax = 10 ;' g.optsfile=g.cmt_user_path+ 'Sim/Gauss/v22r0/options/myopts.opts' – Check with print g the overwritten options

30. 28/11/07 A. Sarti Scuola per utenti INFN della grid Example: Gauss Job # Look at the object print g Out[31]: Gauss ( version = 'v22r0', extraopts = 'ApplicationMgr.EvtMax = 10 ;', package = 'Sim', cmt_user_path = '/afs/cern.ch/user/u/ uegede/public/cmtTutorial', masterpackage = 'Sim/Gauss/v22r0', optsfile = File (name = '.../Gauss/v23r0/options/myopts.opts', subdir = '.' )‏ In case you want to run several different Gauss jobs it is possible to use the GaussSplitter: – GaussSplitter( numberOfJobs = 50, eventsPerJob = 3000 )‏ – Jobs have all the same runnumber (but different Evt num)! Use the splitter with backend != Dirac In case you don't want to split the job put – mySplitter = None

31. 28/11/07 A. Sarti Scuola per utenti INFN della grid Analysis job  DaVinci job. Like in gauss you can specify: – Software versions – Private packages dv.getpack('Tutorial/Analysis v5')‏ – Options – Datasets DaVinci ( extraopts = None, package = 'Phys', masterpackage = None, version = 'v19r8', configured = 0, cmt_user_path = '/opt/exp_software/lhcb_user/ asarti/software', optsfile = File ( name = '', subdir = '.' )‏ – You can either provide your dataset in your option files (check the correct LFN syntax for the grid) or use the ganga utilities to handle the dataset (see later...)‏

32. 28/11/07 A. Sarti Scuola per utenti INFN della grid Do it all from Ganga # Define application and get code dv = DaVinci()‏ dv.cmt_user_path = '/afs/.../public/cmtTest' dv.masterpackage='Tutorial/Analysis/v5' dv.getpack('Tutorial/Analysis v5')‏ dv.optionsfile='/afs/.../myAnalysis.opts' dv.make() # Build code # Create job and submit j = Job(name='DaVinciTest', application=dv, backend=Dirac())‏ j.submit()‏

33. 28/11/07 A. Sarti Scuola per utenti INFN della grid 33 Data on the Grid: A Typical LFN  /lhcb/production/DC06/phys-v2- lumi2/00001760/DST/0000/00001760_00000310_5.dst  From the bookeeping page (http://lhcbbk.cern.ch/BkkWeb/Bkk/ welcome.htm) you typically get:http://lhcbbk.cern.ch/BkkWeb/Bkk/ – EventSelector.Input = { "DATAFILE='LFN:/lhcb/production/DC06/phys-v2- lumi2/00001760/DST/0000/00001760_00000001_5.dst' TYP='POOL_ROOTTREE' OPT='READ'" }; – Replicas should exist at least in 1 site(s): CNAF_Castor, Cambridge_DISK, PIC_Castor,CERN_Castor Should a user be concerned with how to use the same dataset at different sites? – NO! ganga & Dirac can help the user specifying and getting the input data....

34. 28/11/07 A. Sarti Scuola per utenti INFN della grid 34 Providing Access to Input Data  Logical File Names (LFNs)‏ – Unique identifier on the Grid – Namespace starts with /lhcb/ Physical File Names (PFNs)‏ Can have many PFNs per LFN Corresponding to replicas of file at different sites  To use data on the Grid it must be registered on the Grid At least one replica must exist in the LCG File Catalogue  DIRAC automatically translates LFNs into the appropriate PFNs for a given site and feeds this information to the application – Grid (DIRAC) users need only know LFNs – ganga uses LHCbDataFile and LHCbDataset to handle/check the data files and corresponding jobs.....  How to get the LFNs ? Bookkeeping Page!

35. 28/11/07 A. Sarti Scuola per utenti INFN della grid Dataset Handling  Check dataset replicas BEFORE submitting jobs (prevent loosing time waiting for dataset to be ready / job doomed)‏ – ds = LHCbDataset(files=list)‏ – ds.updateReplicaCache()‏ LHCbDataFile (name = 'LFN:/lhcb/production/DC06/phys-v2- lumi2/00001760/DST/0000/00001760_00000306_5.dst',replicas = ['NIKHEF-disk', 'GRIDKA-disk', 'RAL-disk', 'IN2P3-disk', 'CERN- disk'] )‏ goodlist=[] badlist=[] for df in ds.files: if len(df.replicas)==0: badlist.append(df)‏ else: print df goodlist.append(df)‏ – If your datasets is large you can ask to split the main job in subjobs defining the number of files for each jobs and the maximum number of files you want to run. mySplitter = SplitByFiles ( filesPerJob = 10, maxFiles = 1000 )‏

36. 28/11/07 A. Sarti Scuola per utenti INFN della grid Execution and finalisation  The progress of a job is monitored by Ganga – Looking at the overview (jobs) or the status field for the backend will show the current status. – Inside ganga: jobs()‏ – Outside ganga, can use a 'script': ganga query  When job is finished the outputdir will contain: – The stdout and stderr of the job – Any histogram files produced or output files requested!  Ntuple and DST files are stored in a storage element on the Grid (as any other output files exceeding 10 MB dimension). – At CERN/CNAF use the nsls command to inspect the content of your Castor area: nsls $CASTOR_HOME

37. 28/11/07 A. Sarti Scuola per utenti INFN della grid Few remarks about running DIRAC  >90% of what you'd like to do can be safely done using ganga. – 100% if you're a little bit smart with python  Why using Dirac directly to submit jobs? – You can find nice features if you want to run MC production [of small samples _of course_!!!! Otherwise go and ask for a general 'official' production] The Job object has an additional 'feature': possibility to have some 'Steps' attached. Sequential jobs is easily built and submitted. – Needs just python and 'DiracEnv' before.....

38. 28/11/07 A. Sarti Scuola per utenti INFN della grid 38 Example Production Script from DIRAC.Client.Dirac import * dirac = Dirac()‏ job = Job()‏ step1 = Step()‏ step1.setApplication('Gauss', 'v19r4')‏ step1.setInputSandbox(['Application_Gauss_v19r4/Gauss.opts'])‏ step1.setOutputSandbox(['Gauss.hbook', 'Gauss_v19r4.log'])‏ step1.setOption(''' ApplicationMgr.OutStream += {''GaussTape''}; GaussTape.output = DATAFILE='PFN:Gauss.sim' TYP='POOL_ROOTTREE' OPT='RECREATE' ''; ''')‏ job.addStep(step1)‏ step2 = Step()‏ step2.setApplication('Boole', 'v8r4')‏ step2.setOption(''' EventSelector.Input={ '' DATAFILE='PFN:Gauss.sim' TYP='POOL_ROOTTREE' OPT='READ' ''}; ''')‏ job.addStep(step2)‏ jobid = dirac.submit(job,verbose=1)‏ print "Job ID = ",jobid Can add multiple user defined steps for Gauss, Boole, Brunel and DaVinci

39. 28/11/07 A. Sarti Scuola per utenti INFN della grid 39 Example Script for Output Retrieval from DIRAC.Client.Dirac import * import sys dirac = Dirac()‏ jobid = sys.argv[1] dirac.getOutput(jobid)‏ Note that all the DIRAC API commands described here may also be executed directly from the Python prompt.  If additional software packages are required  job.addPackage('XmlDDDB','v25r0')‏  Output data to be placed in Grid storage can be specified  job.setOutputData(['DataFile.dst'])‏  This is automatically stored according to LHCb conventions in an LFN of the following form  /lhcb/user/ / / /

40. 28/11/07 A. Sarti Scuola per utenti INFN della grid 40 More Additional Features  Jobs can be rescheduled in the DIRAC WMS – Triggers resubmission to LCG dirac.reschedule(JobID)‏  Can add multiple lines to be appended to the options file of the application directly, through the API job.setOption('''''' ApplicationMgr.DLLs+={''DaVinciTest''}; ApplicationMgr.TopAlg+={''PrimVertAnalysis''}; ApplicationMgr.EvtMax = 10000; '''''')‏

41. 28/11/07 A. Sarti Scuola per utenti INFN della grid 41 Ganga and DIRAC APIs  Ganga and DIRAC both provide Application-Programmer Interfaces (APIs) in Python – For users familiar with submitting jobs to a batch system using bsub myScript or similar, having to learn a new language to submit Grid jobs may seem excessive – Even persistent users will quickly tire of having to type everything at the (I)Python prompt, and having to retype because of syntax errors  One solution is to use scripts (see next slides)‏  Another solution is a Graphical User Interface (GUI): point and click without worrying about syntax

42. 28/11/07 A. Sarti Scuola per utenti INFN della grid 42 Ganga scripts (1)‏  From the command line, a Python script myScript.py can be executed in the Ganga environment using: ganga myScript.py  Ganga makes available simple scripts allowing submission of Grid jobs from the command line: # Create a job for submitting Gauss to DIRAC ganga make_job Gauss DIRAC test.py [ Edit test.py to set Gauss properties ] # Submit job ganga submit test.py # Query status, triggering output retrieval if job is completed ganga query

43. 28/11/07 A. Sarti Scuola per utenti INFN della grid 43 Ganga scripts (2)‏  Given Ganga job name or id, as returned by query, also have possibilities such as: # Kill job ganga kill id # Remove job from Ganga repository and workspace ganga remove id Running jobs on the Grid no more complicated than running jobs on a local batch system(*)‏ (*) Maybe some day you'll like to comment about that statement

44. 28/11/07 A. Sarti Scuola per utenti INFN della grid 44 Ganga GUI (1)‏  Ganga GUI is under development  Built on top of Ganga’s Python API, so doesn’t always offer fully up- to-date functionality – Synchronisation for public (non-beta) releases – Current best version of GUI in Ganga 4.4.3  GUI based on dockable windows: – Monitor: shows status of user jobs – Job builder: provides for job definition – Scriptor: allows execution of any Python/Ganga commands – Logger: displays messages from Ganga  Windows can be docked, floated or hidden  Start the GUI with --gui option @ run time.....

45. 28/11/07 A. Sarti Scuola per utenti INFN della grid 45 Ganga GUI: default view Scriptor Logger Monitor

46. 28/11/07 A. Sarti Scuola per utenti INFN della grid Hints  Subscribe ASAP to distributed analysis mailing list!  Try ganga -h to see options  Use help() and dir() inside ganga  Get used to using scripts and templates. – Will save lots of typing and errors in the future.  Learn ASAP the splitters and LHCbData* utilities wonders – Your life inside LHCb analysis world will be easier  Have a look also in the config file: ~/.gangarc – Many default settings can be changed if required.

47. 28/11/07 A. Sarti Scuola per utenti INFN della grid Documentation  Ganga Web Site: – http://cern.ch/ganga http://cern.ch/ganga  Ganga Tutorial – General getting started guide – LHCb specific documentation  Python Website: – http://www.python.org - use version 2.2 of documentation http://www.python.org  Excellent Python Tutorial: – http://docs.python.org/tut/tut.html  Other resources: – IPython page: http://ipython.scipy.org/  ganga mailing list: – https://mmm.cern.ch/public/archive-list/l/lhcb-distributed-analysis/ https://mmm.cern.ch/public/archive-list/l/lhcb-distributed-analysis/ – lhcb-distributed-analysis@cern.ch