CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL 1 Panoramix A visualization environment for LHCb

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL2 Goals Gaudi/SEAL/POOL LHCb : detector desc & event model Brunel : recDaVinci : physGauss : sim OpenScientist Geant4 Panoramix:vis Handle event display and statistical visualization in a consistent way. LHCb coarse graining organization in term of “projects” : Panoramix is on top “of everything” (that explains the below “eXtreme debugger”). Panoramix subjected to a software agreement between LAL and LHCb.

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL3 It works !!! From the collection of Thomas Ruf From the collection of Thomas Ruf (done on Windows)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL4 Engineering choices Plugged directly to the data framework (Gaudi). No client / server model. Since the data framework is in C++ (and most of native vis and GUI tools are in C / C++) use C++ for all. Target the three laptops of the moment : Linuxes, Windows, MacOSX by using their native environments. Used one powerful “scene manager”, Inventor, for all 3D or 2D data representation. Try to put in “reusable elsewhere packages” things not fully dependant of the experiment. Here the “reusable elsewhere packages” being the ones put in the OpenScientist distribution. See OpenScientist slides for more on “Engineering choices”.

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL5 Vis and GUI products Rendering layer : more than ever : OpenGL. Scene graph manager : OpenInventor. By using the Coin implementation of System In Motion (Norway). Do all graphic with the uppers, including histogram and function plotting. GUI : see OpenScientist slides for the problematic of a choice. Describe the GUI in XML and use the OnX package to create the GUI by using native toolkit of desktop providers. This is simple and give the best performances. Scripted callback : mainly dld/c++ (string front end to dlopen/dlsym) : OnX ui_echo hello and Python (with LHGDict wrapping) : from import OnX *;ui.echo(‘hello’)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL6 OnX Xt, Win, Qt, gtk+ Dld, Python, CINT GL,Inventor XML

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL7 GUI paradigm “A la PowerPoint” : a compact GUI panel organized around a document area. This is 100% in opposition to a terminal prompt spawning viewers around the screen

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL8 Panoramix on a MacOSX by using native Cocoa (NextStep) / OpenGL (not released yet since all below layers not yet released on OSX).

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL9 “data framework” should be understood as the software permitting to manipulate the “event model” and permitting to connect the “event model” (the “data”) to “facilities” like storage, graphic, GUI, scripting. The data framework is here Gaudi. Gaudi does not do any kind of graphic (no Gaudi::DataObject::draw method around and this is correct). Gaudi manipulates services that permits someone to operate the “transient store” containing, in memory, instances of the “event model”. OnX is here the “interactivity framework”. It does the connection between GUI (from an XML description), viewers, scene manager (Inventor), renderer (OpenGL) and scripting. The connection “data framework” “interactivity framework” is done through the Gaudi OnX service, put in a standalone package : Vis/OnXSvc. Various elements of the event model have a “representation” (a datarep). A datarep code is in general a Gaudi “converter” for the Inventor technology (an SoConverter). An SoConverter, for example SoMCParticle, build from a data instance (here an MCParticle) an Inventor scene graph. When built, the scene graph is send to OnX to be displayed, in general, in the “current viewer”. A “visualization request” starts from a scripted GUI callback ; the OnXSvc is searched, then a “data convertion for Inventor” request is activated for various pieces of data. Note that, due to a massive usage of abstract interfaces, things are nicely decoupled (for example an SoConverter is not a StorageConverter. A direct consequence is that someone can get rid of a technology and migrate to a new one, without touching to everything. Connection to the data framework

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL10 Physicists enter into action by programming some “SoConverter” put in some “Vis/So” package. SoDet (G.Barrand, S.Ponce) : representation of the detector. This is very generic and is now sparely touched. Coupled to the LHCb XML det description, it offers a very flexible way to enter and view a geometry. Note that there are various interactive goodies that permits to “expand” or “contract” volumes (from J.Boudreau SoDetectorTreeKit). The GUI can view the detector and event tree in a tree widget. SoEvent : historical. Used to gather first datareps like SoMCParticle, SoMCHit (P.Mato) and representations for the trackers (J.Van.Tilburg NIKHEF). SoCalo (I.Belyaev ITEP) : dedicated package to gather reps for calorimeters. SoRich (C.Jones Cambridge) : reps for the RICH. SoStat (G.Barrand LAL) : to represent histograms of the Gaudi transient store. It uses the HEPVis SoPlotter nodekit. SoHepMC : to visualize HepMCs. Used in conjunction of the Geant4 simulation (Gauss). People did not seem to suffer of Inventor programming. The Vis/So packages.

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL11 Two packages remain Vis/Panoramix and Vis/LaJoconde. Panoramix : it contains the “main” Panoramix.exe program which is a standard “Gaudi main”. This package contains also the Panoramix.onx and other *.onx XML files describing the GUI. It contains also “C++ callbacks” and various Python scripts sufficiently generals to be put “in the common pot”. It is in scripts that people can setup their views (number of regions with their size and position, background colors, etc…) and decide of what they put in their “scenes”. This could be done in Python or directly in C++ (in case some Python wrapping is lacking, too tricky C++ is needed or intensive speed is needed) by using the OnX “dld/C++ callback system”. LaJoconde : historically introduced to connect to “The DaVinci code” (where the secret of LHCb physic is hidden). It had been introduced for “coarse graining” organization but will probably be “transferred” back to Panoramix in future so that physicists manipulate only “one front end program”. This will depend of the capabilities to be able to build coarse graining “Job options” files (by saying easily “I want or I do not want to use DaVinci”) and be able to access corresponding packages and libs at run time. The applications

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL12 Due to basic choices (stick to desktop providers and be plugged directly to the data- framework) it is fast and the author wait anybody that claims to have something faster…. BUT, we must always “have an eye” on Inventor. It is a powerful tool and then can be dangerous. A scene graph may become “too fat” and slow down the creation of a data representation. But there are numerous ways to optimize things (share color, line styles nodes, create dedicated nodes to handle huge collections of data, etc…). Experience shows that on local machines scene graph with up to nodes are ok. For more orders of magnitude, you have to think at optimizing. BUT SOMEONE MUST BE AWARE THAT A CROWDY SCENE GRAPH OFTENLY MEANS A CROWDY PICTURE ON THE SCREEN…. Then the secret of “being fluent” is not in being able to handle big scene graph, it is in being fluent in “selection of data” (selection of what you really want to see). Up to now, at least for what is done in LHCb, speed is sufficient. (I am more worried by what is (not) done in more bigger experiments…) Speed

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL13 All this works and had been put in place very early. Problem remains around visualization of boolean operations over volumes. It is the “case of coplanar edges”. Dixit E.Chernyaev, the author of the algorithm (the BooleanProcessor of Geant4) there are cases where things cannot be decided (easily) and the best way is to avoid to have “edge to edge situations”. Fine, ok. But it is hard to explain to the detector people that they have to add or remove some matter here or there in order to “pass” a visualization algorithm !!!!! But a non-algorithmic solution perhaps remains ; use the stencil capabilities of OpenGL to do that. Code had been found in Australia but not yet put in an Inventor node and integrated / tested in the system. A side effect ; it probably kill the vector PS production and then only “pixmap” PS production would remain for these views. If someone has ideas ? Other algorithms to try, etc… Problems (1)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL14 Not yet fluent in fully stand alone installation on a (remote) local machine. We remember that a display is on top of everything and that the three common desktop are targeted. Then be fluent in installing the display passes by being fluent in installing “all the below”… Today, binaries (Windows, Linux) are regularly reconstructed on the /afs/cern.ch areas (lot of thanks to Florence Ranjard) and users run them on their local machine. (Doing remote X11 on lxplus is definitely not recommended). Then we : –Use CERN as a central integration / binary construction / distribution place : THIS IS GOOD. (I strongly believe that this is the number one role of CERN). –DO NOT USE CERN at run time but use local machines seeing CERN with afs. But CERN anyway around at run time as a STRONG afs server provider. For more, we strongly rely on efforts done around CMT (and PACMAN ?) to provide at some time straight forward source or binary local installation “of everything”. (See Christian Arnault poster about ATLAS software distribution). Problems (2)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL15 LCG : where are we going here ? Can we build “bosonic software” with “fermionic people” ? Panoramix on top of much more things not really deeply though to work together. MacOSX port had been a pure nightmare… (And the story is not finished here. The only presence of ROOT induces to run in “bind_at_launch” mode that slows down the startup of everything. There are mysteries around dynamic loading and execution of static object constructors done with Gaudi and SEAL / PluginManager). Problems (3) gaudi/seal pool pi root/th cintroot/io lcgdict python

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL16 In the software agreement, was mentioned that a physicist should be involved (at partial time) to do the sync during the whole story. This never happened. For example, despite that Panoramix received code contributions from numerous people and that it is used by various people, no physicist “strongly visualization driven” spontaneously emerged to survey that all element of the event model received a representation. The engineer is NOT paid to have all the LHCb detector in head along with all the event model and physic algorithms and to decide how things should be represented. It would be welcome to avoid a “last minute rush” on that last point. A problem ?

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL17 html pages and paper doc produced with Doxygen from dedicated.h files (and not the ones of the code). Some of the files loadable from the program to form a help pull down menu and contextual helps. But the content is clearly insufficient. Nice user page of Thomas Ruf Basement for a wiki ? But for programmers (writing SoConverter), doc of Gaudi and Inventor is online. But once more, is it to the engineer to document everything and in particular all what can be represented with various options, etc…? Problems (4)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL18 How to fit 48 hours in one day ? (Transplantation of a superconducting positronic brain ? (*)) Need manpower ? Sure, the needed army that LCG will not put in place for doing graphic… Problems (5) * Due to the international context, the US Robots refuses to do that on a French for the moment….

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL19 Next main issue is going to be the integration of “stat tools” to do “analysis”. For LHCb it will consist to operate “The DaVinci code” from the interactive tools. Here the way to deal with events is a little bit different : –Display : pick one event, view things in it, pass to another event, etc… –Analysis : pick and configure one or a bunch of algorithms (that will book histos and tuples), run over thousands of events, look histograms. Repeat again by modifying algorithms. That’s right that what is done BY THE PHYSICIST is different but things to put in place BY THE ENGINEER ARE EXACTLY THE SAME IN BOTH SITUATIONS : a GUI, operate on the event model, produce “graphical representations” from data. The only significant differences are around the data that must be visualized. In one case it is things in the event model, in the other case it is common things like histograms and tuples. In the second case, then a key issue is to INTEGRATE A GOOD PLOTTER, that is to say a good piece of software able to create graphical representation of histograms. Next…

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL20 Due to the choice of Inventor as scene manager, the author targets a plotter done with Inventor : here the HEPVis / SoPlotter one developed mainly be the author. SoPlotter covers now a lot of common cases (see OpenPAW poster for pictures). The Vis / SoStat package does the connection Gaudi-histos / SoPlotter. This is in fact available for long but not yet heavily used due to the fact that connection to DaVinci is a little bit young. (and probably because people had taken some habits with other tools…). Then, with SoPlotter, the same Panoramix environment can then be used also by working on DaVinci. (It opens also the way to do nice things like putting histo within events). For scripting, someone can already get from Python the OnXSvc and do a vis request over an histo managed by Gaudi. Someone can also customize a lot of the SoPlotter. What have to be provided now is to “standardize” the way to do that. The plan is to used the AIDA API and interfaces, mainly because it had been though and discussed by various parties and that, being pure abstract interfaces, it does not compel us to tied to another system having orthogonal backend GUI and graphic. HEPVis / SoPlotter, AIDA It is crucial to keep in sync the LHCb software with AIDA

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL21 Had been demonstrated than from a python shell, someone can spawn a DaVinci task with : –Panoramix (to display an event or plot an histo) –Hippodraw –ROOT Then physicists will have tools. But, shall software engineers be happy with that ? NO. All these tools have incompatible backend technologies (rendering, GUI) and there is no hope to provide one tool within a common GUI that integrate all that in a consistent interactive way. A physicist will be let with a desktop crowded with standalone GUI windows, each having its own logic. An already announced pain. Then, some job (low priority) is done to try to integrate some of these tools within Panoramix… Integration of other tools (1)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL22 Hippodraw : the design of it is so that someone can easily get rid of the default GUI and rendering (all in Qt), and comes with new ones. Had been done an OpenGL driver to hippodraw. From that it is quite easy to integrate a hippo window within Panoramix. An embryo of reimplementation of the famous “hippo inspector” done with OnX (then mainly in XML) exists now (OpenScientist / Hippo package). ROOT : it will go through an implementation over Inventor of the painful TVirtualPad (some kind of strange mixture of GUI, rendering and scene manager virtual methods). A prototype exists (OpenScientist / Mangrove package). Someone can visualize a TGeo with OnX and Inventor and also the bins of a TH. But more work have to be done to have the full THistPainter over Inventor. But this is feasible. (Any volunteer ? No one, as usual). Java / JAIDA / jas : hard to see how we can have jas drawing in an Inventor viewer or an OpenGL window not managed by the awt / swing. Moreover it appeared difficult to have Inventor drawing in an awt / swing window…. Strong expertise required to go on. If no help from the developers of these tools is coming, the author will continue to put the effort on the SoPlotter (and OpenPAW) and to its AIDA front end ; the AIDA::IPlotter. Integration of other stat tools (2)

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL23 Agora is a prototype of visualization environment for ATLAS using the same technologies than Panoramix (OpenGL, Inventor, OnX, AIDA). It makes sense since the data-framework is the same than LHCb (Gaudi). Proof of feasibility done. In fact here, it is a shame that LHCb and ATLAS had not been able to converge on XML for the det desc…. Anyone interested ? Must be mentioned : Agora

CHEP'04 InterlakenG.Barrand, eXtreme debugger at LAL24 Panoramix is regularly released and used and had already given services (for the XML det desc debugging, reconstruction debugging). Even if not yet an overflow, views appear in meetings and workshops ; and some are very very nice. Engineering choices are done (let us say up to the startup). But the overall architecture will permit to migrate parts if needed. More work around the release building and distribution are needed if the three common user laptops are targeted (Linux, Windows, Mac). At the engineering level, efforts will be put now on the final (!) integration of statistical tools : AIDA / OpenPAW for sure, hippodraw probably, ROOT with a gun in the back. Rendez vous at next CHEP for that. Are physicists happy ? (is a physicist ever happy ?). It is not to the author to give an answer but there are not too much screams, shouts, laptop thrown on the wall for the moment… Conclusions