2. Condor Project Computer Sciences Department University of Wisconsin-Madison Extend/alter Condor via developer APIs/plugins CERN Feb 14 2011 Todd Tannenbaum

3. www.cs.wisc.edu/Condor 2 Some classifications Application Program Interfaces (APIs) › Job Control › Operational Monitoring Extensions

4. www.cs.wisc.edu/Condor 3 Job Control APIs The biggies: › Command Line Tools › DRMAA › Condor DBQ › Web Service Interface (SOAP) http://condor-wiki.cs.wisc.edu/index.cgi/wiki?p=SoapWisdom

5. www.cs.wisc.edu/Condor 4 Command Line Tools › Don’t underestimate them! › Your program can create a submit file on disk and simply invoke condor_submit: system(“echo universe=VANILLA > /tmp/condor.sub”); system(“echo executable=myprog >> /tmp/condor.sub”);... system(“echo queue >> /tmp/condor.sub”); system(“condor_submit /tmp/condor.sub”);

6. www.cs.wisc.edu/Condor 5 Command Line Tools › Your program can create a submit file and give it to condor_submit through stdin: PERL:fopen(SUBMIT, “|condor_submit”); print SUBMIT “universe=VANILLA\n”;... C/C++:int s = popen(“condor_submit”, “r+”); write(s, “universe=VANILLA\n”, 17/*len*/);...

7. www.cs.wisc.edu/Condor 6 Command Line Tools › Using the +Attribute with condor_submit: universe = VANILLA executable = /bin/hostname output = job.out log = job.log +webuser = “ zmiller ” queue

8. www.cs.wisc.edu/Condor 7 Command Line Tools › Use -constraint and –format with condor_q: % condor_q -constraint 'webuser=="zmiller"' -- Submitter: bio.cs.wisc.edu : : bio.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 213503.0 zmiller 10/11 06:00 0+00:00:00 I 0 0.0 hostname % condor_q -constraint 'webuser=="zmiller"' -format "%i\t" ClusterId -format "%s\n" Cmd 213503 /bin/hostname

9. www.cs.wisc.edu/Condor 8 Command Line Tools › condor_wait will watch a job log file and wait for a certain (or all) jobs to complete: system(“condor_wait job.log”); › can specify a timeout

10. www.cs.wisc.edu/Condor 9 Command Line Tools › condor_q and condor_status –xml option › So it is relatively simple to build on top of Condor’s command line tools alone, and can be accessed from many different languages (C, PERL, python, PHP, etc). › However…

11. www.cs.wisc.edu/Condor 10 DRMAA › DRMAA is a OGF standardized job- submission API › Has C (and now Java) bindings › Is not Condor-specific › SourceForge Project http://sourceforge.net/projects/condor-ext

12. www.cs.wisc.edu/Condor 11 DRMAA › Unfortunately, the DRMAA 1.x API does not support some very important features, such as:  Fault tolerance  Transactions

13. www.cs.wisc.edu/Condor 12 Condor Database Queue (Condor DBQ) › Layer on top of Condor › Relational database interface to  Submit work to Condor  Monitor status of submission  Monitor status of individual jobs › Perfect for applications that  Submit jobs to Condor  Already use a database

14. www.cs.wisc.edu/Condor 13 Web App Before Condor DBQ Web Application Web Application Condor Pool Condor Pool Schedd DBMS R/W app data R/W app data Submit Job (SOAP or cmd line interface) Submit Job (SOAP or cmd line interface) Check Status (job log file, SOAP, or cmd line interface) Check Status (job log file, SOAP, or cmd line interface) Non- Trivia l Code User log App tables App tables Crash!!! You did implement two phase commit and recovery, to get run once semantics, right? Crash!!! You did implement two phase commit and recovery, to get run once semantics, right?

15. www.cs.wisc.edu/Condor 14 Web App After Condor DBQ Web Application Web Application Condor Pool Condor Pool Schedd DBMS User log R/W app data R/W app data Submi t Job Chec k Statu s condor_dbq Submit Job (cmd line) Submit Job (cmd line) Get Job Updates Check New Work Update Status Check New Work Update Status Single SQL statements Transacti onal Single SQL statements Transacti onal App tables App tables work table work table job table job table

16. www.cs.wisc.edu/Condor 15 Benefits of Condor DBQ › Natural simple SQL API  Submit work insert into work values( condor-submit- file)  Check status select * from jobs where work_id = id › Transactions/Consistency comes for free › DBMS performs crash recovery

17. www.cs.wisc.edu/Condor 16 Condor DBQ Limitations › Overrides log file location › All jobs submitted as same user › Dagman not supported › Only Vanilla and Standard universe jobs supported (others are unknown) › Currently only supports PostgreSQL

18. www.cs.wisc.edu/Condor 17 Web Service Interface › Simple Object Access Protocol  Mechanism for doing RPC using XML (typically over HTTP or HTTPS)  A World Wide Web Consortium (W3C) standard › SOAP Toolkit: Transform a WSDL to a client library

19. www.cs.wisc.edu/Condor 18 Benefits of a Condor SOAP API › Can be accessed with standard web service tools › Condor accessible from platforms where its command-line tools are not supported › Talk to Condor with your favorite language and SOAP toolkit

20. www.cs.wisc.edu/Condor 19 Condor SOAP API functionality › Get basic daemon info (version, platform) › Submit jobs › Retrieve job output › Remove/hold/release jobs › Query machine status › Advertise resources › Query job status

21. www.cs.wisc.edu/Condor 20 Getting machine status via SOAP Your program SOAP library queryStartdAds() condor_collector Machine List SOAP over HTTP

22. www.cs.wisc.edu/Condor 21 Some classifications Application Program Interfaces (APIs) › Job Control › Operational Monitoring Extensions

23. www.cs.wisc.edu/Condor 22 Operational Monitoring APIs › Via Web Services (SOAP) › Via Relational Database: Quill  Job, Machine, and Matchmaking data echoed into PostgreSQL RDBMS › Via a file: the Event Log  Just like the job log, but has events for all jobs submitted to a schedd.  Structured journal of job events  Sample code in C++ to read/parse these events › Via Enterprise messaging: Condor AMQP  Event Log events echoed into Qpid, an AMQP message broker in a highly reliable manner  https://condor- wiki.cs.wisc.edu/index.cgi/wiki?p=CondorPigeon

24. www.cs.wisc.edu/Condor 23 Some classifications Application Program Interfaces (APIs) › Job Control › Operational Monitoring Extensions

25. www.cs.wisc.edu/Condor 24 Extending Condor › APIs: How to interface w/ Condor › Extensions: Changing Condor’s behavior  Hooks  Plugins

26. www.cs.wisc.edu/Condor 25 Job Wrapper Hook › Allows an administrator to specify a “wrapper” script to handle the execution of all user jobs › Set via condor_config “USER_JOB_WRAPPER” › Wrapper runs as the user, command-line args are passed, machine & job ad is available. › Errors can be propagated to the user. › Example: condor_limits_wrapper.sh

27. www.cs.wisc.edu/Condor 26 Job Fetch & Prepare Hooks › Job Fetch hooks  Call outs from the condor_startd  Extend claiming  Normally jobs are pushed from schedd to startd – now jobs can be “pulled” from anywhere › Job Running Hooks  Call outs from the condor_starter  Transform the job classad  Perform any other pre/post logic

28. www.cs.wisc.edu/Condor 27 What hooks are available? › Fetch Hooks (condor_startd):  FETCH_JOB  REPLY_FETCH  EVICT_CLAIM › Running Hooks (condor_starter):  PREPARE_JOB  UPDATE_JOB_INFO  JOB_EXIT

29. www.cs.wisc.edu/Condor 28 HOOK_FETCH_JOB › Invoked by the startd whenever it wants to try to fetch new work  FetchWorkDelay expression › Hook gets a current copy of the slot ClassAd › Hook prints the job ClassAd to STDOUT › If STDOUT is empty, there’s no work

30. www.cs.wisc.edu/Condor 29 HOOK_REPLY_FETCH › Invoked by the startd once it decides what to do with the job ClassAd returned by HOOK_FETCH_WORK › Gives your external system a chance to know what happened › argv[1]: “accept” or “reject” › Gets a copy of slot and job ClassAds › Condor ignores all output › Optional hook

31. www.cs.wisc.edu/Condor 30 HOOK_EVICT_CLAIM › Invoked if the startd has to evict a claim that’s running fetched work › Informational only: you can’t stop or delay this train once it’s left the station › STDIN: Both slot and job ClassAds › STDOUT: > /dev/null

32. www.cs.wisc.edu/Condor 31 HOOK_PREPARE_JOB › Invoked by the condor_starter when it first starts up (only if defined) › Opportunity to prepare the job execution environment  Transfer input files, executables, etc. › INPUT: both slot and job ClassAds › OUTPUT: ignored, but starter won’t continue until this hook exits › Not specific to fetched work

33. www.cs.wisc.edu/Condor 32 HOOK_UPDATE_JOB_INFO › Periodically invoked by the starter to let you know what’s happening with the job › INPUT: both ClassAds  Job ClassAd is updated with additional attributes computed by the starter: ImageSize, JobState, RemoteUserCpu, etc. › OUTPUT: ignored

34. www.cs.wisc.edu/Condor 33 HOOK_JOB_EXIT › Invoked by the starter whenever the job exits for any reason › Argv[1] indicates what happened:  “exit”: Died a natural death  “evict”: Booted off prematurely by the startd (PREEMPT == TRUE, condor_off, etc)  “remove”: Removed by condor_rm  “hold”: Held by condor_hold

35. www.cs.wisc.edu/Condor 34 POP QUIZ!!! Given  Job Wrapper hook  Job Fetch hooks  Job Running hooks Which one is redundent? Why? Quiz? This is so bogus Mr. Todd!!

36. www.cs.wisc.edu/Condor 35 Sidebar: “Toppings”  If work arrived via fetch hook “foo”, then prepare hooks “foo” will be used.  What if an individual job could specify a job prepare hook to use???  Prepare hook to use can be alternatively specified in job classad via attribute “HookKeyword”  How cool is that???

37. www.cs.wisc.edu/Condor 36 Toppings: Simple Example › In condor_config: ANSYS_HOOK_PREPARE_JOB= \ $(LIBEXEC)/ansys_prepare_hook.sh › Contents of ansys_prepare_hook.sh: #!/bin/sh #Read and discard the job classad cat >/dev/null echo'Cmd="/usr/local/bin/ansys"'

38. www.cs.wisc.edu/Condor 37 Topping Example, cont. › In job submit file: universe=vanilla executable=whatever arguments=… +HookKeyword=“ANSYS" queue

39. www.cs.wisc.edu/Condor 38 Job Router Hooks JOB_ROUTER_ENTRIES_CMD - read the routing table from an external program - optional periodic refresh _HOOK_TRANSLATE - transform original job to "routed" job _HOOK_UPDATE_JOB_INFO - periodically update routed job ClassAd _HOOK_JOB_FINALIZE - handle job completion and update original job ClassAd _HOOK_JOB_CLEANUP - handle cleaning up when done managing job

40. www.cs.wisc.edu/Condor 39 Configuration Hook › Instead of reading from a file, run a program to generate Condor config settings › Append “|” to CONDOR_CONFIG or LOCAL_CONFIG_FILE. Example: LOCAL_CONFIG_FILE = \ /opt/condor/sbin/make_config

41. www.cs.wisc.edu/Condor › Allows the administrator to configure hooks for handling URLs during Condor's file transfer › Enables transfer from third party directly to execute machine, which can offload traffic from the submit point › Can be used in a number of clever ways File Transfer Hooks

42. www.cs.wisc.edu/Condor File Transfer Hooks › API is extremely simple › Must support being invoked with the “-classad” option to advertise its abilities: #!/bin/env perl if ($ARGV[0] eq "-classad") { print "PluginType = \"FileTransfer\"\n"; print "SupportedMethods = \"http,ftp,file\"\n"; exit 0; }

43. www.cs.wisc.edu/Condor File Transfer Hooks › When invoked normally, a plugin simply transfers the URL (first argument) into filename (second argument) # quoting could be an issue but this runs in user space $cmd = "curl ". $ARGV[0]. " -o ". $ARGV[1]; system($cmd); $retval = $?; exit $retval;

44. www.cs.wisc.edu/Condor File Transfer Hooks › In the condor_config file, the administrator lists the transfer hooks that can be used › Condor invokes each one to find out its abilities › If something that looks like a URL is added to the list of input files, the plugin is invoked on the execute machine

45. www.cs.wisc.edu/Condor File Transfer Hooks › condor_config:  FILETRANSFER_PLUGINS = curl_plugin, hdfs_plugin, gdotorg_plugin, rand_plugin › Submit file:  transfer_input_files = normal_file, http://cs.wisc.edu/~zkm/data_file, rand://1024/random_kilobyte

46. www.cs.wisc.edu/Condor File Transfer Hooks › As you can see, the format of the URL is relatively arbitrary and is interpreted by the hook › This allows for tricks like rand://, blastdb://, data://, etc.

47. www.cs.wisc.edu/Condor 46 Plugins

48. www.cs.wisc.edu/Condor 47 Plugins › Shared Library Plugins  Gets mapped right into the process space of the Condor Services! May not block! Must be thread safe!  General and ClassAd Functions › Condor ClassAd Function Plugin  Add custom built-in functions to the ClassAd Language.  Via condor_config “CLASSAD_LIB_PATH”  Cleverly used by SAMGrid

49. www.cs.wisc.edu/Condor 48 General Plugins › In condor_config, use “PLUGINS” or “PLUGIN_DIR”. › Very good idea to do:  SUBSYSTEM.PLUGIN or  SUBSYSTEM.PLUGIN_DIR › Implement C++ child class, and Condor will call methods at the appropriate times. › Some general methods (initialize, shutdown), and then callbacks based on plugin type › What’s available? Plugin Discovery…

50. www.cs.wisc.edu/Condor 49 Plugin Discovery cd src/ dir /s Example*Plugin.cpp You will find: ExampleCollectorPlugin.cpp ExampleMasterPlugin.cpp ExampleNegotiatorPlugin.cpp ExampleClassAdLogPlugin.cpp ExampleScheddPlugin.cpp ExampleStartdPlugin.cpp And a ClassAdLogPluginManager.cpp

51. www.cs.wisc.edu/Condor 50 Collector Plugin struct ExampleCollectorPlugin : public CollectorPlugin { void initialize(); void shutdown(); void update(int command, const ClassAd &ad); void invalidate(int command, const ClassAd &ad); };

52. www.cs.wisc.edu/Condor 51 ClassAdLog Plugin Methods virtual void newClassAd(const char *key) = 0; virtual void destroyClassAd(const char *key) = 0; virtual void setAttribute(const char *key, const char *name, const char *value) = 0; virtual void deleteAttribute(const char *key, const char *name) = 0;

53. www.cs.wisc.edu/Condor 52 Other Extending Ideas…

54. www.cs.wisc.edu/Condor 53 Custom ClassAd Attributes › Job ClassAd  +Name = Value in submit file  SUBMIT_EXPRS in condor_config › Machine ClassAd  STARTD_EXPRS in condor_config for static attributes  STARTD_CRON_* settings in condor_config for dynamic attributes

55. www.cs.wisc.edu/Condor 54 Thinking out of the box… › MAIL in condor_config › Green Computing Settings  HIBERNATION_PLUGIN (called by the startd)  ROOSTER_WAKEUP_CMD

56. www.cs.wisc.edu/Condor 55 All else fails? Grab Source! Condor is open source ya know… Thank you! Questions?

57. www.cs.wisc.edu/Condor 56 Extra Slides

58. www.cs.wisc.edu/Condor 57 Lets get some SOAP details…

59. www.cs.wisc.edu/Condor 58 The API › Core API, described with WSDL, is designed to be as flexible as possible  File transfer is done in chunks  Transactions are explicit › Wrapper libraries aim to make common tasks as simple as possible  Currently in Java and C#  Expose an object-oriented interface

60. www.cs.wisc.edu/Condor 59 Things we will cover › Condor setup › Necessary tools › Job Submission › Job Querying › Job Retrieval › Authentication with SSL and X.509

61. www.cs.wisc.edu/Condor 60 Condor setup › Start with a working condor_config › The SOAP interface is off by default  Turn it on by adding ENABLE_SOAP=TRUE › Access to the SOAP interface is denied by default  Set ALLOW_SOAP and DENY_SOAP, they work like ALLOW_READ/WRITE/…  Example: ALLOW_SOAP=*/*.cs.wisc.edu

62. www.cs.wisc.edu/Condor 61 Necessary tools › You need a SOAP toolkit  Apache Axis (Java) - http://ws.apache.org/axis/  Microsoft.Net - http://microsoft.com/net/  gSOAP (C/C++) - http://gsoap2.sf.net/  ZSI (Python) - http://pywebsvcs.sf.net/  SOAP::Lite (Perl) - http://soaplite.com/ › You need Condor’s WSDL files  Find them in lib/webservice/ in your Condor release › Put the two together to generate a client library  $ java org.apache.axis.wsdl.WSDL2Java condorSchedd.wsdl › Compile that client library  $ javac condor/*.java All our examples are in Java using Apache Axis

63. www.cs.wisc.edu/Condor 62 Client wrapper libraries › The core API has some complex spots › A wrapper library is available in Java and C#  Makes the API a bit easier to use (e.g. simpler file transfer & job ad submission)  Makes the API more OO, no need to remember and pass around transaction ids › We are going to use the Java wrapper library for our examples  You can download it from http://www.cs.wisc.edu/condor/birdbath/birdbath.jar

64. www.cs.wisc.edu/Condor 63 Submitting a job › The CLI way… universe = vanilla executable = /bin/cp arguments = cp.sub cp.worked should_transfer_files = yes transfer_input_files = cp.sub when_to_transfer_output = on_exit queue 1 $ condor_submit cp.sub cp.sub: Explicit bits clusterid = X procid = Y owner = matt requirements = Z Implicit bits

65. www.cs.wisc.edu/Condor 64 Repeat to submit multiple jobs in a single cluster Repeat to submit multiple clusters The SOAP way… 1.Begin transaction 2.Create cluster 3.Create job 4.Send files 5.Describe job 6.Commit transaction Submitting a job

66. www.cs.wisc.edu/Condor 65 1. Begin transaction 2. Create cluster 3. Create job 4&5. Send files & describe job 6. Commit transaction Schedd schedd = new Schedd(“http://…”); Transaction xact = schedd.createTransaction(); xact.begin(30); int cluster = xact.createCluster(); int job = xact.createJob(cluster); File[] files = { new File(“cp.sub”) }; xact.submit(cluster, job, “owner”, UniverseType.VANILLA, “/bin/cp”, “cp.sub cp.worked”, “requirements”, null, files); xact.commit(); Submission from Java

67. www.cs.wisc.edu/Condor 66 Schedd’s location Max time between calls (seconds) Job owner, e.g. “matt” Requirements, e.g. “OpSys==\“Linux\”” Extra attributes, e.g. Out=“stdout.txt” or Err=“stderr.txt” Schedd schedd = new Schedd(“http://…”); Transaction xact = schedd.createTransaction(); xact.begin(30); int cluster = xact.createCluster(); int job = xact.createJob(cluster); File[] files = { new File("cp.sub") }; xact.submit(cluster, job, “owner”, UniverseType.VANILLA, “/bin/cp”, “cp.sub cp.worked”, “requirements”, null, files); xact.commit(); Submission from Java

68. www.cs.wisc.edu/Condor 67 Querying jobs › The CLI way… $ condor_q -- Submitter: localhost : : localhost ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1.0 matt 10/27 14:45 0+02:46:42 C 0 1.8 sleep 10000 … 42 jobs; 1 idle, 1 running, 1 held, 1 unexpanded

69. www.cs.wisc.edu/Condor 68 Also, getJobAds given a constraint, e.g. “Owner==\“matt\”” String[] statusName = { “”, “Idle”, “Running”, “Removed”, “Completed”, “Held” }; int cluster = 1; int job = 0; Schedd schedd = new Schedd(“http://…”); ClassAd ad = new ClassAd(schedd.getJobAd(cluster, job)); int status = Integer.valueOf(ad.get(“JobStatus”)); System.out.println(“Job is “ + statusName[status]); Querying jobs › The SOAP way from Java…

70. www.cs.wisc.edu/Condor 69 Retrieving a job › The CLI way.. › Well, if you are submitting to a local Schedd, the Schedd will have all of a job’s output written back for you › If you are doing remote submission you need condor_transfer_data, which takes a constraint and transfers all files in spool directories of matching jobs

71. www.cs.wisc.edu/Condor 70 Discover available files Remote file Local file Retrieving a job › The SOAP way in Java… int cluster = 1; int job = 0; Schedd schedd = new Schedd(“http://…”); Transaction xact = schedd.createTransaction(); xact.begin(30); FileInfo[] files = xact.listSpool(cluster, job); for (FileInfo file : files) { xact.getFile(cluster, job, file.getName(), file.getSize(), new File(file.getName())); } xact.commit();

72. www.cs.wisc.edu/Condor 71 Authentication for SOAP › Authentication is done via mutual SSL authentication  Both the client and server have certificates and identify themselves › It is not always necessary, e.g. in some controlled environments (a portal) where the submitting component is trusted › A necessity in an open environment -- remember that the submit call takes the job’s owner as a parameter  Imagine what happens if anyone can submit to a Schedd running as root…

73. www.cs.wisc.edu/Condor 72 Details on setting up authenticated SOAP over HTTPS

74. www.cs.wisc.edu/Condor 73 Authentication setup › Create and sign some certificates › Use OpenSSL to create a CA  CA.sh -newca › Create a server cert and password-less key  CA.sh -newreq && CA.sh -sign  mv newcert.pem server-cert.pem  openssl rsa -in newreq.pem -out server-key.pem › Create a client cert and key  CA.sh -newreq && CA.sh -sign && mv newcert.pem client-cert.pem && mv newreq.pem client-key.pem

75. www.cs.wisc.edu/Condor 74 Authentication config › Config options…  ENABLE_SOAP_SSL is FALSE by default  _SOAP_SSL_PORT Set this to a different port for each SUBSYS you want to talk to over ssl, the default is a random port Example: SCHEDD_SOAP_SSL_PORT=1980  SOAP_SSL_SERVER_KEYFILE is required and has no default The file containing the server’s certificate AND private key, i.e. “keyfile” after cat server-cert.pem server-key.pem > keyfile

76. www.cs.wisc.edu/Condor 75 Authentication config › Config options continue…  SOAP_SSL_CA_FILE is required The file containing public CA certificates used in signing client certificates, e.g. demoCA/cacert.pem › All options except SOAP_SSL_PORT have an optional SUBSYS_* version  For instance, turn on SSL for everyone except the Collector with ENABLE_SOAP_SSL=TRUE COLLECTOR_ENABLE_SOAP_SSL=FALSE

77. www.cs.wisc.edu/Condor 76 One last bit of config › The certificates we generated have a principal name, which is not standard across many authentication mechanisms › Condor maps authenticated names (here, principal names) to canonical names that are authentication method independent › This is done through mapfiles, given by SEC_CANONICAL_MAPFILE and SEC_USER_MAPFILE › Canonical map: SSL. *emailAddress=(.*)@cs.wisc.edu.* \1 › User map: (.*) \1 › “SSL” is the authentication method, “.*emailAddress….*” is a pattern to match against authenticated names, and “\1” is the canonical name, in this case the username on the email in the principal

78. www.cs.wisc.edu/Condor 77 HTTPS with Java › Setup keys…  keytool -import -keystore truststore -trustcacerts -file demoCA/cacert.pem  openssl pkcs12 -export -inkey client-key.pem -in client- cert.pem -out keystore › All the previous code stays the same, just set some properties  javax.net.ssl.trustStore, javax.net.ssl.keyStore, javax.net.ssl.keyStoreType, javax.net.ssl.keyStorePassword  Example: java -Djavax.net.ssl.trustStore=truststore - Djavax.net.ssl.keyStore=keystore - Djavax.net.ssl.keyStoreType=PKCS12 - Djavax.net.ssl.keyStorePassword=pass Example https://…

79. www.cs.wisc.edu/Condor 78 Quill › Condor operational data mirrored into an RDBMS  PostgreSQL › Job, machine, historical info › Read-only QuillSchedd Job Queue log RDBMS Startd … Master Queue + History Tables

80. www.cs.wisc.edu/Condor 79 Condor Perl Module › Perl module to parse the “job log file” › Can use instead of polling w/ condor_q › Call-back event model › (Note: job log can be written in XML)

81. www.cs.wisc.edu/Condor Event Logging Condor can generate an “Event Log”:  This is a log of significant events about the life of the jobs on a system.  Competing objectives:  Limit the amount of space used by the logging, so that event logging doesn’t become a problem itself  Never “drop” events  C++ reader library