1. IRMIS Crawlers Tutorial
Presentation by
Debby Quock
(Janet Anderson)
IRMIS Workshop
EPICS Collaboration Meeting 2008
INFN - Laboratori Nazionali Di Legnaro

2. IRMIS2 User Interface User Viewers User Editors Global Search Tool AOI
IOC
PLC
Component Type
Network Info
Racks Info
Controls Spares
Component History
User Editors
Installed Components
Cables

3. Other ways of getting data into the IRMIS database…
Other ways of getting data into the IRMIS database… …Automated Crawlers
7 Main Crawlers to Date
ADL
EPICS user displays
ALH
EPICS alarm handler
AOI
Applications Organizing Index
Network PV
SDDS
Self-Describing Data Sets
Sequence
EPICS sequence programs

4. IRMIS2 Database Schema 76 IRMIS Tables
aoi
component_semaphore
fld
aoi_crawler
component_state
fld_type
aoi_criticality
component_state_category
machine
aoi_document
component_type
mfg
aoi_epics_record
component_type_document
person
aoi_ioc_stcmd_line
component_type_function
person_group
aoi_machine
component_type_if
plc
aoi_note
component_type_if_type
plc_old
aoi_plc_stcmd_line
component_type_person
port_pin
aoi_status
component_type_status
port_pin_designator
aoi_techsys
conductor
port_pin_template
aoi_topdisplay
criticality_type
port_pin_type
aps_component
doc_type
rec
aps_ioc
rec_client
audit_action
rec_client_type
audit_action_type
form_factor
rec_type
base_component_type
function
rec_type_dev_sup
cable
group_name
role
component
ioc
role_name
component_instance
ioc_boot
server
component_instance_state
ioc_error
technical_system
component_port
ioc_error_message
uri
component_port_template
ioc_resource
vuri
component_port_type
ioc_resource_type
vuri_rel
component_rel
ioc_stcmd_line
component_rel_type
76 IRMIS Tables
19 tables automatically populated by crawlers

5. IRMIS Crawlers Software Framework
Blowfish database connection encryption
Logs
Cronjob execution
SHELL=/bin/sh
0,30 * * * * /local/home/IRMIS/pvcrawler/run_crawler.sh
20 1 * * 6 /local/home/IRMIS/alhcrawler/run_crawler.sh
25 2 * * 6 /local/home/IRMIS/adlcrawler/run_crawler.sh
50 1 * * 6 /local/home/IRMIS/sddscrawler/run_crawler.sh
15,45 * * * * /local/home/IRMIS/sddsPVdump/run_crawler.sh
10 8 * * * /local/home/IRMIS/networkcrawler/run_crawler.sh
0 4 * * 6 /local/home/IRMIS/rec_client_mainenance/run_clean.sh
Command line execution
Example: PV Crawler
pv_crawler –go –boot-scan=<module-name> [--no-sequence-scan] [--key=<16 char key>] [--test] [--force] [--help] [--debug-level=…]
Perl modules
Example: ADL Crawler
ADLCrawlerDBLayer.pm
ADLCrawlerParser.pm
adl_crawler.pl
APSAdlDirectoryFile
APSEdpDirectoryFile
Blowfish_PP.pm
db.properties
LogUtil.pm
MySQLDBFunctions.pm
OracleDBFunctions.pm
run_crawler.sh

6. IRMIS Crawler Requirements
Perl version 5
5.8.8 in use at APS
Perl modules
DBI (Database Interface module)
DBD::mysql (MySQL driver for DBI)
MySQL version 5
in use at APS
EPICS Macro Substitution and Include (MSI) tool
Reasonably consistent IOC boot process
Some ability to write/edit Perl code
To create *BootScan module
To handle site-specific boot peculiarities

7. IRMIS PV Crawler Mechanics
db.properties D B I
pv_crawler.pl
path.properties
ioc
*BootScan
Parser
ioc_boot, ioc_resource
rec,fld,…
MySQL
bootparams
st.cmd
dbd db
IOC boot file system(s)

8. IRMIS PV Crawler Mechanics, Cont.
Once every 30 minutes, the PV crawler:
Identifies IOCs to be scanned and their last known boot date (i.e., APSBootScan.pm or SNSBootScan.pm)
For each IOC
Checks for new boot date
If so, insert new ioc_boot record in db and proceed
If not, skip to next IOC
Parses st.cmd (collecting all info on dbd and db files)
Checks for any dbd or db file time mods
If so, insert all pv and field data in db for IOC

9. IRMIS PV Crawler Core DB Schema
PV Crawler modifies tables:
ioc
ioc_boot
ioc_error
ioc_resource
rec
rec_type
rec_type_device_support
fld
fld_type
uri

10. IRMIS PV Crawler Stored Information
Boot history
Each new IOC boot is recorded with date, regardless of whether any configuration has changed
If there is an identifiable problem with the boot, or inability to parse its configuration, an associated error is recorded
File system resources (only if changed from last crawl)
Path to each dbd and db file that make up configuration, along with file modified date and any text substitutions applied
Record type definitions (dbd) (only if changed from last crawl)
The set of fields defined for each record type at that time
Record definitions (db) (only if changed from last crawl)
Record name, fields and their default/overriden values

11. Example PV Crawler Log File Output
I| |Processing ioc iocbdg2, boot date:
I| |Parsing /usr/local/iocapps/R /ioc/booster/3/iocBoot/iocbdg2/st.cmd
I| |IOC Resources changed: 0, Parse error: 0
I| |Write data to DB
I| |No sequences found for this IOC
I| |SEQCrawlerDBLayer: update_db_tables finished.
I| |Processing ioc iocbol, boot date:
I| |Parsing /usr/local/iocapps/R /ioc/booster/3/iocBoot/iocbol/st.cmd
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M1, name=mInitMEM1"
I| |SEQ_PV Sequence source file is /usr/local/iocapps/R /ioc/booster/3/bolApp/src/motorInit.st
I| |SEQ_PV Number of sequencer pvs is 17
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M2, name=mInitMEM2"
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M3, name=mInitMEM3"
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M4, name=mInitMEM4"
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M5, name=mInitHEM5"
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M6, name=mInitHEM6"
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M7, name=mInitHEM7"
I| |SEQ_PV seq command is seq &motorInit, "M=BOO:OptLab:M8, name=mInitHEM8"
I| |IOC Resources changed: 0, Parse error: 0
I| |Write data to DB
I| |SEQCrawlerDBLayer: update_db_tables finished.
I| |Processing ioc iocbramp, boot date:

12. AOI Crawler Discovered Attributes
AOI Crawler parses IOC EPICS startup command files to retrieve information about:
- UPCs (IOCs, PLCs, LabView, ...) associated with each AOI
- Process Variables associated with each AOI
- EPICS startup command lines for later viewing directly through IRMIS
sequence programs ld < mpsTesterSingleLatchCard.o
configuration data MpcConfig 0,0x3100,80,3
Parsing requires unique AOI comment lines in the EPICS IOC startup command files:
# load one or more databases here
#<aoi name="aoi_linac_llrf_control_l6">
dbLoadRecords "linacApp/llrfDb/llrf6.db"
#</aoi>
#<aoi name="aoi_linac_rf_modulator_l6">
dbLoadRecords "linacApp/modulatorDb/modulator6.db"

13. AOI Crawler DB Schema AOI Crawler updates tables: aoi_ioc_stcmd_line
aoi_plc_stcmd_line
ioc_stcmd_line
aoi_epics_record
aoi_crawler

14. Sequence Crawler Discovered Attributes
In IOC startup command files, searches for script lines:
ld (load)
registerRecordDeviceDriver
#! (shebang)
seq (sequence)
Utilizes information:
IOC architecture
executable sequence file
sequence library file
EPICS database load file
Performs tasks:
Marks sequence records as old by setting table field rec_client::current_load =0 for any records associated with the current IOC
Inserts new rows into tables uri, vuri, vuri_rel, and rec_client. Table rec_client holds PV name and points to table vuri. Table uri holds the file name for the sequence program.

15. Sequence Crawler DB Schema

16. ADL Crawler Discovered Attributes
Parses MEDM adl files and looks for PV names
Modifies tables rec_client, uri, vuri, vuri_rel with adl file names and PV names
Execution command
adl_crawler.pl --go --adl-dirs=APSAdlDirectoryFile --edp-dirs=APSEdpDirectoryFile --key=********** >> logs/crawler.log 2>&1

17. ALH Crawler Discovered Attributes
Parses alarm handler alh config files for IOC record references
Writes file name and PV name to tables rec_client, uri, vuri
Execution Command
alh_crawler --go –alh-dirs=<file-name> [--key=<16 char key>] [--test] [--help] [--debug-level=[verbose | debug] info]]

18. Network Crawler Discovered Attributes
Reads a file of network switch interface data and determines if there are changes to the IOC primary and secondary Ethernet switch, blade, and port settings
Writes IOC network interface data located in file accel_sw to IRMIS database table aps_ioc
Network data file accel_sw is updated nightly by IT process
Network Crawler is run once every morning

19. SDDS Crawler Discovered Attributes
Parses SDDS files and writes any ControlName column to the IRMIS rec_client table
Modifies tables rec_client, uri, vuri
Execution Command
sdds_crawler -- go –sdds-files=<file-name> --sdds-app-name=<app-name> [--key=<16 char key>] [--test] …

20. IRMIS Crawlers Future Enhancements
Merge the AOI and PV crawlers into one
Code reuse and code maintenance
Separate historical EPICS PVs from current PVs
Leaning towards separate tables for holding historical data
Crawler will push PV data into historical tables when an IOC boot change has been detected
New crawlers
IOC Save/Restore
Controlled Access System (CAS)
Enhancements to existing crawlers
notification
Notification to IOC cognizants in the event of Crawler discovery of IOC-related errors. Same for AOIs.
ADL Crawler
Revise ADL crawler to search for obsolete PVs and search adl files for "adl cognizant". Notify cognizant via that adl file needs updating to remove or change obsolete PVs.

21. Acknowledgements Don Dohan, Brookhaven National Laboratory
Ned Arnold and Claude Saunders, Argonne National Laboratory
Mariana Varotto, former ANL employee
Ron MacKenzie and Judy Rock, Stanford Linear Accelerator Center