IRMIS PV Crawler

1
IRMIS PV Crawler

• Auto-populating the IRMIS PV Schema

2
IRMIS PV Crawler

• Requirements
• Perl 5 with DBI and DBDmysql modules
• MySQL 4.x (4.1 preferred)
• EPICS msi tool (for now)
• Reasonably consistent ioc boot process
• Some ability to write/edit perl code
• To create BootScan module
• To handle site-specific boot peculiarities

3
IRMIS PV Crawler
D B I
db.properties
pv_crawler.pl
path.properties
ioc
BootScan
Parser
Ioc_boot, ioc_resource
Rec,fld,
MySQL
bootparams st.cmd dbd db
IOC boot file system(s)
4
IRMIS PV Crawler

• What pv_crawler does (every 5 minutes)
• Identify iocs to be scanned and their last known
  boot date (ie. APSBootScan.pm or SNSBootScan.pm)
• foreach ioc
• Check for new boot date
• If so, insert new ioc_boot record in db and
  proceed
• If not, skip to next ioc
• Parse st.cmd (collecting all info on dbd and db
  files)
• Check for any dbd or db file time mods
• If so, insert all pv and field data in db for ioc

5
IRMIS PV Crawler

• Sample output from test run

bacchussaundersgt pv_crawler.pl --go
--boot-scanAPSBootScan --test --debug-leveldebug
I20050304105259pv crawler beginning usage msi
-V -Ipath ... -Msub ... -Ssubfile template
Specifying path will replace the default '.'
stdin is used if template is not
given D20050304105259PVCrawlerDBLayer begin
ioc_find D20050304105259iocs to be boot
scanned iocpar01 I20050304105259Processing ioc
iocpar01, boot date 20050118153007 I200503041052
59Parsing /usr/local/iocapps/R3.13.10/ioc/par/3/i
ocBoot/iocpar01/st.cmd D20050304105259script
mod time 20050114070431 D20050304105259sl lt
cdCommands D20050304105259script mod time
20050223113545 D20050304105259sl startup
"/net/helios/iocapps/R3.13.10/ioc/par/3/iocBoot/io
cpar01" lt------cut--------gt D20050304105259sl
dbLoadDatabase("dbd/par01App.dbd") D2005030410525
9parse_function_call D20050304105259function
dbLoadDatabase with args ("dbd/par01App.dbd") D20
050304105259opening /net/helios/iocapps/R3.13.10/
ioc/par/3/dbd/par01App.dbd D20050304105259sl
dbLoadRecords "par01App/commonDb/par01status.db" D
20050304105259parse_function_call D200503041052
59function dbLoadRecords with args
"par01App/commonDb/par01status.db" D2005030410525
9calling parse_db with /net/helios/iocapps/R3.13.
10/ioc/par/3/par01App/commonDb/par 01status.db lt--
----cut--------gt D20050304105259check_for_file_m
ods() no record found in db, so return
1 I20050304105259IOC Resources changed 1,
Parse error 0 I20050304105259Write data to
DB I20050304105259pv crawler done
6
IRMIS PV Crawler

• What info do we store over time for a given IOC?
• Boot history
• Each new ioc boot is recorded with date,
  regardless of whether any configuration has
  changed
• If there is an identifiable problem with the
  boot, or inability to parse its configuration,
  an associated error is recorded
• File system resources (only if changed from last
  crawl)
• Path to each dbd and db file that make up
  configuration, along with file modified date and
  any text substitutions applied
• Record type definitions (dbd) (only if changed
  from last crawl)
• The set of fields defined for each record type at
  that time
• Record definitions (db) (only if changed from
  last crawl)
• Record name, fields and their default/overriden
  values

7
IRMIS PV Crawler
rdbCore PV Schema
8
IRMIS PV Crawler

• Experience to date
• Running continuously for weeks now
• No involvement of ioc developers time
• All but 3 of 265 IOCs crawled successfully
  (including soft iocs)
• Continual crawler parser improvement required,
  but has settled down to trivial cases now.
• Crawler log raised useful issues for about 16
  IOCs
• Iocs registered as active in ioc table, but were
  not
• Some file system resources unreachable due to
  perms
• Identified iocs booting out of home directory
• Identified references to non-existent st.cmd,
  dbd, db files
• Fields defined in db files that dont exist in
  record type
• Identified iocs outside of general boot
  guidelines
• Duplicate record definitions