LHCb Software week 24-26 November, 1999

1
N-Tuples within Gaudi

• Definition
• Usage
• Run through simplified example
• The persistent back-end

2
What are we talking about?
3
What was an N-Tuple in CERNLIB

• Collection of identifiable n-dim. primitive
  items
• LOGICAL4, INTEGER4, REAL4, REAL8
• See HBOOK
• Row wise N-Tuples
• REAL4 array(SOME_DIMENSION)
• Items are single numbers identified by tags
• Column wise N-Tuples
• Structure mapped to a common block
• Items (number, array, ) are identified by tags
• Allow for variable size arrays

4
What is an N-Tuple in Gaudi

• Collection of N-tuple Items
• Items represent primitives bool, long, float,
  double
• N-tuples reside in a data store
• Can be accessed like any other DataObject
• Data management (e.g. writing records) is done by
  the N-tuple service

Things did not change dramatically
5
Whats the job Wishes

• Stick to architecture
• We want to independent from the back-end
  storage mechanism (HBOOK, ROOT, OBJY...)
• Cernlib will die
• Encapsulate implementation Interfaces or ABC
• Easy to use
• Items must behave like ordinary numbers
• Must support basic arithmetic operations and type
  conversions
• N-tuples should not go all into one single file
• possibly, but not always
• Connect several streams to the data store

6
The N-Tuple Data Store

• Top Entry point /NTUPLES
• Logical file names
• User defined
• First useable directory/MC/REC/USER
• Directory structure
• N-tuples
• 1,2,3,4

7
The Example

• Simplified version of GaudiExample/Ntuples
• Explain how to write and read a column wise
  N-tuple
• Differences to row wise N-tuples are minor
• Only record structure differs
• N-tuples should be easy to use
• Tell me if you think stuff is overcomplicated
• I do not use them daily

8
Easy to Use (Hopefully)

• Define data itemse.g. as members of filling
  Algorithm
• Items are based on primitives
• bool, long, float, double

// Items for n-tuple // _________________
// // 0 dimensional item (number)
NTupleItemltlonggt m_ntrk // 1
dimensional items (Arrays) NTupleArrayltfloatgt
m_px, m_py, m_pz // 2 dimensional
items (Matrices) NTupleMatrixltlonggt
m_hits
9
Easy to Use (Hopefully)

• Book and add the items you later want to fill

• Book the N-tuple

NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1")
if ( !nt ) nt ntupleService()-gtbook("/NTU
PLES/MC/1, CLID_ColumnWiseTuple,
"Hello World")
NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1")
if ( !nt ) nt ntupleService()-gtbook("/NTU
PLES/MC/1, CLID_ColumnWiseTuple,
"Hello World") if ( nt ) status
nt-gtaddItem ("Ntrack", m_ntrk, 0, 5000 )
status nt-gtaddItem ("px", m_ntrk, m_px)
// m_hits0m_ntrk05 numbers within 0,8
status nt-gtaddItem ("hit", m_ntrk, m_hits,
5, 0, 8 )

• Add items

10
Easy to Use (Hopefully)

• Fill all items and commit record
• Simple calculations can be done directly using
  the items

• Do not overrun dimensions

for(m_ntrk0 m_ntrk lt numTracks m_ntrk)
if ( m_ntrk gt m_ntrk-gtrange().distance() )
break
for(m_ntrk0 m_ntrk lt numTracks m_ntrk)
if ( m_ntrk gt m_ntrk-gtrange().distance() )
break m_pxm_ntrk trackm_ntrk-gtpx()
m_hitsm_ntrk0 trackm_ntrk-gtadcValue(0)
... m_hitsm_ntrk4 trackm_ntrk-gtadcV
alue(4)
for(m_ntrk0 m_ntrk lt numTracks m_ntrk)
if ( m_ntrk gt m_ntrk-gtrange().distance() )
break m_pxm_ntrk trackm_ntrk-gtpx()
m_hitsm_ntrk0 trackm_ntrk-gtadcValue(0)
... m_hitsm_ntrk4 trackm_ntrk-gtadcV
alue(4) // Commit record (here of a column
wise N-tuple) status ntupleService()-gtwriteRecor
d("/NTUPLES/MC/1")

• Fill items

• Reset values to defaults
• Is this good?

• Commit record

11
Easy to Use (Hopefully)

• At the end of the job
• All N-tuples will automatically be closed
• The directory structure of the data store is
  reflected by RZ directories
• Access N-tuple for reading in the same way as
  for writing
• Dont forget to cd //ltlungt/ltdirgt in PAW

12
Easy to Use (Hopefully)

• Will open the corresponding logical file
• Search for the directory
• Load the N-tuple

• Reading back (or with PAW)

NTupleItemltlonggt ntrk
NTupleArrayltfloatgt px, py, pz
NTupleDirPtr nt(ntupleService(),
"/NTUPLES/MC/1") if ( nt )
NTupleItemltlonggt ntrk
NTupleArrayltfloatgt px, py, pz
NTupleDirPtr nt(ntupleService(),
"/NTUPLES/MC/1") if ( nt ) status
nt1-gtitem ("Ntrack", ntrk ) status
nt1-gtitem ("px", px) ...
NTupleItemltlonggt ntrk
NTupleArrayltfloatgt px, py, pz
NTupleDirPtr nt(ntupleService(),
"/NTUPLES/MC/1") if ( nt ) status
nt1-gtitem ("Ntrack", ntrk ) status
nt1-gtitem ("px", px) ... do
float sumPx 0 if(ntupleService()-gtreadRecor
d(nt.ptr()).isSuccess()) for ( long j 0
j lt ntrk j ) sumPx pxj while
( status.isSuccess() )

• Assign the entries to the N tuple

• Loop over records

13
Easy to Use (Hopefully)

• Configuration (using job options file)
• Reading N-tuples

NTupleSvc.Input MCtuple1.hbook",
USERtuple2.hbook" // Persistency
type of the N-tuple service 6HBOOK NTupleSvc.Typ
e 6

• Writing N-tuples

NTupleSvc.Output MCtuple1.hbook",
USERtuple2.hbook" // Persistency
type of the N-tuple service 6HBOOK NTupleSvc.Typ
e 6
14
Back-End HBOOK Limitations

• Limited type information
• Cannot define all C data types
• Therefor only bool, long, unsigned long, float
  and double
• NOT char, short, int (unsigned)
• Does not affect disk space HBOOK compresses
  internally
• N-tuple name is an integer
• integer translates later to HBOOK ID
• Must be constant independent of booking time for
  kumacs
• Do not mix up N-tuple IDs and Histogram Ids
• There is only one PAWC common block!

15
Back-End HBOOK Limitations

• Only one index variable per item
• RZ directory names are CHARACTER8
• and upper case ONLY
• double/REAL8 data
• PAW has trouble better dont use
• REAL8 must be quadword aligned (64 bits)
• Padding must be done by user
• Not more than 1000 REAL8 entries per N tuple
• Row wise N-tuples
• Type information is stored as first event
• Start reading at the second...

16
Other Back-Ends

• The Present HBOOK
• PAW is the most widely used viewer
• N-tuples dont help if they cant be analyzed
  interactively
• The Future (?) ROOT I/O
• Will give ROOT a try as back-end
• Implementation with trees/branches is straight
  forward
• Would solve probably all the limitations of HBOOK

17
Conclusions

• It is possible to define dynamic structures
  (N-tuples) with Gaudi
• If you think the usage is too complicated,
  please say so!
• Otherwise it will never change
• Depending on the back-end, limitations invade
  transient side

18
The Back-End HBOOK

• (At least for the time beeing)
• Items must be contigous in memory
• Array of reals
• Common block
• Must be exported in detail where ever a variable
  has to be filled

PARAMETER (MAXTRK 100, MAXHIT 300)
COMMON /CMTRK/ NTRACK, PX(MAXTRK), PY(MAXTRK),
PZ(MAXTRK),
HITS(MAXHIT,MAXTRK) CALL HBNAME(ID,'VARBLOK2
',NTRACK, 'NTRACK0,100, '//
'PX(NTRACK), PY(NTRACK),
PZ(NTRACK), HITS(300,NTRACK))
19
Expected Functionality ?

• Like a library
• Store objects for later use by clients
• Retrieve objects when needed

Make only selectedobjects persistent

• Traverse the content (using Agents)
• Retrieve collections

• Ownership
• Responsibility for cleanup

20
Structure of the data store

• Tree - similar to file system

• Identification by logical addresses
  /Event/Mc/MCEcalHits