DBM Files and Persistence

1
DBM Files and Persistence

• Persistence
• DBM files and DB_File
• Storable.pm, MLDBM, and storing complex data
  structures to disk

2
Persistence

• What if the power cord gets unplugged while the
  program is running??
• Many times we need to make the data live longer
  than the application this is called persistence
• the data persists between program invocations
• There are many ways to do this, from flat text
  files to DBM files to relational databases

3
Flat files

• Simply a text file saved to disk.
• Easy to implement, portable.
• Read in data from a flat file and write it back
  out at the end of the program.
• If it's a large data set, you either use a lot of
  memory to read the whole file into memory, or
  have to walk through the file each time you want
  to do a lookup

4
About the DBM alternative

• DBM is just a disk-based hash table (instead of
  memory-based like in a normal Perl script)
• Keys/values just like a Perl hash
• Lookups are very fast!
• Saves memory since you don't have to read in the
  whole data set
• Data persists between program invocations

5
More on DBM

• Many variations
• SDBM (comes with Perl),
• NDBM,
• GDBM (from Gnu free software foundation),
• DB_File
• For comparisons between variants, see
  http//www.mincom.com/mtr/perl/lib/AnyDBM_File.htm
  l
• Files produced by each type of DBM are NOT
  interchangable

6
Using a DBM file

• To use, simply tie a normal in-memory hash (foo)
  to a disk-based hash (foo.db) using the tie
  function (more on tie soon)

7
DBM Format

• Here's the generic format for setting up the DBM
  file
• tie hash, DBM_TYPE, FILENAME, OPEN_FLAG(S),
  MODE
• DBM_TYPE is the type of DBM file to use (DB_File,
  SDBM, etc)
• tie binds a variable to a class (more on classes
  in OO).
• You can think of tie as connecting your hash
  hash to the file FILENAME transparently.
• Once tied, you can use hash just like any other
  hash.

8
Open Flags

• O_RDONLY for readonly access.
• Use 0444 for the mode.
• O_RDWR for read/write access.
• Use 0666 for the mode.
• O_CREAT to create a new db.
• Usually only used in conjunction
• with O_RDWR, like this
• O_RDWRO_CREAT
• Use 0666 for the mode

9
DBM in Action

• use SDBM_File
• my animal_sounds
• Tie the in-memory hash animal_sounds to
• the disk-based hash 'animals.dat'
• 'animals.dat' is a file in the
• current directory, and will persist after
• the script has stopped running
• tie (animal_sounds, 'SDBM_File',
• 'animals.dat', O_RDWRO_CREAT, 0666)
• die !
• see next slide for continuation of code...

10
More DBM in action

• now that it is tied, treat animal_sounds
• just like any other hash.
• animal_sounds'dog' 'bark'
• print animal_sounds'dog' prints 'bark'
• my _at_animals keys animal_sounds
• my _at_sounds values animal_sounds
• when you're done, untie the hash
• untie animal_sounds

11
DB_File

• Puts a DBM wrapper around a C library of database
  methods, including B-tree
• B-tree is a sorted, balanced, multiway tree (a
  binary tree on steroids) meaning it's FAST!
• This is the recommended DBM implementation
• For an interesting (to me at least -) discussion
  of binary trees and b-trees, see
  http//www.plover.com/mjd/perl/BTree/

12
Using DB_File

• You use DB_File w/ B-trees just like DBM, except
  add DB_BTREE as the last argument, like so
• use DB_File
• my hash
• my filename 'foo.db'
• tie (hash, 'DB_File', filename, O_RDONLY,
• 0444, DB_BTREE) die !
• question where does DB_BTREE get defined?
  Why
• doesnt use strict complain about it?

13
DB_File in Action

• my dogs
• my dog_file 'mutts.db'
• tie (dogs, 'DB_File', dog_file, O_RDONLY,
• 0444, DB_BTREE) die !
• dogs'fifi' 'poodle'
• print dogs'muffy', "\n"
• delete dogs'spike'
• untie dogs
• just like using a regular hash! Hooray!

14
What about complex data and persistence?

• Good question!
• After we learn about references and complex data
  structures, we'll come back and answer that
  question.

15
What about complex data and persistence?

• A very good question! I'm glad you asked. I'm
  assuming you know about Perl references.
• First, let's look at Storable.pm. This module
  lets you store your complex data structures to
  disk. It uses C functions and a binary format to
  walk/store Perl's data structures.
• Very fast and efficient
• machine-dependent
• must pull whole structure out at once and store
  it all at once

16
Storable example

• use strict
• use Storable
• my file 'fruits'
• my fruits ( banana gt color gt 'yellow',
• price gt '0.89' ,
• orange gt color gt 'orange',
• price gt '0.79' )
• Call the store function to save the data
• structure to disk
• store (\fruits, file) die "couldn't store
  !"
• continued on next slide...

17
Storable example, continued

• continued from previous slide
• To get our data structure back later
• (or we could do this in another program)
• my rh_new retrieve(file)
• print rh_new-gtbanana-gtcolor . "\n"

18
DBM and Complex Data Structures

• DBM files require that values be plain strings.
  You cannot readily use a DBM file with a complex
  data structure. This is not good.
• However, the MLDBM module will let you do just
  this! It uses the DataDumper module to
  stringify your references before it stores them
  to the DBM.
• To your program, it looks like a persistent
  complex data structure.

19
MLDBM

• the code looks pretty much the same as
• other DBM types.
• here it is using DB_File (usually the
• preferred type)--can also use SDBM and
• others
• use MLDBM qw (DB_File)
• tie (hash, 'MLDBM', 'filename',
• O_CREATO_RDWR, 0666)
• die "Can't tie to filename! !"
• continued on next slide...

20
MLDBM example, continued

• once you have tie'd your hash, you can use
• it like a regular (non-complex) hash
• hashJoel 'Grow'
• hashphone '555-1212'
• print "Barney's last name hashBarney\n"
• HOWEVER, see next slide for special usage
• for complex structures

21
More on MLDBM

• One drawback you can't access the references
  directly. Instead, you have to pull in the
  reference from the database, work with it, and
  then store it back, like this
• assume dogs has been tie'd to MLDBM
• this does NOT work
• dogs'fifi'-gt2 'poodle'
• Correct way to do it
• ra_dog dogs'fifi' pull out record
• ra_dog-gt2 'poodle' modify
• dogs'fifi' ra_dog put it back