Peck: Transparent Distributed Backup Using Chirp

1
Peck Transparent Distributed Backup Using Chirp
Graduate Operating Systems, Fall 2005
Matthew Van Antwerp
December 15, 2005
2
Outline

• Existing Methods
• Strengths and Weaknesses
• Chirp Overview
• Peck Storage and Retrieval
• Mapfiles
• Conclusion

3
Existing Backup Methods

• Dedicated Backup Server
• Portable Media (CD, DVD, etc.)
• Freenet
• USB Thumbdrives
• Chirp

4
Strengths and Weaknesses
5
Chirp Distributed Storage Pool at ND

• Composed mostly of department and lab computers -
  catalog list
• Each system (about 200 in total) offers up spare
  hard drive space.
• Can be accessed through libchirp API, command
  line tools, or parrot.
• Peck sits on top of Chirp.

6

Peck Structure
Peck
libchirp
chirp server
chirp server
chirp server
7
Peck Function

• Input file list
• Attempts upload and download of test file to each
  server to learn permissions
• Locates sufficient servers for upload
• Writes filename and server name to mapfile for
  each uploaded file
• Uploads copy of mapfile to multiple servers

8
Peck Mapfile

• Hypothetical Your hard drive crashes and you
  lose all your data (yes, it will happen to you
  one day).
• Luckily you have been backing up your files via
  Peck.
• Peck scours the servers for your mapfile
  (relatively slow due to no knowledge of Chirp
  servers).
• When it finds a copy, it retrieves the file and
  opens it.
• One by one, Peck retrieves files listed in
  mapfile from the relevant servers.

9
Peck Mapfile Versions

• How do we know we are retrieving the proper
  mapfile when we upload new files or retrieve
  files?
• Also, how can we keep from filling up Chirp
  servers with redundant copies?
• Answer through careful use of mapfiles.
• After first run of peck, we have a mapfile on the
  servers which we retrieve on next run.
• We check mapfile for a given filename when
  attempting to upload and update appropriately.
• Before uploading new mapfile, delete all old
  mapfiles.

10
Analysis

• Checking permissions slow (18.15s avg)
• Upload similar to ftp times
• Mapfile retrieval after total system loss slow
• Download (after mapfile retrieved) similar to
  ftp times.
• Updating mapfile and removing old mapfiles
  fairly slow.
• Very little overhead besides the obvious
  bottlenecks.

11
Conclusion

• Peck achieves many of the strengths of other
  backup methods while avoiding their weaknesses.
• Easy to use simply give the application a list
  of files to upload (easily used in conjunction
  with file age scripts). An ideal cron job.
• Cheap (free), fairly efficient (although not
  largely scalable), easy to setup and maintain,
  and transparent.
• All necessary information for retrieval is stored
  on the Chirp servers.