Supporting Persistent Objects In Python

1
Supporting Persistent ObjectsIn Python Jeremy
Hyltonjeremy_at_zope.com
2
What is persistence?

• Data lives longer than programs
• Files, pipes, relational databases, etc.
• But, discontinuity of representation
• Orthogonal persistence
• Automatic management of program state
• Independent of data type or longevity
• Allow programmer to focus on application data
  model

3
Python approach

• Goals
• Minimize changes to existing programs
• Work with standard interpreter
• Zope Object Database (ZODB)
• Support Zope application server
• Originally targeted for web development
• Separate database and storage layers

4
Not on todays agenda

• Benefits of persistence
• Discussion of related work
• Early work
• PS-algol, Napier88
• Much recent work on Java
• PJava, ObjectStore PSE

5
ZODB key ideas

• Persistence by reachability
• Any reachable object is saved
• Transactions to control updates
• Safe sharing among clients
• Implement with Persistent mixin
• Provides hook for persistence
• Compromise on transparency

6
Persistent base class

• C extension type that defines
• Four _p_ attributes
• Custom attribute accessors
• Efficient C API
• Marks serialization boundaries
• Interacts with database
• Load objects on demand
• Register modifications with TM

7
Persistence by reachability

• Any object reachable from ZODB root is persistent

8
A simple example

• from Persistence import Persistentfrom
  Transaction import get_transactionfrom
  ZODB.FileStorage import DBclass
  Counter(Persistent) _value 0 def
  inc(self) self._value 1def main() fs
  DB(data.fs) conn db.open() root
  conn.root() obj rootmyobj
  Counter() get_transaction().commit() obj.inc()
  get_transaction().commit()

9
Object serialization

• Standard pickle library
• Serializes arbitrary object graph
• Raises TypeError for sockets, files, c.
• Instance vars serialized via dictionary
• Hooks to define custom state
• __getstate__() / __setstate__()
• Persistent mixin ignores _v_ attributes

10
Pickling persistent objects

• Stores objects in separate records
• Persistent objs pickled as oid class
• Works with cache to maintain identity
• Handling non-persistent objects
• Copied into record of containing object
• Sharing by persistent objs is problematic

11
Object identity / caching

• Cache maintains oid ? obj mapping
• Guarantees only one copy of object
• Unpickler loads all referenced objects
• Ghost objects
• Only Persistent header initialized
• No instance state loaded
• State loaded on first object access
• LRU cache of recent objects

12
Attribute access handlers

• Persistent implements C wrappers
• Override tp_getattro, tp_setattro slots
• Mediate access to instance variables
• Crucial Python feature

13
Example __getattribute__() hook

• class Example(object)
• _p_state False
• def _p_activate(self)
• print "activate"
• self._p_state True
• def __getattribute__(self, attr)
• print "intercept", attr
• if not attr.startswith("_p_") and not
  self._p_state
• self._p_activate()
• return super(Example, self).__getattribute
  __(attr)
• gtgtgt obj Example() obj.value "test"
• gtgtgt print obj.value
• intercept value
• intercept _p_state
• intercept _p_activate
• activate
• test

14
Transactions

• Supports multiple threads, processes
• Independent database connections
• Updates visible at transaction boundaries
• Optimistic concurrency control
• When conflict occurs, abort and retry
• On error, abort to restore consistency
• Reverts to last saved state

15
Concurrency and conflicts

• Invalidations sent at commit time
• Clients process at transaction boundaries
• Conflicting transactions aborted
• Write conflict at commit time
• Read conflict on object access
• Application must retry on conflict
• Can use generic wrapper
• Can define conflict resolution method

16
Retrying conflicts

• Example wrapper for retries

def transact(f, retries3) def
wrapper(args, kwargs) n retries
while n try
try return f(args,
kwargs) finally
get_transaction().commit()
except ConflictError n - 1
if n 0 raise
except
get_transaction().abort() raise
return wrapper
17
Other features

• Undo support
• Storage stores multiple revisions
• Transactional undo reverts to earlier state
• BTrees efficient persistent containers
• Storing code in database

18
Limitations

• Schema evolution
• Must code manually in __setstate__()
• Database management
• Manual pack() to remove revisions, do GC
• Sharing of non-persistent objects
• Integration with legacy code
• Multiple inheritance helps
• Factory classes

19
Getting the software

• http//www.zope.org/Wikis/ZODB
• info central for ZODB
• ZODB 3.1 released Oct. 2002
• Uses ExtensionClass
• ZODB4 alpha planned for Dec. 2002
• Based on Python 2.2 type support
• Fewer onions in the varnish