Feeding the Monster Advanced Data Packaging for Consoles

1
Feeding the MonsterAdvanced Data Packaging for
Consoles
Bruno ChampouxNicolas Fleury
2
Outline

• Next-Generation Loading Problems
• LIP A Loading Solution
• Packaging C Objects
• Demo LIP Data Viewer
• Questions

3
Some things never change
4
Next-Gen Loading Problem

• Processing power up by 10-40X
• Memory size up by 8-16X
• Optical drive performance up by 1-4X

5
Next-Gen Loading Problem

• Xbox 360
• 12X dual-layer DVD drive
• Outer edge speed 15 MB/s
• Average seek 115 ms
• PlayStation 3
• Blu-ray performance still unknown
• 1.5X is the most likely choice
• CAV drive should give 6 to 16 MB/s
• Average seek time might be worse than DVD

6
Next-Gen Loading Problem
Memory Size Maximum Bandwidth Time to fill
PS2 32 MB 4.4 MB/s 7.3 s
PS3 192 MB 16 MB/s 12 s
Xbox 64 MB 6.9 MB/s 9.3 s
Xbox 360 480 MB 15 MB/s 32 s
7
Next-Gen Loading Problem

• In order to feed the next-gen data needs, loading
  will need to be more frequent
• Hard drives are optional for PS3 and Xbox 360
• Optical drive performance does not scale with the
  memory/CPU power increase
• Conclusion
• Loading performance must be optimal any
  processing other than raw disc transfers must be
  eliminated

8
Did I Hear Loading Screen?

• Disruptive
• Boring as hell
• Non-skippable cutscenes are not better!
• Conclusion
• Loading screens must not survive the current
  generation

9
Background Loading

• Game assets are loaded during gameplay
• Player immersion is preserved
• Solution
• Use blocking I/O in a thread or another processor

10
Background Loading

• Requirements
• Cannot be much slower than a loading screen
• Must have low CPU overhead
• Must not block other streams
• Conclusion
• Once again, loading performance must be optimal
  any processing other than raw disc transfers must
  be eliminated

11
Proposing A Solution

• Requirements for a next-generation packaging and
  loading solution
• Large amounts of assets must be loaded at speeds
  nearing the hardware transfer limit
• Background loading must be possible at little CPU
  cost
• Data assets must be streamed in and phased out
  without causing memory fragmentation

12
Understanding Loading Times

• Freeing memory space
• Unloading
• Defragmenting
• Seek time
• Read time
• Allocations
• Parsing
• Relocation (pointers, hash ID lookups)
• Registration (e.g. physics system)

13
Reducing Loading Times

• Always load compressed files
• Using N1 compression will load N times faster
• Double-buffering hides decompression time
• Plenty of processing power available for
  decompression on next-gen consoles

14
Reducing Loading Times

• Compression algorithm choice
• Favor incremental approach
• Use an algorithm based on bytes, not bits
• Lempel-Ziv family
• LZO

15
Reducing Loading Times

• Take advantage of spatial and game flow coherence
• Batch related data together in one file to save
  seek time
• Place related files next to each other on the
  disc to minimize seek time

16
Reducing Loading Times

• Take advantage of optical disc features
• Store frequently accessed data in the outer
  section of the disc
• Store music streams in the middle (prevents full
  seek)
• Store single-use data near the center (videos,
  cutscenes, engine executable)
• Beware of layer switching (0.1 seconds penalty)

17
Reducing Loading Times

• Use the flyweight design pattern
• Geometry instancing
• Animation sharing
• Favor procedural techniques
• Parametric surfaces
• Textures (fire, smoke, water)

18
Reducing Loading Times

• Always prepare data offline
• Eliminate text or intermediate format parsing in
  the engine
• Engine time spent converting or interpreting data
  is wasted
• Load native hardware and middleware formats
• Load C objects directly

19
Why Load C Objects?

• More natural way to work with data
• Removes any need for parsing or interpreting
  assets
• Creation is inexpensive
• Pointer relocation
• Hash ID conversion
• Object registration

20
Loading C Objects

• Requires a very smart packaging system
• Member pointers
• Virtual tables
• Base classes
• Alignment issues
• Endianness

21
Loading Non-C Objects

• Must be in a format that is ready to use after
  being read to memory
• Texture/normal maps
• Havok structures
• Audio
• Script bytecode
• Pretty straightforward

22
Load-In-Place (LIP)

• Our solution for packaging and loading game
  assets
• Framework for defining, storing and loading
  native C objects
• Dynamic Storage a self-defragmenting game asset
  container

23
Load-In-Place LIP Item

• 1 LIP item ? 1 game asset
• 1 LIP item ? unique hash ID (64-bit)
• 32 bits for the type ID and properties
• 32 bits for the hashed asset name (CRC-32)
• The smallest unit of data that can be
• queried
• moved by defragmentation
• unloaded
• Supports both C objects and binary blocks

24
Examples of LIP Items

• Joint Animation
• Character Model
• Environment Model Section
• Collision Floor Section
• Game Object (hero, enemy, trigger, etc.)
• Script
• Particle Emitter
• Texture

25
C-Based LIP Items

• Can be made of any number of C objects and
  arrays
• On the disc, all internal pointers are kept
  relative to the LIP item block
• Pointer relocation starts with a placement new on
  a relocation constructor
• Internal pointers are relocated automatically
  through constructor chaining

26
Placement new Operator

• Syntax
• new(ltaddressgt) lttypegt
• Calls the constructor but does not allocate
  memory
• Initializes the virtual table
• Called once for each LIP item on the main class
  relocation constructor

27
Relocation Constructors

• Required by all classes and structures that
• can get loaded by the LIP framework
• contain members that require relocation
• 3 constructors
• Loading relocation constructor
• Moving relocation constructor (defragmentation)
• Dynamic constructor (optional, can be dummy)
• No default constructor!

28
Object Members Relocation

• Internal pointer
• Must point within the LIP item block
• Converted into absolute pointer
• External reference (LIP items only)
• Stored as a LIP item hash ID
• Converted into a pointer in the global asset
  table entry that points to the referenced LIP
  item
• LIP framework provides wrapper classes with
  appropriate constructors for all pointer types

29
Relocation Example

• class GameObject
• public
• GameObject(const LoadContext ctx)
• GameObject(const MoveContext ctx)
• GameObject(HASHID id,
• Script pScript)
• protected
• lipRelocPtrltTransfogt mpLocation
• lipLipItemPtrltScriptgt mpScript

30
Relocation Example (contd)

• GameObjectGameObject(const LoadContext ctx)
• mpLocation(ctx),
• mpScript(ctx)
• GameObjectGameObject(const MoveContext ctx)
• mpLocation(ctx),
• mpScript(ctx)
• GameObjectGameObject(HASHID id,
• Script pScript)
• mpLocation(new Transfo),
• mpScript(pScript) SetHashId(id)

31
Relocation Example (contd)

• templatelttypename LipItemTgt
• void PlacementNew(
• lipLoadContext loadCtx)
• new(loadCtx.pvBaseAddr)
• LipItemT(loadCtx)
• loadCtx.pvBaseAddr pvLoadMemory
• PlacementNewltGameObjectgt(loadCtx)

32
Relocation Example (contd)
33
Load-In-Place Load Unit

• Group of LIP items
• The smallest unit of data that can be loaded
• 1 load unit ? 1 load command
• Number of files is minimized
• 1 language-independent file
• Models, animations, scripts, environments,
• N language-dependent files
• Fonts, in-game text, some textures, audio,
• Load unit files are compressed

34
Load Unit Table

• Each LIP item has an entry in the table
• Hash ID
• Offset to LIP Item

35
Dynamic Storage

• Loading process
• Load unit files are read and decompressed to
  available storage memory
• Load unit table offsets are relocated
• Load unit table entries are merged in the global
  asset table
• A placement new is called for each LIP item
• Some LIP item types may require a second
  initialization pass (e.g. registration)

36
Dynamic Storage

• Unloading process
• Each LIP item can be removed individually
• All LIP items of a load unit can be removed
  together
• Destructors are called on C LIP items
• Dynamic storage algorithm will defragment the new
  holes later
• Locking
• LIP items can be locked
• Locked items cannot be moved or unloaded

37
Platform-Specific Issues

• GameCube
• Special ARAM load unit files
• Animations
• Collision floors
• Small disc ? compression
• Xbox/Xbox 360
• Special LIP items for DirectX buffers
• Vertex, index and texture buffers
• 4KB-aligned LIP items (binary blocks)
• Buffer headers in separate LIP items (C objects)

38
Load-In-Place Other Uses

• Network-based asset editing
• LIP items can be transferred from and to our
  level editor during gameplay
• Changes in asset sizes do not matter
• Used by Maya exporters to store our intermediate
  art assets
• LIP is much more efficient than parsing XML!

39
Packaging C Objects

• Nicolas Fleury

40
Our Previous Python-Based System

• class MyClass(LipObject) x Member(UInt32)
  y Member(UInt32, default1) p
  Member(makeArrayType( makePtrType(SomeClas
  s)))

41
Cool Things with this System

• Not too complex to implement.
• Python is easy to use.
• Introspection support.
• A lot of freedom in corresponding C classes.

42
Problems with this System

• Python and C structures must be synchronized.
• Exporters must be written, at least partly, in
  Python.
• Validations limited (unless you parse C code).
• We just invented a Python/C hybrid.

43
C-based system

• class MyClass public MyBaseCls ...
  LIP_DECLARE_REGISTER(MyClass) uint32 x
• // In .cppLIP_DEFINE_REGISTER(MyClass)
  LIP_REGISTER_BASE_CLASS(MyBaseCls)
  LIP_REGISTER_MEMBER(x)

44
Consequences

• Exporters are now written in C.
• Class content written twice, but synchronization
  fully validated.
• Dummy engine .DLL must be compiled (not a working
  engine, provides only reflection/introspection).
• Need a good build system.
• We just added reflection/introspection to C.

45
Member Registration Information

• Name
• Offset
• Type
• Special flags (exposed in level editor, etc.)

46
(Non Empty) Base Class Registration Information

• Name
• Type
• Offset calculated with
• (size_t)(BaseClassType)(SubClassType)1 - 1

47
Member Type Deduction

• In IntrospectorBase classtemplate lt typename
  TypeT, typename MemberTgtvoid RegisterMember(
  const char name, MemberT(TypeTmemberPtr))

48
Member Type Deduction (Arrays)

• In IntrospectorBase classtemplate lt typename
  TypeT, typename MemberTypeT, int sizeTgtvoid
  RegisterMember( const char name,
  MemberT(TypeTmemberPtr)sizeT)

49
Needed Information in Tools

• Every class base class (to write their members
  too).
• Every class member.
• Every base class offset (to detect missing base
  class registration).
• Every member name, size, type and special flags.
• For every type, the necessary alignment and if it
  is polymorphic.

50
Introspection Classes (Members)
51
Introspection Classes (Base Classes)
52
Result Member Introspection

• Able to know all types and their members.
• Can be used for both writing and reading binary
  data.
• Same class used in tools to fill the data as in
  engine.

53
LipViewer

• Data of any platform, endianness, pointer size
  (binary files have a header with platform id).
• Both for engine data and tools binary formats.
• Hexadecimal viewer integration, edition support.
• Excellent learning and debugging tool.

54
LipViewer Demo
55
Restrictions for Simplification of Implementation

• Polymorphic types must begin with a vtable
  pointer (their first non-empty base class must be
  polymorphic).
• Cant inherit twice from same class indirectly
  (or offset trick doesnt work).
• No virtual base classes.
• All padding is explicit.

56
Explicit Padding

• class MyClass ...LIP_PADDING(mP1,
  LIP_PS3(12))uint16 mSomeMemberlipPaddinglt4gt
  mP2uint32 mSomeOtherMemberLIP_PADDING(mP3,LIP_
  PC(4) LIP_PS3(8))...

57
Particular Things to Handle

• Endianness.
• 64 bits pointers (no more!).
• VTable padding.
• Type alignment.

58
VTable Padding

• __declspec(align(16)) class Matrix
• class MyClass uint32 x, y, z Matrix
  m
• 32 bytes on PS3, 48 bytes on PC.

59
Automatic Versioning

• Create a huge string with member names/types and
  member names/types of pointed classes.
• In the case of polymorphic pointers, all
  sub-types must also be included.
• Hash the huge string.
• Can be integrated in tools dependency tree
  mechanism.

60
Needed Information in Engine

• A hash map of objects to do the placement new of
  the appropriate type.
• Smart pointers/arrays handle the rest.

61
Type Ids

• VTable pointers are replaced by a type id.
• LIP_DECLARE_TYPE_ID(MyClass, id) in .hpp.
• Defines a compile-time mechanism to get id.
• Declares a global object.
• LIP_DEFINE_TYPE_ID(MyClass) in .cpp.
• Defines the global object. Its constructor adds
  itself as a hash node to a hash map. This object
  class is templated to make operations with the
  good type (example placement new).

62
Hash Map Overview
63
Pointers

• Normal pointers like T must be set to 0 when
  exporting an object.
• For relocated pointers, smart pointer classes
  must be used.
• Different types of smart pointers/arrays
• Ownership of pointed data?
• Relocation of pointed data?

64
Smart Members

• Classes can derive from a lipSmartMember class
  to implement a custom writing/reading in tools.
• Class only used as a tag, it doesnt have any
  virtual function.
• Classes deriving from lipSmartMember are
  expected to implement a compile-time interface.
• Useful for smart pointers (normal pointers cannot
  be load-in-placed).

65
Smart Members Full Control Over Writing and
Reading.

• class MySmartArray lipSmartMember
• public
• void Write(lipLipWriter) const
• void WriteExternalData( lipLipWriter)const
  
• void Stream(lipLipReader)
• void StreamExternalData( lipLipReader)

66
WeakRelocPtrltTypegt
67
RelocPtrltTypegt

• Relocation assumes pointed data is not of a
  sub-type and does directly a placement new of
  Type.

68
RelocPolymorphicPtrltTypegt

• Relocation looks in the hash map to do placement
  new of the good type (involves a search and a
  virtual function call).

69
RelocFixedArrayltType, sizegt
70
WeakRelocArray
71
RelocArrayltTypegt
72
RelocPolymorphicArrayltTypegt
73
RelocWeakPtrArrayltTypegt
74
BinaryBlockPtrltalignment4gt
75
Enums

• Concept of exclusivity group masks to regroup
  values in mask in a radio button group in GUI.
• LIP_REGISTER_ENUM(MyEnum)
  LIP_DEFINE_ENUM_VALUE(eNO)
  LIP_DEFINE_ENUM_VALUE(eYES)

76
Other Solutions

• Parse debug info files.
• Compile as C/CLI.
• Parse source code.

77
Questions?
78
Links

• Latest slides
• http//www.a2m.com/gdc/
• How to reach us
• bruno.champoux_at_a2m.com
• nicolas.fleury_at_a2m.com