... (console mods built on PC) Marketing intrusion..

1
Code and data structure implications when
patching without local storage and resource
management overall.
Ken DemarestKen_at_demarest.comKdemarest_at_eidos.com
2
Background on this talk

• This is a thought experiment it all started
  with Jon asking me what hard problems I had heard
  about recently
• Patching consoles Turns out I never did it, so
• Focus on high concepts, ignores numerous
  practical issues
• Make assumptions about a future that may not
  occur
• Goal is to drive discussion, out-of-the-box
  thought
• Gather some current best practices relative to
  future improvements
• And maybe perspective for console guys who might
  not have the depth of online experience PC
  developers have, and PC guys who are unused to
  storage limitations

3
Initial Poll

• Who has ever delivered a patch to their game?
• Who uses late linking of data?
• Who has edit-and-continue in their data path?
• What areas cause level resets?
• What areas invalidate edit-and-continue?
• Whose products support mods / plugins?
• Who is console only (vs PC or consolePC)?

4
Situation

• Next gen consoles (PS3, XB2, PSP, DS) will
  provide connectivity
• Sending data to console games post-sale will
  enable new revenue opportunities
• Patching (should never happen, of course)
• Content unlock (fill every DVD)
• Content extension (Lode Runner maps)
• Community mods (console mods built on PC)
• Marketing intrusion I mean surveys, advertising,
  dynamic product placement, up-sell, cross-sell
• Storage may be limited
• Never as massive as PC storage MMO Patch
  Kings
• Current PS2 no HD Microsofts HD nut trend to
  mem cards

5
Data Transfer Options

• Require storage
• But even then you wont be the only app to
  utilize it
• Send data every time
• Plausible, but slow
• Think of online connection as a remote hard
  drive
• Youve got more bandwidth than storage (ex 60
  broadband, 40 modem on PS2 today)
• Conclusion any data sent should be as small as
  possible but how?

6
Reality Check

• North American PS2 install base is 30M
• 3.2M connected users
• Appears unlikely to break 30 connectivity in
  near- to mid-term
• Europe still hamstrung
• Japan uptake less certain - lt1M today Korea
  likely uptake
• But all new PS2s have network adapter built in
• Still, 30 of 30M is 9M users the size of the
  largest PC game penetration, and about 18x the
  size of the MMO audience
• Assuming XB2 leverages MSN/Zone/Passport user
  base, we could see big numbers here as well
• So still a viable profit center

7
Considered as a patch problem, how do we keep
data transfer small?

• Todays code data layout often resists
  efficient patch sizes
• Code fixups
• Data aggregation, indexes, etc
• The good news
• Simple techniques can ease the problems
• They (might) ripple into greater build efficiency
• The bad news
• Many techniques imply slower loads
• Might have to change your fundamental resource
  management architecture
• But hey, youve got a while to make these
  systemic changes

8
Code Options
Warning Absurd ideas herein for the sake of
illustration

• Stop fixup problems
• DLLs (or equivalent MS TDR)
• Ship the obj files link at run time
• Fixed, predictable link order
• Imagine an order-retaining linker
• Post ship, name everything zz1, zz2, etc
• Code hooks (like a plug-in architecture or a
  mod)
• In main loop, around mem allocations, root
  ancestor constructor/destructor, DLL detection,
  kernel process insertion, etc.
• Code intercept installation on load
• All facilitated with a compact game state
• Manual aspect-orientation
• Crazy VMT redirect applyDamage()

9
Data has troubles too

• Sample data path
• Hot spots
• Getting data game-friendly seldom has order
  guarantees
• Some source data has Butterfly Effect on derived
  data
• Derived data takes a long time to generate often
  large (otherwise you wouldnt need to
  pre-generate it)
• Compression may cross assets (text?)
• Internal linkage lacks persistent references
• Concat indices often resemble code fixups
• Embedded concats only make it worse

Source
Game- Friendly
Derived Data
Link
Compress
Concat
Embed
Illustrative not necessarily in order each
step may also happen multiple times
10
Data Options

• The biggies
• Model data build on code build principles
• Dynamic linking of data at (re)load time
• Concat using relocatable, sector-based approach
• (the irony of FAT32 sector sorting tell FAT to
  order files?)
• The others
• Modularity and predictable ordering in all data,
  to facilitate a diff
• Compression optional (or diff-friendly via window
  borders)

11
When Sending Data or Code

• Date-stamped resources make atomic changes clear,
  QA-able
• Diff then compress before send
• Merge at load time (no local storage, remember?)
• Only send when game-relevant

12
Veering to a related thought experiment

• Lets suppose that you never had to QA anything,
  or worry about load times
• Your end-users could theoretically get build
  results just like the team does (suspend
  disbelief, please and Yes, MMOs appear to do
  this already)

13
Hey, quality upload quality build?

• Conceptually, delivery of patches/upgrades/etc to
  customers is no different than delivery of build
  updates to your team (except for QA, load times,
  delivery infrastructure, blah blah)
• HL2, Halo2, Oddworld all have facets of the data
  delivery techniques discussed earlier
• Trim resource management tends to yield faster
  builds
• Tends to prepare you for efficient online
  delivery too
• Mod community has had many of these techniques
  for years yet projects still let their build
  environment stray from the faith, often for very
  valid reasons
• How might these techniques point us towards data
  delivery nirvana conjoined with build
  environment nirvana? Can TCP/IP distribution of
  code and data be unified across target audiences
  (team/players)?

14
A few pathological cases

• Artist adds wrinkles to a characters forehead,
  and all players see it immediately
• Dev team and the community make simo-changes to
  levels, and everybody sees it at once
• Content rights (ownership) is a characteristic of
  both the dev team and players
• i.e. what content you can see/play, as well as
  your rights to make changes
• Were heading over the edge, so why stop now

15
Straw Man (1)
Warning Blatant impossibilities ahead. No
throwing of rotten tomatoes, please.

• All tools are also servers of their data and
  services via TCP/IP
• Maya, Max, Photoshop, your crazy facial animation
  tool
• Via a uniform format (SOAP? XMLRPC?)
• Capable of evincing change deltas, not just
  entire data units
• Via a perfectly uniform resource system with
  perfect pointer management
• Game displays insta-results and multi-user simo
  editing on live levels
• Late linking, and persistent symbolic links allow
  data-based edit and continue
• All code and data compilation is always
  distributed and cached
• Data validation is so good it isnt possible for
  the team to get stopped
• Oh, and automated code unit testing makes this
  true for code as well
• Derived data is never required only tasked out
  then later loaded when ready
• Just-in-time auto-concatenation (by the game?)
  groups data for most efficient load times

16
Straw Man (2)
OK, maybe one small tomato

• Validation extends to code
• A bot plays through the levels before you see
  them, finds bugs before QA
• Bugs are always fully repeatable, often across
  builds
• Code and scripting both always link late, fast,
  dynamic re-link
• The game never halts the code is so robust it
  even saves game state across major recompiles,
  effectively never halting
• Publish to team members and publish to players
  only vary in the amount of QA
• In fact, player distribution is just a version
  control sync

17
Discussion

• What in the straw man, ignoring constraints, is
  undesirable?
• What are key obstacles to achieving the straw
  man?
• What other high-concept, perfect-world systems
  would you like to see in a next-generation
  resource management / data delivery system? (or
  open discussion to asset chain overall)

18
Cost Analysis Poll

• What of tech time in your org is spent creating
  system (vs. leaf) 20, 40, 60, 80?
• Data full recompile lt10 min, lt30 min, lt1 hr, lt2
  hr, more?
• Code full recompile lt5 min, lt10 min, lt30 min, lt1
  hr, more?
• Who runs metrics on code effort vs. lost team
  time?

19
Next Gen Tools Poll

• Who journals, or uses techniques to improve bug
  reproduction?
• Who uses unit testing? Expects to use it?
• Focusing on techniques to debug multithreaded
  pain?
• Fully generic layer for tweaking data values
  (with a GUI layer where needed)?
• Who sees integration of 3rd party tools as a
  major part of their next product?
• Other techniques?

20
Next Gen Resource Chain Stealth Talk

• Ken Demarest
• Ken_at_demarest.com