Issues and Implementations in Multiplayer Game Development

1
Issues and Implementations in Multiplayer Game
Development

• Online game development is difficult and riddled
  with expensive risks. Whether you are adding
  multiplayer support to a single-player game or
  fielding a massively multiplayer persistent
  virtual world, the additional complexity of
  connecting your players over a network adds
  substantial development time and testing costs.
  Get it right, and online can bring innovative
  game play and a sense of community to your player
  base. Get it wrong, and going online brings you a
  Pandoras box of missed deadlines, frustrated
  developers, and unhappy customers

2
Introduction
Why Online?
Technology
3
Tutorial Takeaway Messages

• Building online games with single-player game
  techniques is painful
• Early changes to your development process
  system architecture to accommodate online play
  will greatly ease the pain
• Lessons Learned from our background will
  provide your project with useful Rules of Thumb

A wise man learns from his own mistakes, a wiser
man learns from someone elses
4
Network Terminology
5
Bandwidth

• A measure of the width or capacity of a
  communications channel. Greater bandwidth allows
  communication of more information in a given
  period of time.
• Important Bandwidth refers only to the amount of
  data a channel can handle. As an analogy, a dump
  truck has a higher bandwidth than a sports car.
  This does not mean that the dump truck is faster,
  just more efficient at moving large amounts of
  material

Source Ubisoft Online Terminology Guide, used
with permission
6
Cheat

• An exception to a games normal rule set that is
  included by the games developers. Usually,
  cheats are enabled only in the single-player mode
  of the game (if it has one) or were included for
  use (and forgotten about) by the developers to
  aid with testing and/or the media.

7
Client/Server

• This is any networking architecture in which one
  machine (the server) is responsible for making
  final determinations on what occurs in the game.
  This can range from game designs in which the
  client merely displays information to the user
  and returns user inputs to the server, to game
  designs in which all machines are running nearly
  complete versions of the game, and the server
  only serves to centralize communications and
  adjudicate any disagreements.
• Important This is an architectural distinction.
  It is entirely possible that the server is a
  consumer operated machine.

Source Ubisoft Online Terminology Guide, used
with permission
8
Consumer-Hosted

• A game in which all of the machines involved are
  run by the consumer.

Source Ubisoft Online Terminology Guide, used
with permission
9
Dedicated Server

• A client-server game in which there is a machine
  which is serving as a server without having any
  local player. This may be hosted by a consumer,
  or by a game service.

Source Ubisoft Online Terminology Guide, used
with permission
10
Dumb-Client

• A client-server game design in which the client
  is largely a dumb input/output device.

Source Ubisoft Online Terminology Guide, used
with permission
11
Exploit

• An unexpected or unintended game result
  generated by the actions of the player that
  modifies the course or outcome the game in a way
  not intended by the games designers. Usually
  the result of players discovering a bug in the
  game.

12
Hack

• A modification made to a game or its data by an
  end user. This can take the form of modifying
  either code or data, in memory while the game is
  running, and/or to the games files on disk. The
  game itself or a service that it uses (such as a
  driver), or the data it shares with other
  computers, or hardware can be modified.

13
Hybrid Game

• Mix between Client-Server and Peer-to-Peer,
  where a server determines the outcome for some
  events, while each players computer determine
  the outcome of some events (usually local to that
  player).

14
Firewall

• A security product that employs a combination of
  hardware and software to prevent unauthorized
  users or traffic from the Internet from gaining
  access to a private local area network. Many
  problems in online gaming arise because firewalls
  block communication between games.

Source Ubisoft Online Terminology Guide, used
with permission
15
Integrated Server

• A client-server game in which the machine which
  has a user playing the game is also serving as
  the server. On some PC applications, even though
  there may be a server running on the same machine
  as a player, they are separate applications.

Source Ubisoft Online Terminology Guide, used
with permission
16
IP Address

• This is the identifying number for any machine on
  the Internet or for any LAN using the IP
  protocol. The original, and most widely used
  form, is IPv4. Under IPv4, each address is made
  up of four octets. If expressed as text, it will
  take the form of "www.xxx.yyy.zzz" (sometimes
  called a dotted IP string). The broader Internet
  is slowly switching over to IPv6 to increase the
  number of available addresses. As the name
  implies, IPv6 addresses are made up of six
  octets.
•  

Source Ubisoft Online Terminology Guide, used
with permission
17
IP Address

• Additionally, under IPv4, the following rules
  apply to network addresses The local machine is
  always reachable by the "loopback" address of
  127.0.0.1, and the following ranges of addresses
  are reserved for use on local LANs
•  
• 10.0.0.0 - 10.255.255.255
• 172.16.0.0 - 172.31.255.255
• 192.168.0.0 - 192.168.255.255

Source Ubisoft Online Terminology Guide, used
with permission
18
Lag

• A slowing of the execution of a game. Lag can be
  caused by two main factors network issues or
  computer performance.
•    
• Network lag cause an online game to miss
  information bits necessary for the execution of
  it. For instance, the last message indicating the
  position of another player can be lost or is
  late, but the game has to display something on
  the screen. Depending on how this situation has
  been taken into account by the developers, the
  display of the other player will freeze and the
  computer will wait for the next message to arrive
  before continuing the execution of the game.   

Source Ubisoft Online Terminology Guide, used
with permission
19
Lag

• Performance lag is caused by the overload of the
  main CPU (too much information to process at a
  time) or the graphical CPU (too many polygons or
  lightning to process).

Source Ubisoft Online Terminology Guide, used
with permission
20
Latency

• A measure of how long it takes a single bit to
  propagate from one end of a link or channel to
  the other. Latency is measured strictly in terms
  of time.

Source Ubisoft Online Terminology Guide, used
with permission
21
MAC Address

• Media Access Control Address, a hardware address
  that has been embedded into the network interface
  card by the manufacturer to uniquely identify
  each node or point of connection of a network.

Source Ubisoft Online Terminology Guide, used
with permission
22
NAT (Network Address Translation)

• Enables a local area network to use one set of IP
  addresses for internal traffic and a second set
  of IP addresses for external traffic.

Source Ubisoft Online Terminology Guide, used
with permission
23
Octet

• An 8 bit value. Because a byte is the smallest
  addressable amount of memory on the machine (6
  bit, 7 bit, 9 bit, and 16 bit machines have all
  been on networks, along with the now standard 8
  bit bytes), network engineers like to use octet
  to specify that we really mean 8 bits.

Source Ubisoft Online Terminology Guide, used
with permission
24
Peer-to-Peer

• In this networking architecture, all games are
  running full versions of the game engine.
  Although one machine may be used to host the
  setup of the game, once the game is running, all
  machines are effectively equal. Disputes may be
  resolved algorithmically (i.e. claims about your
  own damage are always valid) which lends itself
  to cheating, or via a voting mechanism.
• Important This is an architecture. It does not
  mean any game where all the machines involved
  are consumer owned.

Source Ubisoft Online Terminology Guide, used
with permission
25
Server-Validation

• A client-server game design in which the clients
  are running relatively complete versions of the
  simulation, and the server is validating claims,
  rather than determining all results.

Source Ubisoft Online Terminology Guide, used
with permission
26
Service Hosted

• A game in which one or more of the machines
  involved is provided by a game service (often the
  games publisher).

Source Ubisoft Online Terminology Guide, used
with permission
27
TCP

• Transmission Control Protocol. TCP is one of the
  main protocols in TCP/IP networks. Whereas the IP
  protocol deals only with packets, TCP enables two
  hosts to establish a connection and exchange
  streams of data. TCP guarantees delivery of data
  and also guarantees that packets will be
  delivered in the same order in which they were
  sent.

Source Ubisoft Online Terminology Guide, used
with permission
28
UDP

• A connectionless protocol that, like TCP, runs on
  top of IP networks. Unlike TCP/IP, UDP/IP
  provides no error recovery services. UDP/IP
  packets are CRC checked, but if there is a CRC
  failure or part of a UDP/IP packet fails to
  arrive, the packet is lost entirely.
  Additionally, where TCP/IP is stream based,
  UDP/IP is datagram based, with all the data in a
  UDP/IP send arriving at once, if it arrives at
  all. It can be used for broadcasting messages
  over a local network to find games, and is used
  by many games as the primary game data channel.

Source Ubisoft Online Terminology Guide, used
with permission
29
What is a WAN?

• Wide Area Network (WAN) is a network of
  networks
• Allows different networks to talk to each other
• Size and distance doesnt matter

30
LAN Environment

• Local Area Network (LAN) shares common address
  space
• Actually independent of media usually Ethernet
• Also independent of protocol usually TCP/IP
• Can broadcast

31
WAN Environment

• Multiple networks
• Can have overlapping address spaces
• Very media dependent
• Uses edge device to hide the details
• Different types based on connection details

32
WAN Connection Types

• Point-to-point a single leased line between two
  sites
• Circuit Switching like a phone call, connection
  made on demand
• Packet switched Virtual circuits are created

33
Point to Point

• A leased line from the provider
• Always on
• Fixed bandwidth
• Works like a pipe, whatever goes in at one end
  comes out the other
• Can set up an extended LAN environment
• More expensive

34
Circuit Switching

• Similar to a leased line in that there is a
  physical connection between the two sites
• Gets set up and torn down so its only on when
  needed
• Similar to making a phone call
• ISDN is an example

35
Packet Switched

• Uses provider infrastructure
• No real connection between sites, uses a virtual
  circuit
• Routes data packets to final destination by
  whatever path is available
• Two types of circuits Switched Virtual Circuit
  (SVC) and Permanent Virtual Circuit (PVC)
• Examples are ATM, Frame Relay, X.25, SMDS

36
Virtual Circuits

• PVCs are always on permanently established
• SVCs are brought up and down
• Call Setup
• Idle
• In use
• Tear down

37
Quality of Service

• Quality of Service (QOS) requires a queuing
  mechanism
• Differs between media used
• Prioritizes data according to packet information
• Line purchased may or may not support - Make
  sure you buy the right thing!

38
Edge Device

• Any device that hides the complexity of what you
  are talking to from your LAN
• Does the protocol disassembly and reassembly
• Handles signaling
• Handles media conversion
• Router is an example

39
ATM

• Asynchronous Transfer Mode
• Used by Telcos to handle voice
• Breaks everything up into small, fixed size
  packets
• Very common, cheap
• T1s, T3s, OC3s etc.
• Need to test for equipment compatibility!

40
Typical ATM Usage

• Local Exchange Carrier (LEC) provides physical
  line
• Line is located in the carriers Point of
  Presence (POP) on your facility
• You connect your router to your internal Ethernet
  LAN
• LEC connects line to your carrier

41
Typical ATM Usage cont.

• Your carrier (Sprint, ATT, Global Crossing, etc.)
  has a small, fixed number of hops to get you onto
  main trunk
• Main trunk will get your data to destination LEC
  site
• LEC to POP to Router to Destination LAN
• Magic is in how it knows where to go!

42
Break until 1115

• 1115am-1230pm
• Moving from Single Player to Multiplayer

43
NETWORKED GAMES ARE HARDER THAN SINGLE PLAYER
GAMESUse Case Steal Ball Being Dribbled By
Another Player
Remote machine always has an approximation of
ball position
44
Issue Kicking a Shared Soccer Ball

• Your opponent has the _exact_ position of the
  ball
• Your computer has only an _approximate_ position
  of the ball
• When you (the player or the coder) make a
  decision, it may be on incorrect data
• Level of distortion impacts quality of gameplay
• Inaccuracies of data lead to serious coding
  defects

The more computers involved, the greater the
problem
45
Shared Worldview Each Node Has A Potentially
Different Representation Of Every Value
Client A (Local Memory) Clear Views
Game Logic
Player Views
True Values
Damage Levels
Player Position
AI State
46
At any slice in time, fuzzy, loosely coupled
views produce an inconsistent shared worldview
Machine A, time t
Machine B , time t
Network
Shared View _at_ (t) Is ALWAYS DIFFERENT
B
Network
C
Network
A
Machine C , time t
47
The loss of precision

• Compression saves bandwidth
• What to compress and how much?
• Possibly compress then decompress before you use
  it in game to minimise divergence
• Examples
• Car position vibration
• Car position aggressive driving

48
What is an atomic operation?

• Each message becomes an atomic operation
• Interframe and intraframe timing is not preserved
  across the network
• Troublesome case Multiple frame actions such as
  shell transitions
• Troublesome case Multi-step operations in which
  the interim cases are invalid at frame start

49
Event Ordering And Atomicity Across Transactions
Further Impact GamePlay And Coding Errors
Near-Endless (and illogical) Ordering Are Possible
Machine A
Machine B
A1,A2,B1,B2 A1, B1,B2,A2 B1,B2,A1,A2 A1,A2,B1,B2
A2,A1,B1,B2 B2,A2,B1,A1
A1,A2
B1,B2
Machine C
50
Indeterminate timing and sequencing of information

• Player A sends a stream of packets describing
  game state
• Timing
• Player A picks up invulnerability pickup (10s)
• Player A gets shot 9.9s later
• If the second packet is delayed before arriving
  at B

51
Indeterminate timing and sequencing of information

• Sequencing
• Player A picks up rocket-launcher
• Player A fires rocket
• If these arrive in the wrong order at B

52
Atomicity Across Transactions(Sharing Internal
State)
Client Process
Logical Thread (A)
Logical Thread (B)

1. Generate House Info Request (11,11)
2. Set Local State waiting for reply on house
   (11,11)

Local House State (11,11)

1. Generate House Info Request (22,22)
2. Set Local State waiting for reply on house
   (22,22)

Local House State (22,22)
3. EnterHouse (LocalHouseState)
Thread (A) acts off of incorrect data
53
TSO Solution Encapsulate State Logic in a
State Machine
Regulator A
Client Process
Regulator B
Logical Thread (A)
Logical Thread (B)

1. Generate House Info Request (11,11)
2. Set Local State waiting for reply on house
   (11,11)

Local House State (11,11)

1. Generate House Info Request (22,22)
2. Set Local State waiting for reply on house
   (22,22)

Local House State (22,22)
3. EnterHouse (LocalHouseState)
Thread (A) now acts off of correct data
54
Latency (and why it isn't constant)

• The internet

55
Latency (and why it isn't constant)

• Router buffers

56
Latency (and why it isn't constant)

• Different routes

57
Latency (and why it isn't constant)

• Packet loss

58
API options State Replication Versus Event Driven
Client A
Client B
State Changes
State Changes
Best Available representation of state
Best Available representation of state
Shared StateDB
Shared StateDB
Partial Updates
Pros
Cons
Intuitive Model Hides Network Optimizations
API can create new problems Hidden network costs
59
Recommendation Support Shared State Events
Client A
Client B
Immediate Events
State Changes
State Changes
Best Available representation of state
Best Available representation of state
Shared StateDB
Shared StateDB
Partial Updates
Shared State
Events
Instrument for volume latency Optimize to
network budget
Use for stateless transactions, Immedidate actions
60
Shared WorldView Summary(Impact On Players
Programmers)

• Player immersion jerky visuals, occasional
  discontinuities
• Fairplay who killed who?
• Well, who shot first? How can you tell?
• Was Player A actually located where Player B shot
  him?
• Coding issues
• Race conditions
• Event ordering loss of precision can change
  future outcomes per machine
• Bugs of this sort are hard to reproduce

61
Shared WorldView Summary (Mitigating Factors)

• High degrees of precision are rarely required
• Predictable motion patterns simplify hiding the
  latency (dead reckoning, predictive contracts, et
  al)
• Rule of Thumb design game play to minimize the
  required accuracy (and volume) of shared data
• Useful tactic handle all inter-process
  communication via state machines that handle each
  transaction from initiation to conclusion (TSOs
  Regulator template worked well)

62
Scale and Scaleability

• Scale the N2 problem
• Data Transmission Optimizations minimize the
  network cost of sharing the required amount of
  Shared State
• Interest management, predictive contracts (dead
  reckoning), variable resolution, compression,
• Application Scalability minimize the amount of
  required Shared State
• References www.maggotranch.com/ADS
• Bandwidth these days, who cares?
• Caveat see Scale!
• Best bet metrics, metrics, metrics
• Set budget per player, optimize to that point,
  then STOP
• Danger pre-mature optimization
• Danger game play changes that break optimizations

63
Why every and all are dangerous

• SP all information is always available
• MP shared information must be packaged,
  transmitted and unpackaged
• Each step costs CPU bandwidth, and can happen
  10s to 100s of times per minute
• May also cause additional overhead (e.g. DB
  calls)
• Scalability is key many shared data structures
  grow with the number of players, AI, objects,
  terrain,
• Caution early prototypes may be cheap enough,
  but as game progresses, costs may explode

64
Why every and all are dangerous

• Example TSOs Data Service
• Initial design Transmit entire ReservedLotList
  to all connected clients, every 30 seconds
• Initial fielding no problem
• Development testing lt 1,000 Lots, lt 10 clients
• Complete disaster as clients DB scaled
• Shipping requirements 100,000 Lots, 4,000
  clients
• DO THE MATH BEFORE CODING
• LotElementSize LotListSize NumClients
• 20 Bytes 100,000 4,000
• 8,000,000,000 Bytes, TWICE per minute!!

65
Why every and all are dangerous
22,000,000 DS Queries! 7,000 next highest
66
Costs of QA

• 6 player game you need at least 6 testers at
  the same time
• Testers
• Consoles, TVs and disks
• Network connections
• End-user grade network connections (ADSL)
• More players - more likely to see hard-to-find
  bugs so they are marked as online bugs
• Automated testing

67
Debugging Trials Tribulations

• MP games require MP inputs to test
• Often, sufficient machines QA testers not
  available
• Developers significantly handicapped
• Non-determinism is a serious issue
• Running the same test twice does NOT necessarily
  give the same result twice
• Offline code changes frequently breaks online
  code functionality
• Run-time debugging of networked games often
  becomes post-mortem debugging
• What helps
• Automated testing
• Architectural support for forcing causality (but
  )
• Strong isolation of online / offline code

68
Debugging Trials Tribulations
Simple TSO Test One avatar holds the Lot open,
while 3 Avatars jump in out
69
Debugging Trials Tribulations
70
Debugging Trials Tribulations

• TSO EnterLot 30 test runs, 4 behaviours
• Successful entry
• Hang or Crash
• Owner evicted, all possessions stolen
• Random results observed in all major features
  very difficult to track
• What worked
• Massive repetition of tests to establish true
  pass/fail conditions
• Continual repetition of unit tests (monkeys
  rock!)
• Initial calibration tests 400x runs per unit
  test
• 7 to 30 failure of any given unit test

71
Debugging Trials Tribulations
Monkeys Continual Repetition of Critical Path
Unit Tests
72
Impact On The Development Team

• Changes to non-networked gamplay often break
  networked play (shared code implicit
  assumptions timing changes)
• Changes to online code can inhibit content
  development (server down often equals nobody
  working)
• Non-determinisim produces frustrating defects
• Mitigating factors
• Make sure people can work in isolation no
  critical path failures that bring down the team
• Strong testing in the dev team
• Online/offline code sharing via modules, not
  ifdef

73
Stability Analysis What Brings Down The Team?
Test Case Can an Avatar Sit in a Chair?
Failures on the Critical Path block access to
much of the game. Worse, unreliable failures
use_object ()
buy_object ()
enter_house ()
buy_house ()
create_avatar ()
login ()
74
Impact On Others
75
Process Shift Comb Filter Testing
Sniff Test, Monkey Tests - Fast to run -
Catch major errors - Keeps coders working
Smoke Test, Server Sniff - Is the game
playable? - Are the servers stable under a
light load? - Do all key features work?
Full Feature Regression, Full Load Test - Do
all test suites pass? - Are the servers stable
under peak load conditions?



New code ready For checkin
Promotable to full testing
Promotable to paying customers
Full system build

• Cheap tests to catch gross errors early in the
  pipeline
• More expensive tests only run on known
  functional builds

76
Lunch Break until 200

• 200pm 400pm
• Case Studies

77
Break until 430

• 430pm 530pm
• Post Ship Issues

78
Cheating The Big Picture

• A few serious questions every developer should
  ask
• Why do we care if anyone cheats at our game?
• Is it really a problem?
• What is our relationship with our customers?
• What are we willing to do about it?
• What about our publisher?

79
Who is cheating?

• Anyone who can get away with it
• Profiling is valid here Young males are at
  greatest risk.
• We have a win at all costs culture that does not
  value honesty
• The Anonymous nature of the net allows people to
  step outside their normal behavior limits
• Single player cheats have not helped

80
Why are they cheating?

• To Win
• To cause others not to win
• Crashing others is considered a win
• To exercise control
• Almost pathological need for some
• To show off (attention)

81
Matts Rules about Cheating

• Rule 1 If you build it, they will come -- to
  hack and cheat.
• Rule 2 hacking attempts increase with the
  success of your game.
• Rule 3 Cheaters actively try to keep developers
  from learning their cheats.
• Rule 4 Your game, along with everything on the
  cheater's computer, is not secure. The files are
  not secure. Memory is not secure. Services and
  drivers are not secure.

82
Rules, cont.

• Rule 5 Obscurity is not security.
• Rule 6 Any communication over an open line is
  vulnerable to interception, analysis, and
  modification.
• Rule 7 There is no such thing as a harmless
  cheat or exploit. Cheaters are incredibly
  inventive at figuring out how to get the most out
  of any loophole or exploit.

83
More Rules

• Rule 8 Trust in the server is everything in a
  client-server game. Actually its true for all
  game types.
• Rule 9 Honest players would love for a game to
  tip them off to possible cheating. Cheaters want
  the opposite.
• Rule 10 No one likes to have their ass handed
  to them on a platter every time

84
Rules, cont.

• Rule 11 Once a cheat has been perpetrated, a
  patch, sequel, or similar game will suffer
  similar hacking attempts sooner than later.
• Rule 12 There is more than one way to
  perpetrate a cheat.
• Rule 13 It only takes one person to break an
  online community.

85
The Last Rule

• Rule 99 There is no Silver Bullet to make
  your games vulnerabilities go away. It is an
  ongoing, never ending war. What a developer can
  do is to choose which battles are worth fighting.

86
Reflecting on the rules

• What is the purpose of a computer game?
• To entertain create enjoyable experiences
• To make the developer money
• To make users desire your future products
• Who owns the game experience?
• What exactly did the player buy?
• What are the obligations and expectations?

87
General Categories of Cheats

• Reflex augmentation
• Authoritative clients
• Information exposure
• Compromised servers
• Bugs and design loopholes (exploits)
• Environmental weaknesses
• Social Cheats

88
Reflex Augmentation

• Anything that replaces human skill response
  with automated input.
• FPS Aiming proxies or bots.
• Situation analyzers or AI players.
• Timing assistants.
• Changes the game from a contest of two humans to
  one of man vs. machine.

89
Authoritative Clients

• When a players client is the final arbitrator of
  any aspect of the game
• Less of an issue for server based games
• Obviously someone has to make some definitive
  decisions in a peer to peer game.
• Can all players make the same decision and then
  compare their conclusions?

90
Information Exposure

• The revealing of any game information hidden from
  the player
• More of a problem for synchronous simulations
• Can be totally passive
• RTS Map Fog of War hacks
• Skin model hacks
• View of other players status

91
Compromised Servers

• Related to general PC security issues
• More likely done by the server operator, but
  could be done by someone else
• A perceived problem with clans
• How to detect?
• Ties into matchmaking issues.

92
Exploits

• Coding bugs and design loopholes
• Catchall category of problems
• Often discovered by accident
• Question of who makes the call

93
Environmental Weaknesses

• When the games environment is subverted
• Corrupted Communication data
• Hacked Modified Drivers
• OS bugs
• Hacked System Clocks
• Bandwidth and latency calculations

94
Social Cheats

• Social Engineering that takes place inside of a
  game
• Online forms of issues and scams weve had in
  person for thousands of years.
• Player Collusion and betrayal.
• Player in the same physical location.
• Player controls pace of game abusively
• Player text/voice chats others abusively

95
Other problems Identity Theft

• The impersonation or hijacking of a players
  online identity and legacy.
• More of problem as we push MMO games, persistent
  games and meta-games.
• Players often have a lot invested in their
  on-line avatars and identities.

96
Other Reverse Engineering

• Reverse engineering of Game Rules and Stats
• This is a design issue
• Non published Statistics
• Random Number Generators

97
Some Grey Areas

• Engine Game Configurations
• Separates the hardcore player from the causal
  gamer.
• Connection Type
• Not much we can do.
• Equipment Configuration
• Ditto for PC

98
Grey Areas cont.

• Cheaters vs. Impossibly Good Players
• What happens if someone cant tell the
  difference?
• The skill curve of your game
• Can be prohibitive to new players

99
What the developer can do

• Plan for anti-cheating efforts in the schedule.
• Audit your (final) communication formats.
• Audit your data program layout.
• Attempt to make cheats using your insider
  knowledge.
• Plan for a patch.
• Provide auditing tools to your players.
• Create persistent player identities.

100
What developers can do, cont.

• Know in advance the public stance you and your
  publisher will take if a cheat is found.
• Denial can work, but is risky
• Honesty is appreciated by players
• But it could be an admission of liability?
• Communicate with your players
• Active community management.
• Cultivate contacts in the user community

101
How long do you support a game after it ships?

• Problem no available staff
• Problem It costs
• One solution metric based on sales
• Make a decision, be willing to set an EOL date
  for support?

102
Dealing with Authoritative Clients

• Run the full simulation on all machines
• Communicate only requests, not actions
• Harder than you think
• Assume nothing regarding game rules
• Verify the full state of the simulation
• Great debugging benefits

103
Combating Information Exposure

• Add detection processes to client.
• Share information on the engines display
  operation.
• Good for complex data
• Simple data can still be passively extracted
• Audit previous gameplay for information
  assumptions.
• Use Honeypot designs.

104
Dealing with Compromised Servers

• Give the users tools to determine the servers
  state configuration.
• Link Identities to operators.
• Allow users to filter operators
• Use third party products like Punkbuster or
  SecurePlay.

105
Dealing with Exploits

• Patch, baby, patch.
• Most effective patch case
• Have a consistent policy regarding changes
• And document the changes to your users
• changes in patches not communicated can be seen
  as new cheats or bugs.

106
Other things developers can do

• Develop an internal cheater team from your QA
  team during the development process.
• Provide them with assistance to crack your own
  game.
• Provide Tools to combat Social Cheating
• Recorded games and auditing tools

107
Programming Recommendations

• Assuming that your game will be hacked for the
  purpose of cheating.
• Make an effort to determine the likely forms that
  the cheat will take, and put code in your game
  whose sole purpose is to detect evidence that a
  cheat has taken place.
• For very likely cheats, place detection code at
  multiple levels in the game if possible.
• Once cheating has been detected, delay responding
  if possible.

108
More recommendations

• Single player cheats need to go in their own
  separate game system.
• Have your game validate the integrity of its
  executables, dynamic link libraries, and
  non-variant data files

109
Check your output!

• Dont leave debug symbols or developer
  functionality compiled into the games executable
  files.
• Always scan your games executable files for
  strings and other revealing data before releasing
  the files to anyone outside of the development
  team.
• Make sure that all debugging and developer code
  is actually not compiled in external release
  versions of your game.

110
Dont forget to check data files too!

• Make sure that all developer and debugging
  information and functionality is removed from the
  data files that accompany the games executables.
  
• If your game uses a scripting language or other
  data system that exposes human readable function
  or command names, implement a tokenizer or
  compiler that will change that data to a
  non-obvious form, and dont release the
  uncompiled data files.

111
More thoughts about cheating

• Never underestimate the techniques hackers will
  use to rip apart a game.
• Once a code-related hack has been created, fixing
  it with a patch can be a no-win situation.
• Get it right the first time when you patch.
• Consider performing radical changes to the games
  code to fix a hack.
• Previous versions of a game provide a source of
  difference data for a hacker.

112
Patching Multiplayer Games

• Patching on consoles ranges from impossible, to
  very difficult, to only requiring a
  certification pass.
• Who pays for the patch to be built?
• How do you get it to your players?
• Network budget
• Impact on the player experience
• Differential copies game changes to simplify the
  problem
• What conditions require a patch
• How bad do things have to be to require a bug
  patch?
• Or do you patch to deliver new content?
• Remember that any patch has a customer service
  cost.
• Remember that the nature of a multiplayer game
  requires the players to patch if they want to
  play with others.

113
Long-term customer support

• Making support calls not happen must be a
  priority. Especially for games without a
  recurring income stream, any CS call may put you
  cash negative on the sale.
• Poor customer support can drive customers away
• However, many customers may have unreasonable
  expectations and they will never be satisified,
  regardless of the CS quality.
• Official forums tend to be extremely negative.
  Happy players usually dont post, and unhappy
  players want everyone to know they are unhappy,
  and will post in the only forum they know the
  developer/publisher reads.

114
Cross platform support

• Risks All platforms that are interoperable must
  be able to be patched roughly simultaneously
• Risks Small differences in floating point
  processing can give an unintentional advantage to
  some platforms
• Risks Depending on the differences between
  platforms, input devices may give a significant
  advantage

115
New Content

• MMP New content must be added, constantly
• Strong impact on old content
• Gameplay balances, long term additive properties
  (N2ed? Worse?)
• Code rot, tools, and long term maintainability
• Free vs user created vs commercial expansions
• Legal issues
• Distribution installation with N prior
  expansions
• Gameplay imbalances
• Providing hooks for player generated content is
  not cheap!

116
Open QA

• 530pm - 600pm

117
References

• References available as part of the final slide
  set
• Final Slides available online _at_
• www.maggotranch.com/MMP