Tuning The Money Funnel

1
Tuning The Money Funnel
Measure
Change
Measure
OR
Guess
Change
Guess
2
Background Larry Mellon20 years of engineering
_at_ scale
Research era

• Project Jade ARC distributed system research
• Distributed computing, 1983
• Distributed workspaces knowledge acquisition
• Jade Simulations parallel simulation
• Optimistic computation OS for supercomputers
• Fault-tolerant cluster computing
• Synthetic Theatre of War training virtual worlds
• DARPA 50,000 entities, real-time virtual worlds
• ADS, ASTT, HLA RTI 2.0, interest management
  cluster computing

Wife era

• EA (Maxis) The Sims Online, The Sims 2.0
• Scalable simulation architectures
• Automated testing to scale the development
  process, player performance metrics
• Emergent Game Technologies (CTO)
• Architect for scalable, flexible MMO development

3
Darius Kazemi Orbus Gameworks darius_at_orbusgamework
s.com
4
Panelists
5
Why arent we using more metrics in game design,
production and operations?

• TRADITION weve never used many metrics
• It used to be hard to get data

6
Why do we need to switch to Metrics Driven
Development?

• MMOs are a dollars in, dollars out business over
  10 year lifecycle tune your money funnel!
• Use CPM to reduce cost increase revenue
• Need to stay nimble in the marketplace
• Observe and respond to player behavior

7
Business models are changing How can metrics
affect MMO models?

• MMOs are a service industry, not a boxed goods
  business
• That means multiple recurring revenue streams
• Customer retention is king
• Content refresh rate needs to be high, and
  reflect current player interest

8
The problem small simple have become big
clumsy
Team Size
5 to 50 (tightly knit) people
50 to 300 (loosely coupled) people
Implementation Complexity
9

• Next Gen Games
• Increased Complexity
• Increased Complexity of Analysis

Art from Fun Meters for Games, Nicole Lazzaro
and Larry Mellon
10
(No Transcript)
11
Next Gen Games
12
How will metrics affect game play and game
designin the future?
13
Tuning the funnel Lower dev ops costs
Profit
New Content
Regression
Customer Support
Operations
14
Tuning the funnel Lower dev ops costs
Profit
Lower New Content Cost
Regression
Customer Support
Operations
15
Tuning the funnel Lower dev ops costs
Profit
Lower New Content Cost
Lower Testing Cost
Customer Support
Operations
16
Tuning the funnel Lower dev ops costs
Profit
Lower New Content Cost
Lower Testing Cost
Happy Customers Dont Call
Operations
17
Tuning the funnel Lower recurring costsWhat
tuning factors are useful?
Profit
Lower New Content Cost
Lower Testing Cost
Happy Customers Dont Call
Operations
Lower bandwidth CPU
18
Tuning the Incoming side of the Money Funnel

• How do metrics help
• Optimize income per user
• Increase the number of users?
• Retain users over time?
• Other industries report
• MUCH easier to retain a customer than recruit
• 80 of Harrahs Casinos revenue comes from 20
  of players

19
The 3P's of game metrics

• Player
• Performance
• Process

20
Metrics-Driven Developmenteach group needs
different metrics
Production
Metrics
Designers
Operations

• Time on task
• Fun zone
• Dead zone

Engineers
21
Metrics-Driven Development
Metrics

• CPU load per event
• Lag time under load

Engineers
22
Engineering MetricsAggregated Instrumentation
Flags Trouble Spots
Server Crash
23
Metrics-Driven Development
Metrics
Operations

• Number of each type of packet, over time
• Client failure rate
• Number of players per CPU

24
Metrics-Driven Development
Production

• Percent of world terrain completed each month
• Number of animations per month
• Number of automated tests that pass each month
• Broken build time wastage
• Number of supportable clients each month

• MUCH more valuable if you share these metrics
  team-wide!

Metrics
25
Tuning imbalances or exploits can throw your
entire economy out of kilter, but remember to
triangulate!
26
Tuning imbalances or exploits can throw your
entire economy out of kilter, but remember to
triangulate!
27
Measurable targets projected trends give you
actionable progress metrics, early enough to react
Target
Risk!
Any metric (e.g. clients)
Time
Any Time (e.g. Alpha)
28
Unstable builds are expensive slow down your
entire team!
Development
Checkin
Repeated cost of detection validation
Firefighting, not going forward
Build
Impact on others
Smoke
Regression
Play test
29
Correlated data is much more valuable than
single-source data
Click rate
Time
Time
30
Metrics change how you work!
Measure
Change
Measure
OR
Guess
Change
Guess
31
War stories What metrics have saved your bacon?

• Larry
• Engineer efficiency Compile / load / link times
• System Non-determinism level and location
• Bug frequency per source code file
• Team iteration rate Build times failure rate
• Panel?

32
How to boot strap metrics on your MMO

• If you want to get an executives attention to
  justify a metrics project, show how it will save
  them money!
• Server cost, Network cost,
• Cost of content, cost of customer service, cost
  of customer acquisition, cost of customer
  retention, cost of QA, ...
• How did this panel get started with metrics?

33
Where do you get your metrics from?
Indirect (asking players)
Direct (observing players)
Data Mining (Game, CS, bugs, )
34
Implementation complexities

• Being online lets us pull data from client side,
  live servers, the persistent database, customer
  database, etc.
• We have a broader array of data that we can
  deploy against our problems than ever before!
• But, volumes of data effects collection and
  analysis how do you deal with collection,
  correlation and distribution?
• Meta-metrics!
• Flexible reporting interface!

35
GIGO vocal minority metrics bugs
How do you deal with the Garbage In / Garbage
Out Problem?
Measuring the wrong thing, or measuring it the
wrong way, can lead to the wrong result!
Calibration and Triangulation are your best
friends Metrics are an attention
funnel! Valuable feedback to team (My work made
a visible difference!)