Quality of Life through applying Software Quality Management

1
Quality of Life through applying Software Quality
Management

• A Case Study of the use of the Team Software
  Process in Games
• Vicarious Visions
• Tobi Saulnier, VP Product Development
• tobi_at_vvisions.com

2
Why so much overtime? And stress?

• Over-ambitious scope versus budget
• Inability to determine what is possible
• Inability to convince customer or team what is
  possible
• Inability to give hard numbers on the real cost
• Underestimation of time needed for
• Development
• Testing
• Fixing
• When is it really done?
• Inability to manage change
• In schedule
• In scope
• In resources
• Tend to just absorb more and more

3
Case Study Apply Software Quality Techniques

• Games are not just software
• BUT they share a lot of the problems of software
• Carnegie Mellon Software Engineering Institute
  (SEI)
• Funded by the US for tech transfer to the US
  software industry
• Some products of SEI
• Capability Maturity Model Levels 1-5
• Personal Software Process
• Team Software Process
• A bunch of other cool stuff

4
Preparation Personal Software Process

• Teaches software engineers to understand and
  improve their own process of coding
• Better individual estimates, code, problem
  solving
• Involves data collection by the individual
• Estimates versus actuals time, lines of code,
• Defects introduced and found
• Seeding historic data for future planning
• Training is a big time commitment!
• Two one-week training sessions
• Lots of homework - 10 programming assignments
• Expensive! (for game industry)
• All VV engineers attended training

5
Next A TSP Pilot Project

• TSP Team Software Process
• Everyone on team uses principles from PSP
• Not just software folks principles can be
  applied to art and design too
• Additional TSP project management framework
• Launch meetings
• Defined roles
• Change process
• In-project Meeting scripts
• We chose Spidey DS
• Short launch project all the hallmarks of no
  QoL
• Team receptive
• Lots of risk and change expected

6
Launching a TSP Project
Day 1
Day 2
Day 3
Day 4
1. Establish Product and Business Goals
4. Build Top- down and Next-Phase Plans
7. Conduct Risk Assessment
9. Hold Management Review
PM. Launch Postmortem
2. Assign Roles and Define Team Goals
5. Develop the Quality Plan
8. Prepare Management Briefing and Launch
Report
New Teams TSP Process Review
6. Build Bottom- up and Consolidated Plans
3. Produce Development Strategy
7
More about the Launch

• Allowed ALL team members to understand the game
  they were going to create and feel equally
  responsible for delivery
• Granularity of tasks elucidated the quantity of
  work and scope
• Scope reduction scenarios were considered and
  taken to provide a more reasonable schedule
• Task granularity was refined included each
  process step, not just a broad task (e.g.
  Implement Spidey Sense was broken into multiple
  PSP steps DLD, DLD Inspection, Code, Code
  Review, Code Inspection, Unit Test)
• Assumes 4 task hours per day (not 8!)
• We did this all before, didnt we?
• Maybe, sometimes, the structure makes a
  difference!

8
More About Time Tracking

• Team members log time charged to tasks, task
  completion, and defects (aka bugs)
• Provided visibility into tasks that were taking
  significantly more time than planned
• Allowed for course correction such as adding
  resources or cutting scope
• Data also used as a learning tool for planning
  the next project
• This requires trust of management
• No sharing of individual data
• Fear of productivity monitoring will prevent
  accurate data
• For instance, would be hard to implement if there
  are recent or impending layoffs

9
Task Completion Tracking

• Earned Value (EV) and target EV reported at
  weekly group meeting by each team member
• To achieve EV, task needs to be marked as
  completed in workbooks
• Allows tracking of task completion

10
Defect Logging

• Bugs encountered are logged
• Bug logging should take note of when the bug was
  injected
• This data should be used to help reduce of bugs
  in the future and reduce rework (aka increase
  quality)
• Want to remove bugs as early as possible in the
  pipeline (e.g. in design rather than
  implementation or test)
• What is a defect?
• For art?
• For design?

11
Results What went well

• Team Morale
• Launch led to team cohesiveness
• Individual ownership of tasks
• Lessened burden on leads and PM for scheduling by
  distributing work across all team members
• Team responsible for scope reduction
• Planning
• Planning was detailed
• Planning accounted for time spent besides
  implementation (i.e. design, testing, reviews,
  etc.)
• Helped reinforce pipleines, etc as each task was
  broken down into process steps which earned value
• We were able to justify the need for additional
  resources by showing specifically what those
  people would work on

12
More of what went well

• Process
• Design Inspections and Code Inspections led to
  solid, reusable code
• Team management duties divided among the team
  left more time for Leads to contribute
• Adding additional resources was feasible as the
  workbooks allowed us to easily balance and move
  tasks between people
• Reviews scheduled for Art and Design earlier,
  prevented rework
• Gathered data useful for future planning
• Lines of code per hour, animations per day, etc
• Review time
• Task hours per week
• Basic metrics for components

13
Of course, all was not perfect

• Individuals could add forgotten or new tasks to
  their workbooks which could lead to hidden
  schedule slippage
• At Launch did not schedule all tasks needed to
  complete the project, only a subset
• The TSP Excel workbook tools
• Clunky for transferring tasks between individuals
• Overwhelming and have many acronyms
• Team members failed to log defects
• Lack of buy-in from some team members (-gt bad
  data)
• Process self-discipline slipped in final couple
  of weeks

14
Bottom Line

• A huge difference, esp at the end of the project
• Team could identify needs early
• Missing resources
• Estimations that werent working out
• Work that hadnt been planned
• Resources were added relatively early
• Didnt have to fight with team about whether they
  were needed
• Clearly identified tasks already identified that
  new people could be applied to
• Crunch time still happened but shorter, and
  higher morale
• We made launch!

15
Lead Artist

• In the end I believe TSP saved the day. It
  allowed us to see the big picture rather early
  and gave us a focus.

16
Lead Designer

• It was good. I had a sense of what I had to do
  at most times.

17
Lead Engineer

• I consider the use of TSP on Spidey DS to be a
  complete and utter success. It forced us to scope
  back our game early and set out a schedule that
  could be divided among a growing team. Constant
  updates on our progress allowed us to keep
  focused on key features and resist feature creep.
  I would feel very uncomfortable running any
  software project without the TSP process in
  place.

18
Suggestions for future use

• Critical that inspections are occurring
• Process must be tailored so that team members are
  comfortable with the process and feel compelled
  to use it
• TSP needs to be customized further for design,
  art, and animation
• TSP still needs a lot of tuning to be used more
  efficiently

19

• Thanks!
• Questions?
• Also See the Related Talk Wed 9-10 am
• Spider-man 2 DS Launch
• Double the Screens, Half the Dev Cycle
• Also much more info on TSP at
• www.sei.cmu.edu