Overview of Empirical Findings on Agile: eWorkshop Results

1
Overview of Empirical Findings on
AgileeWorkshop Results

• Forrest Shull
• CeBASE Fraunhofer Center Maryland
• and
• Mikael Lindvall, Vic Basili, Barry Boehm,
  Patricia Costa, Kathleen Dangle, Roseanne
  Tesoriero, Laurie Williams, Marvin Zelkowitz

2
Outline

• What is agile development?
• How we are starting to study agile through
  eWorkshops
• a green field approach
• Results of expert discussions to refine
  hypotheses about
• Team size
• Personnel needed
• Critical project constraints
• Implications for future studies

3
Defining agile

• Agile approaches to software development are
• Iterative
• Self-organizing
• Emergent
• Reliant on tacit knowledge
• Agile follows the 4 values and 12 principles of
  the Agile Manifesto (http//www.agilemanifesto.org
  /)

4
Defining agile

• Specific methods include
• XP
• Scrum
• Feature Driven Development (FDD)
• Dynamic Systems Development Method (DSDM)
• Agile modeling
• Key practices include
• Pair programming
• Test-driven development
• Simple design / refactoring
• Collective code ownership
• Continuous integration
• And others

5
Motivation for this Work

• Empirical studies of agile methods are just
  getting started
• Little data collected yet
• Little understanding of when applicable,
  cost/benefits, tailoring for environment
• At this stage, need to
• Understand which are the important tenets
• Understand what empirical information already
  exists to support those tenets (including data,
  case study results, examples from practice)
• Collect, abstract, and disseminate that
  information
• Refine new, testable hypotheses of value to the
  community
• To achieve these goals, we planned and executed
  eWorkshops

6
eWorkshopBackground
Process 1. Choose a topic of discussion
2. Invite participants 3. Distribute
Pre-meeting information sheet 4. Establish
meeting codes for meeting analysis 5. Publish
synthesized info from pre-meeting
sheets 6. Schedule pre-meeting training on
tools 7. Set up control room 8. Conduct
meeting 9. Post-meeting analysis and synthesis
and storage 10. Dissemination of packaged
knowledge

• Roles
• On the Scene
• Lead discussants - leads the technical discussion
• Participants 3 -15 experts in their respective
  domain
• Moderator - monitors and focuses the discussion
  (e.g., proposing items on which to vote) and
  maintains the agenda
• Behind the Scene Support team operating from
  control room
• Director - assesses and sets the pace of the
  discussion
• Scribe - summarizes the discussion on whiteboard
• Analyst - analyzes responses by type
• Tech support

Control Room
7
eWorkshop Background Participants

• Designers of agile methodologies
• Scott Ambler (Agile Modeling)
• Kent Beck (XP)
• Ken Schwaber (Scrum)
• Consultants
• Don Wells (at DaimlerChrysler)
• Development managers
• Peter Hantos (Xerox)
• Gary Pollice (Rational)

• Developers/team leads
• Ken Auer (RoleModel SW)
• Bill Kleb (NASA LaRC)
• Tim Mackinnon (Connextra Ltd.)
• Randy Miller (TogetherSoft)
• William A. Wood (NASA LaRC)
• Researchers
• Hakan Erdogmus (NRC)
• Joel Martin (NRC)
• Frank Maurer (U. Calgary)

8
eWorkshop Background

• Formulated provisional hypotheses, e.g.
• Agile development works better for smaller teams
  and is more difficult for larger teams
• e.g. refactoring can be done only for small
  systems and great developers
• Agile methods fit applications that can be built
  quickly and don't require tight quality
  constraints.
• Agile methods work less well for critical,
  reliable, and safe systems

9
eWorkshop Background

• Initial response Agile is always suitable for
  all projects.
• Wherever people are willing to keep talking and
  learning.
• Projects where people are valued over other
  factors.
• Unless someone is more interested in keeping
  their job than doing the right thing.

10
Agile experiences Small systems?

• Has agile actually been applied to large teams?
• Plenty of experience up to teams of 12 people
• Some descriptions of teams around size 25
• A few data points of larger size teams
• 45 90-person teams, described in A. Cockburns
  Agile Software Development
• Isolated descriptions of teams of size 150, 800
• Face-to-face agile working groups only feasible
  up to 20-40 people
• Past that, need to apply some scale-up strategies

11
Agile experiences Small systems?

• Scale-up strategy Teams of teams
• 800-person team was organized using scrums of
  scrums
• Teams with members from multiple product lines
• Regular meetings of cross-project sub-teams
  (senior people or common technical areas)
• Need a core team responsible for the glue core
  architecture/standards
• Effective ways of coordinating teams include
• Yearly conferences to align interfaces
• Rotating people between teams in 3-month
  internships
• Shared test case results
• Examples
• 800-person team documented in Jim Highsmiths
  Agile Software Development Ecosystems
• Ron Crocker employed agile methods on large teams
  on Motorola projects (http//www.eos.dk/jaoo/2001/
  Slides/Ron_Crocker/jaoo2001.pdf)

12
Agile experiences Great developers?

• Ongoing debate about whether or not agile
  requires good people to happen
• Good people can make just about anything
  happen
• Participants agreed that a certain percentage of
  experienced people are needed for a successfully
  agile project
• Some consensus that 25-33 of the project
  personnel must be competent and experienced
• Might be as low as 10 if pairs can rotate
• Competent here means
• Real-world experience in the technology domain
• Have built similar systems in the past
• Good people communication skills

13
Agile experiences Criticality, reliability,
safety issues?

• Some disagreement about suitability
• Agile works if performance requirements are made
  explicit early, and proper level of testing can
  be planned for
• Example of 15M, 18-month agile success story
• Agile best fits applications that can be built
  bare bones very quickly, spend most of lifetime
  in maintenance
• But consensus seemed to form that the agile
  emphasis on testing is the key to working with
  critical projects
• Need to pass tests enforces that there hasnt
  been too much change to address strict
  requirements
• Customers can write acceptance tests that measure
  nonfunctional requirements (e.g. for number of
  users or memory usage)

14
Agile experiences Criticality, reliability,
safety issues?

• However, not all participants believed this
  approach was adequate to meeting nonfunctional
  constraints
• Performance requirements (e.g. throughput and
  response time) cant be certified by a certain
  number of discrete test cases
• Moreover, tight nonfunctional requirements
  require up-front commitments to design decisions
  not compatible with emergent practices or
  ruthless refactoring!
• In the end, no consensus, but a better indication
  where technology development/evaluation can help.

15
Refining the hypotheses

• Hard to plan empirical studies of full-blown
  agile methods (how to compare development to a
  traditional lifecycle?)
• Can we test the applicability of key practices in
  isolation?
• Many participants felt that the practices are
  synergistic i.e. separating them out is not
  doing agile and may not have any benefit
• Subsets may exist
• Test-driven development, unit testing,
  refactoring
• Collective code ownership and automated
  acceptance testing
• Collective code ownership and pair programming
• Also has significance for trying to introduce
  practices incrementally, for buy-in

16
Conclusions

• eWorkshops were an effective way to approach the
  topic
• Beginning to build up an initial knowledge base,
  showing what data/experiences exist to support
  confidence in the lessons learned
• Results can lead to more detailed hypotheses for
  more rigorous empirical study
• Further eWorkshops are planned
• For more info
• See http//fc-md.umd.edu/projects/Agile
• Contact fshull_at_fc-md.umd.edu