Agile for Infrastructure

1
Agile for Infrastructure

• Applying Agile Software Development Practices to
  other IT Domains

Terry HamiltonIASSIST Computing Services
Inc.http//www.iassist.ca
2
Abstract

• Presentation Title
• Agile for Infrastructure Applying Agile
  Software Development Practices to other IT
  Domains
• Speaker
• Terry Hamilton, President of IASSIST Computing
  Service Inc.
• Abstract
• Agile Practices have shown their value for
  countless Software Development projects across
  many fields and specialties. However Software
  Development is not the only IT endeavor that can
  benefit from Agile. Agile for Infrastructure is
  a demonstration of how the principles of Agile
  can be applied by Systems teams to deliver Agile
  Infrastructure to large scale projects.

3
Not SOFTWARE DEVELOPMENT

• Agile as discussed in this presentation
• NO CODE gt Pure Server and Middleware rollout
• NO DEVELOPERS gt SysAdmin types
• Agile is applied to Infrastructure in many places
• Boeing, Citibank, IBM, more internal and
  engagements
• RFPs and Contracts starting to require Agile
• Young Agile was SWAT Teams, SkunkWorks, etc.
• Agile has same purpose as these tried true
  workarounds
• Simply applying the Lessons consistently

4

• SWAT Teams
• v2.0

• Now includes...
• a Tried and True
• Structure and Framework!

With repeatable action and tangible metrics grip!
5
Agile for Infrastructure Case Study

• AGILE (Scrum) for large cutting edge SOA build
• IBM zSeries, zOS, 200-400 Linux on zSeries,
  VMWARE
• 100s of Web Services, 10ks of users
• 40 Middleware latest versions across zOS and
  Linux on Z
• WebSphere (application layer), Tivoli (sys mgmt),
  Rational (development)
• Application Environments for SOA worth tens M
• Will replace 30yrs of COBOL, 5yrs for application
  transition
• Transactions worth 250M/hour (1500tps _at_ peak)
• Client Public facing applications with
  regulatory implications

6
Obstacles to Agile Adoption

• Obstacles are Opportunities

7
Biggest Obstacle - Chicken Egg

• How to build an infrastructure to run
  applications that have not been designed yet?
• IT Team Tell us what you need!
• Dev Team Tell us what well get!
• Large hardware buy-in costs unwillingness to
  fire the starting shot!
• Months of repeated estimating and planning and
  debate
• This alone sold Agile!
• We promised to deliver tangible, working product,
  
• month by month,
• and still be flexible to changing requirements
  from the application teams!

8
Obstacle New Skills Legacy Staff

• New Infrastructure skills becoming good with
  new Programming Lang.
• Skills with this middleware stack are rare
• Original Project plan had 25 infrastructure
  person years!
• Reality Start Now! Deliver DEV environment in
  6 months or less
• Legacy shop years of cookie cutter solutions
  (CICS, DB2, zOS)
• Same processes, people and applications for years
  and years and years
• Lets start with 10 people for 1 month and see
  where we get.
• Hand picked 8 go-getters from the staff plus two
  contractors
• Only Scrum Master had any Agile experience or
  training
• Had to break from the existing culture
• Heres a Post-it Note go set your own
  deliverables!!!
• Biggest fight dedicated team members
• Had to detach from application dependencies and
  processes

9
Obstacles Existing Processes

• Process is an overloaded word
• Many times a Process is really
• Desk Procedures only known to few individuals or
  a single team
• Habit instead of structure
• Officially following CMMI Level 3 processes
• But existing CMM supported Development, not
  infrastructure
• Years since anything but upgrades have been
  formally required
• You cannot bypass or ignore existing processes.
• But we can work in a way that doesnt engage them
• We isolated the team and the work
• removing dependencies implicitly removed
  ill-fitting processes
• Dont avoid processes just to be independent or
  different
• Re-use what works (or what is needed even if
  painful)

10
Obstacles Management Doubt

• Agile is so new it is scary
• Self Managed teams is a political phrase use
  it wisely.
• Scrum Master had to accept risk of Ownership
• Team had Ownership but additionally SM was the
  face for that risk
• Agile was being evangelized by Middle layer
  (Architects)
• Sold it UP to management and DOWN to technical
  staff
• Evangelize ruthlessly and bring in Experts for
  the outside view
• Scott Ambler could sell Agile Snow at the North
  Pole
• Had to allow oversight (Agile demands it!
  We wanted it!)
• Management updated at weekly Mgmt meetings
• Invited but yet to actually attend a Scrum or
  Planning (but thats another issue)
• Existing processes eventually drove estimates
  that were scarier than Agile
• Risk 22 days on Agile OR risk delivering late or
  never.

11
Implementing Agile

• Our customized techniques
• Localized Process Improvements

12
Daily Scrum Meeting

• Implemented AS IS!
• Typical difficulties
• Get folks to agree to another meeting each day
• Become Tech chats instead of 1 minute Updates
• Typical advantages
• Identify dependencies
• Face time for Team Members
• Early warning indicators
• Sense of dependency between team members
• Sense of urgency to the deadline

13
Deliver Working Systems

• Every 22 days delivered working middleware /
  servers
• Maybe not complete but installed, running and
  usable
• Included Install docs, Standards, Hardened
• Following iterations added tasks
• Configured Performance tuned, automation
• Document procedures, integrate with XYZ, ABC and
  MNO products
• DONE clearly defined to the team
• Reviewed and approved by the Infrastructure
  Architect / Product Owner
• Available to the team so available to review as
  required

14
Release Iteration Planning

• Agile for Infrastructure approach
• Middleware servers prioritized with
  Stakeholders before iterations
• Evolve the Operational Model as we learn
• Leveraged Iteration 0 hardware software
  pre-decided
• Iteration 0 design a cut/paste from Patterns for
  eBusiness
• Refined by learning what was wanted, needed and
  could do
• Surprisingly little re-work required
• Products not up to the marketing glossy but
  worked well
• Stories Middleware Servers (the Release Plan)
  
• Tasks Steps to document, install, configure
  (the Iteration Plan)

15
User Stories
started with none.

• We had none!
• Application couldnt to tell us what their
  requirements were!
• Devised our own A piece of Middleware or a
  Server a Story
• We took holistic approach
• Build it to Install Planning Guides
• Redbooks are great resource for IBM products
• Whitepapers but not just any whitepapers
• Best Practices as documented or consult experts
• Case-by-case asks for Application Architect input
• This input changed constantly as app designs
  floated
• Decisions / Assumptions must favor the generic
• Choices should prefer open-ended solution to
  allow changes later

16
Continuous Testing

• This was Tough!
• How to do this with Middleware and Servers?
• A completely different concept in QA
• Decided on two checkpoints
• IVT Install Verification Tests
• Use sample apps that come with the products
• Peer Reviews
• Architect (Product Owner) walkthru of docs and
  install
• Integration between products a defacto Peer
  Review
• Openly acknowledged that we couldnt be perfect
• Some changes likely required when the apps
  arrived

17
Burndown Charts

• Implemented (mostly) AS IS!
• Tracked interruptions as a separate work task
• 100 Dedication is not the same as 100 focused
• Focus and mind share are required, not just time
  spent
• Staff dont acknowledge time spent on small
  favours
• Skilled people like interruptions reflects on
  their abilities
• Go-To Person syndrome Jim always has the
  answer.
• Interruption here is Business As Usual
• Hours spent on interruptions were tracked by
  individual
• Just like a regular task but with estimated 0
  hours work

18
Burndown Charts
Panic! 5 days in and already 40hrs behind!

• By tracking Interruptions we reveal several
  problems
• Not a dedicated team
• Staff absorb extra work without revealing it
• Skills resided in specific individuals
• People didnt feel empowered to say No.

19
Self Managed Teams

• Often a tough sell
• Even after you put politics and entrenched
  practices aside
• The hardest part of getting to the team to form
• You cant form a team only a group they do the
  rest with encouragement!
• Three sided
• Management needs to know the Agile Experiment is
  in good hands
• A 30 day iteration wasted is still waste even if
  short
• Team needs to learn to trust itself and its
  members
• One loose cannon will put everyone on the
  defensive
• Trust in the PO and SM to assist (lead) and
  protect team
• Old habits these positions of authority are
  deferred to in crisis

20
Information Radiator / Kanban

• Go Big or Go Home use a Task Room, not Board
• All 4 walls of our meeting room covered in
  Post-it Notes
• TODO Wall, WIP Wall, DONE Wall,
• ACCEPTED Wall and ISSUES Wall (yes, that is 5
  walls)
• Information Blanket not Radiator Cant be
  missed!
• Post-it Notes The Best Tool Ever!
• Interactive! Team creates, moves through states,
  signed when done
• Usual boring kickoff became impassioned, engaged
  planning session
• Management stood at the door amazed the usually
  docile staff debating, cajoling, negotiating,
  juggling the to-do list
• Wiki as a team room repository
• Collaborative, version control, attachments, easy
  access
• Easy to learn, easy to maintain, looks good too

21
Information Radiator - Task Room
Above the Line uncommitted
WIP Wall
Iteration Backlog (products servers todo)
TODO Wall
Tasks / Stories committed
Done Wall
22
Accomplishments

• Our Successes and Failures

23
Successes

• Other projects starting to evaluate Agile
• Imitation of the project because it was a success
• 1st Agile project and 1st Iteration by 1st Scrum
  Team
• only 5 off estimated hours (after
  interruptions accounted for)
• 200 Linux on zSeries servers including
  Middleware
• On time Estimated in January, on time in
  October
• On budget actually reduced dedicated team size
  over time
• On quality 2 months of use and 2 only
  mis-configurations (bugs) so far
• PROD ready at least 2 months ahead of
  applications being ready for it

24
Successes

• Demonstrate ability to identify and manage risks
  and issues
• The Burndown is Information
• Project plans, Gantts, Complete dont reveal
  information, only document facts
• Obstacles tracked daily via Scrum
• Risks, deferred workload, assumptions on Wiki as
  they were found
• Sr. Mgmt presented to Executives on our success
• Made Agile more visible throughout organization
  and beyond
• Users are satisfied, Team feels successful, Mgmt
  relieved
• Stakeholders can then start their work happy

25
Failures

• Agile Thats when you skip the documentation,
  right?
• Highly visible but not highly understood
• Not intended as a training exercise but could
  leverage better
• Depth of understanding limited to the guys with
  the post-it project plan
• Suddenly seeing Iteration instead of Phase on
  Waterfall plans
• Iteration is obviously the cool new thing
• The server guys did it so we can do it
• Training and Education
• One Team now has experience with Agile and
  Retrospectives were positive but need to grow the
  of teams individuals with knowledge.
• Have not yet taken opportunity to train more
  Product Owners or Scrum Masters, limiting
  adoption chances.

26
Failures

• Generalists
• Cross-over knowledge gained but not enough to
  support
• Middleware products require too much depth
• Skills, techniques, dependencies are specific
• Department structures hard to break
• Structure separated skills, most returned to
  their traditional groups
• Transition to Business As Usual
• Still negotiating to show that Agile works on BAU
  support
• Works for one time build cycle but unproven on
  support day-to-day

27
Lessons

• Sponsor Absolutely required.
• 2nd Level management or higher
• Person authorized to dictate allocation of staff
  across departments
• Evangelists
• Bring someone from outside to sell Agile Scott
  Ambler works.
• Have someone inside to promote and start/run it
• Start with Agile/Scrum Skeleton and customize to
  fit
• RESIST the urge to fall back to traditional
  approaches
• Yes, Excel is nice. No, it is not a time
  tracking tool!
• Dont mess with the basics when you are starting
  out
• Iterations of fixed length lt 30 days, Daily
  Standup meeting, Dedicated resources

28
Lessons

• Burn Down chart is your absolute best friend
• Simplest tool but hard for uninitiated to
  understand take the time to teach
• Form the team carefully
• 1st Agile team are picked volunteers, not pure
  volunteers, not assigned
• Seed the team with potential tech skills arent
  everything a team needs
• Load the dice
• On Project 1, Iteration 1, be conservative with
  estimates, limit committed deliverables, allow
  time for teaching, do training in advance
  (iteration 0)
• Management willing to risk agile will usually
  understand the value of motivating the team on
  the first iteration
• Do not cheat but make Iteration 1 a success by
  preparing it right

29
QA