Distributed software projects

1
Distributed software projects

• Tim Cornwell
• Associate Director for Data Management

NRAO is a facility of the National Science
Foundation operated under cooperative agreement
by Associated Universities, Inc.
2
Challenges for NVO

• NVO requires well-coordinated, state-of-the-art,
  team-based software development
• Geographical distribution is a major obstacle to
  software development
• Will NVO work as an exercise in distributed
  software development?
• My personal comments
• Based on experience in NRAO
• 4 sites around US
• And as AIPS Project Manager (March 1995
  August 2001)
• 7 sites around the world
• Scale is 20 people, 130 person-years
• And other projects seen from a distance

3
Software development 101

• Develop software iteratively
• Make multiple passes, revising requirements,
  specifications, design, implementation, testing
  as necessary on each pass
• Manage requirements
• Know what you are designing for
• Only modify requirements at specific milestones
• Use component-based architectures
• Test and verify software often
• Smoke-and-burn often
• Control changes to software
• Revision control and discrete releases

4
Things that can go wrong.

• Organizational
• Scope union of all interests (rather than
  intersection)
• Different visions at different locations
• Failure to resolve Research vs. Production
  dichotomy
• Fragmentation of activities to be aligned with
  local interest
• Politicization of technical choices via upper
  management
• Larger management culture cannot accommodate
  iterative development
• Personal
• Not Invented Here limits pace of development or
  degree of library reuse
• Maytag repair man syndrome does X at site Y
  ever do any work?
• Insufficient face time for design
• Community
• No community buy-in
• Too long to delivery

5
Things you can do right (1)

• People
• Recruit very smart, hardworking, social team
  workers
• Management
• Take management seriously
• Project manager/coordinator and deputy
• Each site has a manager
• One key technical leader (with credibility)
• Team with many recognized and respected
  specialists
• Meaningful work packages
• Processes
• Simple but shared processes
• Commitment to iterative design
• Short e.g. 6 - 9 month development cycle

6
Things you can do right (2)

• Communication
• Lots of archived communication
• Regular reporting (weekly, at end of development
  cycle)
• Weekly reports from everyone to everyone
• Attention to sociology
• Regular phone meetings, videoconferences,
  face-to-face meetings
• No flaming allowed in email or via phone or in
  check-in logs!
• Travel

7
AIPS People

• Need core of very smart, hardworking, social
  people
• Personal challenge to work within a team
• Wide range of skills needed across a project
• Everyone contributes
• No flaming in any context
• Meet socially when possible
• Extended visits to project center needed
• Especially for new project members
• Project membership has to be personally rewarding

8
AIPS Processes

• Schedule
• Regular releases (now every 6 months)
• Excellent discipline to adopt
• Testing
• Regular Smoke and Burn tests of builds
• Finds new problems quickly
• Unit testing
• Regression tests
• User testing
• Bugs
• Start bug tracking early
• Follow statistics

9
AIPS Schedules, etc.

• Plan in 6 month development cycles
• Focus and Agree via project wide discussions
• Schedule 4-5 months of few week targets per
  developer
• Tracked in weekly reports
• Each developer spends 30 time fixing defects
• Overall progress detailed in development cycle
  reports

10
AIPS weekly reports and meetings

• Everyone writes a weekly report summarizing work,
  outlining progress, raising questions, etc.
• Sent to everyone
• Appropriate items can be discussed in weekly
  project-wide meetings
• Excellent form of peer-to-peer communication
• Establishes sense of community
• Weekly phone meetings of all project members
• No longer than 20 30 minutes
• Alternates every week between 15UT and 22UT to
  pick up Europe and Australia
• Everyone attends
• Project wide news
• Raise but dont settle technical questions
• Early warning system
• Technical, scientific, personal, sociological,
  political

11
AIPS yearly meetings

• Talk over entire project face to face
• Everyone attends

• Plan next cycle down to week level
• Break bread together

12
End result

• Developed highly complex, multi-featured software
  package
• Many diverse uses
• Highly productive, geographically dispersed group
• 12,000 lines of code per developer per year
• Development has been iterative
• Basically harmonious group atmosphere