Release Engineering

1
Release Engineering

• Frank Pronk
• frank_at_pronk.ca
• April 21, 2008
• Configuration Management, Builds and Releases

2
Responsibilities

• Source code control
• Development process tools
• building, maintaining and configuring
  development/QA infrastructure
• custom development utilities
• developer tool configuration and administration
• builds and releases

3
Development process tools

• Bug tracking and workflow system such as
  Teamtrack or Jira
• Source code control such as CVS, Perforce or
  Visual Studio Team System
• Test case manager
• integration between tools (e.g. check-in refused
  without valid CR or CR for wrong branch)?
• development activity reporting

4
Test Environment

• Servers and LAN isolated from corporate network
• Set-up performance and endurance testing lab
• Simulate customer deployment environment
• If applicable, keep copies of test data on file
  server
• Maintain strict control of server configuration
• Software and patches installed as per vendor and
  industry standards

5
Source code control

• With very few exceptions, everything must be
  under source code control (if it's not in souce
  code control then it doesn't exist)?
• Release Engineering handles branching, labelling,
  and rollback
• Merging is usually a development responsibility
• Avoid accidents by limiting write access as
  necessary

6
Branching Example
7
Builds

• version all build artifacts
• build should contain everything shipped to
  customer (installer, your application, JDK,
  libraries, documentation, etc.)? for all
  supported platforms
• label build in source code control system so
  developers can check-out same code
• smoke test build quickly so QA will know if this
  build should taken for further testing

8
Builds 2

• coordinate build with development to avoid
  including partially checked-in code
• generate build manifest identifying what has
  changed and what is testable by QA, or if
  supported, transition CRs in the ready for build
  state to ready for test
• use dedicated build server(s)?

9
Automation

• Automation is your friend
• Run continuous builds and tests to quickly
  discover broken builds
• Start with automated installation on supported
  platforms, then automated smoke tests, and
  finally full regression tests done nightly
• Create tools and scripts that allow developers to
  properly test their own code

10
Multi-site development

• fast and reliable connection between sites
• scheduling automated builds can be a challenge
• use tools that work well with remote users
• each site should have a Release Engineering
  person to deal with build or source code issues
• concentrate common infrastructure where it is
  most needed and best supported

11
Release Engineering skillsets

• Configuration management
• script development
• application administration
• system administration
• source code maintenance (branching, rollback,
  etc.)?
• builds and releases

12
Best Practices

• use automation build and test frequently
• a broken build is an emergency
• use virtualisation tools such as vmware
• simulate the customer environment
• create standard desktop image for developers
• independent builds and installs
• database must also be under CM control
• data migration can be a huge issue if not managed

13
Best Practices 2

• coordinate code changes with database changes
• maintain strict control of third-party code such
  as libraries
• use pattern matching tools to scan source code
  for doubtful coding practices
• good reporting tools for dev and project managers
  to see what has changed, why and by whom (aka
  blame page)?
• monitor bug find vs. fix rate, rate of code
  churn, failure rate by module