The Rollercoaster of Product Development.

1
The Roller-coaster of Product Development.

• Tony Gibbs
• Presentation to BSc (Hons)
• CRTS course _at_ UWE.
• 22/11/2004

2
Agenda

• War stories and successes of Product Development.
• With thoughts from the University of Life!
• Contexts
• Technical
• Environmental
• Financial
• Organisation
• Managerial

3
My background

• BT apprenticeship before BSc (Hons) degree.
• BSc (Hons) degree in MCI (now CRTS), 2(ii), from
  UWE in 1989 (I.e. here!)
• 15 years product development experience
  post-degree so far.
• Embedded single board computer in small team (me
  1 technician.)
• Complex system in large team (?25 million devt
  spend.)
• Software team lead and Test team lead.
• Now a part of a 2.7 billion UK-wide Public
  Safety radio project.
• Im a S/W Engineer mostly!

4
Technical ups downs

• One bug OFTEN covers another one.
• My customer wanted some software from me
  urgently.
• With a specific bug fixed in about 60K of C code.
• I fixed it on Christmas Eve, tested it, and sent
  it out at 6pm.
• But the C code bug was covering a Assembler bug
  which could not be tested for previously!
• My customer found the 2nd bug first!
• Between Christmas and New Year!

5
Technical ups and downs

• Other people companies make mistakes outside of
  your control or influence.
• A Prototype for a Digital TETRA radio was a PCB
  that was A4-paper sized.
• The products PCB could only be shrunk to its
  required size with arrival of a Digital ASIC.
• The ASICs all failed test because the Fabs
  subcontractor made a ve mask from a ve mask!
• 4 6 weeks elapsed slip in the product plan!

6
Technical ups and downs

• Try to understand other peoples pressures and
  needs, not just your own.
• I needed to host an RTOS (VxWorks) on some new
  custom designed hardware, but using some
  existing available s/w device drivers.
• H/w team lead needed to reduce component count
  and cost.
• A 10 component was expensive.
• RTOS license cost ?3 per board (run-time
  license.)
• HOWEVER SUCCESS!! USING AN RTOS SAVED DEVT
  TIME DEVT MONEY on S/w!
• Cost Devt cost /n Unit Cost.
• But I needed to listen to the H/w team as well.

7
Technical ups and downs

• Launch Timescales are short getting shorter
  (e.g. Motorola with 17 new phones in 2004.)
• Need to develop hardware software in parallel.
• Need to REDUCE development RISK.
• I needed to keep a 10 person software team
  working to design, code and unit test software
  WITHOUT any real h/w being available.
• I needed to help h/w team be able to test
  prototype hardware.
• RTOS simulator on Sun Unix network allowed s/w
  h/w development to continue in parallel.
• Monitor prom helped BOTH h/w team RTOS port.
• Code was free from processor support chip
  manufacturers.
• SUCCESS!! Complex Real-time process ran 1st time
  at real speed (?500?s) !! A relief all-round!!
• RTOS tools allowed parallel devt!!

8
Technical Ups Down

• Problems blamed on Software might be Hardware
  related.
• RTOS being ported onto 2nd iteration of custom
  hardware.
• Hardware chip count being reduced, so a VHDL
  version of 16550 UART was put in an FPGA.
• RTOS could not do RS232 serial comms with a
  bought in device driver.
• Turned out to be a Hardware problem in VHDL UART.

9
Technical Ups Down

• If the Software has not been changed but it does
  not run anymore, then it could be the Hardware.
• A complete batch of Processor boards came through
  the production line, and 100 failed test.
• The software was being blamed, but it had not
  been changed. The RTOS Software was not
  understood.
• Source code control (e.g. using RCS) was is
  important.
• Fault was intermittent.
• H/w team said they had not changed the hardware.
• But it turned out to be a PCB change of the 0v
  ground tracking.

10
Technical Ups Down

• Software needs to be well documented (in the code
  on paper.)
• A student during a summer placement had designed
  some hardware and assembler software for an alarm
  unit.
• But the software was incorrect, and not
  documented well enough.
• Sothe hardware was changed with addition of NOT
  gates (inverters) to get product out on time.

11
Technical Ups Downs

• Learn from other peoples mistakes in our
  industry, before you make them yourself!
• Design Software that avoids Memory Leaks!!!
• Put wrappers around alloc() and free() so that
  you can spot Memory Leaks, use leak detection
  tools.
• Make it easy for yourself in the Lab, because
  they are hard to spot in the field.
• Upgrades in the field are slow costly.
• E.g ?75 per mobile to re-flash it, including
  Logistics!! Disruption/Down-time/Bad News.

12
Technical Ups Down

• Quality cannot be tested in, if it is poor at the
  start.
• Design for Quality for RE-USE.
• Review the design on paper/screen before you code
  it.
• Document it!! Use structured design methods.
• I used Yourdon RT-SA/SD because I knew it
  others knew it!
• Use automated code review tools, if available.
• Use Defensive programming
• ATT Telecomms outage in 1990s for Eastern USA
  seaboard for 24 hours.
• CAUSE? No Default case in a C-language Switch
  statement.
• COST? ?50 million!!!
• For 1 line of code!!!

13
Technical Ups Downs.

• Design for Testability.
• Design in extra interfaces in to H/w, PCBs,
  Software, even if not populated with components
  in final products.
• Design in connectors for debugging equipment,
  e.g. ICEs and Logic Analysers.
• Design in a (RS232) Serial Port or two.
• Useful for output of debugging PRINTF statements.
• Design in LEDs or 7-segment displays you can use
  for debugging or for attaching Oscilloscopes on
  to.
• I learnt this lesson in BSc (Hons) MCI year 1 in
  1986, and it is so useful, e.g. to watch an
  interrupt signal.
• Software is often on Critical Path to launch, so
  make it easier to test debug it.

14
Environmental

• Not really an up or down, but something to be
  aware of.
• RoHS and WEEE
• RoHS Removal of Hazardous Substances EU
  directive.
• WEEE Waste Electrical Electronic Equipment
  EU directive.
• These will affect our industry a lot, and call
  for more attention to recycling, resource
  efficiency and sustainable development.
• Companies are already having to rework their
  equipment and processes to comply with these
  directives.

15
Organisational Ups Downs

• Marketing have requirements which are sometimes
  imprecise, but which are as good as they can do
  at the time.
• Sometimes need to prototype feature / look
  feel.
• so it can be shown to Marketing and to Potential
  customers.
• Need to prototype cheaply though.
• To reduce the business RISK.
• Need to design to target costs.

16
Organisational Ups Downs

• Sales have to have something to sell, even if it
  is being developed now!
• Need to be able to give Sales a clear definition
  of what it will do, even when still in
  development.
• Needs a clear up-front design.
• Design needs to be followed through on.
• Because it will have been sold.
• Sales will need clear progress information to
  pass on to customers.
• Sales brings in the money.

17
Project Management Triangle
18
Project Management Triangle

• The 3 aspects of the triangle have to be balanced
  against each other with 1 or 2 taking priority
  over the others.
• Not infinite time to launch products.
• However, how is Quality v Features balanced?
• Difficult to upgrade an embedded real-time system
  (e.g. mobile phone) if it has a bug in it.
• Mostly due to LOGISTICS of RECALL to UPGRADE.
• If Time is tight, then may to sacrifice some
  Features in order to maintain Quality.
• Shouldnt just squeeze more Features in if
  Quality cannot be maintained.

19
Financial

• Cash flow is the life blood of organisations.
• So, Priorities have to be set and juggled.
• Budgets have to be set and met.
• So spend on Capital items have to be justified.
• Quality improved?
• Timescale shortened?
• Both of the above together?
• Tools might have to be rented or leased rather
  than bought (initially.)
• Can also send them back if not right for the job!

20
Managerial ups downs

• The managers have to hold their nerve when other
  ups downs are going on.
• But the managers need information to brief
  others.
• The good best managers will try to keep the
  rest of the company off the workers, so they can
  focus on the technical work.
• But the workers need to supply the information,
  sometimes to the MD in person.

21
Managerial Ups Downs

• The Boss OFTEN knows better than you.
• Through experience of doing it before.
• The Boss OFTEN has some more information than
  you.
• Because higher in the organisation.
• The Boss is SOMETIMES under more pressure than
  you.
• So it is SOMETIMES not productive to challenge
  their requests.
• Support their requests.
• Not always easy to do this, and this is the
  HARDEST lesson I have had to learn!!

22
Summary (1)

• Timescales are getting shorter, whilst Features
  Sets are becoming richer.
• Good Quality is expected, that meets the
  products Cost Requirements.
• Make sure it is safe for its market.
• Cant sell a Rolls-Royce at Minis price.
• If got Quality S/w or H/w already, then REUSE IT.
  
• PORT IT NOT RE-WRITE IT.
• It has already been tested debugged!!

23
Summary (2)

• A quieter day is NOT coming, so document it now!!
• Lab books, Comments in code, README files.
• Someone will have to maintain it.
• Software is expensive, it is OFTEN re-used when
  it works already.
• RE-USE will happen more, because of pressures on
  time costs.
• Good documentation will help RE-USE.
• Example SRP1000 v SRP2000 radios.
• SRP1K 4 years development.
• SRP2K 12-15 months, with 90 SOFTWARE RE-USE
  from SRP1K.

24
Summary (3)

• Support the Boss, so they can support you.
• Give them the information they need, and listen
  to their advice requests.
• Dont give them unpleasant surprises!!
• Learn from other peoples mistakes in our
  industry.
• Dont design Memory Leaks in to Embedded
  Real-time systems.
• Cant just download a patch for it in the field!!
• Try to understand other peoples pressures
  needs.
• Everyone is under pressure today!

25
Good books.

• Roger Pressman, The Art of Software
  Engineering.
• Explains lots of software engineering basics for
  use in the real world.
• Steve McConnell, Rapid Development, Microsoft
  Press.
• A Microsoft Project Manager who tries to explain
  how to balance pressures needs.

26
Thank you for listening.

• Any Questions?