CS445 CS645 ECE451 Software Requirements Specification and Analysis

1
CS445 / CS645 / ECE451Software Requirements
Specification and Analysis

• Lecture 15
• Validation (and verification)

2
Readings

• Required
• Fagan (course pack)
• Ignore the code-specific stuff
• Weinberg Freedman (course pack)
• Optional
• Leffingwell Widrig, Chap. 32

3
Overview

• Validation vs. verification
• Validation techniques
• Reviews
• Walkthroughs
• Formal inspections
• Focused inspections
• Active inspections
• Checklists
• Testing
• Prototyping
• Formal validation

4
Validation and verification

• Validation Evaluate sw artifact (e.g., SRS) wrt
  customer reqs.
• Are we building the right system?
• i.e., is the spec what the customer really wants?
• Spec, Domain Req
• Verification Evaluate sw artifact wrt existing
  artifacts.
• Are we building the system right?
• e.g., does the design implement the spec?
• Design Spec
• Code Design
• Test Cases Spec
• Our focus is on Validation.

5
Validation criteria

• The criteria include
• correctness
• unambiguousness
• completeness
• consistency
• etc.
• We are checking
• That the SRS captures the stakeholders
  requirements
• User satisfaction that the system as specified
  will be usable

6
Requirements validation

• Recall the reqs engineering waterfall-like
  sub-model
• Elicitation - collect information about
  requirements
• Analysis - understanding / modelling desired
  behaviour
• Specification - documenting behaviour of proposed
  software system
• Validation - checking whether documented
  specification accomplishes customers
  requirements

7
Challenges of req validation

• There are few documents that can be used as the
  basis for validation -(
• When validating a specification, were validating
  it against the stakeholders requirements.
• Some of these may even be documented!
• If they are documented, they are probably
  expressed in natural language, which probably
  means they are open to multiple interpretations.
• All of this means that thoroughly validating a
  document is a time-intensive and error-prone
  process.

8
Validation techniques

• Reviews
• Several flavours
• Testing
• Prototyping
• Formal validation

9
Reviews

• Basic idea
• Humans (often semi-outsiders) read and analyze
  artifacts, look for problems, meet to discuss
  these problems, and agree on a set of actions to
  address the identified problems.
• Often, they will have a good idea of likely
  problem areas both inside and outside problem
  domain.
• Need both domain experts and domain-ignorant
  developers.

10
Reviews

• Broad industrial consensus
• Reviews work!
• They find more errors than testing does.
• They find errors faster than testing does.
• Everyone believes in them, even Microsoft.
• Requirements reviews are the most widely used
  technique of requirements validation.

11
Reviews

• Advantages
• Can review all kinds of software artifacts, not
  just code,
• e.g., specs, test suites, design docs
• Errors are caught much sooner than testing, when
  they are much cheaper to fix!
• Can serve to bring the entire development team
  together into the big picture, to educate
  newcomers.
• Disadvantages
• Some find it dull work.
• Often requires preparation, paperwork,
  follow-ups.
• Its time consuming and expensive
• But its usually cheaper than the alternative!

12
Social problems with reviews

• How can you get the reviewers to do a good job?
• Reviewers are rarely professional reviewers
  usually, they are software developers who have
  their own work they need to do.
• They have their own deadlines and will give their
  own work higher priority.
• Adding responsibility to the reviewer and / or
  taking an egoless (product centred, group
  buy-in) approach often works
• Why not have the author act as reviewer?
• The last person you want proofreading a document
  is the author, because (s)he reads what (s)he
  thinks the document says, not what it actually
  says!
• However, the author often acts as reviewer in
  other kinds of reviews (i.e., not proof reading).

13
Review techniques

• There are a number of specific techniques for
  requirements reviews.
• Reading and signing off
• Walkthroughs
• Formal inspections
• Focused inspections
• Active reviews

14
Reading and signing off

• Reading the document
• Read and look for errors.
• We all dont see mistakes in our own work, and
  its beneficial to have someone else look at
  their work.
• Signing off
• Reviewer signs off (approves) after reading the
  document.
• Best not have have the author do this!
• Makes the reviewer partly responsible if errors
  are subsequently found in the document
• Encourages the reviewer to be more thorough.

15
Hidden slide

• You are already doing this -- peer evaluations
  are commenting that you are reading each others
  work.
• By signing off on a document, a reviewer is
  saying that he or she has reviewed the document
  and approves it.
• It often encourages reviewers to be more
  thorough. If a reviewers name is going on the
  document, he or she will be more inclined to make
  sure that it is correct.

16
Types of group reviews

• Walkthroughs
• Informal, often high-level overview.
• Often led by author/expert to educate others on
  his/her work.
• Goal may be knowledge transfer or finding errors
  or both.
• Inspection
• Structured inspection of, say, code.
• Usually, a very detailed examination of an
  artifact.
• Participants have defined roles preparation
  required paperwork generated often follow-ups
  too.

17
Walkthroughs

• An expert or the author presents the, say,
  specification
• The other participants ask questions and give
  comments
• The tone of the meetings is informal.
• Participants may have different levels of
  understanding going into a walkthrough, so
  walkthroughs can also be tutorials.
• Advantage
• Few demands on the participants, so reviewers may
  be more likely to attend than if they had to read
  the document in order to participate.

18
Walkthroughs

• Walkthroughs may be used more often in reviews of
  requirements documents than in reviews of other
  software documents.
• Reviews of requirements documents involve a large
  number of people, because there are usually a
  large number of stakeholders to consult, and it
  may prove impossible to get everyone prepared for
  a more formal review.
• In such cases, a walkthrough may be the only
  reasonable way to ensure that the stakeholders
  have actually looked at the material.

19
Walkthroughs

• Another perceived advantage is that if the
  requirements are presented to a large audience,
  preferably one that represents a broad cross
  section of skills and viewpoints, then there is a
  hope that there are no major oversights in the
  requirements.
• In other words, multiple heads are better than
  one, and redundancy helps.

20
Course project deliverable

• Each group is asked during the last week or two
  to be reviewers during another groups
  walkthrough of some specific work done after the
  second partial SRS is handed in.
• Fix all known problems of first two partial SRSs
• Walkthrough Billing and Calling
• 3 for fixing problems and leading walkthrough
• 2 for reviewing

21
How to do the walkthroughs

• Each team, T1, should prepare its solution for
  Billing and Calling. When T1 is ready, it should
  inform its customer/TA.
• The TA will pair another of his/her teams, T2,
  with T1. (S)He will arrange a 1 hour meeting at
  a time suitable for the TA and all the members of
  teams T1 and T2.
• At the meeting, T1 will walk through its solution
  for Billing and Calling. This walkthrough will be
  done in front of a workstation so that T1 members
  can point to defining documents in UML and SDL
  online. There is no need to prepare a written
  report.
• The other team, T2, will listen to the
  walkthrough and report any problems it notices
  with the solution. It will ask questions about
  anything that is not clear.

22
Hidden slide

• Make clear to them that 2 marks for review is
  basically how they ask questions
• Emphasize that there is no excuse for not finding
  suitable time for meetings - they have at least
  regular class timeslots free as there wont be
  classes during these weeks.

23
Formal inspections Fagan 1976

• A formal inspection is a managed review process,
  with rules concerning participants and roles, and
  with strict entry and exit criteria for each step
  in process.
• The idea behind formal inspections is to improve
  the quality of the document.
• If the purpose of the walkthrough is to gain some
  assurance that there are no major oversights in
  the requirements document, then the purpose of
  the formal inspection is to strive for a
  zero-defect document.

24
Formal inspections

• Formal inspections are characterized by rules on
  who should participate, how many reviewers should
  participate and what roles they should play
• There should be from 3 to 5 reviewers
• author, moderator (6 ?author), and other
  reviewers.
• The author serves as the presenter of the
  document.
• The moderator initiates the inspection, convenes
  the meeting, assigns roles, controls the meeting,
  decides whether to do another inspection, and
  prepares as the other reviewers.
• The other reviewers prepare for inspection by
  reading the document and documenting errors.
  These reviewers will often have checklists of
  common errors, perhaps different for each reviewer

25
(No Transcript)
26
Formal inspections

• One of the moderators responsibilities is to
  postpone the inspection meeting if it looks like
  any participant has not prepared properly
• After a meeting has been postponed once for a
  particular reviewer, (s)he is not likely to be
  unprepared again.

27
Formal inspection meeting

• Prior to the meeting, there is a walkthrough to
  familiarize the reviewers with the document to be
  inspected.
• The reviewers receive copies of the document, and
  each prepares for the inspection meeting by
  reviewing the document privately to find as many
  problems as possible, possibly according his/her
  checklist.
• The focus of the inspection meeting is on finding
  problems, and not on fixing them.
• No time is wasted to fix problems indeed, a fix
  may be invalidated by a problem or fix found
  later. Fixing is left to the author after the
  inspection meeting.

28
Formal inspections

• The moderators main job at the inspection
  meeting is to keep the focus on finding problems
  and to cut off any digression to solution
  finding.
• According to some if less than 5 of the material
  is reworked, there doesnt need to be another
  inspection.
• But, its often the case that if any of the
  material is reworked, not only if more than 5 of
  the material is reworked, you should do yet
  another inspection.
• It is common to introduce new problems when
  fixing old problems and these may need to be
  found by inspection.

29
Formal inspections

• Inspection meetings are cut off after 2 hours.
• It has been observed that reviewers error
  detection rates go down after 2 hours, and it is
  better to wait and continue only when the
  reviewers are fresh.
• Inspection is considered complete only when
  rework is complete.
• Error data is collected, reported, and analyzed.
• Very Important
• The authors boss is not allowed to sit in on
  review or to see the the data!
• Inspections are not to be used for employee
  evaluation
• They are to be used only for collecting error
  data so that the software can be fixed and future
  inspections can be improved.

30
Formal inspections

• One of the motivations behind formal inspections
  is to give management a way of measuring and
  managing quality assurance.
• What can an analysis of detected errors tell us?
• It can reveal new types of errors that should be
  added to the checklists to help with future
  inspections.
• It can point to projects that are likely to be
  problematic, because significantly more errors
  were reported than usual.
• Evaluation of entry and exit points can help
  determine whether the project is on schedule.

31
BNR code inspection example

• BNR (now Nortel Networks) produced about 20 MLOC
  of source code over 10 years. Their DMS digital
  switching software is about 10 MLOC.
• They inspected 2.5 MLOC, 8 releases over 2 years.
• They found 0.8 1.0 errors per person-hour by
  inspection, which is 2 to 4 times more effective
  than testing.
• They found about 37 errors per KLOC.
• Other studies found 5095 errors per KLOC
• Type of errors
• 50 incorrect statement
• 30 missing statement
• 20 extra statement

32
BNR code inspection example

• An error diagnosed in released software takes 33
  person-hours to diagnose and repair.
• An error detected by a customer after release is
  sometimes called an escalation these are very
  expensive to fix.
• Their coders typically produce 3 to 5 KLOC of
  finished, documented code per person-year.
• This is difficult software to get correct (large,
  real-time system)!

33
Focused inspections

• In a focused inspection, the reviewers have roles
  and each looks only for specific types of errors.
• Focused inspections help avoid the problem of
  reviewers not having the time to read the whole
  document.
• The leader can assign each reviewer to tasks for
  which (s)he is most skilled.
• A reviewer who is an expert on the requirements
  can look for missing and wrong requirements.
• A UML expert can look for modelling errors and
  not be an expert about the systems requirements.
• Those who are skilled at and enjoy finding
  inconsistencies (general purpose pedants), and
  who may not be experts on anything in particular,
  can be set loose looking for inconsistencies.

34
Active reviews

• In a nutshell
• Inspection process where reviewers (who are often
  outsiders/ignoramuses) act as users of the
  artifact.
• Authors pose questions that require reviewers to
  use artifact to answer.

35
Active reviews

• Ask reviewer to use the specification Parnas
• In this case, the author poses questions for the
  reviewer to answer that can be answered only by
  reading the document.
• Not only does this force the reviewer to do the
  work, but it also exercises the document in ways
  that it will be exercised in practice by software
  users.
• One example of such a question is
• Find requirements that justify every specified
  method.
• Another example
• Give each reviewer a different set of scenarios
  and ask him / her to walk through each scenario
  with the specification, to make sure that the
  specification handles the systems role in each
  scenario

36
Active reviews

• Example For each of the access functions, the
  reviewer should answer the following questions
• Which assumptions tell you that this function can
  be implemented as described?
• Under what conditions may this function be
  applied? Which assumptions described those
  conditions?
• Is the behaviour of this function (its effects on
  other functions) described in the assumptions?

37
(No Transcript)
38
Advantages of inspections

• Considered to be more effective than testing for
  finding bugs
• Inspections find the causes of errors.
• Testing finds the symptoms of errors.
• Authors write their documents expecting others to
  be able read and understand it often improves
  work habits!
• Also, author learns from inspections what makes
  documents understandable.

39
Advantages of inspections

• Author often develops blind spots about his/her
  documents
• Fresh eyes may spot errors/flaws more easily.
• Having to explain something is an excellent way
  to learn it!
• Simple, doable, costs only time and effort.
• Some very impressive experiences, and (unlike
  many other claimed SW improvements) they have
  high credibility.
• The goal is detection and product improvement NOT
  evaluation, scorekeeping, management spot checks
• Its OK to be wrong.

40
Advantages of Inspections

• Side effects
• Encourages group buy in, team building.
• Everyone is now familiar with the code, wants to
  make it work
• Encourages handing down of knowledge from old
  hands to new people
• Encourages adherence to coding standards (so you
  have common vocabulary/expectations)
• (Ideally) reduces time needed for testing, with
  less overall effort

41
Potential problems of inspections

• Personality problems
• Person with good ideas may not express them well
• Person with bad ideas might dominate
• Some people dislike disagreements, others love
  arguing for its own sake
• Holy wars sometimes people have fundamentally
  irreconcilable points of view
• Semi-colon wars easy to get lost in trivial
  matters

42
Potential problems of inspections

• Office politics
• All comments get logged formally can get back at
  people you dont like on the record.
• But, recall that the authors boss should not
  being present.
• Its draining loses effectiveness after a couple
  of hours

43
Variations in inspections

• Amount of structure and formality in process
  varies widely
• Go through line-by-line
• Everyone read beforehand, report only problem
  spots
• Asynchronous reviews
• Web-based techniques
• Put documents on the web (usually within company
  network!)
• Reviewers can be geographically distributed,
  different timezones
• Review asynchronously or via groupware
• Groupware also called Computer Supported
  Co-operative Work (CSCW)

44
General SRS checklist

• Is a functional overview of the system provided?
• Have the software and hardware environments been
  specified?
• If assumptions that affect implementation have
  been made, are they stated?
• Has every acronym, constant, variable, and
  timeout been defined in the Data Dictionary?
• Are all the requirements, interfaces,
  constraints, definitions, etc. listed in the
  appropriate sections?

45
Interface checklist

• Are all inputs to the system specified, including
  their source, accuracy, range of values, and
  parameters?
• Are all outputs from the system specified,
  including their destination, accuracy, range of
  values, parameters and format?
• Are all screen formats specified?
• Are all report formats specified?
• Are all interface requirements between hardware,
  software, personnel, and procedures included?
• Are all communication interfaces specified,
  including handshaking, error-checking, and
  communication protocols?

46
Behavioural requirements checklist

• Have all requirements described in the problem
  statement and in subsequent communications with
  the customer been specified?
• Are all inputs to a function sufficient to
  perform the required function?
• Are undesired events/inputs considered and their
  required responses specified?
• Have the types, initial values, units been
  defined for every object attribute?
• Have the parameter and return types of all object
  operations been defined?

47
Behavioural requirements checklist

• Have all UML internal events been defined?
• i.e., the actions/messages appearing in dynamic
  models that are not invocations of object
  operations
• There should be a short description for each
  internal event, specifying the its source,
  destination (or broadcast), parameters, and brief
  definition.
• Have all internal SDL signals been defined?
• There should be a short description for each
  internal signal, specifying the events source
  destination, parameters, and brief definition.

48
SDL notation checklist

• Have all external signals been declared in the
  system spec?
• Have all internal signals been declared in the
  block specs?
• Have all data variables and timers been declared?
• Have all data variables been initialized?
• Are all timers reset when they are no longer
  needed?
• Does every process have a start state?
• Process states should not be followed by anything
  other than an input symbol.
• Input symbols should only appear as the first
  symbol in a transition between states.
• At every state, has the receipt of any possible
  input been considered?

49
Class diagram checklist

• Have the multiplicities of all associations been
  considered?
• Are each objects attributes really data values?
• Attributes that are objects themselves should be
  modelled using aggregation.
• Have generalizations of objects been considered?
• Are each objects operations really operations on
  the objects attributes?
• Operations invoked by the objects dynamic model
  are often operations on another objects data.

50
Statechart checklist

• Are there any state-transitions that can never
  occur (because the event never occurs)?
• Are all remote operations (of other objects)
  invoked as either an action or a message?
• Are all attributes used declared in the object
  model?
• Are all operations called declared in the object
  model?
• Do all superstates have an initial state?
• Have special states (e.g., abnormal termination)
  been considered?

51
Non-behavioural requirements checklist

• Is the expected response time (from users point
  of view) specified for all ops?
• Is the level of security specified?
• Is the reliability specified, including the
  consequences of software failure, the vital
  information that needs to be protected from
  failure, and the strategy for error detection and
  recovery?
• Is the maximum memory specified?
• Is the maximum storage specified?
• Are planned changes specified (i.e.,
  maintainability)?
• Are acceptable trade-offs between competing
  non-behavioural properties specified?

52
Course project

• Your TAs evaluation of your SRS is a review. How
  do they review your document?
• The evaluation is basically a reading type of
  review, but it has some aspect of a sign-off
  review.
• The TAs job is to review your work, thus, they
  are professional reviewers, and they are more
  likely to do a thorough job than a reviewer in
  practice
• They have checklists of common properties to
  evaluate, such as level of detail, clarity, model
  layout, etc.
• They look for common SDL and UML problems.
• They have key scenarios that they use to debug
  your models.

53
Overview

• Validation vs. verification
• Validation techniques
• Reviews
• Walkthroughs
• Formal inspections
• Focused inspections
• Active inspections
• Checklists
• Testing
• Prototyping
• Formal validation

54
Testing

• There are two kinds of testing that affect
  requirements engineering
• Testing the requirements themselves, aka
  validation
• Planning for the testing of the implementation
  against the requirements

55
Testing

• How on earth can you test requirements?
• Type checking
• Running an executable specification (e.g., SDL)
  and checking certain scenarios
• Usually, testing means testing of code, but we
  can also test an executable specification.
• Often, best if not performed by author of
  specification
• Simulating the product good for getting
  customer approval.
• Its unclear where the boundary is between
  testing specs and demo-ing a prototype for
  customer feedback.

56
Testing

• Advantages
• Has some of the benefits of prototyping in that
  the customer can see the product described in
  the requirements executing.
• Low-level details checking is usually more
  reliable when done by tools.
• Can be done earlier in the development lifecycle
  than most other testing

57
Testing

• Disadvantages
• Its always labour intensive and costly.
• Hand holding of tools, designing of test cases,
  etc.
• Not clear when to stop.
• Law of diminishing returns definitely a factor in
  testing.
• Many specification notations are not executable
  or even (usefully) checkable.
• As Dijkstra famously noted
• Testing can be used to show the presence of
  errors but not their absence.

58
Test case planning

• Part of the requirements specification for any
  system should describe how to determine if the
  final product satisfies the requirements.
• Often called acceptance testing.
• That is, it should include a complete test plan
• a complete collection of test cases.
• plus for each test case, what the response of the
  system should be to that case

59
Test case planning

• There are two basic kinds of test cases
• Those generated from the specification
• AKA black box testing
• Test what the system is supposed to do according
  to the spec / interface, treating the
  implementation (at the system level) as a black
  box.
• Those generated from code / implementation
• AKA white box or structural testing
• Design structure, so this is testing of the
  representation of the system, rather than the
  idea of the system (the spec)
• Metaphorically, structural testing is about
  looking for likely weak spots in the structure of
  the system, ignoring the black box semantics.
• Do all loops terminate? Are all if conditions
  tested? Is there any dead code (unreachable by
  any execution)?
• Obviously, at the requirements stage, the only
  kind that can be considered is black-box test
  cases, generated from the requirements.

60
Testing granularity

• When we are testing code, we start with unit
  tests, which are at the level of a class / module
  / file (depending on the language)
• We try to rigorously test each method / procedure
  of each unit.
• We may (should!) have both black box and white
  box tests for each unit.
• The black box tests are designed against the
  externally visible interfaces of the unit
• For each method, say, think of ways of testing it
  using only your knowledge of what it is supposed
  to do, not how it is implemented.
• The white box tests are designed against the way
  in which the code is written
• e.g., try to test all paths through a method, try
  to exercise all test conditions in ifs and loops,
  boundary values, etc.

61
Testing granularity

• Gradually, we combine the units into logical
  subsystems this is called integration testing
• We do more black box testing
• against the interface of the whole subsystem
• and more white box testing
• against our understanding of how the subparts
  depend on and interact with each other.
• For a big system, there may be several phases of
  integration testing as the subparts are merged to
  form larger and larger subsystems.

62
Testing granularity

• When we get to the system (i.e., top) level, it
  is called system testing.
• We have black box test cases based around what
  the system as a whole is designed to do (use the
  top level interface)
• and white box test cases designed around our
  understanding of the structure of the design.
• i.e., its really integration testing at the top
  level.

63
Testing granularity

• But note that these system-level test cases are
  based on what the system can do, not what the
  customer expects.
• This is why we design test cases around the
  requirements with customer input. These are
  called acceptance tests.
• Any system that can pass the acceptance tests is
  capable of satisfying the customer (and the
  requirements model).
• Obviously, we can use the SRS and the customer to
  design the acceptance tests the other tests
  require design information.

64
Scenarios as test cases

• Each of the scenarios that you have developed for
  the purpose of identifying requirements is
  basically a test case.
• A scenario gives for each user input the systems
  response, and lays them out in the order in which
  they should occur in one computation in the
  system.

65
Prototyping

• Prototyping is making a quick and dirty
  implementation of at least the most uncertain
  parts of the system, to allow the users to see
  what the requirements engineers understand of
  their requirements.
• If the specification is executable, it can be
  used as the prototype.
• If not, then its useful even to slap together
  something that fakes the execution of only
  documented scenarios.
• Its especially useful for interface-driven
  systems.
• The purpose of a prototype is to obtain a
  credible validation response.

66
Formal validation

• Ways to check if a formal specification has
  certain desirable properties
• completeness
• consistency
• mutual exclusion
• particular temporal properties
• Techniques
• Model checking (for FSMs and temporal logic)
• Theorem proving (more general for any formal
  specification)
• Key difference from other techniques formal
  verification involves checking all possible
  execution paths of the specification

67
CS445 / CS645 / ECE451Software Requirements
Specification and Analysis

• Lecture 15
• Validation (and verification)