The Importance of Adequate Verification and Validation Strategies in Risk Management

1
The Importance of Adequate Verification and
Validation Strategies in Risk Management

• Prof. Ch. Hirsch
• Vrije Universiteit Brussel
• President, NUMECA International

2
Content

• Introduction
• Error identification and management
• Verification requirements
• Validation requirements
• Beyond VV
• The new generation of simulation tools
• Non-deterministic simulations
• Robust design methodologies
• Conclusions

3
Introduction

• Whats new in simulation and Computer Assisted
  Engineering (CAE)
• CAE covers all areas of physical simulation,
  particularly
• Computational Structural Mechanics (CSM)
• Computational Fluid dynamics (CFD)
• Computational Electromagnetics (CEM)
• Growth of computer power leads to unprecedented
  levels of CAE simulations
• Million of points or degree of freedoms (DoF)
• Complex geometries
• Multiphysics (coupling fluid-thermal
  fluid-structure aero-acoustics,)
• Leading to Multidisciplinary design and
  optimization software systems (MDO)

4
Objectives

• This leads to a growing request for quality
  assurance (QA) on the simulation tools
• In order to respond to this request, joint
  efforts are required to define
• uncertainty bounds on simulation results
• limits on models
• error control methodologies
• appropriate test cases and QA strategies

5
Validation and verification

• Validation The process of determining the degree
  to which a model is an accurate representation of
  the real world from the perspective of the
  intended uses of the model.
• Verification The process of determining that a
  model implementation accurately represents the
  developer's conceptual description of the model
  and the solution to the model.
• Verification is a prerequisite, upstream of the
  validation process and requires a dedicated
  effort towards adequate methodology

6
VV
7
Identification of Error Sources

• Model errors and uncertainties
• Difference between exact solution of the
  equations and real flow
• Turbulence model uncertainties predominate
• Application uncertainties
• Uncertain boundary conditions or geometry
• Discretization or numerical errors
• Difference between exact solution and solution on
  a finite grid
• In industrial CFD, solutions are usually not grid
  independent
• Iteration or convergence error
• Difference between converged solution and result
  after n steps
• User errors
• Mistakes, blunders, carelessness, optimism, etc.,
• Programming or code errors (bugs)

8
Verification process
9
Verification process

• No matter how many tests and empirical
  experiments we perform, we can never prove that a
  software implementation is error free, or has
  zero software defects.
• Experience suggests that one can never totally
  eliminate bugs from complex CFD software,
  certainly not simply by performing algorithm
  testing alone.
• An important approach to algorithmic testing in
  verification is the method of manufactured
  solutions

From W.L. Oberkampf, T. G. Trucano, Ch. Hirsch
(2002), Verification, Validation, and Predictive
Capability in Computational Engineering and
Physics. Paper presented at FOUNDATIONS 02,
Foundations for Verification and Validation in
the 21st Century Workshop, October 2002
10
Validation process
11
Validation hierarchy
12
Validation Application domain

• In case a) we have high confidence that the
  relevant physics is understood and modeled at a
  level that is commensurate with the needs of the
  application.
• In case b) the boundary of the domain indicates
  that outside this region there is a degradation
  in confidence in the quantitative predictive
  capability of the model.
• Outside the validation domain the model is
  credible, but its quantitative capability has not
  be demonstrated.

From W.L. Oberkampf, T. G. Trucano, Ch. Hirsch
(2002), Verification, Validation, and Predictive
Capability in Computational Engineering and
Physics. Paper presented at FOUNDATIONS 02,
Foundations for Verification and Validation in
the 21st Century Workshop, October 2002
13
Validation Application domain

• Figure 5a depicts the prevalent situation in
  engineering which shows the complete overlap of
  the validation domain with the application
  domain. The vast majority of modern engineering
  system design is represented in Fig. 5a.
• Figure 5b represents the occasional engineering
  situation where there is significant overlap
  between the validation domain and the application
  domain. Some examples are prediction of crash
  response of new automobile structures, entry of
  spacecraft probes into the atmosphere of another
  planet, and the structural response of new
  designs for deep-water offshore oil platforms.
• Figure 5c depicts the situation where there is no
  overlap between the validation domain and the
  application domain.
• Predictions for many high-consequence systems are
  in this realm because we are not able to perform
  experiments for closely related conditions.
• The inference from the validation domain can only
  be made using both physics-based models and
  statistical methods. Model calibration, which
  employs explicit tuning or updating of model
  parameters to achieve some degree of agreement
  with existing validation experiments, does not
  fully assess uncertainty in the predictive use of
  the model.
• The requirement for predictive code use far from
  the validation database necessitates
  extrapolation beyond the understanding gained
  strictly from experimental validation data.
• For example, the modeling of specific types of
  interactions of physical processes may not have
  been validated together in the given validation
  database. There is then uncertainty in the
  accuracy of model predictions describing such
  interactions.

14
Dramatic illustration

• A significant example is to be found in the
  recent report from NASA about the failure of the
  scramjet hypersonic vehicle experiment X-43A.
• The report, issued recently at http//www.space.co
  m, mentionsSPACE.com has learned that the
  failure of the NASA X-43A hypersonic aircraft in
  June 2001 was the result of inaccuracies in
  computer and wind-tunnel tests that were based on
  insufficient design information about the vehicle
  itself. According to the NASA-MIB documents, the
  X-43A Hyper X launch vehicle "failed because the
  vehicle control system design was deficient for
  the trajectory flown due to inaccurate analytical
  models, which overestimated the system margins."

15
VERIFICATION AND VALIDATION EFFORTS IN EUROPE
16
European activities

• Various collective efforts in Europe for CFD
  validation
• ERCOFTAC privately funded action on CFD Quality
  and Trust
• QNET-CFD Thematic Network on QT
• FLOWNET Collection of data for validation test
  cases
• ECORA QT for CFD in the nuclear industry

17
PHASE 1ERCOFTAC BEST PRACTICE GUIDELINES
18
Objectives

• Guide to avoid the most common pitfalls
• Guidelines not specific to individual codes,
  methods or applications
• Guidelines that have very wide support
• Not exhaustive 20 of rules to cover 80 of
  aspects
• Modular form for each topic
• Discussion section
• Problem description and discussion, including
  references to important books, articles and
  reviews with relevant examples for the
  user.
• Short simple statements of advice which provide
  clear guidance, are generally accepted, and are
  easily understandable without elaborate
  mathematics.

19
Follow-up
Expert Knowledge-base Expert knowledge on limits
of physical models in specific flow regimes
Extension Extension of scope of BPG to more
difficult flow regimes
RevisionUpdates, corrections andimprovements
within current scope
Application ProceduresAdvice on how to best do
CFD and statements on accuracy for relevant
industrial cases
20
PHASE II THE QNET-CFD NETWORK
21
Objectives

• To assemble, structure and collate existing
  knowledge on the industrial application of CFD
  and to make these available to European industry
• To improve the quality of the industrial
  application of CFD
• To improve the level of trust that can be placed
  in industrial CFD calculations by assembling,
  structuring and collating existing know-ledge
  encapsulating the performance of models
  underlying the current generation of CFD codes
• To establish a shared database of computational
  and experimental results to support industrial
  applications
• To provide a regular state-of-the-art review on
  quality and trust
• To promote technology transfer between industries
  through workshops, regular meetings and
  electronic communication

22
Membership

• 44 participating organisations with
  representatives from
• 8 European Union member states
• 1 EFTA member state
• 2 Pre-accession states
• The membership is principally built from
  industries, although there is a strong
  representation from universities and research
  organisations
• A diverse range of industrial sectors is
  represented, including industries as diverse as
  civil construction and textiles
• Also included representatives from the major
  European CFD code vendors
• The organisations involved range from large
  industrial concerns to SMEs

23
Research Centres
Industries
Q N E T - C F D
DERA Health and Safety Executive NCSRD The
Meteorological Office CEA CIRA CIMNE Vattenfall
Utveckling AB Czech Academy of Science

• WS Atkins Science Technology
• CFS Engineering SA
• Renault
• Sulzer Innotec
• NUMECA Int.
• Electricite de France
• SNECMA Moteurs
• MTU
• AEA Technology
• BMW Rolls-Royce
• MAN Turbo
• ABB ALSTOM Technology
• Rolls-Royce Power Engineering
• Fluent Europe
• Magnox Electric
• Mott MacDonald
• SNPE Propulsion
• HR Wallingford
• Arup RD

Universities
University of Brussels (VUB) University of
Karlsruhe University of Surrey University of
Poitiers University of Southampton University of
Rome Martin-Luther-Universitat NTUA FH
Niederrhein Cranfield University University of
Florence University of Czestochowa

24
Work Procedure

• Development of knowledge base
• Develop Quality Trust knowledge base that
  consist of a library of application challenges
  (AC) within each of the industry sectors and
  information on a series of well-documented flow
  regimes that underlie these industrial
  applications
• Identification and documentation of application
  challenges
• Quality checks of application challenges
• Identification of underlying flow regimes (UFR)
• Documenting underlying flow regimes and
  development of best practice advice
• Quality checks on underlying flow regimes and
  review of best practice advice
• Development of application challenge best
  practice advice
• Review of best practice advice
• Exploitation of knowledge base after end of
  project

25
Quality Control

• Quality Control is a major element through the
  QNET-CFD operation
• Managed by a Quality Coordinator (A. Hutton)
  and a Scientific Coordinator (W. Rodi)
• A Quality procedure document has been generated,
  providing guidelines for acceptance or rejection
  of submitted ACs
• All ACs have been screened and cross-reviewed by
  2 partners
• After acceptance, they are recorded in the
  knowledge and data base.

26
Thematic Areas

• The Network is organised around 6 Thematic Areas
  (TA) aligned with the following industrial
  sectors
• TA 1 EXTERNAL AERODYNAMICS
• TA 2 COMBUSTION AND HEAT TRANSFER
• TA 3 CHEMICAL PROCESS, THERMAL HYDRAULICS AND
  NUCLEAR SAFETY
• TA 4 CIVIL CONSTRUCTION HVAC
• TA 5 ENVIRONMENT
• TA 6 TURBOMACHINERY INTERNAL FLOWS
• Each Thematic Area has a TA coordinator that
  assumes responsibility for the activities of the
  group

27
Underlying flow regimes UFR -

• Subdivided in 4 categories
• UFR 1 -- Free flows
• UFR 2 -- Flow around bodies
• UFR 3 -- Semi-confined flows
• UFR 4 -- Confined flow
• Each UFR can be attributed to several ACs
• Contains experimental data and a variety of CFD
  simulations, from which Best Practice Advice
  (BPA) guidelines are derived
• From the collection of BPAs of all the UFRs
  associated to an AC, a BPA for the AC is to be
  established

28
UFR 1 -- Free flows

• 1-01 Underexpanded jet Partner-07
• 1-02 Blade tip and tip clearance vortex flow
  Partner-09
• 1-03 Buoyant plumes Partner-12
• 1-04 Annular coaxial jets, flow and mixing
• Partner-19
• 1-05 Jet in a cross flow Partner-36

29
UFR 2 -- Flow around bodies

• 2-01 Blunt base flow (streamwise flow past
  cylinder with cut-off end)
• 2-02 Flow past cylinder  
• 2-03 Flow around oscillating airfoil  
• 2-04 Flow around (airfoils and) blades
  (subsonic) 
• 2-05 Flow around airfoils (and blades) A-airfoil
  (Ma0.15, Re/m2x106)
• 2-06 Flow around (airfoils and) blades
  (transonic) 
• 2-07 3D flow around blades
• 2-08 Flow around grid bars
• 2-09 Rotor/stator interaction

30
UFR 3 -- Semi-confined flows

• 3-01 Boundary layer interacting with wakes
  under adverse pressure gradient - NLR 7301 high
  lift configuration
• 3-02 Atmospheric boundary layer with rough wall
  
• 3-03 2D Boundary layers with pressure gradients
  
• 3-04 Laminar-turbulent boundary layer
  transition
• 3-05 Shock/boundary-layer interaction (on
  airplanes)
• 3-07 Natural and mixed convection boundary
  layers on vertical heated walls
• 3-08 3D boundary layers under various pressure
  gradients, including adverse pressure gradient
  causing separation
• 3-09 Impinging jet
• 3-10 Plane wall jet
• 3-11 Plane wall jet in counter current flow
• 3-12 Stagnation point flow
• 3-13 Flow over isolated hills/valleys (without
  dispersion)
• 3-14 Flow over surface-mounted cube/rectangular
  obstacles
• 3-15 2D flow over backward facing step
• 3-16 Wave-driven flow in a basin

31
UFR 4 -- Confined flow

• 4-01 Secondary flow in rotating and non-rotating
  channels
• 4-02 Confined coaxial swirling jets
• 4-03 Pipe flow - rotating
• 4-04 Pipe flow - non rotating
• 4-05 Curved duct/pipe flow (accelerating)
• 4-06 Swirling diffuser flow
• 4-07 Developing channel flow with mass injection
  through wall
• 4-08 Orifice/deflector flow
• 4-09 Confined buoyant plume
• 4-10 Natural convection in simple closed cavity
  
• 4-11 Simple room flow
• 4-12 Flows in chambers with multiple
  inlet/outlets
• 4-13 Compression of vortex in cavity

32
Best Practice Advices

• To be derived from the BPA of individual UFR
• BPA to be strongly supported by the evidence
  examined in the UFR document Section on
  Comparison of CFD Calculations with Experiments
• Recommendations for Future Work
• Such as new experiments to be undertaken for
  which the values of key parameters are much
  closer to those encountered in real application
  challenges
• Visit http//www.qnet-cfd.net

33
THE FUTURE OF VV
34
New approach to VV

• It is generally recognized that, although actions
  towards the reduction of simulation uncertainties
  are still required, numerous sources of
  uncertainties will always remain.
• Therefore new methodologies are required in order
  to incorporate the presence of uncertainties in
  the simulation process and in order to introduce
  the existence of these uncertain simulation
  results in the decision process related to
  industrial design.
• This implies responses to the following questions
• How to manage uncertainties in simulations, that
  is how to quantify, in a rational way, the impact
  of the different sources of uncertainties on the
  simulation results.
• Given the existence of uncertain simulation
  results, how are multidisciplinary design
  optimizations (MDO) techniques to be developed
  and applied in order to support the industrial
  design process?

35
Next steps
36
Required steps 1

• Quantification and management of simulation
  uncertainties 
• Quantification and management of uncertainties is
  required at the individual discipline level (CFD,
  CSM, CHT,) as well as at the integrated system
  level.
• The current practice of CAE simulations is based
  on deterministic parameters such as fixed
  boundary or initial conditions, fixed geometry
  and physical properties, fixed physical model
  parameters, etc.
• However, these conditions are not generally known
  precisely and are attached with unavoidable error
  levels, as listed above. Therefore one should be
  able to evaluate simulation results by
  incorporating these uncertainties in the
  simulation process, in order to approach a
  rational quantification of uncertainties,
  including the establishment of a confidence
  interval for the simulation-based predictions.

37
Required steps 1a

• Quantification of uncertainties
• Methods have to be developed to achieve the
  following objectives
• model and quantify the parameter uncertainties
  and stochastic inputs (such as initial
  conditions, boundary conditions, geometrical
  uncertainties)
• model the modeling uncertainties associated
  with physical models, transport properties,
  source terms
• specification of the nature of the parameter and
  model uncertainties (interval limits, probability
  density distributions, fuzzy logic sets,)
• quantification of the parameter (and model
  uncertainties, based on all possible sources of
  information, including experimental data,
  analytical estimates, results from computational
  processes, .

38

• Quantification of uncertain variables

39
Required steps 1b

• Management of uncertainties 
• Implement probabilistic simulations, for which
  innovative mathematical and algorithmic methods
  have to be developed for the treatment of
  differential equations containing stochastic
  input parameters and model parameters. Several
  methods are to be developed, assessed and
  validated, such as
• perturbation techniques whereby the variables of
  the problem are expanded in terms of Taylor
  series around their mean value (mainly for
  Gaussian or weakly non-Gaussian processes)
• Monte Carlo methods for complete statistics, but
  generally excessively expensive
• Fuzzy logic methods. Fuzzy logic allows to create
  models based on inexact, incomplete, or
  unreliable knowledge or data, and, moreover, to
  infer approximate behavior of the system from
  such models
• Polynomial Chaos methods, whereby the randomness
  of the solution uncertainties is represented and
  the PDF of the different solution components is
  defined at every point and time.

40
Management of uncertainties 

• Examples of probabilistic output for a model
  equation, with a relative viscosity variation

41
Probabilistic Design and Risk management

• Introduction of probabilistic simulations into
  the design and decision process

42
Probabilistic Design approach

• The objectives are therefore
• To develop aerodynamic and structural
  optimization algorithms that provides designs,
  which are robust with respect to uncertainties in
  geometry, operating conditions, and code
  simulation uncertainties.
• To control and reduce risks by providing designs
  with performances insensitive to intrinsically
  uncertain quantities
• To reduce system risks by enabling uncertainty
  quantification and design strategies at the
  conceptual design stage

43
Probabilistic Design approach

• The two major classes of uncertainty-based design
  problems are robust design problems and
  reliability-based design problems.
• A robust design problem seeks a design that is
  relatively insensitive to small changes in the
  uncertain quantities.
• A reliability-based design seeks a design that
  has a probability of failure that is less than
  some acceptable value.

44
Conclusions

• Classical verification and validation require a
  large scale effort, which has to continue,
  although we will never eliminate uncertainties.
• Risk management needs new generation methods to
  allow for
• The quantitative assessment and management of
  simulation uncertainties by Non-Deterministic
  methodologies
• Efficient probabilistic simulation outputs to
  quantify reliability bounds of the predictions
  (mean and standard deviations of relevant design
  quantities) in a rational way,
• The development and application of design
  methodologies incorporating probabilistic based
  simulations

45
Conclusions

• The non-deterministic approach is critical in
  order to
• Reduce system risks by enabling uncertainty
  quantification.
• Increase design confidence and safety by
  measuring and controlling uncertainty in
  performance predictions and to enable estimates
  of probabilities of failure

46

• Thank you for your attention !

47
Verification procedures

• From
• W.L. Oberkampf, T. G. Trucano, Ch. Hirsch (2002),
  Verification, Validation, and Predictive
  Capability in Computational Engineering and
  Physics. Paper presented at FOUNDATIONS 02,
  Foundations for Verification and Validation in
  the 21st Century Workshop, October 2002

48
The KNOWLEDGE BASE

• Created and managed by Univ. Surrey and Atkins
  company
• Collects all the data from the ACs, their
  relevant UFRs
• Covers experimental and the CFD data
• Highly interactive and user-friendly environment
• http//www.qnet-cfd.net

49
Required steps 2

• Applications to subsystems and systems
• The objective at this step is to provide
  efficient uncertainty quantification and control
  strategies and algorithms for multidisciplinary
  uncertainty quantification.
• The general methodologies are to be applied,
  validated and tested for the different
  disciplines CFD, CSM, CHT,for various aerospace
  and other representative applications.
• In a second phase, the validated subsystem
  methods are to be applied for integrated
  components and systems, to be defined by the
  different application areas (aerospace,
  automotive, environmental,).