System Wide Information Management (SWIM)

1
System Wide Information Management (SWIM)
Service-Oriented Architecture (SOA) Testing
2
Agenda

• Traditional Testing
• Migration to SOA
• Conceptual SOA
• Traditional vs. SOA Testing
• SOA Testing Solutions
• Anticipated Challenges

3
Characteristics of Traditional Testing

• In most cases, non-agile but well established
  proven process
• Single-system focused not dependant on other
  systems within your business domain
• Tightly coupled
• Well-defined interfaces specified in the
  Interface Control Document (ICD) and Initial
  Requirements Document (IRD)
• Limited security constraints, if any
• Test program is based on the common Verification
  Validation (VV) test model
• Test team only needs to be technically savvy on
  the system under test, and not the entire
  business domain
• Test Tool Strategy is needed because systems are
  usually furnished with custom tools
• Test requirements only apply to the
  system/subsystem and interfaces

4
Stand Alone Server Architecture

• Applications share data and functions on an
  isolated single machine, no loose coupling
• Each application communicates with others in a
  point-to-point, one off interface design
• Difficult to upgrade or replace

5
Client Server Architecture

• Server applications share data and functions on
  a single machine, no loose coupling
• Each server based application communicates with
  others in a point-to-point, one-off interface
  design
• Supports multiple users

6
Distributed Server Architecture

• Server applications share data and functions
  across multiple server machines, possible loose
  coupling
• Each server based application communicates with
  others in a point-to-point, intranet, or internet
  design
• Supports multiple users and supplies composite
  data

7
Web Services Server Architecture

• Server applications share data and functions
  across multiple server machines, possible loose
  coupling
• External data and/or function provided externally
• Each server based application communicates with
  others in a point-to-point, intranet, or internet
  design
• Supports multiple users, supplies composite data

8
SOA

• Data collections and function hosted by internal
  and external servers loosely coupled
• Collections often communicate using a single
  interface
• Reuse and upgrades are less challenging
• Multiple remote users

9
Software Complexity Timeline
10
The Concept of SOA

• From Oasis (www.oasis-open.org),
• SOA is defined as
• An architectural style whose goal is to achieve
  loose coupling among interacting software
  agents/services.

SOA enables business flexibility in an
interoperable, technology-agnostic manner. It
consists of a composite set of business-aligned
services that support a flexible and dynamically
re-configurable end-to-end business processes
realization using interface-based service
description. SOA is a commonly chosen
implementation architecture process for today's
largely distributed businesses or systems.
11
Some Benefits of SOA are

• Reducing integration expense uses open
  standards, not expensive proprietary middleware
• Increasing asset reuse legacy systems can be
  easily wrapped as services and exposed
• Increasing business agility services and systems
  may be rapidly developed, tested, and delivered
• Reduction of business risk
• Loosely coupled a modified service does not
  affect other service or client
• Interoperable platform-neutral

12
Components that make up SOA

• Service provider
• Service consumer
• Service registry
• Governance Lead
• Administrator

• SOA services are intended to be reusable and can
  be shared within the entire business domain

13
SOA Characteristics

• SOA transport protocols
• Hypertext Transfer Protocol (HTTP)
• HTTP Secure (HTTPS)
• Java Message Service (JMS)
• Messaging
• Extensible Markup Language (XML)
• Web Services (WS)-Addressing
• SOAP
• Interface
• Web Service Description Language (WSDL)
• Quality of Service (QoS)
• Security
• Reliable messaging
• AuthN / AuthZ
• Policies
• Etc.

• Governance
• National Airspace System (NAS) Service
  Registry/Repository (NSRR)

14
Common SOA Testing Challenges

• No visibility below the User Interface to isolate
  errors
• Data driven business logic is service embedded
• Inaccessible external services
• Inability to test components that do not have an
  available User Interface
• Need for continuous validation of application
  functionality
• Inability to test composite solutions due to
  limited access or availability of dependent
  services and data needed for testing
• Inadequate or incomplete testing, resulting in
  costly problem identification and debugging once
  released into production

15
Common SOA Testing Challenges (cont.)

• Poor collaboration between development and
  Quality Assurance (QA), with minimal to no reuse
  of test assets between unit, functional,
  regression and performance testing
• Security testing needs to intensify as threat
  grows
• Test team structure will require alignment to
  business domains (processes) and not to
  technologies to ensure that agility and speed on
  SOA testing do not compromise overall service
  quality
• Organizations will have to develop and maintain
  Test Assets for key services to guarantee
  performance and security throughout the entire
  test coverage cycles

16
Traditional Testing vs. SOA Testing

• Similarities
• Both Traditional and SOA methodologies can
  utilize the success proven V-Model
• The V-Model enforces testing discipline
  throughout the project life cycle
• Both can utilize the top down or bottom up test
  approach

17
Traditional Testing vs. SOA Testing

• Differences

Traditional SOA
Limited scope (system, subsystem, interfaces) Business-wide, based on client size or subscriptions
Black Box testing White Box testing
Non-agile follows a rigid process Flexible, parallel development is a common industry practice
Expensive utilizing proprietary middleware Less expensive incorporating open source products and reusable services
Testing within the system boundary Testing involve elements inside and outside of system boundary
A/B level requirement testing Service-component-level testing
Commonly available test tools Customizable, environmentally adaptable SOA based test tools are needed
Platform specific Universal platform
Traditional systems troubleshooting is straight forward and failures are easy to identify SOA Troubleshooting is more complex end-to-end testing involving multiple systems
18
Traditional Testing vs. SOA Testing
Differences continued
Traditional SOA
Only requirements and regulations need to be validated and verified Governance is involved and industry standards apply (Quality of Availability (QoA) on performance, regulatory policies, business policies, audit policies, and infrastructure policies)
Small scale security testing, usually reserved at the end of test schedule SOA security testing process requires continuous validation throughout the project cycle
Test data are easily to control and configure. Test data generated and modified at different systems, therefore hard to control (data aggregation)
Creating a fall-back backup configuration for the traditional test bed is simple Creating a fall-back configuration for entire SOA type distribution system is complex especially when open source products are involved (frequent version updates)
Larger portion of testing occurs after the development phase of the project SOA testing occurs from the beginning alongside the project business and requirements development
Regression testing occurs only when major functional requirement changes occur or when bugs are fixed (intensive but less frequent) Regression testing is needed when software updates occurs and only affects the updated service units more frequent, but lightly scaled
19
SOA Testing Solutions

• Service Level Testing (SLT) emphasis
• Security Testing and the project lifecycle
• SOA and NAS experts leverage
• Test Tool Strategy implementation

20
SLT Emphasis

• SLT Emphasis
• Test emphasis at the SLT level over Governance
  testing, service-component level testing,
  integration-level testing, process/orchestration-l
  evel testing, system-level testing, and Security
  testing
• Formal Code Peer reviews
• Ensures standards and compliance are met
• Identifies potential interoperability,
  performance, and security defects and weaknesses
• Continuous Functional, Performance and Security
  tests against the Services
• Requires the use of test tools and/or test
  harnesses
• Quality Entrance and Exit Criteria for SLT
• Ensures service usability

21
Security Testing and the Project Lifecycle

• Security requirements should be established
• Security risk assessments should be performed
  during the technical design phases
• Technical deliverables should be validated in
  accordance with the groups security standards
• Security tests should be performed at the service
  level and not just the delivered integrated
  system level
• Security audits should be performed and reported
  on periodically

22
SOA and NAS Experts Leverage

• Subject Matter Experts / Domain Experts
• NAS expertise
• SOA domain expertise
• Software development expertise
• Certified Resources
• Certified individuals
• Certified tools
• Software Support
• Commercial Off-The-Shelf (COTS) support contracts
• Open Source community and committers
• Knowledge Sharing
• Content repositories (SWIM Wiki, Knowledge
  Services Network (KSN))
• Working groups

23
Test Tool Strategy Implementation

• SOA Test Tools
• iTKO LISA, soapUI
• XML Tools
• Altova XMLSpy, XML Copy Editor
• Integrated Development Environments (IDEs)
• FUSE IDE for Camel, Eclipse IDE, NetBeans IDE,
  IntelliJ IDEA
• Web Consoles and Browsers
• Web consoles for ActiveMQ, Camel, Oracle AS,
  Glassfish, etc.
• Fuse HQ, Lightweight Directory Access Protocol
  (LDAP) browsers, management and monitoring tools

24
Return On Investment
25
Business and Financial Metrics

• Efficiencies associated with service reuse
• Integration time savings
• Related opportunity costs
• Cost savings/avoidance
• Reduction in project and maintenance cost
• SOA ROI () Cost Savings/Efficiencies Achieved
  - All SOA-Related Investments
• SOA ROI () SOA ROI () All SOA-Related
  Investments

Source Leo Shuster, who is responsible for
National City Bank's SOA
26
SOA Performance Metrics

• Overall ROI can be projected by per service ROIs
• Overall ROI can be projected by per service
  revenue
• Monitor service growth rate/reuse to ensure reuse
  maximization
• Design to deploy business agility
• Mean time to service change agility measurement
• Reliability mean time to failure
• Service vitality and revenue tracked for 12 months

27
Anticipated Future Challenges

• Services Metadata
• Employing Governance in regards to managing and
  providing information on how services interact
• Level of Security
• Protection against theft of sensitive data and
  application infrastructure vulnerabilities
• Test Coverage
• Continuous services testing by producers and
  consumers to ensure expected functionality
• Scalability
• Specialized support for persistence, failover,
  and load balancing
• XML Message Size
• Additional bandwidth and resources for
  parsing/processing
• Increase in Connections
• Connection management for server burden relief
• Shared services hosting to avoid latency from
  additional hops

28
Conclusion

• Traditional Testing has its place
• Migrate to SOA where appropriate
• SOA brings benefits such as loose coupling,
  code/services reuse, and remote users
• SOA test products are numerous, mature and vary
  for many different use cases
• SOA challenges have been identified with many
  modeled solutions available to pursue

29
Questions and Comments
30
Back up
31
Keys to a Successful SOA Solution

• Definition of SOA
• Find Business Partners
• Aim for Critical Mass
• Franchise