Detecting and Handling Outliers

1
Using Quality Audits to Improve the 2007 Economic
Census
European Conference on Quality in Official
Statistics 2008 July 8 11, 2008
presented by Deborah Stempowski U.S. Census
Bureau Economic Planning Coordination Division
1
2
Agenda

• Economic Programs Directorate
• 2007 Economic Census
• Quality Management and the 2007 Economic Census
• Role of Quality Audits in Quality Management
• Quality Audits and the Macro Processing and
  Dissemination Team Process
• Macro Processing and Dissemination Processes
• Macro Team Process
• Software Quality Assurance (SQA) Evaluation
  Procedure
• Conclusions
• Current Status
• Successes and Areas for Improvement for SQA
• Quality Management Lessons Learned
• Next Steps

2
3
Economic Programs Directorate

• Measure and profile businesses and government
  organizations of the United States
• Nine organizations in the directorate
• Five subject matter divisions
• Four directorate wide support divisions
• Economic Planning and Coordination Division
  (EPCD)
• Economic Statistical Methods and Programming
  Division (ESMPD)
• Office of Statistical Methods and Research for
  Economic Programs (OSMREP)
• Center for Economic Studies (CES)

4
2007 Economic Census

• Economic Census is conducted every 5 years
• Data collection focuses on years ending in 2 and
  7
• Questionnaire mailout occurs in December of the
  collection year
• Data processing and dissemination activities
  take place for 3 years following mailout
• Covers over 23 million business establishments
• Core Census Programs
• Manufactures
• Mineral Industries
• Construction Industries
• Retail and Wholesale Trade
• Service Industries
• Finance, Insurance, and Real Estate
• Transportation, Communication, and Utilities
• Use of enterprise systems to collect, process
  and disseminate data
• There are approximately 1700 separate data
  releases over a 2.5 year period

4
5
Quality Management and the 2007 Economic Census

• Short-Term Goals
• Ensure adherence to Statistical Standards
• Ensure adherence to Project Management process
• Ensure adherence to Software Development process
• Adherence measured through quality audits
• Long-Term Goals
• Improve product quality
• Improve process quality

5
6
Role of Quality Audits in Quality Management

• Evaluation of work products and processes based
  on established standards
• Documentation of audit results
• Communication with project manager, project team
  and SQA staff
• Reporting of noncompliance issues to senior
  management as needed
• Tracking of noncompliance issues to closure
• Documentation of status of audit process
• Evaluation of audit effectiveness against
  objectives of SQA program

6
7
Organization to Conduct Quality Audits

• Separate Staff Dedicated to Quality Audits
• Independently Supervised
• Objective Review of Management Products Produced
• Objective Review of Activities Performed
• Feedback from QA Staff to Manager
• Private Audit Process

7
8
Macro Processing and Dissemination Processes

• Macro Processing and Dissemination -- Phase 3 of
  the Economic Census -- involves all processes
  related to the macro analysis and dissemination
  of the Economic Census data and its related
  programs.
• There are 11 major IT systems
• Each system is supported by a cross-divisional
  team comprised of staff from
• EPCD Provides the Business Analysts and the
  Project Managers. Often times, both of these
  roles are played by the same individual.
• Economic Statistical Methods and Programming
  Division (ESMPD) Provides the programming and
  system development staff.
• Subject matter areas (business units) Provides
  the subject matter (data) experts and system
  users.
• These teams strive to maintain, and if possible,
  improve the standardization of these systems.

8
9
Quality Audits and the Macro Processing and
Dissemination Team Process

• In the conjunction with ESMPD, EPCD developed
  the Macro Processing and Dissemination Team
  Process.
• Re-used best practices from the 2002 Economic
  Census and used them to develop our own software
  development methodology.
• Implemented a minimum process for software
  development methods and tools.
• Developed standardized tools and procedures that
  would be used for all macro teams.
• Ensured clear understanding of roles and
  responsibilities for all team members.
• Throughout the project lifecycle, work products
  and processes are reviewed to ensure compliance
  to agreed upon standards (SQA Audits).

9
10
Macro Processing and Dissemination Team Process
10
11
Quality Audits and the Macro Processing and
Dissemination Team Process
11
12
Conclusions

• Current Status
• Successes and Areas for Improvement
• Lessons Learned
• Next Steps

12
13
Current Status

• Status of the 2007 Economic Census
• All phase 3 projects are currently in
  development and testing.
• Several of the project teams have made
  preliminary deliveries of production
  applications.
• All systems are expected to be ready on or
  before October 31, 2008.
• Status of SQA activities
• We have completed the first two cycles of SQA
  evaluations for all 11 IT systems. Neither of
  these two cycles identified noncompliance issues.
• Third cycle of evaluations is scheduled to be
  complete July 22, 2008.
• Project teams have conducted lessons learned
  sessions to gain feedback.

14
Successes and Areas for Improvement for SQA

• Successful initiatives from our progress to
  date
• Clearly defined project processes and products
  as well as individual roles help to reduce
  ambiguity of tasks.
• Utilization of standard templates and documents
  reduces the learning curve for team members who
  serve on several teams.
• Independent assessment provides valuable feedback
  all during the project.
• Initiatives and processes that could be more
  effective in subsequent projects
• Auditing process and products for presence only
  does not offer much value.
• Having the SQA staff trained to review the
  contents of each document and not just the
  template that is used, would offer more value.
• After several rounds of auditing with no
  noncompliance issues, it might be acceptable to
  only audit a subset of projects.
• When established business processes have been
  working well to date, focus resources on other
  aspects of the process.
• All team members should be included in the
  methodology development process to ensure
  consideration of work from all viewpoints is
  considered.
• Improved communication plans would make
  everything flow much easier.

14
15
Quality Management Lessons Learned

• Quality Management is a continuous process. It
  cannot be turned on for the Economic Census and
  turned off for all projects between Censuses.
• Focus should be on the content of documents
  reviewed in the audit and not just the presence
  of that document.
• Clear, specific, detailed standards must be
  developed early to get everyone educated on the
  process and its expectations.
• Independent review and reporting of adherence to
  standards is of great value.
• Goal of quality audits is to tell senior
  management whether a project is adhering the
  standards. Failure to adhere to standards should
  be viewed as a teaching opportunity to improve
  skills.
• Planning and training for project management
  should include quality assurance processes and
  staff at all stages.
• Decisions on tools used for project management
  should be made early . Some quality problems
  occur because we do not make the best use of
  tools available.

15
16
Next Steps

• Receive feedback from all participants during
  each phase of the project and make modifications
  where feasible.
• Continue to carry out each project and perform
  an evaluation of the Macro Processing and
  Dissemination Team Process upon the closeout of
  all projects.
• Upon final evaluation, make modifications to the
  process and begin planning phases for the next
  cycle of projects to be audited.

16
17
Contact Information
deborah.m.stempowski_at_census.gov
17