ETA Data Validation Methodology December, 2004

1
ETA Data Validation MethodologyDecember, 2004
2
2
The Data Validation Project is the Core of the
Performance Reporting Process

Specifications Reports and Data Elements
Validated Performance Outcomes
Standardized Software
Validation of Report Data
Validation Reports
Performance Objectives

The Data Reporting and Validation System (DRVS)
is central to the reporting process and not just
an 'after the fact' validation of reports.
3
How Does Validation Work?

• Two separate processes are required to ensure
  that performance data is reliable
• Report Validation
• Data Element Validation
• ETA provides software to states and grantees that
  analyzes participant records

4
WIA Report Validation (RV)
5
Purpose of Report Validation

• Validates the accuracy of the values reported on
  the ETA 9091
• The WIA software also allows states to validate
  the accuracy of the ETA 9090 and a quarterly
  rolling four quarters report
• Assesses whether performance measures are being
  calculated accurately
• States, WIBs, and local offices can analyze
  performance outcomes and factors that may impact
  performance

6
Creating RV Extract Files for WIA

• State IT staff produce extract file with
  participant records for all participants and
  exiters included on the appropriate report
• To validate the annual report, the extract file
  should include 11 quarters of exiters and 2 years
  of participants
• Extract file format is based on the WIASRD (with
  extra fields added to help locate participant
  files in one-stops sampling unit, WIB Name,
  Office Name, Case Manager, and User Field)

7
Filling in the Sign-In Screen

• Select the type of report being validated
  annual, quarterly, or rolling four quarters
• Select the program year that the report covers
• For the quarterly and rolling four quarters
  reports, also select the due date from the drop
  down menu

8
Sign-In Screen Example

• The sign-in screen below is for an annual report
  covering program year 2001

9
Loading Extract File into Software

• State staff import the extract file into the data
  validation software
• The user can choose to skip sampling for data
  element validation, which makes the import faster
  but requires re-import to draw samples
• The software processes the file and builds
  performance outcome groups based as specified in
  Appendix A of the WIA Validation Handbook and
  Attachment C of TEGL 7-99
• The software calculates the ETA 9091 report,
  including performance for each of the special
  populations (vets, public assistance recipients,
  disabled, etc.)

10
Error Reports

• Once the extract file is loaded, the software
  creates three error reports for records that
  could not be imported
• Types of error reports
• Edit Check Error Report
• Duplicate Records Report
• Windows Errors

11
Edit Check Error Report

• Two Types of edit check errors
• Critical Errors that occur when records have
  invalid data that effect performance (e.g. youth
  who were older than 21 at registration) records
  with critical errors are rejected
• Warning Errors that occur when records have
  invalid or inappropriate data that does not
  impact on performance (e.g. the value for gender
  3) records with warning errors are not
  rejected.

12
Edit Check Error Report
13
Duplicate Records Report

• The software rejects all duplicate records (two
  or more records with the same SSN, exit date, and
  funding group)
• This is necessary to calculate the states 9091,
  but prevents the counting of participants for
  each office in sub-state reports where
  participants were served by multiple offices
• The software allows the user to easily identify
  which duplicate records are in the extract file
• The user must determine which observations to
  delete from the extract file and then reimport
  the entire extract file

14
Duplicate Records Report
15
Windows Error Report

• The Windows Error Report is located in a file in
  the data directory.
• This report identifies records that had
  significant formatting errors that prevent the
  record from being loaded in the softwares
  database.
• Records with windows errors are rejected by the
  software.

16
Windows Error Report
17
Review Error and Duplicate Reports and Reload
Files if Necessary

• Review error and duplicate reports
• Determine whether the extract file must be
  regenerated or reformatted based on whether the
  number of errors is significant
• Load the entire modified extract file if
  necessary
• The software will not import additional records
  into a file that has already been imported.

18
Analyzing Performance Outcomes

• Software sorts records into performance outcome
  groups
• Groups represent numerators and denominators for
  performance measures
• Groups also break out wage record outcomes from
  supplemental employment outcomes
• Users can view summary of all groups and detail
  for each group
• Users can export performance outcome groups
• Import these group into statistical software for
  analysis
• Export to WRIS
• Export to local areas to inform them of
  participant records that lack wage data

19
Example of WIA Performance Outcome Group

• Performance Outcome Group 1.4
• Adults
• Not employed at registration
• Employed in the first and third quarters after
  the exit quarter
• No earnings in wage records data from
  supplemental sources

20
Summary of Records in Each Group
21
Analyzing Records in Each Group

• User can view detailed data for participant
  records in each group by clicking on left column
• User can sort the records in the group by
  clicking top of column
• Sort by any field, ascending and descending
• Useful for analyzing pre-program wages and
  distribution of wages by participant

22
View Participant Records by Group
23
Downloading ETA 9091 Data

• To calculate the error rate, the software must
  have the states reported values
• Import the item totals from the ETA 9091 into the
  reported count input form
• The software uses the same format as states use
  if they upload the results of their ETA 9091 to
  EIMS
• If states use the validation software to prepare
  the ETA 9091 or ETA 9090, there are no separate
  reported counts to input

24
Report Validation Summary

• After reported values from the ETA 9091 have been
  imported the software produces the report
  validation summary
• Calculates the difference between the validation
  and reported values
• Displays an error rate for each reported value
• The validation software provides the capability
  to e-submit the report to ETA
• If states use the validation software to prepare
  the ETA 9091, the RV summary will show zeros for
  the reported values and 100 for the percent
  difference

25
(No Transcript)
26
Software as a Reporting Tool

• Software produces facsimiles of the ETA 9091 and
  ETA 9090
• These facsimiles include the validation values
• These facsimiles do not include customer
  satisfaction measures, local performance, or
  financial tables
• Software also produces a rolling four quarters
  report for internal state purposes
• Software exports validation values for the ETA
  9091 in the format required by EIMS

27
(No Transcript)
28
Overview of Report Validation
29
(No Transcript)
30
WIA Data Element Validation (DEV)
31
Purpose of Data Element Validation

• Examines a sample of participant records to
  determine the accuracy of the data elements used
  to calculate the programs reports
• Compares records against source documentation
• Data element validation provides estimates of the
  error rate for each data element being validated

32
Timing of Data Element Validation

• Most efficient to wait until all retention data
  are collected and reported
• For the annual report, samples for Program Year
  2003 reported in October 2004 validate exiter
  records for the performance year April 1, 2002
  through September 30, 2003
• Samples are not selected for the quarterly report
• Software selects samples for a rolling four
  quarters report.

33
Validating Data Elements Feasibility

• ETA can validate data elements only where
  supporting evidence exists
• Race, ethnicity, gender, and other elements are
  not validated because they are self-reported

34
Software Selects Random Samples

• Four separate samples are selected
• adults
• dislocated workers
• older youth
• younger youth
• Samples are displayed on a validation worksheet
• Worksheets can be filled in on-line, or printed
  and completed on-line later

35
Sampling

• High risk, high importance participant records
  are oversampled (e.g. employment determined by
  supplemental sources)
• Software weights the results of the validation to
  correct for oversampling and undersampling
• Software creates a report with overall state
  error rates

36
Sample of Offices Within State

• Samples are selected from a sample of offices in
  the state rather than every office for WIA and
  TAA
• Impractical for state staff to conduct validation
  for a small number of cases in a large number of
  locations
• Ensures that results will represent the overall
  quality of the states data
• More practical for state staff to conduct the
  validation on-site

37
Cluster Sampling Table
38
Conducting Data Element Validation

• Validator uses the validation handbook to
  identify the necessary source documents
• After locating source document(s), validator
  checks the accuracy of each data element on the
  worksheet
• Worksheet only contains elements selected for
  validation that apply to the participant
• Many fields on the worksheet will be blank
• Most blanks are not validated,
• The instructions indicate when blanks must be
  validated

39
Location of Source Documents

• Most source documentation will be located in the
  One-Stop center
• Some source documentation will be stored at the
  state level
• Wage record information is checked against
  states wage record files
• There may be other central administrative data
  files

40
Overview of Tasks forData Element Validation

• Determine if validation will be done on a laptop,
  from paper worksheets, a combination of laptop
  and paper worksheets, or from computers connected
  to a central server
• Assemble worksheets and arrange visits to offices
• Gather supporting documentation at office and
  complete worksheets
• Examine summary and analytical reports

41
Various Methods for Data Element Validation

• Onsite validation is essential to preserve the
  integrity of the process
• Ideal for state staff to perform validation
  onsite
• Promotes communication and mutual understanding
  with locals
• In some cases, onsite validation is impractical
• Distances are too great
• Small number of records
• With the approval of the regional office, states
  can therefore pursue a combination of onsite and
  remote validation if necessary

42
Task 1 Assemble Worksheets and Arrange Visits
to Offices

• Identify the offices that must be validated
  notify them of visit
• Assemble printed worksheets or take laptop
• Worksheets are sorted by offices so the validator
  will have a group of worksheets to complete when
  visiting each sample office

43
Task 2 Assemble Supporting Documentation and
Complete Worksheets

• Request the case files for selected cases at
  One-Stops.
• Review the case file for each data element on the
  worksheet.
• Many data elements will be blank on the worksheet
  because not every data element applies to each
  record.
• For example, if employment was established
  through wage records, the data element for
  supplemental sources will be blank.

44
Validation Instructions

• Found in the Data Validation Handbook
• Contain
• element name
• reference number
• element definition
• format
• federal sources
• state version of the sources
• validation rule

45
Validation Instructions

• Federal sources are the generic, federally
  recommended source documentation for data element
  validation
• States enter state-specific version of the
  federally approved source documentation in the
  state version of sources column
• For example, a states WIA registration form
  might be the called the WIA-922. The state would
  then put this in the state version of sources
  column next to the appropriate elements

46
Example of Validation Instructions
47
Recording the Validation Results

• Validator finds the reference number on the
  validation worksheet and matches it to the
  corresponding reference number in the validation
  instructions
• After reviewing source documentation and
  following validation rules, record the results in
  the checkbox on the worksheet

48
Two Validation Rules

• Match the data on the worksheet must be the
  same as the data in the source documentation
• For example, if the worksheet shows the birth
  date to be 7/1/1975, the source documentation
  must show the exact same date
• Support the data on the worksheet must be
  similar to the data in the source documentation.
• This instruction is used when information must be
  processed or interpreted.
• For example, if the worksheet shows a participant
  to be basic skills deficient, the source
  documentation must support this interpretation

49
DEV Worksheet Summary
50
DEV Worksheet Individual Record
51
Task 3 Examine Summary and Analytical Reports

• The software calculates the error rate for each
  element on the summary and analytical report for
  each group.
• The error rates are weighted to account for the
  clustering of offices and the stratification of
  records.

52
(No Transcript)
53
DV for LX, TAA and NFJP

• LX has full Report Validation (same functions as
  WIA)
• Limited Data Element Validation (non-random, 25
  records, only checking vs. state files)
• TAA and NFJP have only Data Element Validation
  (there is no performance report submitted by
  states or grantees)