Data Processing Concepts

1
Data Processing Concepts

• CALPADS
• Training 2f

2

• INTRODUCTION

3
Introduction
WHAT IS DATA PROCESSING?
Data processing converts raw data into usable
information.
WHY UNDERSTAND DATA PROCESSING?
Understanding the way that data submitted to
CALPADS is loaded will provide you the ability to
.

• load data that creates a complete history of a
  students data (i.e. longitudinal data)
• understand how to interpret error messages
• understand how to resolve data loading problems
• correct incorrect existing data
• meet the data submission requirements

4
Introduction
WHO NEEDS DATA PROCESSING UNDERSTANDING?
All CALPADS users need to understand how data is
processed in order to confirm that the data
available in CALPADS is being correctly
interpreted and reported.

• The degree of understanding needed varies
  depending on the type and complexity of the
  interaction a user has with CALPADS

• Interactions Needing In-Depth Understanding
• Excel spreadsheet creation
• SIS extract file creation
• Data reconciliation

• Interactions Needing Basic Understanding
• Online data entry error resolution
• Batch uploads error resolution
• SSID requests
• Data analysis

5
Introduction
HOW MUCH UNDERSTANDING OF DATA PROCESSING IS
NEEDED?
Key concepts of CALPADS data processing are
needed for a basic understanding. The details of
CALPADS data processing are needed for an
in-depth understanding.

• All users should understand key concepts. The
  details needed vary according to the complexity
  of the interaction

Key Concepts for Basic Understanding

• Details for In-Depth Understanding
• Record Layouts
• Processing Methods

• Validation of data
• File Operational key
• Updateable fields

• Importance of dates
• Ownership of data
• Meaning of data elements
• Relationship of data

6
Introduction
WHAT CAN YOU LEARN IN THIS MODULE?
The topics in this module will introduce you to
.

• What longitudinal data is and what it looks like
  in CALPADS
• How dates impact data and reports (e.g.
  enrollment dates and data effective dates)
• What data in CALPADS are key to knowing how to
  add, update, and delete information (e.g.
  operational key)
• How and when certain data are entered (e.g.
  initial enrollment vs. enrollment update)
• What it means to own a students data (e.g. who
  can update a students data)
• How CALPADS data processing differs from SRRTS
• What information is available to explain the
  details of data processing (e.g. CALPADS File
  Specifications, Code Sets)

7

• CALPADS DATA

8
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL
CALPADS collects different types of data at an
individual student/staff/course level. For
students, the following information is submitted
to CALPADS and tracked over time (i.e.
longitudinally) to achieve the benefits of a
complete history.
9
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)

• Enrollment data is available for a students
  current LEA/school as well as for the
  LEAs/schools that have served the student in the
  past.

Note that all student data in this presentation
is test data, not actual student data
10
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Demographic data is maintained by effective dates
to associate the information with an enrollment
and to capture the information used for reporting
at a specific point in time.
11
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Demographic data includes not only legal names
and birth information, it also includes alias
names, race/ethnicity and parent/guardian
information.
12
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Grade level data reveals a students current
grade level at the students current school as
well as the progression through previous grade
levels by date. The effective dates allow CALPADS
to tie the grade level to the LEA/school in the
enrollment history.
13
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Student status data, maintained by date, provides
key pieces of information that determine whether
a student is reported, either currently or in the
past, as a member of certain groups of students,
such as CTE Pathway Completers, Immigrants, etc.
14
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Language History data, maintained by date,
provides key pieces of information that determine
whether a student is reported, either currently
or in the past, as a member of certain groups of
students, such as English Learners.
15
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Address History data, maintained by date,
provides locator information as well as key
pieces of information about a students
residence, either currently or in the past, that
determine whether a student is reported as a
member of certain groups of students, such as
Homeless.
16
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont)
Program data, maintained by start and end dates,
provides a students program eligibilities and
participations, both currently and in the past.
The dates associate the program data with an
enrollment.
17
CALPADS Data
LONGITUDINAL DATA AT INDIVIDUAL LEVEL (cont.)

• Individual detail data enables ..
• analysis of individual placements, services, and
  assignments for a specific point in time
• aggregation to and analysis of summary levels
  such as school, LEA, and state for a specific
  point in time
• Longitudinal data with its effective dating
  enables ..
• the association of individual level data to an
  LEA/school
• analysis of the changes that have occurred in an
  placements, services, and assignments over a
  period of time
• analysis of the changes that have occurred at a
  summary level, such as school LEA, and state over
  a period of time

18
CALPADS Data
DATA SUBMITTED

• The following summarizes the data submitted to
  CALPADS and tracked to achieve the planned
  benefits.

• Students
• SSID (Statewide Student Identifier) and
  Demographic Data
• Participation and Eligibility for Programs such
  as Special Education, Gifted and Talented,
  English Learners, etc.
• Course Enrollment, Completion, and Grade
  Information
• Enrollment and Exit Information
• Discipline Data
• Statewide Assessments
• Teachers
• SEID (Statewide Educator Identifier) and
  Demographic Data
• Course and Job Assignments

19

• DATA SUBMISSION

20
Data Submission
DATA SUBMISSION METHODS
Data is submitted to CALPADS in multiple ways.
Each LEA selects the method that is most suited
to its size, resources, and data management tools.

• Online Data Submission is available for
• LEAs with manual record keeping systems
• Locating and Assigning SSIDs
• Entering small numbers of enrollments and
  enrollment updates

21
Data Submission
DATA SUBMISSION METHODS (cont.)

• Batch Data Submission is available for
• LEAs with both manual and computerized record
  keeping systems
• Locating and Assigning SSIDs
• Entering small and large numbers of enrollments
  and enrollment updates with either of the
  following input methods
• Files extracted from local Student Information
  Management System (SIS) developed for CALPADS
• Excel Spreadsheet developed for CALPADS

22
Data Submission
BATCH DATA SUBMISSION FORMATS
Batch data is submitted to CALPADS in files, each
containing a specific type of record. Each record
contains a specific type of data. Data submitted
to CALPADS is organized into the following Record
Types

• SSID Enrollment
• Student Information
• Student Program
• Student Discipline
• Student Health
• Student Waivers

• Course Section
• Student Course Section
• Student Assessment Details (Pre-ID)
• Staff Demographics
• Staff Assignment

23
Data Submission
BATCH DATA SUBMISSION FORMATS (cont.)
Each Record Type submitted has a pre-defined
layout (i.e. order) for the elements of data to
be included in the record. The layout of the
record is the same regardless of whether it is
created by a computer system, such as a Student
Information Management System, or by an excel
spreadsheet created manually. The Record
Layouts are defined and explained in the CALPADS
File Specifications (CFS). For each Data Element
to be submitted in the record, the following
information is provided
An explanation of each of these items follows for
those who needed a detailed understanding of
CALPADS processing. Note particularly the
elements in bold.
24
Data Submission
DATA ELEMENT DESCRIPTION
25
Data Submission
DATA ELEMENT DESCRIPTION (cont.)
26
Data Submission
RECORD LAYOUTS

• The following record layouts are used to build
  and maintain the longitudinal student information
  in CALPADS..
• SSID Enrollment
• Student Information
• Student Program

27
Data Submission
RECORD LAYOUT SSID ENROLLMENT

• The SSID Enrollment record layout is used to
• Assign and enroll a new SSID in CALPADS
• Locate and enroll an existing SSID in a new
  school
• Exit a student from a school
• The data elements in the SSID Enrollment record
  are processed differently by CALPADS depending on
  whether the enrollment is for a new SSID or an
  existing SSID.
• If the SSID is a new SSID, CALPADS loads the
  demographics and grade level along with the
  enrollment data, establishing the initial
  demographic and grade level data in CALPADS
• If the SSID is an existing SSID, CALPADS does not
  load the demographics and grade level. Only the
  enrollment data is loaded. Changes to the
  initially loaded demographics and grade level are
  made with the Student Information record.

28
Data Submission
RECORD LAYOUT SSID ENROLLMENT (cont)

• The process of enrolling a student in CALPADS
  establishes the students official enrollment in
  the LEA as well as establishes ownership of the
  students data in CALPADS.
• In CALPADS, ownership of data establishes .
• Access rights to the data
• Responsibility for data maintenance
• Responsibility for accuracy and timeliness of data

The LEA in which a student is primarily enrolled
owns the students data.
Following is a partial description of the each of
the data elements of the SSID Enrollment record
layout. Note the Operational Key and Updateable
Fields. Refer to the CALPADS File Specifications
and Code Sets for the most up-to-date and
complete description.
29
Data Submission
RECORD LAYOUT SSID ENROLLMENT (cont)
30
Data Submission
RECORD LAYOUT SSID ENROLLMENT (cont)
31
Data Submission
RECORD LAYOUT SSID ENROLLMENT (cont.)
32
Data Submission
RECORD LAYOUT STUDENT INFORMATION

• The Student Information record layout is used to
  
• submit additional information about a student
• manage demographic information changes
• manage grade changes
• It should be submitted .
• when a new student is assigned an SSID in order
  to enter initial address and ethnicity/race
  information
• at the beginning of each Academic Year for each
  currently enrolled student to validate enrollment
  and record grade change
• when a student transfers between LEAs
• when a students information changes,
  particularly prior to state reporting and prior
  to the use of student data for assessment and
  accountability purposes

33
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
The Student Information record may be submitted
independently of other record types however, the
SSID being reported must have an enrollment for
the Reporting LEA that includes the effective
date for the record. This record is also used
in the Fall 1 Data Submission as a positive
confirmation of a students enrollment in the
current academic year. It must have an Effective
Start Date in the current academic year that is
less than or equal to Census Day (i.e. from July
1 through Census Day)
The Student Information record is used to verify
a current enrollment for the Census Day
headcount. .
Following is a partial description of the each of
the data elements of the Student Information
record layout. Note the Operational Key and
Updateable Fields. Refer to the CALPADS File
Specifications and Code Sets for the most
up-to-date and complete description.
34
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
35
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
36
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
37
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
38
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
39
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont)
40
Data Submission
RECORD LAYOUT-STUDENT INFORMATION (cont.)
41
Data Submission
RECORD LAYOUT STUDENT PROGRAM

• The Student Program record layout is used to
• submit program eligibility or participation data
• manage program information changes
• It should be submitted .
• when a student becomes eligible or begins
  participating
• when a student exits a program
• when an eligible student transfers between LEAs
• when a participating student transfers between
  schools and between LEAs
• each academic year for
• Free and Reduced Price Meal Eligibility
• NCLB Title I Part A Basic Targeted programs to
  capture the services

42
Data Submission
RECORD LAYOUT STUDENT PROGRAM (cont)

• It should not be submitted .
• at the end of the academic year, unless the
  student is exiting the program
• when a student exits from the school

43
Data Submission
RECORD LAYOUT - STUDENT PROGRAM (cont)
The Student Program record may be submitted
independently of other record types however, the
SSID being reported must have an enrollment for
the Reporting LEA and School of Attendance. Also
if the Education Program Membership Code equals .

• Participating - the Membership Start Date and
  End Date must be equal to or within the range of
  the enrollment Start Date and End Date,
  respectively
• Eligible the Membership Start Date and End
  Date may be before and after the enrollment Start
  Date and End Date, respectively

Requested Membership Code is eligibility for
GATE, Free or Reduced Price Meal, Title I Part
C Migrant. If not available provide participation.
Following is a partial description of the each of
the data elements of the Student Program record
layout. Note the Operational Key and Updateable
Fields. Refer to the CALPADS File Specifications
and Code Sets for the most up-to-date and
complete description.
44
Data Submission
RECORD LAYOUT - STUDENT PROGRAM (cont)
45
Data Submission
RECORD LAYOUT - STUDENT PROGRAM (cont)
46
Data Submission
RECORD LAYOUT - STUDENT PROGRAM (cont.)
47

• DATA PROCESSING

48
Data Processing
DATA PROCESSING METHODS
CALPADS uses three main processing methods to
load the longitudinal data.

• Transaction Processing - Records are added,
  updated and deleted using transaction types of
  Add/Update (default), Delete, Replace.
• Effective Date Processing - Data are managed
  using Start and End effective dates. CALPADS
  stores the historical data for an LEA based on
  dates provided in the submission file.
• Replacement Processing - Data for a given school
  and time period are processed as a full
  replacement for the data in CALPADS. Existing
  data are deleted and replaced with the new data.

49
Data Processing
DATA PROCESSING METHODS (cont)
Each one of the CALPADS record types is
associated with one of the processing methods. In
addition to the processing method, an operational
key is identified for each record type. The
operational key for each record type is essential
for identifying a record in CALPADS. The table
below depicts the processing type and operational
key for the various CALPADS record types.
50
Data Processing
DATA PROCESSING METHODS (cont.)
These Record Types will be implemented later.
For the purposes of the CALPADS processing
methods, the current Academic Year ID or Academic
School Year is based on the processing date.
51
Data Processing
TRANSACTION PROCESSING
Transaction processing is used for submission of
the following file formats

• Student Enrollment
• Student Program

Records are added, updated and deleted using the
following three transaction types

• Add/Update
• Delete
• Replace

52
Data Processing
TRANSACTION PROCESSING (cont)
The rules for Transaction Processing are ..
53
Data Processing
TRANSACTION PROCESSING (cont)
Scenario 1a Add/Update Enroll Student
(Add) Submitted Record Record Type
Code SENR Transaction Type Code ltBlankgt
Reporting LEA 1111111 School of Attendance
2222222 Academic Year ID 2008-2009 SSID A
AAAAAAAAA Student School Start Date
7/1/08 Student School Exit Date
ltblankgt Enrollment History Results
No record was found in the ODS with the
Operational Key, so the record was added
54
Data Processing
TRANSACTION PROCESSING (cont)
Scenario 1b Add/Update (continued) Exit
Student (Update) Submitted Record Record Type
Code SENR Transaction Type Code ltBlankgt
Reporting LEA 1111111 School of Attendance
2222222 Academic Year ID 2008-2009 SSID
AAAAAAAAAA Student School Start Date
7/1/2008 Student School Exit Date 1/15/2009 Enr
ollment History Results
Key was found in ODS. Record was updated with the
exit date. Note that the Start Date of the
Enrollment must be known in order to update it
with the Exit.
55
Data Processing
TRANSACTION PROCESSING (cont)
Scenario 1c Add/Update (continued) Enroll
Student in New School (Add) Submitted
Record Record Type Code SENR Transaction Type
Code ltBlankgt Reporting LEA 1111111 School
of Attendance 3333333 Academic Year ID
2008-2009 SSID AAAAAAAAAA Student School
Start Date 1/16/2009 Student School Exit Date
ltBlankgt Enrollment History Results
Key was not found in ODS. Record was added. Note
that a concurrent enrollment is created if the
add for the new school is processed before the
exit from the first school
56
Data Processing
TRANSACTION PROCESSING (cont)
Scenario 2 Delete Delete Student
Enrollment Submitted Record Record Type
Code SENR Transaction Type Code D Reporting
LEA 1111111 School of Attendance 2222222 Acad
emic Year ID 2008-2009 SSID AAAAAAAAAA Stu
dent School Start Date 7/1/2008 Student School
Exit Date ltBlankgt or 1/15/2009 Enrollment
History Results
Note that the Operational Key in the D record
must match the record to be deleted in CALPADS.
If other non-key values are included in the D
record, they must also match the record to be
deleted in CALPADS.
57
Data Processing
TRANSACTION PROCESSING (cont.)
Scenario 3 Replace Replace Multiple Student
Enrollments Submitted Record Record Type
Code SENR Transaction Type Code R Reporting
LEA 1111111 School of Attendance
2222222 Academic Year ID 2008-2009 SSID
AAAAAAAAAA Student School Start Date
10/1/2008 Student School Exit Date
3/1/2009 Enrollment History Results
The data in the R record replaces all records
in CALPADS that have a key that matches the key
in the R record. Note that the key does not
contain the Start Date.
58
Data Processing
EFFECTIVE DATE PROCESSING

• Records are presented with an Effective Start
  Date and/or Effective End Date to indicate the
  first and last date CALPADS can consider the data
  in a record to be effective for processing and
  reporting. Data are managed using these dates,
  allowing CALPADS to create a longitudinal history
  of information.
• Effective Date processing is used to submit file
  formats for .
• Student Information
• Student Health
• Staff Demographics
• Records are added, updated and deleted using the
  following two transaction types
• Add/Update
• Delete

59
Data Processing
EFFECTIVE DATE PROCESSING (cont)
The rules for Effective Date Processing are ..
60
Data Processing
EFFECTIVE DATE PROCESSING (cont)
The rules for adjusting the dates ..
Adding/Updating Current Data (i.e. record
includes only Effective Start Date) During
normal processing, an LEA only needs to populate
the Effective Start Date. When the record is
submitted, CALPADS creates a record in the ODS
with the effective start date of the new record
and no end date, and adjusts existing data as
follows If the Effective Start Date of the new
record is..

• After the effective start date in the existing
  record, CALPADS establishes an end date for the
  existing Start Date that is one day prior to the
  new effective Start Date
• Before or equal to the effective start date in
  the existing record (or records), CALPADS removes
  the existing record(s). (This will delete all
  records from the effective start date through the
  end of the current Academic School Year.)

61
Data Processing
EFFECTIVE DATE PROCESSING (cont)
Scenario 1a Add/Update Current Data Start Date
in New Record After Start Date in Existing
Record CALPADS establishes an end date on the
existing record using the day before the new
effective start date and the new record is
inserted using the provided effective start date.
62
Data Processing
EFFECTIVE DATE PROCESSING (cont)
Scenario 1b Add/Update Current Data Start Date
in New Record Before Start Date in Existing
Record CALPADS replaces the existing record
with the new effective start date and no end date
63
Data Processing
EFFECTIVE DATE PROCESSING (cont)
The rules for adjusting the dates ..
Adding/Updating Historical Data (i.e. record
includes both Effective Start Date and Effective
End Date) LEAs may use effective date processing
to correct historical data in CALPADS. When the
record is submitted, CALPADS creates a record in
the ODS with the effective start and end date of
the new record and adjusts existing Start Date in
the same way as for Current Data. The existing
End Date is adjusted as follows If the
Effective End Date of the new record is..

• After the effective end date in the existing
  record, CALPADS removes the existing end date
• Before the effective end date in the existing
  record (or records), CALPADS establishes a Start
  Date for the existing End Date that is one day
  after the new End Date

64
Data Processing
EFFECTIVE DATE PROCESSING (cont)
Scenario 2 Add/Update Historical Data End Date
in New Record Before End Date in Existing Record
CALPADS establishes a Start Date for the
existing End Date that is one day after the new
End Date
65
Data Processing
REPLACEMENT PROCESSING

• These transactions delete all of the existing
  data in the ODS for a specific school and period
  (e.g., School Year or School Year and Term) and
  replace with new data provided.
• Replacement processing is used for submission of
  the following file formats
• Student Discipline
• Student Waivers
• Staff Assignment
• Course Section
• Student Course Section
• There are two valid transaction types
• Replace
• Delete

66
Data Processing
REPLACEMENT PROCESSING (cont)
The rules for Replacement Processing are ..
67
Data Processing
REPLACEMENT PROCESSING (cont)
Scenario 1 Replace Student Course Section
Records Existing Course Section Records
Submitted Record Submitted Set of Records Record
Type Code SCSE Transaction Type Code ltBlankgt
Reporting LEA 1111111 School of Delivery
2222222 Academic Year ID 2008-2009 Term FY
Course Section Record Results
The records in the input file replace all
existing records that match the key
68
Data Processing
REPLACEMENT PROCESSING (cont)
Scenario 2 Delete Student Course Section
Records Existing Course Section Records
Submitted Record Submitted Set of Records Record
Type Code SCSE Transaction Type Code D
Reporting LEA 1111111 School of Delivery
2222222 Academic Year ID 2008-2009 Term FY
Course Section Record Results
All records in ODS that match the key are deleted
69

• SUMMARY

70
Key Points
COMPARISON - SSID ENROLLMENT
71
Data Processing
COMPARISON - SSID ENROLLMENT (cont)
72
Data Processing
COMPARISON - SSID ENROLLMENT (cont.)
73
Key Points
COMPARISON STUDENT INFORMATION
74
Key Points
COMPARISON STUDENT PROGRAM
75
Questions
76
Updates
RECORD OF CHANGES