4 WorkFlow Manager

1
(No Transcript)
2
4 WorkFlow Manager

• Table of Contents
• 4.1 Introduction 2
• 4.2 Receipt Manager Overview 4
• 4.3 Workflow Manager Login 8
• 4.4 Reviewing File Transfer Results 12
• 4.5 Viewing Data Validation Errors 17
• 4.6 Correcting Data Validation Errors 23
• 4.7 Creating a Batch 28
• 4.8 Batch Data Validation 32
• 4.9 Reviewing Batch Status 35
• 4.10 Viewing Batch Data Errors 41

3
4.1 Introduction

• This portion of the training describes important
  functions for sending data receipt to the GSSIS
  and includes a process flow that provides an
  overview of some of this important functionality.
  
• The following page presents a high-level flow
  chart of the WorkFlow Manager process.

4
4.1 Introduction
Figure 4.1
5
4.2 Receipt Manager Overview

• Training Objectives
• At the completion of this unit, the user will
  understand the purpose of the Receipt Manager and
  the validation rules associated with it. The user
  will understand the role of the Receipt Manager
  in the context of the overall Extract, Transform,
  and Load process.
• Introduction
• Once files are extracted from the local SIS and
  transferred to GDOEs sFTP site, the Receipt
  Manager is launched. The Receipt Manager is a
  tool on the server that checks the file to see
  whether it is eligible for inclusion in a batch.
• Receipt Manager Validation Rules
• The objective of the Receipt Manager is to check
  for error conditions and to provide as much
  feedback, as early in the process as possible.
  Receipt Manager checks the following
• File Name
• File Format ID
• Authorized File
• Valid GTID
• Valid Course Number
• The extract routine produces a number of files.
  For some of these files GTIDs and course numbers
  are not required, so these values are only
  validated when applicable.

6
4.2 Receipt Manager Overview
Receipt Manager data validation rules for
Filename All files must conform to a unified
file naming convention to pass the initial
validation process. An example of the correct
format to be used for naming extracted files is
listed on the next page. Vendors have been
instructed to develop extract routines that
create files in conformance with file naming
convention. The following illustrates the naming
convention for extracted data files DistrictCode_
PrimaryTargetTable_YYYYMMDDHHMM.xxx where Dist
rictCode is the GDOE-assigned District
Code PrimaryTargetTable is the warehouse table
that is the target of the load plan that will
process the extracted file YYYYMMDDHHMM is the
portion of a timestamp value that enables more
than one file of the same template to be sent
daily xxx specifies the following file
formats csv Comma-delimited
tab Tab-delimited del Other delimiter
txt ASCII flat file fixed width fil EBCDIC
flat file fixed width
7
4.2 Receipt Manager Overview

• Receipt Manager data validation rules for File
  Format ID
• The data within the file is not for the District
  from which the file was uploaded.
• Receipt Manager data validation rules for
  Authorized File
• File must be sent by a user authorized to send
  it. District A may not submit District Bs file.
  Districts may not submit State level files.
• Receipt Manager data validation rules for
  matching GTIDs
• The student template files must be checked for
  matching GTIDs.
• Any unmatched GTID values will be logged to a
  reject file in the file staging area and are
  considered to have failed the validation step. If
  this occurs, the user will receive an email
  stating that the file failed validation with an
  explanation that an unmatched GTID exists.
• Receipt Manager data validation rules for
  matching course IDs.
• The course template files must be checked for
  matching Course IDs in a table provided by GDOE.
  
• Unmatched Course ID values are considered to have
  failed the validation step and will be logged to
  a reject file in the file staging area. If this
  occurs, the user will receive an email stating
  that the file failed validation with an
  explanation that an unmatched Course ID exists.

If an extract file submitted to Receipt manager
fails any one of the validation tests, an email
is sent to the user describing the failure.
8
4.2 Receipt Manager Overview

• Summary
• The user should understand
• The Receipt Manager errors and data validation
  errors
• Which items are reviewed in Receipt Manager
• What determines validation errors
• What constitutes a correct file name

9
4.3 WorkFlow Manager Login
Training Objectives At the completion of this
unit, the user will know how to login to the
WorkFlow Manager application. Introduction The
WorkFlow Manager application gives visibility
into the Extract, Transform, and Load process. It
allows users to view the status of files received
by Receipt Manager and to create batches.
Workflow Manager also allows users to view the
status of batches as they are transferred to the
verification database (where data is validated)
and GSSIS (where data is stored).
10
4.3 WorkFlow Manager Login

• Steps for accessing the DW/WFM.
• Use the URL https//portal.doe.k12.ga.us/
  to logon to the GDOE Portal

Figure 4.2
11
4.3 WorkFlow Manager Login

• Steps for downloading file of GTIDs from GDOE
  Website.
• Click on the WFM link.

WFM Link will be inserted here.
Figure 4.3
12
4.3 WorkFlow Manager Login
After the application has been launched, a page
similar to the one below will be displayed.
Figure 4.4
Summary In this section the user learned how to
log on to WorkFlow Manager.
13
4.4 Reviewing File Transfer Results

• Training Objectives
• At the completion of this unit, users will
  understand how to review results from the file
  transfer process.
• Introduction
• Once the extract files are transferred from the
  local SIS to GDOEs sFTP site, District users can
  review the result of each files transfer via
  Workflow Manager. The following results are
  possible
• File is not recognizable, email is sent, but
  status is not reported in WorkFlow Manager
• File has passed initial validation and has been
  transferred with status Validation OK.
• File has not passed initial validation and has
  not been transferred with status Failed
  Validation.

14
4.4 Reviewing File Transfer Results

• Steps for reviewing file transfer results
• Launch the DW/WFM application.
• Click the Review Details button in the File
  Status section of the home page, as shown below.

Figure 4.5
15
4.4 Reviewing File Transfer Results
Once the Review Details button has been selected,
the page shown below will be displayed. The user
can now view more detailed information about each
individual file.
Figure 4.6
16
4.4 Reviewing File Transfer Results
Each file that was submitted will be
automatically assigned a Receipt ID, as shown
below. The Receipt ID is located in the Receipt
ID section of the page and will allow the user to
track the status of an individual file that was
submitted.
Figure 4.7
17
4.4 Reviewing File Transfer Results
The other columns on the page provide the user
with more detailed information about a
corresponding Receipt ID.
Figure 4.8
Summary Users should now understand how to view
detailed information for each file that was
transferred from their local SIS to GDOEs sFTP
site.
18
4.5 Viewing Data Validation Errors

• Training Objectives
• At the completion of this unit, users will
  understand how to review data validation errors
  that result from the rejection of certain files
  by the Receipt Manager during the file transfer
  process.
• Introduction
• Snapshot files (those required on Count Days)
  show a Failed Validation message in Receipt
  manager.

• Steps for viewing data validation errors in
  DW/WFM
• Launch the DW/WFM application (from the GDOE
  Portal).
• Click the Review Details button in the File
  Status section
• Click the Review button of a file with a status
  of Failed Validation, as shown below.

Figure 4.9
19
4.5 Viewing Data Validation Errors
Users can view the transferred file contents and
the date that the file was last modified. The
file will not have a Batch ID, ETL Load Plan,
record insert, or record update information until
it has been added to a batch which you will learn
how to create later in this document.
Figure 4.10
20
4.5 Viewing Data Validation Errors
Steps for viewing data validation errors in
DW/WFM

• Click the SOURCEFILE hyperlink or View File
  Contents button to view more details about the
  file. In the example below, the View File
  Contents button was selected.

Figure 4.11
21
4.5 Viewing Data Validation Errors
The text box of the File Record Details
Workflow page allows the user to view a comma
delimited version of the data that was submitted.

• Steps for viewing data validation errors in
  DW/WFM
• Click the Download button to download the data to
  your local computer.

Figure 4.12
22
4.5 Viewing Data Validation Errors
Steps for viewing data validation errors in
DW/WFM

• Click the Save button to save the file to your
  local computer.

Figure 4.13

• Select the location on your local computer and
  click the Save button.

Figure 4.14
23
4.5 Viewing Data Validation Errors
Steps for viewing data validation errors in
DW/WFM

• Click the Home button to return to the DW/WFM
  home page.

Figure 4.15
Summary Users should now understand how to
review the status of a file that is submitted
through the Workflow Manager and download that
file to the local computer.
24
4.6 Correcting Data Validation Errors
Training Objectives At the completion of this
unit, users will understand how the following
steps are followed to resolve the data validation
errors that may occur. Upon completion of these
steps, files that previously failed validation
are eligible for inclusion in a batch.
Steps for resolving data validation errors

• Login to DW/WFM
• Download the file(s) with a status of Validation
  Failed
• Correct the data in the SIS
• Build a new extract
• Resubmit the file

Introduction If not all files transferred fail
validation, the user may want to prepare a batch
with those files that did pass validation. Files
with a Validation OK status should be marked as
File Ignore because they have already passed
the initial validation process. It is
recommended that all of the files with a status
of Validation OK that were sent at the same
time as the file that failed validation be marked
as File Ignore and not put through the Data
Load process again. The following slides
explain how to change the status of a file to
File Ignore.
25
4.6 Correcting Data Validation Errors
Steps for changing the status of a file to File
Ignore

• Launch the DW/WFM application.
• Click the Review Details button in the File
  Status section
• Click the Review button of a file with a status
  of Validation OK, as shown below.

Figure 4.16
26
4.6 Correcting Data Validation Errors
Steps for changing the status of a file to File
Ignore

• Click on the Change Status hyperlink.

Figure 4.17

• Click File Ignore in the New Status drop down
  menu.
• Click the Submit button, as shown below.

Figure 4.18
27
4.6 Correcting Data Validation Errors
Steps for changing the status of a file to File
Ignore

• Confirm that the status has been changed to File
  Ignore.
• Click the Home button.

Figure 4.19
28
4.6 Correcting Data Validation Errors
Steps for changing the status of a file to File
Ignore

• Re-confirm that the status has been changed to
  File Ignore to complete the process of changing
  the state of the file.
• Click the Logout button.

Figure 4.20
To complete the process of correcting data
validation errors follow the steps below. Once
the files have passed data validation they will
have a status of Validation OK.

• Correct the data in the SIS
• Build a new extract
• Re-Upload the file

Summary Users should now have a better
understanding of how to correct data validation
errors.
29
4.7 Creating a Batch
Training Objectives At the completion of this
unit, users will understand how to create a
batch. Introduction Once the file(s) that are
submitted have a status of Validation OK, they
are eligible to be included in a batch. The
creation of a batch triggers the process of data
loading into GSSIS. The following slides explain
this process. Please note that only one batch can
be processed at a time.
Steps for creating a batch

• Launch the DW/WFM application.
• Confirm that the file(s) have a status of
  Validation OK.
• Click the Review Details button in the File
  Status section

Figure 4.21
30
4.7 Creating a Batch
Steps for creating a batch

• Click the Assign Batch button at the bottom of
  the page as shown below.

Figure 4.22
31
4.7 Creating a Batch
Steps for creating a batch

• Check the Action check box for each file.
• Click the Create Batch button.

Figure 4.23
32
4.7 Creating a Batch
Steps for creating a batch

• Click the Home button to return to the home page.
• Verify that there is 1 batch in process.
• Click the Show Batch Details button.
• Select the batch that is in process.
• Verify that the Batch Summary and the file status
  sections state that the batch is Ready to be
  processed.
• Verify that the batch is assigned a Batch ID to
  complete the process.

Figure 4.24
Summary Users should now have a better
understanding of the requirements and processes
for creating a batch.
33
4.8 Batch Data Validation
Training Objectives At the completion of this
unit, users will understand the steps involved
once the file is included in a batch.
Introduction For a file to be included in a
batch it must first pass the error checks of the
Receipt Manager. Once a batch is created it is
passed through a second set of error checks. The
purpose of the ETL is to provide additional
validation checks and data transformations. Upon
completion of the process, data is eligible to be
sent to GSSIS.
Distinguishing the two types of uploads Regular
Uploads This form of upload occurs on a monthly
basis without any additional action required from
the user. ETL Complete data is copied to GSSIS
on a pre-defined schedule set by GDOE. Count Day
Uploads This form of upload includes FTE-1,
FTE-2, and FTE-3 data. Count Day Uploads are
different from a Regular upload because the
Superintendent must go online and indicate that
the uploaded data is OK to be copied to GSSIS.
This process is not covered in this training
document.
34
4.8 Batch Data Validation
The ETL step is initiated once the batch is
scheduled for processing.

• If the ETL Step fails due to data validation
  errors, the batch will have a status of ETL
  Failed and an email is sent to the Primary
  District user and to the system data
  administrator.
• If the ETL Step completes, then the file will
  have a status of ETL Complete and is eligible to
  be moved to GSSIS.

Fields Validated The ETL data validation rules
are similar to that of the Receipt Manager. The
following items must be validated for the file to
have a status of ETL Complete.

• Field Format
• Logical Key Validation
• Date Fields
• Numeric Ranges
• Duplicates
• Mandatory Fields

35
4.8 Batch Data Validation
ETL Validation Rules for logical keys

• All fields that are part of the logical key must
  contain a value.
• Examples of logical key validation
• District_Code value is checked against the
  District dimension table for a matching entry. If
  it is not found, then an error is triggered and
  the record rejected.
• Location_ID value is checked against the Location
  dimension table for a matching entry. If it is
  not found, then an error is triggered and the
  record rejected.
• Note that if the District_Code is not valid, the
  system is unable to verify location as the
  Location_ID depends upon a valid District_Code.

Example of a Cascading Error Dependency If
there is a format error, it would then be
impossible to identify follow on errors that
exist within the data because the data will be
unintelligible. Only after the first error is
fixed, can the succeeding data be validated. For
example an invalid District Code, which until
corrected, would cause district dependent data
elements to be in error.
Note If a District has a batch that has not
been completed because of an incomplete load and
is still in process, the batch process, once
complete, requires a state level user to initiate
the Move to Production (M2P) step. This ensures
that incomplete data loads (with possible data
integrity or quality issues) are not loaded
through to the warehouse. Summary Users should
now have a better understanding of the reasoning
behind the batch data validation rules.
36
4.9 Reviewing Batch Status
Training Objectives At the completion of this
unit, users will learn about additional
information that is available regarding the batch
status and how to obtain this information. Introdu
ction Once a batch is submitted, WorkFlow Manager
enables the user to monitor its status.
Steps for reviewing the status of a batch

• Launch the DW/WFM application.
• Click the Review Details button in the Batch
  Status section.

Figure 4.25
37
4.9 Reviewing Batch Status
Steps for reviewing the status of a batch

• Click the Review button.

Figure 4.26
38
4.9 Reviewing Batch Status
The date and time that the prior status was
issued can be viewed in the upper portion of the
Batch Status section. The lower portion of the
page will allow the user to view status of the
individual file.
Figure 4.27
39
4.9 Reviewing Batch Status

• Steps for reviewing the status of a batch
• Click the Receipt ID hyperlink.

Figure 4.28
40
4.9 Reviewing Batch Status
After clicking on the Receipt ID hyperlink, the
page below is displayed. The middle portion of
the page allows users to confirm the ETL Load
Plan start and end dates as well as the number of
records that were inserted or updated in the
batch.
Figure 4.29
41
4.9 Reviewing Batch Status
Users can access more information about the file
by clicking on the hyperlinks in the File column
or the View File Contents buttons in the Action
column.
Figure 4.30
Summary Users should now have a better
understanding of how to view the status of all
batches and the dates that they were last
modified.
42
4.10 Viewing Batch Data Errors
Training Objectives At the completion of this
unit, users will know how to view Log Files,
Source Files, Error Source Files, and Error KL
Periods. Introduction Once the batch is created
and submitted, ETL Load Plan errors are checked.
This section helps the user to diagnose and fix
errors identified in the files. Errors must be
fixed in the SIS.
43
4.10 Viewing Batch Data Errors

• Steps for viewing a Log File
• Login to DW/WorkFlow Manager
• Click the Show All Batches Button
• Click the Review button of the batch that you
  would like to review.
• Click on the Receipt ID hyperlink
• Click the LogFile hyperlink or the View File
  Contents button to view the Log File.

Figure 4.31
The Log File hyperlink in the File column allows
the user to view detailed information about the
overall status of the ETL Load Plan for a
particular batch. This link also provides the
user with details regarding why the ETL process
is unable to be completed.
44
4.10 Viewing Batch Data Errors
Once the LogFile has been selected, the screen
will display information similar to the picture
below. The text box of this page displays the
status of the file and the date that it was last
modified in the lower right hand corner.
Figure 4.32
45
4.10 Viewing Batch Data Errors
LogFile details of an ETL failed batch The
example below displays the LogFile information of
a batch that has a status of ETL Failed. Users
can view additional information by sliding the
scrollbar on the bottom of the page to the right.
Figure 4.33
46
4.10 Viewing Batch Data Errors
LogFile Details of an ETL Failed Batch Click the
Download button in the Action column to save or
print the information listed in the text box.
Figure 4.34
47
4.10 Viewing Batch Data Errors
Once the user has clicked the Download button the
following dialog box appears. Users should click
the Open button to open the file.
Figure 4.35
48
4.10 Viewing Batch Data Errors
The application opens the file as a text file
that can be printed and saved. An example of
this file is shown below.
Figure 4.36
49
4.10 Viewing Batch Data Errors
A Source File contains all of the information
that is submitted in a batch.

• Steps for viewing a source file
• Login to DW/WorkFlow Manager
• Click the Show All Batches Button
• Click the Review button of the batch that you
  would like to review.
• Click on the Receipt ID hyperlink
• Click the SOURCEFILE hyperlink or the View File
  Contents button.

Figure 4.37
50
4.10 Viewing Batch Data Errors

• Steps for viewing a Source File
• Click the Download button located in the Action
  column

Figure 4.38
51
4.10 Viewing Batch Data Errors

• Steps for viewing a Source File
• Click the Open button to view the contents of the
  file.

Figure 4.39
This action opens an Excel spreadsheet that
contains a listing of student IDs and
corresponding information regarding the students
infractions, courses, and teachers.
52
4.10 Viewing Batch Data Errors
The errors in a Source File can be viewed by
selecting Error Source File of a particular batch.

• Steps for viewing an Error Source File
• Login to DW/WorkFlow Manager
• Click the Show All Batches Button
• Click the Review button of the batch that you
  would like to review.
• Click on the Receipt ID hyperlink
• Click the Error SOURCEFILE MSG hyperlink to view
  the exact name of the field that caused the
  Source File error.

Figure 4.40
53
4.10 Viewing Batch Data Errors
The File Record Details WorkFlow page now
displays the file details in the text box. In the
example below, an error occurred in the
Stud_Infraction table.
Figure 4.41
54
4.10 Viewing Batch Data Errors

• Required steps for viewing Error Source Files
• Click the Download button.

Figure 4.42
55
4.10 Viewing Batch Data Errors

• The following dialog box appears when user clicks
  on the Download button. Users must Open or Save
  the document in order to download the file
  contents.
• Click the Open button.

Figure 4.43
56
4.10 Viewing Batch Data Errors
The application will open the file as a text file
that can be printed and saved. An example of this
file is shown below.
Figure 4.44
57
4.10 Viewing Batch Data Errors
Users can also view the fields with an Error KL
Period. This function will displays all the
information that was uploaded for a record that
contains Error KL Period.

• Steps for Viewing an Error KL Period
• Login to DW/WorkFlow Manager
• Click the Show All Batches Button
• Click the Review button of the batch that you
  would like to review.
• Click on the Receipt ID hyperlink
• Click the ERROR_KL_PERIOD hyperlink

Figure 4.45
58
4.10 Viewing Batch Data Errors

• The following dialog box will appear when user
  clicks on the Download button. Users must Open or
  Save the document in order to download the file
  contents.
• Required Steps for Viewing an Error KL Period
• Click Open.

Figure 4.46
59
4.10 Viewing Batch Data Errors
The application will open the error file as a
text file that can be printed and saved. An
example of this file is shown below.
Figure 4.47
Summary Users should now have an understanding
of how to view four files that provide
information on the status of batch files. With
this information, users can diagnose and fix
errors.