Systems Integration and Data Management

1
Systems Integration and Data Management

• Presented By Greg Devine
• Director, Higher Education Blackboard Consulting

April 14, 2005
2
Welcome

• Session Objectives
• Introduce 2-3 key points
• Innovation
• Insert
• Results/Outcomes
• Insert

3
Blackboard Snapshot Integration
An overview of the process, tools and
operationsFor Blackboard Learning System
(Release 6.2)

• What Data Which Files
• Snapshot Components
• Snapshot Tool
• Modes, Operators, Syntax
• Data Sources and Data Source Keys
• Field Ownership
• Remote Snapshots
• Snapshot Controller

4
IMS Entities

• Integration Entities Snapshot Files
• Enterprise Data

5
Other Data Types

• Additional Enterprise Data

6
File Formats
ltPERSONgt ltSOURCEDIDgt ltSOURCEgtBlackboard
Universitylt/SOURCEgt ltIDgt39450210223lt/IDgt
lt/SOURCEDIDgt ltUSERIDgtswanglt/USERIDgt ltNAMEgt
ltFNgtMr. Stanley Wang Jr.lt/FNgt
ltSORTgtWang,Stanleylt/SORTgt
ltNICKNAMEgtWang,Stanleylt/NICKNAMEgt ltNgt
ltFAMILYgtWanglt/FAMILYgt
ltGIVENgtStanleylt/GIVENgt
ltOTHERgtFranklinlt/OTHERgt ltPREFIXgtMr.
lt/PREFIXgt ltSUFFIXgtJr.lt/SUFFIXgt
lt/Ngt lt/NAMEgt ltDEMOGRAPHICSgt
ltGENDERgt2lt/GENDERgt ltBDAYgt1959-01-01lt/BDAYgt
lt/DEMOGRAPHICSgt ltEMAILgtSwang5_at_Blackboard_univ
ersity.comlt/EMAILgt ltTEL teltype"1"gt3104591276lt
/TELgt ltTEL teltype"2"gt3104591200lt/TELgt
ltADRgt ltSTREETgtTwin Oaks Valley Rdlt/STREETgt
ltSTREETgtattn S. Wanglt/STREETgt
ltLOCALITYgtSan Marcoslt/LOCALITYgt
ltREGIONgtCAlt/REGIONgt ltPCODEgt92096-0001lt/PCODE
gt ltCOUNTRYgtUSlt/COUNTRYgt lt/ADRgt
ltDATASOURCEgtBlackboard Universitylt/DATASOURCEgt
ltEXTENSIONgt ltX_BB_SYSTEMROLEgt0lt/X_BB_SYSTEMR
OLEgt ltX_BB_INSTITUTION_ROLEgt0lt/X_BB_INSTITUT
ION_ROLEgt ltX_BB_STUDENTIDgt144532lt/X_BB_STUDE
NTIDgt ltX_BB_PASSWORDgtrpetersonlt/X_BB_PASSWOR
Dgt lt/EXTENSIONgt lt/PERSONgt

• XML
• All data in one file
• Reusable
• Standardized (IMS)
• Delimited (preferred)
• Easy to Manipulate
• Multiple Files
• Smaller

EXTERNAL_PERSON_KEYUSER_IDSYSTEM_ROLEINSTITUTIO
N_ROLEFIRSTNAMELASTNAMEEMAIL 0142270sait.tarha
nnonerole_11SaitTarhansait.tarhan_at_college.edu
0001129marcha.hunleynonerole_11MarchaHunley
marcha.hunley_at_college.edu 0001613steven.yeltonno
nerole_11StevenYeltonsteven.yelton_at_college.edu
0003970dorothy.mannnonerole_11DorothyManndo
rothy.mann_at_college.edu 0099821sherry.huttonnone
role_11SherryHuttonsherry.hutton_at_college.edu 00
00311gary.websternonerole_11GaryWebstergary.
webster_at_college.edu 0173103carla.gesell-streeter
nonerole_11CarlaGesellcarla_at_college.edu 000061
2olivia.wattsnonerole_11OliviaWattsolivia.wa
tts_at_college.edu 0000381robert.eveslagenonerole_
11RobertEveslagerobert.eveslage_at_college.edu 000
0987debra.lierlnonerole_11DebraLierldebra.li
erl_at_college.edu 0225494robert.nieldsnonerole_11
RobertNieldsrobert.nields_at_college.edu 0000982c
haralee.allennonerole_11CharaleeAllencharalee
.allen_at_college.edu 0295293david.hoctornonerole_
11DavidHoctordavid.hoctor_at_college.edu 0001393c
olleen.meyernonerole_11ColleenMeyercolleen.me
yer_at_college.edu 00294456clark.stullnonerole_11
ClarkStullclark.stull_at_college.edu 0000577julie.
websternonerole_11JulieWebsterjulie.webster_at_c
ollege.edu 0074937george.armstrongnonerole_11G
eorgeArmstronggeorge.armstrong_at_college.edu
7
Snapshot Integration Components

• 3 Primary Software Components
• Generator (client developed)
• Tool (Blackboard Learning System core)
• Controller (Blackboard Consulting deliverable)

Snapshot Controller
SIS
Blackboard
SnapshotGenerator
SnapshotTool
SIS
Snapshot Files
SIS
8
Default Snapshot Configuration
Administrative Systems
Blackboard Academic Suite
Snapshot Controller
SnapshotGenerator
SnapshotTool
Database
SIS
SIS
Snapshot Files
SIS
JDBC
FTP
9
Remote Snapshot Operations
Campus System(s)
Blackboard Client
Blackboard Learning System
Snapshot Controller
SnapshotGenerator
Application
SIS
SnapshotTool
SIS
Snapshot Data Files
SIS
Database
FTP
SOAP
JDBC
On Campus
Remote (ASP)
10
Snapshot Security
Campus System(s)
Blackboard Client
Blackboard Learning System
Snapshot Controller
SnapshotGenerator
Application
SnapshotTool
SIS
SIS
Snapshot Data Files
SIS
Database
HTTPS
SFTP, SCP
VPN
On Campus
Remote (ASP)
11
Data States

• Available/Unavailable vs. Enabled/Disabled
• Availability (Y/N) is set via the GUI, Snapshot
  or Event
• Row Status (enabled/disabled) is set via snapshot
  only.

Available_Ind Row_Status
User 1 Y Enabled
User 2 N Enabled
User 3 Y Disabled
User 4 N Disabled
See Integration Manual or Data Planning Workbook
for detailed behavior of users, enrollments and
courses.
12
Snapshot Tool - Modes

• Manual (MANUAL)
• Simplest mode, adds and updates records
• Used to manage partial data sets.
• Snapshot (SNPSHT)
• Adds, updates and disables records
• Used to manage full data sets.
• Remove (REMOVE)
• Removes records
• Copy Into (COPYINTO)
• Copies content between courses or organizations

13
Snapshot Tool Modes
Adding records is the same in both Snapshot and
Manual modes

• Adding Records

Bb
ADD
Blackboard Database
Snapshot File
14
Snapshot Tool Modes
Updating records is the same in both Snapshot
and Manual modes

• Updating Records

Bb
No Change
update
disabled
update
User3
Y
enabled
N
No Change
Blackboard Database
Snapshot File
15
Snapshot Tool Modes
The difference between Manual and Snapshot is
seen when records are no longer found in the file.

• Missing Records

Bb
ignore
disable
User3
Y
enabled
disabled
Blackboard Database
Snapshot File
Manual mode just adds and updates.
Snapshot mode disables missing records.
16
Snapshot Tools

• ..\blackboard\apps\snapshot
• \bin
• DSM data source manager
• SNAPSHOT snapshot tool
• SNAPSHOT_OVERRIDE snapshot tool
• \config
• ENV - connection info, class path and connection
  type
• \data
• SNAPSHOT.PROPERTIES primary snapshot
  configuration
• IMS-EP01.dtd DTD for XML snapshots

17
Snapshot Tools Command Lines
snapshot -V ltserver namegt -f ltoperationgt -t ltdata
filegt -C ltproperties filegt

• snapshot V bbls.abc.edu f CRS_SNAPSHOT t
  ..\data\SIS_CRS_SP2003 C ..\data\snapshot.propert
  ies
• snapshot V bbls.abc.com f USR_MANUAL t
  ..\data\SIS_USR
• -C ..\data\snapshot.properties
• snapshot V bbls.abc.org f STAFF_REMOVE t
  ..\data\SIS_MEM_FA2003 -C..\data\snapshot.properti
  es
• snapshot V bbls.abc.gov f CRS_COPYINTO t
  ..\data\SIS_CRS_SU2002 C ..\data\snapshot.propert
  ies

18
Snapshot Tools Snapshot Override

• Snapshot override allows you to override settings
  from the properties file.

• snapshot_override -Ddata.source.keyDL_CRS V
  bbls.abc.edu f CRS_SNAPSHOT t
  ..\data\SIS_CRS_SP2003 C ..\data\snapshot.propert
  ies
• Snapshot_override -Dmax.error.count0 V
  bbls.abc.com f USR_MANUAL t ..\data\SIS_USR
• -C ..\data\snapshot.properties

19
Snapshot Properties
Maximum errors to process. If you want all
valid records to be processed, use -1 which
indicates unlimited If you want an all or none
transaction, use 0 Otherwise, pick some
reasonable number of errors max.error.count-1
Snapshot process settings delimteter used for
parsing snapshot files data.delimeter Used
to denote datasource to use for
operation. data.source.keySIS Determines if
password field should be encrypted encrypt.passwor
dY Denotes string used to mark the beginning
of errors error.delimeter(!) Determines
character used to escape delimeter escape.characte
r/ Designates flag to handle header
validation for configuration property
file header.validationY Determines if logged
information should be sent to console. log.stdout
N Designates flag to establish handling of
enumerated values. parse.allow.defaultN
Designates maximum size of database
transaction snapshot.batch.size300 Used to
configure pause between persistence
actions wait.length-1 Snapshot controlled
field settings for Blackboard override of data
fields for category entities category.bb.controlle
d.fields

• Operations Properties
• delimiter, batch size, error count
• data source key
• Ownership
• bb.controlled fields
• Course Copy Operations
• content areas, settings, memberships

20
Snapshot Properties - Ownership

• Ownership Control
• Fields owned by Blackboard are not updated by
  the snapshot process.
• Fields not owned by Blackboard may be overwritten
  by the snapshot.
• Ownership properties are set and stored in
  snapshot.properties
• Fields owned by Blackboard
  listed under bb.controlled
• Fields are Specific for Each Entity

Snapshot controlled field settings for
Blackboard override of data fields for
category entities category.bb.controlled.fieldsAV
AILABLE_IND fields for link entities categoryme
mbership.bb.controlled.fields fields for
group entities group.bb.controlled.fieldsAVAILABL
E_IND,COURSE_NAME fields for user
entities person.bb.controlled.fieldsINSTITUTION_R
OLE fields for membership entities membership.b
b.controlled.fieldsROLE
21
Ownership - Example

• FirstName is owned by Blackboard
• LastName is owned by SIS

Bb
Smithe
Joe
Freedman
Angie
Jen
Jones-Smith
Francis
Frank
Blackboard Database
Snapshot File
1. Snapshot creates user records.
2. Data is updated via the UI.
3. Data is updated in SIS.
4. Snapshot runs again and last names are
updated, first names are left alone.
22
Data Sources and Data Source Keys

• Data Source Keys can be used to separate and
  identify data from different sources.

SIS
HR
Alumni
23
Data Sources and Data Source Keys

• Data Source Keys can also be used to separate and
  identify data from different logical groups.

Spring
SIS
Summer
Fall
24
Data Sources and Data Source Keys

• Blackboard database records are mapped to a
  particular data source key.
• Anything entered via the GUI is mapped to the
  SYSTEM DSK.

course_ID courseName ext_crs_key dataSourceKey
Course1 Biology Bio01_02F2005 fa_crs_2005
Course2 History Hist03_06F2005 fa_crs_2005
Course3 French Fre01_05P2005 sp_crs_2005
Course4 French2 Fre02_01P2005 sp_crs_2005
Course5 Spanish Course5 SYSTEM
Note Course_ID Ext_Crs_Key will be
identical for GUI entered records.
25
Using Data Source Keys
Bb
111-11-1111
111-11-1111
222-22-2222
222-22-2222
external_person_key
DSK
888-88-8888
888-88-8888
111-11-1111
Medical
666-66-6666
666-66-6666
222-22-2222
Medical
DSK Medical
333-33-3333
Law
333-33-3333
Law
444-44-4444
Law
444-44-4444
Law
555-55-5555
Law
555-55-5555
Law
666-66-6666
Medical
555-55-5555
777-77-7777
Law
777-77-7777
Law
444-44-4444
888-88-8888
Medical
777-77-7777
333-33-3333
The DSK ensures that only the right records are
updated. Without DSK the Law records would be
disabled when the Medical snapshot is processed.
DSK Law
26
Snapshot Tools Data Source Manager
DSM -V ltserver namegt -f ltoperation -b ltdata
source keygt

• dsm V bbls.abc.edu f CREATE b BBCRS_FA2005 d
  Fa05_Courses
• dsm V bbls.abc.com f DISABLE_PERSON b BBUSERS
  t 20050305
• dsm V bbls.abc.org f PURGE_MEMBERSHIP b
  BBSTAFF_FA2005
• dsm V bbls.abc.gov f MODIFY b BBCAT r
  BBCAT_FA05

Only disabled records will be removed using the
PURGE operation.
27
Closing Slide

• Innovating Together in 05
• Recap top 2-3 points
• Resources Available
• Any links, resources available for attendees to
  learn more
• Follow up Contact(s)
• Insert
• IF YOU ONLY REMEMBER 1 THING
• Insert