Further Development of a Digital Library Curriculum: Evaluation Approaches and New Tools

1
Further Development of a Digital Library
Curriculum Evaluation Approaches and New Tools

• Seungwon Yang, Uma Murthy, Seonho Kim and Edward
  A. Fox,
• Department of Computer Science, Virginia Tech,
• Barbara M. Wildemuth, Jeffrey Pomerantz and
  Sanghee OhSchool of Information and Library
  Science,
• University of North Carolina Chapel Hill
• ICADL 2007, Hanoi, Vietnam, Dec. 10 Dec. 13,
  2007

2
Acknowledgements (Selected)

• Faculty / Staff / Others
• Module evaluators, Advisory Board members,
  Digital Library Research Lab members, Susie
  Marion, Sidewalk team (David Maier, Lois
  Delcambre, Sudarshan Murthy)
• Sponsor
• US National Science Foundation Grants
• IIS-0535057 to VT
• IIS-0535060 to UNC-CH

3
Overview

• Introduction
• DL curriculum framework
• Development lifecycle
• Future work
• Invitation

4
Introduction

• Interdisciplinary project
• VT CS UNC-CH SILS
• Goal
• DL lesson modules ? courses
• Community

5
Introduction

• Team
• VT Dr. Edward A. Fox (PI), Seungwon Yang
• UNC-CH Dr. Barbara M. Wildemuth (PI),
• Dr. Jeffrey P. Pomerantz, Sanghee Oh
• Advisory Board
• VT10, UNC-CH14, other12
• Term Jan. 2006 - Dec. 2008

6
Foundations - 5S Framework -

• Theoretical foundation
• Necessary condition to be a Minimal DL

Ss Role in DL Examples
Streams Various types of content text, video, audio
Structures Organize information catalog, metadata, hypertext
Spaces Store and present information interfaces, storage, vector space
Scenarios Provide services searching, browsing, recommending
Societies Form a DL community of users and service managers service managers, teachers, learners
7
Foundations - Computing Curriculum 2001(ACM
IEEE-CS)-

• CS Body of Knowledge
• ? Information Management (IM.14)
• ? Digital Libraries
• DL topics and learning objectives
• Suggestions for CC2001 update

8
Overview

• Introduction
• DL curriculum framework
• Development lifecycle
• Future work
• Invitation

9
DL Curriculum Framework

• Current DL Module Framework
• 10 core modules
• 41 sub modules

10
DL Module Framework ( 1-5 of 10)
11
DL Module Framework ( 6-10 of 10)
12
Draft Module Example

• 5-b Application software

13
Overview

• Introduction
• DL curriculum framework
• Development lifecycle
• Future work
• Invitation

14
Development Lifecycle

• Diagram
• (Selected) six draft modules
• Pilot test for formative evaluation

15
Development Lifecycle Diagram
16
(Selected) Six Draft Modules

• 1st yellow box
• High-priority areas first
• 1-b History of digital libraries and library
  automation
• 5-b Application software
• 6-a Information needs, relevance
• 6-b Search strategy, information seeking
  behavior, user modeling
• 7-b Reference services
• 9-c DL evaluation, user studies

17
Pilot Test for Module Evaluation- Feedback
Meeting during JCDL07

• Prioritize them
• Define the scope
• More consistent relationships
• International inputs in development
• From Asia, Europe, Africa, Oceania,

18
Overview

• Introduction
• DL curriculum framework
• Development lifecycle
• Future work
• Invitation

19
Future work Resource presentation in a module

• Superimposed Information (SI) Technology
• Sub-document level granularity presentation
• Marks references to selected regions within
  base (original) information

20
SI Technology
21
Future work Understanding the communitys use
of module

• Visual User model Data Mining (VUDM) Tool

22
VUDM

• Visualization of DL module usage trends for three
  weeks

23
Future work Plans for module distribution

• Using wikis
• Wikiversity distribute modules, set up a DL
  community
• Wikibooks DL booklet
• Using Second Life (secondlife.com)
• 3D virtual environment
• Supports collaborative, distance learning
• Collaborator in SJSU Dr. Lili Luo

24
New Media Consortium Campus
From http//b2e.nitle.org/media/Howard_beforethemo
b.jpg
25
Overview

• Introduction
• DL curriculum framework
• Development lifecycle
• Future work
• Invitation

26
Would you like to join us?

• Your contribution two ways
• Module development
• (e.g., based on your teaching experiences/classes)
  
• Module evaluation
• Contacts for PIs
• VT Edward A. Fox (fox_at_vt.edu)
• UNC-CH Barbara M. Wildemuth (wildem_at_ils.unc.edu)

27
More Information

• Project homepage http//curric.dlib.vt.edu
• Module evaluation http//curric.dlib.vt.edu/wiki
• Project diary http//tuppence.dlib.vt.edu/blogs/i
  ndex.php?blog2

28

• Thank you!
• Questions and comments?

29
(No Transcript)
30
Foundations

• 5S framework
• Advisory board
• Computing Curriculum (CC) 2001

31
Wikiversity
32
Module Template (1/2)

• 1. Module name
• 2. Scope
• 3. Learning objectives
• 4. 5S characteristics of the module
• 5. Level of effort required
• 6. Relationships with other modules
• 7. Prerequisite knowledge required
• 8. Introductory remedial instruction

33
Module Template (2/2)

• 9. Body of knowledge
• Topic 1
• Lesson
• Topic 2 ...
• Topic 3 ...
• 10. Resources
• 11. Concept map
• 12. Exercises / learning activities
• 13. Evaluation of learning outcomes
• 14. Glossary
• 15. Additional useful links
• 16. Contributors

34
Foundations - Advisory Board
From VT From UNC Other Institutions
Steven Edwards, CS Roger Ehrich, CS Weiguo Fan, ACIS Steve Harrison, CS Gail McMillan, Library Chris North, CS Manuel Pérez-Quiñónez, CS Naren Ramakrishnan, CS Deborah Tatar, CS Layne Watson, CS Catherine Blake, SILS Laura Gasaway, Law School Jane Greenberg, SILS Stephanie Haas, SILS Brad Hemminger, SILS Thomas James, Dean, School of Education Paul Jones, Director, ibiblio SILS School of Journalism Mass Communication Diane Kelly, SILS Gary Marchionini, SILS Montek Singh, CS Natasha Smith, Library Helen Tibbo, SILS Steve Weiss, CS Javed Mostafa, SILS Dan Atkins, U. Michigan Christine Borgman, UCLA Lillian Cassel, Villanova Michael Christel, CMU Raya Fidel, U. Washington Richard Furuta, Texas AM University Elizabeth Liddy, Syracuse University Clifford Lynch, CNI Kurt Maly, ODU Tefko Saracevic, Rutgers Linda Smith, UIUC Ingeborg T. Sølvberg, NTNU (Norway)
35
Pilot Test for Module Evaluation - Procedures -

• Project team
• Provided an overview of our progress in
  developing modules
• Presented the curriculum development lifecycle
• The participants
• Worked in pairs to evaluate a module
• Guided by an evaluation form
• All
• discussed the evaluation process and how it
  should be implemented more widely

36
Formative Evaluation

• Evaluate 5 sections 2
• Objectives
• Body of knowledge
• Readings
• Learning activities
• Level of effort and prerequisites
• Overall structure
• Additional comments

37
Example Guiding Specific Questions

• Body of knowledge
• Guiding question Does the module address all
  areas of the topic that need to be addressed?
• Specific questions
• Will the body of knowledge enable students to
  achieve the objectives?
• Are there any topics that you think are critical
  to add to the body of knowledge?
• Are there any topics that you would remove from
  the body of knowledge?

38
Formative Evaluation Form (1 of 3)

• Objectives
• Guiding question Are the objectives appropriate
  for the topic?
• Specific questions
• Are the objectives observable?
• Will students be able to achieve the objectives,
  given the content in the body of knowledge?
• Body of knowledge
• Guiding question Does the module address all
  areas of the topic that need to be addressed?
• Specific questions
• Will the body of knowledge enable students to
  achieve the objectives?
• Are there any topics that you think are critical
  to add to the body of knowledge?
• Are there any topics that you would remove from
  the body of knowledge?

39
Formative Evaluation Form (2 of 3)

• Readings
• Guiding question Are the readings the best and
  most appropriate for the topic?
• Specific questions
• Are there any readings that you think are
  critical to add to the list?
• Are there any readings on the list that you would
  remove?
• Learning Activities
• Guiding question Are the activities appropriate
  for the topic?
• Specific questions
• Will students be able to accomplish the
  activities, given the content in the body of
  knowledge?
• Will the activities enable students to achieve
  the objectives?
• Can you suggest any other learning activities
  that may be appropriate for this module?

40
Formative Evaluation Form (3 of 3)

• Level of Effort and Prerequisites
• Guiding question Is it feasible to teach the
  module as it is currently constructed?
• Specific questions
• Is the level of effort required in class
  appropriate to the scope of the body of
  knowledge? Prior to class?
• Is the prerequisite knowledge required sufficient
  for students to comprehend the body of knowledge?

41
Current Status of Formative Evaluation

• Sending out emails
• Provided p.w. to evaluators who agreed to
  participate
• Expect to be done in Dec., 2007

42
Future work Resource presentation in a module

• Superimposed Information (SI) Technology
• Sub-document level granularity presentation
• Marks references to selected regions within
  base (original) information