Practice and Theory in Digital Libraries: The Case of Open Video Libraries in the Digital Age LIDA05

1
Practice and Theory in Digital Libraries The
Case of Open VideoLibraries in the Digital Age
(LIDA05)Dubrovnik, Croatia

• Gary Marchionini, PhD
• University of North Carolina at Chapel Hill
• www.ils.unc.edu/march
• march_at_ils.unc.edu
• May 30, 2005

2
Outline

• Digital Libraries as phenomena
• Multimedia and video challenge our text biases
• Open Video concepts and system Moebius
• User studies
• Conclusion

3
Pragmatics

• Useful theory and practice are a Moebius strip
• DL practice in informed by multiple theories
  related to
• Information structure
• Human behavior
• System design
• Social-political-economic constraints and
  organizational behavior
• History and epistemology
• We want principles, not only developedthe work
  of the closetbut applied, which is the work of
  life. Horace Mann, Thoughts, 1867

4
Theories of What and Why

• Digital extensions of physical libraries
• Augmentations of intellect
• Collaborative spaces sharium
• Cultural institutions
• World Brain
• Economic models
• Complex information systems

5
Theories of How

• Reuse and open source information
• Levels of abstraction
• Information retrieval
• Information interaction
• Iterative design and evaluation
• Resource management

6
Digital Library Design Space1999 What Has
Changed in 2005?
Adapted from Marchionini Fox, IPM, 1999
7
Provocation Text no longer rules

• The Net generation depends much less of reading
  (they are entering universities as students and
  soon, as professors Oblinger Oblinger, 2005
  Educause book). In the US
• Children age 6 or younger average of 2 hrs/day
  using screen media, 1.6 hrs/day playing outside,
  39 min. reading
• 13-17 yr olds average 3.1 hrs/day watching TV
  and 3.5 hrs/day with digital media. They
  multitask
• gt2M million US children (ages 617) have their
  own Web site. Girls are more likely to have a Web
  site than boys (12.2 percent versus 8.6 percent).
• Ability to use nontext expressionaudio, video,
  graphicsappears stronger in each successive
  cohort.
• Multimedia and Multitasking the trend of 21st
  century
• Information specialists MUST get over our text
  bias

8
Open Video DL Case

• Open
• Public good
• Reusable
• Files not streams
• Chunking
• Agile views user interface
• Alternative representations (views)
• Agile control mechanisms

9
Open Video Vision/Contributions

• An open repository of video files that can be
  re-used in a variety of ways by the education and
  research communities
• Encourages contributions
• A testbed for interactive interfaces
• An easy to use DL based upon the agile views
  interface design framework
• Multiple, cascading, easy to control views (pre,
  over, re, shared, peripheral)
• Views based upon empirically validated surrogates
• An environment for building theory of human
  information interaction
• A set of methods and metrics that reveal how
  people understand digital video through
  surrogates

10
Background Status

• Begun 1995 with colleagues at UMD BCPS
• Funding NSF, NASA, NSF/LoC
• Collaborators/Contributors I2-DSI, ibiblio, CMU,
  UMD, NIST, Prelinger and Internet Archives, NASA,
  ACM
• 2600 video segments
• 2000 different titles
• 15000 unique visitors per month
• MPEG-1, MPEG-2, MPEG-4, QT
• OAI provider
• Ongoing user studies
• New Preservation initiative

11
Agile Views Interface Research

• Provide a variety of access representations
  (e.g., indexes) and control mechanisms
• Usual search and browse capabilities
• Leverage both visual and linguistic cues
• Create and test surrogates for overview preview,
  shared and history views

12
User Study Framework
13
The Surrogates

• Storyboard with text keywords (20-36 per board_at_
  500 ms)
• Storyboard with audio keywords
• Slide show with text keywords (250ms repeated
  once)
• Slide show with audio keywords
• Fast forward ( 4X)
• Fast forwards 32X, 64X, 128X, 256X
• Poster frames
• Real time clips
• Text titles

14
Surrogate Examples
15
Metrics
16
User Studies

• Study 1 Qualitative Comparison of Surrogates
  (ECDL 02)
• Study 2 Fast Forwards (JCDL 03)
• Study 3 Narrativity (CHI 02 ASIST 03 paper)
• Study 4 Shared views and History Views (Geisler
  dissertation)
• Study 4 Poster frames and text (eye tracking,
  CIVR 03)
• Study 5 TREC evaluations (03 and 04)
• Study 6 cognitive load and ISEE (Mu diss.)
• Study 7 relevance judgments for video (Yang
  diss.)
• Study 8 Surrogate integration study (in
  analysis)
• Others several specific masters papers (Hughes,
  Gruss

17
Study 1 Compare Surrogates

• What are the strengths and weaknesses of
  different surrogates from the users perspective?
• Are any of the surrogates better than the others
  in supporting user performance?

18
The Surrogates

• Storyboard with text keywords (20-36 per board_at_
  500 ms)
• Storyboard with audio keywords
• Slide show with text keywords (250ms repeated
  once)
• Slide show with audio keywords
• Fast forward ( 4X)

19
Method

• 7 video segments (2-10 min), 5 surrogates created
  for each
• 10 subjects with high video and computer
  experience
• Three phases (all multi-camera videotaped)
• View full video then use 3 surrogates, repeat
• Participant observation and debriefing
• Do NOT view full video, use 3 surrogates, repeat
• Participant observation and debriefing
• Complete 3 assigned tasks with surrogates of
  choice
• Think aloud and debriefing
• http//www.open-video.org/experiments/chi-2002/met
  hods/study1.mov

20
Tasks

• Gist determinationfree text
• Gist determinationmultiple choice
• Object recognitiontextual
• Object recognitiongraphical
• Action recognition (2-3 second clips)
• Visual gist (predict which frames belong)
• http//www.open-video.org/experiments/chi-2002/sur
  rogates/index.html

21
Preferences

• In debriefing after each phase, subjects asked
  about preferences.
• Some preferences changed over the phases
• 2 subjects preferred ff
• 4 subjects said ff if audio keywords added
• 1 storyboard with audio keywords
• 2 slide show with audio keywords
• ? drop ss with text keywords, develop ff

22
Performance

• No SRD on gist (both free text and multiple
  choice)
• SRD on action recognition favoring ff
• Near SRD on text object recognition favoring
  SB/w audio keywords
• 81 to 291 compaction rates suitable for tasks
• Psychometric and face validity support for the
  tasks (means and variances relevant to real
  tasks)
• SRD in gist and visual gist for one video
• ?Homogeneity of frames diminishes surrogate value
• ?Keywords help when visual variability decreases

23
Qualitative Results

• Subjects suggested different surrogates for
  different tasks (e.g., ff for judging kid safe,
  sb for identifying images, ff for video styles)
• Three senses of gist
• Topic (T)
• Narrativity (N)
• TNvisual style
• Individual preferences and experiences influence
  surrogate effectiveness

24
Study 2 Fast Forward

• How fast can we make fast forwards?
• 4 ff conditions (32X, 64X, 128X, 256X)
• Four video segments for each condition
• 45 subjects (1/2 UG, 1/2 grad, 2/3 female)
• 6 tasks (full text gist, multiple choice gist,
  word object recognition, graphical object
  recognition, action recognition, visual gist)
• Counterbalance speed and videos
• Web-driven experimental condition, 3-camera video
  tapes, single subject at a time in usability
  laboratory

25
Example Image Recognition Stimulus
26
Results

• SRD on 4 of 6 tasks as speed increases, however,
  reasonable performance at even the highest rate
• Video content/genre interacts with performance
• Preference does not parallel performance (people
  can perform well under extreme conditions but do
  not like/enjoy)
• No user characteristic differences (age, sex)
• ?Give users control but select appropriate
  defaults
• Caveat controlled, independent focus on FF,
  likely a lower bound on performance

27
Speed Effects on Performance
28
Narrativity Study

• CHI walk up kiosk, 20 people used
• 20 one-minute clips ( half bw, no audio)
  selected on 2 criteria contain characters, have
  cause/effect relations between scenes (5 in each
  category)
• SRD on chars, cause, and interaction

29
Shared Views and History Views Studies

• Evaluate AV Design Framework by instantiating and
  evaluating a design
• Shared (based on recommendations) and History
  Views (based on logs)
• Phase 1 compare OV to Views interface (28
  participants). OVgtaccuracy NSRD on time, but
  learning effect AVgtnavigation/efficiency
  AVgtsatisfaction
• Phase 2 qualitative analysis of shared and
  history views

30
Poster Frame Study

• Research Questions
• Given both textual and visual metadata which
  surrogate will be utilized, which surrogate will
  be preferred?
• Does the placement of the surrogates affect how
  they are used?
• Does the assigned task affect how surrogates are
  used?
• Does personal preference play a role in how
  surrogates are used?

31
Study Methods / Procedures

• 12 undergraduate students (paid volunteers)
• Pre-Study questionnaire
• Demographics
• Visual vs. Verbal learning style (VVQ)
• 10 search problems
• Counter-balanced
• Design 1 and 2
• 1 text on left / visuals on right
• 2 visuals on left / text on right
• Eyetracking
• Post-study questionnaire
• Follow up questions

32
Results

• All participants over all tasks
• Mean time looking at text 29.7 sec.
• Mean time looking at pics 6.8 sec.
• 75 of fixations over text
• 18 of fixations over pics
• First fixations over text 65
• First fixations over pics 54
• Text requires and gets more user attention

33
Results contd

• Design 1 vs. Design 2
• When text was placed on the left, mean time per
  fixation was slightly higher
• VVQ
• Balanced group spent more time looking at text
• Tasks
• Varied by task
• Time spent looking at text
• Time spent per fixation over text
• Frequency of fixations over text

34
Screen Shots
35
Screen Shots
36
Screen Shots
37
Tasks

• Please find a video that discusses the
  destruction earthquakes can do to buildings.
  These search results are from a search on the
  word Earthquake.
• Please find a video that discusses nurses and
  their contributions to the United States Army.
  These search results are from a search on the
  word Work.
• Please choose a video from the following list
  that you think would be entertaining for you and
  your friends to watch.

38
Discussion

• In this restricted situation (i.e. pre-formulated
  results page) participants used text as the main
  anchor point
• ? Because text is a better surrogate?
• ? Because text contains more information?
• ? Because text is more familiar to people
• ? Because tasks directed users to text?

39
Discussion contd

• Layout seemed to have little effect on how
  surrogates were used.
• Difference of .03 of a second
• Participants didnt report a significant
  preference for layout
• Some liked design 1 and some liked design 2
• VVQ
• Hypothesis that visual learners would use visual
  surrogates and verbal learners would use verbal
  surrogates was not supported

40
Discussion contd

• Tasks
• Some tasks took more time to complete
• Regardless of
• Counterbalancing order
• Participant
• Layout design

41
Text or Pictures?

• Text was reported as
• Being the search anchor
• Containing significant topical information
• Taking longer to read than pictures
• Visuals were reported as
• Being globally liked
• Being used to quickly narrow down choices
• Taking less time to decode than text
• All participants said the results page would be
  weaker without them
• Often lacking in reference points

42
Conclusion

• Visual metadata was used to make (confirm???)
  relevance judgments
• Combination of visual verbal stronger than one
  or the other
• Generalize with caution
• Small number of study participants
• Specific set of search results pages
• Ten specific search tasks.

43
The Integration Study

• Compare old OV to redesign? Compare to Internet
  archive?
• How do multiple surrogates and agile control
  mechanisms affect understanding of video?
• Accuracy? Time? Satisfaction? Cognitive load?
  Navigational overhead?
• Data analysis underway

44
Relevance Study (Yang)

• 3 task groups (illustration 10 profs,
  collection building 8 video librarians, video
  production 8 producers/editors)
• In-depth interviews
• Text, audiovisual, implicit categories of 39
  different criteria
• Topicality most often mentioned, but far less
  than text studies
• Production groups less varied, more audiovisual
  criteria

45
Theory-Practice Lessons from OV

• User-centered design and user testing pays off,
  i.e. research informs practice
• Production system operation raises new kinds of
  research questions
• Sustainability models
• Curatorial models
• Preservation challenges
• Upgrade paths for universal access

46
DL Research Directions

• Incorporating people into DLs (patrons,
  librarians)
• Leveraging contributions and implications for
  curatorship
• Preservation strategies how much context?
• Hybrid physical-digital library operations

47
Observations

• A moebius strip is infinite the interplay
  between theory and practice goes on
• Need for collaboration between working libraries
  and researchers

48
Selected Open Video Readings

• Yang, M. Marchionini, G. (2005). Deciphering
  visual gist and its implications for video
  retrieval and interface design. Conference on
  Human Factors in Computing Systems (CHI).
  Portland, OR. Apr. 2-7, 2005.
• Yang, M. Marchionini, G. (2004). Exploring
  Users' Video Relevance Criteria -- A Pilot
  Study. Proceedings of the Annual Meeting of the
  American Society of Information Science and
  Technology, pp. 229-238. Nov. 12-17, 2004.
  Providence, RI.
• Yang, M., Wildemuth, B., Marchionini, G.
  (2004). The relative effectiveness of
  concept-based versus content-based video
  retrieval. Proceedings of the ACM Multimedia
  conference, pp. 368-371.
• Mu, X., Marchionini, G. (2003). Enriched
  video semantic metadata authorization,
  integration, and presentation. Proceedings of
  the Annual Meeting of the American Society for
  Information Science and Technology, 40, 316-322.
• Wilkens, T., Hughes, A., Wildemuth, B. M.,
  Marchionini, G. (2003). The role of narrative
  in understanding digital video an exploratory
  analysis. Proceedings of the Annual Meeting of
  the American Society for Information Science, 40,
  323-329.
• Hughes, A., Wilkens, T., Wildemuth, B.,
  Marchionini, G. (2003). Text or Pictures? An
  Eyetracking Study of How People View Digital
  Video Surrogates. Proceedings of CIVR 2003, pp.
  271-280.
• Wildemuth, B. M., Marchionini, G., Yang, M.,
  Geisler, G., Wilkens, T., Hughes, A., and Gruss,
  R. (2003). How Fast Is Too Fast? Evaluating Fast
  Forward Surrogates for Digital Video.
  Proceedings of the 3rd ACM/IEEE-CS Joint
  Conference on Digital Libraries (JCDL 2003), pp.
  221-230. (Vannevar Bush Award Winner for Best
  Paper at JCDL 2003)
• Mu, X., Marchionini, G., Pattee, A. (2003).
  The Interactive Shared Educational Environment
  User interface, system architecture and field
  study. Proceedings of the Annual Meeting of the
  American Society for Information Science and
  Technology, 40, 291-300.
• Mu, X., Marchionini, G. (2003) Statistical
  Visual Features Indexes in Video Retrieval.
  Proceedings of SIGIR 2003, pp. 395-396.
• Marchionini, Gary (2003). Video and Learning
  Redux New Capabilities for Practical Use.
  Educational Technology.
• Marchionini, Gary and Geisler, Gary. (2002). The
  Open Video Digital Library. D-Lib Magazine, Vol.
  8, Number 12, December.
• Barbara M. Wildemuth, Gary Marchionini, Todd
  Wilkens, Meng Yang, Gary Geisler, Beth Fowler,
  Anthony Hughes, and Xiangming Mu (2002).
  Alternative Surrogates for Video Objects in a
  Digital Library Users? Perspectives on Their
  Relative Usability. Proceedings of the 6th
  European Conference on Digital Libraries,
  September 16 - 18, 2002, Rome, Italy.
• Geisler, G., Marchionini, G., Wildemuth, B. M.,
  Hughes, A., Yang, M., Wilkens, T., and Spinks, R.
  (2002). Video Browsing Interfaces for the Open
  Video Project. Proceedings of CHI 2002, Extended
  Abstracts.
• Nelson, Michael L., Marchionini, Gary, Geisler,
  Gary, and Yang, Meng (2001). "A Bucket
  Architecture for the Open Video Project short
  paper." JCDL 01, ACM - IEEE Joint Conference on
  Digital Libraries (June 24-28, 2001, Roanoke,
  Virginia).
• Geisler, Gary, and Gary Marchionini (2000). "The
  Open Video Project A Research-Oriented Digital
  Video Repository short paper." In Digital
  Libraries '00 The Fifth ACM Conference on
  Digital Libraries (June 2-7 2000, San Antonio,
  TX). New York Association for Computing
  Machinery, 258-259.
• Slaughter, L., Marchionini, G. and Geisler, G.
  (2000). "Open Video A Framework for a Test
  Collection." Journal of Network and Computer
  Applications, Vol. 23(3). San Diego Academic
  Press.