InkSeine: in situ search for active note taking

1
InkSeine in situ search for active note taking

• Ken Hinckley, Shengdong Zhao, Raman Sarin,
  Patrick Baudisch, Edward Cutrell, Michael
  Shilman Desney Tan

r
oronto
2
In situ search for digital ink

• InkSeine
• Ink
• Seine
• -n a fishing net
• -v to fish with a seine
• insanely cool ink search!
• InkSeine lets the user fish for useful
  information directly from ink notes.

Fishermen catching salmon on the Columbia River
using a seine.(courtesy Wikipedia)
3
Active note taking

• XLibrispen-based system for active
  reading(reading critical thinking learning)
• InkSeine combines pen-based note taking with
  searching, linking, collecting sensemaking

ActiveNote Taking

• Sketch designs
• Reflect on a topic
• Capture extend creative ideas

4
Core interaction pattern Ink-Search-Gather

• Ink excels at capturing users thoughts ideas
  with minimum distraction from formatting issues

5
Ink-Search-Gather interaction pattern

• Search digging through the file system on a
  tablet is really a pain
• Active note taking benefits from low overhead
  access to supporting materials related docs
• Focus on personal search, but support Web as well

6
Ink-Search-Gather interaction pattern

• Gather Make it easy to use what you find by
  pulling content directly into unstructured notes
• Thumbnail hyperlinks to documents
• Clippings from documents
• Queries that can be revisited later

7
Ink-Search-Gather interaction pattern

• InkSeine unifies these three elements in a fluid
  user experience that is integrated directly with
  inking

8
In situ search

• Leverage preexisting ink to initiate search
• No context switch to search app stay in the
  flow
• No tedious transcription of text to a search
  box
• Queries as first class objects, commingled with
  ink notes
• Flag some ink for search for later
• Visible and salient search history as part of
  notes
• Queries can be copied, pasted, moved, etc.
• Interleave inking, searching gathering
• No barrier between inking / searching note
  ideas while searching, side-by-side searching,
  span app boundaries for content gathering
• Tightly couple queries with application content
• Queries naturally persist in originating context
• Easy re-visitation of prior queries
• Results of queries become new content

9
Rich tradeoffs in cost structure of sensemaking

• Continuum of time cost can defer at any point
• 1 second Capture thought I should search on
  this
• 5 seconds Trigger search see initial results
  list
• 10-20 s Open a doc from result list or drag
  result into notes
• 20 s-2 minutes
• Scroll through list and inspect details of
  results
• Apply filters, revise query
• Grab a snapshot from a short document
• Correct occasional ink reco errors Tablets
  reco is amazingly good!
• Longer sufficient depth for common info needs
  not mired by last 10 of complex queries that
  require full featured search tools
• Inspect result documents to see if they meet
  information needs
• Sideways searches based on Title, Date, or Author
  of a result

10
Video
11
Paper Prototype
12
Paper Prototype
13
Paper Prototype
14
Findings

• Existing desktop search tool difficult to use
• Difficult text entry
• Enter and correct text

15
Findings

• Existing desktop search tool difficult to use
• Difficult text entry
• Enter and correct text
• Annoying context switch and flow interruption
• Switch among apps and context

16
Findings

• Existing desktop search tool difficult to use
• Difficult text entry
• Enter and correct text
• Annoying context switch and flow interruption
• Switch among apps and context
• Tedious information gathering
• Difficult to link to documents and gather pieces
  of information

17
Notebook Control
18
In Situ Search
19
Breadcrumb
Breadcrumb
20
Breadcrumb
21
Trigger Personal Search
22
Search Panel
23
Query Area
Query Area
(CueTIP)
24
Result List
Result List
25
Result List
Result List
Focus Result
26
File and Timeline Filters
Filters
27
Gestures for Filters
Cross Multi-select
Tap - Select
Cross Turn Right Exclude
Disjunctive cross Multi-select
28
Directed Work Flow
Selection
Correction
Search
29
Occlusion
30
Occlusion
31
Crossing Widgets
32
Resize Search Panel
33
Examine Document
34
Document Tracking Menu
35
Multiple Search Panels
36
Design Issue 1 Discovery

• Gesture discovery problem
• Highlighter hints

37
Design Issue 2 Occlusion

• Context preservation occlusion problem
• Smart positioning of search panel

38
Design Issue 3 Poor Reco

• Recognition problem for out-of-vocabulary words
• Personalized vocabulary from desktop search
  index

UIST
?
List?, Dist?, Gist?, Mist?, Hist?
39
Summary

• In situ search experience
• Tailored to pen input
• Optimum workflow / maximum flexibility
• Multiple ways to gather content
• Minimize search screen real estate
• Span application boundaries

40
Future Directions

• Pull in materials for searches from notes

41
Future Directions

• Pull in materials for searches from notes
• Support vector-based searches

42
Questions?