Retrospective and Challenges for ModelBased Interface Development

1
Retrospective and Challengesfor Model-Based
Interface Development

• Pedro Szekely
• University of Southern California,Information
  Sciences Institute, USA
• http//www.isi.edu/isd/Mastermind

2
Acknowledgments
Mastermind Project USC/ISI Pedro Szekely,
Project Leader Martin Frank, Computer
Scientist Pablo Castells, Visiting Scientist (U
Madrid) Ewald Salcher, Graduate Student (U Graz)
Mastermind Project Georgia Tech Jim Foley,
Project Leader Spencer Rugaber, Project
Leader Kurt Stirewalt, Programmer Noi Sukaviriya
(now at IBM)
Funding provided by DARPA USA Defense Advanced
Research Projects Agency Allen Sears, Program
Manager
6/5/96
2
DSV-IS'96
3
Outline

• High quality interfaces
• Model-Based User Interface Development
• Analysis of model based tools
• New directions
• Mastermind

6/5/96
3
DSV-IS'96
4
User Interface Tools Goal
Development Effort
Quality
6/5/96
4
DSV-IS'96
5
High Quality Interfaces
6
Interface Quality

• Task support
• Actively supports the tasks users need to perform
• Presentation
• Show information needed for tasks in an easy to
  assimilate format
• Dialog
• Efficient, easy to learn, natural for task
• Context sensitivity
• Sensitive to platform characteristics, user
  profile, usage history
• Multi-modal interaction
• Speech, language, pen in addition to mouse and
  keyboard

6/5/96
6
DSV-IS'96
7
Task Support

• Systems with the following capabilities
• Understand the tasks the user is performing
• Understand its own capabilities
• Actively assist users in performing their tasks
• Show information needed to perform the tasks
• Anticipates subsequent tasks
• Prevents errors, provides error recovery
• MS wizards a step in the right direction

6/5/96
7
DSV-IS'96
8
Task Support Example MS Word
MS Word Document
6/5/96
8
DSV-IS'96
9
Task Support Example MS Word
1) Invoke answer wizard
2) Fill in question
3) Select topic
MS Word Document
4) Ask for advice
6/5/96
9
DSV-IS'96
10
Task Support (contd.)
5) System shows instructions
6/5/96
10
DSV-IS'96
11
Task Support (contd.)
6) System show which menu to use
5) System shows instructions
6/5/96
11
DSV-IS'96
12
Task Support (contd.)
6) System show which menu to use
5) System shows instructions
7) System shows dialogue box
8) System explains what to do
6/5/96
12
DSV-IS'96
13
Task Support The Correct Answer
Wizard is brittle
The answer I was looking for
6/5/96
13
DSV-IS'96
14
High Quality Presentations

• Show the information that users need to perform
  their tasks
• in a format that makes it easy to assimilate
  information
• sensitive to the amount and kind of data
• using appropriate graphic design principles
• taking into account cultural conventions
• and capabilities and handicaps of user

6/5/96
14
DSV-IS'96
15
Easy To Assimilate Presentations

• Presentations generated with SAGE (Steve Roth,
  CMU)
• Input database tables
• User select columns to view
• System designs presentation

6/5/96
15
DSV-IS'96
16
Context Sensitivity

• Platform sensitivity
• Displays that adapt to screen size, color depth,
  resolution
• Interaction that adapts to network bandwidth and
  latency
• User sensitivity
• Interaction and displays that adapt to user
  experience and skill
• Interaction sensitivity
• Interfaces that remember previous interaction
• Remember window placement, sizes and options
• Remember user choices

6/5/96
16
DSV-IS'96
17
Multi-Modal Interaction
Phil Cohen, OGI

• Pen
• Gesture, handwriting
• Speech input
• Limited natural language
• Deictic input
• E.g. put that there

6/5/96
17
DSV-IS'96
18
Development Tools
19
HCI Development Process
User Requirements
Guidelines
Task Analysis
Past Experience
Developers
Design Specifications
User
6/5/96
19
DSV-IS'96
20
HCI Development Process
User Requirements
Design Knowledge
Guidelines
Task Analysis
Past Experience
Libraries
Context
Developers
Executable Specification
Design Specifications
User
6/5/96
20
DSV-IS'96
21
HCI Development Process
User Requirements
Design Knowledge
Guidelines
Task Analysis
Past Experience
Libraries
Context
Developers
Executable Specification
Design Specifications
User
Analytical Evaluation
User Testing Data
Advice
6/5/96
21
DSV-IS'96
22
Domains Of Expertise

• Developers
• Application domain
• Users
• Tasks
• Work context
• Graphic design
• HCI
• High level decisions

• Tools
• Book keeping
• Search
• Repetitive tasks
• Application independent domains
• Chart generation
• Style rule application and checking

6/5/96
22
DSV-IS'96
23
Tools

• Specification tools
• Word processors, drawing programs
• Layout tools, visual programming
• Transformation tools
• E.g., task model to Petri nets
• Advice/design exploration tools
• E.g., TRIDENT interaction object selection
• Generation tools
• E.g., Adept, Humanoid, Trident, ...
• Evaluation tools
• NGOMSEL, model checkers

6/5/96
23
DSV-IS'96
24
Informal Definition of Formal Specifications

• A formal specification is used by a program to
• derive useful information for developers
• generate an executable specification
• generate another formal specification

6/5/96
24
DSV-IS'96
25
Informal Definition of Formal Specifications

• A formal specification is used by a program to
• derive useful information for developers
• generate an executable specification
• generate another formal specification

• Informal specifications
• Text
• Drawings
• Formal specifications
• Logic
• Semantic net
• Rules
• Programs

6/5/96
25
DSV-IS'96
26
Informal Definition of Formal Specifications

• A formal specification is used by a program to
• derive useful information for developers
• generate an executable specification
• generate another formal specification

• Informal specifications
• Text
• Drawings
• Formal specifications
• Logic
• Semantic net
• Rules
• Programs

Model-Based Tools use formal specifications as
input
6/5/96
26
DSV-IS'96
27
Knowledge Sources of Tools
6/5/96
27
DSV-IS'96
28
Ease Of Use Issues In Model-Based Tools

• Modeling problem
• Tools need detailed application knowledge
• Difficult for application developers to formalize
• Requires expressive notations
• Reuse problem
• Developers need to find and understand the
  relevant portions of tool knowledge bases
• Control problem
• Developers need to understand how their inputs
  affect the observable behavior of tools
• Need to understand what to say to obtain
  desired behavior

6/5/96
28
DSV-IS'96
29
Expected Benefits of Model-Based Tools
6/5/96
29
DSV-IS'96
30
Classification Of Model-Based Tools

• Purpose
• Specification
• Transformation
• Interface generation
• Advice
• Evaluation

• Features
• Task support
• Presentation
• Dialog
• Context sensitivity
• Multi-modal

6/5/96
30
DSV-IS'96
31
Classification Of Model-Based Tools

• Purpose
• Specification
• Transformation
• Interface generation
• Advice
• Evaluation

• Application domain
• Narrow
• General

• Extensibility
• Open
• Closed

• Features
• Task support
• Presentation
• Dialog
• Context sensitivity
• Multi-modal

• Language semantics
• Mathematical
• Defined by tools

6/5/96
31
DSV-IS'96
32
Classification Of Model-Based Tools

• Purpose
• Specification
• Transformation
• Interface generation
• Advice
• Evaluation

• Application domain
• Narrow
• General

• Model contents
• Task
• Application
• Presentation
• Dialog
• User
• Platform

• Extensibility
• Open
• Closed

• Features
• Task support
• Presentation
• Dialog
• Context sensitivity
• Multi-modal

• Ease of use
• Modeling
• Reuse
• Design control

• Language semantics
• Mathematical
• Defined by tools

6/5/96
32
DSV-IS'96
33
Analysis
34
Successful Model-Based Tools(High Quality, Lower
Cost)

• ITS (Charlie Wiecha, IBM)
• Style rules to define presentation
• Small set of style rules for large number of
  windows
• Easy to adapt to different devices
• mouse, touch screen
• small, large displays
• Easy to apply policy changes to many windows
• Less work than defining each window individually
• Industrial strength tool
• Seville World Fair Kiosks
• many others ...

6/5/96
34
DSV-IS'96
35
Successful Model-Based Tools(High Quality, Lower
Cost)

• ITS (Charlie Wiecha, IBM)
• Style rules to define presentation
• Small set of style rules for large number of
  windows
• Easy to adapt to different devices
• mouse, touch screen
• small, large displays
• Easy to apply policy changes to many windows
• Less work than defining each window individually
• Industrial strength tool
• Seville World Fair Kiosks
• many others ...

• Sage (Steve Roth, CMU)
• Automatic presentation generation
• Presentations are of high quality
• Deep knowledge of chart design
• Narrow domain
• Model
• Developer models data
• Simple way to control design

6/5/96
35
DSV-IS'96
36
Why Are ITS and SAGE Successful?

• Application domain
• Narrow
• General

• Model contents
• Task
• Application
• Presentation
• Dialog
• User
• Platform

• Do one thing well
• Extensible
• Cant model it, program it
• Specification language
• Accessible to developers,
• even though not easy (ITS)
• Model covers surface aspects
• ITS presentation dialogue
• SAGE presentation
• Designer control
• Automation,
• but developer has last word

• Extensibility
• Open
• Closed

• Ease of use
• Modeling
• Reuse
• Design control

• Language semantics
• Mathematical
• Defined by tools

6/5/96
36
DSV-IS'96
37
Best Automatic Generation Tool TRIDENT

• Application domain
• Narrow
• General

• Model contents
• Task
• Application
• Presentation
• Dialog
• User
• Platform

• Basic approach
• High level specification
• Application Task models
• Generate
• Presentation Dialogue
• Use rich knowledge bases
• Style guides for choosing interaction objects
• Use sophisticated algorithms
• Decision tree for IO selection
• Metric based layout
• Task-based window allocation

• Extensibility
• Open
• Closed ???

• Ease of use
• Modeling
• Reuse
• Design control

• Language semantics
• Mathematical
• Defined by tools

6/5/96
37
DSV-IS'96
38
Lessons For Automatic Generation

• TRIDENT lessons
• Narrow domain enables progress
• Automatic interface generation is a VERY hard
  problem
• Results are good
• lots remain to be done
• How to make it better?

• Window allocation, IO selection layout cannot
  be done independently
• Space influences everything
• Choice of IO influences space, gt window
  allocation layout
• Layout influences IO selection, window allocation
• Modeling burden?
• Modeling?
• Developers specify what they want?

6/5/96
38
DSV-IS'96
39
Evaluation Tools

• Logic-based
• Still in infancy
• Useful in special situations
• Medicine, factory control
• Modeling burden is too large
• Asking the right questions is difficult
• Too hard!

• Task based (e.g., GOMS)
• Less powerful than logic based,
• Predictions execution time and learnability
• Much easier to do
• Similar to established evaluation techniques
• cognitive walkthrough
• GOMS models automatically derived from task
  models used for generation

6/5/96
39
DSV-IS'96
40
Advisor/Critiquing Tools

• Analogy to word-processing
• Spelling checkers
• Detect errors
• Suggest corrections
• Custom dictionaries
• Grammar checkers
• Marginally useful
• No correction
• Content checkers
• Non-existent

6/5/96
40
DSV-IS'96
41
Advisor/Critiquing Tools

• Analogy to word-processing
• Spelling checkers
• Detect errors
• Suggest corrections
• Custom dictionaries
• Grammar checkers
• Marginally useful
• No correction
• Content checkers
• Non-existent

• Interface design
• Style guide checkers(e.g., Gerhard Fischer)
• Like grammar checkers
• Detect lots of violations
• Little on correction
• Layout metrics
• Low semantics advisors?
• Like spelling checkers

6/5/96
41
DSV-IS'96
42
Summary of Current Model-Based Work

• Generation tools
• Generation from task and application/data models
• Automatically design presentation and dialog
• Progress being made (TRIDENT)
• Successes
• Generation very narrow domain, sophisticated
  algorithms (SAGE)
• Not so narrow, but developer designs presentation
  and dialog (ITS)
• Payoff is in the features it enables platform
  retargetting, reuse

6/5/96
42
DSV-IS'96
43
New Directions
44
Shift Focus

• Interface Quality
• Task awareness/support
• Intelligent presentation
• Context-sensitivity
• Multi modal interfaces

Automatic Generation Automatic Evaluation
6/5/96
44
DSV-IS'96
45
Interface Trends
1984
1995
2000
6/5/96
45
DSV-IS'96
46
Avoid Being One Generation Behind

• Interface Quality
• Task awareness/support
• Intelligent presentation
• Context-sensitivity
• Multi modal interfaces

• WIMP era is ending
• Model-based tools are just maturing for the WIMP
  era
• Generation of WIMP
• Evaluation of WIMP

1984
1995
2000
Tools to construct multi-modal interfaces
6/5/96
46
DSV-IS'96
47
New ChallengePlatform Scalability

• Interface Quality
• Task awareness/support
• Intelligent presentation
• Context-sensitivity
• Multi modal interfaces

• WIMP era single platform
• New era many platforms
• Large monitors
• Laptops
• PDAs
• Powerful pagers
• Networks

Tools to construct interfaces that work across
platforms
6/5/96
47
DSV-IS'96
48
Task SupportIntelligent Presentation
Direct manipulation multi-modal interfaces
are not enough

• Task support
• Embed more semantics in the interface
• Understand tasks
• Cooperate with user to map goals to system
  capabilities
• Prevent recover from errors
• Streamline task performance
• Task models
• Use them for support,
• not only for generation

• Intelligent presentation
• Help users absorb the information they need
• Task models
• The context needed to design presentations at
  run-time

6/5/96
48
DSV-IS'96
49
Summary

• Interface Quality
• Task awareness/support
• Intelligent presentation
• Context-sensitivity
• Multi modal interfaces

• Interfaces that reason
• about tasks
• about information
• about platform capabilities
• about user behavior

6/5/96
49
DSV-IS'96
50
Summary

• Interface Quality
• Task awareness/support
• Intelligent presentation
• Context-sensitivity
• Multi modal interfaces

• Interfaces that reason
• about tasks
• about information
• about platform capabilities
• about user behavior

Model-Based Interfaces
6/5/96
50
DSV-IS'96
51
MASTERMIND

• Showcase of capabilities in the context of
  querying logistics databases

52
Query Interfaces Users Problem
User questions How do I find radio/TV/com
equipment? How do I specify condition codes? How
do I specify that the item must be in AR4? What
information can I get back? How do I know that my
query will work?
Example query Which radio/TV/com equipment is
in less than condition code A or B in AR4
05/28/96
52
MASTERMIND
53
Query Interfaces Users Problem
User questions How do I find radio/TV/com
equipment? How do I specify condition codes? How
do I specify that the item must be in AR4? What
information can I get back? How do I know that my
query will work?
Example query Which radio/TV/com equipment is
in less than condition code A or B in AR4
Solution Displays that show the relevant
terminology Dialogue that matches the user
tasks Find main concepts (radio/TV/com
equip) Specify restrictions (condition
code) Specify info requested (nomenclature,
price)
05/28/96
53
MASTERMIND
54
Presentation Model
Developer View
Derived Implementation
05/28/96
54
MASTERMIND
55
Presentation Model

• Layout scheme based on graphic design theory
• Produce visually pleasing displays
• Pink elements contain LAD-specific behavior
• Reused in alternate designs
• Recursively defined using same technique
• Novel replication mechanism
• Supports dynamic behavior without programming

05/28/96
55
MASTERMIND
56
Task Awareness
Example query Which radio/TV/com equipment is
in less than condition code A or B in AR4
Specify restriction task
Choose attribute to restrict
Specify restriction information
Choose operator
Specify restriction value
05/28/96
56
MASTERMIND
57
Task Awareness
Example query Which radio/TV/com equipment is
in less than condition code A or B in AR4
Questions that task centered interfaces must
answer
Specify restriction task
1. What are the restrictable attributes of my
object? 2. How do I specify the one I
want? 3. Does the attribute match any
keywords? 4. Is it restricted already? To
what? 5. Can I disable the restriction without
deleting it?
Choose attribute to restrict
6. Which widget invokes the restriction
display? 7. What happens if the restriction
display is already active? 8 What else can be
done while specifying a restriction? e.g.,
specify an output?
Specify restriction information
Choose operator
9. Which operators are valid for this attribute?
10. What values are legal for the selected
attribute given a choice of operator ? 11. What
happens when an invalid combination of operator
value is entered?
Specify restriction value
Mastermind provides facilities to enable
interface to address these questions
05/28/96
57
MASTERMIND
58
Facilities For Task Awareness

• Modeling language provides terminology to express
  the answers
• Raises the issues
• Forces developers to think solutions
• Reusable components contain answers

Specify restriction (7) reentrant False
Choose attribute is_a Primitive Choice Task (1)
choices ltltlink to interface state modelgtgt
(2) interactor is_a Choice Inter (6) where
ltltlink to presentation modelgtgt action activate
Specify restr info is_a Dialogue Task
... choose operator, specify restriction value
Done is_a Done Dialogue Task (10)
preconditions (11) recovery
Cancel is_a Cancel Dialogue Task
05/28/96
58
MASTERMIND
59
Summary of Leverage

• Task model makes interface task aware
• Computes and maintains the info that users need
  to perform tasks
• Defines the sequencing of steps to complete the
  tasks
• Presentation model manages information
  presentation
• Easy for developers to define change
  presentations appropriate to task
• Modeling is more productive than programming
• Models are much smaller than equivalent code
• Model checkers ensure generation of correct
  source code
• Model enables adaptation of interface
• Platform scaleable adaptation of LAD interface
  done in 1 day
• 2nd version of LAD interface done in 3 days

05/28/96
59
MASTERMIND
60
Analysis of Interface

• Presentation model success
• Presentation model is small
• Model checker prevents common Amulet errors
• Generated Amulet is large
• Interface state is complex
• Coded by hand LAD
• Largest piece of code
• Currently being modeled
• Update code
• Coded by hand LAD
• Small, but longest debugging
• Change manager will automate it

Application Interface
Generated code
Presentation model
05/28/96
60
MASTERMIND
61
EvolutionVersion 1 to Version 2

• New layout defined with a new grid
• Change of color scheme
• Interface state of outputs and restrictions merged

One new component (based on old one)
05/28/96
61
MASTERMIND
62
Platform Adaptation
Workstation Interface
05/28/96
62
MASTERMIND
63
Platform Adaptation
Workstation Interface
Laptop Interface

• Refinements become pop-up
• Components scale appropriately

Preserve structure of interface to facilitate
user transfer
05/28/96
63
MASTERMIND
64
Platform Adaptation
Workstation Interface
Laptop Interface

• Refinements become pop-up
• Components scale appropriately

PDA Interface

• Title disappears
• Keywords move
• Constraints collapsed

Preserve structure of interface to facilitate
user transfer
05/28/96
64
MASTERMIND
65
AdaptationWorkstation, Laptop PDA

• Workstation to PDA
• Defined thresholds on grid
• When space smaller than threshold, make it zero
• When component disappears, add button to pop it
  up
• Defined conditional settings for
• Position of keyword type-in
• Font-sizes
• Interface state modifications to control contents
  of pop-up

Pop Up
05/28/96
65
MASTERMIND