Recent Work in ModelBased User Interfaces

1
Recent Work in Model-Based User Interfaces

• Jeffrey Nichols
• Lecture 13
• 05-830 Advanced User Interface Software

2
Last time

• Model-based User Interfaces
• Automatic generation of the user interface so the
  programmer wont do a bad job.
• Dialog boxes are relatively easy to generate
• The full application interface is hard to
  generate
• Abstract descriptions of the interface can be
  longer and harder to generate than implementing
  the interface itself.
• Interface builders turned out to be easier

3
But work continued

• Focus Changed
• Task models were leveraged more
• Design assistant aspect emphasized
• A Couple Projects of Interest
• Trident
• Mecano Mobi-D
• FUSE
• AIDE

4
TRIDENT

• Vanderdonckt, J., Knowledge-Based Systems for
  Automated User Interface Generation the TRIDENT
  Experience, Technical Report RP-95-010, Facultes
  Universitaires Notre-Dame de la Paix, Institut
  dInformatique, Namur, 1995.
• An interface design assistant
• Interesting features
• Knowledge-based approach (i.e. expert system)
• Choosing Widgets
• Doing Layout
• Use of Task Models
• Decides where separate windows are needed

5
Choosing Widgets

• Used a decision tree
• Chose abstract interaction objects (AIO)
• Similar to Brads Interactor Model
• Lots of parameters
• Continuous?
• Capacity
• Etc.

6
Choosing Layout

• Uses Right/Bottom Strategy
• Next component is placed to the right or below
  the current component
• Decision made by heuristics or designer

7
Right/Bottom Strategy
8
Windows from Task Models

• Basically used for constructing wizard-like
  interfaces
• What information should be on the first screen,
  etc.

9
What are task models, anyway?

• Description of the process a user takes to reach
  a goal in a specific domain
• Typically have hierarchical structure
• Introduced by GOMS
• Number of different task modeling languages
• GOMS
• UAN
• ConcurTaskTrees

10
ConcurTaskTrees

• Developed by Fabio Paterno et al. for the design
  of user interfaces
• Goals
• Graphical for easy interpretation
• Concurrent model for representing UI tasks
• Different task types
• Represent all tasks, including those performed by
  the system

11
Task Building Process

• Three phases
• Hierarchically decompose the tasks
• Identify the temporal relationships among tasks
  at same level
• Identify what objects are manipulated and what
  actions can be performed on them, and assign
  these to the tasks as appropriate.

• Temporal Relationships
• T1 T2 - Choice
• T1 T2 - Interleaving
• T1 T2 - Synchronization
• T1 gtgt T2 - Enabling
• T1 gtgt T2 - Enabling with Information Passing
• T1 gt T2 - Deactivation
• T1 - Iteration
• T1(n) - Finite Iteration
• T1 - Optional
• T Recursion

12
Example

• Note First example is ambiguous

13
Another Example
14
Building/Editing Task Models

• Tools are available
• ConcurTaskTrees Environment

http//giove.cnuce.cnr.it/ctte.html or Google
for ConcurTaskTrees
15
Recent Systems

• XIML eXtensible Interface Markup Language
• Developed by the makers of Mecano/Mobi-D and
  Trident
• Kitchen-sink language for modeling any part of
  the interface design process
• XWeb
• Now known as ICE Interactive Computing
  Everywhere
• ICrafter
• A system for integrating user interfaces from
  multiple devices
• Personal Universal Controller
• My research

16
XIML

• eXtensible Interface Markup Language
• Designed by RedWhale Software
• Intended to support the full lifecycle of
  interfacebuilding

17
XIML Requirements

• Central Repository of Data
• For one user interface or many
• Comprehensive Lifecycle Support
• Abstract and Concrete Elements
• Relational Support
• Underlying Technology
• XIML must be independent of particular tools

18
Models in XIML

• An XIML document can contain any type of model
• Task
• Domain
• User
• Presentation
• Dialog

19
Example Use for XIML

• Multi-platform interface development

20
Status of XIML

• Used by RedWhale Software to drive their
  interface consultant business
• They have developed many tools
• move interaction data to/from XIML
• Leverage data in XIML to better understand
  various interfaces
• Automate parts of the interface design process

21
Model-Based Interfaces for Control

• XWeb
• ICrafter
• PUC

22
XWeb

• Work by Dan Olsen and group at BYU
• Premise
• Pervasive computing cannot succeed if every
  device must be accompanied by its own interactive
  software and hardwareWhat is needed is a
  universal interactive service protocol to which
  any compliant interactive client can connect and
  access any service.
• The web comes close to solving this problem, but
  is interactively insufficient.

23
XWeb Protocols

• Based upon the architecture of the web
• XTP Interaction Protocol
• Server-side data has a tree structure
• Structured Data in XML
• URLs for location of objects
• xweb//my.site/games/chess/3/_at_winner
• xweb//automate.home/lights/livingroom/
• xweb//automate.home/lights/familyroom/-1

24
XWeb XTP

• CHANGE message (similar to GET in HTTP)
• Sequence of editing operations to apply to a
  sub-tree
• Set an attributes value
• Delete an attribute
• Change some child object to a new value
• Insert a new child object
• Move a subtree to a new location
• Copy a subtree to a new location

25
Platform Independent Interfaces

• Two models are specified
• DataView The attributes of the service
• XView A mapping of the attributes into
  high-level interactors
• Interactors are somewhat like abstract
  interaction objects
• Atomic
• Numeric
• Time
• Date
• Enumeration
• Text
• Links
• Aggregation
• Group
• List

26
XWeb Example

• DataView

27
XWeb Example

• XView

28
XWeb Example

• Interface

29
Other XWeb Details

• Has simple approach for adjusting to different
  screen sizes
• Shrink portions of the interface
• Add additional columns of widgets
• Also capable of generating speech interfaces
• Based on a tree traversal approach like Universal
  Speech Interfaces

30
ICrafter

• Part of the Interactive Workspaces research
  project at Stanford
• Main objective
• to allow users of interactive workspaces to
  flexibly interact with services
• Contribution
• An intelligent infrastructure to find services,
  aggregate them into a single interface, and
  generate an interface for the aggregate service.
• In practice, much of the interface generation is
  done by hand though automatic generation is
  supported.

31
ICrafter Architecture
32
How is aggregation accomplished?

• High-level service interfaces (programmatic)
• Data Producer
• Data Consumer
• The Interface Manager has pattern generators
• Recognize patterns in the services used
• Generate interfaces for these patterns
• This means that unique functionality will not be
  available in the aggregate interface

33
Automatic Generation in ICrafter
34
Manual Generation in ICrafter
35
Personal Universal Controller

• My work with Brad
• Problem
• Appliance interfaces are too complex and too
  idiosyncratic.
• Solution
• Separate the interface from the appliance and use
  a device with a richer interface to control the
  appliance
• PDA, mobile phone, etc.

36
Idea

• Control existing appliances
• Generate multi-modal interfaces

37
Architecture
- Comm. Protocol
- Interface Generators
- Specification Lang.
- Appliance Adaptors
XML-based
38
Language Design

• Approach
• Create reference interfaces
• AIWA Shelf Stereo
• ATT Telephone/Answering Machine
• Test interfaces with subjects
• Users twice as fast and made half the errors with
  reference interfaces as compared to
  manufacturers interfaces
• Analyze interfaces for functional information

39
Language Elements

• State Variables and Commands
• Represent functions of appliance
• State variables have types
• Boolean, Enumeration, Integer, String, etc.
• Variables sufficient for most functions but not
  all
• seek button on a Radio
• Label Information
• One label not suitable everywhere
• The optimal label length changes with screen size
• Speech interfaces may benefit from pronunciation
  and text-to-speech information

40
Language Elements, cont.

• Group Tree
• Specify organization of functions
• We use n-ary tree with variables or commands at
  leaves

41
Language Elements, cont.

• Dependency Information
• Formulas that specify when a variable or command
  is active in terms of other state variables
• Equals, Greater Than, Less Than
• Linked with logical operators (AND, OR)
• For example,
• ltandgt ltequals statePowerStategttruelt/equa
  lsgt ltequals stateRadioBandgtAMlt/equalsgtlt/and
  gt

42
Interface Generators

• Generators for Two Modalities
• Graphical
• Implemented for PocketPC in Java 1.1
• Uses dependency information to generate panel
  structure of interface
• Speech
• Implemented using Universal Speech Interface
  (USI) techniques Rosenfeld 2001
• Uses dependency information to disambiguate
  shortcut words (e.g. play) and resolve
  pre-conditions for a requested function (e.g.
  play CD)

43
Graphical Interface Generator

• Focuses on panel structure of user interface
• Small groups of controls have basic layouts
• Complexity comes from structure of groups
• Structure can be inferred from dependency info!

44
Inferring Structure

• Find sets of variables that are mutually
  exclusive
• Every variable in a set will never be active at
  the same time as a variable in another set
• Create structure with sets, using overlapping
  panels

45
Choosing Panel Types
a)
b)
c)
full screen
partial screen
tabbed
46
Making the Interface Concrete

• Finish conceptual layout
• Choose controls (decision tree)
• Choose row layouts
• (one column, two column, etc.)
• Allocate space
• Examine panel contents and choose sizes
• Instantiate and place controls

47
Generating Speech Interfaces

• Automatically build USI tree from dependencies
• Allows verbal navigation of functional groups
• Automatically generate grammar for parser
• Phrases for query and control
• What is playmode?
• Set playmode to play
• play
• Automatically generate language model and
  pronunciation for recognizer