5. Computer Science and Human-Computer Interaction

1
5. Computer Science and Human-Computer
Interaction

• 5.1. Motivation
• 5.2. User Interface Software Concepts
• 5.2.1. Design Process for Interactive Systems
• 5.2.2. Interaction Techniques
• 5.2.3. Interaction Styles
• 5.2.4. User Interface Management Systems
• 5.2.5. User Interface Description Languages
• 5.3. User Interface Software Tools
• 5.3.1. Window Systems
• 5.3.2. Toolkits
• 5.3.3. UIMS
• 5.4. Future Directions

2
5.1. Motivation

• Easier to use, but harder to program Myers
• Multiprogramming
• Real-time programming
• Robustness
• Testability
• Modularization
• Need to support iterative design and testing
• Software tools

3
5.2. User Interface Software Concepts

• Dialogue independence
• Interface (syntax) vs. application (semantics,
  business logic)
• Model-View-Controller architecture
• Interaction techniques
• Basic interaction styles
• Design levels conceptual, semantic, syntactic,
  lexical
• Generate UI from abstract description
• Early research in user interface management
  system (UIMS) and user interface description
  languages (UIDL)
• User interface software tools
• New interaction styles (non-WIMP)

4
5.2.1. Design Process for Interactive Systems

• Partition into Four Levels Foley Wallace
• Conceptual
• Semantic
• Syntactic
• Lexical

5
5.2.2. Interaction Techniques

• Interaction task
• Interaction device
• Interaction technique
• A way of using a device to perform a generic HCI
  task
• Widget in a toolkit
• Two information processors communicating via
  constrained interface Tufte
• Goal Increase bandwidth across channel

6
5.2.3. Interaction Styles Current

• Command Language
• Question and Answer
• Form-based
• Menu
• Natural Language
• Direct Manipulation

7
Interaction Styles Emerging

• Virtual and augmented reality
• Tangible interfaces
• Ubiquitous, pervasive, handheld
• Lightweight, tacit, passive, non-command,
  context-aware
• Affective computing
• Multi-modal interfaces
• Reality-based interfaces
• Brain-computer interaction

8
5.2.4. User Interface Management Systems Research

• Dialogue Independence Hartson Hix
• User Interface Management System (UIMS) Approach
  Olsen
• Before
• After
• Levels of design Conceptual Johnson
  Henderson, Semantic, Syntactic, Lexical

9
Classes of Programmers and Tool Users

Application
Programmer
UIMS
User

Tools
Interface

Designer
(UIDL)
Runtime
End

Application
User
UIMS

10
Dialogue Independence

• Same UIMS for many applications
• Different interfaces to same application
• Interface to null application (rapid prototyping)

11
5.2.5. User Interface Description Languages

• How does the user interface designer describe the
  desired user interface to the UIMS or toolkit?
• Layout
• Behavior
• With a User Interface Description Language (UIDL)
• May be interactive
• Generate UI from abstract description
• UIDL describes user-visible behavior, but not
  implementation
• More in Section 5.3.3
• Other uses of UIDL (evaluation)

12
5.3. User Interface Software Tools

• Goal Support dialogue independence and
  separation of programming roles Foley, van Dam,
  Feiner, Hughes

13
5.3.1. Window Systems

• Manage shared resources (pixels, keyboard,
  mouse), provide clipping, translation
• Tells client when exposed, resized, or other
  damage
• Could implement in different places
• Single built-in window manager or allow separate
• Dispatcher
• (N.B. Security classifications shown
  are simulated only.)

AccessWindowMouseEvent (Event e) . . .
FileWindowMouseEvent (Event e) If
(e-gttypeMS_LEFT) . . .
14
5.3.2. Toolkits

• Library of reusable routines for implementing
  human-computer interface features
• Basic components
• Set of widgets
• "Intrinsics"
• Does not try to describe or support user
  interface as a whole
• Standard look and feel
• Encourage consistency
• But imposes limits on interaction style

15
Toolkits (cont'd)

• Examples
• Visual Studio (Windows)
• Interface Builder (Macintosh)
• GTK (X/Unix/Linux)
• Based on controls or "widgets"
• (N.B. Security classifications shown
  are simulated only.)

AccessCheckBox (Boolean val)
ClassifTypein (char val)
DisplayMsgFile ()
16
5.3.3. UIMS

• Higher level than procedure library
• May include main program, provide external
  control of application
• Call semantics/business logic as subroutines
• Message to component software
• Design time, run time, and evaluation time tools
  Myers
• Module partitioning Runtime vs. interface design
  time partitioning

17
Model-View-Controller

USER

• Design pattern, originally Smalltalk
• Found, in some form, in most UI toolkits
• Matches dialogue independence, syntactic/semantic
  split
• Extend to web browser

18
User Interface Description Language

• Designer creates it, UIMS executes it
• Specific language or technique suitable for
  describing user interfaces
• Describe user-visible behavior, but not
  implementation
• Goal Generate a UI from high-level, abstract
  description
• Platform-independent if possible (Windows, Mac,
  X)
• Platform-type-independent if possible (desktop,
  cellphone, voice)!
• Model-based UI development Paterno Jacob,
  Limbourg, Vanderdonckt
• Current GUI practice
• Has evolved from general purpose UIDL into small,
  stable set of widgets and classes
• Change with new UI styles, portability needs
• Choice of design language at heart of UIMS

19
Basic Classes of UIDLs

• Based on techniques for specifying static
  languages
• Both can be modified for interactive languages
• BNF
• send-cmd SEND DRAFTNAME RECIPNAME
• State Transition Diagram
• Alternate Form of State Transition Diagram
• start SEND -gtgetdr
• getdr DRAFTNAME -gtgetre
• getre RECIPNAME -gtend

SEND
DRAFTNAME
RECIPNAME
start
getdr
getre
end

20
Early UIMS/UIDL Research

• Language-based
• Menu Network Kasik
• BNF, grammar-based Olsen
• State transition diagram Newman
• Event language
• Declarative
• Non-language-based
• Interactive builder (bottom up)
• Current state of practice, use GUI techniques to
  specify GUI
• Plug together components with pre-defined
  interactive behavior
• Automatic generation (top down) Foley
• By demonstration Myers
• Other categories
• Constraint-based Hudson
• User-oriented Moran

21
Example State Diagram as UIDL

• Command-driven Graphical Report Editor
• Jacob, Using formal specifications...

22
Examples Visual Programming for UI

• Layout vs. Behavior
• UIMS, with state diagrams for behavior
• Jacob, A state transition diagram language...
• Visual Basic, pre-programmed behavior

23
UI Tools Mobile Devices

• Basic HCI development and design principles
• Largely GUI, not new interaction style
  gestures, GPS, ...
• Use context Full attention or not?
• Development tools
• Early Clumsy tools, restricted UI designs
• Much like early GUIs, function key-driven
  (pre-mouse) GUI
• Current UIs and tools very similar to standard
  GUI
• iPhone Interface Builder (Mac, Objective C)
• Droid Java Swing (minor differences)
• UI Consistency
• Low level consistency
• High level models, UIDL could help
• Example IMAP mail clients

24
UI Tools Standardization

• Early
• Many desktop platforms
• No standardization
• Current
• 3 major platforms, standard widgets
• Settled into stable toolkits, widget libraries,
  GUI class libraries
• UIMS seems like overkill
• Emerging
• Mobile, automobile
• Diverse platforms, no standardization
• Wider range of interaction techniques
• Model-based development, UIMS redux
• Future
• Same issue for new non-WIMP interaction styles
• Still wider range of possible interactions

25
UI Tool Trends

• UI toolkits, libraries
• Become important, integral part of software
  development tool sets
• e.g., Macintosh Cocoa UI features are well
  integrated
• Good uptake despite unusual programming language
• Web browser as UI platform
• Cloud computing
• Chrome-only OS
• New tools for web UIs, but still clumsy, poorly
  integrated
• Making cross platform toolkits less significant

26
UI Evaluation

• Directly from UIDL
• Evaluate interface without building system
• Emerging area, but lack underlying formal
  knowledge of UI design
• Analyze user interface to detect specific
  properties Darlington et al.
• Apply human performance models Card
• Automated screen design critique Tullis
• Prove safety-critical properties
• Screen mockup, prototyping tools
• Slide show" of canned screens based on user
  inputs
• UIMS sans application (use stubs for semantics)
• Visual Basic, Powerpoint, plain HTML also used
  this way

27
5.4. Future Directions Emerging Interaction
Style

• Parallel, Highly-interactive (non-WIMP)
• Lightweight, Context-aware, Non-command
• Continuous ( Discrete)
• Tangible Interfaces
• Microprocessors, 3D forms
• Ubicomp, Pervasive, Context-aware
• Sensors, networks
• Reality-based Interaction
• Brain-computer Interaction

28
Implications for Software
29
Reality-Based Interaction

• Connects several emerging interaction styles
• Understand together as a new generation of HCI
  through RBI Jacob et al., Reality-Based
  Interaction...
• Computer interaction more like rest of world
• Exploit skills and expectations user already has
  about simple, everyday, non-digital world
• Reality extensions
• Design tradeoffs

30
Tangible User Interfaces

• Simple, transparent mechanical structures
• Use knowledge of physical world to operate
• Augment physical objects with digital meaning
• Combine physical digital representations
• Exploit advantages of each
• Jacob et al., A Tangible Interface...
• Zigelbaum et al., The Tangible Video Editor...

31
Technology for TUI

• Sensors
• Produce signal in response to change in
  surroundings
• Actuators
• Produce physical change in response to signal
• Microcontroller
• Communicates with devices and with main computer
• Other technologies
• RFID
• Computer vision
• ...
• Wider range of I/O events than GUI
• Development methodology, tools
• None!

32
TUIML

• A Visual Language for Modeling Tangible User
  Interfaces Shaer
• TAC paradigm
• Each TUI consists of token within a constraint
• Same object may sometimes be token, sometimes
  constraint
• Two tier model fits well
• Dialogue (states, storyboard)
• Interaction (especially continuous)

33
Conclusions 1. Research Background

• Background for following current research in user
  interface software
• Interaction styles
• Dialogue independence
• Four levels of design
• User interface management system (UIMS)
• User interface description language (UIDL)
• Goal Generate UI from abstract description
• Types of software tools
• New interaction styles (non-WIMP) and
  requirements
• Easier to use, harder to program?

34
Conclusions 2. Practice

• What research ideas can be applied in building
  user interfaces today?
• Separate interface from application programming
• Model-View-Controller architecture
• Use specialized tools (UIMS) for leverage
• Use specialized languages (UIDL) for perspicuity
  and portability

35
Sources

• Textbooks
• Olsen
• Foley, van Dam, Feiner, Hughes
• Shneiderman, Plaisant, Cohen, Jacobs
• History
• Early UIMS research Buxton et al. Foley and
  Wallace Jacob Kasik Newman Olsen
• Classic surveys ACM Transactions on
  Computer-Human Interaction Myers, Hudson,
  Pausch, Computing Surveys Hartson Hix
• Current research
• ACM UIST Conference on User Interface Software
  and Technology (annual conference)
• ACM EICS Symposium on Engineering Interactive
  Computing Systems (conference)
• CADUI Conference on Computer-Aided Design of User
  Interfaces (conference)
• ACM Transactions on Computer-Human Interaction
  (journal)
• interactions (ACM magazine)

36
References

• W. Buxton, M.R. Lamb, D. Sherman, and K.C. Smith,
  ''Towards a Comprehensive User Interface
  Management System,'' Computer Graphics 17(3) pp.
  35-42 (1983).
• S.K. Card, T.P. Moran, and A. Newell, ''The
  Keystroke-Level Model for User Performance Time
  with Interactive Systems,'' Comm. ACM 23 pp.
  396-410 (1980).
• J. Darlington, W. Dzida, and S. Herda, ''The Role
  of Excursions in Interactive Systems,''
  International Journal of Man-Machine Studies 18
  pp. 101-112 (1983).
• J. Foley, W.C. Kim, S. Kovacevic, and K. Murray,
  ''Defining Interfaces at a High Level of
  Abstraction,'' IEEE Software 6(1) pp. 25-32
  (January 1989).
• J.D. Foley, A. van Dam, S.K. Feiner, and J.F.
  Hughes, Computer Graphics Principles and
  Practice, Addison-Wesley, Reading, Mass. (1990).
• J.D. Foley and V.L. Wallace, ''The Art of Graphic
  Man-Machine Conversation,'' Proceedings of the
  IEEE 62(4) pp. 462-471 (1974).
• H.R. Hartson and D. Hix, ''Human-computer
  Interface Development Concepts and Systems for
  its Management,'' Computing Surveys 21(1) pp.
  5-92 (1989).
• S.E. Hudson, ''Graphical Specification of
  Flexible User Interface Displays,'' Proc. ACM
  CHI'89 Human Factors in Computing Systems
  Conference pp. 105-114 (1989).
• R.J.K. Jacob, "A Specification Language for
  Direct Manipulation User Interfaces," ACM
  Transactions on Graphics, Vol. 5(4) pp. 283-317
  (1986) http//www.cs.tufts.edu/jacob/papers/tog.p
  df.
• R.J.K. Jacob, ''A State Transition Diagram
  Language for Visual Programming,'' IEEE Computer
  18(8) pp. 51-59 (1985) http//www.cs.tufts.edu/j
  acob/papers/ieeecomputer.pdf.
• R.J.K. Jacob, "Using Formal Specifications in the
  Design of a Human-Computer Interface,"
  Communications of the ACM, Vol. 26(4) pp. 259-264
  (1983) http//www.cs.tufts.edu/jacob/papers/cac
  m.pdf.
• R.J.K. Jacob, A. Girouard, L.M. Hirshfield, M.S.
  Horn, O. Shaer, E.T. Solovey, and J. Zigelbaum,
  "Reality-Based Interaction A Framework for
  Post-WIMP Interfaces," Proc. ACM CHI 2008 Human
  Factors in Computing Systems Conference pp.
  201-210, ACM Press (2008) http//www.cs.tufts.edu
  /jacob/papers/chi08.pdf.

37

• R.J.K. Jacob, H. Ishii, G. Pangaro, and J.
  Patten, "A Tangible Interface for Organizing
  Information Using a Grid," Proc. ACM CHI 2002
  Human Factors in Computing Systems Conference pp.
  339-346, ACM Press (2002) http//www.cs.tufts.edu
  /jacob/papers/chi02.pdf.
• R.J.K. Jacob, Q. Limbourg, and J. Vanderdonckt
  (Eds.), Computer- Aided Design of User Interfaces
  IV, Kluwer Academic Publishers, Dordrecht (2005).
• J. Johnson and A. Henderson, "Conceptual Models
  Begin by Designing What to Design," interactions,
  Vol. 9(1) pp. 25-32 (January 2002).
• D.J. Kasik, ''A User Interface Management
  System,'' Computer Graphics 16(3) pp. 99-106
  (1982).
• T.P. Moran, ''The Command Language Grammar A
  Representation for the User Interface of
  Interactive Computer Systems,'' International
  Journal of Man-Machine Studies 15 pp. 3-50
  (1981).
• B.A. Myers, ''User Interface Software Tools,''
  ACM Transactions on Computer-Human Interaction
  2(1) pp. 64-103 (March 1995).
• B.A. Myers, Creating User Interfaces by
  Demonstration, Academic Press, Boston (1988).
• B. Myers, S.E. Hudson, and R. Pausch, "Past,
  Present, and Future of User Interface Software
  Tools," ACM Transactions on Computer- Human
  Interaction, Vol. 7(1) pp. 3-28 (March 2000).
  Also reprinted in Human-Computer Interaction in
  the New Millenium, ed. J.M. Carroll,
  Addison-Wesley/ACM Press, Reading, Mass., 2002,
  pp. 213-233.
• W.M. Newman, ''A System for Interactive Graphical
  Programming,'' Proc. Spring Joint Computer
  Conference pp. 47-54, AFIPS (1968).
• D.R. Olsen, Building Interactive Systems
  Principles for Human-Computer Interaction, Course
  Technology Inc. (2009).
• D.R. Olsen, User Interface Management Systems
  Models and Algorithms, Morgan Kaufmann, San
  Mateo, Calif. (1992).
• F. Paterno, Model-Based Design and Evaluation of
  Interactive Applications,, Springer-Verlag,
  London (2000).
• O. Shaer and R.J.K. Jacob, "A Specification
  Paradigm for the Design and Implementation of
  Tangible User Interfaces," ACM Transactions on
  Computer-Human Interaction (2009)
  http//www.cs.tufts.edu/jacob/papers/tochi.shaer
  .pdf.
• B. Shneiderman, C. Plaisant, M. Cohen, and S.
  Jacobs, Designing the User Interface Strategies
  for Effective Human-Computer Interaction,
  Addison-Wesley, Reading, Mass. (2009).
• E.R. Tufte, ''Visual Design of the User
  Interface,'' IBM Corporation, Armonk, N.Y.
  (1989).

5-37
38

• T.S. Tullis, ''A Computer-Based Tool for
  Evaluating Alphanumeric Displays,'' Proc.
  INTERACT '84 Conference, B. Shackel, ed. (1984).
• J. Zigelbaum, M. Horn, O. Shaer, and R.J.K.
  Jacob, "The Tangible Video Editor Collaborative
  Video Editing with Active Tokens," Proc. TEI 2007
  First International Conference on Tangible and
  Embedded Interaction pp. 43-46 (2007)
  http//www.cs.tufts.edu/jacob/papers/tei07.zigel
  baum.pdf.

5-38