Building Whole Applications Using Only Programming-by-Demonstration

1
Building Whole Applications Using Only
Programming-by-Demonstration

• Richard McDaniel
• January 14, 1999

2
Purpose

• Allow nonprogrammers to build applications with
  original, dynamic behavior
• Use programming-by-demonstration
• No written language at all - programs created
  solely through demonstration
• Implement in system called Gamut

3
State of the Art

• Full-fledged languages and libraries
• Visual C, Java
• Simplified languages with syntax-supporting
  editor
• Authorware, Agentsheets
• Tools with macro languages
• Director / LINGO, Excel / Visual Basic

4
Outline

• Introduction
• Interaction Techniques
• Video
• Inferencing
• User Study
• Conclusion

5
Domain

• Board Games
• Has background board
• Pieces above board show state
• Pieces follow rule-like behavior
• Autonomous behavior controlled by timers
• Examples
• Tic-Tac-Toe, Pacman, Reader Rabbit

6
Example Pacman
7
Innovations in Gamut

• Interaction Techniques
• Allows developers to specify important details
• Inferencing
• Infer conditions from sources unaffected by the
  behavior
• Infer chains of expressions

8
Inferencing - Conditions

• Relationship can be formed between arbitrary
  objects
• Relating a toggle switch to decide which way to
  move

?
Run Away
9
Expression Chain

• Expressions form chain of relationships
• In Monopoly, the position to move a piece is,
  the square which is the sum of the dices number
  of squares from the original position

Reds Turn
GO
10
Interaction Techniques

• Nudges
• Do Something, Stop That
• Hint highlighting
• Guide Objects
• Temporal Ghosts
• Cards and Decks

11
Nudges
12
Negative Examples

• Standard technique for demonstrating examples
  Macro Recorder
• Requires special controls to make a negative
  example
• Negative examples in Gamut provided by Stop That
• No special set-up required

13
Guides, Ghosts, and Hints
14
Cards and Decks
15
Video
16
Stages of Inferencing
Translation
Matching Actions
Code Generation
17
Translation

• Read events from undo history list
• Convert events to actions
• Eliminate redundant and inconsequential events

18
Matching Actions

• Matches actions in trace to original behavior
• Unordered
• Typically not many (three or four)
• Considerations
• Degree to which actions match
• Priority of actions in original behavior
• Actions which create new objects

19
Recursive Difference

• Propagates changes to descriptions
• Each description contains heuristics for
  comparing values
• Builds an And/Or tree result

20
Code Generation

• Resolves each value difference by creating a new
  description
• Search for previously defined description
• Use heuristics to generate description from
  highlighted objects
• Assemble values into a table and use Decision
  Tree learning

21
Decision Trees
Condition For Pacman Monster

• Created at places where Gamut chooses between
  different behaviors
• Generate attributes using heuristics and hint
  highlighting
• Build table of examples that relate behaviors to
  attributes

Is Eaten
Y
N
Run Home
Is Blue
Y
N
Chase Pacman
Run Away
22
User Study

• Goal
• Determine if nonprogrammers can use Gamut
• Perform an informal study
• Participants
• Nonprogrammers, CMU community
• Location
• HCI Institutes usability lab

23
Tasks
Safari
Pawn Race
G-bert
24
Expertise with Techniques
25
Problems

• Reluctance to create guide objects
• Needed explicit instructions in tasks
• Once told, participants would create appropriate
  objects
• Highlighting inappropriate objects
• Some objects seem too obvious to highlight, like
  ghosts

26
Future Work

• Apply to other domains
• Feedback for just inferred relationships
• Display for language
• Higher-level inferencing

27
Conclusion

• Builds entire applications without a written
  programming language
• Infers a broader range of behavior than prior
  systems
• Is usable by nonprogrammers