A Case Against Testing Tools

1
A Case Against Testing Tools

• CSER Meeting November 2001
• Andrew Walenstein

And For Testing Theories Instead
2
Our research product ideas

• we produce and investigate tool ideas
• SHriMP fisheye views
• RMTool reflexion analysis visualization
• PBS landscape views, clustering
• Rigi visual clustering
• etc.

3
Uuusuallyits the idea not the tool

• idea is adopted and propagated, not the tool
• individual tools die, ideas propagate evolve
• whither Rigi v. 2?
• theyll be desiging Rigi-like tools, SHriMP-like
  visualizations, etc.
• ideas are the main lasting legacy of research

4
Ideas generalized statements

• a tool idea is a generalized statement
• about a class of tools/interfaces/techniques
• Rigi-like, PBS-like, etc.
• (i.e., NOT about Rigi v5.4.2)
• concerning their usefulness or utility
• makes things easier, faster, better

5
Butwe test TOOLS!

• specific implementations of the idea are examined
• myriad specifics involved
• colour choices, undo features, help, search
  capability, syntax, window managers, layout
  tweaks, keyboard bindings
• unix environment, windowing system, etc.
• is this OK?

6
Some problems?

• ideas are general, tools are specific
• what is our basis for generalizing?
• can ideas be tested with just 1 implementation?
• can we eliminate effects of implementation?

7
Example SuperBook (1980s-90s)

• pitted hypertext system vs. regular text
• evaluation Q faster / better than a book?
• ver. 1 (fisheye views) slower, but close
• ver. 2 (search idea) comparable
• ver. 3 (presentation idea) better than books

8
Idea evaluation?

• ver. 1 FAILURE.
• ver. 2 marginal pass.
• ver. 3 empiricism says OK. (5 years later?)
• so the fisheye view idea was eventually
  validated. Success!!
• BUT

9
Questions to raise

• fisheye view idea was probably good
• but how do we really know?(maybe it was the
  presentation?)
• fixups were needed
• do we have time/money to polish each tool?
• when do we know when to stop tinkering or to just
  abandon the idea?
• which features do we copy when making a new tool?

10
One problem is informality
ideas intuitions
interpret stretch
informal mapping
abstract
concrete
test results
implementation
11
How to fix?

• NOT by pitting one tool against another
• NOT by writing lists of concrete tool features
• NOT by surveying user attitudes / opinion
• NOT by studying work practices
• NOT by modeling comprehension / cognition

12
OK, Dr. Smartypants, how?

• One way formalize ideas
• test them directly

13
Theory-based experimentation

• build theories
• of tool usefulness, utility, etc.
• then
• map theory to implementation as a claim
• test claim by testing claim predictions
• theory forms basis for generalizing
• measure usefulness?

14
Hrmph!

• not easy
• need to understand psychological effects (eeew)
• few existing theories
• how to formalize classes of tools?
• how to formalize usefulness?
• but we must begin!
• when do we plan on being scientific about our
  ideas?

15
Simple Example Rigi

• a key tool idea visual cluster perception
• theory outline thinking hard, seeing easy
• Rigi implementation thinking
  cohesion/coupling comparison seeing visual
  clutter

16
Potential experimental methods

• standard time measurement
• traditional stimulus variation
• one variant makes visual operator difficult
• one variant displays inappropriate clusters
• paper implementation?
• eye trackers?
• all of the above (triangulation)
• (Note to hard scientists falsifiable,
  quantifiable)

17
One possible direction

• we will need some theories
• engineers think a lot
• I have studied cognitive support theories
• how why tool features aid thinking
• guiding belief let others do the psych
• they do it better get money prestige for it
• we pillage results and adapt to our problems

18
My current status

• I scoured psych, cog sci, HCI, distilled
• 3 basic cog supp principles
• 16 more specific variations
• application
• dozens of ideas categorized idea catalog
• two RE tools analyzed in relative depth
• field study explored experimental techniques
• shows some promise

19
Conclusions

• experiment at the claim level, not tool level
• build theories of usefulness, aid, assistance
• make claims about tools explicit/formal
• because
• stability hacking tool wont invalidate result
• early feedback dont need perfect prototype
• (validity our intutions may not be right!)