Hipikat: A Project Memory for Software Development

1
Hipikat A Project Memory for Software Development

• The CISC 864 Analysis
• By Lionel Marks

2
Whats the Typical Scenario for a New Developer?

• A new developer joins the team and knows little
  of the project
• An existing member offers advice and helps
  newcomer finish first tasks
• Information generally about basic ideas on the
  problem domain and pointers on using tools
  effectively
• Mentors give information on the eccentricities of
  the system and why things are done that way

3
What is the New Scenario?

• Team members are not co-located
• Hard for a newcomer to get up-to-speed on the
  project on the virtual team
• Using electronic media such as mailing lists,
  code repositories and work tracking programs, can
  create a project memory with Hipikat

4
What is Hipikat Good for?

• With the volume of information being so enormous,
  sifting through the documents would take too long
• Hipikat allows the user to search these archives
• Searching CVS for code to clone for your new
  method
• Searching configuration management tools for a
  history of changes, to help in setup maintenance
  or to redo a setup.

5
Related Work

• Group Memory
• Design Assistant and TeamInfo had the information
  generated by humans specifically for their tool
  to query
• Recommender Systems
• Remembrance Agent and Code Broker did not use
  multiple information sources

6
Related Work

• Programming from Examples
• Reuse View Matcher and Explainer which has
  libraries of specifically created code to query,
  not existing code from the current system.
• Mining Software Repositories to help in SW
  Maintenance
• CVSSearch search comments from CVS commits
• Expertise Browser and Expertise Recommender
  recommend developers with expertise at certain
  parts of a project
• Zimmerman et al., Ying et al. found change
  patterns in repositories to recommend potentially
  relevant files when working on a change in the
  code of a project

7
How Hipikat Works

• Hipikat forms the project memory from source
  code, documentation, and other electronic media
  such as e-mail messages and bug reports. All are
  put in a central database that can be queried.
• It can then be queried to recommend artifacts
  that are relevant to the users current task.
• Information sources can be monitored periodically
  or continuously, and when they are committed to
  the database, then they can be included in
  searches

8
How Hipikat Works

• A user makes a query by selecting an artifact in
  the Eclipse project workspace and choosing Query
  Hipikat from the context menu. (Interesting!)
• User can also search the Hipikat database through
  Search Hipikat

9
The Digg Effect

• Training the System (Increasing the rank of
  certain artifacts in queries)
• Just by clicking on the artifact (that it looks
  useful)
• Clicking a thumbs-up or thumbs-down
• Problems (Can anyone see them?)

10
The Digg Effect

• Training the System (Increasing the rank of
  certain artifacts in queries)
• Just by clicking on the artifact (that it looks
  useful)
• Clicking a thumbs-up or thumbs-down
• Problems (Can anyone see them?)
• Clicking on artifact ? What if relevant on first
  sight, but not really upon inspection? Inc. rank
  of bad match
• Clicking on thumb ? Asking to verify usefulness
  before even used the data. Cant rank like this

11
Evaluation Criteria

• Precision vs. Recall
• In a perfect world, whatever the system
  recommends is exactly what you need
• Giving recommendations that have examples of
  use for relevant APIs rather than just giving
  the place in code where you must make your
  changes (i.e. The solution) (Interesting!)

12
Eclipse Newcomer Study

• Finding Participants
• Require adequate programming experience in Java
• Have experience developing large or medium-sized
  SW
• Very small pool of recruit-able computer science
  undergrads with the above criteria (Good point!)
• Also wanted to compare the solutions
• Got some of experienced members of the Eclipse
  Development team
• Asked them to perform the same tasks for baseline
  comparison

13
Eclipse Newcomer Study Results

• The first task, implementing the pop-up window
  when hover over the side of code to suggesting
  breakpoint properties
• Experts performed worst on handling the special
  cases
• 25 of experts got them
• 38 of newcomers got them perfectly right
• 63 of newcomers got it basically right
• Harder task
• 75 of experts solved it correctly
• 38 of newcomers got it right

14
Use of Hipikat

• More queries made on the harder task
• Almost all queries made within the first hour
• Generally once the person knew what file(s) to
  change, and had a general plan, no more queries
  were made.
• Criteria for finding a recommendation useful
• Is the problem report interesting
• Similar enough to current task code re-use
• Learn relevant information
• For difficult coding problems, searched and found
  filenames that look potentially relevant
• Did not look at source code changes, but rather
  built their understanding from scratch looking
  at the file in an editor. (easier, did not want a
  false lead and have to continue searching)

15
My General Thoughts

• Good paper in general
• The paper did a good job at stating its facts and
  shortcomings.
• Would like to see it used in a different way, as
  a data miner for teaching the person about the
  system rather than providing assistance when in
  doubt (too much recall for when trying to perform
  a single task)

16
Likes and Dislikes of this Paper

• Likes
• Its study of how people thought about the problem
  
• The tool itself is an intriguing idea
• The style of the paper very readable!
• Dislikes
• Asking that people take into account special
  cases when first coding something in a system a
  bit of a reach
• Saying that the participants must have worked on
  a medium to large SW system before. Many
  undergrads will go into industry working on their
  first large system ever. Would be interesting for
  your tool to get someone who is really new to
  these kinds of SW systems see if they use
  Hipikat more