Hints on System Design and Research Methodology

1
Hints on System Designand Research Methodology

• CS290F
• Winter 2005

2
System Research

• Change happens
• problems have changed in last 20-30 years
• but research methodology stays constant
• Success in graduate research
• choosing the right research problem
• doing the work, addressing the problem
• presenting the results
• writing/publishing papers
• giving talks
• writing / giving the perfect job talk

3
Wisdom of the Ages

• Nuggets of wisdom available
• perspectives from veterans of the trade
• Well try to summarize a couple of them
• 3 guidelines on choosing a research problem???
• Hints for computer system designButler Lampson
  (Microsoft Research)
• How to have a bad career in researchDavid
  Patterson (U. C. Berkeley)
• Various hints on giving talks, writing
  papersFox, Levin, Redell, etc

4
Choosing a research problem

• Work on the edge of what you know
• doing so expands your horizon of knowledge
• Choose a problem that has impact
• either applicative or scientific
• if in doubt, choose scientific over application
• Choose a problem youre uniquely qualified to
  solve
• stay away from common problems well within reach
  of competitors

5
Hints for Computer System Design

• Butler Lampson
• PhD Berkeley 1967
• Faculty Berkeley, MIT
• Researcher Xerox PARC, Digital Research,
  Microsoft Research
• Turing Award, 1992
• Hints circa 1982 (SOSP-IX)

6
General Hints

• Hints from building and studying real systems
• For each hint, two key points
• why it helps in building a good system
• where in the system building it helps
• Organized along the Why axis
• functionality
• speed
• fault-tolerance

7
I. Functionality

• Most basic motivation for a system
• Design of correct interfaces is key
• it is a programming language
• Keep it simple
• capture the minimal essence of an abstraction (do
  not generalize, easy to err)
• simplify
• minimize implementation/debugging complexity
• critical for heavily used abstractions (files,
  V/M )less so for rarely used components (passwd
  gen)

8
Keep It Simple cont.

• Make it fast
• programmers can build on top of a complete, fast
  API
• can use profiling tools to gauge frequency of
  calls
• Dont hide power
• if the system can do something fast, open that
  functionality to programmer
• Leave it to the client
• if its simple to switch back forth, solve one
  problem and let the client do the rest

9
Continuity

• Keep basic interface stable
• changing code to follow is expensive
• Keep a place to stand
• design APIs with upgradability in mind
• support backward compatibility when possible
• e.g. method deprecation in Java

10
Making Implementations Work

• Implement a strawman
• the first try never works out
• but you learn what you really want
• can solidify APIs, system design
• Keep secrets of the implementation
• hide things that can change
• tension with making assumptions for performance
• Divide and conquer
• attack a hard problem by dividing into smaller
  problems

11
Handling All the Cases

• Handle normal worse cases separately
• normal should be fast/optimized
• worst must make progress
• optimize for the common case (e.g. caching)

12
II. Speed

• Split resources between consumers
• tradeoff utilization for performance
• dedicated management is simpler/faster
• Use static analysis if you can
• but fall back on dynamic if not possible
• Cache answers to expensive computations
• only if f(x) is functional (same result w/ same
  args)
• must choose cache size correctly to avoid
  thrashing
• Use hints to speed up execution
• speculate optimistically on results of
  calculations
• must check for correctness (cheaply)

13
More Speed

• When in doubt, use brute force
• choose b.f. rather than optimizations based on
  weak/incorrect assumptions
• easy analysis over complexity
• leverage Moores Law
• Compute in background if possible
• Use batch processing if possible

14
III. Fault-tolerance

• End-to-end
• error recovery always necessary e2e
• all others are optional, for performance
• Log updates
• cheap, append-only
• allow regeneration of current state
• functional updates using constant args
• Make actions atomic or restartable
• simplify recovery

15
Break

• Then well talk about Pattersons advice how to
  have a bad career in research
• Hints on giving talks (Fox)
• Hints on writing papers (Fox)

16
Hints for Giving a Talk (Fox)

• Know the audience and watch the jargon
• Keep the big picture in mind
• Tell me a story, dont read an article
• Pace yourself
• What did you just say again?
• There will be questions

17
Know the Audience Watch Jargon

• Many talks have varied audiences
• the most critical one your job talk
• Use the hourglass model
• start broad (high level), hook everyone
  inconvince audience the problem is useful and
  hard
• dive down into details, impress expertsjust
  sufficient detail to show convincing depth
• wrap up, recapture entire audience
• Avoid jargon / technical terms
• define terminology before usage

18
Keep the Big Picture in Mind

• What is the big picture?
• motivate the problem well
• can take up to 1/3 of your talk
• What is the context?
• what are the subproblems?
• where does your work fit in?
• how does it relate to existing work?
• What is your high level contribution?
• come back to high level throughout talk

19
Tell Me a Story

• Slides
• avoid sentences, use short phrases
• slides are hints for you to speak from
• Presentation style
• use minimal notes
• maintain eye contact, move across audience
• use personal experience to elaborate bullets
• Humor
• can animate talk, grab attention of audience
• can be annoying if overdone

20
Pace Yourself

• Time management is critical
• going over is very bad form (esp. job talk)
• Keys
• practice, practice, practice
• use key timepoints to gauge progress
• rule of thumb 2 min / content slide
• Be adaptive
• speed up if behind schedule
• know when and what to skip over
• be prepared for extra details if time left

21
What Did You Just Say Again?

• Use recaps to highlight high-level ideas
• especially important if middle of talk focused on
  technical details
• easy to lose track of the point
• remind audience of significance of work
• Future work an implications
• use slides near end to discuss consequences of
  your work (new things to be done, new issues
  discovered by your work)

22
There will be Questions

• Questions are unpredictable
• know material one level deeper than slides
• think about issues ahead (prepare backup slides)
• Time management
• answer questions with composure
• handle nonstop persistent questions with lets
  take this offline
• Key
• practice talks you care about
• get ruthless (and varied) audience members

23
How to Write a Good xxxx Paper

• Three classes of papers
• presents a real system (implementation)
• what did you learn from the experience
• what was unexpected?
• if measurement paper, determine which graphs
  convey your message, then wrap paper around them.
• presents an unimplemented system (vision)with
  useful ideas/techniques
• you must be convinced of your vision!
• make concrete statements that can be backed up by
  citations, real data, or prelim. results
• others (theory papers, survey papers)

24
Before You Write

• Give 5-10 min talk to colleague
• forces you to collect thoughts
• others will help identify strength/weaknesses and
  contributions
• Do not cram
• last minute papers do not work out
• corollary multitasking on papers is generally
  bad
• Know when to take a break
• can get too close to the work
• take break to reestablish focus and perspective

25
Starting Checks

• Write from an outline
• figure out main message (from 10000 feet)
• everything else follows
• use bulleted list
• Do not write abstract/title until end
• the paper will change as you write
• intro will also morph
• Make small of points
• usually 2-4
• appear in intro, body, conclusion

26
Writing

• start from outline
• make outline reflect subsections
• write out succinct points of each section
• what must be said
• write out topic sentences for each section
• read through for logical flow
• give to colleague in diff. area for validation
• fill in paragraphs

27
Editing

• Organization/structure can change
• this is ok, 3-4 major revisions are normal
• Pause for top-down reading
• do this occasionally to maintain perspective
• Get early feedback
• external reviews on drafts are invaluable
• Cite cite cite
• be complete in related work

28
Writing Style

• See Strunk and White
• Use active sentences
• Omit needless words
• Be brief and concise, not flowery or formal
• Check for spelling and grammar

29
Final Checks

• Does paper make contributions clear?
• Does outline of paper flow at each logical level?
• Before telling me about X, convince me I should
  care
• Have you made each important point 3 times?
• Have you gotten reviews from each relevant area?
• Have you searched for and acknowledged related
  work?
• dont put down previous work, be positive about
  your differences
• Do the abstract/conclusion capture the top-level
  message?