Cross cutting themes SOC Workshop

1
Cross cutting themesSOC Workshop

• Objectives - primary secondary
• Phases of a collaboratory - understanding
  confusion frustration
• Motivation intrinsic extrinsic
• Management skills and processes
• Formal and informal roles
• Functions of evaluation

Gillian Kerr www.realworldsystems.net June 19
2003
2
1. Objectives

• Very difficult to manage complex projects without
  clear objectives.
• Primary, core objectives Research
  collaboratories should increase
• Research productivity
• Research quality
• Research impact

3
Key performance indicators

• From June 2001 SOC Workshop
• Possible measures frequency and impact of
  publications diversity among collaborators (on
  multiple dimensions) satisfaction of
  collaborators frequency of travel number and
  prestige of honors accumulated by collaborators
  proportion of collaborations composed of
  first-time collaborators impact of the
  collaborations on the training of young
  scientists attracting newcomers to the field,
  frequency and duration of delays level of
  interpersonal trust extent of shared mental
  models degree of mutually consistent work
  practices and level of public interest.
• Most are secondary or tertiary objectives, not
  primary. Not useful for managing projects. Cost
  is not mentioned.

4
What does success look like?

• Crystallized collaboration like common standards
  and shared data
• Taking on larger questions scaling up
  scientific enterprise
• Reduced communication reduced costs of social
  interaction
• Doing science rather than other stuff (e.g.,
  building trust and relationships)
• Creating and maintaining structures that permit
  productive collaborations, meaning allowing new
  relationships rather than being plugged up in old
  collaborations. Cycle of new, thrashing,
  crystallization, new
• Success may look like peaks and then drops in
  communication careful not to incorporate
  secondary objectives that may be unrelated to
  your overall goals into key performance indicators

5
Scaling up research?

• From November 2000 to September 2001, NASA
  Ames Research Center invited net-based volunteers
  to identify and mark craters on images of Mars.
  As they described it, this was an experiment to
  see if public volunteers (clickworkers), many
  working for a few minutes here and there and
  others choosing to work longer, can do some
  routine science analysis that would normally be
  done by a scientist or graduate student working
  for months on end (http//clickworkers.arc.nasa.g
  ov/top). More than 85,000 volunteers marked and
  classified craters.1 The quality of their work
  was virtually indistinguishable from that of a
  geologist with years of experience in identifying
  Mars craters (http//clickworkers.arc.nasa.gov/do
  cuments/crater-marking.pdf).
• From Lee Sproull 2003 via recommendation from
  Mike Cohen, at http//cip.umd.edu/sproullpaper.doc
  

6
2. Phases

• Form
• Storm
• Norm
• Perform

• Confusion (boundaries objectives)
• Frustration (calibration)
• Focus
• Productivity

What are the tasks involved in each phase? Are
they all necessary? Whats going on in each
phase, and are there any shortcuts?
7
Confusion and frustrationwhats going on?

• Standish Group reports only 34 success rate for
  IT projects (2002)
• Lack of user involvement was major cause of
  project failure, followed by executive support
  and skilled project manager. User involvement
  takes a long time but essential for success.
• Costs and benefits of diversity of users vs
  common ground higher transaction costs, but
  more potential for new information and connections

8
3. Project management skills processes

• 4 types of projects by types of uncertainty
• Variation
• Foreseen uncertainty
• Unforeseen uncertainty
• Chaos
• Each requires different sets of management roles,
  skills and processes.
• Meyer et al, Managing project complexity From
  variation to chaos, MIT Sloan Management Review,
  Winter 2002.

9
Managing in chaos

• The project team must continually create new
  decision trees based on incremental learning.
  Managers must repeatedly and completely redefine
  the project. Execution involves repeatedly
  verifying goals on the basis of learning
  detailed planning only go to next verification
  rapid prototyping, and making ruthless go/no go
  decisions.

10
Managing in chaos, cont

• Necessity of funders to be flexible when dealing
  with uncertainty. Changing, evolving model is not
  necessarily a sign of poor management.
• As we learn, we can move up the scale to
  unforeseen uncertainty and then foreseen
  uncertainty. Projects will get more predictable,
  and can be managed more formally

11
4. Motivation

• Extrinsic vs intrinsic dangers of emphasizing
  extrinsic awards
• Bibliometric analysis recognition of other
  forms of contribution besides articles like
  Natures acceptance of molecule pages. Will they
  be accepted for tenure and promotion?
• Other disciplines will be facing this issue soon
  (physics is already).
• Team awards? Awards for specific roles like
  academy awards?
• Karma points for 2000 authors?
• Intrinsic social norms should be investigated
  more systematically

12
5. Formal and informal roles

• Leaders (Dr. Gilman)
• Project managers
• Community and technology facilitators (BIRN
  boxes)
• Social norm transmitters whats acceptable
  expected behaviour (dressing in referee outfit?)

13
6. Functions of evaluation

• Social scientists should be watching and thinking
  and contributing best practices. But they
  shouldnt get distracted by niceness of talking,
  trust, communication in themselves. Collaboration
  involves costs as well as benefits.
• Projects should have a qualitative evaluation
  independent of measurable objectives
  unanticipated consequences are vital to look at,
  though not related to project management.
  Important distinction.
• The project should have ongoing evaluations that
  are related to management objectives, but that
  include measures of social engagement to track
  user involvement, since thats largest failure
  point.