Reading Discussion

1
Reading Discussion s519 by Peter Hall 1/21/09

• Ryan, J., McClure, C. R., Bertot, J. C. (2001).
  Choosing measures to evaluate networked
  information resources and services Selected
  Issues. In C. R. McClure J. C. Bertot (Eds.),
  Evaluating networked information services
  Techniques, policy and issues (pp. 111-135).
  Medford, NJ Information Today Inc.

2
Abstract

• IT and information managers see the importance in
  measuring their organizations networks and
  information flow important
• Two questions to consider What are the key
  elements of a strategy that measures these
  networks? What are the key issues that may arise
  when employing such a strategy?
• Research suggests successful managers develop a
  strategy and consider key issues prior to
  implementing these measures and continuing with
  data collection, analysis, and use procedures

3
Examples of Measurements

• Quantitative (Statistics)
• Performance measures (Statistical ratios)
• Qualitative (Interviews, surveys, observations,
  shadowing, etc.)
• Any others?

4
The Research

• Sponsored by United States Institute for Museum
  and Library Services
• Multi-method approach Site visits, case
  studies, interviews, focus groups, surveys,
  observation, path analysis
• Iterative approach Study team adapted
  collection and analysis based on learning
• Limited study scope Study participants were all
  library information managers
• Drawbacks and/or limitations?

5
Selection Issues

• What were some potential concerns and issues that
  these library information managers identified
  prior to adopting data collection, measures, and
  analysis?
• Why are you measuring something?
• What are the potential uses?
• Tradeoffs New measures bring additional costs,
  time, and resources of an organization
• What will you do with your findings?
• May have different outcomes and effects for
  different stakeholders in organization (i.e.
  management, employees, shareholders)

6
Measure Types

• Capacity Ability of an organization to use or
  deliver information through network
• Use Utilization measure of these resources or
  services
• Efficiency Relates services or resources used
  to services or resources provided
• Impact Effect of resource or service on an
  activity or situation
• Outcome Are goals achieved?
• Examples of measure types?
• Other considerations
• Defining measures, data collection, and analysis
• Fair measures All of an organizations
  operations

7
Data Collection

• Sources Are all relevant stakeholders input
  collected?
• Can data be collected uniformly for accurate
  analysis?
• Which data collection measures should be used?
  Qualitative, quantitative, both? Any new
  collection techniques that should be used?
• Privacy Are stakeholders free to give an
  accurate portrayal?
• Balancing data collection across organization
• Proper instruction and preparation is key
• Any other potential issues?

8
Data Analysis and Use

• Combine new data with existing information
  Internal, external
• Example Your new data with national studies or
  internal performance or composite measures
• Is this information compatible for comparison?
• How do you analyze the results generated by a new
  measure?
• Is further instruction necessary?
• Who interprets and how?
• How is this data presented?
• How do you use it?
• Does the end cost outweigh the benefit?
• Table 6.11 - Examples

9
Conclusions / Recommendations

• Information managers should choose network
  measures based on context of change, complexity,
  utility, and resources
• Proper strategy required prior to implementation
  of network measures, data collection, and
  analysis for success
• Reducing and balancing staff burden of collection
  and analysis
• Be able to provide proper instruction/training to
  properly educate important stakeholders
• Future and additional considerations and research

10
Discussion Topics

• Has anyone been involved in this process in their
  organizations?
• Success/failures, most important metrics, time
  consumption, cost/benefits?
• Did anything come out of this data?
• Any other analytics tools used?
• i.e. External, web-based tools
• Internal work flow documents
• Thoughts on the reading Agree/disagree, glaring
  deficiencies, techniques used, research methods?

11
Work Flow Example