Measuring IT Success

1
Measuring IT Success

• Reza Torkzadeh
• Department of MIS
• University of Nevada Las Vegas
• www.unlv.edu/faculty/reza/
• Organizational Systems Research Association
• 21st Annual Conference
• Las Vegas, Nevada

2
Measuring IT Success

• Success paradigms
• Success measures
• Usefulness
• Satisfaction
• Impact
• Usage
• E-commerce success measures
• Theory and practice issues

3
Design vs outcome perspective

• There are at least two perspectives on
  evaluating information systems
• Design perspective - has a strong tradition in
  the MIS field, calls for evaluating information
  systems relative to design specifications or user
  needs.
• Outcome perspective - calls for
  performance-related evaluations that focus on
  outcomes.

4
Design vs outcome perspective

• The measurement of IT success has progressed from
  design paradigm to an outcome paradigm.
• As we move from user satisfaction to downstream
  success measures, evaluation must look at
  outcomes rather than design intention.
• User acceptance, perceived usefulness, and ease
  of use are appropriate criteria for evaluating
  developmental progress.

5
System to Value Chain
6
Alternative theory base

• The innovation process model provides a framework
  for describing stage dependent information
  technology success criteria and suggests
  alternative theory bases for developing success
  measures.
• Cooper and Zmud
• Mgmt. Sc. 36(2) 1990

7
Innovation Process Model

• Process Model
• Initiation
• Adoption
• Adaptation
• Acceptance
• Routine Use
• Infusion

• Changing Standards for
• Evaluating Info. Systems
• Perceived usefulness
• User satisfaction
• Usage time
• Usage pattern/impact

8
Perceived usefulness

• Usefulness
• e.g., work more quickly, job performance,
  increase productivity, effectiveness
• Ease of use
• e.g., easy to learn, controllable, clear
  understandable, flexible
• Davis, MIS Quarterly 13(3)

9
End user computing satisfaction

• Content
• Accuracy
• Ease of use
• Format
• Timeliness
• Doll Torkzadeh, MIS Quarterly 12(2)

10
IT impact on work

• Task productivity
• Management control
• Customer satisfaction
• Task innovation
• Torkzadeh Doll, Omega 27, 1999
• Task learning?

11
IT impact

• When we evaluate the impact as people use
  technology, we are not directly assessing the
  technology or the people. Rather, we are
  evaluating a complex socio-technical phenomenon
  defined by the interaction of people and
  technology in a social and organizational
  context. This unpredictable nature of the impact
  construct enhances its theoretical significance
  for both domains of MIS and social sciences.

12
IT impact

• A conception of technologys impact that is
  limited to productivity and/or management control
  is rooted in an industrial paradigm that ignores
  organizationally relevant impacts essential to
  the success and survival of modern organizations.

13
IT impact

• Modern management perspective on IT must go
  beyond routine use and explore ways that have
  substantial implications for the nature of work,
  productivity, learning, and diffusion of
  management functions throughout work force.

14
E-commerce success

• A five-factor 21-item instrument that measures
  means objectives in terms of
• Internet product choice
• Online payment
• Internet vendor trust
• Shopping travel
• Internet shipping errors

15
E-commerce success

• A four-factor 16-item instrument that measures
  fundamental objectives in terms of
• Internet shopping convenience Internet ecology
• Internet customer relation
• Internet product value
• Torkzadeh Dhillon ISR, 13(2)

16
User evaluation

• What is needed for user evaluations to be an
  effective measure of information technology
  success is the identification of some specific
  user evaluation construct, defined within a
  theoretical perspective that can usefully link
  underlying systems to their relevant impacts.
• Goodhue, Mgmt. Sc. 41, 1995

17
System-use

• In Labor and Monopoly Capital, Braverman argues
  that the widespread introduction of information
  technology in manufacturing, services, and white
  collar work has improved productivity and
  management control, but has not changed the labor
  process itself.

18
System-use

• Braverman sees information technology being used
  to facilitate the vertical specialization of work
  with management/staff
• planning the employees work,
• monitoring performance, and
• taking corrective action.

19
System-use

• Using a post-industrial perspective, Hirschhorn
  argues that information technology is used in new
  ways that have decisive implications on the
  general texture of life. Worker skill is not the
  skill of executing, but the skill of problem
  solving and the ability to learn and innovate.
  Work becomes heavily dependent on information
  processing, increasingly abstract, and mediated
  by sensing mechanisms.

20
System-use

• In studying the impact of IT, we cannot just
  study what workers do, but rather how prepared
  they are to do what they might have to do. As
  Hirschhorn argues, the overall impact of
  technology is to diffuse the management
  functions. Empowered workers can use information
  technology to solve problems in innovative ways,
  plan their own work, and monitor/control their
  own performance. We must re-think the old
  organizational designs.

21
System-use

• Zuboff distinguishes between the deskilling
  impact of automating and enhanced cognitive
  skills (sense making, inferential reasoning,
  systemic thinking) needed in an informated
  environment.

22
System-use

• Weick argues that information technology
  utilization requires ongoing structuring and
  sense making. Work is largely cognitive and time
  alone is not a good measure of either the amount
  of work done or the extent of computer usage.

23
System-use constructs

• Decision support
• Problem solving
• Decision rationalization
• Work integration
• Horizontal integration
• Vertical integration
• Customer service
• Doll Torkzadeh , IM, 33, 1998

24
Theory and Practice

• The productivity of substantive research
  activities depends upon efforts to improve theory
  and measurement development
• No instrument should be used without questioning
  the procedures used to develop it and the
  appropriateness of the measure for the research
  being examined.

25
Theory and Practice

• As an applied discipline, the development of
  information system measurement is impaired by
  confusion concerning the opportunities and
  challenges of
• relying upon other disciplines for theory
  development,
• modifying such theories to fit the IT research
  domain, and
• developing original IT theory.

26
Theory and Practice

• Measurement development can occur in one of three
  contexts
• strong theory,
• no theory, and
• weak theory.

27
Theory and Practice

• It is important to identify underlying
  assumptions concerning the opportunities and
  challenges posed by borrowing theory from other
  disciplines or conceptualizing original theory.
• These underlying assumptions frame individual
  thinking on whether and how to develop IT
  theories.

28
Theory and Practice

• Important theoretical questions that guide
  instrument development
• What is the domain of MIS research?
• What is the purpose of an instrument? Is it to
  evaluate the effectiveness or value of an
  application or to predict attitudes/behaviors
  vis-a-vis an application?
• To what extent is the attitude-behavior research
  tradition of social and cognitive psychology
  applicable to MIS research and issues important
  to managers?

29
Theory and Practice

• What Theory
• The attitude-behavior literature evolved out of
  research domains in social and cognitive
  psychology that may be quite different than those
  experienced by the MIS research community.
• The research in social and cognitive psychology
  contains a rich variety of behavioral phenomena
  and often focuses on emotionally charged issues
  (e.g., capital punishment, birth control
  prejudice, etc.)

30
Theory and Practice

• Thus, attitude research in this domain emphasizes
  the affective rather than the cognitive (e.g.,
  belief) dimension of attitudes.
• The bulk of attitude research focuses primarily
  on affect rather than cognitive and behavioral
  dimensions.
• This emphasis on emotional issues of attitudes
  suggests the need for caution when borrowing
  concepts and techniques for use in MIS.

31
Theory and Practice

• How much do we want evaluations of information
  systems to be based on affective (emotional) vs.
  cognitive responses from users?
• Moreover, research results have been
  disappointing.
• Simple models of the attitude-behavior
  relationship are yielding to more complex models
  of the conditions under which attitudes can
  predict behavior.