Noor Azizah KS MOHAMADALI

1
EVALUATION STUDIES IN HEALTH INFORMATICS AND A
PROPOSED INTEGRATED MODEL OF USER ACCEPTANCE OF
TECHNOLOGY

• Noor Azizah KS MOHAMADALI
• Supervisor
• Dr. Jonathan Garibaldi
• IMA SEMINAR
• 28th April 2009

mnk_at_cs.nott.ac.uk School of Computer Science
2
Presentation Outline
Introduction An overview of Evaluation
Study An overview of Existing Model of
Technology Acceptance (Information Systems
Theory) An overview of Existing Work on
Technology Acceptance in HealthCare Proposed
Integrated Conceptual Model of Technology
Acceptance Current Work and Future Work
Conclusion
IMA SEMINAR 28th April 2009
3
Introduction

Implementation of new information systems in the
organization costs thousand of millions of
dollars each year. Decision makers often
believe that technology will bring benefits.
However, evidence from various studies on
implementation of new system in healthcare
sectors does fail (Southon et. Al.,
1999). Effective evaluation of healthcare
information systems are necessary in order to
ensure systems adequately meet the requirements
and information processing needs of users and
health care organizations.
IMA SEMINAR 28th April 2009
4
An overview of Evaluation Study in Health
Informatics

• Evaluation study in health informatics is a
  study that measure or explore the attributes of
  health information systems (in planning,
  development, implementation or operation) with
  the aim to informs a decision to be made
  concerning that systems in a specific context.
  (Mohd Yusof and Papazafeiropoulou,2008)
• Evaluation is carried out to seek answers to the
  following (Friedman and Wyatt, 1997)
• Why objective of evaluation?
• Who which stakeholders perspective is going
  to be evaluated?
• What aspects of evaluation?
• When which phase in the system development
  life cycle?
• How method of evaluation?

IMA SEMINAR 28th April 2009
5
Why Objective of Evaluation
Authors Purpose of Evaluation
Yusof and Papazafeiropoulou (2008) Evaluation can be used to improve HIS through past experience, to identify more effective techniques or methods, investigate failure and learn from previous experiences.
Meijden et. al (2003) Only a thorough evaluation study can show whether or not specific system was successful in a specific settings
Nahm et. al (2007) The assessment outcome of CIS implementation is vital not only to justify the cost within organization but also to promote the national agenda to improve healthcare information technology
IMA SEMINAR 28th April 2009
6
Who and What ?
Stakeholders Concerns
Management/ Organization Whether investment justified? Will the users accept the system? How committed are the users to use the system? Etc
User Are all necessary facilities (e.g. training support, network infrastructure, etc.) provided? How useful and easy is to use the system? How safe and secure is the system? How is the quality of the information provided by the system? Etc.
Patient How will it improve the quality of services?
Developer Has it met all the user requirements?
IMA SEMINAR 28th April 2009
7
When Stage of Evaluation
Evaluation can be carried out during each of the
three main phases of the system development life
cycle
Pre-implementation (development)
During implementation
Post-implementation (Yusof and
papazafeiropolou, 2008)
IMA SEMINAR 28th April 2009
8
How Method of Evaluation
Author (s) Objectivist Subjectivist
Wyatt and Liu (2002) Objectivist evaluation is an evaluation approach that uses experimental designs and statistical analyses of quantitative data Subjectivist approach is an evaluation approach relies on qualitative data which can be derived from observation, interview and analysis of documents and other artefacts.
Limitation of objectivist approach - Cannot
provide answer as to why and how a system works
within a specific settings. Many researchers now
tend to use subjectivist approach when
undertaking evaluation work (Collen, 1986
Friedman and Wyatt, 1997, Kaplan, 2001a Gremy
et. al., 1999)
IMA SEMINAR 28th April 2009
9

• Among all, user acceptance is one most important
  area of research.
• User acceptance is a risk for successful of any
  IT project (Louise K. Schaper, 2007)
• Clinical Information System (CIS) experienced
  high level of user resistance, thus understanding
  of a successful CIS implementation is critical to
  improve health care services as a whole.
  (Jean-Marc Palm, 2006)

IMA SEMINAR 28th April 2009
10
Research Methodology (Part 1)

• Steps Involved
• Identify critical success factors (LR).
• Analyse Existing Theory/ Model of User Acceptance
  (IS Theory).
• Analyse Existing work on user acceptance of
  technology.
• Development of Proposed Integrated Model of
  Technology Acceptance.
• Evaluate Proposed Model.

11
Step 1 Some of Identified Critical Success
Factors
Factors (s) Sources
It improves job performance Garfield (2005) etc
Allow to work quicker Despont-Gros et. al(2005)
Save time Ting-Ting Lee(2008) etc
I find the system to be easy to use William J.Doll(1991) etc
Instruction are clear and easy to remember William J.Doll(1991) etc
User-friendly Sicotte et. al(2006) etc
Conciseness and Completeness Maryati Yusof et. al(2008) etc
Helpdesk support Martens, Weijden (2008) etc
Training etc Mahmood et al (2000),
12
Step 2 Overview of Existing Model of Technology
Acceptance (IS Theory)
Unified Theory of Technology Acceptance and Use
of Technology by Vankatesh, 2003 (UTAUT) The
basic concept underlying this model is that
individuals will form various beliefs and
attitudes regarding the technology these will,
in turn, have an impact on their intentions to
use the technology and therefore, affect their
actual use of the technology. Information
Systems Success Model by DeLone McLean, 2002,
2003 A system can be evaluated in terms of
information, system, and service quality these
characteristics affect the subsequent use or
intention to use and user satisfaction.
Task-technology fit (TTF) by Goodhue and
Thompson,1995 Task-technology fit (TTF) theory
holds that IT is more likely to have a positive
impact on individual performance and be used if
the capabilities of the IT match the tasks that
the user must perform.
13
Our Observation

• UTAUT and IS Success Model addressing the same
  issue
• but with different construct defined.
• Behavioural Intention to Use/ Intention to Use/
  Use.
• -IS Success Model Information Quality, System
  Quality and Service Quality
• -UTAUT - Performance Expectancy, Effort
  Expectancy, Social Influence And Facilitating
  Condition.
• The importance of fit between the factors
  (Ammenwerth et al., 2006 Kaplan,2001b
  Goodhue,1998).
• The importance of moderating factors such as age,
  gender, experiences that may or may not have
  influence on new systems.

IMA SEMINAR 28th April 2009
14

• Step 3 Analyses of Existing Work on User
  Acceptance of Technology

15
The design-reality gap model (Heeks,2006)
Information
Information
Technology
Technology
Processes
Processes
Limitation Moderating factors
Objectives and values
Objectives and values
Staffing and skills
Staffing and skills
Management systems and skills
Management systems and skills
Other resources
Other resources
Design
Reality
Gap
Strength ?Identification of most of the
important factors for evaluation ?Introduction
of gap features
IMA SEMINAR 28th April 2009
16
ICT and OTs A model of information and
communication technology acceptance and
utilisation by occupational therapists (Schaper
and Pervan,2007)
INDIVIDUAL CONTEXT
Computer anxiety
TECHNOLOGICAL CONTEXT
Computer self-efficacy
Performance Expectancy
Computer Attitude
Effort Expectancy
Behavioural Intention
Use Behaviour
IMPLEMENTATION CONTEXT
Social Influences
Compatibility
Organisational Facilitating condition
MODERATORS Age Gender, Experiences,
Voluntariness of Use, Access, Clinical
speciality, Clinical workload, setting type,
geographic area
Limitation Fit factors
IMA SEMINAR 28th April 2009
17
CHEATS a generic information and communication
technology evaluation framework (Shaw, 2002)
This proposed framework identified six
dimensions for evaluation which are clinical,
human and organizational, educational,
administrative, technical and social.
Strength Detailed measurement for each of
above factors.
Limitation ? Fit factors ? Moderating factors
IMA SEMINAR 28th April 2009
18
Understanding IT acceptance by individual
professional Towards an integrating view (Yi et
al., 2006)
Perceived Behavioural Control
H10
H5
H6
Perceived Ease of Use
H11
H15
H8
Behavioural Intention
H10
Result Demonstrability
H7
H12
Personal Innovativeness in IT
Perceived Usefulness
H16
Image
H13
H2
H3
Subjective Norms
H1
H10
Limitation Moderating factors, Fit factor,
Organizational factors
IMA SEMINAR 28th April 2009
19
Step 4 Proposed Integrated Model of Technology
Acceptance

1. To make use of IS Success Model and Unified
   Theory of Technology Acceptances and Use of
   Technology.
2. To incorporate fit factors as proposed by Goodhue
   (1995).
3. Proposed Conceptual Model of User Acceptance of
   Technology

IMA SEMINAR 28th April 2009
20
(No Transcript)
21
Step 5 Process of Evaluating a Model

• Ability to explain past observations
• Ability to predict future observations

http//en.wikipedia.org/wiki/Scientific_modeling
22
Model Evaluation (Phase 1)
Factors (s) Construct (s)
It improve job performance Performance Expectancy (Perceived Usefulness)
Improve communication Performance Expectancy
Save time Performance Expectancy
I find the system to be easy to use Effort Expectancy (Perceived Ease of Use)
Instruction are clear and easy to remember Effort Expectancy
User-friendly Effort Expectancy
Conciseness and Completeness Information Quality
Excellent helpdesk support Service Quality
Training etc Facilitating Condition etc
23
Model Evaluation (Phase 2)

• Case Study Strategy - Qualitative methods
• Initial contact with clinical collaborators from
  Nottingham Breast Institute was made on April
  2008, who have deployed the Distiller Software
  (Slide Path, 2008).
• Users Medical Researcher Students
• Data collected through audio and hand-recording.
  
• The data will then be transcribed into filed
  note and will be analyzed.
• Emerging themes will be identified.
• NViVo software

IMA SEMINAR 28th April 2009
24
Current and Future Works

• Investigate techniques to assign priority among
  factors.
• Analytic Hierarchy Process (AHP) and Fuzzy
  Cognitive Map (FCM)
• Work on Knowledge Representation
• Identify appropriate techniques to represent the
  knowledge (critical success factors) and to
  predict rate of successful implementation of new
  system.

IMA SEMINAR 28th April 2009
25
Conclusion
Identifying those critical factors for
successful implementation of new information
systems in health care sector may help decision
makers to make better investment decisions in new
technology more effectively. Proposed
integrated model hopefully will serve as
guideline in conducting evaluation study,
particularly on user acceptance of technology.
IMA SEMINAR 28th April 2009
26
References
1 Ting-Ting Lee, Mary Etta Mills, Barker
Bausell, Ming-Hui Lu (2008). Two-stage evaluation
of the impact of a nursing information system in
Taiwan. International Journal of Medical
Informatics, 77, 698-707. 2 Gray Southon,
Chris Sauer, Kit Dampney (1999). Lesson from a
failed information system initiative issues for
complex organisations. International Journal of
Medical Informatics, 55, 33-46. 3 F. Gremy,
J.M. Fessler, M. Bonnin (1999). Information
systems evaluation and subjectivity.
International Journal of Medical Informatics, 56,
13-23. 4 Morris F. Collen (1986). Origins of
Medical Informatics. The Western Journal of
Medicine, 145, 778-785. 5 Maryati Mohd Yusof,
Anastasia Papazafeiropoulou (2008). Investigating
evaluation frameworks for health information
systems. International Journal of Medical
Informatics, 77, 377-385. 6 Eun-Shim Nahm,
Vinay Vaydia, Danny Ho, Barbara Scharf, Jake
Seagull (2007). Outcome assessment of clinical
information system implementation A practical
guide. Nursing Outlook, 55, 282-288. 7 Maryati
Mohd Yusof, Jasna Kuljis, Anastasia
Papzafeiropoulou, Lampros K. Stergioulas (2008).
An evaluation framework for Health Information
Systems human, organization and technology-.t
factors (HOT-.t). International Journal of
Medical Informatics, 77, 386-398. 8 Nicola T.
Shaw (2002). CHEATS a generic information
communication technology (ICT) evaluation
framework. Computers in Biology and Medicine, 32,
209-220. 9 M.J Van Der Meijden, H.J. Tange,
Troost, A.Hasman (2003). Determinants of Success
of Inpatient Clinical Information Systems A
literature Review. Journal of the American
Medical Association, 10, 235-243. 10 Mo Adam
Mahmood, Janice M.Burn, Leopoldo A. Gemoets,
Carmen Jacquez (2000). Variables affecting
information technology end-user satisfaction a
meta-analysis of the empirical literature. Int.
J. Human-Computer Studies, 52, 751-771. 11 J.D
Martens, T. van Der Weijden, (2008). Feasibility
and Acceptability of a Computerized System with
automated reminders for prescribing behavior in
primary care. Int. J. Medical Informatics, 77,
199-207. 12 Jeremy C. Wyatt, Sylvia M. Wyatt
(2003). When and how to evaluate information
systems?. International Journal of Medical
Informatics, 69, 251-259. 13 J C Wyatt, J L Y
Liu (2002). Basic concepts on medical
informatics. Journal of Epidemiology and
Community Health, 56, 808-812. 14 Bonnie
Kaplan (2001). Evaluating informatics
applications clinical decision support systems
literature review. International Journal of
Medical Informatics, 64, 15-37. 15 Elske
Ammenwerth, Carola Iller, Cornelia Mahler (2006).
IT-adoption and the interaction of task,
technology and individuals a .t framework and a
case study. BMC Medical Informatics and Decision
Making, 6, 1-13.
27

• 16 Jean-Marc Palm, Isabelle Colombet, Claude
  Sicotte, Patrice Degoulet (2006). Determinants of
  User Satisfaction with a Clinical Information
  System. AMIA 2006 Symposium Proceeding. 614-618.
• 17 Christelle Despont-Gros, Henning Mueller,
  Christian Lovis (2005). Evaluating user
  interactions with clinical information systems A
  model based on human-computer interaction models.
  Journal of Biomedical Informatics, 38, 244-255.
• 18 Richard Heeks (2006). Health information
  systems Failure, success and improvisation.
  International Journal of Medical Informatics, 75,
  125-137.
• 19 Louise K. Schaper, Graham P. Pervan (2007).
  ICT and OTs A model of information and
  communication technology acceptance and
  utilization by occupational therapists.
  International Journal of Medical Informatics,
  76s, S212-S221.
• 20 William J.Doll (1991). The Measurement of
  End-User Computing Satisfaction Theoretical and
  Methodological Issues. MIS Quarterly, 5-10.
• 21 Gray Southon (1999). IT, Change and
  evaluation an overview of the role of evaluation
  in health services. International Journal of
  Medical Informatics, 56, 125-133.
• 22 Clifford S. Goodman, Roy Ahn (1999).
  Methodological approaches of health technology
  assessment. International Journal of Medical
  Informatics, 56, 97-105.
• 23 Mun Y. Yi, Joyce D. Jackson, Jae S. Park,
  Janice C. Probst (2006). Understanding
  information technology acceptance by individual
  professionals Towards an integrative view.
  Information and Management, 43, 350-363.
• 24 Bonnie Kaplan (2001). Evaluating informatics
  applications-some alternative approaches theory,
  social interactionism, and call for
  methodological pluralism. International Journal
  of Medical Informatics, 64, 39-56.
• 25 C.P. Friedman, J.C. Wyatt (1997). Evaluation
  Methods in Medical Informatics, Springer-Verlag,
  New York.
• 26 Joan Ash, Marc Berg (2003). Report of
  conference Track 4 socio-technical issues of
  HIS. International Journal of Medical Informatics
  69, 305-306.
• 27 Monice J. Garfield (2005), Acceptance of
  Ubiquitous Computing. Information Systems
  Management, 22, 24-31.
• 28 Claudie Sicotte, Guy Pare, Marie-Pierre
  Moreault, Andree Paccioni (2006). A Risk
  Assessment of Two Interorganizational Clinical
  Information Systems. Journal of the American
  Medical Informatics Associations, 13, 557-566.
• 29 http//www.istheory.yorku.ca/UTAUT.htm
• 30 http//www.fsc.yorku.ca/york/istheory/wiki/in
  dex.php/Delone_and_McLean_IS_success_model
• 31 http//www.fsc.yorku.ca/york/istheory/wiki/in
  dex.php/Task-technology_fit

28

Questions?
IMA SEMINAR 28th April 2009