Studies of Effectiveness of Learning Networks

1
Studies of Effectiveness of Learning Networks

• Starr Roxanne Hiltz, Murray Turoff, and Yi Zhang

2
Completed Study From Virtual Classroom to
Virtual University (1997- 2001)

• Some new findings and publications
• Replication of earlier study on undergraduate CIS
  majors only, with graduate and undergraduate
  students from a variety of disciplines

3
Findings

• Overall effectiveness measures and correlation
  coefficients among variables very similar
• Paper The Effects of Course Level, Course Type,
  and Gender on Course Outcomes in the "Virtual
  University"
• Raquel Benbunan Fich and Starr Roxanne Hiltz,
  HICSS 2002 (draft available)

4
Abstract

• Data for almost 2000 students allow us to
  contrast outcomes for courses in three different
  modes of delivery (completely online, mixed, and
  completely on campus), for different levels and
  types of courses and students.

5
Virtual U Project- Findings

• In terms of grades, we found that online students
  achieved higher grades than those in
  face-to-face (FtF) courses.
• There was no significant interaction of mode of
  delivery with course level (graduate vs.
  undergraduate), course type (more technical
  computer science and engineering courses vs. less
  technical courses in CIS, humanities, and
  management), or associated with gender.

6
Virtual U Project- Findings

• For one measure of subjective satisfaction,
  Perceived Learning, there was a significant
  interaction of mode with course level graduate
  students in mixed mode courses reported the
  highest levels.

7
Conclusions

• Whereas the previous NJIT project was limited
  to undergraduate CIS courses, the results of
  this project enable us to further generalize the
  finding that ALN modes of delivery tend to
  produce results equal to or better than those for
  FtF modes of course delivery. (Of course, this
  is dependent on faculty being effective virtual
  profs....)

8
Becoming a Virtual Professor Pedagogical Roles
and Asynchronous Learning Networks

• Nancy Coppola, Starr Roxanne Hiltz, and Naomi
  Rotter
• HICSS 2001 revised Draft paper, to appear in
  JMIS

9
Becoming a Virtual Professor

• This paper presents a qualitative study of role
  changes that occur when faculty become online or
  "virtual" professors.
• In 20 semi-structured interviews of faculty,
  coded with pattern analysis software, the authors
  captured role changes enacted by instructors in
  ALN settings -- cognitive roles, affective roles,
  and managerial roles.

10
A Virtual Professor- Findings

• Overall, faculty reported a change in their
  teaching persona, towards more precision in their
  presentation of materials and instructions,
  combined with a shift to a more Socratic
  pedagogy, emphasizing multilogues with students.
• Further work coding conference transcripts for
  swift trust formation

11
Current Project Learning Networks Effectiveness
Research Program (started 1/2001)

• Starr Roxanne Hiltz, Murray Turoff, Ricki Goldman
  ( NJIT) and Sharon Derry ( U. of Wisconsin), plus
  several Ph.D. students
• Paper for this workshop Studies of Effectiveness
  of Learning Networks
• (Starr Roxanne Hiltz, Yi Zhang, and Murray
  Turoff)

12
Goals

• The goal of this research program, which began in
  January of 2001, is to increase the quality,
  quantity, and dissemination of results of
  research on the effectiveness of Asynchronous
  Learning Networks (ALN).

13
Planned Activities I

• Create a WebCenter for Learning Networks
  Research.
• It will include a series of online knowledge
  bases that are regularly updated and will be
  available through the project web site to
  researchers, faculty, the press, and the public.
• Users are invited to contribute to these
  knowledge bases, as well as to use the
  information already there.

14
Planned activities

• Build/strengthen an ALN evaluation research
  community that will create and share improved
  research methods, theoretical frameworks, and
  instrumentation for assessing the outcomes of
  online learning.
• Face to face workshops and asynchronous
  communication will be used to achieve this. The
  results of these activities will be used to
  enhance the materials on the web site.

15
Workshops

• For beginners and intermediates-- at ALN
  annual meetings ( Maryland, Florida)
• Advanced, for practicing ALN researchers 1.5
  days at NJIT, spring of 2002 (expenses will be
  paid for approximately 12 participants)

16
Building a Virtual Community for
ALNhttp//www.alnresearch.org
17
Features of a Virtual Community

• Community members should be able to contribute
  to, discuss and learn from the communitys
  knowledge, and from each other (Ram et al. 1999)

18
Project Motivation

• The group of researchers in the ALN field is
  expanding
• We need to have a knowledge base for ALN
  researchers, so that new projects build on past
  findings
• A place for researchers to easily find and
  communicate with each other
• A place to find resources for research, such as
  published studies, research instruments, etc.

19
Major Features

• The Knowledge Base
• Contribute and Get Members get information
  that they want from the WebCenter, also they
  contribute what they think might be useful for
  other researchers.
• Communication
• Use discussion boards for exchanging ideas,
  getting feedback, conducting paper fairs, etc.

20
Research Questions

• How effectively does the ALN WebCenter serve
  researchers at different levels? Will the
  researchers find it really useful?
• Is the combination of discussion boards and
  enhanced digital library a good way to implement
  a virtual community for researchers?
• Will the idea of Get and Contribute make the
  community more vigorous?

21
Structure of WebCenter

• See the screen shots of our Alpha version
  http//www.alnresearch.org

22
Home page of ALN WebCenter
23
Criteria for Inclusion

• Empirical studies of the effectiveness of
  learning networks published in a refereed journal
  or conference proceedings, in the English
  language.
• They must be full papers, not just extended
  abstracts. This is operationally defined as at
  least five pages in length.

24
Criteria for Inclusion

• To be considered as an empirical research study,
  the paper must include research questions or
  hypotheses ( at least implicitly), describe some
  data collection methods, and report some
  empirical results.
• It must have a reasonable number of subjects-
  tentatively defined as 20 or more.

25
Page of list of papers
26
Coding of the studies

• A template was created for coding of the most
  important aspects of the methods and findings of
  the research papers. Most of the coding of
  studies is done by Ph.D. students.

27
Database entry Part I
28
Database entry Part II
29
Contributions Page
30
Research Instrument Contribution Form Part I
31
Research Instrument Contribution Form Part II
32
Resources

• Annotated links to web sites of interest
• Tutorials (hypertext form) -- the first is on
  educational theories
• Second tutorial, on research methods used in
  educational research, with links to examples of
  studies using them, almost completed
• http//www.sit.wisc.edu/steinkuehler/NJIT/

33
Alternative designs for beta

• http//www.alnresearch.org/newversion/home.html
• http//www.alnresearch.org/newversion/new-photo6/p
  review1/home.htm.

34
Analysis of findings of the studies

• We decided to focus on a key subset of the
  papers those that compare the effectiveness of
  ALN courses in terms of student outcomes, to that
  of traditional "face to face" courses.
• We identified 19 of the studies that clearly meet
  this criterion.

35
Procedure

• Two of the authors categorized the types of
  individual measures used, and whether each
  individual finding reported showed ALN to be
  better than traditional courses, no different, or
  worse.
• The second step was that all of the results for
  each study were categorized in terms of whether
  they, in total, showed ALN to better, worse, or
  no different on the whole.

36
Measures used in the studies

• Objective and subjective used about equally
• Objective measures- number of studies
• Grades, for specific projects or exams or for the
  entire course, compared to sections or students
  using other delivery modes (16).
• Measures of the quality of work (e.g., group
  projects may be judged on creativity,
  completeness, length, etc.) (9).

37
Objective measures, cont

• Course completion rates (3 studies)
• Counts or measures of activity levels or patterns
  (5)

38
Subjective measures

• Student self-assessments (through questionnaires
  or interviews) of course learning outcomes
  (absolute or compared to traditional courses) 8
• Perceived effectiveness of the mode or system
  used for delivery (including convenience,
  motivation, usability, time required, access to
  professor) 18 studies
• Quality of the instruction or materials (3
  studies).

39
Overall results (p.6)

• Most of the studies either measure effectiveness
  in more than one way (e.g., grade distributions
  plus subjective student assessments) and/or study
  different courses, resulting in many "mixed
  results."
• Looking at all measures, 28 of the comparisons
  showed ALN to be better than FtF 24 no
  difference 12 worse

40
Overall Results

• Pattern of results for each study
• Eight studies showed largely positive results in
  terms of ALN being better than FtF
• Eleven showed mixed or no difference sets of
  findings

41
Conclusions and Discussion

• The evidence is overwhelming that ALN tends to be
  as effective or more effective than traditional
  modes of course delivery, at the university
  level.
• There really is no need for more studies to
  explore this question. What we need is more
  research that will enable us to make it even more
  effective, especially as new technologies
  proliferate.

42
Conclusions and Discussion

• We need to develop both more sophisticated and
  more comprehensive theoretical frameworks, and
  also more valid methods and instruments than
  those which have characterized a majority of
  studies to date.
• Need for more longitudinal studies (over a series
  of courses for the same students)

43
Summary We hope the WebCenter can

• Attract people who are interested in ALN field.
• Help new researchers to get information to design
  studies
• Help researchers to exchange ideas
• Help students find the research literature
• Create a warm, productive environment for
  researchers all over the world

44
Thank you!Any SUGGESTIONS ???