Using PLA to Liberate Learning (PLA: participatory learning approach)

1
Using PLA to Liberate Learning(PLA
participatory learning approach)

• Michael Bieber, Jia Shen, Dezhi Wu, Vikas
  Achhpiliya
• Information Systems Department
• College of Computing Sciences
• New Jersey Institute of Technology
• http//web.njit.edu/bieber
• November 2003

2
Outline

• Motivation
• PLA Participatory Learning Approach
• A bit of theory
• Experimental results
• Interesting issues

3
Motivation

• To increase learning of course content
• Learning through active engagement
• involve students as active participants
• with the full problem life-cycle
• through peer evaluation
• Minimize overhead for instructors

4
Outline

• Motivation
• PLA Participatory Learning Approach
• A bit of theory
• Experimental results
• Interesting issues

5
PLA Process
All entries posted on-line

• Each student creates 2 exam problems
• Instructor edits the problems if necessary
• Each student solves 2 problems
• Students evaluate (grade) the solutions to the
  problems they authored, writing detailed
  justifications
• Ph.D. students evaluate each problem a second
  time
• Instructor gives a final grade
• optional Students can dispute their solutions
  grade, by evaluating it themselves and writing
  detailed justifications
• Instructor resolves the dispute

6
(No Transcript)
7

Instructor Control Process
Student Learning Process
Course Design
Process Flow Learning from doing the PLA
activities
Make up problems
Set up on-line environment
Read- other problems - other solutions - grade
justifications - disputes
additional learning from reading everything peers
write
Solve problems
Exam Process Control Assign ID Edit
questions Assign who answers questions Assign
level-2 graders
Level-1 and Level-2 graders grade solutions
Determine Final Grades
Dispute final grade
Resolve Disputes
8

Instructor Control Process
Student Learning Process
Confirmation ID, understand process
Course Design
Make up problems
Set up on-line environment
Read- other problems - other solutions - grade
justifications - disputes
Solveproblems
Exam Process Control Assign ID Edit
problems Assign who solves problems Assign
level-2 graders
Level-1 and Level-2 graders grade solutions
Determine Final Grades
Dispute final grade
Resolve Disputes
9
Evaluation (grading)

• Evaluation includes
• Written critique or justification (positive or
  negative)
• Optional separate sub-criteria to critique
• Solution result is correct and complete (40)
• Solution was well explained (30)
• Solution demonstrated class materials well (10)
• Solution cited appropriate references (20)
• Grade (optional recommended to save instructor
  time)
• Evaluation/grade may be disputed (optional)
• Student must re-evaluate own solution when
  disputing

10
Instructor should provide

• Detailed instructions and timetable
• Solution what is expected
• Critiquing and grading guidelines

11
Outline

• Motivation
• PLA Participatory Learning Approach
• A bit of theory
• Experimental results
• Interesting issues

12
Constructivism(Learning Theory)

• The central idea is that human learning is
  constructed, that learners build new knowledge
  upon the foundation of previous
  learninglearning throughout the exam process
• Two classic categorizations
• Cognitive Constructivism (Piagets theory)
• Social Constructivism (Vygotskys theory)

13
Cognitive Constructivism (Piaget 1924)

• Knowledge is constructed and made meaningful
  through individuals interactions and analyses of
  the environment. --gt knowledge is constructed
  in the mind of individual
• Knowledge construction is totally
  student-centered.

14
Learning

• Learning is a constructivist, often social
  activity occurring through knowledge building
  (Vygotsky, 1978)
• Knowledge building activities include
  contributing to, authoring within, discussing,
  sharing, exploring, deploying a collective
  knowledge base (ONeill Gomez 1994 Perkins
  1993).

15
Learning

• People learn as they navigate to solve problems
  (Koschmann et al, 1996) and design
  representations of their understanding (Suthers
  1999)
• Learning requires cognitive flexibility (Spiro et
  al. 1991), and results from interaction with
  people having different experiences and
  perspectives (Goldman-Segall et al. 1998)

16
Expert-like Deep Learning

• Categorizing knowledge and constructing
  relationships between concepts are likely to
  promote expert-like thinking about a domain
  (Bransford 2000).
• To design appropriate problems for their peers,
  students must organize and synthesize their ideas
  and learn to recognize the important concepts in
  the domain.
• This results in deep learning (Entwistle 2000)
• seeing relationships and patterns among pieces of
  information,
• recognizing the logic behind the organization of
  material
• achieving a sense of understanding

17
Where is Knowledge Constructed in PLA?

• In all PLA stagesconstructing problems,
  solutions, grade justifications, dispute
  justifications
• When reading everything their peers write
• Students also are motivated to learn more when
  peers will read their work (McConnell, 1999).

18
Assessment Learning

• Main goals of tests
• To measure student achievement
• To motivate and direct student learning
• The process of taking a test and discussing its
  grading should be a richly rewarding learning
  experience (Ebel and Frisbie 1986)
• Assessment should be a fundamental part of the
  learning process (Shepard 2000)

19
Outline

• Motivation
• PLA Participatory Learning Approach
• A bit of theory
• Experimental results
• Interesting issues

20
Course Information

• NJIT CIS677 Information System Principles
• Graduate level core course (Masters/Ph.D.)
• Aim study how IS/IT can be used effectively
• Both on-campus and distance-learning sections
• software Virtual Classroom/WebBoard
• Traditional Exam
• Three-hour, in class, 3-4 essay questions, 6
  pages of notes
• Used PLA 5 times between Fall 1999 and Summer
  2002
• We compared control groups without PLA and
  treatment groups with PLA
• Also, we used with shorter essay questions in CIS
  365, undergraduate course on file structures in
  Fall 2002, with similar survey results.

21
Enjoyability
Cronbachs Alpha0.68
Questions SA A N D SD Mean S.D.
I enjoyed the flexibility in organizing my resources 26.2 48.9 16.7 3.6 4.6 3.88 1.00 221
I was motivated to do my best work 23.5 42.9 28.2 3.4 2.1 3.82 .92 238
I enjoyed the examination process 17.2 42.3 22.6 10.5 7.4 3.51 1.13 239
SA - strongly agree (5 points) A - agree (4) N
- neutral (3) D - disagree (2) SD - strongly
disagree (1) the mean is out of 5 points S.D. -
standard deviation
22
Perceived Learning
Cronbachs Alpha0.88
Questions SA A N D SD Mean S.D.
I learned from making up questions 17.9 42.5 21.3 13.8 4.5 3.55 1.08 240
I learned from grading other students answers 17.7 48.1 19.4 9.3 5.5 3.63 1.06 237
I learned from reading other peoples answers 15.8 45.0 22.1 11.3 5.8 3.54 1.07 240
I demonstrated what I learned in class 13.6 50.2 22.6 10.9 2.7 3.61 .95 221
My ability to integrate facts and develop generalizations improved 21.8 49.2 25.6 2.1 1.3 3.88 .83 238
I learned to value other points of view 17.6 51.9 27.6 1.3 1.6 3.82 .81 239
I mastered the course materials 7.4 51.6 31.4 6.9 2.7 3.54 .84 188
23
Recommendation Do Again!
Question SA A N D SD Mean S.D.
Would you recommend in the future that this exam process used? 20.7 40.1 24.5 8.9 5.8 3.60 1.10 237
Similar results for CIS365 undergraduate file
structures course using short essay questions
(Fall 2002)
24
Outline

• Motivation
• PLA Participatory Learning Approach
• A bit of theory
• Experimental results
• Interesting issues

25
What students liked best

• Active involvement in the exam process
• Flexibility
• Reduction in tension

26
Trade-offs

• Trade-offs for students (traditional vs. PLA)
• Timing Concentrated vs. drawn-out (2.5 weeks)
• Access to information limited vs. the Internet
• Experimental integrity we couldnt justify the
  process to the students fully
• Trade-offs for professors
• Fewer solutions to evaluate, but each is
  different
• Timing Concentrated vs. drawn-out process
• Much more administration

27
Timing

• PLA for exams took 2.5 weeks
• For frequent activities PLA processes could
  overlap
• e.g., quizzes, homeworks
• Students could be creating problems for one
  quiz,while solving problems for the prior quiz,
  while evaluating solutions from the quiz before
  that
• Benefits to overlapping PLA activities
• working with materials from several classes at
  the same time
• could reinforce class materials
• could result in synthesis (combined understanding)

28
Scope

• Which activities?
• so far exams
• what about quizzes, homeworks, larger projects,
  in-class projects
• Which problem types?
• so far short and long essay questions
• what about multiple choice, short answer,
  computer programs, semester projects
• Sub-problems
• computer program design implementation
• semester project outline execution

29
Scope, cont.

• Course Level
• Graduate, undergraduate, secondary school (high
  school, junior high)
• Disciplines
• IS/IT, business, science, engineering,
  humanities, medical, all of secondary school

30
Scope, cont.

• Degree of Evaluation (assigning grades)
• Currently solutions
• What about
• quality of problems
• quality of evaluations/grades
• All could be disputed
• Degree of Participation
• students could evaluate each
• students could arbitrate disputes

31
Evaluation Results

• Written critique (positive or negative)
• Grade (optional recommended to save instructor
  time)
• Recommendation to accept or reject the artifact
  (problem, solution, evaluation)
• If rejected, optionally
• the artifact would have to be redone and
  re-evaluated
• the evaluator or instructor would substitute an
  acceptable artifact, and the PLA process
  continues
• The evaluation/grade could be disputed

32
Full Collaboration

• Groups for
• Problems, solutions, evaluation, dispute
  arbitration
• Requires group process support
• Group roles leader, scheduler, etc.
• Process work on each activity together or
  separately, internal review
• Grading of individual group members
• Process Tools brainstorming, voting, etc.

33
What can go wrong

• Students are late students drop the course
• Entries posted in wrong place
• Inadequate critiques
• Good
• I agree with the other evaluator
• and of course, technical difficulties

34
PLA Environment Software

• Guide the process
• Form groups
• Assign problem solvers, evaluators, dispute
  arbitrators
• On-line templates to ensure full entries
• Guide people to post entries in correct place
• Incorporate group process tools
• Handle problems as much as possible
• Remind people who are late
• Reallocate who does what
• Based on a workflow management tool

35
Anonymity/Privacy Issues

• Should student entries be anonymous?
• Will students reveal their IDs?
• Is it fair to post critiques if not anonymous?
• Is it fair to post grades if not anonymous?
• Will anonymity work in small classes?

36
Issue Perceived Fairness

• Should students evaluate/grade peers?
• But they must evaluate others in the workplace
• Its the instructors job to evaluate and grade
• PLA is a (constructivist) learning technique
• Students have no training in evaluation
• Evaluation is a skill that must be learnt (and
  taught)
• Many evaluators inconsistent quality
• safeguards in the PLA process

37
Grading Issues

• Disputing high grades
• Award bonus points if students dispute (and
  justify with a critique) grades that are too high
• Encouraging honest grading
• For successful disputes, deduct points from
  evaluators

38
Grade Inflation

• Detailed grading guidelines for sub-criteria
• great 20 points
• very good 18 points
• good 14 points
• OK 10 points
• poor 6 points
• Student does good on 5 problems, grade 70
• U.S. students will protest vigorously
• Evaluators will hesitate to assign good
• Result pressure for highly skewed grading rubrics

39
Other Cross-Cultural Issues

• In some cultures
• Students are so competitive, they would only give
  failing grades to peers
• Students would not hurt peers feelings, and
  would only give good evaluations
• Some systems only have pass/fail, so numeric
  grades are mostly irrelevant

40
PLA Contributions

• Systematic technique to increase learning
• Constructivist approach, actively engaging
  students in the entire problem life-cycle
• Minimizes overhead for students and instructors
• Experimental evaluation
• Supporting software
• PLA liberates learning from its traditional
  instructor-controlled structure!