Analyzing CS Competencies using The SOLO Taxonomy

1
Analyzing CS Competenciesusing The SOLO Taxonomy
ITiCSE'09 Keynote
Claus Brabrand ((( brabrand_at_itu.dk ))) (((
http//www.itu.dk/people/brabrand/ ))) Associate
Professor, IT University of Copenhagen
Denmark
2
Outline

• 1) Introduction
• Constructive Alignment
• The SOLO Taxonomy
• 2) From Content to Competence
• Advocate a shift in perspective
• Elaborate The SOLO Taxonomy
• 3) Analyzing CS Competencies
• using The SOLO Taxonomy
• Compare CS vs NAT vs MAT

3
Introduction to

• Constructive Alignment SOLO Taxonomy

John Biggs popular and heavily cited book
Teaching for Quality Learning at
University - What the student does
Note 3rd Edition now available J.Biggs
C.Tang, 2009
Teaching Teaching
Understanding Understanding
19 min award-winning short-film on Constructive
Alignment (available on DVD in 7 languages,
epilogue by John Biggs)
4
Activation Exercise
T

• Discuss with your neighbour

What are the main messagesof the film (which
did YOU findparticularly relevant, if any)?
5
Outline

• 1) Introduction
• Constructive Alignment
• The SOLO Taxonomy
• 2) From Content to Competence
• Advocate a shift in perspective
• Elaborate The SOLO Taxonomy
• 3) Analyzing CS Competencies
• using The SOLO Taxonomy
• Compare CS vs NAT vs MAT

6
From Content to Competence

• My old course descriptions (Concurrency 2004)
• Given in terms of a 'content description'
• Essentially

• Goal is
• To understand
• deadlock
• interference
• synchronization
• ...

This is a bad idea for two reasons...!
7
Problem 1 !

• Problem with 'content' as goals !

analyze ... theorize ...
analyze systems explain causes
define deadlock describe solutions
agreement
Stud. C

• Goal is
• To understand
• deadlock
• interference
• synchronization
• ...

tacit knowledge from a research-based tradition
not known by student
Teacher
name solutions recite conditons
analyze systems explain causes
Stud. B
?
P.S. even if it were possible to agree, we know
that the exam will dictate the learning anyway.
Stud. A
Censor
8
Problem 2 !

• Problem with 'understanding' as goals !

• Goal is
• To understand
• deadlock
• interference
• synchronization
• ...

'concept of deadlock' ?!
?
The answer is simple
It cannot be measured !
9
Competence !

• 'Competence' as goals !

Competence knowledge capacity to act
upon it
Have the student do something and then "measure"
the product and/or process

• Objective !
• To learn how to
• analyze systems for...
• explain cause/effects...
• prove properties of...
• compare methods of...
• ...

Note 'understanding' is of course
pre-requisitional !
?
Note' inherently operational ( verbs)
'SOLO' Structure of the Observed Learning
Outcome
10
SOLO Advantages

• Advantages of The SOLO Taxonomy
• Linear hierarchical structure
• Aimed at evaluating student learning
• Converges on research (at SOLO 5)

ResearchProduction ofnew knowledge
11
SOLO (elaborated)
Note the list is non-exhaustive
QUALITATIVE
QUANTITATIVE
SOLO 2 uni-structural
SOLO 3 multi-structural
SOLO 4 relational
SOLO 5 extended abstract

• theorize
• generalize
• hypothesize
• predict
• judge
• reflect
• transfer theory (to new domain)

• analyze
• compare
• contrast
• integrate
• relate
• explain causes
• apply theory (to its domain)

• combine
• structure
• describe
• classify
• enumerate
• list
• do algorithm
• apply method

• define
• identify
• count
• name
• recite
• paraphrase
• follow (simple) instructions

12
Using SOLO in Practice

• Recommendations on course descriptions

1) Use 'standard formulation' a) puts
learning focus on the student b) competence
formulation "to be able to"

• Intended Learning Outcomes Algorithms 101
• After the course, the students are expected to be
  able to
• identify and formulate algorithmic problems
  
• classify and compare algorithms
• construct and analyze algorithms using
  standard paradigms
• implement algorithms for simple problems.

4) Avoid 'understanding-goals' "To
understand X", "Be familiar with Y",
"Have a notion of Z", ...!
V
N
V
N
V
V
V
V
N
V
N
3) Use 'Verb Noun' formulation What the
student is expected to do with a given
matter
2) List sub-goals as 'bullets' Clearer than
text
N
V
13
Activation Exercise
T
Which do you predict are key CS competences ?
Concurrency analyze systems compare models
14
Outline

• 1) Introduction
• Constructive Alignment
• The SOLO Taxonomy
• 2) From Content to Competence
• Advocate a shift in perspective
• Elaborate The SOLO Taxonomy
• 3) Analyzing CS Competencies
• using The SOLO Taxonomy
• Compare CS vs NAT vs MAT

Joint work with Bettina Dahl at Aarhus University
15
Grade Scales
Conversion (between EU countries)
7 steps
4 steps
4 steps
8 steps
8 steps
ECTS
10 steps
10 steps
SCALE
...
...
...
...
...
21 steps
21 steps
A, B, C, D, E, Fx, F
Grade Degree of realization of
course objectives!
All Universities Explicit ILO's The SOLO
Taxonomy!
16
Massive DATA set

• Unique Opportunity
• Systematically formulated ILO's for all courses
• Quantifiable (analyzable) via The SOLO Taxonomy

competencies
5,608
courses
734
21
institutes
universities
TWO
17
SOLO Mapping

• Mapped by
• B. Dahl C. Brabrand
• With help from
• 3 Educational research colleagues (medicine)
• J. Biggs C. Tang

18
Top 10 Competencies

• Top 10 Competencies

Natural Sciences
" " Physics,
Chemistry, Biology, Molecular Biology
19
Histogram of Top Competencies

• If we look closer (comparative visualization)...

More than 2x

• also
• program
• construct
• structure

More than 3x
More than 3x
MAT

NAT
NAT
MAT
CS
MAT
MAT
MAT
CS
CS
MAT
CS
CS 15 NAT 1.0 MAT 0.3
CS 4.5 NAT 4.4 MAT 40
CS 14 NAT 14 MAT 60
Legend
Computer Science
Natural Science
Mathematics
with apply
20
SOLO Distribution

• SOLO distribution
• The 15 "programming competences" (all at SOLO
  4)
• implement, program, design, construct,
  structure

? 15 ?
EX 3.7
EX 3.4
EX 3.1
Legend
SOLO 2
SOLO 3
SOLO 4
SOLO 5
21
Assumptions
Assumptions

• SOLO is an appropriate competence measure (we
  refer to J.Biggs K.F.Collis, 1982 )
• Context independence of SOLO mapping (for
  each competence we inspected several goals)
• Subject independence of SOLO mapping (we
  limit ourselves to a 'science context')
• Equal weight assumptions (Competences in a
  goal goals in a course have equal weight)
• Outcomes intended ? formulated ? achieved
  (we analyze formulated, but reason about
  achieved)

Biggs studies
approximation
approximation
approximation
implicational
22
Conclusions

• Most frequent CS Competences are
• describe (13), explain (10), apply method
  (9), implement (7), analyze (6),
• "Programming-related" skills
• 15 of CS-curriculum
• The "Essence of Math" is
• reproducing, formulating, proving, solving,
  argueing,(and applying)
• SOLO-levels of subjects
• CS gtSOLO NAT gtSOLO MAT

15
23
Outline

• 1) Introduction
• Constructive Alignment
• The SOLO Taxonomy
• 2) From Content to Competence
• Advocate a shift in perspective
• Elaborate The SOLO Taxonomy
• 3) Analyzing CS Competencies
• using The SOLO Taxonomy
• Compare CS vs NAT vs MAT

24
Keynote Points

• Constructive Alignment
• addresses many teaching / learning problems
  e.g.
• Esp. student motivational issues (learning
  incentives)
• ...and student performance issues (learning
  support)
• The SOLO Taxonomy
• is good for reasoning about competencies
• Esp. for designing courses and curricula
• DATA
• Study, analyze, and reflect on teaching /
  learning
• using (objective) DATA!

25
Questions...
Cognitive processes
Course descriptions
My research and teaching
"understanding" content ? competence
Association new old
The SOLO Taxonomy
'TLA' Teaching / Learning Activities
Teacher models levels 1 - 2 - 3
The Short-Film
Susan Robert
The Book
?
Student activation
Tips'n'Tricks
CS v. NAT v. MAT
recite generalize
15 programming
Students at University
"What is good teaching?"
Constructive Alignment
John Biggs
Top 10 Competences
26
Thank You!
Film's homepage
((( http//www.daimi.au.dk/brabrand/short-film/
)))
27
Related References

• Teaching for Quality Learning at University
  (what the student does)John Biggs Catherine
  TangSociety for Research into Higher Education,
  2007. McGraw-Hill.
• Evaluating the Quality of Learning The SOLO
  TaxonomyJohn Biggs Kevin F. CollisLondon
  Academic Press, 1982
• Teaching Teaching Understanding
  UnderstandingClaus Brabrand Jacob Andersen19
  minute award-winning short-film (DVD)Aarhus
  University Press, Aarhus University, 2006
• Using the SOLO Taxonomy to Analyze Competence
  Progression of University Science
  CurriculaClaus Brabrand Bettina DahlHigher
  Education, 2009
• "Constructive Alignment The SOLO Taxonomy a
  Comparative Study of University Competencies in
  Computer Science vs. Mathematics"Claus Brabrand
  Bettina DahlCRPIT, Vol. 88, ACS 3-17, R.
  Lister Simon, Eds., 2007

28
Implementing Alignment

• Alignment Implementation Process

1) Think carefully about overall goal of
course (what are the stud. to learn?)
2) Operationalize these goals and formulate
them as SOLO intended learning outcomes
alignment
learning incentive
learning support
3) Choose carefully the form(s) of assessment
( intended learning outcomes)
4) Choose carefully the form(s) of teaching
( intended learning outcomes)
Think of teaching activities as training for
exam
29
SOLO Progression

• SOLO Progression
• Computer Science vs. Mathematics vs.

30
Conclusion (Progression)

• What have we really shown?!?

A) SOLO has "proved" that progression exists in
curricula (since we "believe" in SOLO as a
measure)
xor
B) SOLO has "been proven" to be a good tool for
analyzing competence progression (since
we "believe" in the existence of progression)
31
Progression Assumptions
Extra assumptions wrt. Progression

• Numeric quantification of SOLO
  assumption (i.e., numeric step from 2-3 is
  comparable to 3-4 and 4-5)
• Progression manifests itself as competences
  assumption (i.e., in 'verb'-, not
  'noun'-dimension)

32
SOLO Calculation Method

• Calculation Example (for a course)
• "SOLO average"
• (23)/2 (34)/2 (44)/2 4 / 4
  3.50
• "SOLO distribution"

• identify (2) and formulate (3) algorithmic
  problems
• classify (3) and compare (4) algorithms
• construct (4) and analyze (4) algorithms
  using standard paradigms
• implement (4) algorithms for simple problems.

"double weight averaging"
33
Neighbour Discussion
T
Discuss with neighbour "does this make sense
?!?" (content ? competence)
E.g. ("Learning about programming" vs.
"Learning to program" )
34
Activation Exercise III
T

• Discuss with your neighbour

Discuss what you predict wewould find in the
DATA set ?

• Questions
• a) most frequent CS competences?
• b) percentage of "programming-related"
  competences?
• c) CS v. NAT v. MAT (wrt. SOLO levels)?

35
Post-It exercise
T
Write down 1-2 key competences (i.e.,
verbs) (for your course)
Concurrency analyze systems for deadlock
compare models wrt. behavior
36
Tips'n'Tricks (activation)

• Neighbour discussions

• Post-It exercise

• focus zoom in
• anonymous (!)
• swap'able
• everyone will engage
• empathetic control
• shared knowledge pool

• more questions (students dare ask them)
• better questions (students had a chance
  to discuss)

Phil Race
1-2 min timeout

• Frequent breaks

• Form variation

pulse reader measurements
lecturing blended with in-class activation
exercises
37
Tips'n'Tricks (cont'd)

• Use many examples(build on student
  pre-knowledge)

• Explicit structure

1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4.
wwwwwww
1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4.
wwwwwww
1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4.
wwwwwww
1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4.
wwwwwww
?

• self evident to you teacher
• not to a learner student
• (esp. during learning process)

• "Less-is-more"

• Student 'recap' at end

• analyze
• compare
• relate

common deadlock, uncommon deadlock,
A-synchronization, B-synchronization, hand-shake,
multi-party synchronization, multi-party
hand-shake, binary semaphores, generalized
semaphores, blocking semaphores, recursive locks,
...
vs.
now
after 1 day
after 1 week
after 2 weeks
after 3 weeks
Emphasize depth over breadth (coverage)
38
Now, please "3-minute recap"

• Please spend 3' on thinking about and writing
  down the most important points from the talk
  now!

Immediately
After 1 day
After 1 week
After 2 weeks
After 3 weeks
39
Problematic Courses

• E.g. course Databases (at RUC/Roskilde)
• Note almost entirely non-operational(!)
• i.e. measure how?!

• obtain knowledge about the structure of
  database systems
• be familiar with design of databases by use of
  special notations like E/R and analysis
  through normalization
• get an overview of the most important database
  models and a detailed knowledge about the
  most important model - the relational model
  as well as the language SQL
• get an overview of database indexing and query
  processing
• obtain knowledge about application programming
  for DB systems.

Familiar with ?!
40
BONUS SLIDES
41
Based on John Biggs' Theories

• 2nd edition
• (3rd edition expected this fall)

"Teaching for Quality Learning at University",
John Biggs
42
UNALIGNED COURSE
?
Teachers intention
Students activity

• e.g.
• explain
• relate
• prove
• apply

"Dealing with the test"
Exams assessment

• e.g.
• memorize
• describe

• e.g.
• memorize
• describe

43
ALIGNED COURSE
?
Teachers intention
Students activity

• e.g.
• explain
• relate
• prove
• apply

• e.g.
• explain
• relate
• prove
• apply

• e.g.
• explain
• relate
• prove
• apply

Exams assessment

• e.g.
• explain
• relate
• prove
• apply

• e.g.
• explain
• relate
• prove
• apply

44
Top 10 Competencies

• Top 10 Competencies

Natural Sciences
" " Physics,
Chemistry, Biology, Molecular Biology
45
SOLO (elaborated)
Note the list is non-exhaustive
QUALITATIVE
QUANTITATIVE
SOLO 2 uni-structural
SOLO 3 multi-structural
SOLO 4 relational
SOLO 5 extended abstract

• theorize
• generalize
• hypothesize
• predict
• judge
• reflect
• transfer theory (to new domain)

• analyze
• compare
• contrast
• integrate
• relate
• explain causes
• apply theory (to its domain)

• combine
• structure
• describe
• classify
• enumerate
• list
• do algorithm
• apply method

• define
• identify
• count
• name
• recite
• paraphrase
• follow (simple) instructions

46
Exercise
T

• Buzz Session

1) Discuss w/ neighbour 2) Write it on a
Post-It 3) Swap Post-Its
"which film messages did you find
particularly relevant?"
Just Keep Swapping
47
Student Motivation

• Susan (intrinsic motivation)
• - wants to learn !
• Robert (extrinsic motivation)
• - to pass exams !

48
Constructivism

• Transmission is Dead
• (lectures )
• Knowledge is Actively Constructed !

!
active teacher passive students
risk
49
SOLO Taxonomy

• Hierarchy for Competences
• Deep learning (not surface) !

5 generalize, theorize, predict, 4 explain,
analyze, compare, 3 describe, combine,
classify, 2 recite, identify, calculate,
50
Stud Learning Focus

• Focus on Student Learning !
• (instead of what teacher does
• labelling students good/bad)
• Student activitation ? learning

51
Alignment

• Make explicit ILOs
• (Intended Learning Outcomes)
• (and tell this to students)

Exam ILOs Teaching
52
The Role of the Exam

• Alignment
• A theory of planning (over the course of a
  course)
• A theory of motivation (and incentive)
• The exam as a...

"The exam does not come after, but before the
course!"
"Necessary evil"
application of alignment
Motivational and learning-guiding pedagogical
tool for the teacher(!)
53
Di-Transitive Verbs

• Mono-Transitive verbs
• Di-Transitive verbs

54
Data Set (XML and XQuery)
( Extracts all mathematics courses w/
maximum 1 goal and 2 competencies ) xquery
version "1.0" ltresultgt for course in
fndoc("data-au.xml") //institute_at_name
"MAT"//course let goals course/goal
where (fncount(goals) le 1) and
(fncount(goals/competence) eq 2) order by
course/_at_name return course lt/resultgt
XQuery
Data set http//www.itu.dk/people/brabrand/sol
o.xml http//www.itu.dk/people/brabrand/data-a
u.xml http//www.itu.dk/people/brabrand/data-s
du.xml
XML
55
The BLOOM Taxonomy (1956)

• The BLOOM Taxonomy

Analysis
Evaluation
Synthesis
SOLO 45
Qualitative
Application
Comprehension
Quantitative
SOLO 23
Knowledge

really intended to guide the selection of
items for a test rather than to evaluate the
quality of a students response to a particular
item -- (Biggs Collis, 1982)
56
CS vs Math Distributions

• Computer Science
• Mathematics

?(? 3.68, ? 0.39)
?(? 3.06, ? 0.24)