RMIT Open Program

1

• RMIT Open Program
• SET Program Annual Review Using the Student
  Outcomes Survey (SOS) Data
• http//www2.rmit.edu.au/departments/planning/ircu/
  ircu_home.php
• Developed by A J Calderon
• Angel J Calderon
• Marielle Vanderydt
• Fiona Wahr
• July 2006

2
Learning Outcomes

• Confirm understandings of
• The what, when, how and why of the SOS.
• How the data drives /is used in institutional
  rankings.
• Impact of RMIT program rankings on community
  perceptions.
• Understand the relationship between SOS and the
  Program Annual Review process.
• Ability to assess program risks via SOS data.
• Consider and develop program improvement
  strategies based on the risk assessment.

3
Presenters

• Angel Calderon
• Head Institutional Research, Policy and Planning
• ext 59708.
• Marielle Vanderydt
• Manager, PQA SET Porfolio
• marielle.vanderydt_at_rmit.edu.au
• Fiona Wahr
• Manager, TL Strategic Projects SET Portfolio
• fiona.wahr_at_rmit.edu.au

4
Key Points

• SOS is a snapshot of domestic graduates /
  module completers perceptions of their TAFE
  experience.
• Same instrument administered nationally six
  months after completion of training.
• The National Vocational Centre for Educational
  Research (NCVER) administers SOS nationally.
• Instrument has changed over time.
• There are methodological issues with instrument
  improvements are required.

5
Key Points - 2

• SOS gathers information about training and
  employment activities before, during and after
  completion of training.
• Findings have been consistent over time.
• RMIT has consistently performed poorly compared
  to the national average.
• SOS data is used by OTTE in assessing
  institutional performance in delivering training
  and meeting students expectations.
• SOS data is one set of data used by OTTE in
  determining state training priorities.

6
RMIT Performance

• Good Teaching Scale and Overall Satisfaction
  SOS Poor performance compared to Other TAFE /
  State.
• Improving but still lagging other providers in
  the state
• Good Teaching and Overall Satisfaction CEQ
• Poor performance compared to Other Universities
  / National.

7
Link with RMITs Strategic Plan Overall
Satisfaction Index (OSI)

• RMITs Strategic Plan Goal, by 2010
• 5 above National/State Average OSI.
• Measured from the Course Experience Questionnaire
  (CEQ), Student Outcomes Survey (SOS) and
  Postgraduate Research Experience Questionnaire
  (PREQ).

8
1st Group Discussion

• Table Discussion
• How is the SOS used in your School or work area?
  10 mins.
• Group Discussion
• Share your findings with the whole group 20
  mins.

9
SOS at RMIT

• NCVER draws sample from data provided to OTTE by
  Stats and Reporting.
• NCVER contacts graduates directly.
• RMIT pays an additional fee to NCVER in order to
  boost RMIT sample of domestic graduates from
  previous year.
• Planning Research is responsible for the
  analysis and reporting of the SOS.

10
RMIT Response Rates

• Traditionally, the national response rate has
  been around 45 nationally.
• Conversely, RMITs response rate has been below
  national average.
• Only domestic graduates are surveyed.
• Fewer follow ups are conducted by NCVER so
  response rate has declined in comparison to
  previous years.

11
SOS Items

• Before starting training, particularly asks
  information about training and employment
  activities over the six months prior to
  commencing their program.
• During training, asks information regarding
  program characteristics, sources of income,
  reasons for doing the training, and employment
  activities.
• At the end of May in the year after completion of
  training, obtaining information about employment
  status and opinions of the training.

12
SOS Scales Good Teaching Items

• My instructors had a thorough knowledge of the
  subject content.
• My instructors provided opportunities to ask
  questions.
• My instructors treated me with respect.
• My instructors understood my learning needs.
• My instructors communicated the subject content
  effectively.
• My instructors made the subject as interesting as
  possible.

13
SOS Scales Generic Skills Items

• My training developed my problem solving skills.
• My training sharpened my analytic skills.
• My training helped me develop my ability to work
  as a team member.
• My training improved my skills in written
  communication.
• My training helped me to develop the ability to
  plan my own work.
• As a result of my training, I feel more confident
  about tackling unfamiliar problems.

14
SOS Scales Overall Satisfaction Item

• Overall, I was satisfied with the quality of this
  training.
• Note Analysis in interpreting SOS data have been
  developed using similar methodology to that
  employed in interpreting CEQ data for HE.

15
Rankings and Graduate Surveys (1)

• TAFE prospective students are less aware of
  rankings and rating surveys.
• OTTE produces an assessment of TAFE providers but
  data is only available for institutional
  analysis.
• OTTE is encouraging institutions to make more
  direct use of SOS data for institutional
  improvement and for determining training
  priorities in key and selected industries.

16
What data is available at RMIT?

• Online tools and resources.
• Time Series available
• 2004 to 2005.
• Instrument has changed so no comparisons can be
  made for early years.
• Output options HTML, Excel and PDF
• Program / School / Portfolio level.
• Comparisons by Field of Education to State
  averages.

17
SOS Analysis Options

• Main Program Experience Items Itemised
• SOS Scales
• Good Teaching
• Generic Skills
• Assessment
• Overall Satisfaction
• Time Series
• Detailed
• For all RMIT programs with a response

18
Student Outcomes Web Page
19
SOS Screen
20
What does the data tell us?

• SOS Items and Scales
• Nature of responses
• Percent agreement
• Performance/trends over time
• Strengths, improvement areas
• Other relevant program data used to unpack the
  story.

21
2nd Group Activity

• In groups/pairs, analyse the program data in the
  example given to you. Focus on
• Good Teaching Scale
• Discuss key features and the issues indicated by
  the data.
• In groups/pairs, discuss possible TL
  improvements that could be made to the program.
• What lessons about the programs performance can
  be inferred from the data?

22
Program Annual Review

• Data gathering
• Data review for
• Risks
• Highlights
• Plan improvements
• Reporting
• Acting on the improvements
• Where does SOS apply?

23
Linking with the PAR

• SOS data is
• a critical data set for reviewing program
  performance
• particularly relevant to the Quality, Viability
  and Relevance section of the PARR template
• to be attached with the submitted PARR

24
Finale Reflection and Feedback
What is the most useful thing you have learned
today? Please complete the Feedback Sheet to
help us keep improving this session. Thankyou
for participating ?
25
SET Portfolio Resources and Support Available
Professor Mark Shortis, Dean, Academic
Development Ext 59628, mark.shortis_at_rmit.edu.au
Ms Marielle Vanderydt, Manager, PQA Ext 54293,
marielle.vanderydt_at_rmit.edu.au
26
Related Open Programs Developed by Policy and
Planning

• Understanding and Using the CEQ/GDS.
• CEQ Qualitative Comments.
• VTAC Analysis.
• Using Statistical Information for Decision
  Making.
• Industry and Government Priorities - Trends and
  Implications for Profile Development.
• Refer to http//www.rmit.edu.au/PC/OPENPROGRAM