Quality Improvement Seminar: Academic Audit

1
Quality Improvement Seminar Academic Audit

• A Presentation Supporting the
• 2006 Year of the Department
• A Call to Engagement

2
Overview of the Year Of The Department (YOTD)

• Develop a culture of quality improvement at the
  department level
• Leverage application of technology in traditional
  and virtual classrooms
• Mobile computing
• Team design of VC and popular general education
  classes
• Focus on quality improvement in design,
  implementation, and assessment of curricula

3
Overview of the Year Of The Department (YOTD),
cont.

• Quality education processes include the following
  quality domains
• Desired learning outcomes
• Design of the curricula
• Design of teaching and learning processes for
  selected courses/programs
• Student learning assessment
• Results and feedback for quality assurance
• Academic curriculum audit track

4
Rationale Why Now?

• New mandatory KBOR Performance Agreement goal
• Outcome of the FHSU System Appraisal and Strategy
  Forum
• NCA mandate for assessment of student learning
  outcomes
• Specialized accreditation requirements

5
Expectations and Outcomes

• By June 2007, selected academic departments will
  have completed the following
• Revise affinity diagrams of all programs to
  reflect curriculum review
• Plan for implementing changes included in
  affinity diagrams, including assessment
  activities
• Final departmental/program academic audit report
  (integrated into Department Annual Report)

6
Expectations and Outcomes, cont.

• To accomplish the YOTD, departments must
• Educate faculty about the five quality processes
• Research and determine best practice with
  respect to curriculum and assessment
• Look objectively at curriculum and assessment and
  assessment data
• Make changes
• Plan tracking for quality improvement
• Report results of the process

7
Where Do We Start?

• Get the faculty commitment
• Gather the documents
• University/college documents (mission, KBOR
  Performance Agreements, catalog)
• Department documents (Current AFFINITY DIAGRAM,
  mission, DAR, curriculum guides, comparable
  programs)
• Make time
• Assign, delegate, and set deadlines
• Establish milestones

8
Audit Step 1 Learning Objectives

• Examine learning objectives of the major program
  of study and specific learning objectives of each
  course
• What should the students in the program know and
  be able to do?
• How would the students educational experiences
  in the program contribute to their capacities as
  citizens, their employment success and their
  quality of life?
• Are the specified learning outcomes based on the
  needs of our enrolled students rather than the
  ideal students often envisioned by faculty?

9
Audit Step 1 Learning Objectives, cont.

• How do you create the best program level learning
  objectives?
• AFFINITY DIAGRAM
• Specialized accreditation?
• Comparable programs at peer institutions
• Honest input, consensus, spirit of improvement
  (akaThe Ghost of AQIP Present)
• Talk to experts (NCATE team, UASC members)
• Derived from mission? Mission derived from LOs?

10
Audit Step 2 Curriculum and Co-Curriculum

• Curriculum is the formal educational experience
  that students select for their program
  co-curricular activities extend to out-of-class
  experiences in student groups and volunteer work
• How does the curriculum relate to the programs
  learning outcomes?
• What is being taught in the courses, in what
  order, and from what perspective?

11
Audit Step 2 Curriculum and Co-Curriculum, cont.

• Does the curriculum have an organizing principle
  which produces coherence and consistency between
  the program learning outcomes, course goals and
  assessment results (e.g. globalization,
  geographic information systems, etc.)?
• Does the curriculum build cumulatively on
  students prior knowledge and capacity?
• To what extent does the co-curriculum, program
  activities and department student organizations
  support the curriculum and the programs learning
  objectives generally?

12
Audit Step 2 Curriculum and Co-Curriculum, cont.

• How do I create the best curriculum/co-curriculum
  for our majors?
• Consult best practice who is doing it well?
• Staying current requires us all to be aware of
  our changing discipline.
• What are common areas of concern by the students?
  Faculty? Start there.
• What is the history of the degree program?
• Get in touch with alumni and employers.
• Spend some time with soon-to-be graduates (focus
  group them).
• Are cognate areas aligning differently?

13
Audit Step 3 Teaching and Learning Methods

• Teaching methods, students learning styles, and
  common learning strategies must be addressed as
  outcomes to curriculum and co-curriculum delivery
• What teaching and learning methods are being used
  across the department? For example, what methods
  are employed for introducing students to new
  materials, for interpreting those materials and
  answering student questions, for stimulating
  student participation, and for providing feedback
  on each students work (e.g. course management
  techniques and best practices)?

14
Audit Step 3 Teaching and Learning Methods, cont.

• Is the program sensitive to generational and
  individual learning styles and approaches
  (especially important for distance learning
  courses)?
• Is learning interactive?
• Is technology being used, and if so, is it
  producing improved learning or at least better
  communication, productivity and work flow?

15
Audit Step 3 Teaching and Learning Methods, cont.

• How do we teach most effectively?
• Become learner focusedinvolve students
• Let the methods emerge from the curriculum,
  generally speaking
• What kind of learners are your students it
  makes a difference
• Being current helps here, too
• Adopt teaching methods that support assessment
• Investigate new methods that might recruit and
  retain students
• Study exit points (all students leave after
  StatsI wonder why?)

16
Audit Step 4 Student Learning Assessment

• Assessment of student learning is becoming the
  most important element in effective pedagogy.
  Without assessment, there can be no systematic
  improvement.
• What approaches and techniques are being used to
  assess the programs specified learning outcomes?
  
• Are they aligned with the outcomes and how often
  are they being used?
• Do they compare beginning and ending performance
  by students to ascertain added value?

17
Audit Step 4 Student Learning Assessment, cont.

• Is there a mix of direct and indirect
  (self-reported) approaches and key performance
  indicators (KPIs) for assessment of outcomes?
• Who is responsible for each or all of these
  assessment approaches on the affinity diagram?
  Individual faculty members? A department
  committee?
• Are the assessment results trending in a
  particular direction?
• How do the results inform quality improvement
  efforts?

18
Audit Step 4 Student Learning Assessment, cont.

• How do we assess students most productively
• Make a statement about how much assessment is
  enough to know your students have learned
• Next to none is not a good answer
• Adopt different forms of assessment at multiple
  access points
• Direct, indirect, embedded, satisfaction,
  qualitative, quantitative
• Pre-test / post-test design is still useful
• Gather examples of course/program assessment from
  colleagues at other institutions

19
Audit Step 5 Implementation and Quality Assurance

• Faculty improvement and implementation of revised
  curriculum are the final components of the
  academic audit
• What processes have been institutionalized to
  assure department faculty and the college dean
  that the designs/strategies for the curriculum,
  the teaching and learning methods, and the
  assessment of student learning is being
  implemented as intended?
• How can the department assure that the teaching
  of the learning outcomes is the target of robust
  evaluations?

20
Audit Step 5 Implementation and Quality
Assurance, cont.

• How does the department assure that there will be
  regular application of the previous four
  educational quality processes?
• How does the department focus on other aspects of
  continuous quality improvement in addition to the
  target program?

21
Audit Step 5 Implementation and Quality
Assurance, cont.

• What does an effective implementation look like?
• Does it all need to be done today? NO
• Adopt a QI perspective
• Know that all change may not yield results
  immediately
• Essential to have commitment of all faculty
  teaching the program
• Different interests, politics, adjuncts
• Chair serves as a critical piece in the
  implementation? Or does she?
• Report what worked, be good role models
• Tell your students what to expect, give them
  something to look forward to

22
Academic Audit Map
23
Reporting for YOTD Department Annual Review

• Identify and describe the relevance of the
  new/revised student learning outcomes the
  department developed for the target academic
  program.
• What processes has the department developed
  during the YOTD to continue/institutionalize a
  review of how students learn, how course
  consistency for multiple sections in the target
  program will be maintained and how assessment of
  learning outcomes will be conducted?

24
Reporting for YOTD Department Annual Review,
cont.

• Briefly describe how the department will continue
  the process of making Quality Job One a team
  effort after the YOTD is complete. (In other
  words, how will the department become/continue as
  a collegial learning organization focused on
  continual quality improvement?)

25
Reporting for YOTD Department Annual Review,
cont.

• What forms of evidence will you continue to
  collect and use in your ongoing quality
  improvement efforts? (e.g. student preparation
  information, generational learning styles,
  requirements for employment and citizenship,
  assessment of learning outcomes, and so on)

26
Reporting for YOTD Department Annual Review,
cont.

• A program should be considered a plan of
  learning with an organizing principle,
  intentional learning outcomes, and courses that
  drive those learning outcomes and assessment
  techniques at a minimum. Briefly describe how
  your academic audit generated a more coherent
  plan of learning for the target program.

27
Reporting for YOTD Department Annual Review,
cont.

• What best practices relevant to teaching in
  general and your target program in particular did
  you find during your YOTD research and which of
  those did you adopt to improve instructional
  delivery skills, instructional design or
  course/program management?

28
Reporting for YOTD Department Annual Review,
cont.

• Briefly describe how your department will make
  the pursuit of educational quality by individual
  faculty and the department as a whole part of the
  criteria for merit evaluation, tenure and
  promotion?
• National Center for Postsecondary Improvement
  (NCPI)

29
Resources for Academic Quality

• AQIP www.aqip.org
• QI at FHSU www.fhsu.edu/aqip/
• Academic Leadership Development www.fhsu.edu/alq/
• Academic audit
• www.unc.edu/depts/pubpol/auditprojectstatement.htm
  
• http//www.apsu.edu/senatef/TBR Acad
  Briefs/Academic Affairs Brief v2 n1.pdf
• http//quality.massey.ac.nz/download/Audit
  Overview.pdf

30
Quality Improvement Seminar Academic Audit

• A Presentation Supporting the
• 2006 Year of the Department
• A Call to Engagement