Assessing LearningCentered Leadership

1

• Assessing Learning-Centered Leadership
• Andrew C. Porter
• University of Pennsylvania
• Joseph Murphy, Ellen Goldring, Stephen N.
  Elliott
• Vanderbilt University
• Spring 2008

2
Acknowledgments

• The VAL-ED instrument will be published and
  distributed by Discovery Education Assessment
  starting in July 2008.
• We are authors of the VAL-Ed, and while we have
  made every effort to be objective and data-based
  in my statements about this instrument in this
  presentation, readers should judge the facts and
  related information materials for themselves and
  make independent decisions regarding the use of
  the instrument.

3
Outline of Presentation

• Background on VAL-ED
• The VAL-ED instrument
• Psychometric properties
• Score reports
• VAL-ED and professional development

4

• Background on VAL-ED

5
The Case for Leadership Assessment

• Most school leadership improvement focuses on
  professional development, mentoring, licensing
  policies, and standards.
• Minimal attention has been paid to assessment,
  feedback, and subsequent action.
• Leadership assessment and feedback is an
  important missing link to improving and
  strengthening school leadership.

6
Learning-Centered Leadership

• Leaders should be assessed on leadership
  behaviors associated with student learning.
• Learning-centered leadership is leadership for
  student performance.
• Learning-centered leadership is the framework for
  our leadership assessment system.

7
Our Conceptual Model
8
Leadership Behavior Framework
9
Definitions of Core Components

• High Standards for Student Learning There are
  individual, team,
• and school goals for rigorous student academic
  and social learning.
• Rigorous Curriculum (content) There is ambitious
  academic content provided to all students in core
  academic subjects.
• Quality Instruction (pedagogy) There are
  effective instructional practices that maximize
  student academic and social learning.
• Culture of Learning Professional Behavior
  There are integrated communities of professional
  practice in the service of student academic and
  social learning. There is a healthy school
  environment in which student learning is the
  central focus.
• Connections to External Communities There are
  linkages to family and/or other people and
  institutions in the community that advance
  academic and social learning.
• Performance Accountability Leadership holds
  itself and others responsible for realizing high
  standards of performance for student academic and
  social learning. There is individual and
  collective responsibility among the professional
  staff and students.

10
Definitions of Key Processes

• PlanningArticulate shared direction and coherent
  policies, practices, and procedures for realizing
  high standards of student performance.
• ImplementingEngage people, ideas, and resources
  to put into practice the activities necessary to
  realize high standards for student performance.
• SupportingCreate enabling conditions secure and
  use the financial, political, technological, and
  human resources necessary to promote academic and
  social learning.
• AdvocatingPromotes the diverse needs of students
  within and beyond the school.
• CommunicatingDevelop, utilize, and maintain
  systems of exchange among members of the school
  and with its external communities.
• MonitoringSystematically collect and analyze
  data to make judgments that guide decisions and
  actions for continuous improvement.

11
The Development of VAL-ED

• The development of VAL-ED has been supported by a
  3-year grant from The Wallace Foundation.
• Three phases of our work
• Phase 1 Leadership conceptualization and
  assessment system development
• Phase 2 Field testing the behavior rating scale
  and exploring its properties
• Phase 3 Dissemination of results and products

12

• The VAL-ED Instrument

13
The VAL-ED Instrument

• The instrument consists of 72 items defining six
  core component subscales and six key process
  subscales.
• Principal, Teachers, Supervisor provide a
  360-degree, evidenced-based assessment of
  leadership behaviors.
• Respondents rate effectiveness of 72 behaviors on
  scale 1Ineffective to 5Outstandingly effective.
• Each respondent rates the principals
  effectiveness after indicating the sources of
  evidence on which the effectiveness is rated.
• Two parallel forms of the assessment facilitate
  measuring growth over time.
• The instrument will be available in both paper
  and online versions.

14
Purpose Uses

• The VAL-ED can be used as part of a comprehensive
  assessment of the effectiveness of a leader's
  behaviors.
• The VAL-ED reports principal performance through
• Norm-referenced scores and
• Criterion-reference scores.
• VAL-ED can be used annually or more frequently
  to
• Facilitate a data-based performance evaluation,
• Measure performance growth, and
• Guide professional development.

15
Implementation

• Identify respondents and invite participation.
• Discuss use of results confidentiality.
• Decide paper or online version.
• Time and Timing
• Average respondent requires 20 to 25 minutes.
• Schedule completion after respondents have had a
  reasonable time to observe/experience the
  leaders work and its effects on the school.
• Designate person(s) to manage collection and
  submission of response forms, if paper version
  used.
• Ensure teacher confidentiality.

16
Directions for Completing Rating Scale
17
An Example Set of Responses
18

• Psychometric Properties

19
Assessing Learning-Centered LeadershipThe
VAL-ED vision

• A leadership assessment system that has the
  following properties
• Works well in a variety of settings and
  circumstances,
• Is construct valid,
• Is reliable,
• If feasible for widespread use,
• Provides accurate and useful reporting of
  results.
• Is unbiased,
• Yields a diagnostic profile for summative and
  formative purposes.
• Can be used to measure progress over time in the
  development of leadership, and
• Predicts important outcomes.

20
Psychometric Evidence

• Item and response scale development
• Based on review of learning-centered leadership
  literature and alignment to ISLLC standards.
• Critiqued by education leaders and leadership
  researchers.
• Item sorting study
• Established content validity by asking education
  leaders to sort the items into 36 cells.
• Cognitive interviews of paper/pencil version
• Two rounds of cognitive interviews in three
  districts each.
• Three respondents evaluated the format and items.
• Nine-school pilot test
• Estimated reliability of each of 12 scales.
• Established construct validity through factor
  analysis.
• Established face validity through questions to
  respondents.

21
Psychometric Evidence

• Cognitive interviews of online instrument
  including revisions based on 9-school pilot
• Bias review
• Submitted to urban districts to evaluate
  language.
• 11-school pilot test
• Confirmed changes made after 9-school pilot test.
• 300-school field test
• Conducting differential item functioning to
  determine biases.
• Establishing norms.
• Setting performance standards.
• Proficiency standard setting
• Using bookmark approach with 24 education leaders.

22
Cognitive InterviewsValidity

• First round
• Sources of evidence cumbersome
• Item stem lost
• Instructions wordy
• Modifications
• Instructions bulleted
• Stem included in each item
• Not done added
• Conclusions
• VAL-ED captured all the relevant leadership
  behaviors
• Response scale was clear
• Respondents able to complete task on their own
• Cognitive interviews of online prototype

23
9-School Pilot StudyValidity Reliability

• Methods
• Confirmatory factor analysis
• High Parsimonious Goodness of Fit Index (.93-.96)
• First, second, and third-order factor loadings
  salient
• Responses to final survey questions
• Understood items, sources of evidence
• Teacher and principal
• effectiveness ratings were
• correlated r .47
• Cronbachs Alpha above
• .92 for 108-item-form scales
• Scores high (generally gt4
• on the five-point effectiveness
• scale)

24
11-School Pilot StudyValidity Reliability

• Methods
• High agreement between schools
• r .79 for teacher-principal,.51 for
  principal-supervisor, .68 for
• supervisor-teacher
• Mean effectiveness
• ratings lower, more
• variable
• Alpha above .89 for
• all 72-item-form
• scales

25
Respondent Feedback
26

• Score Reports

27
Interpretation of Rating Scale Results

• Descriptive Analysis
• Total Score
• Core Components Subscale Scores
• Key Process Subscale Scores
• Norm-Referenced Profiles
• Principal
• Teacher
• Supervisor
• Total respondent composite
• Criterion-Referenced Profiles
• Distinguished
• Proficient
• Basic
• Below basic

28
VAL-ED Results Performance Descriptors

• Distinguished
• A distinguished leader exhibits leadership
  behaviors of core components and key processes at
  levels of effectiveness that over time are
  virtually certain to influence teachers to bring
  the school to a point that results in strong
  value-added to student achievement and social
  learning for all students.
• Proficient
• A proficient leader exhibits leadership behaviors
  of core components and key processes at levels of
  effectiveness that over time are likely to
  influence teachers to bring the school to a point
  that results in acceptable value-added to student
  achievement and social learning for all students.
  
• Basic
• A leader at the basic level of proficiency
  exhibits leadership behaviors of core components
  and key processes at levels of effectiveness that
  over time are likely to influence teachers to
  bring the school to a point that results in
  acceptable value-added to student achievement and
  social learning for some sub-groups of students,
  but not all.
• Below basic
• A leader at the below basic level of proficiency
  exhibits leadership behaviors of core components
  and key processes at levels of effectiveness that
  over time are unlikely to influence teachers to
  bring the school to a point that results in
  acceptable value-added to student achievement and
  social learning for students.

29
Aggregated Effectiveness Ratings
30
Comparisons Across Respondent Groups
31
Comparisons Across Respondent Groups
32

• VAL-ED and Professional Development

33
Cell-by-Cell Feedback
34
VAL-ED and Professional Growth

• Cell-by-cell feedback highlights up to 6
  potential areas of growth.
• Behaviors from these 6 domains are listed.
• Areas of growth provide principals with
  information about key targets for professional
  development.

35
Leadership Behaviors for Possible Improvement
Example of a potential area of growth
36
Supporting Research Publications

• Goldring, E., Porter, A.C., Murphy, J., Elliott,
  S.N., Cravens, X. (2007, March). Assessing
  learning-centered leadership Connections to
  research, professional standards, and current
  practice. New York, N.Y. Wallace Foundation.
• Murphy, J., Elliott, S.N., Goldring, E.,
  Porter, A.C. (2007). Leadership for learning A
  research-based model and taxonomy of behaviors.
  School Leadership Management, 27 (2), 179-201.
• Murphy, J., Elliott, S.N., Goldring, E.,
  Porter, A.C. (in press). Leaders for productive
  schools. In M. Brundrett M. Crawford (Eds.),
  Developing school leaders An international
  perspective, London Routledge.
• Murphy, J., Elliott, S.N., Goldring, E.B.,
  Porter, A.C. (2006). Learning-centered
  leadership A conceptual foundation. New York,
  NY Wallace Foundation.
• Porter, A.C., Goldring, E.B., Murphy, J.,
  Elliott, S.N., Cravens, X. (2006). A framework
  for the assessment of learning-centered
  leadership. New York, NY Wallace Foundation.
• Murphy, J.F., Goldring, E.B., Cravens, X.C.,
  Elliott, S.N., Porter, A.C. (2007, August). The
  Vanderbilt Assessment of Leadership in Education
  Measuring Learning-Centered Leadership. Journal
  of East China Normal University.
• These and other publications are all available
  for download at http//www.vanderbilt.edu/lsi/vale
  d/featured.html.

37
To Learn More

• Visit http//www.val-ed.com
• Contact Discovery Education/ThinkLink
  Assessment 2416 21st Avenue, South, Suite 300
  Nashville, TN 37212 Jackie Shrago (o)
  615-324-6091 (e) Jackie_Shrago_at_discovery.com
  or William Dycus (o) 615-324-6090 (e)
  William_Dycus_at_discovery.com