Reporting School and District Performance: Issues, Tools and Solutions

1
Reporting School and District PerformanceIssues
, Tools and Solutions
Gregory H. Leazer
UCLA Graduate School of Education Information
Studies
1999 CRESST National Conference
2
Overview

• Personal Introduction
• Information System Design
• School Report Cards
• Whats Being Done Where
• Existing Report Formats
• Issues in Reporting School Results
• CRESST Approach Example

3
My background

• Information Systems, especially Information
  Retrieval
• I teach in the Department of Information Studies

4
Information System Design

• Provide evidence for informed decision-making
• Preserve and disseminate human knowledge and
  understanding

5
A Critical Element The User Interface

• When we reason about quantitative evidence,
  certain methods for displaying and analyzing data
  are better than others. Superior methods are
  more likely to produce truthful, credible and
  precise findings. (p. 27).
• Source Edward Tufte. Visual Explanations
  Images and Quantities, Evidence and Narrative.
  Graphics Press, 1997. Visual and statistical
  thinking displays of evidence for making
  decisions.

6
A case study Challenger launch

• Lack of legend explaining charts
• Chartjunkvisual clutter that draws attention
  away from the illustrative data
• lack of clarity depicting cause and effect
• poor positioning and ordering of data (scale
  problems)

7
Good Information System Design

• Document the sources of the data
• Enforce appropriate comparisons
• Demonstrate the mechanisms of cause and effect
• Express those mechanisms quantitatively
• Recognize the multivariate nature of analytic
  problems
• Provide evidence for alternative explanations.

8
Bottom Line

• Convey an understanding the dynamics of the
  variables and processes of observed phenomena.
• A deeper understanding of the information and
  data.
• Source Tufte

9
Applying Tufte to School Reports

• Scientific reasoning where there is a smaller set
  of variables, and the dynamics, while complex,
  are more deterministic.
• The identification of variables, articulated as
  part of the scientific theory, is a critical
  phase of the results.
• In social science research, for example about
  educational institutions, research is often in a
  more exploratory phase, with many variables, and
  a less clear connection between cause and effect.

10
Why School Report Cards?

• To Inform the Public
• Accountability
• Informed Consumers of Public Education
• To Inform the Educators
• Feedback for Improvement
• Monitor Progress Toward Goals
• Identify Schools for Targeted Intervention
• Opportunity for Schools to Highlight
  Accomplishments
• Identify Poorly Performing Schools

11
States Producing Annual School Report Cards (N36)
Source Education Week
12
States Where Report Cards Are Available on the
Web (N26)
Source Education Week
13
States Requiring School Report Cards Be Sent Home
(N13)
Source Education Week
14
Have you ever seen a School Report Card in your
community?
15
Whats Being PresentedStudent Achievement
Source Education Week
16
What Do People Want to See?

• Safety and Teacher Qualifications
• rated highest by Parents, Taxpayers, and
  Educators
• Class Size, Graduation Dropout Rates
• Student Performance Data
• Test scores are important but not the only thing
  of interest
• Student Demographics (Ethnicity and LEP)
• considered least important

17
Comparison Information

• National Averages
• State Averages
• District Averages
• Similar Schools (e.g., LAUSD Cluster)
• Established Standard
• School Goals
• Prior School Performance

18
States Providing Comparison Information on
Student Test Scores
19
Combining Data Into a Summary Score

• A Difficult Technical Issue
• How to Weight Data Elements
• Most States Have Avoided Doing It
• Only 12 States Assign Overall Ratings to All
  Schools
• Another 7 States Identify low performing
  Schools only
• Other States and Districts are Considering It
• Different Preferences (Scores vs. Grades)

20
Current Report Card Formats

• Range from a Few to Several Pages
• Data Usually Presented in Tables with Explanatory
  Text
• Usually Some Comparison Information
• Relative to some Average (District, State,
  National)
• Relative to some Standard
• Some Previous Year Information
• Few Graphics

21
School Report Cards Examples
22
School Report Cards Examples
23
School Report Cards Examples
24
Commercial Efforts at School Report Cards

• School Reform Advocates
• Real Estate Agencies
• Relocation Specialists
• Newspapers
• Weaknesses
• Often Charge per Report
• Limited Data Elements

25
Issues to Consider

• Define the Audience
• Educators and Public Dont Always Agree on What
  is Important
• Selection of Indicators
• Format Presentation
• Interpretability
• Delivery Availability

• Data Availability Quality
• Relevant Comparisons
• Illustrates Relationships among Indicators
• Credibility of Messenger

26
Perceived Credibility of Sources
Source A-Plus Communications
27
Potential Indicator Elements

• Student Demographics
• School Characteristics
• Teacher Characteristics
• Funding
• Academic Engagement

• Stability/Mobility
• Professional Development
• Parent Involvement
• Safety/Security
• Student Achievement
• Qualitative Information

28
Categories of Indicators

• Inputs
• Processes
• Outputs
• Environment or Context

29
Issues When Evaluating Indicator Sets

• Focus on Measures Under the Schools Control
• Keep in Mind the Purpose of the Indicator Set
• No Single Indicator or Set of Measures Can Serve
  All Purposes

30
Los Angeles Example

• Collaborative Effort
• Mayor, District, CRESST/UCLA
• Complements Existing LAUSD School Accountability
  Report Cards
• Single-page, Graphical Representation for Each
  School
• Additional Detail and Data Available
• Element Description and Information on How to
  Interpret the Report Provided

31
Research Questions (1)

• Can we develop effective techniques for the
  dissemination of results from educational
  research, including especially assessments of
  school and district performance?

32
Research Questions (2)

• Which indicators are the most appropriate for
  evaluating schools, districts, educational
  programs?
• Can we develop effective methods for the
  representation of educational indicators?

33
Research Questions (3)

• What are the appropriate uses of school
  performance data? Can we develop current and
  comprehensive information services to support
  these uses?
• Can we determine whether the dissemination of
  indicator data will result in improved policy
  decisions, more informed public debate, improved
  educational programs?

34
REPORT CARD OBJECT TYPES
School Level Percentages District or Cluster
Comparisons School Goals
Counted Data Number of instances
Standardized Test Assessment Scaled
Data National/District Comparisons Percentile
Rank Reporting
35
(No Transcript)
36
Method of Delivery

• Web-based
• more flexibility
• different comparisons
• greater detail (links to explanations, more data,
  etc.)
• Phonebook style compilation
• renewable
• short life span

37
Public Awareness Availability

• Will Be Widely distributed
• Public locations (e.g., libraries, community
  centers, etc.)
• Every Public School in LAUSD
• Possible Alliance with Local Print Media
• Descriptive Article
• Web-site hosting or linking

38
Future Directions

• Improving the Design of the Reports
• Software Development
• Project Evaluation

39
Improving the Report Design

• Multiple Years Representations
• Adequate Yearly Progress
• Progress Toward Goals
• Combining Data
• School Comparisons
• Relationships among Variables

40
Software Development

• Integration into QSP Software
• Authorable Stand-alone Software
• Provided to Schools Districts
• Determine their own Indicators
• Create their own layout

41
Project Evaluation

• Comprehension of data an evaluation of the user
  interface
• A school comparisons test