What Are Educational Indicators and Indicator Systems?
Richard J. Shavelson and Others

Source: Shavelson, R. & others. (1991). What are educational indicators and indicator systems? Washington, DC: ERIC Clearinghouse on Tests, Measurement, and Evaluation. [ED338701]

Return to: | Readings in Educational Psychology | Educational Psychology Interactive |

Educational indicator systems serve similar purposes to indicator systems that are used to monitor the economy, the criminal justice system, or other social systems. Statistical indicators are used to monitor complex conditions that we would probably judge imprecisely or miss altogether in day-to-day observations. Governments recognize the value of statistics that provide current information, analyze trends, and forecast impending changes. Consequently, it is not surprising that policymakers and researchers are seeking better statistical indicators of education.

The overriding purpose of indicators is to characterize the nature of a system through its components--how they are related and how they change over time. This information can then be used to judge progress toward some goal or standard, against some past benchmark, or by comparison with data from some other institution or country.

WHAT IS AN INDICATOR?

The term indicator was defined above as a statistic. After reviewing the literature on social indicators, Jaeger (1978) concluded that indicators are "anything but clear and consistent. Review of a dozen definitions has produced much that is contradictory and little that is concise and illuminating" (p. 285). He recommended that: all variables that (1) represent the aggregate status or change in status of any group of persons, objects, institutions, or elements under study, and that (2) are essential to a report of status or change of status of the entities under study or to an understanding of the condition of the entities under study, should be termed indicators. "I would not require that reports of status or change in status be in quantitative form, for narrative is often a better aid to comprehension and understanding of phenomena than is a numeric report" (pp. 285-287).

Jaeger's recommendation to leave the definition of an indicator open and to determine the status of potential indicators on pragmatic rather than strict definitional grounds is a wise one.

An education system can be conceived as having underlying properties that are not directly or perfectly measurable. For example, we can talk about the quality of the teaching force but also recognize that there is no direct way to measure it. At best, several statistics can be combined into an indicator that gets at our notion of teacher quality. An indicator of teacher quality might be some aggregate of years of academic training in the discipline taught; possession (or lack of) a credential in the subject matter taught; measured subject-matter knowledge; measured pedagogical knowledge; measured ability to translate subject-matter knowledge into a form that communicates to students of a given age, background, and prior knowledge; and so on.

Education indicators are statistics that reflect important aspects of the education system, but not all statistics about education are indicators. Statistics qualify as indicators only if they serve as yardsticks. That is, they must tell a great deal about the entire system by reporting the condition of a few particularly significant features of it. For example, the number of students enrolled in schools is an important fact, but it does little to tell us how well the education system is functioning. However, data on the proportion of secondary students who have successfully completed advanced study in mathematics can provide considerable insight into the health of the system, and can be appropriately considered an indicator.

We propose the following working definition as a heuristic guide: An indicator is an individual or composite statistic that relates to a basic construct in education and is useful in a policy context.

WHAT IS AN INDICATOR SYSTEM?

Another central concept in the discussion of indicators is that of the indicator system. Whether indicators are single or composite statistics, a single indicator can rarely provide useful information about such a complex phenomenon as schooling. Indicator systems are usually designed to generate more and more accurate information about conditions. However, an indicator system is more than just a collection of indicator statistics. Ideally, a system of indicators measures distinct components of the system and also provides information about how the individual components work together to produce the overall effect. In other words, the whole of the information provided by a system of indicators is greater than the sum of its parts.

National indicators should be conceived of as something more comprehensive than a time series of educational outcomes (e.g., achievement, participation). Simply monitoring outcomes does not provide explanations for observed trends. For example, trends might be explained by demographic changes, by educational improvements, or by some combination of these. Moreover, education policy indirectly influences outcomes by actions such as increasing standards for teacher certification or for high school graduation. The direct effects of these policies will be reflected in changes in teachers' qualifications (e.g., an increase in teachers with bachelors' degrees in the disciplines they teach, not degrees in education), in better matches between teachers' subject-matter and pedagogical training and their teaching assignments, and in the number of academic courses students take in high school.

National indicators must represent, at least roughly, the important components of an educational system. In addition to monitoring outcomes, indicators should reflect the characteristics of students and communities served by schools, the financial and human resources (especially teachers) available to the schools, and other educational inputs. Moreover, they should reflect the adequacy of the curriculum and instruction received by students, the nature of the school as an organization in pursuit of educational excellence and equity, and other educational processes. Finally, indicators must be related to one another so that their relationships, and changes in these relationships, can be ascertained to suggest possible explanations for observed changes in outcomes.

WHAT ARE REASONABLE EXPECTATIONS FOR AN INDICATOR SYSTEM?

A good education indicator system is expected to provide accurate and precise information to illuminate the condition of education and contribute to its improvement. The information generated will be neither possible to grasp through casual observation nor generally available from other efforts to collect, report, and analyze data about schooling. Indicators are thus expected to assist policymakers as they formulate schooling goals and translate those goals into actions.

Whenever social indicators have been heralded as a stimulus for reform, their promise has quickly given way to realism. Promises of policy applications have been overly optimistic. Indicator systems were, for example, unable to provide detailed and accurate enough information for evaluating government programs. These events led to more realistic assessments of what indicators can and cannot do.

What Indicators Cannot Do.

The literature on social indicators appears to agree on what indicators cannot do

Set goals and priorities. The public establishes educational goals and priorities through its elected representatives. The information generated by an indicator system can inform those objectives, but it is just one factor among many in shaping decisions about policy preferences and priorities.

Evaluate programs. Social indicators cannot substitute for well-designed, in-depth evaluations of social programs. Indicators do not provide the level of rigor or detail necessary.

Develop a balance sheet. Social indicators lack the common referent available to economic indicators. Evoking an economic analogy and proposing a parallel development for social indicators is misleading because education cannot put each of its constructs on a common dollar metric as can be done, say, for Gross National Product (GNP). As Rivlin (1973, p. 419) pointed out, "No amount of disaggregation of inputs...will provide a basis for answering the how-are-we-doing question in the education sector. As long as cost is used as a proxy for value there is no way to compare inputs with outputs or to see whether a given amount of education is being produced with fewer resources." Rivlin also noted that because students help produce education, it is difficult to disentangle the quality of the output from the quality of student input.

What Indicators Can Do.

The expectations for social indicators are now quite modest: to describe and state problems more clearly, to signal new problems more quickly, to obtain clues about promising educational programs, and the like. The following statements illustrate the realistic tone currently taken by the social indicator movement:

We will be able to describe the state of the society and its dynamics and thus improve immensely our ability to state problems in a productive fashion, obtain clues as to promising lines of endeavor, and ask good questions.

The fruit of these [social indicator] efforts will be more directly a contribution to policy-makers' cognition than to their decisions. Decisions emerge from a mosaic of inputs, including valuational and political, as well as technical components. (Sheldon and Parke, 1975, p 698)

REFERENCES

de Neufville, J.I. (1978-79) Validating policy indicators. Policy Sciences, 10, 171-188.

Jaeger, R. (1978) About educational indicators. In L.S. Shulman (Ed.) Review of Research in Education, 6, 276-315.

Rivlin, A.M. (1973) Measuring performance in education. In M.Moss (Ed), Studies in income and wealth. New York: Columbia University Press, 411-437.

Shavelson, R., L.M. McDonnell, & J. Oakes (Eds, 1989) Indicators for Monitoring Mathematics and Science Education: A Sourcebook. Santa Monica: RAND Corporation. This Digest was adapted from material appearing in this book.

Sheldon, E.B. & Parke, R. (1975) Social indicators. Science, 188, 693-699.

-----

This digest was created by ERIC, the Educational Resources Information Center. For more information about ERIC, contact  ACCESS ERIC 1-800-LET-ERIC.

This publication was prepared with funding from the Office of Educational Research and Improvement, U.S. Department of Education under contract number RI88062003. The opinions expressed in this report do not necessarily re flect the position or policies of OERI or the Department of Education.