Recent Articles

Allowing for Known Unknowns: The Dangers of Ignoring Reality.

  • Written by Andrew Elliott McBurney
  • Hits: 2603
  • Email

Decisions must be based on reality.

As teachers, there are some things we have direct control over. Some things we influence to one degree or another. There are other things we have no control over. Yet several of the more-recently developed accountability systems inspired by NCLB don’t necessarily take those things into account. Sometimes, when there are factors teachers have no control over, those factors are treated as if they don’t exist.

This is not a plea for no accountability. We need an accountability system, but it needs to be a fair and accurate assessment of how schools and teachers are really doing. Any system we adopt needs to be based on reality—all reality, or at least as much as possible, and not just some selected parts of it.

Donald Rumsfeld has taken a lot of flak for his famous, or infamous, statement that:

Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones. (Dept. of Defense News Briefing, February 12, 2002.)

Putting aside the contentiousness of the War on Terror and the decisions made by the Bush Administration, this is as about as succinct an explanation as you can get of the factors that influence decision-making, and it applies as much to the profession of teaching as to any other. There are things we know we don’t know, and we have little, if any, influence over them. We ignore the known unknowns at our peril.

So, how do we account for them in an accountability system?

I think the answer lies with the idea that while the actions of one human being are unpredictable, if you get enough humans together, a statistician could give you odds on exactly what they are going to do. In other words, we should look at each school or each teacher relative to all others that are similar—we should make use of standard deviations. This is contingent upon data that is already generated and generally available.

Let’s take attendance rates for a fictional Central High School. Central will be compared with all other schools in the state that are comparable, taking into account the number of students, demographic data, numbers of at-risk, economically disadvantaged, limited English proficient, &c. All these schools together will be the population. So long as Central’s attendance rate is within two standard deviations below the mean, then Central would not be negatively impacted in the accountability system (or perhaps the state will decide on one standard deviation, or perhaps no negative impact for only one standard deviation, a slight negative impact for two standard deviations, and a greater negative impact for being beyond three standard deviations, or whatever a low outlier is determined to be). If, on the other hand, Central is two standard deviations above the mean, or a high outlier, then Central would be readily identified as a school that should be looked at for why it is having success with its attendance rate—and so that other schools might then learn from Central.

But, what if the state sets a standard that all schools should have 95% attendance, and the population that Central is in has a mean of 89%? First of all, schools in that population within a standard deviation or two (however the state decides) would not be negatively impacted in their accountability rating, since they are really no better or worse than most of the other schools in that population. The high outliers (again, however the state defines this) would be looked to as examples for what strategies to implement. Schools below the mean would be motivated to improve against the possibility that many of the other schools in the population will themselves improve, thus raising the mean for the total population.

If, after a period of years, the mean attendance rate of the population is still below the state standard, then this would be indicative of concerns greater than any one school that the state itself, working with those schools, should figure out how to address—but in any case without penalty to those schools that are within one or two standard deviations from the mean.

This could be applied to anything the state wants to consider for accountability, such as passing rates on state-mandated exams, SAT scores, rates of college acceptance, &c.

Not only would this be more fair to the schools than many existing accountability systems, but it would be better at identifying areas of concern as well as successes that other schools could learn from.

It might also discourage schools and districts from developing policies and programs that meet certain accountability standards without actually doing anything to improve the education of their students. For example, there is a school district in my state that, shortly after the state decided to consider the number of students successfully completing Pre-AP/AP courses in its accountability rating system, announced that henceforth, all of their middle school students would be “Pre-AP.” I have no idea how that’s working out for them.

When he was pushing for NCLB, President Bush was right to warn against the “soft bigotry of low expectations.” Here is a way that we can be fair, have high expectations, and more effectively help schools achieve those expectations, by basing our evaluations and decisions in reality.

(NOTE: This post one of a set of four posts, which, altogether, describe four dangers I believe are facing 21st century public education in the United States. The others are: “Traditional Equals Bad: The Dangers of Definitions,” “Whatever Happened to Lifelong Learning?: The Dangers of Measurable Objectives,” and “Sponges and Bombing: The Dangers of Keeping Up Appearances.”)