The Public Policy Forum posed an interesting question on their Facebook page last week:

“Is it helpful to compare student performance across school districts with differing demographics?”

The question was spurred by the reaction to the recent release of their annual report on school performance in Racine.  I answer their question with an enthusiastic yes.  In fact I will go further, it is not only helpful, it is essential.

Why?  Well, graduates of districts across the state will be competing for the same spots in universities, and/or for the same jobs.  It is crucial that districts know how their students are faring compared to others, regardless of their students’ demographics and socio-economic situation.  If we do not know how they compare, how can we even begin to close achievement gaps?

It is not enough to simply present data on how students of any given school district are faring given the non-classroom challenges they face.  That does not mean we shouldn’t measure and celebrate (or lament) how Milwaukee is performing compared to other urban districts.  We should.  But we also need to know where students stand compared to others they will compete with in the real world.

My enthusiastic yes to the Public Policy Forum’s question comes with an important caveat; we should not judge teachers or schools based on how their students compare to students from significantly different demographic groups.  Accurate comparisons of the impact specific schools or teachers have on student learning require accounting for the impact of outside factors, such as socio-economic status, known to at the very least serve as a proxy for non-classroom factors known to negatively impact performance.  Using raw test scores to judge school or teacher quality unfairly holds professionals and institutions accountable for things they do not control.

An analogy I have heard several times is that swapping all of the staff between Brookfield Central and Milwaukee’s Bay View High would likely not significantly change the academic performance of either school.  That is probably true. However, that does not mean the differences in student performance in those schools should not be understood and explained.  And it also does not mean value-added analysis cannot be considered (as it is in the Public Policy Forum report on Racine) to get a better idea of how schools are serving students, regardless of their demographics.