The CREDO Study; Dubious Conclusions About New Jersey Charter Schools

I am far from an expert when it comes to statistics.  In fact, reading parts of the CREDO study published today about charter performance in New Jersey makes me feel more than a little intimidated.  I'm diving in though, because with the glowing conclusions in the press release I may have to shut down this blog and run out and enroll my kids in the local charter!

The CREDO at Stanford University New Jersey analysis found that 30 percent of the charter schools have significantly more positive learning gains than their traditional school counterparts in reading, while 11 percent of charter schools have significantly lower learning gains. In math, 40 percent of the charter schools studied outperform their counterparts and 13 percent perform worse. In comparison, CREDO’s 2009 national study of charter schools in 16 states found at that time that 17 percent of the charter schools had exceeded their district school counterparts’ growth . 

A significant finding came from the results of the urban charter schools in the state. Students enrolled in urban charter schools in New Jersey learn significantly more in both math and reading compared to their traditional public school peers. In fact, charter students in Newark gain an additional seven and a half months in reading per year and nine months per year in math compared to their traditional public school counterparts. Students enrolled in suburban charter schools also learn significantly more in both math and reading compared to their peers in traditional public schools; however, students in rural charter schools learn significantly less than their district school peers in both reading and math. 

“Charter schools in New Jersey, specifically in Newark, have some of the largest learning gains we have seen to date. These results demonstrate that charter schools can thrive in a constructive policy environment and prove to be a high-quality option for parents and students,” said Dr. Margaret Raymond, Director of CREDO at Stanford University. (emphasis mine)

Before filling out applications for my daughters I will wait patiently for far smarter folks like Rutgers' Bruce Baker and ELC's Danielle Farrie to sink their teeth into this one, but a few things jump out from the actual study, even at a novice like me. 

First, check out this graph:

According to these numbers, if your kid attends a charter in Newark they are going to show gains.  But how about if your kid attends a charter in one of the "other major cities" which are defined as Camden, Trenton, Jersey City, and Paterson?  

Sorry, no gains for your kid.  

But wait!  The press release stated that "students enrolled in urban charter schools in New Jersey learn significantly more in both math and reading compared to their traditional public school peers"?  

I repeat, NO gains in four of the five major cities in New Jersey!

And what about the statement that "charter students in Newark gain an additional seven and a half months in reading per year and nine months per year in math compared to their traditional public school counterparts"?  

Here is the chart in the report that demonstrates these numbers:

And here is the paragraph that precedes it:

The data is analyzed in units of standard deviations of growth so that the results will be statistically correct.  These units, unfortunately, do not have much meaning 
for the average reader.  Transforming the results into more accessible units is challenging and can be done only imprecisely. Therefore, Table 3 below, which presents a translation of various outcomes, should be interpreted cautiously. (emphasis mine)

Yet, these gains are stated boldly in the press release as "fact" without this caveat.  (And not to nit pick, but since when do we round 7.2 to 7 and a half, not 7??)

Joy Resmovitz, Huffington Post education reporter extraordinaire, seems to have noticed the same Newark effect:

The study, which looked at the state's performance relative to charter schools nationally, found that while New Jersey charters tended to have more promising outcomes, Newark's schools are responsible for the bulk of the gains. 

"The real story here is how Newark’s middle-school charters are lifting otherwise low-achievement youths," said Bruce Fuller, a University of California, Berkeley education professor who was not involved in the study. "That’s where the encouraging action is ... Once you go outside of Newark and into elementary schools, the results are quite disappointing."

Well, that sure has far less punch, doesn't it??  Maybe I was a bit hasty to talk about shutting down the blog...

Look at this graph that is supposed to "determine whether performance remained consistent over all the years of study:"

 Notice the dip in 2010?  Here is how it is explained in a footnote:

The atypical result for the 2010 growth period may reflect changes in the state achievement testing regime in the prior two years, which resulted in new standards and higher performance requirements.

Does anyone else read that as an indicator that charters are teaching to the test?  If a change in the test results in lowered scores, followed by increased scores the next year, does this reflect growth or merely preparation?  Or, as is the case with Robert Treat, something even worse.

Here's one last graph:  

Looks like there are fair gains in urban schools, very little in suburban, and loses in rural schools.  But remember, the graph above showed that ALL of the urban gains are in Newark.  The other four cities showed no gains in Math and lost ground in Reading.  

How do you draw a conclusion from any of this other than that at best charters across New Jersey are a crap shoot? 

Maybe putting my kids into a lottery to take my chances that the local charter might be slightly better isn't the best idea...

Bruce Baker has pointed out time and time again, that the results in the "high flying" charters in Newark are more likely related to segregation than any other factor.  

He did it again today, and concluded:

So, when all is said and done, this new “charter school” report like many that have come before it leaves us sadly unfulfilled, at least with respect to it’s potential to provide important policy insights. Most cynically, one might argue the main finding of the report is simply that cream-skimming works – generates a solid peer effect that provides important academic advantages to a few – and serving a few is better than serving none at all (assuming the latter is really the alternative?). 

And that right there my friends gets to the heart of the matter.  When at their best charters only provide academic advantages to a few, and many, many more fail to do so, why are we allowing charters to take resources from district schools struggling to serve ALL students?

Looks like I'll be keeping the blog up after all, and no need to fill out those lengthy applications!  Whew!

This parent is not only sticking with New Jersey's traditional public schools, I'm going to keep fighting for them.   

Who's with me?