Tuesday, November 27, 2012

The CREDO Study; Dubious Conclusions About New Jersey Charter Schools

I am far from an expert when it comes to statistics.  In fact, reading parts of the CREDO study published today about charter performance in New Jersey makes me feel more than a little intimidated.  I'm diving in though, because with the glowing conclusions in the press release I may have to shut down this blog and run out and enroll my kids in the local charter!
The CREDO at Stanford University New Jersey analysis found that 30 percent of the charter schools have significantly more positive learning gains than their traditional school counterparts in reading, while 11 percent of charter schools have significantly lower learning gains. In math, 40 percent of the charter schools studied outperform their counterparts and 13 percent perform worse. In comparison, CREDO’s 2009 national study of charter schools in 16 states found at that time that 17 percent of the charter schools had exceeded their district school counterparts’ growth . 
A significant finding came from the results of the urban charter schools in the state. Students enrolled in urban charter schools in New Jersey learn significantly more in both math and reading compared to their traditional public school peers. In fact, charter students in Newark gain an additional seven and a half months in reading per year and nine months per year in math compared to their traditional public school counterparts. Students enrolled in suburban charter schools also learn significantly more in both math and reading compared to their peers in traditional public schools; however, students in rural charter schools learn significantly less than their district school peers in both reading and math. 
“Charter schools in New Jersey, specifically in Newark, have some of the largest learning gains we have seen to date. These results demonstrate that charter schools can thrive in a constructive policy environment and prove to be a high-quality option for parents and students,” said Dr. Margaret Raymond, Director of CREDO at Stanford University. (emphasis mine)
Before filling out applications for my daughters I will wait patiently for far smarter folks like Rutgers' Bruce Baker and ELC's Danielle Farrie to sink their teeth into this one, but a few things jump out from the actual study, even at a novice like me. 

First, check out this graph:
According to these numbers, if your kid attends a charter in Newark they are going to show gains.  But how about if your kid attends a charter in one of the "other major cities" which are defined as Camden, Trenton, Jersey City, and Paterson?  

Sorry, no gains for your kid.  

But wait!  The press release stated that "students enrolled in urban charter schools in New Jersey learn significantly more in both math and reading compared to their traditional public school peers"?  

I repeat, NO gains in four of the five major cities in New Jersey!

And what about the statement that "charter students in Newark gain an additional seven and a half months in reading per year and nine months per year in math compared to their traditional public school counterparts"?  

Here is the chart in the report that demonstrates these numbers:
And here is the paragraph that precedes it:
The data is analyzed in units of standard deviations of growth so that the results will be statistically correct.  These units, unfortunately, do not have much meaning 
for the average reader.  Transforming the results into more accessible units is challenging and can be done only imprecisely. Therefore, Table 3 below, which presents a translation of various outcomes, should be interpreted cautiously. (emphasis mine)
Yet, these gains are stated boldly in the press release as "fact" without this caveat.  (And not to nit pick, but since when do we round 7.2 to 7 and a half, not 7??)

Joy Resmovitz, Huffington Post education reporter extraordinaire, seems to have noticed the same Newark effect:
The study, which looked at the state's performance relative to charter schools nationally, found that while New Jersey charters tended to have more promising outcomes, Newark's schools are responsible for the bulk of the gains. 
"The real story here is how Newark’s middle-school charters are lifting otherwise low-achievement youths," said Bruce Fuller, a University of California, Berkeley education professor who was not involved in the study. "That’s where the encouraging action is ... Once you go outside of Newark and into elementary schools, the results are quite disappointing."
Well, that sure has far less punch, doesn't it??  Maybe I was a bit hasty to talk about shutting down the blog...

Look at this graph that is supposed to "determine whether performance remained consistent over all the years of study:"
 Notice the dip in 2010?  Here is how it is explained in a footnote:
The atypical result for the 2010 growth period may reflect changes in the state achievement testing regime in the prior two years, which resulted in new standards and higher performance requirements.
Does anyone else read that as an indicator that charters are teaching to the test?  If a change in the test results in lowered scores, followed by increased scores the next year, does this reflect growth or merely preparation?  Or, as is the case with Robert Treat, something even worse.

Here's one last graph:  
Looks like there are fair gains in urban schools, very little in suburban, and loses in rural schools.  But remember, the graph above showed that ALL of the urban gains are in Newark.  The other four cities showed no gains in Math and lost ground in Reading.  

How do you draw a conclusion from any of this other than that at best charters across New Jersey are a crap shoot? 

Maybe putting my kids into a lottery to take my chances that the local charter might be slightly better isn't the best idea...

Bruce Baker has pointed out time and time again, that the results in the "high flying" charters in Newark are more likely related to segregation than any other factor.  

He did it again today, and concluded:
So, when all is said and done, this new “charter school” report like many that have come before it leaves us sadly unfulfilled, at least with respect to it’s potential to provide important policy insights. Most cynically, one might argue the main finding of the report is simply that cream-skimming works – generates a solid peer effect that provides important academic advantages to a few – and serving a few is better than serving none at all (assuming the latter is really the alternative?). 
And that right there my friends gets to the heart of the matter.  When at their best charters only provide academic advantages to a few, and many, many more fail to do so, why are we allowing charters to take resources from district schools struggling to serve ALL students?

Looks like I'll be keeping the blog up after all, and no need to fill out those lengthy applications!  Whew!

This parent is not only sticking with New Jersey's traditional public schools, I'm going to keep fighting for them.   

Who's with me?


  1. It may just be me, but the thing that jumps out in Figure 5 is that they have from 37 "persisting" schools in 2008, add 2 that same year, and have 50 "persisting" schools the next year.

    I didn't go to a charter school, so I never learned where 37 + 2 = 50. Or how you could add two new schools in each of the following two years and end up with 51 unless three of those "persisting" schools no longer persist.

  2. Hi Ken,
    There is this footnote to that graph:

    "Note the jump in the number of persisting schools between the 2008 and 2009 periods.
    Eleven schools appeared in the data with tested students for the first time, although they
    were opened in prior years."

    One thing they don't seem to be clear about in the study are how many charters closed. This is what they mean by persistent: didn't close or get shut down. It's unclear to me whether the charters that are closed are just removed from the data set all together or only removed in the year they close. Would be interesting to know that. If they aren't included, one would think this would skew the numbers artificially high....

    To me, this is remarkably flawed. How can you judge the efficacy and success of charters if you don't look at how many charters open only to close shortly thereafter and what effect this has on both the charters AND the districts?

    If I missed where the study addresses how closed schools are handled (which is entirely possible), someone please enlighten me.

    Thanks for stopping by Ken!

  3. What intrigued me was Figure 9. It showed that those with 3 years of continuous attendance in a Charter,students actually showed a decreased SD in their third year of attendance.

    Why? Will their SD continue to decline over years 4,5,6...?

  4. Newark is the only NJ city with a significant national Charter Management Organization (CMO) presence - KIPP and Uncommon schools. These are by far the two biggest operators in the state, and their scores on state assessments are routinely among the best in the city.

    The conclusion here is staring folks right in the face - replicating successful CMOs should be a priority; more skepticism should be given to startup/'Mom and Pop' single-site charters.

    Would you support an amendment to the authorizing process that eases the process for organizations with a track record of success while heightening scrutiny for new providers?

  5. Thank you for this information, particularly your contribution as to exactly who is paying for this study. Personnally, I find that information a little too muddy for anyone to truly know who's funding exactly what. This is disgraceful. And it's a perfect example of the lack of transparency there is in NJ government, Particularly from Cerf.

    So the next question has to be: Why can't our own State Department of Education issue a comprehensive report on charters or on all public schools? What is so difficult? Or is it that Cerf doesn't want to do it.

  6. Probably students must needs to occupy each and every aspects mentioned here, because, these generally helps to move along with some certain principles.