Sections

Personal tools

Blog Entries: NATO & All the Unanswered Questions May 09, 2012
Why hasn't your Chicago press pushed for answers to these questions about the prep by Chicago criminal justice agencies for NATO summit?; “Justice” in the police accountability system in Chicago? Apr 10, 2012
My thoughts on how to define justice in the Chicago Police Accountability system.; Closing Districts Dec 06, 2011
Shedding a little reality in the discussion about closing districts. Not something you will hear from the Fraternal Order of Police.; [read more]

Take me some credit-ville

On Wednesday the 16th, I attended an event at the Institute on Public Safety and Social Justice at the Adler School of Professional Psychology. The title of the event was "Attention Felons:

On Wednesday the 16^th, I attended an event at the Institute on Public Safety and Social Justice at the Adler School of Professional Psychology. The title of the event was "Attention Felons: Chicago's Project Safe Neighborhoods”, (PSN).

One of the speakers at the event was Dr. Andrew Papachristos, who co-authored the evaluation of the project during his time in Chicago. The evaluation showed that PSN was successful at reducing gun crimes in the west side neighborhoods they operated in. The funny thing about social science research in Chicago, you never really hear about a project evaluation that showed the project was not successful, weird isn't it? I mean PSN might be a great project, but with all the glowing evaluations of the other projects running in Chicago it is hard to trust the claims of results of any of them.

When the presentation was over I asked a question about how the evaluators of the program were able to control for other possible factors such as differing policing strategies in the districts they observed vs. their control districts. To his credit, Dr. Papachristos basically said that they could not control for most of the factors I mentioned. That got me thinking about how all these programs from criminal justice agencies, community agencies, and now the feds might all just be taking credit for the same occurrence, a reduction in the homicide rates in Chicago, when in reality most likely had little or nothing to do with the reduction.

It is important for academics, politicians, and the public to understand that evaluating the precise cause for a complex social phenomenon such as crime is really more art than science. Unfortunately for Chicagoans, most of the data that would allow anyone to judge the art/science is only available to the people doing the evaluations. Even Papachristos, in his answer to my inquiry, said that he tried to get a few other types of data to look at in an effort to consider other external effects and he could not because that data was being examined by other academics. I guess that means it is off limits to others while one academic is using that particular set of data.

Key to understanding the limits of social science research and especially project or program evaluations is that the researchers can almost never truly account for all the possibilities that could have been responsible for the outcome. These possibilities in social science research are called external effects. Here is just a short list of possible external effects that could account for the drop in violent crime in Chicago over the last 19 years or so, because crime has been dropping in Chicago since the very early 1990.

Chicago's Alternative Policing Program (CAPS)
Gentrification
The economic changes during this period that greatly improved employment opportunities for the youth of Chicago
New programs put into place by the Cook County State's Attorney's Office
Increased incarceration
Better re-entry programs that help ex-offenders reenter society
Improvement in schooling by the Chicago Public Schools that has prepared Chicago's youth to enter the job market
Tougher sentencing policies in the Cook County Circuit Court
Redevelopment of the Chicago Housing Authority
Better work by Chicago Police Officers independent of any specific program

Now any of these ten could possibly (although I seriously doubt most had any effect) account for some of the drop in crime and violence in Chicago. The reality is we will never really know. We must be careful when individual academics evaluate a program and find that it is working. Public policy is too often driven by this research without the benefit of additional independent researchers corroborating the results. You can take a look at most major programs working on crime and violence in Chicago and find some academic who has done a evaluation of the project to find only too glowing results. In Chicago there seems to be a pandemic of this type of research.

I am not saying that all project evaluations are not worthy of some degree of trust; some are actually valid to one degree or another. We need to be cognizant of the role restricted access to data plays in anyone's ability to evaluate how any agency or project within the criminal justice sphere is working. Restrictions are in place to prohibit prying eyes from looking upon data that will detail the patterns and practices of criminal justice agencies. This goes the same for evaluating some community based organizations because most of the data you need to evaluate their work comes from criminal justice agencies. The agencies typically do not provide access unless they are pretty sure of the outcome of the study before it even takes place. This gatekeeper function played by local criminal justice agencies allows criminal justice officials the ability to regulate what data is used to evaluate their practices and the practices of the agencies they work with. All project evaluations needs to be take with at least a teaspoon of salt, sometimes a gallon.

Document Actions

Send this

Bookmarks

Comments (5)

john hagedorn Sep 21, 2009 01:46 PM

I've read the evaluation, and you are right Tracy, so many alternative explanations are not considered. But most important is why the rates are so high in Lawndale, much higher than in nearly all "violent" (meaning poor black) communities in New York or other large cities? Everyone claims credit for any decrease in murders but we hear nothing from anyone when rates go up. But who is responsible for Chicago's homicide rate that is nearly 3 times New York City's? If policing strategies are so effective, what effect does police corruption and brutality have? Frankly, I'm pretty fed up with evaluations like those of CeaseFire that claim credit where no credit is due.

Steve Rhodes Sep 21, 2009 06:06 PM

I also become suspicious of locals touting success strategies when local trends follow national trends. If crime is dropping *everywhere,* then why should we attribute the decrease to a local strategy that isn't being employed elsewhere? When a local trend differs from a national trend - up or down - that's when I tend to take note.

andrew papachristos Sep 22, 2009 02:14 PM

Excellent comments! As one of the authors of the PSN report, let me make a few points:

John definitely gets at the bigger issue: why are rates higher in these areas than in other? Even after any reductions that we find, crime rates in these areas are still 5xs the national average! That is indeed the central issue, make no doubts about it. Those of interested in such issues are asking these questions all the time and our life work hopes to at least shed some light on these issues. So, let me re-iterate that these are things I take VERY seriously in my work and have written about such issues elsewhere.

However, the questions raised by tracey, john, and steve, are bigger issue than any single program (or its evaluation) can get at. No single program or policy can eliminate racism, hunger, poverty, genocide, police brutality, lack of economic opportunities, and so on. Instead, programs tend to focus on patricular issues and try to solve those. In the case of PSN it focused on a specific population and a specific crime: gun offending among people on probation and parole. One needs to judge/evaluate a program based on its goals, what is sets out to do, and how these two things are important relative to the larger issues in its city/neighborhood. Gun violence is a serious issue. The other issues raised are also serious, but not the focus of PSN. To put it in an extreme, you should fault a housing program on the Southside of Chicago for not paying attention to, say, a string of rapes on the Northside of the city.

Someone will always say, "yeah, but X also had an effect on crime rates." All of this is fine. But, I'm in the business of testing such claims with systematic data (qualitative or quantitative) and analysis. If it's gentrification, then let's test it. If it's street workers making a difference, then let's test it. If you can test more than one thing at a time all the better. And, if you have more data and of different types, the better still!

The point of any "good" evaluation, is to try and "do its best" to rule out as many competing hypotheses/causal factors as possible. We covered at least three in our paper that we had data on: (1) changes in socio-demographic-economic indicators; (2) spatial diffusion/issues (which may include gentrification, but that is debatable) ; and (3) a highly visible competing program, CeaseFire. We were unable to obtain any detailed data on policing strategies, although we tried. And we were unable to acquire other public health data (on other injuries), though we tried.We openly confess our sins as well as our unsuccessful attempts to gather such data. It would have been much better had we had such data, no arguments from me on that one.

As for Steve's and Tracy's comments, we did indeed try to control for overall declining crime rates and the observations we saw were NOT attributable to the overall decline in crime/violence. So this is not, to use a statisticians term, simple "regression towards the mean" here. In other words, crime when down much more dramatically in these areas than in other parts of the city or in other neighborhoods across the U.S.

Thanks for the comments.

Tracy Siska Sep 22, 2009 04:01 PM

Thanks for your comments professor. I hope that this blog was not taken as a slight against your work evaluating PSN, it definitely was not intended to do so. It is more towards the problem of community members, policy makers, and academics in general in Chicago who think that just because their is a glowing evaluation of the project that the project must be working. We discussed this issue after your presentation at Adler in reference to CeaseFire. CeaseFire was able to get a very positive evaluation form the academics at Northwestern, just like about all the CRJ related projects do.

Now, maybe they are all accomplishing their small piece they are concentrating on or something else is at work. That something else needs to be dug up and brought to the light of day because policy makers see a glowing evaluation without ever being able to understand that their might be junk social science behind it. Programs like CAPS and CeaseFire will always get glowing recommendations as long as their is little capability of the public to scrutinize the data and understand the connections some academics have to the programs they are evaluating.

As to the societal context, I worry about how larger societal issues are accounting for reductions (or increases) in crime and violence. I worry that partly because of social science based project evaluations that reinforce the belief in the public that the policing strategies are the cause of violence reductions that when crime goes up it will in result in greater levels of police abuse because they are "fighting the gangs". If the reality is that policing strategies generally have little impact on crime and violence it is important for the public to know that upfront so as to not force the police in to a untenable situation down the road. New York city is a good example of how it seems that policing strategies may not have a significant impact on crime and violence, just as it looks like incarceration may not also.

My blog was gearing at starting a conversation about the veracity of some social science based project evaluations. It seems there might be a need of looking at all the work a particular academic has done to see if there are any negative evaluations. If not, then it seems they are either really lucky to evaluate on the best of the best, or their is something else going on here. In Chicago, it always seems like there is something going on somewhere behind closed doors that the public never really learns about until it is way to late.

As for PSN, I knew little about it before your presentation. I have reservations about the long federal sentences but I intrigued by the approach to reentering felons and meeting with them to inform them about what the feds are doing.

Thanks for the comments this has been constructive.

andrew papachristos Sep 22, 2009 04:58 PM

No offense taken: this is the fun part of my job, as far as I'm concerned. I welcome such discussions.

As for the long-federal sentences, our eval shows that, in fact, that the traditional "get tough" strategies had very little effect whatsoever on the decline in violent crime. The greatest impact came from those issues that focused on successful re-entry and the "buy-in" of the young men returning to these community (my comments on "Why Criminals Obey the Law", for instance). The effective of such non-enforcement efforts were considerably higher than any of the enforcement efforts we measured, which was part of our/my excitement about PSN.