The Ministry of Justice aims to test whether, by investing in resettlement support post-release, the reconvictions of prisoners leaving Peterborough Prison after short sentences can be reduced by 10 per cent. The Social Impact Bond (SIB) is a novel form of financing in which investors who pay for the rehabilitation programme will be rewarded by its success – or left out of pocket if it fails.

Measuring the degree of success of this scheme is critical, since public money is involved.  But, as Straight Statistics has reported before (here and here), this is far from a simple task. It involves creating a control group of matched prisoners released from other prisons against whom to measure whether the cohort of Peterborough prisoners who received SIB’s support had fewer reconvictions in the first year post-release.

Social Impact Bond (SIB) selected the private prison HMP Peterborough, (which mainly serves two police force areas, and associated courts) and also the re-settlement teams to work in its environs. Peterborough Prison was purposely selected, not picked at random. Nor was it considered possible to randomize prisons, or prisoners, to intervention or non-SIB.

Rather, the plan is to create a control group by the technique of propensity score matching – for each male released from Peterborough Prison,  identifying a set of control prisoners whose profile matches his but who were released from other prisons. Begun on 10 September 2010, the first Peterborough cohort will eventually comprise 1,000 consecutive eligible prisoners released from HMP Peterborough having served sentences of less than 12 months.  

Once the ex-prisoner is on SIB’s books, he remains there for the next 18 months because success is to be measured by each court appearance that results in a conviction (regardless of how many offences were dealt with at that court appearance) for crimes committed within 12 months of release provided that the reconviction is registered on the Police National Computer (PNC) within 18 months of the offender’s index release. 

Finding a control group that matches this Peterborough cohort without the safeguard of randomization is a task and a half.

For the Peterborough cohort, how can one prevent (or detect) inadvertent shuffling of later charges into an already-scheduled court appearance, which would reduce the count of court-appearances resulting in a reconviction? Or the use of discretion in favour of fixed penalty to avert a court-appearance altogether? Or selecting controls in such a way that recidivists, who have already been released before from the same control prison since 10 September 2010, remain eligible ? Rather you than me . . .  

Nothing daunted, an independent assessment team (IAT) from QinetiQ and the University of Leicester has been appointed and recently http://www.justice.gov.uk/downloads/publications/research-and-analysis/m... ">reported on the methodology to be used. Their first key task was to find out, using 2008 data from the Police National Computer, how propensity-score matching could be used to select controls (averaging nine per case) from 36 other local male prisons. Unfortunately, and somewhat surprisingly, the documented characteristics available for matching do not include opiate-dependency, including by injection, which has been known for some 20 years to be associated with high recidivism. But, the number of past incarcerations is taken into account, which is, perhaps, a reasonable proxy.

Credit is due to the team for delivering on its brief, except perhaps in respect of ineligibility for ‘cohorting’ of recidivists who have had a prior eligible-release from the same control prison since 10 September 2010 (the start-date for SIB ‘s first cohort). I do not think that this aspect is yet fully covered.

However, the team did ensure that liberations from court were ineligible at control prisons – because they are ineligible at HMP Peterborough. They also documented how many control prisoners were selected from each of the 36 control prisons but did not report this as a proportion of the establishment’s capacity. These proportion fall broadly into three subgroups with 24/36 local prisons having greater ‘proportional’ similarity to HMP Peterborough, that is: selected controls were 18 per cent or more of their capacity.

How better could the pilot have been designed? Well, suppose that we had determined in advance, as IAT has now done, that 24 other local prisons can provide prisoners who are, on 2008 data, a reasonable match to the Peterborough releases.  

Together with HMP Peterborough itself, we would thus have had a total of 25 local prisons. Let us assume that all would be willing (or obliged, as now) to participate. To avoid the statistician’s hell of n=1, we might then choose at random:

1. five of the 25 eligible propensity-match prisons to be the establishments at which SIB operates;
2. five to receive extra cash per eligible release, matching the cost per prisoner of the SIB scheme, to be deployed by the prison and its local partners to improve eligible ex-prisoners' re-integration into the community and to reduce theur recinviction count, and;
3. the randomly-remaining 15 establishments serve as cluster-randomized controls who receive neither additional cash nor SIB services.

This randomized design differentiates SIB-specific services from additional cash for locally-improved resettlement and from the status quo (control group of 15 prisons); and it removes SIB’s advantage of purposeful-selection.

The Ministry of Justice (MOJ) also deserves congratulation.  The IAT found that the matched controls from 2008 had a slightly higher reconviction rate than did the Peterborough releases: a mean of 1.71 court convictions (sd 2.48) for the 6,591 controls from the 36 other local prisons against a mean of 1.64 (sd 2.56) for the 694 male offenders from Peterborough. The apparent head-start for HMP Peterborough (0.07 fewer reconvictions) was not formally significant but is a goodly step on the way to the reduction of 0.17 (10 per cent of control mean, see IAT’s Table 3.14)) that SIB has to demonstrate to profit early.

If the Peterborough’s head-start were to persist in 2009 and/or after adjusting the analysis for match-covariates’ influence on reconvictions, then the SIB-cohort might be faced with having to demonstrate a reduction (relative to the control-mean) of 0.07 + 0.17 = 0.24 rather than just 0.17.

The 0.17 reduction is about twice the likely associated standard error{when based on 1000 vs 9000}and so MOJ’s power calculations look to be in right public-purse ballpark for 50:50 power per SIB cohort of 1,000 – if the SIB intervention indeed delivers a 10 per cert reduction and the HMP Peterborough effect in 2008 were happenstance. Of course, if the SIB-effect-size were genuinely 0.24, then the power would deservedly be 80:20 for the SIB cohort of 1,000 to be in profit. Nice work, MOJ.

Of course, ahead of SIB’s first cohort being completed, the independent assessment team will wish – if possible - to repeat its test-drive on data from 2009/10 to see how stable its regression coefficients for propensity-match characteristics are between cohort-years; whether the same set of local prisons offers a proportionately high yield of controls; and how stable the intrinsic HMP Peterborough effect appears to be (with and without covariate adjustment). Did SIB select well?

The advantage that retrospective test-driving has is that in 2008/09 {& 2009/10} no-one had any particularly vested interest in stacking offences per court appearance, or in promoting differential exercise of discretion to avoid court-appearances. That advantage cannot entirely be relied upon to apply in prospect . . .

Conflict of Interest: SMB has been a pro-bono, independent commentator on draft work on how to select controls for the HMP Peterborough cohort.