The Ministry of Justice has recently launched a “payment by results’ pilot study at the privately run Peterborough Prison, which holds 480 male and 360 female prisoners.  
 
The scheme aims to reduce reoffending by rewarding the organisers, the social investment organisation Social Finance (SF), if their clients’ convictions in the first year after their release are cut by 10 per cent. Investors in SF will be rewarded if this 10 per cent cut (recorded on the Police National Computer within 18 months of release) is achieved per cohort of 1,000 eligible male prisoners, or if a 7.5 per cent cut is achieved across all 3,000 clients SF is contracted to assist.
 
Only detected crimes count and, of course, a first conviction for a crime committed in the year after my release may not feature on PNC until much later even than 18 months.
 
The study protocol for the Peterborough Pilot should include a statistical analysis plan which sets out clearly how various potential difficulties should be handled. For instance, against what comparator group will the clients be matched?  How was this group determined? And what are the prior expectations on the number of convictions?
 
No randomisation has been planned. The SF pilot began in August but the contract was only let in November for the Independent Assessor who will determine the controls and how they will be (retrospectively) selected.  
 
The plan, apparently, is to nominate a set of prisons which are comparable in function to HMP Peterborough though not necessarily run by the same private company (Kalyx); and, for each client of SF released, to select (how?)10 male prisoners released on the same day from comparator prisons.
 
The intention is that SF will work with, or be assessed on, all male prisoners released from HMP Peterborough who were sentenced to less than 1 year in jail. How will the independent assessor know, in retrospect, the list of all who were eligible? And how will he (or she) know about male offenders who were sentenced to less than a year, served most of their sentence at HMP Peterborough but were conveniently transferred to another jail prior to release?
 
The first flaw in this outline is different start-times. To be eligible, an SF client must have been there for a sufficient period of time for Social Finance to introduce him to its services, which are delivered both in prison and after release. A minimum pre-release period, whatever it may be, will be needed.  Is the same minimum pre-release period applied when selecting eligible controls from comparator prisons? We need to know.  
 
The second flaw is the potential for losing SF clients before they are released. The candidate may seek, and be granted, a prison transfer, or the prison’s director may instigate one; worse, SF may decide that a prisoner is not shaping up and so, pre-release, have him or her shipped out of the programme so that comparators are never sought but SF has successfully ditched a likely loser.  
 
In principle, the last of these cannot happen but the assessor’s contract does not specify how he, or anyone else, can determine whether all eligible inmates have been ascertained, and for whom SF should be accountable. All of these difficulties could be controlled by randomization and the principle of “once randomized, always analysed”. Without randomization, the potential for bias and wrongful reimbursement from the public purse is considerable, as Department of Health learned to its cost and ours when its non-randomized reimbursement scheme for drugs for multiple sclerosis patients fell apart.
 
Within 6 years (recruitment period + 2 years’ follow-up, I assume), if first-year-convictions have dropped by more than 7.5 per cent, investors should receive a payment representing a proportion of the cost of re-offending. Apparently, MoJ’s costs have been capped at a generous £8 million for 3,000 releases, corresponding to a 13 per cent reduction in first-year-reconvictions. However, SF-investors’ outlay is also guaranteed, by an amount which I have not discovered.
 
The SF spend of between £1,000 and £3,000 per client is in addition to whatever is currently expended to ease the transition of released prisoners back into their community. Thus, MoJ’s “payment by results” pilot actually compares a minimum undisclosed payment of £X million for 3,000 HMP Peterborough releases against no additional outlay for 10 times as many releases from other prisons. The fact that ten times as many controls are being sought makes it clear that no additional expenditure is involved in their cases. The only additional cost is that for the independent assessor.
 
Spending will be increased – to a maximum of £8 million – if the mean number of first year convictions for SF clients is 10 per cent less than for controls. Other providers might spend £X million for 3,000 male releases more cost-efficiently, but this pilot will not answer that question either.
 
“No convictions” does not equate with “no crimes”, and detection rates - at about a quarter - are low for volume crimes. So what accounting adjustment will be made for undetected crimes?
 
I requested the study protocol from Social Finance, from Prisons Minister Crispin Blunt, from Kalyx, and from MOJ’s professional staff, but was not holding my breath because design issues were fixed as part of SF’s contract, and it was “commercial-in-confidence”.  
 
I was therefore pleasantly surprised, but not altogether reassured, by MoJ’s answers to key questions and the specification for the Independent Assessor which, it was hoped would serve as a study protocol despite stating that: “It is not envisaged that the Contractor’s evaluation will be published; however, the MoJ reserves the right to do so if required.”  Here are my questions (bold), and MoJ’s answers (italicised):
 
Why no randomized controls?
Practically and operationally, difficult to randomize within a single prison: Yet, daily, doctors randomize within single practices or clinics. Cluster-randomization of prisons, the alternative, would have allowed neither the same concentration of SF’s resources, nor guaranteed its preferential choice of locality.
 
Lack of investor appetite/ethical issue, SF keen to intervene with all offenders in Peterborough Prison, rather than leave some without support: I’ll bet -  thereby SF guarantees no like-with-like comparison while offenders  elsewhere are without their support.
 
Difficult to maintain ‘clean’ within-prison control group, so that control group would be unblinded: Sounds as if the controls, and perhaps even the control prisons, will not be informed that they are part of a non-randomized experiment. The SF-intervention cannot anyway be masked.
 
What statistical power does the Ministerial pilot have against a well-reasoned target effect-size?
 
Effect size of 10 per cent is dependent on there being 1,000 unique offenders (does that mean that the same man, if re-released from HMP Peterborough after a short sentence, is SF-ineligible? And how can same restriction be applied with equal force to controls in other prisons?) and 10,000 matched-controls for 80 per cent  power at 10 per cent significance level, based on using the latest available estimates (mean and sd not provided; 10 per cent significance level is un-conservatively liberal) for reconviction events for prisoners sentenced to less than 1 year’s incarceration.
 
The MoJ’s Business Plan warns that, in August 2011, even the number of pilot projects established, and their participants, will only be released “subject to commercial confidentiality and Office for National Statistics guidance” – a marriage of terms that ONS might wish early divorce from. Unacceptably, the same marriage-vows bind publication (on the first 1,000 SF-clients & 10-member control sets) of an interim evaluation of SF-rehabilitative projects and their payment-by-results information.
 
Rehabilitation of offenders is a public service for which Ministers must be held account publicly, whether that service is delivered by private companies or by public servants. Kevin Dunion, Scotland’s Information Commissioner, has ruled that contracts with private companies for the delivery of public services must be in the public domain – as the Reliance contract for prisoner escorts now is.
 
He also deprecated, but could not overturn, the decision by ministers that allowed Scotland’s original prisoner escort contract to be written so that monthly outcomes on a range of key performance indicators (including prisoner self-harm) were hidden from public and parliamentary scrutiny. This was outrageous.
 
We want no repetitions of this sorry saga when it comes to MoJ’s rehabilitation pilots – they are too important, both for prisoners and for their potential victims. The rehabilitation pilots’ design (preferably randomized), analysis and costs need to be fully in the public domain.