Hooray! A proper survey of what doctors think of the Health Bill
Credit where it’s due: the Royal College of Physicians today reports the results of a properly-conducted survey of what its members and fellows think of the Health and Social Care Bill.
I’ve previously criticised the Royal Colleges of General Practitioners and of Psychiatrists for online polls of their members that generated very low response rates and from which anybody with a respect for evidence would hesitate to draw conclusions.
The physicians, by contrast, engaged the Electoral Reform Society and sent out questionnaires by post and e-mail (where known) to its 25,417 fellows and members, achieving a response rate of 35 per cent (8,878) which is respectable, certainly much better than the 9 per cent achieved by the RCGP and the 11 per cent by the psychiatrists.
The BMJ, which claims to be read by 100,000 doctors in the UK, did a similar online poll, with a response rate of less than 3 per cent – 2,710 (92 per cent) saying the Bill should be withdrawn, and 241 (8 per cent) saying it shouldn’t.
It’s interesting to compare results from the RCP’s survey with these online “voodoo polls”. The RCP found that 69 per cent favoured rejection of the Bill as it stands, with 6 per cent accepting it and 22 per cent neither completely accepting nor completely rejecting it.
When we look at the question most closely aligned with those asked by the RCGP and by BMJ – that is, should the Bill be withdrawn – 49 per cent favoured the call for withdrawal with 46 per cent calling for continued engagement to improve it. A further 4 per cent had no opinion.
This result is quite different from those of the RCGP and BMJ online polls, both of which scored more than 90 per cent in favour of calling for withdrawal of the Bill. The psychiatrists didn’t ask this exact question, but did ask whether respondents thought the bill “fundamentally flawed” and more than 80 per cent said yes.
The more nuanced response from the physicians is in spite of the fact that they may have more to lose from the bill than GPs, since it aims to put the GPs in charge of commissioning care, a prospect not all hospital doctors welcome.
The RCP’s president, Sir Richard Thompson, said: “We believe that this is the single biggest survey among the medical royal colleges, with the highest turnout, and while it shows there is a clear majority of RCP’s fellows and members who would personally reject the Bill, opinion is divided almost equally on whether the RCP should continue to critically engage or now call for the withdrawal of the Bill.”
The results, available in full on the RCP website, are a textbook example of why I rail constantly and unavailingly against self-selected online polls that draw opinion only from the strongly committed. These polls produce biased results, and the RCP deserves great credit for spending a bit of money to do it properly and showing what a difference it makes.
Update, 19 March
And since we're handing out bouquets, another one is due to the Readers' Editor of The Observer, Stephen Pritchard, who wrote in his column yesterday that the "poll" underlying a story in the paper last month was not to be trusted.
The story said nine out of ten members of the Royal College of Physicians wanted the health bill withdrawn. But it came from a website, callonyourcollege.blogspot.com, set up to oppose the bill, and was a self-selecting open-access poll. Junk, in plain language. Stephen's column indicates that the one of the journalists responsible for the story is reluctant to accept this.
But junk is what it was. It's no excuse to say that readers were given enough information about its provenance to work out for themselves it was junk. That's a journalist's job.
Gabriel Scally (not verified) wrote,
Fri, 16/03/2012 - 18:06
Nigel, I suspect you may be wrong about the RCP doing the survey properly.
The first of four questions read: 'Do you accept or reject the plans laid out in the Health and Socia; Care Bill (as it stands now)?' This is an entirely objective question. In my view it could only be improved upon by reversing the order of the words 'accept' and 'reject' in a random 50% of questionnaires, and similarly alter the order of the response boxes.
There were three options given: 1) 'Accept'. 2) 'Reject'. 3) 'Neither completely accept, nor completely reject'.
The second question was a different kettle of fish. It read: 'The RCP's position has always been that we do not accept the Bill in its current form and seek to engage critically with all parties to amend and improve the legislation. We have been anxious that requesting withdrawal of the Bill has the potential to impair our ability to influence the healthcare policy agenda in the future. In future, do you think the RCP should:'
The two options given were: 1) 'Continue to engage critically on further improving the Bill.' 2) 'Seek withdrawal of the Bill.'
The final two questions were invitations to rank key issues of concern out of a list of 12 in question 3, and 16 in question 4. Question 3 was about 'health reforms' and question 4 about 'issues that may effect the quality of care that physicians are able to deliver to patients'.
There can be no serious doubt but that the design of question 2 introduces serious bias both in the wording of the two wholly unnecessary sentences that precede the actual 'question', and also in the wording of the responses.
I will certainly be using the questionnaire in teaching as an example of the deliberate introduction of bias in the framing of questions.
Colin Forster (not verified) wrote,
Mon, 19/03/2012 - 15:12
It is not even the case that 90% of the respondents to the January 2012 RCGP survey were against the bill.
Q4 of the survery asks: Do you think it is appropriate to seek the withdrawal of the Health and Social Care Bill?
The 1200 or so who did not answer “agree” or “strongly agree” to Q4 take themselves out of the survey (count the numbers – they dont answer Q5 or Q6). The results of Q5 and Q6 are then 98% and 90% “yes”, but only because the “no”s have already stopped answering!
In fact, 1863 voted strongly or very strongly to withdraw from an a total response of 3119 (59.7%)
Contrast this with a similar poll in October 2011 where 1413 voted strongly or very strongly to withdraw out of a total response of 1906 (74%).
So the percentage of respondents keen to withdraw FELL substantially between Oct and Jan. Of the extra responders in Jan, more than half were against!