Macmillan Cancer Support recently claimed that young Britons have 200 Facebook friends but can only turn to two friends for support in a crisis. The survey was released for Cancer Talk Week, which was last week. The claim was widely reported.

The press release was vague about how this information had been elicited, beyond saying it was the result of an online survey of 1,000 adults aged 18-35. My thanks to Macmillan Cancer Support for providing more detail, including the polling organisation, Toluna, and the questions asked. 

Toluma is an online organisation claiming four million members around the world who participate in polls for points for which they can claim prizes or enter draws, or simply just for fun. The respondents are self-selected and the results uncorrected for any discrepancies between the sample and the public at large in terms of sex, ethnicity, education and so on.

Academic studies have shown that such polls are less accurate than those where the respondents are randomly selected, as one would expect. There is also evidence that the errors they make are not easily corrected by post-stratification (weighting).

As evidence, such polls are clearly second-best, but are they entirely worthless? I find it hard to make up my mind about that. Nate Silver, who writes a statistical/political blog for the New York Times, FiveThirtyEight, posted an interesting piece about this last month. His conclusion is that online polls should be deemed “guilty until proven innocent”. Some, such as YouGov, have a reasonably robust track record; others do not. “Providing more information to the reader is usually the right default” he concludes.

There is one area where self-selected online polls are clearly defective, and that is anything to do with respondents’ own presence online. Those who respond are likely to be more web-savvy and active online, so what they say about, for example, the number of Facebook friends they have will not be typical of the public at large. So we can probably discount that part of the Macmillan survey.

More specifically, if Macmillan wanted to know if young people would talk to each other about cancer, why didn’t they ask this specific question, rather than asking “If you had a serious problem, how many friends would you turn to for support?” It might have produced a different answer.

Those polled were aged 18 to 35 (or claimed to be) and were therefore likely either to have parents alive and/or partners or spouses. They would be far more likely to turn to these family members than to friends, so the relevance of 61 per cent saying they had “only” two friends to turn to is questionable. The poll asked how often respondents spoke to their mothers, but the answer was not reported.

In a quote in the press release Jeannie Wilkinson, a Macmillan-funded Relate counsellor, says: “It is surprising and concerning that people confide in such a small number of friends and family ...” But the survey didn’t ask about confiding in family members, only friends.

This is a far less egregious survey than the recent one by Barnardo’s, but it triggers the same suspicion – that it set out to get a particular answer in order to chime with a campaign. The week was about encouraging people to talk about cancer, so a survey showing they don’t was required.

How can charities avoid the charge of spinning surveys to suit their fund-raising purposes? I suggest they should publish the methodology used in much greater detail, name the survey company used, acknowledge the sample wasn’t random, and provide the full questions and answers. (To Macmillan’s credit, the press release did say the figures had not been weighted.)

I accept that charities are not going to commission gold-plated surveys, given their cost. But if they are going the cheap route, they should admit that the results have limitations. And they shouldn’t pretend to have answered questions they didn’t ask.