The dangers of surveys

We'll show why surveys, polls, figures and statistics, and conclusions derived from them, can't always be relied upon. Just don't ask to see our sources.

81% of journalists know that statistics make it 64% easier to come up with a gripping introduction to our articles 40% of the time.

That's probably why every day (that's 100% of days) I am guaranteed to receive at least one press release from one company or another with the results of their latest survey or report, which frequently demonstrates why we need to buy more of their products.

The figures can be interesting and entertaining, and the conclusions and analysis that the companies supply with them full of soundbytes that journalists can use as the basis of their articles, or to hook readers in, or to pad out a piece to the required number of words.

But check the footnotes to many of these reports and they appear to be nothing more than quick polls or back-of-envelope calculations. This may get the company's name in the papers in a positive light, but it doesn't mean they should be relied upon by readers.

This doesn't stop us journalists from using them though. Here's just a handful of surveys and reports that have been quoted in the press in recent days.

3m more non-retired adults are “not at all ready” for retirement compared to last year

Retirement income specialists MGM Advantage carried out a survey to see how ready we are for retirement. It reckons that 54% of non-retired adults are “not at all prepared” for retirement. That represents an increase of more than three million people in one year, it claims.

So, from being at least somewhat prepared for retirement, three million extra people in just 12 months are suddenly not in the slightest ready.

That sounds like quite an extreme jump to me. The problem may be because MGM Advantage is extrapolating from the results of a poll of just 1,600 or so non-retired adults. About 850 of them must have said they weren't at all prepared. But let's say that the 54% figure is wrong by just five to six percentage points (an error of 80 or so people in its poll) and, when we extrapolate to the entire non-retired adult population, we can see that MGM Advantage's figures could easily be too high by around two million people. And that all depends on whether they used appropriate population data both times, which, as far as I can see, they don't reveal.

This is a common issue with small surveys. This one questioned far less than 1% of the group involved. Modest errors matter a lot when extrapolating figures from small samples onto an entire population.

UK consumers losing £3.6bn in missed credit card rewards

I quoted this figure from American Express myself in a recent article, but then went on to explain that it was probably misleadingly high. It shows what rewards we'd be getting if we switched from plastic that doesn't reward to plastic that does.

Looking at the footnotes to Amex's press release, you can see that the cashback and reward card provider assumes that every adult should have a reward or cashback card, and that they are all able to get one.

But many consumers would find it hard to get such credit cards, as you generally need a better credit rating than for other sorts of cards. Many of us already have a wallet full of other cards, too.

Furthermore, these credit cards are frequently not a good choice, because most UK citizens are paying off debts. A credit card that enables us to do that more cheaply, such as a balance-transfer card, makes much more sense and will save far more than £3.6bn in interest payments between us altogether.

48% of 25-50 year olds don't have a will

One of MyVoucherCodes's recent surveys doesn't have the whole shebang of flaws, but it could have at least three of them.

To start with, it too extrapolates from a small sample size (just 1,219) up to many, many millions of people. You know about the inaccuracies this could create already, so I'll move on.

This report also causes the reader to infer something that may or may not be true. The survey didn't find that “48% of people who need wills don't have them”. What it found was that “48% don't have wills”. But when you read that statistic “48% don't have wills” in a newspaper report, accompanied by words of how frightful and disturbing this is, you could be forgiven for getting the meaning of the survey confused, and for inferring there's more to worry about than there really is.

If I had died in my late teens and early 20s, I wouldn't have cared a damn who got my meagre possessions. I'm sure I wasn't the only one.

The survey goes on to break it down as to why these people don't have wills and here at least one more error is made.

One of the reasons given for not having a will was “don't need one”. 13% of those without a will gave that explanation. The survey also makes a big thing about how a third of them – 32% – are parents.

What a less mathematically aware reader might easily infer from that is that 32% is a worryingly large proportion of parents. But let's look at those numbers a bit more carefully.

48% don't have wills. 13% of them say “they don't need one” and 32% of them are parents. That's 32% of 13% of 48%. Doing the maths, that means less than 2% of all the people surveyed said that they are parents who don't have a will because they don't need one. It's not such a frightening number when you spell it out like that.

Furthermore, some of them could be right that they don't need a will. For sure, lots of parents do; many of the 52% who said they do have a will are also parents. However, there are scenarios where you won't need a will. If you have insignificant assets and high debts, or if you're married and your half of your assets aren't worth a great deal, you might have nothing to worry about, as any assets you have could end up where you want them to be by legal default.

That said, most people do need wills. I am sure that many people who need them don't have them because they assume that their spouse, partner or children will get the lot, which isn't necessarily true.

Furthermore, even if you don't need a will, it could be worth writing one. Wills can reduce administrative costs and inheritance tax, which means more of your pot goes to your loved ones.

But you shouldn't let a survey lead you to the spontaneous decision to visit a lawyer and write out a cheque for their services without further research. A good starting point for getting information would be our wills archive.

Mostly harmless

It looks like none of the surveys I've picked out this week are going to end up devastating any readers' wealth. For the most part, I think, they will probably even encourage positive and sensible research or action. That's why using these surveys is so tempting for us journalists. But do not assume corporate research peddled to the media is always harmless: the results can cause you to make expensive or even poverty-inducing mistakes.

Always take surveys quoted in the press with a vat of salt. Do your own research, think for yourselves, and ask questions in Q&A.

More: Compare credit cards through lovemoney.com | Finally, some good news about car insurance! | Ditch these current accounts – quick!