Science corrections - replication is much more important than retraction

(The original article appears at The Conversation under the title "Retraction of a journal article doesn't make its findings false.").

The American Medical Association recently retracted six papers co-authored by food consumption and psychology researcher, Brian Wansink, in three of its journals. These studies include two showing that large bowl sizes encourage us to eat more, and that shopping when hungry leads us to buy more calorie-dense foods.

A prolific academic researcher, Wansink has provided many thought-provoking ideas about the psychology of food consumption through more than 500 publications which have been collectively cited more than 25,000 times.

His research has shown that people will eat a lot more from a bottomless soup bowl; they will eat more from larger portions, even if it is stale popcorn or food served in a dark restaurant; and they will eat less if a portion is made to appear larger using visual illusions.

Retractions are a permanent means by which journals endeavour to preserve the integrity of scientific literature. They are typically issued for some form of misconduct, but it does not necessarily mean the results are false.

Are retracted studies false?

A number of challenges have been made against more than 50 of Wansink’s publications. At present, 15 corrections have been published and 13 retractions have been made.

The retractions follow a range of allegations of misconduct including auto-plagiarism (copying your own work), data mismanagement and data manipulation. But none of this means Wansink’s results are entirely discredited.

The American Medical Association made its retractions based on Cornell University (Wansink’s employer) being unable to provide an independent evaluation in response to an Expression of Concern regarding Wansink’s studies issued in May 2018.

The absence of evidence does not prove his results are false.

Science relies far more on whether results are repeatable. And many of Wansink’s results – including some which have been retracted – have been replicated.

Two of the most recently retracted studies showing that adults and children eat more from larger bowls form a part of a larger literature as reflected in their having been cited nearly 300 times and 40 times respectively.

The bigger the plate size, the more people will eat (if they serve themselves). NeONBRAND/Unsplash

Importantly, multiple reviews of the scientific literature reveal that others have replicated the findings of Wansink and colleagues on how the plate or bowl size affects consumption.

In a meta-analysis I authored with others, the combined studies in this area show that doubling the plate size increases consumption by 40% on average if people are serving food onto the plate themselves. However, there is no effect of plate size on consumption if a fixed or constant amount of food is served on the plate. (Disclosure: this meta-analysis was published in a journal issue for which Wansink was one of the editors.) 

The evidence that retraction does not necessarily falsify a result is shown in the meta-analysis reported above showing that doubling the plate-size to people who are self-serving reduces consumption by 40%. In fairness, this meta-analysis contains one of Wansink's retracted papers. A re-analysis excluding this one study sees the self-serving plate-size effect (reported in the previous paragraph) reduce from 41% to 39%. A more conservative approach would be to exclude all papers in which Wansink was an author. In this case, the self-serving plate-size effect reduces to 38%.

Replication is more important than retraction

The problem of reproducing findings in science (called replication) is a much bigger issue than retractions. Retractions attract attention, but are relatively minor; replication does not attract attention, and is critically important.

The replication crisis facing social sciences, health and medicine suggests that 50% or more of published findings may not be repeatable.

For instance, in social science, a large team of researchers replicated 100 studies published in three high-ranking journals. The results showed only 36% of the replications found statistically significant results, and the average size of the observed effects was half of that seen in the original studies.

The high rate of replication failure arises, in part, from the arcane statistical approach used for analysing research data. In essence, researchers seek statistically significant findings. Statistical significance is typically defined as when the probability (p-value) of the observed data assuming there was no effect is less than 5%.

The statistical approach has a convoluted logic which many students and even many academics, misunderstand. In simple terms, if 100 studies are conducted in which there is no effect, about five of them will produce "statistically significant" results. Even though there is no effect. That is, if I test a shark detection system in a fresh-water swimming, and it goes off less than five times in a 100 trials, it has passed the standard of statistics and we can safely conclude that it works - which is clearly absurd.

One of the simple ways in which people misunderstand this analysis is by concluding that only about 5% of published studies will be false. But this is wrong, and why few seem to genuinely understand that over 50% of published studies are false - as has been suggested for both medical and psychological research.

Journals and academics wish to publish novel, statistically significant results. They tend to ignore studies with null results, and put them in a file-drawer.

Replications that are successful add nothing new, and replications that fail (not statistically significant) are uninteresting to publishers albeit critically important to science.

A related problem is that academics may dredge through data and cherry pick statistically significant results, a practice called p-hacking.

The misconduct of journals and academics through their obsessive focus on statistically significant findings is reflected by the replication crisis and the prevalence of p-hacking.  The implications are that many, even most published studies are probably false.

If Wansink differs from others, it is in his disarming openness in a 2016 blog post admitting to data dredging. This post attracted intensive scrutiny from his peers and began.

To put this into context, Wansink has published more than 500 articles. If 250 of them prove to be false in the sense that the results cannot be replicated, then he is on par with social and medical science in general.

The retraction of thirteen of Wansink’s articles - some of which have been replicated by others - is a blip receiving much more attention than it deserves.

Science ought to be interested in what it true, rather than baying about the missteps of its practitioners. If science were to follow the principle of 'wrong in one thing, wrong in everything' (fallus in uno, fallus in omnibus), science would be a very thin volume and even then, likely to reflect only the findings of those whose errors have not been detected.

Science makes mistakes and missteps. The advances are achieved through new ideas and repeated testing.

Retractions may be important signals of reduced confidence in a finding, but they do not prove a finding false. This requires replication.

Science doesn’t provide certainty. The good scientist is one who embraces uncertainty. 

Claims of absolute certainty made by authoritative figures are probably false (c.f. Clarke's first law).

Tim van der Zee, one of Wansink’s lead detractors states on his website “I am wrong most of the time.” 

The challenge for scientists is to believe this. Few are prepared to accept that 50% or more of their published findings might be false. 

My mistake

Life is long

... if you are lucky.

Making mistakes is inevitable,

learning is not!

Mistakes are made for lack of reflection*, but

learning is achieved only on reflection.

A daily practice:

Today, I made a mistake. I was wrong. I erred. 

What were the mistakes I made today?

What - if anything - could I do to avoid making such mistakes again?

---------

* My mistake... only some mistakes result from lack of reflection! To repeat, 'making mistakes is inevitable'

Getting it right by being uncertain

Certainty for humans is like a flame is to a moth.

Blind instinct takes us there, but we may well get burnt.

We might get more things right in this world if we were a little less certain about everything.

How can we avoid the trap of certainty?

The wrongs of the righteous

There's no lack of certainty in the world, and lots of evidence that despite that certainty, we're wrong!

Marketing is plain sailing

Marketing is easy, but only if you really understand it.

And most people do not.

Most people think that marketing is powerful, that it can make people do things that they would not otherwise do.

We have a long history of being scared of such things - like hypnosis, subliminal messaging, hard sell salespeople, etc.

But just because we're scared of the bogeyman doesn't mean that he exists! Subliminal advertising was debunked decades ago - but many still believe it.

Marketing is like sailing. It does have the potential to be incredibly powerful, to drive ships around the world at considerable speed.

But the ship itself has no power. Rather, it is designed to harness the power of the wind.

And good marketing is designed to harness the power of the market, most notably, the will of the customer.

The customer's will, like the wind, is mindless. There is a clear direction in both wind and customer will, but knowing why it is going that way is difficult to know. Individual consumers are like the wind - drawn ineluctably in a particular direction, but without knowing why. Many times as consumers, we are drawn to things without really understanding why or how. 

The smart marketer feels the will of the customer on her face, and uses it to navigate in the direction she wants to go. 

In this view, the marketer no more leads the customer than the sailor leads the wind.

Yes, the sailor has a destination in mind, and sometimes, those destinations are difficult to reach. For those destinations, the sailor needs special skills, plotting angles and using the wind while tacking towards their destination. 

Sometimes, the destination is difficult to reach, it is a point within a cove laying to windward. In this situation, the wise sailor will lay off, and wait until the wind is blowing in a more favourable direction.

If the marketer is finding her job difficult, it may be because she is trying to oppose winds and currents that make up the market.

Marketing is like sailing. Progress is made by marshalling the power of the market. Good sailors use the wind, great sailors anticipate the wind.

Good marketers meet the market's will, great marketers anticipate the market's will.

The blame game: sports, alcohol, & violence

A raft of headlines reported on a fascinating finding that linked State of Origin matches to a spike in domestic violence.

For instance, the SBS report ran the headline "Study exposes 'clear' Origin link to DV" (where DV is domestic violence).

Big story because basically, the data showed that between 6pm and 6am the next morning on State of Origin night, domestic violence increases by 40%. Incidentally, non-domestic violence (blokes beating up other blokes) went up by 70% as well.

Despite this sobering result, the media have leapt on this story, and spun a long drinking yarn. Specifically, they have drawn conclusions about the involvement of alcohol in all of this - even though alcohol consumption is not directly observed in the original study in any way.

 

More specifically, the media reports claims that alcohol advertising and sponsorship of these games is to blame (e.g., see NBN's reporting).

But that is not what the data show. The study is very simple and it simply shows that on game nights, there is a spike in violence. 

It is certainly reasonable to speculate that sports night encourages more socialising, more alcohol consumption, more excitement, more aggression perhaps. But it is difficult to claim that alcohol sponsorship is the cause. 

Run the counterfactual - if ALL advertising and sponsorship of such events by alcohol was eliminated, would the consumption of alcohol stop? More importantly, would the violence stop?

Claiming that the advertising of alcohol is the cause of violence on sports nights is like claiming that promotion of champagne is the cause of violence on New Year's Eve. 

And yes, there's a spike in violence on New Year's Eve as has been reported in Australia and overseas. 

This is not to say that alcohol is not a contributing factor. The sports event (like New Year's Eve) is a social event. Many people like to drink on a social event, some will drink too much, and some regrettably, may be violent.

That is, it is sporting and other social events encourage alcohol consumption. But some will argue, surely alcohol advertising is cuing people to drink? 

Surprisingly perhaps, there is not strong evidence to suggest that this is the case. A review of the cuing effects of food and beverage advertising suggests that it has little effect on consumption by adults.

Besides, a review of multiple studies show that the drinking patterns of children tend to be associated with the drinking patterns of their parents. 

We parents worry about the impact of alcohol advertising on children's drinking, but completely overlook the influence of our own drinking and that of our friends.

I am dismissive of media and public fears of alcohol advertising when I see adults who will down a beer to celebrate each time their team scores. And then ask one of their children to get them another beer from the fridge in readiness for the next score.

Not only is the story short on hard data, but it encourages more twittering about short advertising spots where everyone runs to the toilet to drain the last beer, all while ignoring the far more pervasive impact of how the whole culture in which we are immersed consumes alcohol.

Alcohol advertisers are not creating these problems. They are simply like seagulls drawn to chips on the beach. Sports events, public holidays and large public, social celebrations are when some people like to drink, and alcohol advertisers will squawk away in the hope of capturing their share of the chips.

To be clear, I have no aversion to the alcohol advertising and sponsorship being removed - partly because I simply am not much of a sports spectator. But I do have an objection to it being removed on false and specious grounds, and particularly if it removes what appears to be a pleasure and a joy for many.

The claim that alcohol advertising and sponsorship is the problem is a furphy.

Curiously, no-one seems to have correctly understood the real implications of this research. What this research evokes a deeper, darker concern.

The shows that big sports events tend to be associated with violence. The same result has been observed elsewhere in the world with large televised soccer matches in France and the UK leading to an increase of admissions to emergency departments.

The implication is that large sports events should be stopped! 

Suddenly, the supporters of sport will rally round protecting their pleasure from the fun-police. The methodology will be questioned, the conclusions challenged, the results dismissed. 

The game goes on, and we are reminded that much of research is really just a sport. 

To do: nothing

"Just don't do it."

Just do nothing.

Ethics is the philosophy of action. So part of its domain surely therefore includes, the philosophy of inaction.

And this is a defence of doing nothing.

But there's so much to be done!

How can I defend inaction in a world with so many inequalities and injustices that plague our lives?

My answer? For precisely that reason! Precisely because so many are raging at inevitable inequalities and injustices which will forever plague our lives.

Glorious food, inglorious super-marketing bastards & the obesity crisis

Blaming supermarkets for obesity is a little chopping down a tree we are standing on, like tying our bootlaces together so we don't trip up on them.

"No wonder we have an obesity epidemic" the article announces as an introduction to "the nutritionist's eye-opening video post to Faceback".

Sure is eye-opening - if you have never been to a supermarket before.

Or eye-opening if you have been, but didn't notice the end-aisle displays because you were headed into the aisles which are filled to the brim with stuff that most nutritionist's would find to be equally awful.

No wonder we don't make any progress on obesity when we continue to run stories like this.

This kind of story about the evil of supermarkets is popular because it is an awesome cover story. It allows us to completely shift the responsibility to someone else.

Every parent has heard it: "He made me hit him."

Supermarkets, like siblings, are easy targets.

But supermarkets don't even make the products that people are so angry about. And if people are so damn angry, why are they still buying the stuff?

We're listening to the righteous leaders, while the masses (excuse the pun) are right behind waving their packet of chips and saying how they are right behind their nutritionist friends.

What about the good that supermarkets do?

Supermarkets have replaced the old family-run corner store groceries.

We've lost a way of life, rather like horses, carts and buggy whips. The wheel turns, and it does so because it generates something good.

Modern supermarkets give us access to an enormous range of food stuffs of excellent quality at a good price. Oh, and lots of choice!

And so many good things that supermarkets have done to support healhier life-styles. Supporting activities for local schools, introducing and supporting organic and other specialised food ranges, giving fruit away to the kids as they walk around the supermarket.

The reason why we don't talk about the good is simple. It ain't a story.

Are we serious about tackling obesity? If so, there is no "us" and "them." Sure supermarkets want to make a profit, and we want good food - meaning both good for you and just plain yummy even if not good for you - and we want it cheap.

We both individually and jointly contribute to the problem of obesity.

Time for us to make up our minds. Do we want to solve obesity?

Yes? Well then we need to make changes. And it's not about just one side or the other, but both sides together. Obesity is a community problem, one that consumers and marketers need to work on together.

Pointing fingers at the other side is like junk-food -- it has a really satisfying mouth-feel in the moment, but isn't doing us any good.

Don't lie if you want the Easter bunny to visit

The Easter bunny is apparently a German innovation and brings painted eggs, candy and even presents to children who have been good.

What a delicious irony! A fiction to encourage honesty along with other good behaviour.

Meanwhile, neither parents nor clerics tell their respective flocks that there is no "bunny" in any of the stories about Jesus' death and resurrection.

We are not told that the word Easter commemorates a pagan goddess, Eostre. Or at least, might do if St Bede's account of Eostre is not a lie as at least some scholars contend!

What a web we weave,  when we practice to deceive.

Does it matter if we lie? Well, how do I know when to believe?

The Bible (to stay with the Easter theme) would have us believe that Jesus was crucified on Friday and resurrected on Sunday.

Is it true or is it a lie? That's an awful lot of consequences if that is a lie!

But let's back up a little. Let's start by being a little clearer about what we mean by lying.

The truth about eternal life is... we don't know!

We like to believe that our beliefs come from good solid reasons. 

But given that we selectively choose evidence to reinforce what we already believe suggests otherwise (i.e., the confirmation bias).

We only question our beliefs when evidence comes up to suggest it just ain't so. 

Or so I like to believe... but maybe that simply ain't so!

Even in the face of overwhelming contrary evidence, the human mind remains remarkably resistant to doubt.

Consider Leon Festinger's fascinating work on a group who believed that the world would end on December 21, 1954. How would they deal with the evidence that their predictions of the date were wrong - assuming they were wrong?

What is truth?

   

• The truth is very often not knowable
      
• The truth is that many beliefs are formed without knowledge
     
• The truth is that people claiming to know the truth are generally deluded
     
• The truth is that a fiction can be powerful (think placebo, nocebo, etc.)
     
• The truth is that truth may not be a supreme virtue
     
• The truth is that what works, practical wisdom, may be all that matters

Facebook & fast-food - superficially satisfying, but bad for you

Social media is to human connection what junk-food is to human nutrition.

Facebook like fast-food is convenient, it meets a desire, is superficially satisfying, it even offers some of what we need (connection or calories respectively), but by and large and in the long run, it is not all that good for us.

Look at the change in the fundamental unit of community — the family. Today, one in five children are raised in a home where the other biological parent is absent.

Some still see the nuclear family, mum dad and 2.5 kids, as the ideal. But the ideal community unit had already gone the way of REAL hamburgers and real food.

Prior to the nuclear ideal, the extended family lived together, providing a community that supported and sustained them all.

Even now, grandparents have time for kids that parents do not. But now the grandparents have their place and we have ours. We live our lives apart: each child in their own room, each parent may even have their own house. And the grandparents? Live elsewhere.

even though it's not good for me!

Mmm? I may be lovin’ it, but I am pretty sure it’s not good for me.

What is good for me? I’m not sure, but it seems more like when I live in a community, a village, surrounded by say 10–20 people whom I know and trust.

Facebook with 500 or more so-called “friends” cannot replace my village, the people I know and trust.

And trust should not be confused with like. All humans are flawed, and there may be things that I might not like about someone, but that does not mean I am not able to trust them. Trust means I know the person, I know her or his limitations, I know of the contexts in which I might trust them to do the right thing and which ones where they might not.

They are not even necessarily dear friends, I don’t even necessarily like them so much of the time. Rather, I know them, and can trust — as in rely — on them to behave in certain ways.

Uncle Frederic might be an odd-bird, a bit of a bore at social gatherings, still a bachelor and frankly, we can understand why. Nonetheless, he is good with the kids, and they love him, and at a pinch, he could be called in to take Cathy out for a talk and a soda as she wrestles with life as an adolescent.

Just as fast food has supplanted simple, real food cooked at home from scratch, so Facebook and the other online copy-cats have supplanted community created at heart in the home.

A “like” from some 500 or more of my Facebook “friends” may give me a little frisson of pleasure, but a more limited gesture of friendship is difficult to imagine. Pressing a thumbs-up symbol, and then moving on to the next image in a continuously changing stream of images and words is hardly much display of friendship let alone commitment.

The problem is that convenience is king in so many things. We love quick-fixes. And fast food and Facebook offer just that.

Unfortunately, what we want is not necessarily what we need. And what we need — healthy food, exercise, and making time and space for our village, some of whom are frankly a pain in the butt — is bloody hard work.

So we don’t bother. We choose the junk option, the quick-fix, the one that sets off the pleasure centre in our brain. We hear a ping on our phone and leap to look: someone likes me!

That’s not good, that is sad.

To feed your body, you need to eat well. To feed your soul, you need to connect well.

Data bites: confusing cross-tabulations

Some recent research from a sample of 957 members of PureProfile's Australian panel showed that people who classified themselves as "Early Birds" were two times more likely than "Night Owls" to earn over $70k per annum.

Specifically, 23% of Early Birds earned over $70k p.a. vs just 11% of Night Owls.

Does that mean that 23% of those who earn $70k+ are Early Birds and 11% are Night Owls?

Nope. If that were true, that leaves two-thirds (66%) of $70+k earners who are neither Early Birds nor Night Owls.

Does the result mean that there are more Early Birds than Night Owls earning above $70k per annum.

Not necessarily.

If the Night Owls are far more numerous than Early Birds in the total sample, then it is quite feasible for there to be more Night Owls who earn $70k+ even while Early Birds are two times more likely to earn $70k+ than Night Owls.

Making this error is very easy - unfortunately - and even downright confusing in some situations. Here's an example that can seem particularly confounding.

PureProfile's research showed that in the Australian population, men are more likely to be Early Birds than women. About 56% of men are Early Birds compared with just 45% of women (see yellow shading in table below).

However, when we turn the result around so it expresses the proportion of Early Birds (and Night Owls) who are male vs female, we may be surprised to see that 50% of Early Birds are women and 50% are men. (In actual fact, there are slightly more women who are Early Birds than men as we will see in a moment).

Whaaaaaaaaaaaaaat? How can that be?

The problem is one that often confronts us when we do crosstabulations. A crosstabulation (often shortened to crosstab) is simply breaking down the frequency of responses on one variable by groups (in this instance, the groups are male and female).

People tend to get confused because they see the first result (56% of males are Early Birds), and think that this is equivalent to saying that 56% of Early Birds are males.

But this simply ain't so.

Let's break this example out. First, here's the raw counts in each cell. In this sample, there are 945 males who are Early Birds - or 945 Early Birds who are male if you prefer. It is the same thing!

And note that there are slightly more women who are Early Birds than men: 951 women vs 945 men.

The proportion (or per cent) of males who are Early Birds depends on the total number of males there are in the column.

The proportion of Early Birds who are male depends on the total number of Early Birds there are in the row.

So, in a nutshell, there are 945 males who are Early Birds. This represents 56% of the total number of males (column %), but just a fraction under 50% of the total number of Early Birds (row %).

The key takeout is this. Whenever a percentage is being reported, take note of the base. Are you looking at the % of the column (in which case the sum of the column is 100%) or the % of the row (in which case the sum of the row is 100%).

Understanding this distinction is important - and surprisingly often misunderstood. Here's one extreme example to highlight the problem.

Nearly 100% of sexual assaults are perpetrated by males but that does not mean that all males (or even a high percentage of males) are molesters/rapists - thankfully.

However, that doesn't stop many parents, airline policies and even national news anchors from treating all men as potential molesters. Most molesters are male, but most men are not molesters. Again, thankfully.

Drawing this conclusion, and worse, enacting policy based on this result reflects a gross misunderstanding and misinterpretation of the statistics. And it happens to lead to inappropriate stereotyping of a lot of good men. If interested, you can read more about this case here.

How to minimise the danger of this error?

Whenever reporting a percentage, be very clear about what is the base, ie x% of what? Quite simply, % of men is not the same as % of Early Birds.

Meghan Trainor - it's all about the base!

If you're preparing crosstabulations (crosstabs), I generally recommend (and myself, generally present) column percentages only. That way, you know you're always comparing the % of column 1 to the % of column 2.

But what goes into the column and what goes into the row? Generally, we try and put the Causal factor into the Column, and the Result into the Row. As sex is generally decided many years before we begin to decide whether we like to get up early or stay up late, sex is the cause (put it into the column) which is thought to determine the result, namely, whether or not you are an Early Bird.

If you do want to swap it around (and see what proportion of Early Birds are female vs what proportion of Night Owls), swap the row and column variables and rerun your crosstabulation. That way, you are still reading column percentages. (It can still be confusing, but hopefully less so).

And practice. Swap the rows and columns, see if which makes most sense.

Above all, do not mistake the per cent of the column to be the same as the per cent of the row.

Sharing wisdom: reflections on the road to the country for old men

M.C. Escher

Not everyone is lucky enough to journey into the "country for old men" (and women). And even among those who are lucky enough to reach old-age, not all learn as much from the journey as they might. Some arrive at the destination having missed the journey.

Growing old is inevitable, growing up is optional.

Is practical wisdom valuable?

Even in a society which is increasingly technical and technological, there is some sense that wisdom is a worthy goal, a knowledge worth attaining. Adapting the words of Aristotle in Nichomacean Ethics only a little, we can see that his observation applies still today:

"Although the young may be experts in geometry and mathematics [and technology and computers] and similar sorts of knowledge, they nonetheless lack practical wisdom. Such wisdom is gained from experience which the young do not possess, for experience is the fruit of years." (adapted from Nichomachean Ethics 1142a).

So even while we may rely on our children, our grand-children or both to help us download apps to our mobile phone, to show us how to connect our "smart" television to Netflix and how to stream music through Spotify to our tablet and other devices, there is still a place for wisdom.

Skepticism: are you willing to change your mind?

Joel Pett (2009) USA Today

"The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt." 
-- Bertrand Russell

Skepticism has received an increasingly bad name - but most of that is from mis-attribution. 

Skepticism can be usefully distinguished from denialism - even though the word skepticism is used to describe anti-vaxxers to climate-change deniers, 

In some instances, skepticism is used as a synonym of denial as in "I am skeptical about your claims", or "I doubt that what you say is true".

However, skepticism is more uniquely used to describe the notion of uncertainty - as in "I'm uncertain about the truth of what you say", or more simply, "I am uncertain about what is true".

We need to talk about bias: The census

Reliably biased

One thing that virtually everyone seems to believe about survey research is that a bigger sample is better: a sample of 500 respondents is better than 100, and 1000 is better still. And best of all is a census where, technically, the margin of error is reduced to zero.

However, the truth of the research adage about bigger sample sizes holds if, and only if, there is no bias. And the probability of zero bias is rare.

Researchers know this – but the media and the general public do not. The press and their audiences fixate on the number of respondents in research. To overcome this problem, we need to talk about bias.

When the Census 2016 online form was inaccessible for more than 48 hours from 7.30pm on census night (August 9), it created a media flurry. However, when the ABS reported that more than 96 per cent of households had completed the census by the closing date (September 23), many concluded that the problem was resolved.

However, the problem of bias remains, and this is not necessarily resolved with a high response rate. The real #censusfail is less a data collection glitch and more the threat it has posed to data quality.

Research lessons from Census 2016 - making sense of a senseless fail

Let's admit - it's a trainwreck 

The ABS is threatening two million households with fines if they do not complete their census at the same time as being "adamant the quality of data has not been compromised."

It's not so long ago that they reassured the public they would be able to cope with the demand on census night. They were wrong then - and they are wrong now.

The data quality has been hopelessly compromised. For one thing, at this point in time, there are two million out of "close to 10 million dwellings"  that are yet to complete. That is, 20% who have not completed yet.

In the short run, #censusfail was about a data collection problem. The website for collecting the census data was inaccessible on the night of the census at the time that most people would have completed the census form. It remained inaccessible for a further 48 hours. Even longer for some.

The initial response to this colossal data collection glitch was a flurry of fingerpointing and promises that "heads will roll."

However, the bigger problem appears to have been totally overlooked. The real crux of #censusfail is less a data collection fail and more a data quality fail.   

Metric madness: "how likely are you to recommend me?"

Marketers love a good metric system, but the numbers we’re so keen on crunching aren’t always the main objective, says Honorary Adjunct Professor, Macquarie Graduate School of Management, Stephen Holden.

Posted byB&T MAGAZINE

Marketers are mad about metrics! By that, I mean both manic-mad and misguided-mad.

The manic side is clear in the availability of marketing metrics beyond measure: Net Promoter Score, brand valuation, brand equity, purchase intention, ServQual, social media “likes”, CTRs (click-through rate), etc.

The misguided side is that we have become like a child fascinated by a tape measure or a digital scale. A little girl pulls out the tape measure and delights as the tape automatically retracts. A little boy puts his hand on the kitchen scales, the digital display lights up, a number appears.

We get a number, but what does it mean? Who knows! But everyone is measuring it, so we do it too.

As a consumer, I’m baraged by requests to complete little surveys for my bank, my mobile provider, my internet-provider, the airlines I fly, etc. And so many of them end with what we all know is the Net Promoter Score:

“How likely is it that you would recommend [brand] to a friend or colleague?: 0=not at all likely, 10=extremely likely.”

But what does this mean? I suspect that many think that a high recommender score means high satisfaction and/or high future purchase intention – and a low recommender score the converse. But even just a little thought highlights that ain’t so.

What mobile network would I recommend? In accord with MBA-speak, the answer is “it depends!” If you want coverage in rural areas, go with Telstra – and no, that is not my network provider!

What internet provider would I recommend? Not mine because they suck. However, they have rather neatly promised me the world wide web, tied me into a 24-month contract and then delivered crap service. But how good are the alternatives? Not much better – as reflected perhaps in the low NPS norm for internet service which is three!

Purchase intention, satisfaction, and recommendation measure three different things. I might recommend a wedding dress designer, but I have no intention of using them again!  I loved going to Tahiti, I might even recommend it if I thought it suited the person I am talking to, but I’m never going back: “been there, done that.”

I am dissatisfied with my bank, but I have no intention of changing: it’s just too hard. I might even recommend them as being the best of a bunch of bastard bankers.

And I have no intention of talking about, let alone recommending, my favourite guilty pleasures (my favourite treat at the local bakery which is in limited supply), my dirty little secrets (the toilet paper I use), things so trivial or personal, I simply don’t know the brand and/or would never would discuss with others (butter, toothpaste).

Another meaningless measure is the widely touted brand value. What can this measure mean when multiple agencies come up with different scores?

Marketing Week columnist and academic, Mark Ritson has eloquently railed on this nonsense in print and in a public debate with the companies which hawk this BS.

Just because I can measure something does not mean it is useful. I could measure the average height and weight, of the marketing department, but what does it mean? If you’re measuring marketing muscle, you need measures to match.

Make the measure to fit the effect, not the other way round.

Originally posted at B&T Magazine