Are Governments making Polling more difficult?

The most recent GMR report, covering the calendar year 2016, paints a positive picture of the research, insights and analytics sector. All regions – and the vast majority of individual markets – recorded growth in local currency terms, which on a constant currency basis was +3.7.

In addition it is promising to see that the traditionally defined sector, as well as the extended market definition, BOTH record growth – implying that the demand for evidence-based guidance and actionable insight continues to thrive.

Much of the most important work that is guided by research often does not get the publicity it deserves – examples such as the research that helps to reduce child-sex-trafficking in India, the research that is now monitoring the medical fight against pneumonic plague in Madagascar, or the research that helped the NZ government to reduce its outstanding student debt, thereby making more money available for further investment into education – largely go unpublicized and unrecognized.

Ironically, it is one of the smallest sub-sectors of the research profession – opinion polling – that gets the highest profile. The most recent valuations estimate that polling represents less than 2% of the global research, insights and analytics profession, and yet almost everyone you meet will express an opinion on the veracity (or otherwise!) of an opinion poll.

Arguably, opinion polls are one of the most important sources of information about citizens and society.  Social and opinion research measures and reports views and behaviours based on scientifically gathered data and statistical evidence.  Such research is used to report on citizens' views about the society in which they live – reporting that information to the citizens themselves -- and therefore any hindrance to the free flow of information impacts on the delivery of effective, evidence-based policy-making and the reporting of fair elections.  To this day, opinion research remains the best method to report the views of ordinary citizens.

Since 1984, ESOMAR and WAPOR (the World Association for Public Opinion Research) have been proud to conduct the global "Freedom to Publish Opinion Polls" survey – a worldwide measure of the conditions under which social researchers must operate, in order to be able to provide these objective measures of society.

This year, the sixth in the series, we were fortunate to collect information from 133 countries.  Up from just 49 countries in the first survey in 1984, and a 50% increase from the 85 countries examined in 2012, the most recent previous report.  This year the scope was truly global, and for the first time included strong representation from both Asia and Africa, as well as nearly complete coverage of Latin America.

We are delighted to be able to report on some of the as yet unreleased topline, preliminary findings from this survey for this article.  There will be much more official analysis to come.  

This was a study of all sorts of poll restrictions, as well an assessment of research skills and of the quality of poll reporting, but of course a major focus, as always in these studies, was the "blackout" period before an election, when polls cannot be published.  

Most striking and unfortunate is that over the last 33 years, there has been little change in the percentage of countries reporting restrictions of some kind on polls.  In this year’s study, more than half the counties report some restrictions on when polls can be published before elections, including a majority of the countries in Europe, nearly all the countries in South America and every Central American country from which we have data.  That marks little change from the ongoing pattern.  The share of countries with restrictions in all the previous reports has also always stood at about half the countries.

Naturally, there are differences by region: most European countries with restrictions have a relatively short blackout period – less than a week, with half having a blackout of only one day.  However, nine of the 42 European countries surveyed have blackout periods of a week or more, including several members of the European Union: Belgium, Cyprus, Spain, Italy, Luxembourg and Slovakia.  

But nearly every country in South America with a blackout period imposes its blackout for a week or longer – and that is also true for half of the countries in Central America.

In total 30 countries report a blackout period of a week or more. That list of countries is a mix of developed democracies and emerging democracies, and more than one-third are in South and Central America.  In fact, for the most part the most restrictive countries are in Latin America, where several countries – Bolivia, Chile, El Salvador. Honduras, Panama and Paraguay -- have blackout periods of two weeks or more, with some forbidding pre-election poll publication within the last 30 days of the campaign!!  The only countries reporting longer blackouts are Cameroon and Tunisia.

We are led to believe that such restrictions are put in place in order to prevent "misleading" information from being circulated amongst the populace. Given the recent media storm surrounding the UK Brexit election, or indeed the US Presidential election, you might be forgiven for thinking that these restrictions had some basis in fact.

But this is not true.  A database of more than 30,000 published opinion polls, recently compiled by Kantar, and launched at the ESOMAR Congress in September of this year, clearly demonstrates that the polls conducted within one week of the voting period, are uniformly the most accurate of all the polls prior to a vote.

Polls conducted 2 weeks before an election are 28% less accurate than poll conducted in the final week. Polls conducted a month before the election are nearly 50% less accurate.  

Source: Kantar international polling database 2017. The figures quoted are the average absolute errors (top 4) and are based on the analysis of 31,310 polls from 473 elections and voting events across 47 countries.

Government interest in limiting public opinion polling only adds to the well-known difficulties faced in conducting polls.  In almost 40% of the countries providing data on this question – 47 countries in all – researchers claimed it was more difficult to conduct public opinion polling now than it was 5 years ago.  That’s twice the number who said conducting polls was easier.

Will things get better? In 60% of countries surveyed, no change is expected.  In 28%, people don’t know what to expect.  In 10 countries, further restrictions are expected, and four of those countries are in South and Central America, an area where restrictions are already severe.

So what is the status quo?

In April of this year, AAPOR, WAPOR and ESOMAR emphatically condemned all forms of political intimidation targeting opinion researchers imparting relevant information and trends during an election, thereby allowing citizens to make a free and informed decision about their country's future. This followed on from the Ecuadorian Government's forced entry and arrest of staff from the Ecuadorian polling agency CEDATOS - an action which sets a dangerous precedent.  Under democratic law, there is no justification for such an action, and it thus contravenes international law that supports the freedom to share and disseminate information, the freedom of speech, and the right to fair and free elections.

This followed on from a situation in September 2016, when Russia’s most respected independent polling centre received the news that it had been labelled a "foreign agent".  The ruling came less than two weeks before Russia’s parliamentary elections and just after the Levada Centre had published findings of an 8% drop in the approval rating of the ruling party.  

A law established in 2012 requires all nonprofit organizations that receive (any) foreign funding and are engaged in loosely defined political activity, to register and declare themselves as foreign agents, a term widely associated with spying in Russia.  The measure hampered the Centre in conducting research as it required their staff to misrepresent themselves as foreign agents and not as independent researchers when conducting social and opinion research. This had an enormous impact on the organisation’s ability to assemble representative samples through voluntary participation of respondents in line with internationally accepted professional standards.

There are also countries – like the People’s Republic of China – where polls are not conducted around elections.  Pollsters do not ask about government leaders and parties.  But they do ask about some policies, and about social issues.

Furthermore, there are also some counter-intuitive trends in certain countries – such as Turkey and the Ukraine – where there are relatively few legally specified restrictions on polling, and the official blackout period is just one day prior to the vote!  But the survey may not have been able to assess any informal restrictions, including those that researchers might impose on their own work.

Notwithstanding these specific instances, the responses from the study suggest that in many places, there have been improvements, in methodology or in the variety of ways polls can be conducted.  In fact, in two-thirds of the countries surveyed, the methodological quality of most published polls is rated of high quality.  

However, what is not of high quality is the reporting of those polls. Less than one in four said the quality of poll reporting was of high quality.  

This is a more challenging issue, as in the tradition of "fake news", the explicit quality and possible limitations of the research methodology is very rarely reported.  In fact, quite the contrary… These days, the expectations placed on polls by the media, often far exceed the capability of the polls, and then when the results don’t fulfill those expectations, it is the research that is criticized, rather than the unreasonable expectations!

The three highest profile polls of recent times (the UK election, UK Brexit and The US Presidential election) all led to the polling industry becoming the subject of very fierce criticism, because of their "failure" to predict the outcome.  What we tend to forget is that the "prediction" part of a poll is usually made by the reporters or journalists – not the researchers – and so if a poll does not predict the outcome the first reaction is to question the poll, rather than the prediction.  

In statistical terms, the chances of three "missed" polls occurring in such close succession were calculated to be less than 1 in 100, and from a purely objective evaluation of the polling error data, the level of criticism was entirely undue. 

When we consider just how close these three elections were – all with victory margins of 4% or less, making any outcome prediction extremely difficult – as well as the fact that they were conducted in two markets with much lower than average voter turnout and higher than average undecided voters (and outcomes not directly reflecting popular vote share), you could even suggest that in many respect these polls were considerably more accurate than might have been expected under the circumstances.

As a direct response to this problem of misinterpretation of polls, AAPOR (the American Association for Public Opinion Research), WAPOR and ESOMAR have together produced an internationally applicable online training course for journalists, which provides clear guidance on what can (and cannot) be reasonably expected from a poll. The link to this course is: 

http://www.newsu.org/courses/understanding-and-interpreting-polls-international

In summary, the key finding from this most recent survey – supported by the data from the Kantar database – is that it is more important than ever to support and promote the factual reporting of opinion polls. Social and opinion research is used by officials and management around the world to gain better insight into citizens’ beliefs, attitudes and behaviours, to improve decision-making. In addition, such research is used by the media to report on citizens’ views about the society in which they live.  Any infringement on the right to conduct such research has a clear negative impact on the delivery of effective, evidence-based policy-making.

Kathy Frankovic : retired head of surveys CBS News, former President of AAPOR and WAPOR, member of ESOMAR’s Professional Standards Committee

Jon Puleston : VP of Innovation, Kantar Lightspeed,  member of ESOMAR’s Professional Standards Committee

Finn Raben:  ESOMAR Director General