Avi Mayer did the cause of honesty and openness a service when he dug into the data of the Dialog poll published in Haaretz this week. According to the report in the paper, Israelis support apartheid, want separate roads for Arabs and are all bad people.
Avi was suspicious, and managed to get a fuller version of the survey:
I approached a number of the organizations and individuals involved in commissioning the survey. One of the individuals I contacted was Professor Amiram Goldblum [who]… kindly shared the full findings with me and openly shared his thoughts, as well.
The actual results were more nuanced. While not positive, they were misrepresented by the Haaretz article:
In the op-ed, Levy writes: “The majority wants segregated roads in the West Bank and does not flinch in the face of the implications. Even the historic connotation does not bother it in the slightest….”
The question reads: “In the territories, there are roads on which only Israelis may drive and others [on which] only Palestinians [may drive]. Which of the following views is closest to your own: This is a good situation, this is not a good situation but there is nothing to be done, or this is not a good situation and it must be stopped.” 24% selected the first option, 50% the second, 17% the third, and 9% said they “don’t know.”
There is no “74 percent majority in favor” of separate roads, and the majority does not “want” them, as Levy claims. Fully two thirds — 67% — of Jewish Israelis say the existence of separate roads in the West Bank is “not good.”
Who is to blame for this bad reporting?
This is a busy time for polling. a few Israeli election polls are published every week. With the American elections only days away, national and swing state polls are coming out almost hourly. Britain, my birthplace, has a daily tracking poll from research company YouGov, as well as regular polls from other companies.Political operatives rely on polls. They aren’t perfect, but they’re the best guide we have for predicting how people will behave in elections and the easiest way of testing public opinion. But they change behaviour as well as passively predicting it. Parties live and die by polls; leaders resign and laws are made. Politics is the art of the possible, and polls tell us what’s possible and what is not. So it’s important that we can trust our polls and our pollsters.
Of the three countries I mentioned above, Britain has the best system for creating confidence in the polling firms. All of the major research companies are members of a voluntary body, the British Polling Council (BPC).
By joining the BPC, these pollsters agree to a standard of openness and disclosure. If they’re hired to do a private poll they’ll only share the results with their client. However, they always release the full data for their public polls. A BPC member publishes all the questions asked in the poll, who commissioned it, what the unweighted data looks like – usually broken down by sex, region, age, socioeconomic class and party ID – and then the weighted subgroups to produce the final numbers. The data released for just a single question in a poll covers a page with hundreds of numbers:
This is really useful. Because a good poll doesn’t just ask 500 random peole what they think about an issue. That wouldn’t work. A good poll needs a good sample – one which is representative of the whole. Different companies have different ways of ensuring that their poll samples are representative; one, for example, uses what newspaper people read. Another weights for public vs private sector workers. Often they weight according to what party people voted for at the last election, but even here most companies don’t weight to the actual results of the previous election, because people have a tendency to “misremember” who they voted for according to what parties are popular today.
These different approaches have their fans and opponents. A real political expert (or geek – take your pick) can argue for their favourite pollster and pick holes in the methodology of those they don’t like, can do sneaky subgroup analysis while admitting that it’s probably statistically invalid and, when a poll comes out, always asks “Can I see the full tables?”.
That’s not all though. If someone commissions a private poll from a BPC member and then later publishes some of the results, the company will release ALL of them whether the customer likes it or not. There’s no way to cover up what questions were asked.
Israeli polls, by comparison, are a black box. Every couple of days a new voter intention poll arrives with final numbers only and a sample size. Israeli polls usually have a margin of error of 4%, which in a voter intention poll is about ± 4 seats. That’s pretty high.
How are Israeli polls weighted? I don’t know because we aren’t told. Were there originally other questions on the poll that are being covered up? We have no idea. If Avi Mayer hadn’t got hold of the full summary of the Haaretz poll, we still wouldn’t know whether to trust it or not.
Presumably the political parties have access to this fuller data in their own private polls. The Labour party is reportedly hiring Stanley Greenberg to do their private polling, who I’ve worked with before and always been impressed. The new Likud-Yisrael-Beitenu uberparty apparently uses Arther Finkelstein. Both are serious professionals who would give their clients every piece of useful information.
But the public pollsters – Dialog, Teleseker, Panels Politics and the rest – don’t give this same information to the public. So we never know how they reached the numbers they produced; we never know what findings they’re covering up; and we never know whether to trust them.
The British Polling Council was founded to restore public confidence to polling, and was modelled on the USA’s National Council for Public Polls, which does a similar job in America. There’s no reason why these big polling companies couldn’t join together to found an Israeli Polling Council. The public would be better-informed if they did. It wouldn’t stop dodgy poll reporting like that in Haaretz, but it would make it easier to expose in the future.