Meta-analysis of Scottish independence polls (update 17th September 2014)

Since my post on Sunday a number of new polls have come out. This post shows the results of the meta-analysis using the six polls whose results have come out since Sunday.

As before I'm taking the rounded percentage results data from the whatscotlandthinks.org website (I should really go back to the original poll data and get the actual numbers). Six polls have been reported since Sunday night, when the last meta-analysis was posted. Using their data (as extracted from whatscotlandthinks.org), the approximate numbers in the polls (excluding undecideds) and Yes's are as follows:

Paper Poll company Poll start Poll end # exc. undecideds # Yes
Daily Telegraph Opinium 12/09/2014 15/09/2014 1087 520
Scotsman ICM 12/09/2014 16/09/2014 1011 482
Daily Mail Survation 12/09/2014 16/09/2014 920 440
STV Ipsos Mori 15/09/2014 16/09/2014 1349 660
Self-Funded or Unknown Panelbase 15/09/2014 17/09/2014 954 452
Self-Funded or Unknown Survation 16/09/2014 17/09/2014 910 430

Details about the meta-analysis methodology used can be found in my previous post. The results, based only on the these last six polls are shown in the forest plot below:

scottishindypollupdate1

The random-effects estimated mean is now 0.479 (95% CI 0.467 to 0.491) (full results from R are shown at the bottom of the post). There is thus reasonably strong evidence, as has essentially already been reported in the news this evening, that the polls are collectively pointing towards a very narrow victory for the No side.

In contrast, from last week's polls the 95% confidence interval straddled the 50% mark, due to the presence of the Sunday Telegraph poll which showed the Yes side in the lead.

Given how close the polls are to 50% now, a major deficiency in the analyses I have presented is that they are based on the % results of the polls (including undecideds) to the nearest whole percentage point. What I should be doing (if there were more time!) would be to go back to each poll's original data and extract the actual numbers.

Of course the above is based on the poll data. As many commentators have pointed out, there still remains a possibility that a different result could occur tomorrow due to a whole host of factors.

Results from R
In case anyone wants to see the raw results from R, they are:

                proportion           95%-CI %W(fixed) %W(random)
Daily Telegraph     0.4784 [0.4483; 0.5086]     17.45      17.45
Scotsman            0.4768 [0.4456; 0.5081]     16.22      16.22
Daily Mail          0.4783 [0.4456; 0.5111]     14.77      14.77
STV                 0.4893 [0.4623; 0.5163]     21.68      21.68
Panelbase           0.4738 [0.4417; 0.5060]     15.30      15.30
Survation           0.4725 [0.4397; 0.5056]     14.59      14.59

Number of studies combined: k=6

                     proportion           95%-CI  z  p.value
Fixed effect model       0.4789 [0.4665; 0.4913] NA       --
Random effects model     0.4789 [0.4665; 0.4913] NA       --

Quantifying heterogeneity:
tau^2 = 0; H = 1 [1; 1]; I^2 = 0% [0%; 0%]

Test of heterogeneity:
    Q d.f.  p.value
 0.85    5   0.9739

Details on meta-analytical method:
- Inverse variance method
- DerSimonian-Laird estimator for tau^2
- Logit transformation
- Clopper-Pearson confidence interval for individual studies

Leave a Reply