A deep dive into comScore and Nielsen’s measurement tactics
Over the past decade, digital measurement heavyweights Nielsen and comScore have become much better at measuring audiences with precision, which makes it that much more perplexing when their benchmarks for audience size and engagement are wildly different.
The discrepancy is huge, with comScore tracking total traffic that is roughly four times that of Nielsen. And the gap is significant, not just because the figures inform the industry’s perception of consumer trends, but because they influence ad buyers and industry-wide investment flows. If consumption is overstated, maybe companies and marketers have overinvested and should seek better places to put their money. Keep that in mind while Ad-Vantage does a long, deep dive into digital video measurement.
The issue got fresh legs recently when Wall Street ad industry equity analyst – and regular AD-Vantage contributor – Brian Wieser stirred the pot by digging into discrepancies between Nielsen and comScore’s measurements of online video consumption in the U.S.
That, in turn, got Ad-Vantage doing some more digging of its own to try to reconcile such a wide gap, but we’ll get to that in a minute. First, let’s look at Wieser’s findings, which were published in a Pivotal Research Group report and were picked up in some industry media coverage in the U.S.
Wieser tabulated and compared U.S. consumption data from monthly comScore Video Metrix reports and Nielsen quarterly VideoCensus figures to figure out how much video consumption had grown over the last five years. What he found was a dramatic contrast. ComScore data showed overall monthly video consumption in the US grew 448% from Q4 2008 to Q4 2013. Nielsen data, on the other hand, showed video consumption grew only 228%.
Data from Pivotal Research Group, 2014
Over that period, Wieser found comScore has at various times reported between two and five times more video consumed in the U.S. than Nielsen (3.9 times on average). As of December 2013, comScore’s total was three times that of Nielsen. That is a huge gap. Now, before we go any further, let’s add that Wieser says both measures are clearly inaccurate, given they don’t yet include tablet usage and have other imperfections. But they are also the best the industry has. In his report, Wieser argues the real amount of consumption is very close to where Nielsen puts it.
AD-Vantage decided to do a little more digging to better understand the data, and spoke to both companies to get information about their methodologies. Why? Clearly, we’re measurement nerds and gluttons for pain. We found more digging reveals how complex measurement can get, even in the age of digital advances, and raises even more questions about how accurate a picture these leading measures paint.
WHERE DO THESE NUMBERS COME FROM?
We looked at the most recent figures released by the two companies: comScore’s February video rankings and Nielsen’s Q1 2014 Cross-Platform Report. ComScore says U.S. Internet users collectively watched 3.3 billion hours of video in February, while Nielsen says they watched 1.2 billion hours in the same month (a Q1 monthly average). To put those numbers in perspective: online video equates to either 2.4% (Nielsen) or 6.5% (comScore) of all video watched across TV and desktop web combined.
Total monthly consumption is calculated by multiplying total unique viewers by the average amount of time each viewer watches per month. Breaking down consumption into those two constituents provides some indication of where the discrepancy may be coming from.
How? Well, while comScore counts 30 million more viewers, the bigger factor appears to be viewing time: comScore says each viewer is watching, on average, 10 more hours of content each month. That’s roughly five more episodes of How I Met Your Mother, or 50 cat videos on YouTube, each week.
Data from Nielsen, Cross-Platform Report Q1 2014 and comScore, Video Rankings for February 2014
Digging into both companies’ measurement practices, we found that on most points, Nielsen’s and comScore’s measurement of total video consumption is similar. According to information supplied by both companies, both comScore’s Video Metrix and Nielsen’s VideoCensus count all measurable kinds of video. These include: progressive downloads and streams; auto-play and user-initiated video; and all common online video players. Neither includes mobile video.
Both count only time actually watched, not the full duration of videos loaded. ComScore has a three-second cut-off, below which a video is not considered viewed; Nielsen doesn’t. But if anything, that suggests the amount of time comScore measures should be slightly smaller compared to Nielsen, not larger.
There are a few key differences. Nielsen only credits streams where audio is present, which may have some impact. Perhaps more importantly, comScore’s measurement of total video consumption does not exclude any video based on its content – its Internet-wide viewership figures include adult entertainment, pirate streams, etc, even though comScore does not rank these sites or do business with them. Nielsen’s total viewing time specifically excludes adult content.
IS PUBLISHER DATA THE REASON FOR THE DISPARITY?
As far as methodology for determining audience numbers from collected data, both comScore and Nielsen use a complicated combination of passive panel measurement and direct publisher measurement, which essentially means they use data submitted from publishers about how many videos are played on their sites, and adjust this using data from panelist’s recorded viewing habits.
Both sides of the measurement are necessary: publisher data catch more views and expand the universe of measurement, but tend to be inflated, since publishers have an incentive to aggressively measure traffic. Panel data are limited in reach, but more grounded for determining how much video was actually viewed, versus how much was served or loaded. Panel data also tend to be better than publisher data at counting unique viewers, but as we’ve seen comScore’s and Nielsen’s unique viewer numbers are at least in the same ballpark.
ComScore prides itself on its “tag penetration” — the number of publishers that submit data for direct census measurement. It has been using publisher data for longer than Nielsen has, and has more publishers and networks on board. “Currently, our solution has tagging participation (i.e. direct measurement) from the vast majority of the top 100 video content publishers, which enables a more accurate accounting of video viewing duration,” comScore said in a statement about the discrepancy. ComScore says it has 80% penetration with the top 50 publishers, ranked by viewing time per viewer.
“Video Metrix includes measurement of video ads, long tail content publishers, and non-traditional video formats. All of these factors can ultimately provide significant sources of difference at the total market level between measurement services that have a more limited reporting universe and/or technological limitations to what they can report.” (Note that although Video Metrix does measure video ads, ad time and content time are counted separately in their reports.)
ComScore vice-president, marketing and insights, Andrew Lipsman, added: “We feel very good about our measurement. The strongest validation is the number of clients that use it.”
Nielsen, following its strategy in broadcast measurement, has focused on tagging major publishers as a proxy for the broader market. According to Nielsen vice-president, product leadership, David Wong, it receives data from 13 of the top 25 publishers.
When asked whether Nielsen is concerned the disparity generally reflects poorly on the video measurement industry, Wong said: “Not at all. We stand by all of our published data and methodologies for measuring video content and advertising. Our rigorous measurement practices are among the reasons the media industry trusts Nielsen data – both in the linear TV world and in the rapidly growing digital ecosystem – and uses our metrics to facilitate the transaction of billions in ad spend annually.”
One possible reason comScore numbers track much higher could be its greater tag penetration, which enables it to catch more traffic from medium-sized and long-tail websites. According to comScore’s data, 51% of total viewing time comes from the top 25 publisher sites ranked by total time, which includes all of the heavyweights like YouTube, AOL and Netflix. Nielsen, on the other hand, says that 83% of viewing time comes from the top 25 publisher sites.
That means that per comScore, all other sites made up 1.6 billion hours of total viewing time in February. Per Nielsen, it was 957 million hours.
Data from Nielsen, Cross-Platform Report Q1 2014; comScore, Video Rankings for February 2014; and data supplied by Nielsen and comScore
One reason for the gap may be that comScore measures a much larger long-tail because its greater tag penetration gives it a better view into the nooks and crannies of the web. But on the other hand, both comScore and Nielsen note that publishers are more likely to measure their own views “aggressively,” which is why panels have to be used to adjust publisher data. So it may be that because Nielsen relies more heavily on its panel than on publisher data, it’s doing a stricter job of screening out publisher-inflated impression counts. It may also be a little of both.
To cast some doubt on the long-tail answer, there’s also the fact that Nielsen and comScore differ significantly in their measurement of the top 25 publishers’ combined viewing time. In February, comScore said viewers watched 1.7 billion hours of video on the top 25 publishers’ sites, which was more than the total consumption measured by Nielsen for all sites that month. So it’s not just measurements of the long-tail that diverge, but measurements of prime web properties as well.
WHY DOES THE DISPARITY MATTER?
At this point, let’s remind ourselves why all of this matters, as Wieser took pains to stress in his piece: Could there be a real investment gap in video?
According to estimates from eMarketer, the U.S. spent $4.1 billion on desktop online video and $66.4 billion on TV in 2013. That means online video spend was 5.8% of combined budget for TV and online video. Based on comScore’s estimate that online viewership makes up 6.5% of all video seen (excluding mobile), that would suggest advertisers are slightly under-investing in video (more so in Canada). But if Nielsen is right, and online video is only 2.4% of all video time, advertisers are significantly over-investing in video, and should be dialing back in favour of TV or other digital channels like search.
Wieser makes it clear in his report that he believes Nielsen has more accurate measurement for total viewing consumption. ComScore’s consumption numbers are at odds with the realities of the market over the past decade, he says. It seems unlikely that online video had reached more than 10% of TV viewing levels by 2011. “If the absolute scale is correct, Cisco would have made far more money than they ever did. Nortel would never have gone bankrupt. It tells a completely different story of the universe,” he told AD-Vantage in an interview.
However, he also notes in his report that comScore data is typically used for publisher rankings, not overall consumption. He writes that digital media buyers generally prefer comScore for much of the work they do because “its products are perceived to measure online audience share more accurately, and share is critical in determining which publishers should be included on an RFP associated with a media buy.” He says that when it comes to rankings, knowing how much video is being watched in total across the Internet is usually not relevant.
It’s impossible to say with any certainty whether comScore or Nielsen is closer to the real amount of video being watched. But with the two estimates continuing to diverge, it’s unlikely total video consumption will be reconciled any time in the near future. If nothing else, the discrepancy serves as a reminder to marketers to be careful of blind faith in data.
