Nowhere has big data made such an impact than in politics, so at EMC we're watching with interest to see how statistics reported on a daily basis in the media influence the voters' decision on September 18.
It was in 2012, during Barack Obama's campaign for a second term in office at the White House, that big data analytics earned its reputation as the tool which ensured victory.
Over a two-year period, the US President's campaign team collected, stored and analysed voter data collected from pollsters, fundraisers, field-workers, social networks and Democrats' voter files in the swing states.
They then knew what voters liked to eat, how long it took them to take their kids to school, what they did for a living, and how they planned to celebrate their birthdays.
This same data was also mined to help Obama raise $1 billion, hone TV ads and create models of swing-state voters that could be used to boost the effectiveness of phone calls, door-knocks, direct mailings and social media. Campaign messages were micro-targeted to appeal to individual voters. Just as Dwight Eisenhower used radio in the 1950s and John F. Kennedy deployed the power of television in the 1960s, Obama leveraged big data analytics to win in 2012.
Few events in American life other than a presidential election touch 126 million adults, or even a significant fraction that many, on a single day. Certainly no corporation, no civic institution, and very few government agencies ever do.
Obama did so by reducing every American to a series of numbers. Yet those numbers somehow captured the individuality of each voter, and they were not demographic classifications. The scores measured the ability of people to change politics - and to be changed by it.
So is this big data analysis being replicated by the Yes and No campaigns currently in Scotland? There's no doubt that within each campaign team there will be expert analysis taking place on a daily basis in relation to potential Scottish voters, whether those voters know it or not.
Each and every one of us generates vast amounts of information, or 'digital shadows', on a daily basis. Smartphones, social networks and other devices, including PCs and laptops, have allowed billions of people around the world to process huge amounts of information.
We upload pictures, audio and personal information to Facebook, while Twitter records billions of online conversations between people across the planet. This information is creating opportunities for organisations to generate individual-specific profiles, uncovering patterns to produce insights into what makes each one of us tick.
Just think of some of the online interactions many of us conduct daily. Facebook and LinkedIn use patterns of friendship relationships to suggest other people we may know, or should know, with sometimes frightening accuracy.
Amazon saves our searches, correlates them with others' searches and product reviews, and uses the results to generate surprisingly appropriate recommendations. How this feedback loop is mined is the beginning of big data analytics.
Evidence from various polls and studies so far in the referendum campaigns clearly show that Yes is winning the battle for online support through social media - but this lead doesn't translate through to the opinion polls, where Better Together is still significantly ahead.
With just over two months to go it will be fascinating to see both sides pick up the pace of their campaigns and see if they adopt any changes or new approaches that come from their analysis of voter data gleaned through these social media interactions.
Doubtless there are insiders in each camp who are documenting everything that's influencing their respective campaigns and we will hear about what worked and what didn't in due course.
Consider all this in the context that the digital universe is doubling in size every two years according to research results announced by EMC recently, and will multiply 10-fold between 2013 and 2020, from 4.4 trillion gigabytes to 44 trillion gigabytes.
The massive amounts of 'useful data' - data that can be analysed - contained within this digital universe is also growing. In 2013, only 22% of the information in the digital universe was considered useful, but less than 5% of it was actually analysed. By 2020, more than 35% of all data could be considered useful, if businesses are able to analyse and use it.
One thing is for certain: big data is here to stay and it's not just politicians who are seeing the enormous potential it can deliver to today's society.
Martin Brown is EMC Country Manager for Scotland. It has been based at Livingston since 1999, and a second base for the oil and gas industry operates in Aberdeen.
Why are you making commenting on The Herald only available to subscribers?
It should have been a safe space for informed debate, somewhere for readers to discuss issues around the biggest stories of the day, but all too often the below the line comments on most websites have become bogged down by off-topic discussions and abuse.
heraldscotland.com is tackling this problem by allowing only subscribers to comment.
We are doing this to improve the experience for our loyal readers and we believe it will reduce the ability of trolls and troublemakers, who occasionally find their way onto our site, to abuse our journalists and readers. We also hope it will help the comments section fulfil its promise as a part of Scotland's conversation with itself.
We are lucky at The Herald. We are read by an informed, educated readership who can add their knowledge and insights to our stories.
That is invaluable.
We are making the subscriber-only change to support our valued readers, who tell us they don't want the site cluttered up with irrelevant comments, untruths and abuse.
In the past, the journalist’s job was to collect and distribute information to the audience. Technology means that readers can shape a discussion. We look forward to hearing from you on heraldscotland.com
Comments & Moderation
Readers’ comments: You are personally liable for the content of any comments you upload to this website, so please act responsibly. We do not pre-moderate or monitor readers’ comments appearing on our websites, but we do post-moderate in response to complaints we receive or otherwise when a potential problem comes to our attention. You can make a complaint by using the ‘report this post’ link . We may then apply our discretion under the user terms to amend or delete comments.
Post moderation is undertaken full-time 9am-6pm on weekdays, and on a part-time basis outwith those hours.
Read the rules hereComments are closed on this article