Confused About Big Data? Here Are 5 Things You Need To Do

It’s a funny line, but it seems positively quaint now.  It’s hard to imagine young entrepreneurs these days hoping to find their fortune in resource extraction. The days of wildcat wells have been replaced by an industry of geologists and geopolitics.

Every age has its great opportunity.  At the turn of the century, automobiles attracted a flurry of startups.  The consumer culture drove the post-war period and, more recently, finance was hot.  The killer technology of our time is undeniably big data and building a data science capability is crucial for every enterprise.  Here’s what you need to know:

1. Identify the Opportunity

For people who aren’t in a tech related industry, data science can seem hopelessly far afield.  After all, successful executives have gotten that way because they work hard, know their business and serve their customers well. Going to the time, trouble and investment to bring in some eggheaded data scientists just isn’t a priority for most managers.

Yet firms ignore big data at their peril.  A broad based study of 179 companies carried out by researchers at MIT and the University of Pennsylvania found that data driven firms performed 5%-6% better.  Compounded annually, that’s a decisive advantage.  A report by McKinseyfound that the gap could be as high as 60% in the retail sector.

James Manyika, a partner at McKinsey and a chief author of the report, points out that big data capabilities have little to do with company size, IT investment or even the type of business.  In fact, he sees some of the biggest opportunities for components suppliers and distributors who generate valuable data through their normal course of business.

2. Build Skills

Gartner predicts that there will be demand for 4.4 million big data jobs by 2015, with 1.9 million of those positions being created in the United States. However, the McKinsey report estimates that there will be a shortage of 140,000-190,000 data scientists and a deficit of 1.5 million managers who are capable of utilizing data driven insights.

The numbers tell a clear story.  Just as data skills are becoming crucial to compete in today’s marketplace, big data talent is becoming a scarce resource.  Any company without a concerted effort to recruit, train and integrate people with data skills will find themselves at a significant disadvantage.

Fortunately, as demand for talent is growing, so are the resources to meet it.  UC Berkeley has launched a fully online Master’s program in data sciencewhich executives can take while they remain at work. IBM IBM -0.17% has partnered with 1000 universities around the world to help meet the skills gap and even offers an open source curriculum for data science programs.

3. Collect Data

Probably the most important thing for managers to do is make data a priority.  We’re used to using information for a specific task, such as analyzing quarterly sales numbers together to generate market projections, but thinking about data as a resource in its own right is something relatively new.

McKinsey’s Manyika suggests that we focus on “three P’s”: Proprietary, Public and Purchased data.

Proprietary Data:  Many companies are sitting on treasure troves of information, without even knowing it.  As noted above, ordinary distributors and components suppliers often generate important data that can be reborn as entirely new businesses.  A company called Bounce.io has even figured out how to make a business out of the data from bounced emails.

Public Data:  A second important source are public databases that are offered free of charge, such as Data.gov.  So someone with proprietary industry data can correlate that information with broader economic trends.  These can be used to build a predictive model that can help improve decision making.

Purchased Data:  In order to satisfy demand a number of information marketplaces, such as Microsoft’s Windows Azure Marketplace, have sprung up that offer access to valuable resources such as point-of-sale data.

In the digital economy, information is power.  So you can’t afford to ignore the resources that are available to you.  Once you have data, you can start looking for valuable patterns that can propel your business forward.

4. Unify Your Architecture

We’ve come to expect consistency from the brands we do business with.  Go to a major retailer like Bloomingdales or The Gap GPS +0.69% and you’ll see similar brand communication, selection and level of service, no matter which location you find yourself in.

However, that consistency often breaks down between physical and digital brand channels.  See something you like on a mobile site as you walk down the street and then enter a store, chances are the brick and mortar staff won’t have the first idea what you’re talking about.  Their organizations—and therefore their systems—just aren’t designed that way.

Bobby Emamian, CEO of Prolific Interactive, a mobile strategy and development firm, thinks that much of the problem is due to a historical focus on platforms, rather than functions.  Firms build an in-store POS system, then a web site and then a mobile app, all with separate architectures.

In order to move quickly but remain effective, he recommends using an APIdriven approach to integrate legacy systems before building a completely new architecture.  His firm is currently working on proprietary technologies that will help manage and facilitate this process.

5.  Adopt a Big Data Mindset

The power of data to transform industries is often not so much a matter of resources, or even skills, but mindset.  A good example is the story ofGoldcorp, which was once a struggling mining business.  With the business in jeopardy and frustrated by the lack of progress, CEO Rob McEwan decided to do something truly revolutionary.

In what became known as the Goldcorp Challenge, he took 400 MB of proprietary data, put it online and offered $575,000 in prizes for anybody who could locate promising seams.  More than 1,400 contestants identified 110 new targets, 80% of which resulted in substantial new discoveries of gold. Goldcorp is now a $25 billion company.

Much like Goldcorp’s McEwan, to compete in a big data world, managers will have to defy convention and adapt a new mindset.  The traditional, step-by-step strategic planning approach will no longer do.  We’ll have to adopt a more adaptive, Bayesian approach to strategy, where we look to capture value from patterns we find in real time data.

One thing is clear—there is a growing dividebetween firms that can manage in a big data world and those that can’t.  If you want to be able to compete in the future, you had better start now.