• Junk Pile #1 – Walmart
• Junk Pile #2 – Best Buy
• Junk Pile #3 – US Federal Government
• Junk Pile #4 – Unnamed Retail Company
• Strategically Taking on Big Data
• What About the Value of Small Data?
• Better Performance = Happier Customers

“Hooray for big data! Let’s all bathe ourselves in it. It will refresh Internet opportunity, serving its timeless function as the Digital Fountain of Youth.” – anonymous

“This whole new phenomenon seems like nothing but job security for data nerds who want to protect their huge budgets.” – Infofree.com founder Vin Gupta

Big data is a huge movement right now in business. It’s easy to assume that the buzzword is a mythical bandwagon, but IDC predicts the total CAGR for big data hardware and software will be 26.4% through 2018, that year achieving $41.5 billion in revenue.

As you can see above, though, not everyone is onboard. That’s because many data projects are overfunded and ill-conceived.

Junk Pile #1 – Walmart

Walmart has a massive database, including information on their customers, prices, and products. They probably expend more than $1 billion each year on data management and analytics, but their revenues remain flat.

Junk Pile #2 – Best Buy

Similarly, Best Buy is pouring money into wide-scale projects designed by data scientists. However, sales are falling because their site isn’t user-friendly.

Junk Pile #3 – US Federal Government

The federal government also seems to not know what it is doing with data. Even at a time when they were using surveillance in a basically unfettered manner, tapping our phones and reading all our emails, they couldn’t stop the Boston Marathon bomber.

Junk Pile #4 – Unnamed Retail Company

Gupta described the experience of discussing big data with the chief executive of a huge retail company, which has a sophisticated system that is capable of geo-locating customers anywhere, even within shopping malls.

“He bragged to me that they had spent millions of dollars on software and so-called expert data miners to identify the high-value customers and use social media to create buzz and generate revenue from these customers,” Gupta said. However, the CEO started talking in circles when asked about revenue.

Big data certainly has promise. But Gupta is right: many organizations both large and small will botch their big data projects. Without an adept data scientist at the helm, organizations often just spin the wheels and waste a bunch of money.

Strategically Taking on Big Data

The issue is that big data has become a sort of panacea concept: leverage your data, and it will cure all your ills. All the analytics services are competing so that you can market and build your business in the most intricate ways. Silicon Valley data scientists are the new 49ers, entering your data mines to look for gold. The fact is, though, expert data scientists are an extremely rare breed.

Without that expertise, you have a junk pile. Yes, there is a lot of value in that junk pile. It’s impossible to take it apart piece by piece though. You must have someone with the tools and know-how to convert that junk into profit (a data-flipper?).

Enterprises gather and scrutinize data related to everything that they sell, everyone who uses their systems, and customer shopping patterns from various places – POS systems, web polls, tech providers, and social accounts. They love to gather the information, assuming that it will prove itself valuable.

Once the data is in the hands of an organization, it often just stays in junk form.

“Corporations will spend millions of dollars to analyze worthless data for worthless results,” said Gupta. “These expensive, arrogant data miners will create big reports, complicated analytics, but … they have nothing to show for it.”

What About the Value of Small Data?

We all know the basic idea that less is more in many scenarios. The rush of activity related to big data can forget the idea that you might be able to glean more value from a small amount of extremely relevant information. You can find out what that information might be by talking to a few of your customers directly. Figure out which ones aren’t using you anymore and try to determine why they departed.

You don’t necessarily need huge amounts of data. Just because your data is there doesn’t mean it has to be analyzed, either. You don’t always need to organize what’s in your closet. Sometimes, just throw it all away. Small data is often more effective to improve the direction of your business.

“In most cases problems are very simple, and the answers are simple, too,” said Gupta, “like answering the phone by a live person, helping customers right away, and treating them like family.”

Better Performance = Happier Customers

Whether you want to work in the realm of big data or small data, you will need systems that allow your customers to experience cutting-edge speed and reliability. Using InfiniBand rather than Ethernet and distributed architecture over centralized, mainframe-style storage creates a 100% HA cloud that can quadruple the pace of Amazon and SoftLayer.

By Kent Roberts