Big data holds amazing promise, as indicated by the way its being used to study medicine. However, the vast majority of data is not analyzed. What holds back full use of your data?

• The Promise of Big Data: Example of Medicine
• Big Data Going Underutilized
• Archnemeses of Big Data
• Conclusion

The Promise of Big Data: Example of Medicine

Everyone in their first or second year at NYU School of Medicine must complete an assignment that taps into huge amounts of information. Students are able to study a massive database that contains hospital records throughout New York for two full years, totalling over 5 million records. Students can see diagnostics and basic demographics, with specific identities scrubbed for privacy.

The students are given analytic programs so that they can search for patterns and develop insights, notes the school’s associate dean, Marc Triola. These tools allow them to “look at quality measures for things like heart failure, diabetes, smoking and high blood pressure,” he says. “and drill down and look at the performance of the practice as a whole, and [the performance of] individual doctors.”

The school believes this project is fundamental in preparing students for an increasingly technological world.

Big Data Going Underutilized

Big data is incredibly valuable. In the context of medicine, it could even save lives. It‘s not always getting put to use though. Specifically, only about 0.5% of all data is studied today, says the MIT Technology Review. Why?

Let’s first look at the obvious issue of size. How do we fully grasp the huge amount of data we now have essentially at our fingertips? In any practical terms, the sheer volume of information that human civilization has accrued is in the realm of the absurd. As Lauren Browning of Business Insider notes, the amount of information that enabled the 1969 moon landing is actually less than what’s now in a typical PC. The total quantity of data is expanding very rapidly, doubling approximately every two years.

Data can give us incredible insights if we study it, as seen above. Of course, a major reason to collect information, other than simply wanting to access it, is to learn from it. However, that’s not easy to do, notes University College of London big data professor Patrick Wolfe. “The rate at which we’re generating data is rapidly outpacing our ability to analyze it,” he says. “The trick here is to turn these massive data streams from a liability into a strength.”

In other words, the vastness of big data can get in the way of developing these types of projects because what could be a treasure trove can at first seem to be an impossibly big, disorganized mess.

Archnemeses of Big Data

Size is just one of the things that aren’t working in favor of big data; let’s call it big data “archnemesis #1.” What else is holding back the use of your data?

Archnemesis #2 – Computing design

A major hurdle to overcome is that it’s difficult to integrate everything and know that the information is legitimate. The many sources from which data is collected make it challenging to maintain data integrity. It’s important to have safeguards in place that can tell if data might be inaccurate.

Archnemesis #3 – Junk science

With the rise of access to open source big data projects, some people who aren’t trained in data analysis have started releasing sloppy findings. Their findings often don’t completely make sense because they don’t know the skills or tactics used by data science professionals.

Archnemesis #4 – Lack of talent

You may simply not have the people-power to run big data projects and make use of things like Splunk managed services in order to derive reasonable, actionable conclusions. Their simply aren’t enough pros who are well-trained in analytics available to fill the business demand. You can train current employees, though, and work with automated software.

Archnemesis #5 – Inertia

Companies essentially use trial-and-error, through their own actions and by studying those of others, to refine their growth strategy. However, businesses do often get into unhelpful patterns that aren’t based on real evidence, notes Infogix CEO Sumit Nijhawan. “[B]y leveraging analytics, organizations can evolve their existing classical decision making process based on past learnings or intuitions, into a logic-based decision support system that is based on evidence,” he says. The only issue is that many organizations don’t yet use analytics when they make decisions.

RELATED: If you want to try a big data project at your organization, you need the speed and efficiency of a cloud server. At Superb Internet, we use a distributed design (no bottlenecks) and InfiniBand (dozens of times lower latency than the theoretical minimum of 10 GigE). Learn more.

Archnemesis #6 – Labeling

Clearly it is not easy for companies to manage or interpret the massive amount of data that their business now controls. Business leaders want the IT directors to tell them where the data is and what it’s value might be. IT pros often don’t know what the data is exactly, though. It’s not labeled or classified, so it’s unclear if the information relates to customers or sales or even employees. Data classification is a must: it will give you a sense what you need to keep and what can be tossed, both now and in the future.

Conclusion

There are many forces working against big data, as indicated above. Hopefully knowing what might hold you back will make it easier for you to succeed and leverage your information to its fullest possible extent. Clearly big data is valuable and, if used wisely, can give you a large competitive advantage.