The Wrap: What to Know Before Taking a Deep Dive into Dark Data

As more emphasis is placed on collecting and storing electronic information of all types, one of the critical problems most companies face today is how to make best use of an increasing mountain of structured and unstructured material.

Only about 0.5% of the data that exists in a typical enterprise is actually being analyzed and utilized as business intelligence, according to management consulting firm McKinsey & Co.  In most cases, the data is just sitting there taking up space although, if managed, organized and analyzed to uncover patterns and outliers, it can provide valuable intelligence for current and future needs of a business.

The CIO Executive Council (CEC) recently tackled the issue of the data deluge in a Power Hour webcast that specifically focused on the area of so-called Dark Data, which is essentially information assets that are collected, processed and stored by organizations that may have short-term use but may not be used for other purposes.  Leading the session were: David Tennenhouse, Chief Research Officer at VMware, Inc.; and James Rinaldi, CIO at NASA’s Jet Propulsion Laboratory.

Some key highlights of the interactive session:

  • Most companies have too much data and no strategic plan to sift through it to discover useful intelligence that can be used. Much of the data may be invaluable or long past its period of usefulness. But, you won’t be able to find valuable information unless you have a plan to identify the wheat from the chaff.
  • Right now, there are not a lot of useful tools or knowledge to adequately analyze Dark Data. Although technologies like machine learning promise to make the process go faster and be more reliable, there will be a data analysis learning curve to overcome before these tools can be trusted.
  • Sifting through massive amounts of data to find what is useful and not useful takes time. There are no shortcuts, say the experts, since you can’t really skip any steps in the analysis and discovery processes. If you do, then you may miss seeing insights that lead to other insights that eventually uncover that intelligence needle in a haystack.

The CEC members leading the discussion both noted the importance of establishing a data analysis game plan before diving into the deep end of a Dark Data pool. Most important, have some idea of what you are looking for and how you plan to use it before searching for business intelligence.

“Make sure a data strategy is part of your digital transformation plan and that you take the opportunity to look at what you’re going to do with the data, how you want to do it, and how you free the data so it can be used for multiple purposes in your whole transformation scenario,” says JPL CIO Jim Rinaldi.  “You also have to convince people their data is valuable…and you do that with great demonstrations and visualizations and things that show what has potential.”

If you’re interested in participating in virtual sessions like this and others that target strategic IT issues or would like to learn more about the CIO Executive Council, please contact us