Applying Moore’s Law to Data Growth

When Gordon Moore, co-founder of Intel, made this observation in 1965, he sought to a simple measure for the growth in integrated circuits, not the exponential growth in data.

Moore’s Law — which has since been refined to hold that chip density doubles approximately every 18 months — can not only apply to transistor density, but is equally applicable to enterprise data growth.

Consider this. Without the continual, exponential growth of hardware processing capabilities, would we have experienced a correlative explosion in data?

Which Came First, Hardware or Data?

It’s not simply computers generating terabytes of data, it’s all sorts of hardware, instruments, and sensors generating and collecting data. One estimate finds a single Boeing jet engine generates 10 terabytes of information for each 30 minutes it operates. That means a single trans-Atlantic flight of a conventional passenger jet creates 640 terabytes of data.

Tech research firm IDC, in a recent report on data growth, estimates the amount of data available doubles every two years. Across industries, across the world, more and more data is being generated and collected each day.

Does data play a role in the evolution of technology? Which came first: the data or the ability to process it?

Faster, bigger, cheaper … You could easily argue hardware spurred the growth in data collection; however, Ion Stoica, a director at the University of California, Berkley, Algorithms, Machines, and People Laboratory (AmpLab), observes:

“Data grows faster than the Moore’s law. According to one recent report, data is expected to grow 64% every year, and some categories of data, such as the data produced by particle accelerators and DNA Sequencers, grow much faster. This means that, in the future, we will need more hardware resources just to make the same decision!

Does More Data Lead to Better Decisions?

Stoica makes the case that collecting more data about a decision leads to more processing poser needed to arrive at a decision. Under that scenario, it could be that having more data is leading to more computing power and more time devoted to decisions that were made much more quickly and economically when there was not so much data to sift through.

Given the current focus on the enormous volumes of data, important issues are being glossed over. Dan Woods, writing on the Forbes website, contends we don’t fully grasp the issues associated with having so much data in the enterprise. He says it’s a mistake to be distracted by discussions of Moore’s Law and enterprise data:

A closer look at the world of data shows that there is no Moore’s Law in effect. More data just means more data. In many cases data is a liability. More data means more costs for storage, for governance and having too much unorganized data may make it more difficult to find what you need. … It is time to stop paying attention to the growing volume of data and start working on the mechanisms that will allow it to help create a better world.

Data needs to be processed to be converted into useful information. The database administrator and all her tools are needed to help the enterprise extract meaning from vast amounts of data, transforming it into actionable business intelligence. To do so effectively, she also must concentrate on challenges associated with system maintenance and issues commonly faced in tandem with the organization’s data growth. These challenges are compounded when new projects or new hardware goes online.

It takes persistent effort to thoughtfully, successfully marshal and manage your data. If your or your staff need more training or assistance with new data processing tools, you may wish to contact Datavail. We can help you design custom solutions as well as assist with a wide range of other tasks designed to help your organization more effectively and efficiently work with your data. By allowing us to tackle these tasks, we free your staff to focus on high priority, high value projects.

If you would like to know more about our outsourcing services, please contact Datavail now to discuss a custom solution for your enterprise.

Image by kentoh/123RF.