He’s dead, Jim! … musing on Open Data in the age of peta-data

I made a startling realization today. an aha moment if you like. In a few years, the clamor for open data will die out.


Human behavior, and social economics, or at least my three pence thought on this issue, tells me as much. My reasoning is simple; we will only throw away what we no longer need (or ever will).

Over the past few years, the talk around open data, data science, big data … and all things data has grown louder. Everyone wants a piece of the action. Everyone wants to use their data to optimize processes, to make better decisions, to illuminate spending, … to be better. In advertently, what was previously trash (non-usable) is now being horded in abundant stores and servers. Suddenly, data has become that diamond in the rough. And with reactions reflective of its newly acquired status, the grip on data has tightened, and the grip is likely to grow tighter with every new spackle of the data diamond. We have collectively become compulsive hoarders

In 2012, every day 2.5 quintillion bytes of data (1 followed by 18 zeros) are created, with 90% of the world’s data created in the last two years … more

The question that arises is just how much of this data is open, or will ever be open. The reality is that almost a negligible percentage will ever be open, and even then, the data will have been hopelessly summarized and anonymized to be realistically usable. Human behavior suggests that we only throw out what we consider as garbage; what we do not hope to use; ever. And with such growth of data, the data dumps will be growing at an unprecedented rate; rates that will never be matched by the release.

Watching BBC documentaries trashopolis give a glimmer of hope. That even in the putrid mess of outdated, highly summarized, and distorted data, there will be those who will dive in and make some semblance of order, some meaning, some ideas. May be one may be able to build a beautiful island of trash data like these three amazing trash islands . May be, just may be, some George Waring(s)  will help make some order in the chaos of trash data.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s