Werner Heisenberg

Mr. Heisenberg meets #BigData?

1927 was a very good year for Werner Heisenberg, and, in an odd twist, those wrestling with Big Data and the identification of global events and trends that are shaping our future, a mere 85 years later.

Heisenberg was a brilliant physicist, yet his work on Quantum Theory and the Uncertainty Principle may help us shape how we look at many of the issues that we face today in the non-Brilliant-Scientist realm.

“One can never know with perfect accuracy both of those two important factors which determine the movement of one of the smallest particles—its position and its velocity. It is impossible to determine accurately both the position and the direction and speed of a particle at the same instant.” ~ Werner Heisenberg

Heisenberg’s statement has been quoted, mis-quoted, adapted and modified to suit any number of ideas over the decades, so excuse me if I twist it myself to make a point.

In 1926 and 1927, when Heisenberg was laying the foundation for, and publishing, the Uncertainty Principle, we were in a world where Big Data didn’t exist as we know it today. We were also far from being globally hyper-connected, and the idea of Pervasive Communications was a dream of the future.

TAKE A QUANTUM LEAP

I was recently having an interesting, and ongoing, Twitter discussion about Big Data and the value of Curation with some friends (Alan Berkson, Colin Hope-Murray, Peter Bordes and Robert Moore). In response to a question about the value of too much data, or data that was too old, I tweeted “old info doesn’t die, it reveals long-term trends”.

As I looked at what I had written, Heisenberg (oddly, also part of the ongoing discussion) kept coming to mind, ultimately prompting the question “How do we determine the long-term value of an event or data point, and ultimately the value of a trend if it lacks the right context?” This question became all the more important as the different perspectives that frame “context” began to come to light. No two people see the same particle or event from exactly the same personal perspective.

THE RIGHT STUFF

It became increasingly apparent that our discussion of “too much” Big Data was really about having the “right data”. But how do you determine the right data? In many cases, you can’t. We’ve plugged ourselves into this giant fire-hose of Social Media and can’t digest it all.

In the end, most of us can only “sample” off the feed. But in sampling, we get a very accurate description of what is happening at that particular moment, but we can’t tell where what we are sampling fits into the bigger picture. Is this data “byte” the beginning of a trend? Is it supporting a trend that already exists? Or is it perhaps signaling the evolution, or end, of a trend? Is it possible that we can’t answer these questions unless we are continuously sampling from the buffet that is available courtesy of Pervasive Communications and our always-on data feed?

THE MEANING OF LIFE

As we talked a bit about this issue offline (if you consider a couple of hours on a Skype video call “offline”), I came back around to the tweet about the value of old data revealing trends. Perhaps we’re looking at Big Data and the online fire-hose in the wrong way. Too often we think we already know the questions, or we already know the trends, and we look at data points as a way to support our pre-existing notions (numerologists often have a particular knack for this – you can find anything if you look hard enough in the wrong direction).

So rather than always trying to consume information to answer questions, what if we just taste the data, and let it help us form the right questions, regardless of the sector, the market or even what the data was originally supposed to represent? Why not let information from the Transportation sector be co-mingled with information from Politics, or Economics, or Energy. By doing so, we’re helping to erase preconceived notions about the value of the data, and the answers we expect to get.

Ultimately, that’s what it’s all about, isn’t it? As Alan pointed out in his recent post Big Data: Is The Answer 42?, answers are meaningless if you don’t understand the question, and with today’s glut of data, events and trends, figuring out the right question, and understanding why it’s the right question, is more difficult than ever.

FINDING THE INFLUENCE OF UNCERTAINTY

Heisenberg talked about particles, their position and their velocity. I’m talking about events, their impact and their influence (their ability to form trends). In either case, the more certain we are of something, the less certain we are of something else. To me, that raises the question of value in being uncertain, to an extent.

Knowing the present state of an event or data-point is extremely valuable, as is knowing the direction it is heading. But equally important is the value of knowing why it is where it is at a particular moment and why it is heading in a particular direction (what influenced it, what shaped it). Following that lead, it’s also important to know where it is heading and what it is going to hit (how will it influence something else).

“Why a trend exists is just as important a question as asking what impact will result from the trend. It’s all about context.”

As the data reveals more potential trends, so too does it raise more interesting questions:

  • What value do individual events have, either as singular events or as part of a larger data set?
  • How important are multiple layers of context and different perspectives?
  • How do you anticipate when or how trends may collide or intersect?

The next time you sift through the data, as you swim through the stream, try squinting your eyes a bit. Don’t focus so much on what you see, but rather let some uncertainty creep in, and see what patterns emerge when you see things just a bit “fuzzy”.

In the end, you might be surprised at what you do see, and the questions you start to ask.

  • Philip Sheldrake

    Thanks for the good read Fred. I think you’re touching on real-time complex event processing here (eg, http://www.oracle.com/technetwork/middleware/complex-event-processing/overview/index.html )

    • http://fredmcclimans.com/ Fred McClimans

      Thanks for the feedback and the link. Yes, CEP fits into this model, with the use of a rule-based decision engine as the EDA (to handle semantic structures, classification, linkages between events/trends, etc.). Not an easy task, and deeper than I can dive ;) But well worth the effort!

  • CHopeMurray

    I agree the Heisenberg Uncertainty Principle appropriately describes Big Data. Your observation definitely challenges us to step back and view events and information less discretely in order to detect context and patterns.  I also agree that old information never dies as it can reveal long term trends.  However I believe it has more value than just a data point in an unknown number of trend cycles.  I see old information as a meme, or thought, waiting to be rediscovered either individually or as part of the previously noted pattern.  An item of information is a meme or set of memes packaged for consumption , so old data can always be relevant they just require the efforts of the perceptive and diligent to see the thought(s) behind the wrapper.

    • http://fredmcclimans.com/ Fred McClimans

      Hey Colin – Thanks for bringing up the meme concept. You are definitely on to something there. I think the key to finding/interpreting the “old data/new memes” is in varying the perspective. In that sense, old data never dies, it just waits for a new perspective to arrive. Moving forward, I’m very curious how much information will, by default, get bundled into super-memes out of necessity, and if we will lose a bit of insight into the data (and its then-current application) or if this will actually simplify things – complex data > simple memes. I’d welcome your thoughts.