1927 was a very good year for Werner Heisenberg, and, in an odd twist, those wrestling with Big Data and the identification of global events and trends that are shaping our future, a mere 85 years later.
Heisenberg was a brilliant physicist, yet his work on Quantum Theory and the Uncertainty Principle may help us shape how we look at many of the issues that we face today in the non-Brilliant-Scientist realm.
â€śOne can never know with perfect accuracy both of those two important factors which determine the movement of one of the smallest particlesâ€”its position and its velocity. It is impossible to determine accurately both the position and the direction and speed of a particle at the same instant.â€ť ~ Werner Heisenberg
Heisenbergâ€™s statement has been quoted, mis-quoted, adapted and modified to suit any number of ideas over the decades, so excuse me if I twist it myself to make a point.
In 1926 and 1927, when Heisenberg was laying the foundation for, and publishing, the Uncertainty Principle, we were in a world where Big Data didnâ€™t exist as we know it today. We were also far from being globally hyper-connected, and the idea of Pervasive Communications was a dream of the future.
TAKE A QUANTUM LEAP
I was recently having an interesting, and ongoing, Twitter discussion about Big Data and the value of Curation with some friends (Alan Berkson, Colin Hope-Murray, Peter Bordes and Robert Moore). In response to a question about the value of too much data, or data that was too old, I tweeted â€śold info doesn’t die, it reveals long-term trendsâ€ť.
As I looked at what I had written, Heisenberg (oddly, also part of the ongoing discussion) kept coming to mind, ultimately prompting the question â€śHow do we determine the long-term value of an event or data point, and ultimately the value of a trend if it lacks the right context?â€ť This question became all the more important as the different perspectives that frame â€ścontextâ€ť began to come to light. No two people see the same particle or event from exactly the same personal perspective.
THE RIGHT STUFF
It became increasingly apparent that our discussion of â€śtoo muchâ€ť Big Data was really about having the â€śright dataâ€ť. But how do you determine the right data? In many cases, you canâ€™t. Weâ€™ve plugged ourselves into this giant fire-hose of Social Media and canâ€™t digest it all.
In the end, most of us can only â€śsampleâ€ť off the feed. But in sampling, we get a very accurate description of what is happening at that particular moment, but we canâ€™t tell where what we are sampling fits into the bigger picture. Is this data â€śbyteâ€ť the beginning of a trend? Is it supporting a trend that already exists? Or is it perhaps signaling the evolution, or end, of a trend? Is it possible that we canâ€™t answer these questions unless we are continuously sampling from the buffet that is available courtesy of Pervasive Communications and our always-on data feed?
THE MEANING OF LIFE
As we talked a bit about this issue offline (if you consider a couple of hours on a Skype video call â€śofflineâ€ť), I came back around to the tweet about the value of old data revealing trends. Perhaps weâ€™re looking at Big Data and the online fire-hose in the wrong way. Too often we think we already know the questions, or we already know the trends, and we look at data points as a way to support our pre-existing notions (numerologists often have a particular knack for this â€“ you can find anything if you look hard enough in the wrong direction).
So rather than always trying to consume information to answer questions, what if we just taste the data, and let it help us form the right questions, regardless of the sector, the market or even what the data was originally supposed to represent? Why not let information from the Transportation sector be co-mingled with information from Politics, or Economics, or Energy. By doing so, weâ€™re helping to erase preconceived notions about the value of the data, and the answers we expect to get.
Ultimately, thatâ€™s what itâ€™s all about, isnâ€™t it? As Alan pointed out in his recent post Big Data: Is The Answer 42?, answers are meaningless if you donâ€™t understand the question, and with todayâ€™s glut of data, events and trends, figuring out the right question, and understanding why itâ€™s the right question, is more difficult than ever.
FINDING THE INFLUENCE OF UNCERTAINTY
Heisenberg talked about particles, their position and their velocity. Iâ€™m talking about events, their impact and their influence (their ability to form trends). In either case, the more certain we are of something, the less certain we are of something else. To me, that raises the question of value in being uncertain, to an extent.
Knowing the present state of an event or data-point is extremely valuable, as is knowing the direction it is heading. But equally important is the value of knowing why it is where it is at a particular moment and why it is heading in a particular direction (what influenced it, what shaped it). Following that lead, itâ€™s also important to know where it is heading and what it is going to hit (how will it influence something else).
â€śWhy a trend exists is just as important a question as asking what impact will result from the trend. Itâ€™s all about context.â€ť
As the data reveals more potential trends, so too does it raise more interesting questions:
- What value do individual events have, either as singular events or as part of a larger data set?
- How important are multiple layers of context and different perspectives?
- How do you anticipate when or how trends may collide or intersect?
The next time you sift through the data, as you swim through the stream, try squinting your eyes a bit. Donâ€™t focus so much on what you see, but rather let some uncertainty creep in, and see what patterns emerge when you see things just a bit â€śfuzzyâ€ť.
In the end, you might be surprised at what you do see, and the questions you start to ask.