Feet of Clay

From Feet of Clay to Glimpses of Tomorrow

I’ve been thinking about a way to alter our perception of a raw material and I started off by pondering the properties of clay. I studied clay during soil mechanics classes at University and learned a lot about how this wonderful material behaves under pressure. I realize that I may have an appreciation of clay that most people don’t have. Clay is a relatively unattractive substance found in the ground and of seemingly little value. It can often have an offensive odor and it’s hard to perceive that this material could be the basis for future value.

London Clay

I lived in London for a while and many of the buildings are built from a famous type of brick made from yellow clay. These buildings, in many cases, carry an extremely high perceived value and yet began their life as humble clay. It seems that if you can think forward across the whole value chain from the basic material through the various conversion processes to the ultimate product that delivers value, then your perception of the original basic material will change.

Where am I going with this and how can clay have anything to do with Intelligent Transportation Systems? Well I would now like to apply the same thinking to transportation data. I think that if we were really honest we would admit that our perception of the value of raw data is fairly low. In fact, I’ve talked with some agencies who summarize the data and then throw the raw stuff away, while others don’t even bother collecting the data.

So if we can take the clay approach and develop a future vision along the process that converts data to information, insights and actionable strategies, then perhaps our perception of data will change.

Data is the raw material that will lead to insights that will change future transportation. So what do we do with this new higher value perception of data? Many people consider that the main issue is the inability to store large volumes of data because of cost constraints. Fortunately, our new perception of the value of data coincides with new abilities to store huge volumes of data at relatively low cost.

Data Lakes

Advances in data science allow us to store and manipulate data in ways that were not thought possible in the relatively short-term past. So now we have convergence between a new perception of the value of data and some incredible new abilities to store and manipulate it thanks to Hadoop, Google and Amazon. We also have the ability to structure and restructure the data and other to optimize our ability to retrieve and manage it. A new concept has emerged known as an “data lake” in which data from numerous sources is brought together into a single repository.

Note that this is called the “data lake” and not an “data swamp”. Then the data lake the data is clean, contiguous and easily accessible. This then enables you to take an enterprise wide view of data. I’ve noted in my work with several public transportation agencies that there is a natural tendency for staff within the agencies to build a “cockpit” around themselves. Making use of Excel spreadsheets and other tools, the staff create a toolbox of data and manipulation tools that are specifically designed to support the tasks involved in their job. Unfortunately, while this provides specialist support for the job in hand, it prevents an enterprise wide view of data.

With the capabilities of data science these days, it is possible to leave the staff with the cockpit while also copying the data to the data lake. Just like a real lake, it is possible then to use tools to waterski and to deep dive, exploring the data and revealing insights. It is worth noting that the concept of a data lake also means that you shouldn’t be too judgmental to early about the usefulness of data. It is possible that a seemingly useless piece of data can combine with another piece of data in the data lake to create a valuable insight.

Greatness and Grandeur

Next time you see a magnificent brick building give a thought to the humble clay that could have been perceived as worthless and yet served as the foundation for greatness and grandeur. Do you know where your data is and what it’s doing today, it could also be the basis for greatness and grandeur tomorrow?