One of the more important revolutions within the data analytics community is the surge of data from the Internet of Things (IoT). IoT is a network of “things”: Things such as devices that are worn, buildings, automobiles and more. These things contain sensors and software that enable each of them to connect to the other and exchange information.  

What is the point of all of this connectivity? Well, the thing is (sorry, I couldn’t help myself), there are many benefits to this vast network of connections. In Manufacturing, digital control systems can be used to automate process controls, operator tools, and service information systems to optimize plant safety and security. In infrastructure management, events or changes in structural conditions can be monitored to determine structural risk or security problems. In environmental monitoring, sensor data enables us to create early warning systems and prepare more effectively for natural disasters. In healthcare and human services, remote patient monitoring and preventive care services can be provided to a wide swath of patient groups in addition to being able to personalize such care over time.  And there are many more examples, as you are no doubt aware.

But all these benefits accrue only with smart analysis. In fact, while the devices and sensors are important, it is the data that comes out of those sensors that matters more. Insights from these data won’t just magically fall onto our plates, however. Serious analytics are needed and the challenges in implementing those are significant. Ignoring these challenges results in inflated expectations about what the IoT can deliver and a huge gap between promise and reality. In its latest Hype Cycle for Emerging Technologies, Gartner put the IoT at the top of its so-called hype cycle, which means that the technology is currently the main target of inflated expectations. With that said, let’s look at some of the challenges that affect IoT analysis and how these challenges can be successfully addressed.

Interoperability and lack of standards.

Interoperability is the ability of different information technology systems and software applications to communicate, exchange data, and use the information that has been exchanged. In the context of the IoT, this means that each device should be able to easily communicate with other devices and automatically send out triggers, warnings, and messages based on set threshold levels. This implies that each device manufacturer conforms to a standard that would then make this information exchange seamless, but that is hardly the case. In fact, according to a recently published World Economic Forum (WEF) report, the absence of a consensus regarding manufacturing standards or data exchange protocols makes it very hard for economies to leverage the tremendous potential in IoT analytics.

Inadequate IT architectures.

Simply put, IoT involves deep and wide data at scale. This means we could have large volumes of data coming out of sensors, wearables, machines, buildings, and whatnot that needs to be harnessed in in one system. That system must enable decision makers to analyze these sources and make critical choices in his or her given field, whether healthcare, entertainment, manufacturing, retail, and so on. This also means that IT architectures must not just be prepared for present day volumes but also for the future when data scales are likely to far outstrip today’s levels.  

Fortunately, some technology companies are investing tremendous resources and committing intellectual capital to enabling IoT analytics. Furthermore, independent ventures such as the European Lighthouse Integrated Project have been working for several years on IoT architecture and have created a proposed architectural reference model along with the definition of an initial set of key building blocks.


Perhaps one of the biggest concerns about the IoT is the possibility of devices being tampered with, particularly as more of them are decentralized and manufactured in an environment with no unified standards. This decentralization and the presence of low cost devices may enable malware to bypass traditional firewalls. Furthermore, the sheer diversity of integration software and middleware, APIs, and disparate access points make the effort even more Sisyphean. Unless there are clear-headed policy-driven approaches that systematically address security concerns, the ability to unlock the true potential of IoT will be severely hamstrung.

Slow awareness and adoption.

While the sheer number of devices has increased and continues to do so exponentially, the essence of the importance of this data and the benefits of advanced analytics are only starting to be emphasized. In some ways, the proliferation of multi-modal devices and sensors is far ahead of the understanding of what these devices could mean if they were viewed in a concerted manner. Big data professionals talking among themselves about the benefits of the IoT, though important, is hardly enough.  A multi-pronged industry wide communications effort is needed to convey the critical need for and ROI of IoT analysis. This effort could include use cases, analytic methodologies, architectural recommendations, customer testimonials, analytic apps, and more.

With these challenges clearly in mind, let’s focus briefly on where the opportunities lie.  I can point to at least three areas of encouraging work.

First, tremendous advances have been made in rationalizing and delivering upon the promise of a unified architecture that can ingest and analyze varieties of data at scale and volume.  This is truly a game changer in that we at least now can speak of solutions that can compete with the ebbs and flows of data from disparate devices without imposing artificial technical constraints.

Second, the idea of near zero touch management of analytics systems is slowly, albeit surely, finding considerable cachet in the marketplace. Companies that focus on an application driven environment where users can quickly see interactive visualizations of key metrics are likely to be winners. These are not just winners in the sense of increased revenues and market share, which are no doubt important, but these are tools that truly have the ability to shape the course and discourse of IoT analytics.

Third, there is no one size fits all when it comes to analytics, let alone IoT analytics. What this means is that the amount of data that we see today is so large and varied that a single analytic technique is likely to speak to only small bits of the story.  Analysis in today’s environment needs to be truly multi-genre, where multiple techniques are intelligently applied in concert to deliver insights to the business. Some organizations offer solutions that support multi-genre analytics with pre-built capabilities and others are starting on this path.

The potential of the IoT to improve people’s health, manufacture useful consumer products, improve energy efficiency at a time of global warming concerns, meet consumer tastes and preferences more quickly, and do many more things to advance the human condition is enormous.  Now is the time for analytics professionals and companies to think through these challenges carefully and create solutions that focus on information heterogeneity, ease of use, and consistent innovation.