The Internet of Things (IoT) has been among us for a while, but in recent years we have seen a change in scale, in part due to cheaper sensors that are emerging. Cities are deploying sensors to improve the quality of life for their citizens, while factories are connecting more and more machines and collecting more data about the production processes. Supply chains are being revolutionized by tracking in real-time not only position but also movement (shaking, dropping), humidity, etc… In almost every industry you can see the impact of IoT.
Due to this change in scale new challenges are starting to emerge that are demanding a rethink of the Cloud only paradigm, and the silo approach to IoT.
IoT typically means deploying application intelligence and analytics at the edge (the area between Cloud/data centers and end points such as sensors, factory robots, etc…), or pushing data directly to the Cloud for processing. Both approaches have their advantages as well as potential drawbacks.
The network between the edge and the Cloud can be relatively expensive (especially if you send all data to the Cloud) or has limited capacity (capacity is of course correlated with price). Latency to the Cloud can also be relatively high, and often lacks determinism. For example changing the color of traffic lights via the Cloud might not be optimal.
More and more solutions are being deployed at the edge to address the challenges Cloud faces. But these solutions have their drawbacks too. Many different solutions (hardware and software) make it more challenging to manage these edge services in a consistent and coherent manner.
IoT deployment is typically not confined to the traditional enterprise IT domain (au contraire). This means that traditional security solutions do not always apply, resulting in potential high risk security breaches: it is not only about stealing data, but also about controlling machines For example manufacturing robots, location of vehicles, …
One of the trends that we are seeing is that providers of edge services want to focus on their service (application) as this is where their expertise is. Today however many providers also need to provide the hardware (not always a good source of revenue), a certain level of security (not always their primary level of expertise), and a way to manage their services and devices (which can pose a challenge if a customer deploys multiple silos of IoT services).
A Unified platform beyond Cloud only
To address the challenges described above, a rethink is needed. On one hand the Cloud only paradigm is not sufficient, yet such a new platform needs to support a Cloud like methodology for the edge.
Fog, a driver for IoT
The emphasis here is on “like”, as the edge differs from a Cloud/data center in several important aspects such as: limited resources, limited network capacity, security challenges, and resource distribution. However, such a platform will also have things in common with Clouds. Just like in a Cloud environment it needs to manage the (edge) service life cycle and orchestrate deployment.
With such a platform in place, edge service providers can focus on their core business as this new platform provides them with hooks to develop, deploy, scale, monitor, and manage their services in a secure and safe environment while seamlessly connecting to the Cloud.
Moving to Cloud and beyond
The vision of such a unified platform has been described by Bonomi et.al. and labeled Fog Computing. We are now seeing this vision unfold in several distinct stages.
Unified (IP based) connectivity is typically the first stage. For example, cities offering free Wi-Fi in the city center, or factories that are consolidating their different networks.
Once unified connectivity is in place, it becomes easier to deploy services at the edge by connecting hardware to this IP network. This can lead to service silos, which are sometimes difficult to avoid due to legacy applications and hardware.
The next stage is the deployment of a unified platform (Fog platform) between Cloud and the endpoints to enhance service deployment beyond the Cloud but also to spur innovation by making it easier to share data between these services. This stage is where there is a true added value, as service management is unified and hardware platforms can become more consolidated.
This paradigm shift to think beyond Cloud towards a unified platform, will lead to new products, services and business models, but can also increases the risk of fragmentation due to lack of standards, architectural vision and abstraction. In order for this paradigm shift to truly succeed it is therefore important to have a continuous conversation between the IT and OT industry.
To ensure companies capture the value of IoT, it is important to start the thought process on a Fog and IoT vision early on: service deployments, connectivity capacity beyond Cloud, data filtering and analytics at the edge, device consolidation, real-time requirements, etc…
Such an IoT vision will enable companies to better prepare and understand the risks and opportunities in an increasingly connected world.
Tags: Corporate Technology Group, CTG, Fog, IoT
Companies with many employees face various challenges w.r.t. their size. One of these challenges is to identify key people, skills and information across (and outside) the organization and use them in the most effective way to drive innovation, new initiatives, but also sales.
The natural way for employees to cope with such a challenge is to build social networks (I mean networks of people not software) and collaborate across the organization. This organic social network building happens through various activities such as projects with colleagues, social activities, company events, etc… However, such social networks take time to create, and are typically not that extended. The effectiveness of these social networks is hard to measure (unless perhaps you equip your employees with location trackers). This social network building, I call the qualitative approach to organizational collaboration.
Connect with the right people inside and outside the enterprise
Many companies have deployed technology solutions (tools) to cope with the challenge described above. Companies have personnel directories that show employees’ groups and the official organization hierarchy. Some of these personnel directories allow users to add more personalized information (but this information is not always up to date). More sophisticated personnel directories (or other collaboration tools) also feature timelines of activities/tasks, blogging, integrated search, etc…. .Video conferencing enables people to connect people remotely. Despite these new social tools, email and mailing lists still play an important role in connecting people and disseminate information as well as external social networks and resources. All these tools provide a wealth of information. In essence: collaboration is Big Data.
Do all these social/collaborative enterprise tools help us doing our job better and promoting innovation? From a personal point of view I am tempted to say yes, but much more can be done. My main concern with most of these tools is the lack of analytics features to quickly identify user-relevant information or contacts. New tools – and newer versions of already existing tools – are starting to provide some of these capabilities, but IMO that is still not enough (or not accurate enough) to fully understand the evolving social networks or the relations between people and information (documents, emails, etc…).
What can be Improved
The goal of exposing more of the right analytics to end users would be, for a user, to faster gather insights, new ideas, and enable quicker decision making and eventually translate these insights and ideas to new opportunities, projects and/or costs savings.
Correlate different sources to identify information
To achieve this goal, users should be able to identify patterns in their organization’s data, specifically on threads or evolving thoughts and interactions that can be relevant for their particular projects or questions. In essence, analytics should foster more and improved collaboration with like-minded people, or people that share a common goal. As mentioned earlier, people naturally do this already, but in large organizations it is humanly impossible to scale this effectively and fast, without the help of analytics tools. This type of analytics I call the quantitative approach of organizational collaboration, which I see as complementary to the qualitative approach.
When I look for example at mailing lists or video conferencing, a few questions always pop up in my mind that modern enterprise collaboration tools should be able to answer in just a few clicks:
- What topics are trending during the last week/month? Perhaps type a topic in and get trend information or have the computer generate topics based on a context analysis of your posts or email conversations.
- How are the groups and hierarchies evolving over time (who is talking to who)? Can software recommend groups of people that are relevant for me and my projects?
- What people can be considered as experts on certain topics, based on their posts, replies, published articles, etc…?
- For particular topics, who are the top contributors and how do they relate to the experts? Are people clustering around certain topics?
- Who are the influencers/thought leaders, and how do they relate to experts?
From a strategic point of view companies can leverage analytics from social/collaborative tools to answer questions like:
- Are best practices shared across the organization between the appropriate groups?
- Is there an alignment between strategy and direction of the company?
This is not an exhaustive list and as a software engineer I think that an additional relevant feature for any tool should be the ability to provide an environment to mashup and integrate data by employees, to answer some of these questions.
How can it be Leveraged
Various groups and people (MIT, Virginia Tech, …) do research on this subject and translate this research into strategic insights at the enterprise level. The next step will be to provide the insights to individual employees as well. Enterprise tools with more sophisticated analytics capabilities (many focused on machine learning) are beginning to emerge. Perhaps the biggest challenge is integration of such capabilities across multiple internal and external tools and platforms.
Organizational collaboration is for me not limited to an enterprise environment. Groups with different affiliations who organize themselves as “virtual organizations” to work together towards common goals (for example, Open Source communities or standard bodies) can benefit from this type of analytics too.
To be more successful in capturing the value of collaboration, companies not only need to deploy the right tools, but also need to foster a mashup environment to leverage the organizational insight and tacit knowledge of its employees through analytics.
Special thanks to Marco Valente and Yannik Messerli for the discussions and insight on this subject.
Tags: analytics, collaboration, Corporate Technology Group, CTG, innovation, social, strategy
Data has always been important for many (if not all) companies. Today however it is becoming increasingly easier to collect data about customers, business transactions, devices etc…, . This data enables companies to (more dynamically) optimize their business models and processes and differentiate themelves from their competitors. Before embarking on a (Big) data strategy it is important to understand what is driving this evolution and what are basic architecture building blocks to take into account when transforming into a more data driven enterprise.
Several trends are fueling the evolution towards a more data-driven enterprise: Price/performance improvements in network, storage and computing and the rise of Cloud computing , make it more cost effective to deploy large IT infrastructures and capture data. New data management technologies and products such as Hadoop (MapReduce) and NoSQL/NewSQL provide scalable and more cost effective solutions than traditional SQL databases for various classes of problems. The IT consumerization trend results in departments more actively pursuing analytics capabilities. Another important trend is the Internet of Things (IoT): The advent of cheaper sensors and improved connectivity are bridging the gap between the physical and digital world, enabling collecting data from more devices and environments. This sensorization is unlocking the potential to gather enormous amounts of data and details about almost everything.
These trends are creating new challenges and opportunities to harness and understand the data tsunamis, and leverage the analytics for decision making purposes, to better monitor, control, and understand complex systems from business dashboards to IoT eco-systems. Read More »
Tags: analytics, architecture, Big Data, Corporate Technology Group, CTG, Enterprise
More and more enterprises are managing distributed infrastructures and applications that need to share data. This data sharing can be viewed as data flows that connect (and flow through) multiple applications. Applications are partly managed on-premise, and partly in (multiple) off-premise clouds.
With the advent of the Internet of Things (IoT) the need to share data between applications, sensors, infrastructure and people (specifically on the edge) will only increase. This raises fundamental questions on how we develop scalable distributed systems: How to manage the flow of events (data flows)? How to facilitate a frictionless integration of new components into the distributed systems and the various data flows in a scalable manner? What primitives do we need, to support the variety of protocols? Read More »
Tags: Amazon Cloud Connector, Corporate Technology Group, CTG, internet of things, SDN, Service Provider
More and more enterprises are managing distributed infrastructures and applications that need to share data. This data sharing can be viewed as data flows that connect (and flow through) multiple applications. Applications are partly managed on-premise, and partly in (multiple) off-premise clouds. Cloud infrastructures need to elastically scale over multiple data centers and software defined networking (SDN) is providing more network flexibility and dynamism. With the advent of the Internet of Things (IoT) the need to share data between applications, sensors, infrastructure and people (specifically on the edge) will only increase. This raises fundamental questions on how we develop scalable distributed systems: How to manage the flow of events (data flows)? How to facilitate a frictionless integration of new components into the distributed systems and the various data flows in a scalable manner? What primitives do we need, to support the variety of protocols? A term that is often mentioned within this context is Reactive Programming, a programming paradigm focusing on data flows and the automated propagation of change. The reactive programming trend is partly fueled by event driven architectures and standards such as for example XMPP, RabbitMQ, MQTT, DDS.
One way to think about distributed systems (complementary to the reactive programming paradigm) is through the concept of a shared (distributed) data fabric (akin to the shared memory model concept). An example of such a shared data fabric is Tuple spaces, developed in the 1980’s. You can view the data fabric as a collection of (distributed) nodes that provides a uniform data layer to the applications. The data fabric would be a basic building block, on which you can build for example a messaging service by having applications (consumers) putting data in the fabric, and other applications (subscribers) getting the data from the fabric. Similarly such a data fabric can function as a cache, where a producer (for example a database) would put data into the fabric but associates this to a certain policy (e.g. remove after 1 hour, or remove if exceeding certain storage conditions). The concept of a data fabric enables applications to be developed and deployed independently from each other (zero-knowledge) as they only communicate via the data fabric publishing and subscribing to messages in an asynchronous and data driven way.
The goal of the fabric is to offer an infrastructure platform to develop and connect applications without applications having to (independently) implement sets of basic primitives like security, guaranteed delivery, routing of messages, data consistency, availability, etc… and free up time of the developer to focus on the core functionality of the application. This implies that the distributed data fabric is not only a simple data store or messaging bus, but has a set of primitives to support easier and more agile application development.
Such a fabric should be deployable on servers and other devices like for example routers and switches (potentially building on top of a Fog infrastructure). The fabric should be distributed and scalable: adding new nodes should re-balance the fabric. The fabric can span multiple storage media (in-memory, flash, SSD, HDD, …). Storage is transparent to the application (developer), and applications should be able to determine (as a policy) what level of storage they require for certain data. Policies are a fundamental aspect of the data fabric. Some other examples of policies are: (1) time (length) data should remain in the fabric, (2) what type of applications can access particular data in the fabric (security), (3) data locality, the fabric is distributed, but sometimes we know in advance that data produced by one application will be consumed by another that is relative close to the producer.
It is unlikely that there will be one protocol or transportation layer for all applications and infrastructures. The data fabric should therefore be capable to support multiple protocols and transportation layers, and support mappings of well-known data store standards (such as object-relational mapping)
The data fabric can be queried, to enable discovery and correlation of data by applications, and support widely used processing paradigms, such as map-reduce enabling applications to bring processing to the data nodes.
It is unrealistic to assume that there will be one data fabric. Instead there will be multiple data fabrics managed by multiple companies and entities (similar to the network). Data fabrics should therefore be connected with each other through gateways creating a “fabric of fabrics” were needed.
This distributed data fabric can be viewed as a set interconnected nodes. For large data fabrics (many nodes) it will not be possible to connect each node with all other nodes without sacrificing performance or scalability, instead a connection overlay and smart routing algorithms are needed (for example a distributed hash tables) to ensure scalability and performance of this distributed data fabric. The data fabric can be further optimized by coupling this fabric (and its logical connection overlay) to the underlying (virtual) network infrastructure and exploit this knowledge to further optimize the data fabric to power IoT, Cloud and SDN infrastructures.
Special thanks to Gary Berger and Roque Gagliano for their discussions and insights on this subject.
Tags: application centric infrastructure, cloud, Corporate Technology Group, CTG, data fabric, distributed systems, IoT, SDN