A consequence of the Moore Nielsen prediction is the phenomenon known as Data Gravity: big data is hard to move around, much easier for the smaller applications to come to it. Consider this: it took mankind over 2000 years to produce 2 Exabytes (2x1018 bytes) of data until 2012; now we produce this much in a day! The rate will go up from here. With data production far exceeding the capacity of the Network, particularly at the Edge, there is only one way to cope, which I call the three mega trends in networking and (big) data in Cloud computing scaled to IoT, or as some say, Fog computing:
- Dramatic growth in the applications specialized and optimized for analytics at the Edge: Big Data is hard to move around (data gravity), cannot move data fast enough to the analytics, therefore we need to move the analytics to the data. This will cause a dramatic growth in applications, specialized and optimized for analytics at the edge. Yes, our devices have gotten smarter, yes P2P traffic has become largest portion of Internet traffic, and yes M2M has arrived as the Internet of Things, there is no way to make progress but making the devices smarter, safer and, of course, better connected.
- Dramatic growth in the computational complexity to ETL (extract-transform-load) essential data from the Edge to be data-warehoused at the Core: Currently most open standards and open source efforts are buying us some time to squeeze as much information in as little time as possible via limited connection paths to billions of devices and soon enough we will realize there is a much more pragmatic approach to all of this. A jet engine produces more than 20 Terabytes of data for an hour of flight. Imagine what computational complexity we already have that boils that down to routing and maintenance decisions in such complex machines. Imagine the consequences of ignoring such capability, which can already be made available at rather trivial costs.
- The drive to instrument the data to be “open” rather than “closed”, with all the information we create, and all of its associated ownership and security concerns addressed: Open Data challenges have already surfaced, there comes a time when we begin to realize that an Open Data interface and guarantees about its availability and privacy need to be made and enforced. This is what drives the essential tie today between Public, Private and Hybrid cloud adoption (nearly one third each) and with the ever-growing amount of data at the Edge, the issue of who “owns” it and how is access “controlled” to it, become ever more relevant and important. At the end of the day, the producer/owner of the data must be in charge of its destiny, not some gatekeeper or web farm. This should not be any different that the very same rules that govern open source or open standards.
Last week I addressed these topics at the IEEE Cloud event at Boston University with wonderful colleagues from BU, Cambridge, Carnegie Mellon, MIT, Stanford and other researchers, plus of course, industry colleagues and all the popular, commercial web farms today. I was pleasantly surprised to see not just that the first two are top-of-mind already, but that the third one has emerged and is actually recognized. We have just started to sense the importance of this third wave, with huge implications in Cloud compute. My thanks to Azer Bestavros and Orran Krieger (Boston University), Mahadev Satyanarayanan (Carnegie Mellon University) and Michael Stonebraker (MIT) for the outstanding drive and leadership in addressing these challenges. I found Project Olive intriguing. We are happy to co-sponsor the BU Public Cloud Project, and most importantly, as we just wrapped up EclipseCon 2014 this week, very happy to see we are already walking the talk with Project Krikkit in Eclipse M2M. I made a personal prediction last week: just as most Cloud turned out to be Open Source, IoT software will all be Open Source. Eventually. The hard part is the Data, or should I say, Data Gravity…
Tags: Big Data, core, Data Gravity, Eclipse, edge, Enescu, ETL, Fog computing, IEEE, internet of things, IoT, krikkit, M2M, Moore, Nielsen, Open data, open source, virtualization
March is a rather event-laden month for Open Source and Open Standards in networking: the 89th IETF, EclipseCon 2014, RSA 2014, the Open Networking Summit, the IEEE International Conference on Cloud (where I’ll be talking about the role of Open Source as we morph the Cloud down to Fog computing) and my favorite, the one and only Open Source Think Tank where this year we dive into the not-so-small world (there is plenty of room at the bottom!) of machine-to-machine (m2m) and Open Source, that some call the Internet of Everything.
There is a lot more to March Madness, of course, in the case of Open Source, a good time to celebrate the 1st anniversary of “Meet Me on the Equinox“, the fleeting moment where daylight conquered the night the day that project Daylight became Open Daylight. As I reflect on how quickly it started and grew from the hearts and minds of folks more interested in writing code than talking about standards, I think about how much the Network, previously dominated, as it should, by Open Standards, is now beginning to run with Open Source, as it should. We captured that dialog with our partners and friends at the Linux Foundation in this webcast I hope you’ll enjoy. I hope you’ll join us in this month in one of these neat places.
As Open Source has become dominant in just about everything, Virtualization, Cloud, Mobility, Security, Social Networking, Big Data, the Internet of Things, the Internet of Everything, you name it, we get asked how do we get the balance right? How does one work with the rigidity of Open Standards and the fluidity of Open Source, particularly in the Network? There is only one answer, think of it as the Yang of Open Standards, the Yin of Open Source, they need each other, they can not function without the other, particularly in the Network. Open Source is just the other side, the wild side!
Tags: Big Data, cloud, Eclipse, Fog, IEEE, ietf, internet of things, IoE, IoT, Linux, Linux Foundation, M2M, network, Open Daylight, open source, open standards, social networking, virtualization, Yin Yang
Now that OpenDaylight has arrived, it’s time to explain why I made the Open Source choices eventually embraced by its Founders and the community at large. One doesn’t often see such leaders as Cisco, IBM, Intel, HP, Juniper, RedHat, VMWare, NEC, Microsoft and others agree, share and collaborate on such key technologies, let alone the latter engaging in a Linux Foundation based community (some thought hell will freeze over before that would ever happen, though it got pretty cold at times last Spring).
For those of you not familiar with OpenDaylight (see “Meet Me On The Equinox”, not a homage to Death Cab for Cutie or my Transylvanian homeland), IBM and Cisco have actually started this with an amazing set of partners, nearly that ephemeral Equinox this year (~11am, March 20th) though we couldn’t quite brag about it until all our partners saw the daylight, which by now, we’re hoping everyone does. It was hard not to talk about all this as we saw those half baked, speculative stories before the Equinox – amazing how information flew, distorted as it were, but it did; I wish source code would be that “rapid”, we’d all be so much better for it…
The Open Source model for OpenDaylight is simple, it has only two parts: the community is hosted in the Linux Foundation and the license is Eclipse. The details are neatly captured in a white paper we wrote and published in the Linux Foundation. Dan Frye, my friend and fellow counterpart at IBM and I came up with the main points after two short meetings. It would have been one, but when you work for such giants as our parent companies and soon to be OpenDaylight partners, one has to spend a little more time getting everyone to see the daylight. It boils down to two things, which I am convinced are the quintessential elements of any successful open source project.
1) Community. Why? Because it trumps everything: code, money and everything else. A poor community with great code equals failure (plenty of examples of that). A great community with poor (or any) code equals success (plenty of examples of that too). Why? Because open source equals collaboration, of the highest kind: I share with you, and you with me, whatever I have, I contribute my time, my energy, my intellectual property, my reputation, etc.. And ultimately it becomes “ours”. And the next generation’s. Open Source is not a technology; it’s a development model. With more than 10 million open source developers world wide, it happens to be based on collaboration on a scale and diversity that humanity has never experienced before. Just think about what made this possible and the role some of the OpenDaylight partners have already played in it since the dawn of the Internet. Dan Frye and I agreed that the Linux Kernel community is the best in the world and so we picked the closest thing to it to model and support ours, the Linux Foundation.
2) Fragmentation, or anti-fragmentation, actually. Why? The biggest challenge of any open source project is how to avoid fragmentation (the opposite of collaboration). Just ask Andy Rubin and the Android guys what they fear the most. Just ask any open source project’s contributors, copyright holders, or high priests, how much they appreciate an open source parasite that won’t give back. Though we would have liked to go deeper, we settled on Eclipse, largely because of the actual language and technology we dealt with in the OpenDaylight Controller: most, if not all the initial code is Java, and though some are worried about that, I’m sure Jim Gosling is proud (btw, I’m not sure the Controller has to stay that way, I actually agree with Amin Vahdat), but we had to start somewhere. Plus having a more friendly language NB (northbound, as in the applications run on top of the Controller) is such a cool thing, we think that the #1 open source (Eclipse) and the #1 commercial (Microsoft) IDE’s are going to be very good to it, so why not? There are more reasons that pointed in the Eclipse direction, and other reasons for such wonderful alternatives (as APL or MPL, perhaps the subject of another post, some day). But when it comes to understanding the virtues of them all, no one understands them better than the amazing founders of these license models, most of them from IBM, of course (I wish they did that when I was there).
What happened between the Equinox and Solstice is a fascinating saga within the OpenDaylight community which I think played its course in the spirit of total and complete openness, inclusion, diversity, respect of the individual and the community, and most of all, that code rules – we do believe in running code and community consensus. I tip my hat to all my fellow colleagues that learned these two things along the way, the enormous talent at the Eclipse and Linux Foundation that helped us launch, and even the analysts who tried (and did incredibly well at times) to speculate the secret reasons why these partners came up with the model we did: there is no secret at all, my friends, we’re simply creating a community that is truly open, diverse, inclusive, and never fragmented. Just like a big, happy family. Welcome to OpenDaylight, we hope you’ll stay!
Tags: Android, apache, Cisco, community, controller, Eclipse, Fragmentation, IDE, java, Linux Foundation, Linux Kernel, ONE, onePK, open source, open source model, opendaylight, SDN