Microservices Infrastructure – Mantl – Release 0.4

During the OpenStack Summit last week, we released Mantl 0.4. In this blog I would like to go into more details about the release. But first I’d like to start by explaining what Mantl is – and what it is not.

System Integration as Open Source

Mantl is a layered stack that takes care of system integration. It does this by using tools at different layers – Terraform to provision Virtual Machines and Apache Mesos & Kubernetes for cluster management. Higher level services are taken care of by tools, such as Consul for service discovery, or by custom Apache Mesos frameworks, which are currently used for processing data.

You could say that Mantl create the “glue” to enable hybrid cloud. This is too dry an explanation for us. The truth is that Mantl has three design goals: Build; Deploy; Run

• Firstly, it aims to shorten the development cycle. Most programmers recollect feelings of joy when they first coded. However, as web-development rose in conjunction with the monolith, coding was as much, if not more, about configuration management as it was application development. The extension of the feedback cycle, as well as not been much fun, seriously stunted productivity.

Currently it’s the same for cloud applications. Developers spend excessive amounts of time provision machines, opening ports and managing clusters when they could be developing their applications. One of the tenants of Mantl is it creates a ‘place to innovate’. It does this by making the cloud invisible and thus allowing developers to do what they do best: build innovative applications and get them into user hands as quickly as possible.

• Secondly, Mantl aims to gently coach developers, helping them to write cloud native applications. Many developers, understandably so, design their first cloud applications as they would have their old, three tier systems. With a gentle opinion, Mantl nudges developers towards containerized services and multi-language systems while at the same time creating a bridge between the traditional and the cloud native.

• Thirdly, Mantl aims to make interaction with the cloud as simple as possible. Famously, Joel Spolsky said that all abstractions leak. What this means is that you can never hide the underlying abstraction: virtual machines are bound by the hardware they run on; compilers are bound by underlying machine architectures. It’s the same for cloud: you cannot totally abstract the platform away. However, if you must interact with it, you should do at the right level of abstraction. Mantl provides a number of tools that make this easier. It relies on Docker containers and Terraform, for example, but also provides custom tooling, such as MiniMesos.

In summary, Mantl coaches, shortens the development life cycle and provides abstractions at the appropriate levels. In addition to this, it provides data-tooling.

Let’s now look at some of the innovations from release 0.4.

MantlUI

Mantl 0.4 includes a new WebUI that connects to the various applications (Mesos / Marathon / Chronos / Consul). For example, users can now access Mesos agent logs through an authenticated UI.

Backed by Consul service discovery, the new UI automatically connects to the correct Mesos masters and agents.

Mantl-api

We’re very excited to announce support for the first release of Mantl-API.

Mantl API provides a new way for you to manage Mantl clusters. With the first release, you can easily install pre-built applications and Mesos frameworks. With a single API call, you can now spin up Cassandra on your Mantl cluster.

We think Mantl-API will be useful for anyone who is currently running Mesos.

Glusterfs

Support for deploying GlusterFS as a shared filesystem has been added.

DNS provider support

Support for DNS providers. We’ve added example code to configure DNS registration of Mantl nodes in DNSimple. Thanks to contributors, we will be adding support for other DNS providers like Route 53 and Google Cloud. We’ll make these more configurable when terraform supports conditional logic.

Calico IP per container networking (tech preview).

Calico is a new virtual network solution that enables the IP per container functionality. Calico connects Docker containers through IP no matter which worker node they are on.

Data Tooling Built In

The ELK stack is built into Mantl as Apache Mesos frameworks. This means that developers can use Mantl’s Terraform modules to provision a cluster, setup the system, and immediately start building data-driven applications.

On its own, this functionality is powerful. However, because Mantl uses Apache Mesos frameworks for its data tooling, it can (and does) take advantage of Mesos’ scheduling and hardware utilization features. In addition to this, the frameworks provide extra functionality.

Let’s look at three features of the ElasticSearch framework. Firstly, the framework allows the scaling of the cluster via a GUI – it thus provide the right level of abstraction for developers to interact with the cluster. Secondly, it provides a visualization of the cluster, including where the PRIMARY and REPLICA shards are located. Thirdly, through the GUI, developers can search the cluster, which is handy for testing and debugging.

Please note, although these features are in progress, they are currently on the experimental branch.

Image 1 – ElasticSearch Framework GUI with the works of Shakespeare on a three machine cluster.

The Mantl Developer Tools – MiniMesos
One of the problems with Apache Mesos is that it’s hard to set up. In his O’Reilly article, “Swarm v. Fleet v. Kubernetes v. Mesos”, Adrian Mouat says that, ‘Mesos is a low-level, battle-hardened scheduler that supports several frameworks for container orchestration including Marathon, Kubernetes, and Swarm’. However, he goes onto say that for small clusters it may be an overly ‘overly complex solution’.

Mantl uses Mesos because its battle hardened. But since one of Mantl’s goals is to make interaction with complex tools as simple as possible, the teams building Mantl created MiniMesos.

MiniMesos provides an abstraction layer over Apache Mesos. Minimesos allows developers to run, test and even share their clusters. Since Minimesos can bring a cluster up in milliseconds and lets developers test their code before checking in, it radically shortens the developer lifecycle. Importantly, Minimesos can be used from the command line or via its API, thus making automated system testing easy.

Minimesos now has its own Twitter account and website. It is one (of many) innovations to come out of the Mantl program and has captured the imagination of the community. Pini Reznik, CTO of Container Solutions, who are part of the team working in Mantl, says that ‘Minimesos is to Apache Mesos what Docker is to LXC’.

Image 2 – MiniMesos Command Line Interface as it is implemented in Mantl 0.4. More commands to come, including ‘install’ for quickly adding frameworks.

Check out the video on MiniMesos.

Use Cases
There are many uses cases for Mantl. One of the most interesting patterns that is emerging is around IoT. At DockerCon, in November, we hope to reveal the Wheel of Fortune application. The Wheel of Fortune connects a physical wheel to a REST endpoint. The endpoint is part of an application that scales automatically and displays the data via a web-application.

At first glance the Wheel of Fortune may seem like a bit of fun. However, collecting data, big or otherwise, from the IoT for storage and analysis is a key aim of Mantl. Because Mantl abstracts the underlying infrastructure away or makes it invisible, developers can get busy building and deploying their big data applications without worrying about system integration.

Another interesting use case is hybrid devops. Hybrid devops is the ability for enterprises to develop their applications leveraging Cisco Shipped (ciscoshipped.io) the way they always have. Then leverage Mantl to deploy their application on any external cloud environment supported by Mantl (AWE, GCE, Digital Ocean, Rackspace, Cisco Cloud) in a CI/CD framework that enables internal and external services to be leveraged by the application.

Whats next

We are making Mantl more modular, so that you can select the scheduling, logging and networking components you want to deploy.

The team is also committed to automated testing, and we’ll be testing Mantl against multiple cloud providers daily.

Features on the roadmap include:

Better haproxy support
Improved docker storage leveraging Cisco Contiv.
Full integration of Hashicorp Vault
Kubernetes/OpenShift support
Modular networking leveraging Cisco Contiv
Simplified API management
Application Policy Intent leveraging Cisco Contiv
New deployment and management tools

Conclusion
Modern enterprises face three often competing tensions. Firstly, they have to learn how to build cloud native applications. This involves much more than recreating monoliths in the cloud. It involves changes in process but also in structure. As enterprises encompass small and medium sized companies in their supply chains, they have to have a structure that supports language agnostic microservices.

Secondly, the challenge of big data is calling all companies. Enterprises not only need to tap into the power of data scientists and developers but they have to actively work around organizational scar tissue. It is impossible to work with large amounts of data and to test new algorithms against production data whilst carrying decades worth of old processes and procedures around. The new enterprise can be agile and take advantage of big data. What it can’t be is bureaucratic and take advantage of big data – these two concepts simply cannot coexist.

Finally, all enterprises must deal with governance. This includes security, operations and a shift towards DevOps or NoOps.

Mantl helps enterprises resolve the tension between these three challenges. Mantl enables repeatable and simple deployment procedures through its use of programmable infrastructure tools, like Docker and Terraform. Mantl promotes the microservice architecture and by default supports systems built in multiple languages by multiple teams. This means that enterprises can take advantage of an extended, horizontally aligned, supply chain. Finally, Mantl is both IoT and Big Data ready and friendly. Through its use of abstraction, programmers and data scientists can focus on what they do best whilst leaving system integration the Mantl.

References
● Mantl’s website, http://mantl.io/.
● MiniMesos’ website, http://minimesos.org/.
● Cisco Shipped website, http://ciscoshipped.io
● Cisco Contiv website, http://contiv.io
● ‘The Law of Leaky Abstractions’, Joel Spolsky, http://www.joelonsoftware.com/articles/LeakyAbstractions.html.
● ‘Swarm v. Fleet v. Kubernetes v. Mesos’, Adrian Mouat, http://radar.oreilly.com/2015/10/swarm-v-fleet-v-kubernetes-v-mesos.html.
● ‘Mini-Mesos: What’s a Nice XPer Doing in a Company Like This?’, Jamie Dobson, http://thenewstack.io/mini-mesos/.