OSPod #29: Jay Pipes
Jay may or may not have 99 problems, but passion isn’t one! Jay, like previous guests John Dickinson and Monty Taylor also started with OpenStack in the early days at Rackspace. We learned a lot about the “Big Tent” initiative in this podcast, and how OpenStack is working to balance inclusivity and innovation. (They aren’t mutually exclusive.) Watch the video, download the podcast, or read the transcripts below to learn more about:
- Jay’s accidental foray into tech
- The importance of quality code and asking questions
- Being realistic when it comes to definition of “production ready”
- Defaulting to “open” (and still being able to monetize)
- Emerging tech and dealing with legacy investments
Jeff and I are taking the show to the OpenStack Summit in Vancouver! If you’d like to be considered for a guest spot, tweet us at @openstackpod
See past episodes, subscribe, or view the upcoming schedule on the OSPod website.
To see the full transcript of this episode, click “Read more” below
Jeff Dickey: All right good morning everyone, I’m Jeff Dickey from Redapt.
Niki Acosta: You look dashing today Jeff.
Jeff Dickey: Thank you.
Niki Acosta: I’m Niki Acosta from Cisco and we have an awesome guest with us today. I am so excited to have Jay Pipes, Director of Engineering at Mirantis joining us. I’ve known Jay for quite a long time now. There are probably few people who’ve been involved in OpenStack as long as Jay have and has such a wide range of experience with OpenStack as Jay has, so welcome to the show Jay. We are so incredibly happy to have you. You can expound on your title a little more if I butchered it in any way.
Jay Pipes: Nah, you did just fine.
Niki Acosta: You focused on …
Jay Pipes: Nah…I work on OpenStck stuff…. how’s that?
Niki Acosta: That’s great, love it. Awesome and you’re even sporting your OpenStack shirt. Love that.
Jay Pipes: Yeah, this is the Icehouse one. Actually, this is the only summit I didn’t get to go to. The Icehouse was Hong Kong, right? I did not get to go to Hong Kong. I was working for AT&T at the time and I think only one of my colleagues went. Josh Kleinpeter and Toby Forth. Yeah, this is the only one I didn’t get a chance to go to so I’m actually looking forward to going to Tokyo in November and doing a summit in Asia.
Niki Acosta: Yeah, exciting. We typically like to start the show by asking you how you got into tech and you can take that all the way up through your very colored OpenStack career.
Jay Pipes: Okay, well let’s see. I actually went to school for Political Science which is kind of odd. Right before I went to continue my studies after my first college and went to my second college, I was working in the records retention area at UPS in Columbus, Ohio. The job was to maintain this set of bankers boxes. This is mid-90s and the accountants … It was at the Midwest regional accounting office. This is grand stuff.
The accountants would come back from accounts receivable or payable or whatever and they would ask for a set or records for a customer and a time period. Me and this other fellow were responsible for going through this warehouse and it was very much sort of, you know, the end of Raiders of the Lost Ark? It was like that, completely disorganized, boxes everywhere, nobody took records of anything and so we would go back into the shelves, walk up the little staircase shelving unit things and find these sets of records. Very manual process. Anyway, we had this one desk in the records retention area and there was this old DOS 286 green-screen computer. There was this big book of semantic Q&A database programming. Like you look back at it now and it wasn’t really database programming, it was more like draw a screen form, then put in your data or whatever. Anyway, so I took those books home over a weekend and came back and I programmed this little database application that stored all the records, all the bankers boxes and where they were, what row they were, how high and up, all that kind of stuff. That was my first foray into computing.
The accounting people got word that there’s this weird guy in the back that works in the records area that created this weird program, maybe we can ask him to do some collections stuff. They brought me into the accounting office and I programmed up a little database application so that they could track their collections calls. Anyway, went from there into doing other database stuff and then I did a lot of work in Microsoft MySQL server programming. Then I switched directions, I went into Open Source. I wrote a book on the MySQL database server. Then I got hired by MySQL to run their community relations. Then went back into engineering. Did work in the Drizzle Database Kernel which was like a fork of MySQL. It was kind of like an R&D thing. One year at Sun Microsystems, then I landed at Rackspace. Moved from doing database programming to working on OpenStack when it became OpenStack.
Myself and a guy named Eric Day were one of 2 engineers that a guy named John Purrier had asked to do the due diligence about what would be the next sort of generation of the Rackspace cloud servers platform which at the time was sort of like Slicehost and it was it’s own thing. They were investigating what new platform of framework they could use to build out the next generation of stuff. We had looked at a bunch of different solutions that were out there and at the time, Nova had just been open sourced and kind of like thrown over the wall by the guys at ANSO Labs. We got wind of it and we checked it out, and we’re like, “Well, I mean this should work. This actually has a lot of pluses to it. We can work with this.” It’s sort of like in the wheelhouse of what the Rackspace cloud folks were used to as far as Python and you know, some of the technologies that were used. That’s how we got involved in OpenStack, before it was OpenStack. When it was just like let’s find out what the next generation of the Rackspace cloud might look like or something. That was kind of how I got into tech and all the way into OpenStack land. Condensed into whatever that was, 3, 4, 5 minutes.
Niki Acosta: You’ve actually been with a few companies with OpenStack? You spent some time …
Jay Pipes: Yes. What was it you said earlier?
Niki Acosta: You’re like the OpenStack record, you get around?
Jay Pipes: Yes, that was a great way of describing it. Thank you, Niki. I started in the OpenStack world at Rackspace because Rackspace acquired the whole team from Sun Microsystems that worked on Drizzle, the Drizzle database kernel. From Rackspace, I did a short stint at HP and then went to work at AT&T where I focused on a Dev Ops type of role. We were using a lot of Chef, we did a lot of automation GLU code. From AT&T, I left and went to Mirantis which is where I’m at now. Been here for about a year and a half or so.
Niki Acosta: For a company its size, you know Mirantis, it always floors me to see the amount of contributions that Mirantis makes to OpenStack. Certainly understanding that they were a company kind of born out of OpenStack, right?
Jay Pipes: Well, I mean certainly for the last 4 or 5 years we’ve been focused on OpenStack. I mean, Mirantis has actually been around for awhile. A lot of the early Mirantis stuff was doing sort of offshore development work for network companies. The Ciscos of the world and that kind of stuff. We did network engineering, software engineering, foreign network devices and load balancers, that kind of thing. That was kind of the bread and butter of Mirantis before OpenStack became our sort of soul as a company. You know, since we joined OpenStack, the company has grown substantially. I think we’re at some hundred and something folks now. When I joined, I think it was just shy of 600. I think we’ve added like a 120 or so in a year which is a decent size growth for a company our size. We’re totally focused on OpenStack now. We do …
Niki Acosta: But the proportion of people at Mirantis actually contributing to OpenStack is still quite high, right?
Jay Pipes: Absolutely, yeah. I mean it’s … We have a few folks that contribute to external things on the Linux platform layer. For the most part, if you’re an engineer working at Mirantis, you’re working on OpenStack stuff. Whether it’s foundational type stuff like Nova, Neutron, Glance, Ironic, that kind of thing or whether it’s the higher level stuff like Heat and Sahara and Murano or Rally or those kind of projects that consume the other projects, yeah. If you’re an engineer working at Mirantis, you’re working on OpenStack, pretty much.
Niki Acosta: Brilliant. Jeff, you have a question? I think we wanted to talk about some of the big tent stuff, right?
Jeff Dickey: Yeah, absolutely. Jay, can you first describe kind of Big Tent and then kind of go into that?
Jay Pipes: Sure, I’ll give it my best shot. I think there’s a number of grey areas that I think people have gotten confused about with the whole Big Tent thing. Just to lay the groundwork. All of last year, literally the entire year, the technical committee debated how we define what is “core OpenStack”. The whole idea of what is core was something that we debated hotly between all of the members of the technical committee and on the public mailing list as well … Well, all the mailing lists are public. On the OpenStack dead mailing list, there were a number of conversations about like what is OpenStack supposed to be? Is it just this small group of compute and storage things and maybe some networking in there? Is it that and platform layer stuff? Do we allow competition within a specific space? For instance the deployment space? Should we be welcoming of Puppet and Chef and Ansible and SaltStack and CFEngine and TripleO … It’s sort of way of doing deployments. There were back and forth debates on what OpenStack should be and a number of us were pushing the idea of being a Big Tent. Being very inclusive of newcomers and also allowing competition within specific spaces as well as focusing on the API’s, the public restful API’s that define that interface between the client and the server.
At the end of the year, we had this sort of giant project structure reform proposal that Terry and Doug Hellman and a number of other folks had crafted to sort of represent the distillation of all the debates that we’ve been having over the year. It boiled down to a number of major changes. The first one was that we got rid of the idea of programs being, like there was a compute program and a deployment program and a network program, right? Let’s take an example of the deployment program. The deployment program was TripleO. One of the things that we said was if we say the deployment program for OpenStack is TripleO, then by the nature of saying that we’re kind of blessing that as the TC, then that’s the way to deploy OpenStack. We are on the same token saying, “Well then, maybe it’s not appropriate to deploy OpenStack with Chef or with Puppet or Ansible or SaltStack.” That’s not the real world, right? In the real world, people are deploying OpenStack with Chef and Puppet and Ansible. We were saying, “Well, let’s get rid of these broad categories like the deployment program and have it be blessed by the TC and instead just let other things into the OpenStack ecosystem and not necessarily bless them, but say you are as much a part of the OpenStack ecosystem as this thing called TripleO.” That was the first big thing, is breaking down these sort of silos that we had to put everything in and then have the TC bless whatever was in that silo as being OpenStack. That was the first thing.
The second thing was being much more objective about how to get into the OpenStack code namespace. Prior to the Big Tent initiative and the reform of the project structure, we had this thing called incubation. A project would apply to be incubated and over a period of time, the project would come before the technical committee and the technical committee would go through an incubation review and then a series of graduation reviews where you would graduate from incubation into the integrated release. Then, I guess supposedly if you get into the integrated release then you are part of core OpenStack and the technical committee blesses these set of projects as being production ready and scalable and mature and all these stuff.
Well, the problem with that is that these incubation and graduation reviews partly by nature of the technical committee being elected in 6 month cycles and yearly cycles, would get really inconsistent feedback in these reviews. Some of the feedback was entirely subjective. If you take an example of Zaqar which is the message queue as a service or the queuing service but tenant facing. We had a number of reviews, I think it was 3 graduation reviews for Zaqar, where they were incubated and they were trying to get into the integrated release and go through the graduation review with the technical committee. We had folks on the technical committee bringing up some fairly valid points, but were pretty subjective and even inconsistent from the last time that Zaqar had gone through their graduation review. We were telling them sort of like different things from one graduation review to the next based on some of the TC member’s ideas about appropriate architecture or appropriate infrastructure or appropriate dependencies that the project might have.
In the end, it just kind of stifled innovation. It was preventing a good project contributor team from making progress because we were sort of almost throwing arbitrary restrictions at them. The second big part of the Big Tent initiative was to make the set of requirements for going into the OpenStack code namespace be as objective as possible. Now, the list of things that you need to do to get into the OpenStack code namespace is you need to follow the 4 opens of OpenStack, so Open Design, Open Collaboration, Open Source and Open Reviews. Those 4 things are very objective. You can go and see whether or not you’re doing open code reviews, whether you’re discussing design in the open and all that kind of stuff. You also need to say that if you’re in the OpenStack code namespace that you will defer to the technical committee for any reasons where you can’t reach an agreement within your project contributor team. There’s one other objective requirement that I’m forgetting, but the point being, it’s not this subjective, arbitrary set of requirements that individual TC members might come up with about, “Well, the architecture isn’t as scalable or as distributed as we’d like and therefore we are going to deny you access into the OpenStack code namespace.” Those were the first 2 big things, which is opening up the OpenStack code namespace to more applicants and being more objective about the requirements for getting into the OpenStack code namespace.
Finally, the third big thing of the project structure reform called Big Tent is instead of having this one binary integrated release bit of information that we apply to projects in the OpenStack sphere. Instead, break that into very fine grain tags. Instead of just you’re in the integrated release and that somehow would mean that Celiometer or Neutron or Nova and all the projects that were in the integrated release are of the same quality, same code maturities, some production readiness, it doesn’t make any sense. To say that Swift, the code base of Swift and the code base of Ceilometer were at the same point in their maturity or the release cycle is ludicrous. In the real world, that’s just not the case and so we wanted to come up with a set of tags that were much more fine grained and informative to our audience, whether the audience was a developer, whether the audience was a deployer, a packager of OpenStack or an operator. We are in the process of coming up with these set of tags, that hopefully will indicate to these 4 different audiences, more descriptive information other than just you’re in or out of the integrated release. We’ve come up with a set of tags that describe the release process whether or not it’s managed, what cycle it’s on. We’ve come up with a tag that describes the diversity of the contributor community whether or not it’s entirely dominated by a single company or whether it’s spread out amongst multiple companies. You know, that’s an indication of whether or not the road map for a particular project is going to be driven by a product management within one company alone versus a road map that’s drive more by compromising consensus. These tags are not meant to be blessed or not blessed. They’re meant to be very specific pieces of information that we give to users to let them make a decision. If they don’t want to use any projects that are predominantly or entirely driven by one company, well now they have a tag that will give them that information. That’s the third piece of the Big Tent initiative that, I think, was an important component of it.
Niki Acosta: The net of this is that anyone who goes and chooses to participate in OpenStack will have some guard rails it sounds like, but probably from what I can tell, I think the benefit really is going to be to people who are actually consuming OpenStack, right? To be able to not make the decision for them but to give them the information they need to make the best decision based on their use case or their application or how they want to operate it or whatever, right?
Jay Pipes: It’s kind of always been my opinion that having these 13 individuals on the technical committee pronounced whether or not they think something is production quality or whether they think something is scalable is just kind of ridiculous. I mean, first of all, the 13 individuals on the technical committee have a very wide range of experience in operating OpenStack. Some have no experience whatsoever, others have a little bit, some have quite a bit. To have that set of individuals pronounce project X is production ready, I don’t think that’s all that useful and in fact, having them make that stance and bless something as production ready, I think actually did a disservice to our users because you’re giving out false information or misinformation.
Instead, I prefer to say have the operators provide feedback on what they think is passed and experimental phase or that they have experience in running at scale, in a stable way for X number of months or release cycles or whatever it so that you give the operator community and the deployer community and the packaging community more information than just, “Oh, it’s in the integrated release.” I know there’s a lot of people that want that sort of check box like, “Oh, it’s in the OpenStack integrated release.” I actually made this case about this to a customer of Mirantis recently, you know, would you rather have misinformation or sort of pretend information just so that you feel good about some project being in the integrated release or would you rather have information that’s coming directly from operator and deployer feedback that’s saying, “Maybe this needs another 3 to 6 months to bake.”
That’s where I stand on that. I think that the distributions of OpenStack where there’s Mirantis or RDO or a bunch of cluster archive and all these other things, those distributions will make a value judgment on the projects based on what they are able to test, functionally and at scale in their labs. I think that should be the indication to the operator community that they should take to hear, not something that the technical community has proclaimed.
Niki Acosta: In all of these, I mean, there’s obviously a point to be made about community contribution and the process of growing core teams. How do you flush that out in your mind?
Jay Pipes: That is like the perennial question that any of us that are on the core team, for any of the sort of busy or high velocity projects, we struggle with all the time. I mean, we’ve gone through all sorts of different proposals, like how do we merge more codes but at the same time keep the quality high? How do we get more individuals into the core teams so that we can spread the load, the review load, out? Nova, we struggled with this mightily. We’ve lost a lot of core team members because they move on to other things. There are more important things that their companies assigned them to work on and that’s completely natural, but if you’ve got 10 individuals that are responsible for [inaudible 00:24:05] a piece of code which allows it to go through gate and get merged into the code trees. For a project like Nova, we typically have anywhere between 550 and 650 active patches in review.
I mean, for 10 people to take that weight, it’s just impossible. What ends up happening is that the core team members sort of just get into this, I don’t know, flow of prioritizing stuff in their head and a lot of times, you’ll get core team members that simply have to ignore large chunks of the code base because they have to focus on one particular area. Otherwise, they’ll never get anything done. They become sort of lieutenants of a particular coding area where other core members trust them more than other areas of the code. I hate to say it but the squeaky wheel, it gets the grease a lot of times and a lot of people come to me and say, “Why does it take 3, 6 months to get my code merged?” and I’m like, “Well, did you come onto IRC and asked anyone to do code reviews?” “No.” A lot of it is like as simple as just hop on IRC and see if you can get one of the core team member’s attention and just get someone to look at it.
Niki Acosta: What are you looking for in a good code review? Can any part of this be automated in some way?
Jay Pipes: A lot of the stuff is automated when it comes to style and white space checks and what we call [PEP 8 checks 00:25:51]in Python land, but there are … There’s not a whole lot more that we could automate in my opinion. There are some things around like code coverage, like unit test code coverage percentage that we might be able to automate somewhat but not entirely. It’s not something I would support as a gating measure that would like download your patch and keep it out of the gate.
There are a few things that when I’m trying to sort of mentor newer contributors in the OpenStack space to how to do a code review and also how to respond to code reviews. It’s not just like how do you provide useful and insightful feedback to others, but it’s also how do you take that feedback that core reviewers and non-core reviewers are giving you in a way that’s constructive.
Niki Acosta: In a way, it might be like, “Hey, your baby is ugly,” and someone is [crosstalk 00:26:56].
Jay Pipes: Exactly, everyone always thinks that. My baby is ugly, you scream, run away and your patch gets abandoned. There’s a lot of that. I think I tweeted, I don’t know maybe a couple of years ago now, but it cost nothing to be kind and courteous in a code review. It really is true. You see many times, reviewers will get frustrated and I completely understand that. You’re overworked from the review queue. Any core reviewer on any of these high velocity projects will get requests 10 times a day or more if you’re on IRC to do code reviews. It can get very frustrating but if you just take a deep breath and just say, “You know what? All this person is trying to do is just get some feedback on their code or understand something that you might not have expressed clearly enough and they’re not trying to bother you.” The last thing anyone is trying to do is bother you. Like if they didn’t have to come on IRC and ask to get a review on something, they wouldn’t. It’s not like the first thing someone who wakes up in the morning and goes, “You know what? I’m going to go on IRC and bug some core reviewers today.” It’s not like that. You got to keep that in mind.
You also have to remember that nobody knows the answer to everything. People that sort of pretend that they know the answer to all questions, like I avoid them at all costs because you know that it’s just not realistic. I always say to new contributors, ask questions on reviews. That’s the number 1 thing that tells other reviewers and other patch submitters, you’re interested in learning how to make the code better, you’re interested in the code that you submitted a patch to and you honestly want to learn, so if you don’t understand something, ask the question. Ask good questions. Also, I often tell people that if you have a question about how a particular piece of code in a patch works, it’s typically an indication that that piece of code could use a good doc string or a good code comment explaining why the patch submitter is doing what they’re doing.
These seemed like really small and like petty things but if you think about it, a code comment … Code is read, what? A hundred times, a thousand times more often than it’s written. If you can write a good code comment or if you’re going to ask a question that will prompt a code comment being introduced into the code that will benefit a thousand people that are reading that code after you, then you’ve done a really great review. If you think about it, you’re not doing the review saying, “You should do this, you should do that, you should do this.” If you’re asking a question and that question results in a good code comment that lives on, that explains to people why that code is doing what it’s doing, you’ve effected a hugely important thing in the code base just by asking a question. That’s my biggest advice to new people is ask questions on reviews. The +1, -1 stuff, you know, I don’t like to encourage people to be nit picky about style or stuff like that. I can say, “You know what? Next time, do this. If you re-spin this, maybe change this,” but it’s the questions that you ask and how you ask them and the curiosity that you show, that really makes a difference for new reviewers. That would be my best piece of advice to people that want to do more code reviews.
Jeff Dickey: Jay, talking about the Big Tent and the code review stuff. Do you think OpenStack is too open?
Jay Pipes: Great question, Jeff. I don’t. I’d like to think that everything that I personally do, both in my community work as well as Mirantis, I default to open. This is something that … Actually, when I use to work MySQL, Marten Mickos, the CEO at MySQL at the time who then went to Eucalyptus and then HP. He used to say this is default to open. If there’s not a reason … There should be a reason to keep something from being open, not the opposite of that. In my mind, I’d rather have more information, more open and transparent discussions even if it is chaotic, even if it is sometimes very difficult to reach consensus and to get decorum on something. I’d rather it’d be that way than have 2 people fight it out in a room and then let a bunch of other people go do some work and then you throw it over a wall.
I see that too many times both internally at Mirantis, you know, there are some teams that have a tendency to do that because they come from a proprietary software development world. There’s a number of companies in the OpenStack business that would do the same thing. I try and discourage that as much as possible because I do think that long term, the benefits of everything being open and on the table and transparent are … They outweigh the cons of the chaos that sometimes ensues.
Niki Acosta: Love what you’re saying. If you’re defaulting to open, how do you make money? Like how do you make money in OpenStack?
Jay Pipes: Well, there’s … Just because you’re defaulting to open and having an open road map and an open conversation about your designs and your implementation, doesn’t mean that you can’t charge for things. For instance, the Mirantis OpenStack distribution just … RDO and any other company that’s attempting to productize a distribution of OpenStack, there’s nothing preventing them from charging subscription rates for the work that is being done by those companies. That doesn’t mean that the road map for the products and the projects that are being worked on, that make up that distribution in any way, needs to be closed. I would turn the question on its head and say what prevents companies that have an open road map for making money? I don’t think anything does.
Niki Acosta: That’s a good point. I think it [crosstalk 00:33:50].
Jay Pipes: It strengthens the product to be able to say to any customer or competitor, “Hey, is your road map completely transparent? Can I just see what you’re working on for the next 6 to 12 months on Launchpad and can I go see what road map your engineers are working on for deployment?” If they say no then you can say to a customer, “Here you go, see what we’re going to be working on. If you want to influence that, come check us out. Here’s our online IRC meetings. If you want to really influence that, pay us some money. Here’s a subscription to Mirantis OpenStack or RDO or a support subscription or what it is.” I don’t think open and making money are as sort of competing objective as a lot of people think.
Niki Acosta: I’m glad you said that because I think there are definitely some sentiments out there that pit those 2 against each other, not realizing that due to the vendor participation in OpenStack that a lot of the items that are being … Features that are being added, projects that are being designed, are actually coming from end users that some of these vendors are trying to serve.
Jay Pipes: Absolutely. I mean, you take a look at Mirantis. I mean, this 5 or 6 features in the last couple of releases that it’s not like a Mirantis engineer just thought, “Let’s do this.” These are things that are coming from our customers, that the customers are saying, “Hey, you know what? It’ll be really nice if we would be able to do to support this from our existing infrastructure and make it happen in the OpenStack world.” That gets funneled into a blueprint on Launchpad and tracks via all the open design processes. Yes, it may take a little more time than if you’ve had 2 engineers working in a room night and day and then they throw the code over the wall and try and get it merged upstream somehow.
I personally think it’s a more constructive process to have the design and the development and the road map be done entirely in the open from the start. I think if you do it halfway in between where you have production management people working on like the spec or design or use case of something and then they get open developers working on it at the end, I don’t think you get a lot of the benefits of a collaborative design environment by doing that.
Niki Acosta: Agree. What about the whole thing about vendor lock-in? Certainly being … Working at Cisco, you’re at Mirantis. Whether or not customers actually really care about that is to be determined. Some probably do and some probably don’t and some probably say they do, but when it comes down to it, it all comes down kind of to APIs, right? What do you do with your automation? How hard would it be to move like do APIs matter, Jay?
Jay Pipes: They do immensely. I actually have a talk with Chris Dent from Red Hat called Why APIs Matter at the Vancouver summit. I think it’s going to be like 10 minutes of me prognosticating about why APIs matter on a general sense and then Chris showing the tool that he developed for doing sort of declared testing on a functional level of APIs. A long time ago in the OpenStack ecosystem, we had this debate. When I say long time ago, I mean, like the Bear summit so this was 4 years ago, 4-1/2 years ago. We had this debate about is OpenStack the implementation or is it the API?
I actually argued that it was the API. The OpenStack cloud APIs were what was most important that would drive interoperability that would allow innovation at the implementation layer. If we said that OpenStack was a particular set of implementations, then we were by default limiting ourselves to whatever implementation we happen to have at the time that we would grown into which was the Python implementations for everything. If you take this to its logical extreme, I would have loved to have had a stable set of evolvable public REST APIs for controlling compute network storage of all ways and not have OpenStack be dependent on the implementation that we currently have.
Why do I say that? Because I’m clearly a Python developer and I love coding in Python and all of the OpenStack projects are developed in Python. Python isn’t the be-all and end-all solution for every single type of service. There’s also a huge ecosystem of golang and java code. I can’t stand that I’m saying java code, but yes. There is a huge ecosystem of java code that is running at scale distributor systems in great ways. I think we have limited ourselves in a lot of ways by saying that the Python implementation of these projects is OpenStack versus focusing on the APIs and allowing innovation to happen at the implementation layer. We’ve done a good job … We’ve done a really good job actually in OpenStack of promoting the adapter pattern which is having drivers enabled that will allow you to switch out the underlying infrastructure implementation of things.
Let’s say the database or a message queue or any of the network drivers and that kind of thing, you can flip a bunch of switches in OpenStack which is probably you could say too many switches on OpenStack to deploy different kinds of networking solutions and different database systems and all sorts of stuff. We’ve done a great job of allowing that kind of plug ability. What we haven’t necessarily done and there’s a lot of people that completely disagree with me on this is say, “You know what, let’s think of the next Nova. We’re not going to change the API, we’re going to continue to evolve the Nova API as being the OpenStack compute API but we’re going to make a part of it golang,” or “We’re going to rewrite the scheduler to be in java, whatever.” I’m just throwing things out here, but the point being we’re limiting ourselves by saying this has to be in Python because way early on we said OpenStack was both the implementation and the API.
I personally that that’s a limiting thing. The reason APIs matter is because it is the first impression that any developer has with the OpenStack ecosystem is via the public REST APIs. They may be using the Python clients or FOG or GeniCloud or [LipCloud 00:41:23] or whatever, but those are all just speaking the OpenStack APIs, the public REST APIs. The consistency of those APIs, the user-friendliness and ease of use and whether or not they make sense, structurally make sense are all things that go into the very first impression that our ecosystem gives to its developers which developers are what is going to drive adoption of OpenStack more than anything else. It’s what drove the adoption of AWS. It’s what drove the adoption of Heroku. Its developers. If we don’t have a good first impression to those developers and that first impression is generated from those APIs, then we’re going to lose out which is why there’s an API working group where we focus on things like consistency guidelines, making sure that little things that seemingly, you know, taking one edit at a time or piece-wise don’t seemed to be that big of a deal.
Whether or not, you know, one post to some resources returned to 201 created or 200 okay http response code may seemed trivial and petty but when you look at the flourishing of projects especially under the Big Tent model and the project structure reform and all of these new projects that expose REST APIs, what impression does it give our users if you go to each of these REST APIs and they’re all different, they’re all returning slightly different responses or structure of the JSON payload, some support different response codes, some don’t care. That type of consistency is that first impression that we give developers and right now, it’s not a good one. That’s one of the reasons why that working group was created and folks like Everett Towes and Chris Dent and Ian Cordasco, Miguel [Greenberger 00:43:22] all working to make the APIs more consistent and produce guidelines for both new projects and existing projects to evolve to a more consistent API style over time.
Jeff Dickey: Jay, do you think OpenStack is kind of ready for the masses? I keep thinking about this lately. It seems very much like high school where I think the folks doing OpenStack, you know …
Jay Pipes: I challenged a high school to deploy OpenStack.
Jeff Dickey: No, I mean, it’s the cool kids that are doing OpenStack right now and the more enterprises I’m working with on the OpenStack projects, they’re not there. They may never be the cool kid doing OpenStack and the developers are … They’re not going to take these applications that are making millions and billions of dollars that they’re updating and just completely convert them to cloud native and do different things. Again, do you think this is a for the masses infrastructure platform or it will stick with the cool kids for awhile?
Jay Pipes: Some things are. I was actually never one of the cool kids in high school. I sympathize with the legacy of IT folks. Even though I tried to push modern next gen type of cloud application architectures inside of Mirantis and also in the OpenStack community. I do sympathize with the fact that a lot of these IT guys are saddled with legacy infrastructure that they just … That it’s very difficult to change. The AT&T, it was like a change of the direction of a battleship or an aircraft carrier. It takes a long time just to change any sort of little direction.
I think there are pieces of OpenStack that are easier to wrap heads around for the legacy folks. I think the work with the VMware team has done in Nova and Neutron to allow compatibility between your traditional VMware data center, IT stuff and OpenStack services has gone a long way towards that sort of mismatch that you saw. Legacy IT is very much still a VMware-centric type of place and being able to move workloads between your sort of legacy VMware clusters and also run them in OpenStack using the Vcenter driver was a daily thing to get introduced into the OpenStack ecosystem.
When it comes to some of the storage layer, we have a lot of vendors in the center that I’ve introduced drivers for the more legacy type of non-distributed block storage systems, your NetApp’s, your EMCs, that kind of thing. Yeah, we’re slowly adding in stuff that will seem more familiar for those legacy folks. Whether or not it’s easy for an IT manager to deploy an OpenStack solution and make that available to her internal clients is something that we’re working on. It is not easy … You compare OpenStack with something like CloudStack which has this beautiful viewing installer, it’s super easy to setup and you compare it to your stock OpenStack, you know, how do deploy OpenStack with Chef or Puppet, whatever. It takes a lot of work. I personally think that Fuel goes a long way, which is a Mirantis thing, but a Fuel goes a long way towards a graphical installer that works. It’s got limitations just like the CloudStack thing has lots of limitations, but it allows sort of folks that aren’t used to this highly distributed systems and the sort of open install type stuff to get their feet wet in OpenStack without sort of jumping in the deep end.
Jeff Dickey: I know Mirantis has great VMware support. What do you think that legacy folks … You think it’s going to be more of a trend for OpenStack managing ESX or do you think it’s going to be OpenStack on top of ESX?
Jay Pipes: That’s a good question. I could sort of bandy on about some architectural issues in Nova that are sort of limiting both Ironic and the cluster of hypervisor and stuff like vCenter that hopefully we’ll try and address in the next couple of release cycles and make it more of a better fit than the KVM model. I imagine that you’ll see a lot of sort of converge type appliances come out in the next 3 to 5 years where you’ve got either Ironic … Hopefully, Ironic but maybe some other systems that manage the bare-metal provisioning and allow ESX or Hyper-V directly on bare-metal and then OpenStack managing work loads across those different … Then also have KVM installed all within a single rack or a converged appliance.
I think that’s probably within the next 3 or 5 years, you’ll see quite a bit of that which will help with the legacy IT stuff to moving towards the more modern architecture where it can be completely managed by an OpenStack solution, but your guess is as good as mine as to who will be the first to market on that particular type of thing.
Niki Acosta: Yeah, certainly there’s people trying to do that now. We talked to Platform9 awhile back and that’s their whole schtick, you know, ex-VMware folks kind of coming together to bridge legacy and OpenStack footprints under one, dare I say, single pane of glass. You know, VMware made some announcements yesterday. It looks like they’re with Pivotal and their photon and light wave stuff and it seems almost like they’re kind of stepping ahead as infrastructure as a service and going straight to platform as a service.
Jay Pipes: Makes sense with Cloud Foundry. I’m not familiar with the light wave or photon you said?
Niki Acosta: Yeah, light wave and photon.
Jay Pipes: I’m not familiar with those. I mean, but Pivotal’s … Pivotal role with Cloud Foundry. It makes sense for VMware to pursue the platform layer as well. I mean, they’ve already got a pretty lock on infrastructure stuff. At Mirantis in the Murano project, we kind of take the viewpoint that we’re not going to choose … We’re not going to take sides in the platform sort of wars, I guess. Murano, we wanted to happily install Cloud Foundry and allow users to use the Cloud Foundry APIs, same with Kubernetes for continuing orchestration or Mesos…
Niki Acosta: For those new to … Because I’ve actually done a lot of studying up on Murano and I think it’s incredibly cool. Can you explain that for people who are not familiar with that project?
Jay Pipes: Sure, I mean I’m not … I am certainly not going to do like some sort of product pitch or anything like that but Murano is your sort of generic application catalog and installation system so that you can describe your applications and things that go into your VMs and it will catalog the instructions for installation and the architecture, the topology of the VMs and into this nice manifest and then it just sort of point and click integration with Verizon go install Kubernetes on X number of VMs, go install Mesos to all GMVMs, go install Cloud Foundry on this topology of VMs. It just leaves it at that. It doesn’t try and communicate with Cloud Foundry by the Cloud Foundry APIs or anything like that. It just sends up the application by the developer to use those applications where there’s Kubernetes or Mesos or Cloud Foundry to use the native APIs that the developers use to for those particular applications.
There’s this … I won’t go into that, yeah, I won’t get into the Magnum thing. I still have some open ended questions on what exactly Magnum is doing versus what can be done with something like Heat or Murano, but I’ll leave that for another day. I know there are people that are like screaming in the back of their heads like, “What is Jay talking about? He doesn’t know anything about Magnum!” I’ll just leave it at that.
Jeff Dickey: Can I plug something real quick before we get off that topic?
Jay Pipes: Of course.
Jeff Dickey: Redapt and Mirantis and Google came together last month and did a demo showing kind of Kubernetes and OpenStack working together. Kind of just Google that. It was a great thing.
Jay Pipes: With Murano?
Jeff Dickey: Yeah.
Jay Pipes: With Murano installing Kubernetes?
Jeff Dickey: Yes, it was a demo and then a panel with everyone. Maybe you’ll Google that. It’s a pretty good one.
Jay Pipes: Cool.
Niki Acosta: I’ve listened our podcast today and I can’t help but think that if I was somebody that was thinking, just entertaining the thought about getting into OpenStack that I might be a little scared to get an OpenStack. It sounds like there’s a lot of maybe some politics and pettiness, you know, ugly babies being called out. Hypothetically speaking, if I was someone who would be leaving with that impression, what could you say to me to change my mind? What’s good about OpenStack?
Jay Pipes: The open is good about OpenStack. And the stack as well. I mean, there are so many different areas of technology as well as documentation and blogging and writing and advocacy about cloud computing that you can get into without getting involved in code reviews. You could do a whole heck of a lot of stuff without ever firing up Gerritt and participating on a code review. I wouldn’t let my talk of the chaos of code reviews and the core review teams dissuade anyone from getting involved or go into the mailing lists.
There’s been actually a couple of people recently that have posted to the OpenStack dev mailing list saying, “Hey, I’m a student. I’m studying this. I want to get involved,” and literally within an hour, you’ll get people saying, “Hey, I’m from Ironic or I’m from Heat. We’ve got these things like if you’re interested in this kind of stuff, come join us on IRC. We’ll walk you through some things that we’d love for you to take a look at.” There was a guy recently that … I can’t remember his name, but did the same thing on the mailing list. He’s like, “Hey, I have a experience in this particular networking piece,” and immediately people were like, “Yeah, please come on IRC or let’s discuss this on etherpad….”
You’ll find the community, for the most part, extremely welcoming. Getting involved is really not a problem and finding things to do is certainly not a problem. I think that just maybe you’ll find so much to do and each project is different. It’s not like they’re all exactly the same. Depending on the area of interest that you might have, whether it’s networking or storage or compute or who knows? Governments.
Niki Acosta: Just to point out too, there is a tremendous amount of demand for people who know OpenStack. I mean, any project. Huge demands. Every company that’s involved in OpenStack is hiring for OpenStack positions, left and right. It’s not limited just to engineers or developers.
Jay Pipes: Right, it’s product management, product development kind of thing, yeah. It’s a highly competitive ecosystem out there. Trust me, I’ve been trying to gobble up as many people into our team as possible. It’s kind of funny. When we talk to people that have OpenStack experience that we have come in to interview with us, 9 times out of 10, they have already been contacted and/or already interviewing with 1 or 2 of the other major players in the ecosystem. It is a very highly competitive market out there.
Niki Acosta: It pays well, it’s fun and the community is a blast. I mean, I’ve thrown back a few with Jay and with Jeff actually. They’re probably …
Jay Pipes: Vancouver is coming up.
Niki Acosta: Spectacular group of people. Speaking of Vancouver, do you have any talks like where you’re going to be? What you’ll do?
Jay Pipes: Well, Chris Dent and I are doing I believe it’s on Tuesday is that API … Why APIs Matter talk. That’s the only thing that I’m doing on the conference side. Most of my time will be spent in the design sessions. I mean, feel free to find me in the hallway and say hi, introduce yourself. I’m always happy to meet people.
Niki Acosta: He’s nice. He’s a nice guy.
Jay Pipes: Generally, yeah. I haven’t taken a look at any of the, sort of, non-design summit, non-conference schedule, you know, the night activities but I’m sure there will be quite a few. My wife, Julie, is going to be joining me in Vancouver just like she did in Paris and we’re taking a week vacation afterwards and staying around the area so very excited for that. I think we’ll go like take one of those little flow planes out to Vancouver Island and sort of tootle around there for a little while. I’m looking forward to it. I went to Vancouver, it was probably 10 years ago when I was … 9 or 10 years ago, I was working for MySQL and did a couple of seminars in the University of Vancouver, British Columbia and it’s absolutely gorgeous. I don’t know if you’ve ever been there but it’s really a great place. I’m really looking forward to that.
Niki Acosta: We’re looking forward to seeing you there for sure and we can’t confirm now, without a shadow of a doubt, that we will be podcasting from the OpenStack center, right Jeff?
Jay Pipes: Sweet.
Jeff Dickey: Yes, that’s going to be awesome.
Niki Acosta: Monday and Thursday, probably a shorter format but if there’s anyone out there who would love to participate, we’d love to hear from users, operators, project team leads, people like you Jay. If you are interested, definitely hit up Jeff or myself on Twitter. We would love to have as many people as possible. Recording times are Monday and Thursday from 1:00 to 6:00 pm during the summit. Good stuff.
Jay Pipes: Cool.
Jeff Dickey: It’s going to be awesome. Can I give a Vancouver travel advise for folks going there?
Niki Acosta: Yes.
Jeff Dickey: Do not see say you’re going up there to work, say you’re going for a conference. They will detain you and question you.
Jay Pipes: Really?
Jeff Dickey: Yeah. Say you’re going on a conference.
Niki Acosta: You’re there for … You’re going for a conference?
Jeff Dickey: Yeah.
Niki Acosta: I didn’t know that the Canadians … Obviously, that’s your neck of the woods more than it’s mine down in Texas but I didn’t realize that they have that kind of I don’t know.
Jeff Dickey: They’re very strict up there with their jobs there. They’re just doing their job.
Niki Acosta: I did see an episode of That’s 70 Show where they cross the border to buy beer, like one of the [inaudible 01:00:24] guy is on there, I can imagine maybe it’s something like that, I don’t know. All right, last question Jay. Who do you want to see on the show? 2 people.
Jay Pipes: I had a think and … There’s a guy named James Bottomley who is a Linux kernel guy that I’ve seen a couple of presentations of his and although he doesn’t work sort of actively in the OpenStack projects, he’s been at the OpenStack conferences and I think would be really, really interesting to get the early container technology story from. Then also, Tim Bell. I don’t know if you’ve had Tim Bell on the show but he is from Cern and he’s one of the folks on the OpenStack Foundation User Committee and have him talk about how they scale OpenStack would be awesome.
Niki Acosta: Great suggestion.
Jeff Dickey: Talk about massive … I think they’re the largest I’ve heard of.
Jay Pipes: Yeah, one of the largest.
Niki Acosta: [Crosstalk 01:01:44].
Jay Pipes: Tim is always fun.
Niki Acosta: He is a good guy and you know, they’re doing this incredible work at Cern with creating black holes and I’ve read something recently about them finding like … Yeah, they’re like finding many tiny black holes and the possibilities of extra dimensions, just crazy stuff that’s way cool. That’s a good suggestion. We should definitely try to get him on the podcast. Who is the other guy? James Bottomley?
Jay Pipes: James Bottomley, yes.
Niki Acosta: James Bottomley.
Jay Pipes: I can e-mail you his e-mail address.
Niki Acosta: That’d be great.
Jay Pipes: I think he’d be really, really interesting to chat with about the birth of container technology and sort of chat with him about how container technology is going to change OpenStack and cloud computing in the next 5 to 10 years.
Niki Acosta: Awesome. We love those kind of conversations for sure. Jay, we cannot thank you enough.
Jay Pipes: Thank you for letting me on and babble.
Niki Acosta: I know you got like 4 dogs and they’re very well-behaved.
Jay Pipes: Yeah, they’re all outside.
Niki Acosta: They’re being great. Thanks for giving us a peek on what happens and kind of look at the end of the day, you’re sharing what’s the hard work that goes on that leads up to something being truly open source and freely available for anyone, literally anyone on the planet, to use and consume. I think that’s what makes kind of OpenStack quite spectacular. Thank you for sharing your knowledge, your expertise and a few good laughs and who do we have next week, Jeff?
Jeff Dickey: Next week, we’ve got Edgar Magana.
Jay Pipes: Say hi to Edgar for me.
Niki Acosta: We will do.
Jeff Dickey: He’s from Workday, so it’ll be good.
Niki Acosta: He was also in the deck of cards for the most influential OpenStack people so definitely looking forward to seeing him and then, we have … Anne Gentle’s confirmed, correct? Yeah?
Jeff Dickey: Yes, she’s the week after.
Jay Pipes: Excellent.
Niki Acosta: Anne Gentle will be on and by the way, I did notice Jay that you said she as an operator. Thank you for doing that, I love when someone goes an occasional she in there. It’s good for the women in the tech.
Jay Pipes: You’re welcome.
Niki Acosta: All right, cool. Well, thanks everyone for watching the show. Tweet us. We look forward to seeing you.
Jay Pipes: Have a wonderful day, guys.
Niki Acosta: Thanks, Jay.
Jay Pipes: Bye.
Jeff Dickey: Thanks, Jay. Bye, bro.