Have you heard Weston Jossey explain his company’s hybrid cloud strategy before? If you haven’t, get comfortable and enjoy the recording of our most recent OpenStack Podcast. In it, Wes explains why his company chose to deploy a private cloud, what they use it for, and how they use it in conjunction with AWS to achieve maximum efficiency. Seriously–if you’re looking for tips about how to do hybrid well, this is the podcast for you. Wes makes it easy to understand both the upside and downside of private clouds in general and OpenStack-based private clouds in particular. Plus, he’s extraordinarily honest about what OpenStack is good for and what it’s not so good for. Among other things, in this interview, Wes discusses:
- Tapjoy’s big data architecture
- How they handle 5-10 billion transactions per day
- The TCO of their OpenStack-based cloud
- How he finds great hires in a competitive market
- How Tapjoy automates deployment
Have a show idea? Tweet Jeff and Niki at @openstackpod
See past episodes, subscribe, or view the upcoming schedule on the OSPod website.
To see the full transcript of this interview, click “Read more” below.
Niki Acosta: Hello, Hello, Hello! It’s Niki Acosta from Cisco filling in for Jeff Dickey from Redapt, and I’m so excited about today’s guest, Weston Jossey! Introduce yourself.
Weston Jossey: Hey, Niki! Thanks for having me on. As you said, I’m Wes; I’m Director of Engineering over at Tapjoy. Man, I guess, Niki, you and I have known each other for what, almost coming up on a year…
Niki Acosta: Yeah!
Weston Jossey: … since I started working with Metacloud and doing a bunch of stuff on OpenStack.
Niki Acosta: Yes, and we’re really excited! We’ve received a ton of feedback from our listeners. Thank you to our loyal listeners, who said, “Man, it would be really great to hear from more users.” So, we reached out to Wes. Wes has spoken at an OpenStack, two OpenStack Summits, right?
Weston Jossey: Yep. Well, no, one OpenStack Summit and then recently spoke at Inter- or not Interop, spoke at-
Niki Acosta: Cisco Live.
Weston Jossey: Cisco live in San Diego, yeah.
Niki Acosta: What was that like to speak at the … That was the Paris Summit, right?
Weston Jossey: That was the Paris Summit. Yeah, it was terrifying, is probably one way to define that. We flew out to Paris on a Sunday night and I spoke on a Tuesday night and I was still majorly jet-lagged by the time I had to give the presentation on Tuesday morning. It was great; it was a gigantic amphitheater. Where were we Niki? It was …
Niki Acosta: The Palais de Congrès or something.
Weston Jossey: Palais de Congrès, yeah. So when they walk you in and you have to do your rehearsal the day before, you’re going through the same entrances where all of the theater people normally walk through, with gigantic posters everywhere of like all the different shows they’ve put on, all the different artists that they’ve actually had there, the gigantic concerts. Then you ride up this elevator into the back of the amphitheater and then you walk out on stage and it’s, what, 2, 4, 6,000 seats? I have no idea how many seats are in that place, but it’s just these gigantic lights beaming down on you and it’s completely empty. They’re like, okay, now, do what you’re gonna do tomorrow, but for an empty room. You’re just absolutely terrified!
Then yeah, gave that talk in front of thousands and thousands of people and hopefully it resonated the story of how Tapjoy came to be involved in OpenStack and our journey to get to where we are.
Niki Acosta: You geeked out meeting Tim Bell back there, didn’t you?
Weston Jossey: Yeah, so you had Tim Bell on, what, a couple weeks ago when you were up in Vancouver, is that right?
Niki Acosta: Yes.
Weston Jossey: Yeah, and if anybody hasn’t had the opportunity to meet Tim Bell yet, one, probably one of the nicest people in tech, right? If we had a “Nicest Person in Tech Award,” Tim Bell would at least be nominated, if not be the winner. He gives tons of great talks about the story of CERN and how they started using OpenStack and how they’ve integrated into their technology. But more than anything it was just great sitting back there and talking to somebody who makes you stare at him and wonder at how he and his team manage to do the things that they do at CERN, and the good that he does for the entire world obviously with the research that they’re doing there. It’s pretty spectacular in getting to be involved in it!
Yeah, I sat backstage and was a little bit nervous and hadn’t actually met him before, but knew who he was, and kind of fumbled my first conversation with him. Then sat backstage after he gave his talk and we chatted for a little bit. Just a really great guy, it was good to see him in Paris. I’m hoping he’ll be in Tokyo in the fall. I know he was up in Vancouver. I don’t know if you guys fly him, or if the OpenStack Foundation flies him everywhere, but hopefully they should.
Niki Acosta: Yeah, funny thing about Tim, I was talking to him at the last summit in Vancouver, and some of the guys on his team were having some beers. He had a few, but I’m like talking to him about his family and the fact that his wife knows who Niki is, because she follows him on Twitter and I tweet on him every once in a while. I was asking him about life at CERN and his wife is actually a particle physicist at CERN. He was telling me about how his kids go to school with the kids that are basically the other UN kids, and the area that they live, and how everyone hates them, because they kick everyone’s ass in the science fair. It’s not even a close game!
Weston Jossey: Daddy let me run a quick experiment on the particle accelerator, won me the national prize for the science fair. Yeah, those are the perks of having a father who has 20,000 computers at his disposal, right?
Niki Acosta: Totally! Then he was talking about how right now they’re doing an upgrade at CERN, and I don’t even remember … I think he told me, it was probably in Celsius, but right now the CERN Hadron Collider is like the coldest place on Earth right now, because they have to-
Weston Jossey: Right, it takes a while to cool it down, and so they have to do it over the course of weeks and weeks and weeks.
Niki Acosta: So cool! Anyway, fascinating stuff.
Weston Jossey: I don’t know; I lived in Boston this past winter and I’m pretty sure Boston was the coldest place on Earth for a period of time, for about four weeks. At least the most snowy place on Earth it felt like.
Niki Acosta: I know; my cousin sent me a picture of her house and she had carved the walkway, and the snow was past her head.
Weston Jossey: Yeah, I’m still recovering. It’s a beautiful day outside; it’s probably about 78 degrees, sunny, gorgeous, and I still whenever I close my eyes, I just see mounds of snow just above my head, and I see my dog trying to run through it and just disappearing and then popping up and then disappearing again, and then popping up.
Niki Acosta: Where are you from Weston? You’re not used to snow.
Weston Jossey: No, that’s the thing; I’m used to snow. I was born in California actually. I was born in Los Angeles, but then grew up in Ohio and spent my entire childhood there. We would get snow … I mean, we would get snow! But like a big winter would be, “Oh, man, we got a foot of snow this year,” like over the course of the entire year. It was not normal to get 6, 7, 8 feet of snow like we got this year, and I’ve lived in Boston now since 2008. I moved here after college, had the pleasure of going to Northeastern University for a year, got into their PhD program, summarily dropped out of their PhD program to go into industry. So I’ve lived through a couple of Boston winters, but nothing compared to the experience of last year.
I live right next to the Tapjoy offices where I work, and we’re about two blocks away, so I can walk into work. I basically have no excuse to ever not get into work, other than today where I’m doing a podcast, because if I did it at the office I’d have people behind me teasing me and taunting me. We had a coworker who lives down in Quincy which is down in South Boston, and he, I am not exaggerating … He could not get into the office for an entire month, could not get into the office for an entire month! It was absolutely insane, so yeah, you should be lucky and be happy to be in Texas Niki.
Niki Acosta: Because it’s like 80% humidity and like 95 today or something?
Weston Jossey: That’s what air conditioning is for. It’s fun.
Niki Acosta: Yeah, right.
Weston Jossey: You got central air.
Niki Acosta: Totally! I know; I’m wearing a sweater inside, because I’m actually kind of cold in my own house.
Weston Jossey: Well, I’m wearing my OpenStack sweater.
Niki Acosta: Yeah! Oh, my God!
Weston Jossey: I figured I’d do some branding.
Niki Acosta: No, I was so distracted, that I forgot to ask you the question we always start off with. You told some of it, but how did you get started in tech?
Weston Jossey: Oh, that’s an interesting question. I don’t know; we’ve got, what, another 50 minutes, so I’ll do the next 40 … no. I’ve always been a bit of a nerd, as I’m sure most of the people who are joining in on this podcast. I got into computers probably when I was about 8 or 9 years old. I remember getting our first Gateway computer playing Myst in my basement. I was always interested in computers, but my parents didn’t really want me to spend all that much time on it. They kept telling me to go outside and go do other things, but I always had showed a lot of interest in it, so in high school my high school girlfriend, her father was a software architect. He actually was one of the first people to ever graduate with a computer science master’s from Duke University.
Niki Acosta: Wow!
Weston Jossey: I foreshadowed, I guess it would have been my junior year of high school, I went with him to his work and walked around with him and got to see what his job was like. I remember actually thinking back then, wow, he just did a ton of meetings! That doesn’t seem all that interesting … And because he was a software architect, and software architects sit in a lot of meetings and talk to a lot of people and don’t necessarily get to write a lot of code themselves. But really had … It kind of made an imprint on me. I stayed interested in it, but really didn’t do much around software and tech until I got to college, and I actually went to a really small, private liberal arts college in upstate New York, actually where his daughter was going to school. That was part of why I went to New York.
So they had a small computer science program. There were five people in each of the graduating classes, but I actually went in originally for physics, did a semester of it, loved physics, but also took my first computer science class and loved computer science! One of those … Most kids by the end of their sophomore year of college are still trying to figure out what they want to do in their career and what they want to study, I knew … Niki, are you still trying to figure out what you want to do, Niki?
Niki Acosta: Oh, totally! I don’t know what I want to be when I grow up.
Weston Jossey: Yeah, you’ll figure it out. Yeah, I ended up taking a bunch of computer science courses. As I said, it was a very, very small department. I would have classes of ten … I had one class of one where I was the only person in the class. I had a fantastic advisor by the name of Tom [O’Connell 00:10:10] who’s still at the department today, who really mentored me. More than anything, because it was a liberal arts college, it was all about problem-solving, and so you’re always trying to learn how to solve problems, less so than you’re learning a trade skill. When I was interviewing for a bunch of jobs out of college, I wasn’t necessarily doing that great in terms of getting the placements that I wanted. Secretly I just wanted to work at ESPN, but they didn’t let me.
So I ended up being able to get into a PhD program at Northeastern, as I said, because I thought it’d be interesting to teach … Did it for a year, loved it, but just turned out to not be for me, and then kind of jumped into industry. Then I’ve had a completely different story over the last, I guess now, what, six, seven years of how I got here, but that’s sort of my tale of how I got into actual technology.
Niki Acosta: So in a nutshell the last six or seven years, I know you were acquired into Tapjoy?
Weston Jossey: Yeah, that’s right. I worked for a company called Viximo; we took social games that were on Facebook, because don’t forget, about five years ago social games were really big on Facebook … Farmville, and we would take those games, work with the publishers and actually port them to other social networks around the world. Because believe it or not, five years ago, there were also other social networks around the world other than Facebook. We would take games and put them onto Spanish social networks like Tuenti, we would take them onto German social networks like StudiVZ, and so a lot of what I was doing was around trying to read API documents in German. That was exciting, trying to get all that stuff to work.
It turns out good business, not a great business, just weren’t quite making enough money to justify all the venture capital we had taken. So a sister company of ours, Tapjoy, through North Bridge Partners who is our VC firm, basically said that they thought there was some overlap in skills, ended up making a bit of an acqui-hire, and myself, I think along with nine or ten other people joined Tapjoy. Basically it’s a company that’s all about helping app developers make money. That’s really what it all comes down to. We got acquired by them and then I started my journey from basically just being a regular software engineer to starting to dive into systems and diving into operations. So I basically knew nothing when I started at Tapjoy, when it came to actually how to run systems. At Viximo we had maybe 100 servers, somewhere in there, all running inside of AWS, but it wasn’t really something that I was actively involved in. We had two other guys who were fantastic at doing that kind of stuff, so it was all off in the netherworld for me.
But when we joined Tapjoy, they were understaffed on that front. There was one full-time guy running 500 plus servers for their entire infrastructure, and we were on fire all the time. Everything was on fire! We had tons of downtime every quarter, 20 to 30 hours a quarter. It was just constantly like we were just trying to bail water out, and so I’m a fixer. I don’t know if that’s maybe the best description I have for me, and so when I see a problem I just want to fix it. Myself and this other guy by the name of Eric [Abbott 00:13:34] were just like, “All right, we’re just gonna jump in and we’re gonna just start working on this stuff.” For the next year we basically started the process of overhauling everything we could under the sun, upgrading their databases, getting them onto the latest technologies, making sure that everything was properly automated, really trying to take Tapjoy and evolve it, and so we had a bunch of people working on it, a bunch of great stuff happened. Then started transitioning into what eventually ended up becoming our journey to OpenStack.
Sorry, I’m just rambling, so I’m gonna …
Niki Acosta: No, that’s great. This is great! You’re making my job super easy today.
Weston Jossey: Yeah, yeah. Yeah.
Niki Acosta: We don’t even need Jeff today.
Weston Jossey: I’m sorry?
Niki Acosta: We don’t even need Jeff today.
Weston Jossey: Yeah, well it would have been nice. Ironically, one of the two people that I worked with at Tapjoy who was part of the systems team when I first joined was a guy by the name of Jeff Dickey. I was like, “Oh, man, does he have a podcast with Niki? Is this like a thing that he does now?” A different Jeff Dickey apparently.
Anyways, we got to a point in Tapjoy where things were stable, things were working, but it was still not terribly efficient, is probably the best way that I can put it. We ended up bringing in somebody who had been at Tapjoy early on at the beginning. He understood our systems well, so by the request of one of our board members who was going into every single board member, or board meeting, pounding on the table saying we’re spending too much money, we’re spending too much money on AWS, we gotta figure stuff out, started to explore a bit of the OpenStack community, and explore what it would possibly look like if we were to build it ourselves. Being a cloud company, being a company that basically was totally invested inside of AWS, totally invested in the concept of elastic resources and being able to spin up VMs and spin down VMs and the core concepts that come with operating inside of a cloud, we didn’t feel like it was prudent to go down the route of just running just racks and racks and racks of bare-metal.
So what we decided was that we were gonna basically start to explore a bunch of different partners in the OpenStack space and price it out and try and understand what it was gonna cost if we wanted to use AWS, use one of the bare-metal providers that are out there now, use an OpenStack provider themselves, or try and build it ourselves and put all that together. It ended up being the last two years of my life, basically this journey of how we ended up getting here. So we basically started this whole process in April and actually at the Cisco Live Conference that you and I were at, I had that diagram threw up on the board, and really I wish it was Paul Santinelli, board member, yells at board and says, “Need to spend less money on AWS!” That’s probably like the beginning line. Then a couple months go by and we basically start to narrow in our focus and decide, okay, it looks like this is gonna work, it looks like we can actually do this, what kind of ROI can we get.
We worked with a ton of different partners and we looked at the big companies like a Dell, like a HP, we looked at smaller resellers, and what we basically settled on was we wanted to build it ourselves in terms of we wanted to build- design the gear that we wanted that fit our use case. Then we desperately needed help, and that’s basically where my relationship with you actually started, because we got to a point where we were gonna end up having to hire four or five OpenStack engineers to basically try and help us build, deploy, manage, upgrade, do all this work to be on OpenStack, and we knew we wanted to be on OpenStack. But then we actually got introduced to Metacloud, which is now … I’m sorry; [inaudible 00:17:45]. Go ahead.
Niki Acosta: Cisco OpenStack Private Cloud.
Weston Jossey: Yeah, you guys need an abbreviation. I just can’t do it. I’m sorry.
Niki Acosta: I’m sorry, Wes. No, I will pass that video feedback to the folks that make those decisions.
Weston Jossey: It’s a bit wordy. But at the time Metacloud was still an independent company, had just raised a good amount of VC funding, was nice up into the right … Or I guess I was up into the left, I’m not sure which direction it’s gonna look like … Up into the right trajectory and man, the first time they came in, it was magic! You guys have such an incredible team! It’s really remarkable! I don’t know how often you actually get to see them nowadays now that you’re still down in Texas?
Niki Acosta: I troll them online.
Weston Jossey: Yeah?
Niki Acosta: Yeah.
Weston Jossey: Yeah, so they came, they pitched us on this idea of why having a managed, private infrastructure with Metacloud was such a great idea, totally sold it, totally bought it!
Niki Acosta: Did they take you through that the TCO/ROI tool? Did they do that with you?
Weston Jossey: They didn’t have to.
Niki Acosta: Oh!
Weston Jossey: We had already done it ourselves. Yeah, so we had already done basically the modeling, so for us we had basically gone through and decided we wanted to price in everything. We wanted head count included, we wanted gear included, we wanted just general maintenance included, warranties, everything, everything was gonna get priced into the ROI. Because Amazon, that’s part of what Amazon is providing to you, right? They’re providing cloud engineers in effect behind this … You don’t see it, but they are. They’re running all that infrastructure. We priced it all in and long story short, it ended up depending on the metrics that you were using, saving us over 10X what we would have paid if we had decided to stick all of our stuff back into the public cloud. Yeah, and it’s been a glorious event ever since!
Niki Acosta: Gah-lee! I wish every customer was like you! You make my job so easy Weston Jossey!
You actually have a lot of the data behind this too, which I think is really cool, because you can actually show … It was the GigaOm article that came out that I think may have had a screenshot of that data, an older version, or what have you.
Weston Jossey: Yep.
Niki Acosta: Talk to us about what you’re actually do- because you guys sort of have a hybrid infrastructure. Tell us about all the components in your hybrid infrastructure and what you’re running on each other’s environments.
Weston Jossey: Yeah, because we were born in the cloud and we’re so heavily invested inside of AWS, we knew that we wanted to have ourselves living in AWS and inside of OpenStack. We didn’t want to have to make a choice, because we didn’t feel like we had to make a choice. I think there’s a lot of the conversations that you hear in the OpenStack community, it’s like how do we win, how do we beat AWS, how do we basically become the dominant solution. I don’t know if that’s necessarily the way that I would even have to think about it in that I think that OpenStack and AWS are very complementary systems, exceptionally complementary systems really! What we decided we were gonna do is when we were picking out our colo provider, who ended up being Equinix, which is also a short plug for Equinix, a fantastic company, great to work with.
We decided that we wanted to plant ourselves down right next to AWS in Virginia. We wanted to be as close as we could be geographically to their facilities for a multitude of reasons; one, it makes it easy to get a direct connect between the two sites, and then also because it’s just better to be geographically in the same area in terms of latency. We wanted to have a low latency connection between our two systems. What we’ve basically done is we’ve built this kind of bridge and this gap between our two systems, and we have inside of AWS we have our core production API systems, so when you’re interacting with Tapjoy, you’re interacting usually with servers that are sitting inside of AWS. But when you’re interacting with our data science department, when we’re making an optimization or we’re making a decision about what to do with one of those requests, that’s actually getting funneled all the way down into our OpenStack environment.
A huge number of the requests that are flowing through our production system right now actually hit both AWS and OpenStack all of within milliseconds of one another to basically make us better. You were asking what we run in there, and the reason why we have to hit our OpenStack cloud is that we’re basically running all of our Big Data solutions inside of OpenStack. So those sort of solutions for us are Hadoop, HBase, MemSQL, and then a multitude of other myriad of things that you do when you’re engineers and you build out stuff all the time. We have those servers running in these two different areas, and then we’ve written a bunch of technology to try and gap that stuff. We recently open-sourced a project called Dynamic, we’ve written some stuff called circuitbreakers that we use internally, which is all about intra-datacenter communication to make sure that if the link is ever broken, we have clear fallback plans, kind of a waterfall, as you will, if things don’t work correctly.
We’ve basically just built everything around this concept of duality of OpenStack and AWS.
Niki Acosta: Are you in a situation where you own your base and write your spike, or are you just kind of keeping those environments separate by function?
Weston Jossey: I would say we’re closer to owning by function at this point. I think we could easily go down the road where it is by spike, and I think that that’s the most cost-effective solution out there. So for a particularly large company, and I would even put Tapjoy in that category, it is cost-effective to basically think about your infrastructure as a moving walkway. At the beginning of the moving walkway is something like an AWS or a big, public cloud where you’re paying for elasticity, you’re paying for flexibility, and then as you keep moving down that walkway, everything becomes more predictable. You understand what you already have, you understand what you need, and that’s when you think about transitioning it into your own private infrastructure and transitioning it into an OpenStack.
That’s the beauty of it, because then everything kind of gets transitioned, and then all you’re ever using the public cloud for is new stuff and spike, which makes it exceptionally cost-effective, and it basically uses the two solutions for what they’re best at.
Niki Acosta: How many transactions are you guys doing per day at this point?
Weston Jossey: Oh, gosh.
Niki Acosta: Or requests?
Weston Jossey: Yeah, requests, there’s a couple of numbers. External to internal, I want to say we’re coming up on 3 billion a day, something like that, external to internal. Internal, like if you think about micro services, there’s a lot of chatter inside of the network. If we took all of our services and probably combined them together, I mean, we’re probably talking about, I don’t know, 5 to 10 billion requests per day, or 5 to 10 billion transactions that we’re basically internally trying to handle and do calculations on. We were consuming 30 to 50 terabytes a day of data that we basically consume, process, run through Hadoop, analyze, store. We have well over a petabyte worth of storage that we use for basically near-term storage, that’s our near-term storage is I think close to 1-1/2 petabytes.
Then if you talk about long-term storage, we actually are pretty heavy users now of Amazon S3 and Glacier just because the pricing’s pretty good, and so that extends out that lifespan even further. Tapjoy is actually a really kind of cool company in that we’ve basically held onto every transaction, everything that’s flowing through our system for the last five years. We have every single record of it. So I can actually replay a bunch of our production traffic going back five years ago. It’s pretty-
Niki Acosta: You probably have a file on Niki Acosta. You’re the folks responsible for showing me videos for like cleaning products and stuff that fit my demographic, right?
Weston Jossey: Your persona, yeah, it would be technically your persona. Yeah, someone who cleans I think would be your persona. I don’t know what that would actually categorize into our system. Yeah, I mean, we have a great data science department that does some really good stuff around making sure that the ads that we’re showing we’re showing at, we like to say at the right moment and with the right context. So it’s more than just I want to show Niki an ad. The majority of our ads are opt-in, so we’re actually a rewarded ad network, and so users choose to interact with us, which is kind of a different thing than you’re used to than when you go onto ESPN.com and something pops up in your face. We tend to do our advertising model is through opt-in, so people are saying, “I want to get something in this game, so show me a video.” Or, “I want to get something in this game, so I’m gonna go fill out a survey or I’m gonna go check out an app.” There’s a bunch of different things that we allow users to do.
So if we saw that you were requesting a video, we want to show you a video that’s relevant to you, because we want to make sure that you’re getting in front of the right brand that matters to you. So that’s what we need, this massive amount of infrastructure that we purchased and put inside of our OpenStack deployment, is to make sure that we can make those decisions and we can make them in quite literally milliseconds. We need to make those decisions [inaudible 00:27:18].
Niki Acosta: Is that why some of the apps that I download for my son ask when your birthday is?
Weston Jossey: That’s COPPA compliance that you’re getting.
Niki Acosta: Oh, okay.
Weston Jossey: That’s because-
Niki Acosta: He’s always running up to me going, “Mommy, I need this app!” It’s because it was one of those [inaudible 00:27:31] videos.
Weston Jossey: That’s because, and I’m not a lawyers\, so I may butcher it, but I’m sure our lawyers are very good in making sure that we’re COPPA compliant. But basically if our users are under a certain age, you’re not allowed to collect information about them and use it for advertising. That’s why they will ask you for that stuff is to make sure they’re respecting your son’s privacy.
Niki Acosta: Oh! Well, in that case, I want to be 15 forever. Vallard is actually one of our Cisco team members, he’s our micro-services geek. I think he got kind of excited. Val asks, “Can you tell us more about your Big Data architecture? Do you run instance per hypervisor, why OpenStack instead of just a standard Hadoop cluster? Thanks.”
Weston Jossey: Yeah, it’s a really good question. Yeah, when we went through and designed our infrastructure we basically decided that we were gonna look at one hypervisor per node and then we were going to have one Hadoop instance per physical node, so everything devolved down to what would that physical piece of infrastructure look like. We kind of were a little bit creative, and this was two years ago remember when we were doing all this work, and the Haswell chips were just about ready to get done inside of Intel, and so we were looking at the super low-ell chips, because Haswell was doing a really good job of getting the power really far down. We specifically bought, I’m pretty sure they’re the 1265LV3 series, if anybody wants to look them up. They were the low-power Haswell 1265’s, and the whole purpose of that was we wanted to draw the power down as low as we possibly could, because we wanted to cram as much as we possibly could into a single rack and we wanted to go basically as dense as we could outside of going purely down the open compute route.
So on every single node we basically said, okay, we got four cores running on this node, we’re gonna max out the RAM, so no those particular instances it’s about 32 gigs or RAM. Then we basically had slots for four physical hard drives, so we put 4 TB hard drives, and that became a physical Hadoop node, because we were looking at basically having a one-to-one spindle to core ratio, so every core on there wanted to have a single hard drive against it, make sure that we had sufficient IOPS to basically do our workloads. In our previous environment we were starting to see that we were running into some IOPS issues and so we wanted to make sure that we improved that. At the time when we were making all these purchases and I guess technically today, the SSDs just hadn’t come quite down enough in price yet where we could really justify the cost. It was close, it was really close. I think if you looked at the three-year TOS … Not TOS …
Niki Acosta: TCO, ROI?
Weston Jossey: ROI, some three-letter acronym, yeah. Now it’s probably cost-effective, but two years ago it wasn’t quite there, so that was basically how we landed at that. Then the question of why not just Hadoop running on bare-metal; a lot of it came down to flexibility. We don’t just have Hadoop running on these servers, and so we wanted to make sure that we had as much flexibility as we could around making it a virtualized environment which people were already comfortable with, and making sure that the spin-up and spin-down was easy, the bootstrapping, imaging, everything was really simple, because we didn’t want to have to hire operations engineers who would maintain that stuff for us. We wanted this to mirror what we were used to in a virtualized environment.
I mean, quite literally, one of the nice things with OpenStack is you can run the same image of an OS in OpenStack as you can run inside of AWS. You can actually have the same one running in both places. It was that sort of simplicity, it was that sort of ease of use, and so one of the questions that I get that derives out of that question is, “But what about the performance trade-off?” “Oh, my gosh! It must be so much slower.” It doesn’t even register on the radar, it’s not even one of the problems that we’re really thinking about. People undervalue people’s time, people undervalue ease of maintenance and maintainability and flexibility. What happens if we want to move that Hadoop infrastructure off to new infrastructure a year down the line and we want to repurpose all of that infrastructure? Having something like an OpenStack running makes it really easy to do that, it makes it very simple. For us it was all about maximum flexibility at the small trade-off of basically negligible performance hit.
Niki Acosta: Wow! Thanks for that. I hope, Val, if you’re still listening, he answered your question. Talk to us about your micro-services layer and what you’re doing in between and how you’re able to programmatically connect AWS and your Cisco OpenStack Private Cloud.
Weston Jossey: Right. See, you can get the Cisco Private OpenStack Cloud … See, I still can’t even do it … You can get it out, you’ve got it like conditioned.
Niki Acosta: Sorry!
Weston Jossey: Yeah, so micro-services, we had three years ago when I first joined Tapjoy, we had the traditional monolith, right, a big one, single architecture, everything was in the same app. So we started a process led by our architect, Chris Chiodo, of starting to break some things up into smaller services and a lot of things came with that. We wrote a bunch of different small, light frameworks that helped us to interact with different components. Then we used technologies that make it easy to decouple yourself from being tied into the same infrastructure. So if we think about a request that flows through Tapjoy’s system, somebody’s on their phone, I’m on my iPhone and I’m playing a game and I’m looking at an app, it flows into our system. It’s gonna hit a set of servers, which are basically gonna do first-level triage. They’re gonna do what is this request, what do we need to do.
Then it’s gonna basically branch out and start to hit different micro-services, so maybe I needed to look up some information about the device, have we seen it before, is it a new user, do I have any context about what games they play, does that stuff matter? So we wrote a light framework that basically handles the interaction there between those two systems. Then it flows down into now let’s say that I need to show an ad to you, and so then we’re going to make a request off to our data science infrastructure, and that’s going to now flow through what we call the circuitbreaker. The circuitbreaker is all about detecting and watching for anomalies between the two systems. For example, the circuitbreaker could be configured to say if requests start taking over 100 milliseconds in the 95th percentile, we’re going to consider that the infrastructure is starting to fail, and so we’re going to trip the circuitbreaker.
We trip the circuitbreaker in the same way that your house trips the circuitbreaker to keep your house from catching on fire. It’s much better to have your hairdryer go off than it is to have your whole house catch on fire, and you just go downstairs and push the button. Our circuitbreaker then says in the event that this is failing, what do I then go off and do, and then we have redundancy built in there, so that could either hit a different micro-service that has maybe a more simplified system that doesn’t require us to go into OpenStack, maybe it hits a pre-cached assessment of what we’re supposed to do in this particular situation that’s been pre-computed behind the scenes, so not quite real-time, but still quite good. Then if that system recovers, that circuitbreak automatically re-trips itself, comes back online, and then a request flows in.
On the other side of it, we have to be able to consume a bunch of data. So inside of our OpenStack cloud we’re having to consume just tons and tons and tons of data that’s flowing into this AWS infrastructure, and so we actually have a couple of different systems that handle that. One, we have a system that is basically logging everything that is happening in our system and throwing it into Amazon S3, which is basically just written in pure JSON and can basically get consumed by any consumer that can understand that JSON format. We also then do some real-time pass-through, and so I was mentioning that we open-sourced a project called Dynamic, which allows us to basically do … It’s a topic subscriber system, similar to what you would get out of like an Amazon SNS, or a Zaqar, which is the OpenStack equivalent, and basically it allows us to durably and consistently publish messages into this and pull the data out of it inside of our data science infrastructure in a very durable, clean and consistent way, all over the course of milliseconds.
We’re able to consume all this data in real-time, so that way if you then connect to an app, I can see that information. Then seconds later if you want to request an ad, I can make a decision based off that data that I already just received just seconds ago. Then just to make it that much more fun, we also now have a third layer which is basically our reporting infrastructure, which now is starting to flow through a new system, which is still being worked on by our engineers which uses your traditional Kafka/Spark infrastructure for roll-ups and a bunch of other stuff, the story getting Postgres so that way everybody connects with the data. A lot of moving pieces, but when you’re talking about moving around billions of requests per day, everything gets a little complicated.
Niki Acosta: It’s amazing how much you guys have automated. Look; we talked to enterprise customers who were not born in the cloud, and obviously this is a very difficult mind-shift for people to make. Obviously you came in … There was bare-metal I think, right?
Weston Jossey: Yeah, we did have some bare-metal.
Niki Acosta: And moving from bare-metal and being born in the cloud with Amazon, the whole self-healing … You gave us, I can’t remember it was … The average length of time that your AWS instances were live, were online was like …
Weston Jossey: It’s about four days, less than four days now. Yeah, because we’re constantly elastically scaling ourselves up and down to meet traffic, and we basically have a policy that says the oldest server gets shot in the head. Yeah, I think one of my favorite things is, and one of my favorite reminders that I always tell myself, is never name your servers. Never name your servers, ever! Ever, ever, ever! Never have like a Cedarwood 7 and, “Oh, man, Cedarwood 7 is such a good database.” “I love Cedarwood; Cedarwood is like the least temperamental of all the servers that I’ve ever owned!” The second that you start to name your servers, the second that you start to think about them as like physical entities, then you’re gonna be sad if they go away, which means you’ll do things on them and you’ll do things to them that are not repeatable, and that it’s way too difficult to … Oh, see there’s the dog. My dog wins the award of being first to bark.
So you’ll basically start to do things that are not actually good for your infrastructure, and so for us everything is ephemeral. If a node is misbehaving, we shoot it in the head. If something isn’t working correctly, we shoot it in the head. If a disk fails, we shoot it in the head. We basically also treat our OpenStack infrastructure the exact same way. The node failures we don’t try and move the VM over, we don’t try and do anything cute behind the scenes; we literally just bring it down, it’s over. It’s lived its life. We’ll get a new hard drive brought in, we’ll bring it back online, everything’s good. I’m sorry; my dog is hiding behind the scenes, because my wife decided to leave. She’s very sad. She’s a very sensitive animal!
Niki Acosta: Mine’s passed out. We’re good.
Weston Jossey: No, it’s good.
Niki Acosta: So when you say you shoot it in the head, Wes, is that all happening through automation? If so, what are you using to test performance degradation?
Weston Jossey: Oh, that’s a great question. We use a variety of tools to basically monitor our infrastructure, whether that’s in OpenStack or inside of AWS. The front-line system is a system that we built on top of Sensu, which is a great monitoring framework written in Ruby, which basically we use in conjunction with things like CollectD and StatsD to monitor individual system performance. Imagine we’re seeing anomalies through CPU usage or we’re seeing anomalies in terms of the disks not behaving correctly; that sort of stuff our systems will pick up and if it’s a critical issue, it may page somebody, whoever’s holding the pager. Fortunately not me at the moment. Or it’ll send out an email and basically the next possible person can log in and triage it.
For the most part we haven’t gotten into systems yet where we proactively have systems that kill other systems, which I know is something that we talk about internally, and something that companies like Facebook are really aggressive about, and for the better. To be perfectly honest, the reason we don’t do it is because I’m really paranoid, and I’ve always had this fear that we would write these scripts that okay, if you detect this anomaly I want you to go ahead and just turn off that server. I’ve always had this fear where we’re gonna have a production issue and everything’s gonna look like it’s having an anomaly, and it’s gonna be like, “All right, great. We’re turning off that entire infrastructure,” because the code itself was written in just slightly inconsistent way where that sort of stuff happens. For the most part we will terminate things ourselves, or they will naturally age themselves out as part of our infrastructure.
That’s in terms of actually terminating, but to make sure that that server isn’t misbehaving actually in production, we use things like HAProxy health checks and running health checks on our systems to make sure that they’re behaving, and if they fail the health checks they basically get bounced out and they don’t run. They’ll be online, they’ll be sitting around, they’ll be hanging out. We can use them to inspect them and see what maybe happened, maybe it was a bad code deploy or maybe there was a bug in some of our automation code, but it won’t actually serve traffic. It’s just hanging out, just waiting to die I guess. It’s a sad ending for those servers, but yeah, that’s their end.
Niki Acosta: What about your teams, thinking about obviously you said when you joined Tapjoy you knew nothing.
Weston Jossey: Yeah.
Niki Acosta: There were people who joined with you. What skills have you had to learn, or were you looking for people who knew stuff to bring in or was it pretty much just kind of scrappy and you all had to go out and learn this stuff on your own? What does that transformation process look like, moving from bare-metal and where you’ve got one guy who is responsible for X number of servers to now being responsible for these systems?
Weston Jossey: We’ve had this really interesting group of people on the team over the years. As I said, when we joined we had basically one full-time guy. He was an operations engineer and he continues to be an operations engineer to this day. Myself, obviously I’m a software engineer, so I was learning a lot of it. The other fellow who joined with me was a systems engineer/systems engineer. This guy would look at graphs all frickin’ day! He taught me how to look at a graph like nobody else’s business! It’s something that I’ve tried to instill down to the rest of my team over time. Between the three of us we kind of got the ball rolling. In terms of who then joined over time, it was a lot of I would say people early on in their career. We weren’t necessarily bringing on people who had ten years of AWS experience, because nobody has ten years of AWS experience. It hasn’t been around that long.
Outside of saying I’ve been an infrastructure engineer at Amazon since 2000, there are people who understand the infrastructure, so for us it was trainability, it was coaching, it was can we get you to think about how we approach problems and how we try to tackle issues. So for us we had a really great guy join, who now runs the team for me. I’m just a figure head now at this point, and a guy by the name of Marcus Walser, who has done incredible work, leveling up that team, and leveling himself up too! He had joined, he had some AWS experience, but really just took all of our automation, took all of our infrastructure to the next level, kind of backing into these core concepts, these concepts of resiliency and redundancy and how we think about our problems and how we think about solving solutions. He’s way better at it than I am! I mean, just monumentally just totally better at it than I am, because he just sees things differently than I do!
Then on top of that we managed to take people who came from a background who had never done any of this stuff before … I have a guy on my team who was part of the Startup Institute, which is a great program out of Boston, New York, and I think they have a couple of different areas where you can kind of pivot and reinvent your career and get into something new … Had zero operations experience, zero background in ops. We brought him in, two years later he’s a great senior engineer, really fantastic, has built our entire deployment system along with one other engineer that’s now handling, I mean, literally three to four dozen deploys a day through our system that’s flowing through this entire infrastructure, just some really great work! For us, and for anybody who’s out there working with people who well, we gotta go out and find somebody with 15 years of networking experience and 15 years of they know how to do X, they know how to do Y, I don’t know if that’s what you need anymore.
I think you need people who are hungry, you need people who care. I don’t know; is this a family-friendly podcast? Am I allowed to curse on it?
Niki Acosta: You can curse.
Weston Jossey: I can curse?
Niki Acosta: Yeah.
Weston Jossey: What it really comes down to is can you find people who give a f***! That is really what it comes down to. I don’t care if you have a background in ops, I don’t care if you have a background in engineering, I don’t care if you’re a Japanese poetry major, because guess what, our old head of operations was a Japanese poetry major. That’s the sort of stuff, but it just comes down to he cared, and people who care do a great job. Can you basically sort of define how you want your process to look, how do you want your systems to behave, and then find the right people who will work within that.
Niki Acosta: What do you use for … You mentioned automated deployments. Do you guys use Chef?
Weston Jossey: We use Chef for our infrastructure. Yeah, nice job Niki, fishing for that one.
Niki Acosta: I couldn’t remember if it was Chef or Puppet or Ansible.
Weston Jossey: Insert Chef, Puppet, Ansible here. It doesn’t matter, right?
Niki Acosta: Yeah. I’m thinking about SaltStack.
Weston Jossey: We use Chef, but we don’t use Chef for deployments actually; we use Chef for bootstrap. We basically will build an image and then that image is used as the base layer, and then every single server that comes online gets bootstrapped depending on what its function is and whatever its job is. They usually come from the same base image, more or less, and so then we use Chef to install any packages and software that we need to run on that system, which obviously a lot of this stuff is set up pre-Docker, how can we integrate Docker into the system. I’m hoping that maybe in a year or two we’ll be a bit further down the line, but for now a lot of our stuff works great. We don’t really have an ongoing problem there, but in terms of deployments, we actually built out an entire open-source project, also another plug for one of our open-source projects called slugforge, which is basically how Tapjoy does deploy, so similar to Docker and similar to how other systems will do deployments.
We actually build these tiny encapsulated slugs, which are basically tarball. It’s an executable tarball that includes information about how to install itself, how to run itself like a dpkg on Ubuntu. We basically can solve in on our box and then deploy to those servers through an automated system, and this whole abstraction layer is written called slugforge. It does all the packaging, building, deployment, and that’s basically how we are just constantly rolling out code throughout the day. We then took that, and because we’re engineers, did another abstraction on top of it, because slugforge is a command line tool, and nobody likes using command line tools anymore. So now we also have basically a dashboard inside our system called Deploy Board which handles all of our deployments.
Any engineer in the company can go in and basically take a GitHub pool request, plug in what the GitHub PR is, it starts the automated process, back-merges in master, checks all the specs, checks all of our continuous integration tests to make sure everything looks good, deploys out canaries to a set of servers so they can hop on New Relic, check to see how the server’s performing, is it doing what I expect it to, do some test requests, get literally there’s a green thumbs up or a red thumbs down button. They push that button and then it rolls the code out wide, and then it’s all done. That’s basically we have a complete and total self-serve deployment system that every engineer in the company is empowered to use to deploy out, whether it’s our core Tapjoy server product or to deploy out our micro-services around the company, or now we also, because a lot of our stuff is JavaScript-based, we also have asset deploys that go through there, so we’ll update our JavaScript. Just a lot of cool stuff.
Niki Acosta: It’s a lot of trust that you give your people. Do you set quota limits on usage for your folks?
Weston Jossey: Do we set quota limits? No. We right now have a high trust sort of system where everything right now is peer-reviewed, so at least two people have to sign off on everything. We have good process around what we expect people to look at in the peer review process. We have good tests that tend to catch things before they go out live, and then we also have this, as I was talking about, this verification process. We don’t just roll code out and have the engineer walk away and go get lunch, there’s actually a process where they roll it out to a set of servers. We watch it, we make sure that it’s behaving correctly, and then we go wide. That tends to keep major issues from every cropping up in production or suddenly it rolls up typo or something like that.
But we don’t necessarily institute anything like the Facebook Karma system where if an employee screws up, they can potentially get bumped down or bumped up, or whatever the case is. We haven’t needed it yet. We’re only 70 engineers, 75 engineers, only.
Niki Acosta: Only?
Weston Jossey: Only, yeah. But yeah, the system so far has worked well for us. We used to have ops do every deploy, and we couldn’t scale. We literally could not scale our business that way! We wanted smaller, quicker atomic deploys and having an operations team that it’s their sole job, it was way too expensive! It was time-consuming! Do I really want to drop 200 grand on the loaded head to basically roll out code on a daily basis? It’s just …
Niki Acosta: Or would you rather hire the great folks at Cisco OpenStack Private Cloud?
Weston Jossey: Yeah. Yeah, you could do that too, but will you roll out my code for me Niki?
Niki Acosta: Let it be known that I’ve never even taken you to dinner, but this is all you. I’m not promising you, I don’t know, a t-shirt or anything.
Weston Jossey: No, I don’t think you’ve ever actually even given me a t-shirt.
Niki Acosta: Oh, man! Do you want a Cisco OpenStack Private Cloud t-shirt?
Weston Jossey: I would take some retro Metacloud t-shirts. I don’t know if you still have some anywhere.
Niki Acosta: You know what; you have one. I’m gonna send it to you.
Weston Jossey: Okay.
Niki Acosta: I have two; I have a medium and a large, and I’m gonna send them to you.
Weston Jossey: Yeah. I mean, I don’t know if this is a good thing or a bad thing; does it just show I can be bought for nothing? Is that basically what the moral of the story is that you don’t give me anything?
Niki Acosta: I think that’s within the Cisco customer policy, whatever, and they’re like pre-date Cisco. I think it says Metacloud Austin, American Apparel, like hipster fit. They’re great, you’ll love ’em!
Weston Jossey: Well, I’m very fortunate like the trip down to San Diego. I mean, Cisco was great about making sure to bring me down there. I actually, when I was walking around the night before my talk, I was walking down the street and ran into the sister of one of my really good friends from college, and she was with her dad and her mom. Her dad is a network engineer and I completely forgot about that. He was asking me, he was like, “Oh, how are you liking the conference?” I was like, “I haven’t really done anything yet.” He was like, “I can’t believe you haven’t done anything yet.” I don’t take advantages of the things that you’ve already given me, Niki, in terms of free conference passes and checking out a lot of the cool stuff that you guys do.
Niki Acosta: But you also self-admit that as an operations guy that anytime you hear the word VXLAN or whatever, your head explodes and you’ll change the subject?
Weston Jossey: My eyes glaze over, I take a shot of tequila, because that’s the only way I can get through it.
Niki Acosta: It sounds like what I’m gonna do after I get off of this podcast so I can do my expense reports.
Weston Jossey: Oh. Yeah, hey, I did one on Sunday too. It took me three hours, it has not been approved yet. Speaking of tequila, I also have a gigantic amount of booze behind me.
Niki Acosta: Is that some Bitters over there? That’s pretty bougie.
Weston Jossey: It is. Yeah, my wife is very good with cocktails.
Niki Acosta: Well, that’s not PC.
Weston Jossey: Speaking of tequila …
Niki Acosta: Oh, yeah, Patrón! That’s good stuff! You’re so cool, Wes!
Weston Jossey: I try. I mean … Do you still have your dreads? I can’t even tell from here.
Niki Acosta: Yeah, totally! But I took the extensions out, so now they’re just like half dreads.
Weston Jossey: Now just kind of quasi-dread.
Niki Acosta: Yeah, but now that they’re started, they’re just gonna grow out on their own.
Weston Jossey: Yeah, anybody who hasn’t had a chance to meet Niki in person, she literally is as crazy in person as she is, probably. Well, actually probably way more so.
Niki Acosta: I don’t even drink.
Weston Jossey: You don’t? Yeah. Well, when I showed up at the party in Paris and everybody was just having such a good time, and you were like, “Hey!”
Niki Acosta: I know; I was so excited to meet you in real life, because I think at that point we had like talked on the phone, and I meet you up with the OpenStack folks and then I didn’t really see you until that party I think.
Weston Jossey: Yeah. Well, I’m lucky; OpenStack, one, it’s such an incredibly growing community too. I heard Vancouver it went up like another 2,000 people or something like that.
Niki Acosta: Vancouver was nuts! It was nuts!
Weston Jossey: There you go! There’s your dog, see.
Niki Acosta: There’s my dog.
Weston Jossey: Do we know what the Tokyo numbers are looking like so far?
Niki Acosta: No clue. I mean, the request for papers is already out, which is crazy, because I feel like we just finished Vancouver and it’s already time to submit new papers. I’m sure there’ll be quite a few people there. I mean, I think this is … Other than Hong Kong. Now that we’re a lot bigger and there’s a huge following of people in Asia, I think it’ll be good. Australia too, so it’ll be great.
Weston Jossey: Are you going to be doing a Women of OpenStack talk when you’re out in Tokyo?
Oh, no! I think my Hangouts has crapped out on me.
Niki Acosta: [inaudible 00:54:21].
Weston Jossey: Oh, sorry. Hold on, my camera’s …
Niki Acosta: No! Aaaaah! I have a private message. Can you hear me?
Weston Jossey: Yeah, I can hear you.
Niki Acosta: Someone’s asking me how often do you push code to production?
Weston Jossey: How often do I push code? That’s a good question, so yeah, we’re doing over two dozen deploys a day almost every day Monday through Thursday, although we also have an office based out of Korea. So it’s actually more or less a 24 hour continuous cycle, because they’re deploying code while we’re sleeping. I would say almost to deploy an hour, if not more every Monday through Thursday. Friday’s are reserved for kind of special occasions and special events that need to get stuff rolled out. Then Saturdays and Sundays … Well Sunday’s not in Korea, but Saturdays and Sundays tend to be a bit lighter, because police don’t roll out code on Saturdays and Sundays.
Niki Acosta: Aren’t you glad you don’t have to have a pager on you anymore when stuff breaks?
Weston Jossey: I am.
Niki Acosta: Talk to us about your new role, because I think you’re a good example. We always talk about if you can get to a point where you’re automating, then you can actually start providing value back to the business instead of just keeping the lights on. You’re like the perfect example of someone who’s done just that.
Weston Jossey: Yeah. Sorry, I’m just gonna turn off my camera, because it seems to be glitching.
One of the really cool things for us is that I have a great boss who’s very supportive and allows us to transition between roles. So I was actually out to dinner last November with a bunch of our product managers and a couple of our engineers and had had a couple cocktails, and started pounding the table saying, “You give me five engineers and I’ll get this company $5 million in incremental revenue.” Because I’m that kind of guy. Then two months later I had a team of engineers and a revenue goal, so I got what I asked for, whether or not I meant it. Yeah, I transitioned, so now I run two different teams; I run this red team and then I also run my ops team as well. The red team’s focus is on core business delivery, so how can we drive value to our users.
Our users are multiple different people, that can be publishers, so the game developers who make awesome games and we want to make sure we’re doing good stuff for them. So the users and making sure that we’re putting the best possible ads in front of them, because they’re trying to get rewarded for stuff that they want in the game. Then our advertisers, making sure that we’re taking their ads and delivering it to the right user on the right game and at the right moment. I get to live in this kind of cool realm where I get to look for problems to fix that are potentially great revenue generators for the company, and I have an unbelievable team who have been working at Tapjoy, most of them for north of two years, so they understand the business, they understand what we’re doing, and have done some really just fantastic work over the last six months.
Niki Acosta: You want to give a shout-out to them? I mean, you just mentioned some names along the way.
Weston Jossey: Yeah. No, I mean, I would just say shout-out in particular to my boss, Sean Lindsay, who’s obviously done a lot to help me and my career, but then obviously everybody on my team … James Moore who did a ton of the work on our infrastructure and actually getting our OpenStack deployment. He’s also on the red team now, and he helps me build out a lot of our cool stuff. I’ve got people like [Xia Fang 00:57:38] who was just recently promoted, so congratulations to her. Ryan Detzel, Craig Smith. I have a new designer on my team who I just started full-time, she was an intern/contractor and now full-time employee, [Catherine Wy 00:57:53]. I have a great team across the board, great ops engineer, Jeff Clark.
I don’t know; I feel like I’m at the Oscars now and I’m gonna forget somebody, but I love my team. I love the people that I work with. I’m very, very fortunate! I come to work every day, because I have a family there. That’s the best possible situation to be in.
Niki Acosta: Awwww! All right, last question on the podcast, and if you mention somebody that’s already been on, that’s okay.
Weston Jossey: Yeah, so the number one person … I’m sorry; you broke up, but I’m pretty sure I know who … The person that I want you to go after is a guy by the name of Craig Tracy. He works for Blue Box. I’ve had drinks before with him in Boston, we’ve met a bunch. He retweeted this. He’s a great guy, he’s really smart, understands OpenStack really well, has done a lot of cool work over his career in OpenStack. I think he would be a really interesting person to talk to.
Niki Acosta: He understands that you’re not looking for a solution from Blue Box, because you’re happy …
Weston Jossey: I’m a happy Cisco customer, yes. I-
Niki Acosta: I know Craig. Hi, Craig!
Weston Jossey: Yeah, Craig’s a great guy and the folks over at Blue Box are really great at what they do too.
Niki Acosta: Cool! Anyone else?
Weston Jossey: Off the top of my head, I don’t know, can you just do Tim Bell Hour once a week?
Niki Acosta: Yeah, we should get Tim Bell on for a full hour.
Weston Jossey: You know what; I would be-
Niki Acosta: Have we had him on for a full hour? I can’t remember.
Weston Jossey: I don’t have a specific person in particular that I think you should have on; I have a specific idea. I think you should have somebody on who’s thinking about OpenStack and have somebody who is not a vendor, not a pitcher, and just have them basically talk about the pros and cons of OpenStack, somebody who’s maybe inside of AWS thinking about moving over and maybe just have a big conversation about what that might look like.
Niki Acosta: Not a salesperson?
Weston Jossey: Not a salesperson.
Niki Acosta: Got it. Cool! That’s a good idea!
Weston Jossey: Yeah.
Niki Acosta: That’s lots of fun. Yeah, that’d be a really good talk for the OpenStack Summit too!
Weston Jossey: That would be! Oh, that would be fascinating actually! Yeah, that would be great. Well, because it’s that live dialogue, it’s that angel on one side and devil on the other of what are the trade-offs, what are the pros and cons, because everything has a trade-off, and if you can have an open dialogue about that kind of stuff.
Niki Acosta: Great! Yeah, it’s a balance.
Weston Jossey: Yeah.
Niki Acosta: Cool! Well, Wes, thank you so much! Like I said, I wish every customer was as willing to do this type of stuff. I know we’re not like bending over backwards [inaudible 01:00:15], but it’s really awesome stuff, forthcoming and transparent about what you do in terms of your use of OpenStack and AWS, and the growth that Tapjoy’s had has just been amazing! So thank you so, so much! I can’t see you; all I see is like [inaudible 01:00:32] of your face.
Weston Jossey: Yeah, sorry. I put [inaudible 01:00:35]. Hey, I love you! You’re fantastic!
Niki Acosta: I love you too! Thanks Weston!
Weston Jossey: I love everyone in the OpenStack community!
Niki Acosta: Yeah, and hopefully we’ll see you in Tokyo maybe.
Weston Jossey: That’s my plan. Yeah, just tell the OpenStack people to make sure to ship me out there and I’ll gladly go.
Niki Acosta: I’m on it. I’ll tweet them.
Weston Jossey: All right, thanks Niki.
Niki Acosta: All right, bye Wes. Bye everybody.
Weston Jossey: Bye everybody.
If you’d prefer to listen to the podcast, check it out here: https://itunes.apple.com/us/podcast/an-openstack-podcast/id913351029?mt=2