Cisco Blogs


Cisco Blog > Cisco Interaction Network

Hack, Whack, Chop that…firmware…

I had a customer at Interop NYC yesterday ask me if I still hack stuff? Still?!?! Of course!! It’s something that’s just in your blood daddy-o! I just like to hack stuff. It doesn’t matter really what it is, I just enjoy the challenge of figuring out how stuff works and how to bypass certain controls. I am not trying to be a whank about it and post how to steal a case of Sundrop from a Dixie-Narco vending machine, I just want to know from an engineering stand point. When I see electronic firmware based stuff work I always wonder; “How did they code that one up?”

 That’s all it takes to get me started. Now my wife is not a fan of guns but if I started purchasing all the stuff I hack around here weekly, my guess is that she would change her mind real quick… So I need another method…a more…low cost method of hacking a device without ever purchasing the device. Firmware baby!!! Matter of fact, it is very rare for me to get actual gear. I just go for the low hangin’ fruit! Firmware! say with me…What do we want! Firmware!! f.i.r.m.w.a.r.e!!

 Many vendors out there today offer up firmware freely without authentication or with only a email address so they can gather marketing data. I just use a 10 minute emailer like Mailinator or I give them Robb’s email and then I start downloading firmware. The firmware can be like the wardroom door to Narina if you look deeply into it.

 Here’s the thing. Many vendors out there today do not have firmware developers in house. They have a marketing plan, money, call centers , etc…but code jockeys are something that is normally outsourced. These code houses do not just buzz the code for one vendor but for 50 or more. Now to keep this code straight from vendor to vendor many code houses place comments in their firmware.

 These can be comments about debug interfaces, HARD CODED ACCOUNTS!!!! Private keys, hidden commands and yes even backdoor passwords. (I just found two days ago in a vendor device) Basically, low rent firmware hacking is really a piece of cake to understand.Plus it can really yield huge…benefits. Remember Stuxnet? Oh Yeah…  Most firmware out there today is unsigned and unencrypted which means I can read it in a simple hex editor. But before your go download firmware and opening it up in your favorite hex editor, here are a few pointers to get ya started:

 Tip 00×01 I normally like to look for bootcode if I have the choice. If not, I just deal with what I have.

 Tip 00×02 Firmware files are in hex and have a ton of unreadable data in them. I am interested in about 1% of the file which contains ASCII text. The first thing I do is run firmware thru the *Nix command; Strings. Strings is a command that will print out any ascii character sequences that is followed by a an unprintable character. The default character limit is 4 but you can change that. A example command would look like this:

strings -t firmwareName.bin

Simple but powerful command with few options. The -t is an option I use to tell me the offsets just in case I need to…use them later on…

 Tip 00×03 Some firmware will be compressed in what is called ZLIB compressed chunks. In outsourced code larger then 3Meg this is very common. There is a great tool called DeeZee which is part of the Black Bag Tool Kit from Matasano. It is older but works really well still for binary dissection. DeeZee will search thru a binary file for ZLIB signatures then extract them and print out the results. Human behavior is such that we write and comment stuff out all the time. Look at the best practices for a simple ACL. If I run a file thru strings or view it in a hex editor and see nothing but unreadable crap, then I assume it must be ZLIB’ed or encrypted but that is very rare. I run it thru DeeZee, with the command:

./deezee firmwarename.bin

DeeZee will chew on it few a while then spit out the results into the same directory I run it at. I just do a LS to see the results, then view those results in my hex editor and Kazam! it’s hammertime!

 Now you have some readable ASCII extracted from firmware. Some of my results have been stuff like:

– FTP passwords

– Backdoor passwords for various ports

– Hidden SNMP community strings

– Mountable filesystems that actually allow me to mount the firmware and interact with it. Heck I have pulled off .pem files that allowed me to do a super effective bogus SSL connection!

– Internal server ip addresses of the code house

– Contact information, which is great for social engineering

– Debug interfaces with access commands

and of course some of the funniest comments you have seen. If you decode a OEM suppliers firmware you may have hit a real jackpot since most C code is modularized and reused in other devices, so this the gift that keeps on givin’!! At this point you can gather the data and test certain conditions you have mined OR you can move up to the graduate level of hacking and start looking at disassembly with IDA Pro and start installing rootkits in firmware. That is the true Holy Grail of hacking embedded systems, but we’ll cover that next blog.

What’s that? You want a how to and not just tips? Hmmm…OK, here is an excellent blog showing you the step by step details of reversing some code from a…product…. http://www.devttys0.com/2011/05/reverse-engineering-firmware-linksys-wag120n/

 Jimmy Ray Purser

 Trivia File Transfer Protocol

In the rockin’ age of 390B.C. Playwright Aristophanes wanted to show the world just how pretentious db whack he really was. In his play Ecclesiazusae the characters feasted on a dish called; lopadotemachoselachogaleokranioleipsanodrimupotrimmatosilphioliparomelitoaktakexhumeno-kichlepikossuphophattoperisteralektruonoptopiphallidokinklopeleioplagoosiraiobaphetragalopterugon

MMMMM…Please sir may I have another helping of…lopa, lopadotema.. just forget it. Where’s the gyro stand?

Contact Center: Experience Matters

Our latest episode is out!  This one is all about the Contact Center and we spent a lot more time showing examples of Social Media integration.  Everyone agrees that ‘the internets’ are a huge watering hole of opinions and valuable business data…but its overwhelming when trying to figure out just how you might leverage it.  Now you can integrate all that goodness with the maturity of business relevant call center and all the process that implies.

Jimmy Ray shares his personal experience volunteering to work in an actual (non-Cisco) call center all night long to gain better experience for how we developed this show.  Check out his blog for more details on this.

This show featured Packaged CCE (Contact Center Enterprise), Feature Rich Reporting, Finesse Agent, Social Miner and even a few field trips to see what Cisco is doing with their ‘Social Media Listening Center’ and a very cool mobile application we internally refer to as ‘Roadside Demo’ that will open your mind as to how mobile devices SHOULD be part of your Contact Center Strategy.

Enjoy! 

 

Robb

Tags: , , , , , , , ,

The Making of TechWiseTV Episode 133; Call Center Software…Experence Matters…

It’s certainly true that voice isn’t my favorite technology. Not that is bad or that I dislike it…it just doesn’t hold my interest like routing and switching or data center or even wireless. Man I just fall all over myself on that stuff!! When we are asked to do a TechWiseTV episode, the first thing I do is go back and watch any previous shows that we had before. After watching the Contact Center shows I came to one glaring conclusion; I don’t get it.

<insert ego here> Sure I understand the technology well enough. Also how to implement it and really support it like an IT geek should. I just didn’t understand WHY things like Social Miner, Stats, Desktop deployment were any more important than anything else. So for example; if desktop layout is so important, use a thin client and roll ‘um out. Or if stats and tracking is so important then buy a projector, write script and display that stuff on the wall man. Why is this a real selling feature?

So I contacted a peer of mine that works in a Call Center and I volunteered to work an eight hour shift (from 7PM to 4AM…Yowza…)  to see what it’s like and how users actually use the product in a production environment. I wanted to work in a non Cisco environment to get an untainted experience so that I would be better prepared to really analyze what we did and did not do.  He gladly took me in and after about an hour of online training and two hours of strictly supervised mentorship, I added my name to the “score board” and logged in as “Jimmy Ray” with a dreams off earning a green up arrow showing that I was moving up the leader board.

Wholly smokes! What an eye opening experience. First off, the turnover rate of staff is huge. I had three new folks also starting with me.  It’s not because of the management. Heck, they are actually very cool. Giving out bonuses, prizes right on the spot. Anything to keep their employees happy. I was impressed and surprised because I was expecting to be yelled at, hard drove to stay on the phone all the time, stick to the script, but no folks worked hard to win stuff and compete with each other.  The team of folks that manage the phones come from all walks of life. In one shift I met folks fresh off divorces, two laid off PhD’s from a medical company up the road, a few recent college grads, former educators, etc…basically, high quality folks looking to take a Mulligan and move on to the next phase of life.

To sum up my phone experience, it’d go like this; being yelled at, hung up on, cursed at, air horns and whistles  blew into the phone and loathed more than the United States Congress. Why? because I’m interrupting their chillin’ at home and sometime work life and boy howdy they like me know real quick by not be friendly or patient. Hey I worked as a product developer, a presales engineer and was in the United States Navy so I can handle rejection and being cussed at well. But man alive hearing not just “No” but some tasty cuss words can take its toll on your attitude and overall morale.

Wow! Now I get it. Certainly 8 hours does not make me an expert. It did really change my viewpoint. On the drive back to the Code Cave, I called Robb and woke his butt out of bed and told him we need to reformat this show.  He mumbled something about gummy bears driving on the A10 in Norwich then hung up the phone. I understand why call center users need to have a dynamic desktop that we can update with apps as skill levels/responsibility changes. I understand why an IT department wants to have a solution they can tweak on the fly without calling in a bus load of consultants. I understand why managers know that a call center is expensive to a company and the need to have customized reports in a ton of different formats to show success or failure. Even the ability to scale and add features with limited servers I knew would be important on really any network discipline.  However, in a normal call center solution takes 17+ servers with a strict mandate of “Don’t Touch These” from the consultants that seemed to be more like staff and consultants to be honest. Reminded me a lot of the old main frame days. Come on man!!

Take a look at this episode of TechWiseTV.  It airs on Thursday 19Sep2013. It’s a little different based upon me coming off this experience.  Robb and I formatted this show not based on an all IT experience, but a true usability story. The Contact Center team is really a great group of folks. I have had the pleasure of working with a lot of teams in our seven years of doing TechWiseTV. Without a doubt, the Contact Center team is one of my favs. The team is great, they honestly listen to customer feedback with a pencil to see what they can change. Although Cisco is a large company, this team moves with the nimbleness of a small startup. Watch this episode and even attend our follow on workshop http://www.ciscoworkshops.com If you need a call center solution, I betcha you’d really like the solutions package and licensing they come up with.

Oh…and if your phone rings during suppertime…cut ‘um a little slack…it’s a tough gig for sure.

Jimmy Ray Purser

Trivia File Transfer Protocol

Clarence Birdseye founded his frozen food empire with $7. It bought him an electric fan, cakes of ice, and buckets of brine.

My Top Three Fav Cisco Features!!

We all have things we love and hate about our job. Sometimes it’s silly process stuff like TPS reports…but many times the good far out weight the bad. Just like picking out who to go out fishing with. Folks are gonna get on your nervous and make you want to hit them with the boat paddle, you just pick the folks that make you wanna do that less than others. I think that was my wife’s strategy for picking out a husband too. Not to say that I haven’t been clocked with a boat paddle, skillet, metal detector, etc…

<Long pause of reflection> Anyway…

Cisco has many cool technologies for sure. And by technologies, I mean features. When I was competing against Cisco way back in the dark ages when we pushed packets up baluns…both ways!  if a customer said they are going with Cisco for one of these features I cut our losses and headed for the next customer.  This is a blog of my three favorite Cisco features that only Cisco has and is a great reason to buy Cisco besides the neat-o blue console cables.

Cool Feature 00×01: ISSU (In Service Software Upgrade)

Engineer’s dilemma; How Do I upgrade my switch code when my network is 24x7x365 with minimal to ZERO downtime in a five nines world?  When I FIRST came over to Cisco as an employee after getting my badge (and having a burrito at the Cisco café in building 17), I had to talk to the folks that developed ISSU. At my former gig we tried all kinda of things to get this feature to work on our gear. We just ran into tons and tons of problems. At first we tried modifying VRRP with something called XRRP and it just sucked. Customers hated it. Then we tried other snapshotting technologies and just couldn’t get close. See the problem is how do you GRACEFULLY start two primary software threads to the same process then keep increasing the NICE value so the secondary in now primary without panic-ing the kernel AND losing packets.  Oh and remember I need a back out process if I abort or lose power, a roll back process if the new code starts causing problems, and a solid commit process to give me one more chance to back out so it’s not a permanent change until I agree.  It’s a huge development challenge that took Cisco many many thousands of engineering hours to figure out. As a coder, I really admire the team that designed this technology and overcame all the obstacles to make this work. ISSU is dependent on NSF (Non Stop Forwarding) and SSO (Stateful Switch Over) and is hands down my single favorite feature at Cisco among all of products. If there was a Noble prize for networking; this would win hands down.

Cool Feature 00×02: STP (Spanning Tree Protocol)

STP is in nearly every switch out there today. However….with STP in its IEEE state you can really get in trouble quick.  The Cisco difference is HUGE here. First off, STP is enabled by default. In an Auto-MDIX world, this is critical. I don’t know how many customers I walked into that have had ports cross connected by users. Sometimes, for fun and sometimes, well, I had one user tell me that they cross connected a port because the cable was a trip hazard…Yep… safety…ummm…first I reckon…. Now consider all the options Cisco gives ya to protect your network from easy hacks and misconfigs; Port Fast, Uplink Fast, Bridge Assurance, BPDU Guard, Port Guard, Root Guard, Loop Guard, Etherchannel Guard, all the filtering AND PVST simulation  to allow MST to interop with Rapid PVST+. Nobody does STP better the Cisco. This is a deal killer if you’re a competitor. And, if you wanted the best description and write up on Spanning Tree protocol outside of the book; “Interconnections” by Radia Perlmann it’s hard to beat Cisco LAN Switching Fundamentals.

Cool Feature 00×03: OTV (Overlay Transport Virtualization)

When I was first induced to this feature, I was told it was “mac routing” ????WTF??? Mac….Routing?? Humph…Marketing people. I need to talk to the coders to see what this REALLY is. Well, their kinda right…it’s really more like a distributed control plane among all DC’s using the MAC address to advertise reachability. Designed for the data center interconnectivity it works by encapsulating MAC frames destined for remote DC’s in an IP packet, transports it across the network to the target DC where it unwraps it and forwards it on. OK, so how is that different from routing right? Here’s the thing, first, it doesn’t use the data plane for this. Honestly, (with respects to John Moy) to me, it’s more like L2 OSPF. Each Nexus head end switch keeps a MAC table for each remote destination and can recognize a MAC frame bound for another DC without having to climb the stack to do a routing look up. Heck it even handles multicast efficiently too. PLUS you can config and set this up in about 4 to 8 commands. Cisco likes to say four (marketing pays the bills baby) my experience is about 8. I also like that broadcast storm containment is built in so we don’t flood rogue frames out to all DC’s and it has loop prevention, multi-path, a rudimentary form of load balancing (layer two right?). My fav feature inside of a feature is the fact that we can add a new DC onto the OTV diameter and only config the new DC. The others will automagically update their tables to include the new DC to the OTV par-ty!! This is one excellent and well thought out feature. It’s everything TRiLL should have been.

If you could boil it down to three features, what would be your top three? List ‘um out and let’s see what ya think! It just goes to reason that there is a light and a dark side to the Force. Tune in next week to see my three Cisco features that I really dislike and why. Sure to piss some folks off without a doubt!!

Jimmy Ray Purser

Trivia File Transfer Protocol

-40 degrees Fahrenheit and -40 degrees Celsius are the same temp. It’s the point when both temperature scales converge and everyone agrees that wholly friggen smokes it’s too cold to be measuring anything out here, let’s get our tail back inside.

 

The Soundtrack of Technology

I’ve mentioned before that I grow up around a very musical family. I remember on Sunday’s sitting around on the porch with the whole family playing music. Heck anytime I go back home to Tennessee, we go and play music until early in the morning. Now I was never much of a musician at all.  Although to be quite honest, I’d stand up to the crap today for sure. “You kids git off my lawn!!!”

The one thing I noticed is that life itself has a sound track, so it makes sense to me, that technology also should have a soundtrack. So from the somewhat legal music files of Jimmy Ray Purser, here is my recommendation for tunes based on the technology.

Switches: Traveling Wilbury’s. There’s just something about the smooth melodic sounds of this group that just ties the tech together. Switches are the focal point of the network so the band has to stand the test of time and have a happy tempo for sure. Hey it’s switching, one of the coolest technologies out there. Getting in the Ether groove is easy with the Wilbury Boys.

Routers: Rush. With routing, you’re climbing the stack a little higher and getting into more complex stuff. Oh sure you still a command line commando using your weapons of Command-Tab (Alt-Tab for Bill’s crew) between sessions using alias commands to shortcut longer commands, etc. You need some heady-er tunes that you can sing a bar or two too but also just enjoy the more intellectual groove it puts you in. Hard to beat Rush for that stuff for sure! Dad gum I feel all Mozart smart and stuff listen to this without being a whanker about it.

Wireless: Any Industrial or Trance Electronica. Personally I grab Enigma most of the time but I feed Trance Soma thru the lab and soak it up. With wireless, you have to imagine most of it, then test. There are so many variables in wireless you really need to use music to expand your mind and float ya down Maxwell’s currents to Fourier’s shores.  Other bands come with lyrics and memories that divert your focus too much.  Honorable Mention and super close second: Yes.

Security:  Radiohead. Namely the OK Computer album. Oh man you have to remember you’re plugging your network into the world and challenging everyone. Radiohead gets you in the groove, smashes the record then eats it. That’s what you need as a security person. Think outside the friggen box, against the grain and break the beer bottle over your head.

Hacking:  Ramones. OK when you start to hack, you are breaking the rules and the law in most places. You can’t take prisoners and have to be more like Francois L’Olonnais. He was a French pirate that actually ate an enemy’s heart to terrify the enemy prisoners to make them show him an escape route out of certain capture. They did and he escaped to fight another day. That’s the mind set of a good hacker. Whatever it takes to get your data will be done. Honorable Mention: Sex Pistols/PiL

 Data Center: Boston. Data Center technology is truly a solution that you have to not only work with multiple vendors but every single piece has work perfect for the server to boot off the SAN. So it needs something different, ground breaking and high energy. Hard to beat Boston for that task for sure.  I mean come on man! Tom Scholz (lead guitar picker) didn’t like the sound of rock music so he invented his own!! Plus he was such a perfectionist he delayed an album over two years until the sound was just right. Oh yeah, that is a data center mindset right there!  Honorable Mention: Daft Punk. It puts you on grid baby!

Storage: Pink Floyd. Storage seems to be the most elitist of all technologies. They do EVERYTHING different on that side of the world for sure. Kinda like eating over at a friend’s house for the first time. Not that it’s a bad thing at all. Their methods and practices are tried and true. They are responsible for the data itself so the rules of zero loss tolerance are non negotiable. When I was a young boot camp storage rookie, I’d question why this or that and the result was the same; “You don’t understand yet, but you will in time if you keep at it” That was the same answer I got when I said; “WTF???” after listening to Pink Floyd’s Animals album.

VOIP:  Bjork. Yeah get it config’ed so you can turn this off and go grab a beer and chicken wings at 80’s throwback bar and jam to some hair band tunes!

And to finish it up; my personal happy song…When I’m in a great mood and everything seems to be going A+ you can just bet that the song; “Walking on Sunshine” by Katrina and Waves is on a loop in my head!

So…music tends to be a really personal thing. I think I’ve seen more fist fights over music then girls or sports teams. (I need new friends, I know…) Do you agree or disagree with my song selection? What is YOUR happy song in the sound track of your life?

Jimmy Ray Purser

Trivia File Transfer Protocol

The first record to sell one million copies was The Glen Miller Orchestra’s “Chattanooga Choo-Choo” way back in 1942