Tell me if this sounds familiar… you are asked to perform a penetration test on customer’s network to determine the security posture of their assets and the first thing they do is give you a list of assets that you are NOT allowed to test, because they are critical systems to the business. Ironic isn’t it? This is exactly the difficulty you can expect when performing penetration testing in the cloud, but multiplied by ten.
There is a lot to think about and plan for when you want to perform a penetration test in a cloud service provider’s (CSP) network. Before we get into the technical details, we need to start with the basics.
Questions to ask yourself
- What does my contract and SLA state about penetration testing?
- Does the CSP already have a team of penetration testers? Is this enough to meet your security requirements or compliance objectives?
- Are we hiring a third-party company to perform the penetration tests for us?
- Do we have our own penetration testing team?
CSP In-House Pen Test Team
If your cloud service provider has their own penetration testers that is great news! Not only does it show that they take security seriously, but it means that you can leverage their internal testing results for your own audits. If you do not have the money for your own penetration testing team (either in-house or third-party), you may be able to request detailed audit reports from the CSP relative to your company. Have information disclosure policies built-in to your SLA and monthly vulnerability reports delivered to you. Some types of compliance will say this is not enough and still require third-party testing. Honestly, you should perform your own penetration tests, either with your own team or by hiring a trusted and skilled third-party. If your CSP claims that they are secure, are you just going to believe them? How do you know they didn’t leave anything out of their reports?
Make sure you ask a lot of questions about what they tested & how:
- What was in scope? What devices, web apps, services, databases, hosts, subnets, and network storage? Make sure that any part of their Infrastructure (IaaS), Platform (PaaS), or Software (SaaS) that is relevant to you is included in the scope of testing and reports that are delivered to you.
- For IaaS, you may want to view their architectural design and security review documents, security policies and patch management policies, as well as any hardening guides that they used. In addition, you should request vulnerability scans to be performed on your part of the VLAN or subnet. Other than providing you facilities with HVAC, physical security, and hazard controls -- most of the technical security is your responsibility. Your job is to identify what security gaps have not been filled by the CSP and fill them, or get the CSP to do it.
- For PaaS, in addition to everything listed above, you should have your CSP verify that your platform is fully patched and up-to-date. Ensure that your installation of Apache, Ruby on Rails, Java, PHP, Bugzilla, Drupal, etc. is up-to-date and not running a default installation with default user names, passwords, and configuration files. If it is not protected, it is your job as a PaaS tenant to protect it. Some of these security responsibilities will be shared by the CSP and yourself, so make sure you know where the line is drawn and who is responsible for what through an SLA.
- For SaaS, security mostly depends on the CSP. Make sure that in addition to everything listed for IaaS & PaaS, that you also have web application & web service penetration testing done. Request web application vulnerability reports, and make sure the reports have a web application security policy that requires the CSP to test for vulnerabilities on a regular basis. At a minimum, request that they test for the OWASP Top 10 web application vulnerabilities. If they don’t have penetration testers, they should at least be performing web application vulnerability scans using one of the several great tools that they can setup in-house. Coincidentally, this type of scanning can also be done from the cloud by certain vendors as well.
- How did they test those things? Were the tests performed black box, white box, or grey box? Vulnerability scans & fingerprinting are a precursor to the actual penetration testing. That being said, you want to ensure that from the very beginning you have an accurate vulnerability report by reducing the number of false positives that are discovered. The best way to do this is to use credentials (user + password). If your vulnerability scanner says one of your servers is running an unpatched, old version of Apache…what is the best way to verify this? You login and see what version of Apache it is actually running. This is what white box testing does for you. It reduces the amount of “unknowns” in your testing. You don’t want any “unknowns,” because that is what will kill you. Be warned you still MUST sort through any false-positives, but the amount you will have to deal with should be far less.
- What tools were used? If they used a vulnerability scanner, what scan policy (what vulnerabilities) did they scan for? Top 10? Top 20? Make sure that anything you expect to be covered is tested in their penetration tests & vulnerability scanning. If you are using a SaaS product, you might want to know whether or not their website enforces SSL encryption, or is checked for code injection vulnerabilities such as Cross-site scripting (XSS) and SQL Injections.
Third-Party Pen Test Team
If you are going to hire outsiders, you will be playing the middleman and working out a lot of contract/SLA issues between all three parties that are involved. You will have due diligence to perform when selecting your third-party pen test team. Many of the considerations suggested next should be used when hiring third-party pen testers. You should ask the same questions and have similar requirements as listed above from the third-party pen test team just as one would with the CSP.
Your Own Pen Test Team
So you have a team and you want to pen test your CSP. Great! Let’s get started. First, you should check your SLA and contractual agreement between you and the CSP and then consider the type of cloud you will be testing, because each of them has different consideration you must account for:
- IaaS (Infrastructure as a Service)
- PaaS (Platform as a Service)
- SaaS (Software as a Service)
Next you must know the scope of what you want tested to meet your security requirements:
- Web application pen test (this includes mobile apps)
- Web service pen test (this includes mobile back-end services)
- Network/host pen test (this includes databases, firewalls, and other systems in your cloud network)
IaaS/PaaS Web Apps & Web Services
The good news is that IaaS cloud grants the customer the most granular control over the entire environment. That means most of security built-in will have to be setup by your organization and you should have an easier time planning how to properly conduct your test with the given architecture. Since it is up to you to supply the web application and web services, you will have an easier time testing for vulnerabilities and performing penetration tests. Because security is left to the cloud user’s responsibility, you usually will not have to interact too much with the CSP in order to perform your own penetration tests on your own system as part of your cloud security program. You should state upfront in your CSP contract that you will be performing your own penetration tests and what that scope is going to be and how often. This, ideally, should have been planned early on in the contract, not after the fact. If you are lucky, the CSP has already put this language into their service level agreement (SLA). You should know what application and technology you are running in your own cloud since it was provided by your organization, and you may not have to worry about multi-tenancy issues (but you will for network & host level pen tests). You should be able to easily reach your web application/services remotely for penetration testing. However, if you have deployed web application firewalls (WAF), or reverse proxy servers into your cloud it can interfere with your penetration testing. For the most accurate results, it is recommended to pen test against these web apps/services without your WAF or reverse proxy in front of you. The reason is simple…you want to know what vulnerabilities actually exist so that you can fix them. WAFs are not replacements for secure coding practices. You wouldn’t run an un-patched OS on the web with only a firewall, and expect it to be “secured” would you? Testing with these types of mechanisms in place gives you a false sense of security by hiding what really exists. However, you may also want to test with your IPS, WAF, and other security devices in place; this can also give you another valuable point-of-view. You just need to decide if the goal is to test how well your devices are configured and operating, or if the goal is to test the CSP’s security team and how well attacks are detected on their network? An exercise can be coordinated with the CSP’s security team, by which a live penetration test is conducted to test the responsiveness and effectiveness of their security team. You should opt to perform authenticated pen tests against web apps and services, and authenticated vulnerability scans against hosts. You will be able to get more accurate test coverage of your cloud.
Iaas/PaaS Network & Hosts
If you want to pen test your cloud network meticulously, you should perform internal and external pen tests. This will provide you with two dissimilar points of view, and likely a different set of vulnerability results.
To conduct an internal pen test, you should already have access to all your servers and hosts in the cloud (including the databases and storage components). You should also have access control credentials and network architecture. Start with an authenticated vulnerability scan against your systems. The idea behind an internal assessment is that you want to know what kind of damage a hacker (or disgruntled employee) can do, if he/she had access inside the perimeter. This will also help discover potential pivot* points used by hackers.
It may be possible in your IaaS/PaaS environment that multi-tenancy exists! If this condition exists, you may not be allowed to perform authenticated tests or any test for that matter against certain systems (due to the impact it can have against other CSP customers). You will have to work with your CSP to determine what is allowed, and what is not, which should be documented in your contractual agreement and SLA. Your goal should still be as much testing coverage as possible within legal bounds. For an external and more realistic viewpoint of an average hacker, you will want to remotely test from outside the DMZ. This should be performed as any typical black box remote penetration test. External white box testing here is not really necessary if you are already performing internal white box pen tests.
SaaS Pen Testing in General
It is up to the CSP to provide the security to the customer in this cloud environment. That includes host-level security all the way up to web application security. Performing internal and external pen tests of any type will usually require very difficult contract negotiations, because of multi-tenancy. It is very common that SaaS products share backend components (e.g. databases and storage), and as such certain vulnerability tests (e.g. DoS and SQL injection) may not be permitted. Since you will have very little access beyond the public facing web application(s) & web service(s), you don’t have many options for network pen tests. To test in a SaaS environment, it will require very special permission and you will have to take what you can get. Remember that your objective is still to get as much test coverage as possible. So try to push for an internal pen test on their network through a documented mutual agreement. However, you will most likely only receive permission to pen test the web app and web service itself, and maybe the web servers that those are hosted on. Make sure that you perform authenticated penetration tests for the best coverage.
What you should take away from this is that penetration testing in the cloud takes a large amount of coordination and contract negotiations before the fun stuff can begin. Before you negotiate the terms of penetration testing in your contract, you have homework to do. Know what kind of penetration testing you want to perform. For areas that you are not able to test, make sure you take note of where security gaps may exist, and check to make sure you have compensating controls in place to mitigate the risk. For example, if you are unable to perform penetration tests against a database, ask the user CSP to provide design details, sanitized device configuration files, the hardening guide lines used, results from the CSP’s own penetration tests, and any other relevant audit reports. You have less control over the cloud environment than with your own network, so try to maintain as much of that control as possible through contractual agreements and SLAs.
Pivot* A “pivot” is when a hacker breaks into a computer and uses that computer to hack into another. This has a few advantages:
1) It disguises the hacker’s true origins
2) It allows a hacker to attack other targets from the point-of-view of the victim (i.e. inside the firewall).
If the attacker couldn’t port scan you before, he can now! The best type of pivot is a VPN pivot. The hacker sets up a secure VPN connection between the compromised host inside your network and the outside world, effectively giving him a backdoor into your corporate network.