Is Penetration Testing a Complete Waste of Time?

Let’s face it, hacking is fun. For a lot of cybersecurity professionals as well as black hat hackers, it’s a kind of game, one that pits intellect and skill against unknown adversaries.

So for many people who get into cybersecurity, getting a license to hack as a penetration tester is a dream job. You get all the excitement of cracking systems using every dirty trick you can think of, with none of the potential consequences. You even get paid to hack.

But does penetration testing—also known as ethical hacking—actually accomplish anything? Does the success or failure of a necessarily limited series of attacks in a specific time period against a tiny sample of possible security posture configurations actually tell security administrators anything useful about their defenses?

Opinions are mixed, and the secretive nature of penetration testers doesn’t make it easy to objectively evaluate. But penetration testing has some systematic weaknesses as a cybersecurity defense mechanism that might make you think twice about specializing in or using it.

Enumerating Badness Is Bad, and Penetration Testing Doesn’t Even Do It Comprehensively

In “The Six Dumbest Ideas In Computer Security,” security consultant Marcus Ranum’s diatribe against poorly considered cybersecurity techniques, he outlines the concept of enumerating badness… the idea that all possible weaknesses can be listed and defended against specifically.

Penetration testing, of course, is working off just that catalog of ideas, testing only vulnerabilities that are, by definition, enumerated somewhere.

Ranum lists penetration testing on its own as another of those dumb ideas but in fact it’s a compound problem. It’s an approach to security that attempts to find and document a particular vulnerability and doesn’t even pretend to discover all possible holes.

The downfall of that general approach is easy enough to spot. But a more subtle problem is that penetration testing only even delivers this admittedly limited information for a particular snapshot in time. A phishing attempt or other social networking attack that succeeds once in a penetration testing engagement might not work a second time, or against another set of employees. The dynamic nature of network internals in most corporate networks also means either positive or negative results in a test are only valid for a short period of time when it comes to assessing security posture.

A lot also depends on the individuals involved in the test. Neither all hackers or all attacks are created equal. A penetration tester may be better or worse than the malicious hacker that eventually comes at your system. The relative difference in skill can mean you are either under-prepared or over-prepared… not to mention that two equally capable hackers may have completely different thought processes, so as skilled as your guy might be, he could still miss something.

To provide some guidance in hopes of avoiding these pitfalls, several different groups of information security professionals have attempted to develop standardized frameworks to guide penetration testers and to avoid gaps in testing methodologies:

• Penetration Testing Execution Standards (PTES)
• Penetration Testing Framework
• Open Source Security Testing Methodology Manual

All these guidelines are both dauntingly complex and woefully out-of-date, and once again stand as examples of little more than enumerations of badness.

Penetration Tests Can Do Real Damage

There’s also risk simply in bringing in outside parties and inviting them to root through your systems. Although the legal boilerplate involved is daunting, no piece of paper can make testers forget what they have seen or how they gained access to your systems.

Today’s gray hat could be tomorrow’s black hat. Penetration testing schemes have resulted in Trojans being seeded in client’s networks and data being stolen. In other cases, even well-intentioned pen testers have left behind malware used in their testing that could be used by less benevolent attackers.

To prevent or limit such damage, many clients outline rules of engagement as a part of the test. Of course, this dramatically limits the utility, since real-life hackers won’t stick to such artificial restrictions.

But the need to avoid such damages points out another way in which penetration tests cannot prepare you for the full range of possibility in cyberattacks. Many such attacks are less about penetration than about vandalizing or breaking things. But very few companies are willing to DDOS their own production systems just to see if things break.  That leaves out a whole category of mischief that even the best testers can’t tell you about.

The Benefits of Penetration Testing May Be Mostly Psychological

Of course, security flaws that are found and fixed through penetration testing are a net benefit, although it’s not clear if it’s a particularly economical way to identify them.

At least one security expert, Bruce Schneier, believes that economics are a serious argument against penetration testing, although for the opposite reason. Schneier thinks that most penetration tests will reveal far more vulnerabilities than most companies can afford to fix.

In a cybersecurity version of the old programmer’s trope “Never test for an error condition you don’t know how to handle,” he asserts that it’s pointless to uncover problems you can’t fix—or, worse, opens the door to significant legal liability in the event any of them are eventually exploited.

Schneier makes an exception for vulnerability scanning, which is arguably a type of penetration test, but not one that most professionals would place in that rubric. Regularly and automatically scanning for the top vulnerabilities in the OWASP Top Ten or Nessus database is a quick check against frequently exploited vulnerabilities and has a low cost associated with it.

Penetration testing may not be entirely without value, however. The sparring aspect of the testing can help sharpen your internal security team as an exercise, even if the specific outcome of the match doesn’t tell you much.

Simply the act of having to work against organized attackers can help concentrate internal security staff. Moreover, getting a debrief from those attackers on the aspects of your defenses that were and were not working can help provide a perspective that is sometimes hard to find, that of the malicious attacker. There is a danger of becoming overly familiar with your own systems and losing the ability to view them from outside the firewall. Penetration testers provide that set of fresh eyes.

It’s also a way in which cybersecurity managers can constructively practice or train their staff in response protocols. A penetration test, assuming it is detected, can be an effective simulation for assessing internal policies and procedures for responding to attacks. Since you can know when a penetration test is coming (while genuine black hat hackers are rarely so courteous as to provide a schedule), it allows you to monitor and assess how staff perform under attack conditions.

The psychological benefits also come with psychological dangers, however. Winning the match against a penetration test can offer a false sense of security.

Like most tools, then, penetration testing is one that can be useful in the right hands and with a clear idea of its capabilities. As a blanket or critical aspect of security strategy, however, it’s probably overrated.