10 Questions to Check Your Pen Testers Fees Match Their Expertise

With increased awareness around cybersecurity, the demand for pen test services has grown rapidly. With it has come an explosion of new pen test service providers, each with its own methods, service packages, credentials, and prices.

Variations between vendors can make comparison difficult. The due diligence required before contracting a test can seem more arduous than the testing process itself.

Being able to cut through the marketing-speak of security consultants and understand the true value of the services they offer can help organizations improve security while staying on budget.

This guide contains 10 questions you can ask any pen test provider to help determine whether the testing being offered really matches the prices quoted. We’ve also included notes on what you want to hear in a tester’s response to these questions.

1) How Does Your Penetration Test Differ From Other Types of Security Testing? 

This is a good introductory question to get a prospective tester speaking about their solutions. 

The truth is, you may know the answer to this question already from reading the material on the firm’s site or coming across their adverts. At the very least you’ll know some of what they’ll say. Still, it should still be asked to ensure that the prospective vendor can articulate the differences that make their penetration testing unique.

Listen Carefully

One thing, in particular, to look out for is a vendor that uses the terms “penetration test” and “vulnerability scans” interchangeably. These two are not the same thing. What you are looking for is a pen test, a detailed hands-on examination by a real person that tries to detect and exploit weaknesses in your unique system. 

A “scan” or “assessment” often means the application of a pre-built, one-size-fits-all framework that is a) generic and b) won’t uncover as much as an aggressive and thorough test.

(IMPORTANT: This free PDF report shows pricing data from 10 real penetration testing contracts so your firm can gauge costs and avoid overpaying for your own tests.)

2) Do Your Testers Hold Industry Standard Certifications? 

It’s important to know that the individuals conducting your test (no, not the firm as a whole, the actual people doing the testing) are certified and experts in what they purport to do. There are a variety of certifications that demonstrate knowledge in information security and technology in general, but penetration testers often hold certifications such as CEH, CISSP, GPEN and GWAPT. 

Another layer to this question is if the members of the team are involved in research and development activities in the cyber domain. To some, this may sound overly inquisitive. But in the world of IT, experts will openly advertise their R&D activities as a display of credentials. 

What’s the Answer You Want to Hear?

An effective penetration testing firm will have the certifications of their team members available for viewing. Furthermore, they’ll have multiple recent research projects to review. Pen testing companies in particular very often display the tools and methods they themselves have developed in order to carry out their service. 

What to be Wary of

If the company cannot provide any official certification of their testing team, that is a red line.

Additionally, if there’s no research work to view that should raise some skepticism on your part. Private research is common in IT like other industries. But be concerned if they can’t showcase anything they’ve done in this space. 

3) How do You Maintain Internal Security in Your Company?

This may be an uncomfortable question to pose, but it’s still very important.

When you take on a pen test firm, you are exposing all of your system’s vulnerabilities and weaknesses to them. That is, of course, the whole point.

These vulnerabilities need to be recorded and logged for the test to be effective. All this information remains stored with the service provider even after the penetration test has been completed. This is especially true if the company offers remediation services. You should ask how the prospective service provider will ensure the security of confidential data and what are the steps taken to maintain an adequate level of security.  

4) How Will You Protect my Data During and After Testing? 

This is in some ways a follow-up to the last question. But it is really its own distinct concern as well.

In a pen test, the operation itself can be quite compromising not just to your system’s tools and functions but also to the data stored on that system. 

That is why you should confirm the tester’s methods of securing your data during the test and throughout delivery. 

If devices will be shipped to your location or testers will be visiting your location with their own devices (which is commonly the case) ensure that disk-based encryption is being used to protect data obtained or accessed during the test. 

5) What is Your Process for Performing the Penetration Test? 

Penetration testing methods and techniques do differ slightly from organization to organization and every firm will have its own take. But there are some core activities common across all penetration tests. 

Even if they do not use a pre-defined methodology, the vendor should still be able to outline the steps and stages involved, if for no other reason than letting their clients know what to expect. 

Why is it Important to Ask This?

The first reason this is important to ask is straight-up expertise confirmation. As we’ve already mentioned, a penetration test is not a singular action or set of actions. It’s a multi-stage, complex process (often highly complex) that is meant to literally mimic the strategies and procedures of sophisticated cybercriminals. Whoever is going to do your pen test will need to be able to describe that process to you.   

What’s the Answer You’re Looking For?

You don’t need to be an expert in penetration test stages yourself to have a conversation with a potential vendor. But it would help to have some basic knowledge of how these things generally work. 

1. The first stage of any test is the planning, during which an experienced developer will define the goals of the testing exercise. This includes which elements of the system will be ‘targeted’, taking into account the network processes a hacker would attempt to exploit. Based on these definitions, testers will define rules for the pen test operation and determine the methods and tools to be used. During this slot, there’s supposed to be a lot of questions posed to you and your IT team about your system. So something like “at first we’re going to put forward a lot of questions to you and your IT team about your system” is what you’d like to hear in a tester’s description. 
2. Typically, the planning will be followed by the reconnaissance stage. Here, the analysts gather intelligence on the network and its vulnerabilities using various methods. The goals are to get as much data as possible for identifying potential vulnerabilities to exploit and create attack plans for execution. 
3. Testers will then move on to the actual penetration attempts using both automated and manual tools. 
4. Finally, if any of these attacks are successful, the testers will need to determine what’s called the persistence factor. This means assessing how long it takes for security protocols, either automatic or human initiated, to kick in and respond. 

What Can Go Wrong?

If a tester leaves out any of these elements in describing their test procedures, this is not a good sign. 

Either they don’t know what they’re talking about, or they’re planning to leave out essential components of a high-quality test. 

Now, granted a tester may skim over some details when portraying their test, especially when it comes to the more technical stuff. Indeed any competent professional will not bore or overwhelm potential customers with too much information. In that case, all you have to do is ask. 

If the tester doesn’t mention anything about reconnaissance and interviewing your IT department, bring that up. If the developer doesn’t say anything about checking persistence in vulnerabilities found, put that forward.

If after you ask, a tester still doesn’t have anything to say, you’re almost certainly better off looking elsewhere.     

6) How Will You Ensure the Availability of our Systems and Services During the Test?

Because penetration tests are actual attacks against your systems, it is impossible to guarantee the process will not have an effect on your regular business operations. Availability or functionality throughout the test of any services, apps, or other part of IT infrastructure being tested, can never be one hundred percent guaranteed. However, most of the time, competent testers should be able to carry out simulations with little to no noticeable effect. 

What’s The Answer You’re Looking For?

A competent tester will be able to give you a general picture of what could be disrupted during the test. This can range from slow internet service to whole applications being unavailable for a period of time. 

Additionally, most testers have some idea of whether or not a particular attack runs the risk of bringing down your system or causing a service to slow down. The tester should also be able to tell you ways of mitigating those risks–conducting certain components of the test during off-hours for instance. 

In short, the ideal penetration testing vendor will work closely with you to address operational concerns, monitor progress throughout the test, and work with you to mitigate disruption.

[elementor-template id=”27183″]

7) Who on Your Staff Will be Personally Involved in the Project? 

In professional services industries, it’s an unfortunately common tactic to sell clients on the firm’s most senior, qualified experts, then quietly use junior personnel for the actual services. 

While this technique has historically been a move of larger organizations, today, it’s common for even smaller firms to resort to the tactic in what’s come to be known as “resource dilution.” At the end of the day, it’s really just a method for misleading clients to put more trust in whatever service they’re providing them. 

As far as pentests go, this practice not only results in a poor quality test (ie, missed vulnerabilities) but also leads to a higher chance of testing accidents and disruption to your business continuity during the test itself.

What’s The Answer You’re Looking For?

When meeting with a prospective firm, ask for the names and qualifications of the pen testers that would be on the engagement. 

Confirm any promoted “rock stars” are actually the engineers in your engagement, and what their roles would be.

What to Look Out For

Be alert to vague language around who “may” be involved in your test. What you want to hear is “developers so-and-so will be handling your test.” 

A more likely scenario is that a senior expert will be given ‘charge’ of your project, but will ultimately be playing a managerial role only. This is not necessarily a bad thing. Much of a pen test involves menial, simple work (relatively speaking of course) which can be executed by junior staff or even entry-level programmers. 

The question is how hands-on will the senior manager be? If he or she is overseeing multiple projects simultaneously, this may be a not-so-good indicator. 

In that case, you’ll want to clarify how available the managing experts will be during the course of the work. If they’re able to be on-site with some regularity during the course of the test (in a case where the test isn’t being done remotely) or if they guarantee you constant reachability via phone or other means, this can mitigate the problem of the experts not being the actual executors of the test. 

8) How Much of the Test Uses Automated Tools? 

This question is actually a great opportunity to have the pen tester demonstrate their knowledge and proficiency. But more fundamentally, it’s also another way to ensure what you’re getting is a bonafide pen test as opposed to the ‘scans’ and ‘assessments’ often touted as pen tests.

The distinction between automated testing tools and hands-on, human-initiated procedures is very important to executing an efficient penetration simulation. 

To be sure, any good test begins with automated tools. These programs will run a variety of common and known exploits to see how your system responds. The tester may configure the payload based on the circumstances and specification of your network, but running the actual ‘test’ part of those exploits is in fact automated. 

Automated doesn’t necessarily mean lower quality. In fact, because many threats and vulnerabilities are generic and common, it is easiest to execute them without too much thinking. This saves a lot of valuable time and manpower that you’d otherwise be charged for if a programmer was doing it all by hand. 

With that said, the majority of an effective pen test should consist of manual procedures executed by (or at least under the direct guidance of) a senior programmer. Some pen testers out there boast as much as eighty or ninety percent of their tests done through manual tools. While this may sound impressive, it isn’t really necessary. Certainly not as far as getting an idea if your pen tester is the real-deal or not. 

What’s the Answer You Want to Hear?

When describing their process, what you would like to hear from a prospective testing firm is a little bit of talk about their automated procedures and then the rest of the conversation describing the manual side. 

If they’re not providing detail, inquire about what types of external attacks they’ll simulate. 

Here, it’s worth being a bit familiar with some of the terminology. 

Common cyber attacks that take place in a pen test include web application attacks, such as cross-site scripting, commonly known as XSS attacks. This type of attack involves injecting malicious code into an application that will then target the users of that app. This type of test is particularly important, especially if you’re a firm that collects and stores the private data of clients or other third parties. Knowing the programs used to access and manipulate that data are secure, should be a central concern of any penetration test you contract. 

Another common form of testing is SQL injection designed to give the attacker backend access to manipulate application features. If an SQL attack can be conducted during a pen test, it means attackers have the ability to exfiltrate or even permanently delete certain features of your system. It is essential to know this vulnerability exists if it does since it is often the leverage used in for-profit hacks such as ransom attacks.  

If the vendor you’re interviewing indicates most of the test is automated or doesn’t ask many questions about your particular system or environment, be wary. This is a telltale sign the pen test is simply not going to have the quality you’re paying for. These vulnerability assessments (which are of course priced and advertised as full penetration tests) will only give you and your organization a false sense of security while costing you a lot of money. 

9) Has Your Firm Ever Provided Integration Solutions to Our Organization?

The answer to this question should not be particularly complicated. It’s either a yes or a no.

But it is worth clarifying for a very specific reason: conflict of interest.

Imagine this scenario: A pen tester claims to have ‘successfully breached a system’ by detecting flaws. They deliver an extensive report detailing all the issues at hand and charge you for a full pen test. Later the client finds out that the pen test company also provides security solutions. And lo and behold the client is one of those security solutions customers. 

What has essentially happened is the pen tester was paid to ‘uncover’ all the vulnerabilities they already were familiar with since they were the ones to provide much of the defensive infrastructure.   

Or take the inverse case.

Pen tester is hired. Pen tester conducts test. Remarkably, few to no vulnerabilities are found. But the client’s relief is short-lived. As it turns out, the pen tester is the same company that designed (or sells) many of the client’s IT components. The ‘clean bill of health’ delivered by the tester was essentially the company giving itself a high-quality rating.  

Unfortunately, cases such as these are far from rare in the IT industry. Indeed, with the interconnectivity of the cyber / infosec world, the issue of conflict of interest has become a real concern. And this is especially true when it comes to pen tests which are essentially meant to measure the security and efficacy of IT tools. 

If you want to know you’re getting an honest assessment, it is worth confirming this conflict isn’t present with a potential testing firm. 

[elementor-template id=”27183″]

10) How Will You Be Reporting Your Findings? 

Reporting as far as pen tests go, is divided into two phases: during the procedure and once the test is completed. 

As far as the duration of the test, the tester should be in pretty regular contact with you if for no other reason other than to give you a progress report. It is very important to clarify how those reports will be made and to whom they’ll be delivered. The frequency of communication should also be agreed upon. 

This is something you can negotiate with your penetration tester but a good baseline is to update once when the test kicks off and then at the conclusion of every workweek. But that is a minimum. Ideally, whenever a new phase of testing begins (beginning malicious payload testing for instance), the tester should at least send a message to alert you.  

Additionally, the tester should be given multiple emergency points of contact and should also give you multiple ways of contacting them in case of mishaps.  As we discussed above, a real penetration test does involve a certain level of risk which may result in an inadvertently downed system or two. If those systems are ones your organization relies on to power its business, then you’ll want to be able to contact the tester immediately. 

The method of communication is also crucial. Since you’ll be discussing sensitive information on the nature of your system, you should expect some form of encrypted communication such as specialized email (PGP).

Lastly is the post-test report document. Unfortunately, this is where a lot of firms don’t do a good job. 

After all is said and done, you’ll want to have a clear and understandable recap of what your pen test uncovered. This is important for a few reasons. The first is for post-test remediation.  

When going about fixing the problems detected in a pen test, you’ll need to describe to your IT people or whoever else you put on the job, what the problems are that need fixing. In order to do that, you’re going to want a high-quality pen test report.

The next reason you want a solid and detailed report is for compliance purposes. 

While pen tests are primarily used for identifying what’s wrong, they also identify what’s right. This is extremely valuable for regulation compliance and demonstrating responsible practices on your end to secure your system and any data on it.