Read Before You Contract 3rd Party PCI Penetration Testing

PCI Penetration Tests

Any organization processing, storing, or transmitting cardholder data (CHD) must attain certification or submit a self-attestation of compliance, according to PCI-DSS standards. PCI-DSS v3-2-1 has been published and in effect since 2018, with the most recent (4-0) being released in March of 2022, both of which are available in PCI-DSS Document Library. As part of this standard, PCI-DSS v3-2-1 requires that an organization perform internal and external penetration testing by an independent and qualified penetration tester.

This article will discuss some of the key requirements that need to be considered when engaging or implementing a penetration testing regime for PCI compliance, and how these requirements may differ from non-PCI penetration testing.

(IMPORTANT: This free PDF report shows pricing data from 10 real penetration testing contracts so your firm can gauge costs and avoid overpaying for your own tests.)

What Kind of Pen Testing is Needed for PCI DSS Compliance?

First, to understand what is being asked for, penetration testing needs to be defined. Penetration testing, according to PCI-DSS, is considered a separate activity from vulnerability scanning or assessments, which are also a requirement.

Vulnerability scanning or assessment is the act of identifying, ranking, and reporting on vulnerabilities. Penetration testing adds the act of exploitation to circumvent or defeat security controls.

The definition further delineates the difference between a vulnerability assessment and a pen test in how the testing is being conducted. A vulnerability assessment is highly automated and may combine multiple tools with manual verification. Penetration testing is a manual process of exploitation, that includes or may include, automated tools to identify vulnerabilities.

With the definition of vulnerability assessment and penetration defined, further review of how PCI-DSS penetration testing differs from other penetration testing, the frequency, and further analysis of how results can impact compliance will be discussed.

PCI Pen Testing Standards & Methodology

PCI-DSS 3-2-1, requirement 11.3 stipulates that an organization must implement a methodology or capability of testing that is based on NIST SP800-115. Further, it is required that testing include the full cardholder data environment (CDE), critical systems, and supporting systems for that environment. PCI requires penetration testing of both the external network and internal network. The internal network test must also include segmentation validation testing (testing of VLANs or segmentation that is deployed at the network layer to limit scope).

In short, any environment that may be interacting or storing CHD must have internal and external testing conducted against it to validate that the CHD is not exposed and that the environment has appropriate controls in place and operational.

This testing must also incorporate network-layer testing and application layer testing as part of the scope. PCI-DSS further defines that app pen testing must include tests against the OWASP Top 10 for any application or service that is residing within the CDE.

Now that PCI pen testing is defined and scope is considered, how does it differ from a normal penetration test?

Depending on the consultancy, the methodology may not differ much at all. Most consultancies will perform network penetration testing with any applications at least partly in scope.

However, the biggest differences will be in the network segmentation testing, and the depth of documentation. This is not a typical approach for an internal penetration test and will need to be asked for specifically.

It is always recommended that the contracted consultancy is informed that the test will be used for PCI-DSS.

Qualifications to Perform Penetration Testing

When it comes to who can perform penetration testing for an organization, it comes down to impartiality and qualifications. PCI-DSS does not require a third party to conduct the testing and has guidance on when a third party may not conduct the testing for an organization.

For both internal and external testing, a tester must not have material daily duties in the management or building of the systems related to the CDE. The tester needs to be free from conflict of interest and have the ability to perform an impartial test against the environment.

As for qualification, there are two ways to show that a tester is qualified. The first is through certifications like the Offensive Security Certified Professional (OSCP) or Certified Ethical Hacker (CEH). These two are only examples, and other certifications can be attained to show technical capabilities to perform testing.

The other way a tester can show technical skill is through experience. This will require more evidence gathering by the auditor, and an organization may need to be prepared to provide this. This will include evidence or information on previous engagements, details on previous work experience, references, and types of testing performed with information on the scope. In most cases, a consultancy experienced with PCI will assist with providing this information, if requested, as part of the engagement.

PCI Penetration Testing Frequency

Regardless of the compliance level, from 4 to level 1, the frequency for penetration testing is defined by requirements 11.3.1 and 11.3.2 of the PCI-DSS document. External penetration testing must be conducted at least annually and after any significant changes in infrastructure or applications. Refer to the PCI-DSS Penetration Testing Guidance for detailed information on what are considered significant changes requiring a penetration test. An example of an event that would require an external penetration test would be a new application that has been deployed that resides in and supports the CDE environment.

For internal penetration testing, requirement 11.3.2 states it must take place annually with segmentation testing occurring every 6 months. Just like with external penetration testing, any significant changes would require additional testing to validate that the environment is secure, and the controls are operational. For this requirement, most organizations choose to do an internal penetration test every 6 months and do both segmentation testing and internal penetration testing at the same time.

If an organization were to have any findings that were above a medium risk, remediation would need to take place. This would then require additional testing to validate that the findings are remediated properly.

The PCI Penetration Test Process

When it comes to PCI penetration testing, the certification council and auditor will be looking for key steps during the process and key information to be provided, as an outcome of the test.

Scoping: Prior to an engagement, it is recommended that detailed information on the CDE be gathered to assist with proper scoping. This would include information on the number of hosts, the number of segments in scope, and a network diagram. This information will need to be provided to the consultancy during scoping not only to price, but to ensure the organization does not have any compliance issues when submitting the report. When gathering information for the scoping, review all in-scope systems and networks to identify any missing components, as the test needs to be against all access points, critical systems, and segmentations for the CDE.

Define Rules of Engagement: Once documentation is gathered and the scope of the effort is determined by the consultancy, rules of engagement can be defined. This will be documented as part of the test and as part of the evidence to be submitted with the test to the organization’s auditor. This would include information about testing windows, any systems that should be excluded from automated scanning, controls that would prevent or hinder testing, communication channels, and any other concerns or issues that could arise that the tester should be aware of.

Define Success Criteria: The next step would be defining the success criteria for the penetration test. An organization may immediately think that success would be no major finding, but that is not the only success criteria. Keep in mind that the end goal that PCI has for this engagement is to simulate malicious actors attempting to compromise or pivot within the organization’s network.

Properly defined success criteria will assist with determining techniques and depth of testing to be conducted, as part of the engagement. In some cases, an organization may want to see a system accessed through an exploit to assist with validating the impact and attaining appropriate logs to be able to detect in the future. Careful thought and conversation should be had in this step.

Review Past Issues & Test Results: The next step that is recommended is the review of past issues or test results. This is not overly common within other penetration testing engagements. Many times, these engagements are more point in time. PCI recommends this to allow for validation of remediation but, also, to provide additional information on previous issues.

This provides some information on what vulnerabilities or issues could be in the environment. Additional details that should be reviewed during this step are controls deployed, which could include how these controls are deployed. Again, keep in mind that PCI-DSS wants to validate that the controls are operating correctly as part of the penetration test, so detailed information will assist with this validation.

As evidence, there are a few more steps that are recommended to be followed for PCI penetration testing that are not typically followed for other penetration testing engagements. Everything discussed to this point has been pre-engagement activity. It may seem that there will be a great deal of time upfront, but the reality is that the above steps should be a small portion of the engagement.

Test >> Post-Test >> Reporting >> Remediation: Just like any other penetration test, the bulk of the time will be spent in the testing phase. Once this is complete, post-testing activities can take place. This should involve reporting, which will include best practices and recommendations. Additionally, after reporting is complete, a review meeting should take place to discuss any findings and details on how the vulnerabilities were found or exploited. At this time, if it is included in the contract, retesting can be scheduled for any findings that will need to be remediated.

How Long Will a PCI Penetration Test Take?

When engaging in a PCI penetration test, like any other penetration service, the overall time required to complete the test is dependent on the scope (number of systems, segments, and complexity) and the consulting company.

For a low complexity environment, it could be completed in as little as a week. However, it is not uncommon for a highly complex organization to have a penetration test that takes over a month to complete, especially when it comes to internal penetration testing with segmentation validation included.

As more systems and segments are in scope and need to be tested, the overall time to complete will go up. One final factor that will impact the effort, unlike network penetration testing, is the number of applications and APIs in scope. While the testing against these systems may not be a full application penetration test, the consulting company will still need to show that the OWASP Top 10 was tested against the applications.

The testing only needs to be against the exposed application and will not require credentials, but this does increase the overall effort to complete the test.

Risks of Non-Compliance on Pen Testing

What can happen when not in compliance with PCI penetration testing requirements? There are multiple ways that an organization can be out of compliance.

The first, and most common, is that a risk rating of medium or above is documented as part of the remediation test. In this case, there are two options for an organization to stay in compliance, 1. Remediate or 2. Document a mitigating control. In both cases, retesting will need to take place to validate remediation or that the mitigation has lowered the overall risk of the finding to an acceptable level. An organization should be aware that most mitigation is temporary and will need remediation eventually.

If an organization were to have findings on their penetration testing report and not remediate or have time to remediate before certification, one of two things will happen. Either the certification will be postponed, essentially left open past the expiration of the existing certification, to provide time for remediation, or the certification will not be issued. Most of the time, it is the first case, as there are severe impacts to an organization if the certification is not reissued.

One other way that an organization can be out of compliance is by not conducting the required testing. A yearly penetration test, whether internal or external, could be conducted last minute to meet this requirement, but if proper planning and action did not take place to do segmentation testing every 6 months, an organization likely will have its certification revoked. Once certification is revoked, that organization is not able to process or handle any more credit cards until such time as they become certified.

The absolute worst case for an organization is to be levied with a fine for not meeting the PCI DSS requirements. Depending on the factors of non-compliance and how the payment brands interpret the indiscretion, fines can be from $5,000 to $100,000 per month for the violation until PCI certification is attained. This could also lead to the bank or partner terminating the relationship with the organization and forcing all payments to cease.

How Much Does PCI Penetration Testing Cost?

More factors can impact the overall cost of a PCI DSS penetration testing service. As discussed in this article, a PCI pen test involves network and application layer testing, which adds complexity to the cost model.

Even with this, it is still possible to find a lower-cost consultancy company to assist with providing this service. This is primarily due to this service being considered a commodity and offering the opportunity to upsell additional services to an organization.

Most consultancies assume that the CDE in scope for testing is a very small part of the total organization network and applications. So, in many cases the consultancy would prefer to offer this for a low cost, to be able to attain the additional services. This can be evidenced by the range in the total cost of testing; it is possible to find tests for as little as $10,000 per penetration test, but it is just as common to find testing costing into the six figures (especially when full environments are in scope).

How do PCI Pen Tests Vary By Provider?

pci penetration testing providers

There are a wide range of testing providers to choose from to meet this requirement.

Some firms specialize in PCI penetration testing and do not provide much else beyond this. Usually, these companies are offering other services around PCI and will be less expensive than a full penetration testing consultancy.

In these cases, these pentest vendors will take a more automated approach, with limited manual testing, as the focus is on doing the bare minimum to meet the requirement. This may involve more automated testing or junior testers doing the testing, with senior testers validating and writing the report to show expertise in penetration testing.

This will be quite noticeable if a full penetration testing consultancy is engaged. The two big differences, in this case, will be the total cost and time to complete the engagement.

How Often Do The Rules Change?

Many in the industry can be frustrated by the lack of changes in PCI. For the first time since 2018, PCI released an update in March of 2022 to the PCI requirements. Before that, PCI updated the requirements every 3 to 4 years.

While the requirements for certification have been updated in the past and were just updated, this requirement does not change much. This is due to it being an accepted best practice across the security industry to perform penetration testing.

It can be expected that additional testing requirements may come out to account for new technologies like cloud, containers, and SaaS solutions. But it can be expected that the core requirements for penetration testing will not alert much from one version to the next. 

Balancing Compliance With Costs

When evaluating what level of penetration testing services to engage, whether it is commodity PCI penetration testing or full red team engagement, an organization should keep in mind that it is a balancing act.

Being compliant with PCI does not mean an organization is secure, and it also does not mean it is free of fault or that due diligence has been met.

While the organization is certified, if a breach of cardholder data were to happen, and it was deemed that the organization did not do their due diligence under the standard, additional fines could be levied against the organization and the certification auditing company. This will most likely come in form of fines from the payment brand of $5,000 to $100,000 per month, until in compliance again, but also in additional fines from processors or banks of between $50 and $90 per credit card.

When evaluating the cost of a cheap PCI penetration test, remember that just checking the box does not mean due diligence has been completed and may be viewed as insufficient in a court of law, or by card brand auditors.

Disclaimer: This article is not intended as legal or technical guidance on penetration testing for PCI-DSS. Further information on recommendations, definitions, and requirements should be pulled from the standard or the PCI-DSS Penetration Testing Guidance. This guidance document will provide detailed information on testing requirements, and scope, and provide additional definition examples and details beyond this article.

(REMEMBER: This free PDF report shows pricing data from 10 real penetration testing contracts so your firm can gauge costs and avoid overpaying for your own tests.)


Published by Nathaniel Cole
CISO and Security leader with experience as a business and technical leader across multiple industries. After 15 years in security, I have a track record of building, deploying and managing modern security programs that not only transform technical s...


Download Now >>

FREE GUIDE Shows How Much Your Firm Should be Paying for Penetration Testing in 2022

  • Case studies of 10 real penetration tests & their costs.
  • Cost benchmarks for each pen test type. (network, cloud, web app, internal, external etc)
  • Advice on how to reduce pen testing costs.
  • How to get more value from your pen test without increasing cost.
By submitting this form I acknowledge that I have read and understand the Privacy Policy and agree to the Terms of Use.

    
Copyright © 2022 Network Assured