Penetration testing versus vulnerability scanning.

The difference in one table

Both look for weaknesses in the same systems, which is why they get confused. They answer different questions. A scan asks which known weaknesses might be present. A penetration test asks which weaknesses an attacker can actually use, and how far that gets them. The card industry’s own guidance draws the line in the same place.1

PCI DSS, the card industry’s security standard, makes both mandatory as separate controls: scans at least every three months, penetration tests at least every twelve months and after any significant change.2 The split is a useful pattern even if you never touch a card number. Two instruments, two rhythms.

What a vulnerability scan does well

A scanner takes stock of what it can reach, works out which operating systems, applications and versions are running, and matches them against a database of known vulnerabilities. That makes it good at questions with a lookup answer: an outdated library, a missing patch, a weak TLS setting, a default page left on a server.3

It is also cheap to repeat, which is the point. A scan that runs every week, or on every deploy, catches the drift between tests: the package that fell behind, the debug endpoint someone switched on for an afternoon. Scans run with credentials see more than scans from the outside, because they can read installed versions instead of guessing them from responses.3

Run one before a pentest, too. The UK’s National Cyber Security Centre advises sharing your current vulnerability assessment with the testers,4 so they do not spend paid days rediscovering a missing patch your scanner already reported.

Where scanners stop

A scanner judges each weakness on its own. NIST’s testing guide calls these surface vulnerabilities, weaknesses that exist in isolation, and is blunt about the limit: scanners cannot find vulnerabilities that only appear when attack steps are combined, and may rate every part of a chain as low.5

Its ratings are generic, too. A scanner does not know that the server it flags only serves public files, or that its ‘medium’ sits on your admin console. Scanners also report weaknesses that are not there; NIST warns that they can have a high false positive rate, and tells assessors to set the real risk level themselves.6

Two classes of flaw a scanner cannot see

The first is business logic: a discount that applies twice, a refund approved by the person who asked for it, a step in a sign-up flow that can be skipped. The OWASP Web Security Testing Guide is explicit that a vulnerability scanner cannot detect this type of flaw, because automated tools find it hard to understand context.7

The second is authorization between users. Take an invoicing API in which user A asks for invoice 1042 and receives it. If user B asks for the same invoice and also receives it, nothing in that response looks wrong to a machine: the request is well formed, the status is 200, the body is valid. Only someone who knows the invoice belongs to user A sees the breach. OWASP ranks this, broken object level authorization, first in its API Security Top 10 and calls it extremely common in API-based applications.8

What a penetration test adds

NIST defines penetration testing as security testing in which assessors mimic real-world attacks to find ways around a system’s security features, and notes that most tests look for combinations of vulnerabilities that give more access than any single one.9 That is the work: two findings a scanner would each rate low can add up to one that matters, as figure 1 shows.

This is why the card industry’s guidance calls penetration testing essentially a manual endeavor, in which judgment finds the attack paths that automated means typically cannot.10 Tools still carry much of the volume (crawling, replaying requests, sorting responses), but a person decides what deserves an afternoon and recognises when a response means something.

Three things come out of a pentest that a scan cannot produce:

• Demonstration. A pentest finding is shown, not inferred: it comes with the requests and responses, or the steps, that make it happen. Anything that cannot be demonstrated does not reach the report as a finding.
• Severity in your context. CVSS v4.0 separates a vulnerability’s base score from threat and environmental metrics, and FIRST, which maintains it, stresses that CVSS is not just the base score.11 A tester can rate a finding for your environment; a scanner reports the generic base.
• A path, not a list. The report shows how far one weakness reaches, which tells you which fix removes the most risk first.

One caution from the UK’s National Cyber Security Centre is worth keeping: a penetration test is a way to gain assurance, not the primary method for finding vulnerabilities.12 If your pentest report is full of missing patches, the real finding is about your patch process, and you paid tester rates for a scanner’s work.

Which one do you need?

Almost always both, at different rhythms. Scanning is hygiene: it belongs in your pipeline and your weekly routine. A pentest is a checkpoint, and it earns its cost at these moments:

• Before a launch that exposes new data or new roles: a customer portal, a public API, an AI feature that can act on a user’s behalf.
• After a significant change to authentication, authorization, tenancy or infrastructure. PCI DSS uses the same trigger.2
• When a customer asks. Enterprise security questionnaires ask for a recent third-party report. Under the Dutch Cyberbeveiligingswet, organisations in its sectors must manage the security of their supply chain,13 which is how those questionnaires reach their software suppliers.
• For an audit. SOC 2 does not strictly require a pentest, but penetration testing appears in a point of focus under criterion CC4.1;14 ISO/IEC 27001:2022 has a control for security testing in development and acceptance, 8.29.15

A scan alone is enough when the question has a lookup answer: are we patched, is this server configured as intended, did anything new appear on our external surface this week. For those, a pentest is the wrong tool at the wrong price.

How to tell them apart in a quote

Price is the quickest signal. Published prices for automated tests run from about EUR 1,25016 to EUR 3,500 per test,17 while a human-led web application or API pentest in the Netherlands typically costs EUR 5,000 to 15,000 excluding VAT.18 A quote far below the human-led range buys mostly automation. That can be the right purchase, as long as the report says so.

Then ask four questions. The answers separate the two faster than any brochure:

1. How many tester days, and whose? A pentest is time-boxed work by named people. A scan has no days to count.
2. What does a finding look like? Ask for a sample report. Look for reproduction steps and evidence, not a list of plugin numbers.
3. Is business logic in scope? If the vendor cannot say how they will test your roles, tenants and flows, the test will not cover them.
4. What happens after the fixes? A retest that replays each finding tells you the fix closed the cause, not just the symptom. Ask whether it is included, and when it happens.

To see what our answers look like, read the sample report: a fictional client, in our real format.