Continuous Penetration Testing vs Annual Penetration Tests
Skip to content

Continuous Penetration Testing

Continuous penetration testing is penetration testing that runs through the year, started by changes to systems and to the threat picture as well as by a schedule.

On this page

Each run targets a defined system, attempts to exploit what it finds, and keeps proof of what worked. In most organizations it runs beside the annual penetration test and does a different job.

This guide explains the gap an annual test leaves, what starts a test in a continuous program, what standards and regulators expect on penetration testing frequency, how annual and continuous testing work together, and how to measure the result.

The Gap After an Annual Test

An annual penetration test describes the systems as they stood during the test window, and everything published or changed afterwards waits for the next one. The UK National Cyber Security Centre makes the same point in its penetration testing guidance: a test can only validate that systems are free of known issues on the day of the test, and it is not uncommon for a year or more to pass between tests.

Take a mid-sized organization that tests each March. The agreed scope is a customer portal, a remote access gateway, a mail gateway and the public website.

Month What changes Covered by the March test
March The annual test runs against the agreed scope. The report arrives in April. Yes.
May A portal release adds document upload and a new API endpoint. The code did not exist in March.
June An agency launches a campaign site on a new subdomain, hosted outside the main cloud account. The host was outside the scope.
August A critical advisory is published for the remote access gateway. The patch is applied three weeks later. The flaw was unknown in March.
September A firewall change during a migration leaves a database administration console reachable from the internet. The exposure is new.
November An acquisition brings a second domain and a file transfer server. The assets belonged to another company in March.
January The team marks a March finding as fixed, and the fix goes unchecked. The finding was covered. The fix is unverified.
March The next annual test begins.

Every row is ordinary change, and each one opens a period in which a weakness may be reachable from the internet. The API endpoint released in May waits ten months for its first test.

How Attackers Test Continuously

An attacker’s testing of internet-facing systems runs all year, with no booked window and no end date. The activity is largely automated, and it is aimed at every system the internet can reach, whoever owns it.

Each part of that activity reads a record that a defender can also read.

  • New hosts: Public DNS and certificate records show a new hostname soon after it is created, and automated scanners typically reach a new address without knowing who runs it.
  • New vulnerabilities: When an advisory is published for a product, exposed instances of that product are located and probed, typically long before any one organization’s next scheduled test.
  • Leaked credentials: Passwords taken in one breach are tried against the sign-in pages of other services.

Government guidance published in 2026 reflects this pace. CISA’s Binding Operational Directive 26-04, issued on June 10, 2026, states that threat actors’ use of AI may further narrow the time defenders have between a patch release and possible exploitation. For an exploited vulnerability on a publicly exposed asset, where exploitation can be automated and the technical impact is total, its implementation guidance gives US federal agencies a three-day patching deadline.

Our guide to AI hacking covers how attackers use AI agents and how quickly the documented intrusions moved.

Set against deadlines counted in days, a 12-month interval serves a different purpose. It gives periodic assurance, and the day-to-day checking between tests falls to a continuous program.

Annual and Continuous Testing Compared

The models differ in what starts the work, and most other differences follow from that. An annual test starts because a date was booked, and a continuous test starts because something changed.

  • Scope: An annual test covers the systems named in the statement of work, while a continuous program follows the asset inventory as it changes.
  • Depth: An annual engagement gives a tester days on one application, while a continuous run is narrower and repeats often.
  • Output: An annual test ends in a report, while a continuous program delivers each finding as it is confirmed.
  • Retest: An annual retest is booked as further work once the report is in, while a continuous program retests when the fix is reported.
  • Assurance: An annual test by an independent party gives customers, auditors and insurers a dated statement, while a continuous program gives the security team a running record.

Our guide to penetration testing explains the types, phases and rules of engagement.

Test Triggers and Their Records

A continuous program starts a test when a recorded event shows that the attack surface, or the threat to it, has changed. Each trigger needs a record that can be watched and a defined test that follows.

Trigger Record that shows it Test that follows
A new or changed internet-facing host A new DNS record, a new entry in Certificate Transparency logs, a new open port in scan results A first test of what the host serves: sign-in pages, exposed services and default configuration
A release or infrastructure change A change ticket, a release in the deployment pipeline, a firewall or cloud configuration change A test of the changed function and the access controls around it
A new advisory for an exposed product A vendor advisory, or a new entry in the CISA Known Exploited Vulnerabilities catalog A check of whether the exposed version is affected and exploitable as configured
A leaked credential A credential exposure record matched to a corporate account With approval, a check of which internet-facing portals accept the account and whether multi-factor authentication holds
A reported fix A remediation ticket moved to fixed A retest of the original finding using the original proof
A request A request from the system owner, for example before a launch A scoped test of the named system

A system that rarely changes would rarely be tested by triggers alone, so a rolling schedule underneath visits every in-scope system at a set interval.

The first trigger depends on discovery, since a host that is missing from the inventory cannot start a test. External attack surface management is the practice that finds those hosts.

Changes That Warrant a Test

A change warrants a test when it could alter who can reach a system or what they can do once there. The PCI Security Standards Council’s penetration testing guidance applies a similar measure: a change may be considered significant if it could affect the security of the network or allow access to cardholder data.

In most environments the following changes qualify.

  • New exposure: A host, port, API or storage location becomes reachable from the internet.
  • Authentication changes: A sign-in flow, single sign-on configuration, password reset path or multi-factor setting is added or altered.
  • Authorization changes: Roles, tenant separation or the rules that decide which records an account can read are rewritten.
  • Network boundary changes: Firewall rules, segmentation or remote access paths are modified.
  • Platform changes: A system moves to a new major version, a new hosting environment, or a more sensitive class of data.

Content edits, styling changes and routine patches that leave behavior unchanged typically fall below the line. The rule works best when it is written into the change process, so that a ticket of a qualifying type queues a test.

The Same Year Tested Continuously

Under a continuous program, each change on the March calendar becomes a trigger with its own test. The systems and the changes are the same as before, and the difference is the wait between a change and its first test.

Change Record that starts the test Test and outcome
May, portal release A release ticket of a qualifying type The upload function and the new API endpoint are tested when the release reaches production.
June, campaign site A certificate for the new subdomain in Certificate Transparency logs Ownership is confirmed with the marketing team, the hosting provider’s testing policy is checked, and a first test covers the sign-in page and exposed services.
August, gateway advisory The vendor advisory for a product in the inventory A check on the day of publication shows whether the gateway runs an affected version with the vulnerable feature enabled, and a confirmed result moves the patch ahead of the normal cycle.
September, firewall change A new open port in external scan results A test confirms that the administration console accepts connections from the internet, the rule is corrected, and a retest confirms the port is closed.
November, acquisition New domains and addresses added to the authorization The file transfer server receives a first test once ownership has transferred.
January, reported fix The March finding’s ticket moved to fixed A retest with the original proof closes the ticket or reopens it.

In the annual model the changes from May to November waited between four and ten months for a test. In the continuous model the wait depends on how quickly each record is noticed, which is why the measures later in this guide begin with the time from change to test.

What Standards Expect on Frequency

Most standards set a minimum interval and add change as a second trigger. The table lists what each primary document says.

Standard Reference What it says on frequency
PCI DSS v4 Requirements 11.4.2 and 11.4.3 Internal and external penetration testing at least once every 12 months, and after any significant infrastructure or application upgrade or change.
PCI DSS v4 Requirement 11.4.4 Exploitable vulnerabilities found in testing are corrected, and testing is repeated to verify the corrections.
PCI DSS v4 Requirements 11.4.5 and 11.4.6 Where segmentation isolates the cardholder data environment, its controls are tested at least once every 12 months and after any change to them. Service providers test them at least once every six months.
ASD Information Security Manual, September 2026 Control ISM-2118 Vulnerability assessments and penetration tests before a system is deployed, before significant changes are deployed, and at least every six months thereafter.
APRA Prudential Standard CPS 234 Paragraphs 27 and 31 The nature and frequency of testing must be commensurate with factors that include the rate at which vulnerabilities and threats change and the frequency of change to information assets. The testing program is reviewed at least annually or on material change.
APRA Prudential Practice Guide CPG 234 Testing control effectiveness A sufficient set of controls is tested at least annually, and controls protecting assets exposed to untrusted environments would typically be tested throughout the year.
NIST SP 800-53 Revision 5 Control CA-8 Penetration testing at a frequency the organization defines, on the systems it defines.

Read together, the documents describe one pattern: a minimum interval, a test when something significant changes, and a retest after a fix. A continuous program is one way to meet the change and retest parts.

ASD’s Guidelines for security assurance, the part of the manual that holds these controls, sit closest to the continuous model. Beside the six-month interval, control ISM-2119 calls for suitable AI models to augment vulnerability assessments and penetration tests. Our guide to AI penetration testing explains how agents plan and run a test.

Who performs the test matters as much as when. PCI DSS accepts a qualified internal resource or a qualified external third party, provided the tester is organizationally independent, and CPS 234 requires appropriately skilled and functionally independent specialists. Where continuous testing is meant to satisfy a compliance requirement, the assessor should confirm that the tester, method and evidence meet the standard’s wording.

Setting Frequency By System

Testing frequency is best set for each system, since a single interval for the whole estate tests quiet systems too often and busy ones too rarely. The factors that paragraph 27 of CPS 234 lists for regulated entities work as a checklist for any organization.

  • Exposure: A system reachable from the internet, or hosted where the organization cannot enforce its own security policies, is tested on every qualifying change and on the shortest rolling interval.
  • Rate of change: A system released every week produces a steady stream of triggers, while a system changed twice a year depends on the rolling schedule.
  • Sensitivity: A system that holds payment, health or other regulated data follows the strictest interval among the standards that apply to it.
  • Consequence: A system whose compromise would stop the business also receives human-led depth in the annual engagement.
  • Threat change: A product type that attackers target often, such as a remote access gateway or a file transfer server, is checked each time an advisory names it.

The rolling interval then sets the longest period any in-scope system goes without a test. Where a standard applies, the interval is no longer than the standard allows, which is six months under the ASD manual.

Each visit can take a different focus from the last, for example authentication on one visit and access control between accounts on the next, so depth builds through the year.

Systems run by a supplier need the same decision. Where an entity relies on a related or third party’s own control testing, paragraph 28 of CPS 234 has the entity assess whether the nature and frequency of that testing match the same factors.

Running Both Models Together

The models cover each other’s limits, so most programs keep the annual test and add continuous testing around it. The annual engagement keeps the work that needs sustained human attention and independence, and the continuous program takes the work that depends on timing.

  1. Agree one scope. The asset inventory, exclusions and rules of engagement are written once and used by both, so a host found in June is in scope for the annual tester the following March.
  2. Fix and retest in one cycle. Findings go to the owning team’s existing ticket queue, and the retest closes or reopens the ticket.
  3. Brief the annual tester. The year’s findings, fixes and open items go to the annual tester before the engagement starts.
  4. Spend the annual test on depth. With common exposure already handled, the tester’s days go to business logic, chained attacks, internal networks, and social engineering or physical testing where those are in scope.
  5. Feed the report back. A weakness the annual test finds on a system the continuous program had covered becomes a new check or a new trigger.

A goal-based exercise against the organization’s detection and response is a further step, described in red teaming vs penetration testing.

Ways to Deliver Continuous Testing

Continuous testing describes when tests run, and the organization still decides who runs them. The options differ mainly in how soon a test can start once a trigger is recorded.

  • In-house testers: An internal team takes tests from its own queue, which suits larger organizations with enough change to keep testers occupied and a reporting line separate from the system owners.
  • Repeated project tests: Each test is scoped, quoted and booked as its own engagement, often once a quarter, which suits systems that change on a planned release calendar.
  • Penetration testing as a service: A standing agreement holds the scope, and tests are requested or triggered through the provider’s platform during the term.
  • Agentic testing: AI agents start a test when a trigger is recorded and work inside limits agreed in advance, which suits internet-facing systems where the time from change to test matters most.

Services may be staffed by human testers, by agents or by both, so the buyer should confirm who performs each test. Our guide to penetration testing as a service compares the subscription with project testing and covers how to choose a provider.

Standing Scope and Authorization

Continuous testing needs authorization that stands for the year and a scope that updates as assets appear.

  • Written authorization: The system owner authorizes testing of named domains, address ranges and applications, with a contact for each.
  • Ownership check: A newly discovered host is confirmed as belonging to the organization before it is tested, since a third party’s server sits outside any authorization.
  • Hosted systems: Systems run by a cloud or software provider are tested within that provider’s published testing policy.
  • Windows and freezes: Each system carries its permitted hours, and change freezes such as a retail peak or a financial close suspend testing.
  • Notice to the SOC: The security operations team knows the source addresses of test traffic, so that analysts can tell a test from an intrusion.

Excluded techniques, request rate limits and approvals for higher-impact tests are agreed in the same document.

The authorization is kept current in the same way as the scope. It is reviewed at a set interval and whenever ownership changes, for example when a company is acquired or hosting moves to another provider. Each discovered asset is added with the date its ownership was confirmed and the name of the person who confirmed it.

From Finding to Detection

A confirmed finding also gives the detection team something to check, because a weakness that an authorized test can exploit today may have been exploitable for weeks.

  • Search the logs: The request pattern that proved the weakness becomes a search across web, identity and cloud logs for the period the weakness existed.
  • Check the alert: The test is a known attack at a known time, so the team can confirm whether monitoring recorded it and whether an alert fired.
  • Watch while open: Monitoring on the affected system can be tightened until the fix is in place.
  • Close with evidence: The ticket closes with the retest result attached.

The closed ticket is also the compliance record. PCI DSS requirement 11.4.1 expects penetration testing results and remediation results to be retained for at least 12 months. In a wider exposure program this work is the validation stage of continuous threat exposure management.

Measures of a Continuous Program

A continuous program is measured by how quickly change is tested and how reliably findings close. Counts of tests run or findings raised say little about either.

  • Time from change to test: The interval between a trigger being recorded and its test finishing, tracked for each trigger type.
  • Coverage: The share of internet-facing assets in the inventory that were tested within the rolling schedule’s interval.
  • Time to fix: The interval from a confirmed finding to a passed retest, by severity.
  • Retest result: The share of reported fixes that pass on the first retest.
  • Oldest open finding: The age of the oldest exploitable finding still open, with its owner.
  • Annual test overlap: Findings in the annual report on systems the continuous program had already tested, each traced to a missing check or trigger.

Targets depend on the organization’s risk appetite and release rate, so the first quarter is best spent recording a baseline.

The same measures feed management reporting. APRA’s CPG 234 expects test results to be reported to the appropriate governing body or individual, with follow-up actions formally tracked. Paragraph 29 of CPS 234 requires escalation to the board or senior management of testing results that identify control deficiencies which cannot be remediated in a timely manner, and the oldest open finding shows when that point is reached.

Starting a Continuous Program

A continuous program starts with the internet-facing systems and the triggers that already have a reliable record. The order below gives a team of any size a working program, typically within the first quarter.

  1. List what is exposed. The security lead, or the IT manager in an organization without a security team, compiles the internet-facing domains, addresses and applications with an owner for each.
  2. Sign the authorization. Each system owner authorizes testing in writing, and the exclusions, permitted hours and excluded techniques are recorded beside it.
  3. Choose who tests. The organization selects a delivery model and confirms that it meets any independence requirement that applies.
  4. Run a baseline. Every listed system receives a first test, so later tests measure change against a known state.
  5. Switch on the first triggers. New hosts and reported fixes come first, since discovery records and ticket status are usually available from the start.
  6. Connect the change process. The change manager adds a test flag to the qualifying ticket types, which brings releases and infrastructure changes into the program.
  7. Route findings to owners. Each finding opens a ticket in the queue the owning team already works from, with a named owner and a severity.
  8. Review after a quarter. The security lead compares the measures with the baseline, sets targets, and extends the triggers to advisories and leaked credentials.

In a small organization one person may hold several of these roles, and a provider typically supplies discovery and testing. Many receive it through their managed service provider, a model covered in white label penetration testing.

Continuous Testing With ThreatDefence

In ThreatDefence agentic penetration testing, each trigger in this guide starts a test without waiting for a booked date. The agents work inside an AI SOC platform, backed by a 24x7 SOC, that automates security monitoring, compliance and cyber risk management.

  • A new or changed host: When the ASM agent sees a host, service or certificate appear or change, the testing agent tests it.
  • A new advisory: A product the organization exposes is checked when a vulnerability in it is published.
  • A leaked credential: A credential the Brand agent finds points to a sign-in page worth checking.
  • A request: The organization asks for a test of a named system.
  • A reported fix: The agent retests when the fix is reported, and a passed retest closes the finding.

Systems that see no change are still reached by a rolling schedule. An independent human-led test can run alongside for internal networks and social engineering, which keeps the annual engagement in the program.

Frequently Asked Questions

What is continuous penetration testing?

Continuous penetration testing is a program in which tests run through the year, each started by a change such as a new host, a release, a new advisory or a reported fix, with a rolling schedule covering systems that stay quiet. Every run is a real test with an exploit attempt and proof. It differs from an annual engagement mainly in what starts the work and how soon a fix is rechecked.

Does continuous penetration testing replace the annual penetration test?

No. The annual test by an independent party still provides the dated assurance that customers, auditors and insurers ask for, and it gives a human tester time for business logic, internal networks and social engineering. Continuous testing covers the months between and the systems that change in them, so the annual tester can spend the engagement on depth.

How often should penetration testing be performed?

At least as often as the strictest standard that applies, and again whenever a significant change is made. PCI DSS sets 12 months plus significant changes, ASD’s Information Security Manual sets six months plus deployment of significant changes, and APRA CPS 234 ties frequency to the rate of change in threats and assets. Internet-facing systems that change often justify testing on every qualifying change.

Does PCI DSS require penetration testing more than once a year?

Yes, in some cases. Requirements 11.4.2 and 11.4.3 call for internal and external testing at least once every 12 months and after any significant infrastructure or application upgrade or change. Requirement 11.4.6 has service providers test segmentation controls at least once every six months, and requirement 11.4.4 has testing repeated to verify that exploitable findings were corrected.

What counts as a significant change for penetration testing?

A change is significant when it could alter who can reach a system or what they can do there. Typical cases are a newly exposed host or API, a changed sign-in or access control model, a modified firewall or segmentation rule, a major platform upgrade, and a system that starts holding more sensitive data. The organization’s own risk assessment draws the line, and the change process should record it.

Can continuous testing results be used as compliance evidence?

Yes, where the tester, method and records meet the wording of the standard. PCI DSS accepts a qualified internal resource or external third party with organizational independence, and APRA CPS 234 requires skilled and functionally independent specialists. Dated findings, proofs and retest results are useful evidence of testing after changes. The organization’s assessor decides whether they also satisfy the periodic test.

When should a new system first be tested?

Before it goes live, where possible. ASD’s Information Security Manual calls for vulnerability assessments and penetration tests before a system is deployed and before significant changes are deployed. A system that reaches the internet unannounced, such as a campaign site or an acquired server, should be tested as soon as discovery records it and ownership is confirmed.

How is a continuous penetration testing program measured?

By speed and closure. The main measures are the time from a recorded change to a finished test, the share of internet-facing assets tested within the schedule, the time from a confirmed finding to a passed retest, and the age of the oldest open exploitable finding. Findings in the annual report on systems already covered show where a check or trigger is missing.

Is continuous penetration testing the same as penetration testing as a service?

No. Continuous penetration testing is a practice, in which tests start on change and on a rolling schedule. Penetration testing as a service is one way to buy that practice, through a standing agreement with a provider. An in-house team or a series of project tests can also deliver continuous testing, with a longer wait between a change and its test.

Talk to Our Experts

See how ThreatDefence automates security monitoring, compliance and cyber risk management in one platform.

    I'm an MSP, MSSP, or reseller

    By submitting this form, you accept the terms in our privacy policy.