Is Your Pentest Firm Drafting Your Next Breach Headline?

By now I'm sure you are all aware of what proper AI governance means for your organisation, but for those of you who don't, AI governance refers to the framework of rules, practices, and processes used to ensure that AI systems are developed and deployed responsibly, ethically, and safely.

Most organisations vet their software vendors for AI risk, but what about your third parties / service providers your organisation utilises? How do you gain visibility into their AI usage, security & controls? This is super important. Especially in the world of pentesting, most organisations I have spoken to who regularly undertake penetration testing haven't even thought about this risk! As a trusted advisor, we are expected to maintain a high standard of security when it comes to the protection of our customer data, the data we hold and the findings we identify, if exposed, can lead to a direct breach of a customers environment, effectively the blue print for becoming the next breach headline.

This is where AI, if not used appropriately, can inadvertently expose your organisations pentest findings to data exposure, privacy issues, data sovereignty, compliance breaches and a number of other risks. Putting the actual testing processes to the side for a minute, how do you know if your pentest firm is not putting your sensitive data into LLMs such as ChatGPT and Gemini? How do you know if they are using an AI engine which was trained using your pentest data?, or if your data is located in an AI located in a different region or reporting deliverables are exposed to an AI?, all super important questions that need to be answered up front by your pentest firm(s).

Then we have testing - with another set of questions that need to be answered, how do you know if your pentest is being delivered by real people, and not 80-90% AI and final touches by testers or reporting staff. I'm seeing this a lot by other pentest firms at the moment, clients are offered bargain prices for their pentests, which they snap up engaging the firm, then AI is doing all the work behind the scenes and delivering a sub-par report (imo), you always get what you pay for, right :)

The obvious exclusion here is CTEM (Continuous Threat Exposure Management) or Automated/ Continuous pentest platforms (CPT), these are transparently AI powered in the backend, and this is communicated to the customer so it's to be expected, but the question remains, how accurate are the results if they are not being reviewed by an experienced and skilled tester in the loop? (ps if you are interested in employing CTEM / CPT in your environment please hit me up!)

As testing firms, we absolutely should be utilising AI to speed up some processes within tooling (hence why AI was built into the latest Kali version), it can be an enabler, allowing a tester for example to replay a significant number of attacks against a large number of endpoints or variables without the tester having to manually launch every one and taking a large amount of time to complete that, so it can definitely increase pentest coverage if used appropriately, AI is also well suited to craft payloads, scripts and code that bypass EDR tools which is one of the ways we use it, but transparency is key.

So here is my checklist for every organisation when it comes to engaging a pentest firm and shoring up their AI usage.

Your AI Security Checklist:

• What AI technologies and practices does the pentest firm leverage in engagements?

• Does the firm have a formal AI governance and/or AI disclosure statement?

• Can I get a copy?

• Does the statement outline where your data lives and how it is used with respect to AI?

• Does the statement define what toolsets are used by testers that leverage AI?

• Does the statement outline their AI policies around LLM usage and tooling?

• Does your AI tooling ensure data sovereignty and ensure our data remains in Australia?

• What percentage of the work is delivered by AI and what percentage is humans? (If it's 90% AI, you aren't buying a pentest; you're buying a scan).

• Are AI outputs vetted by experienced and knowledgeable testers?

• Are AI identified findings validated through manual testing methods?

• Is AI leveraged in reporting? If so, how?

• Does my data get used for training of AI?

It's also recommend that you engage long standing reputable pentest providers (10+ years experience) that prioritise validated results over 'AI-generated noise'. Firms that prioritise validated results and manual testing over AI will generally have well defined and transparent policies around AI usage and will favour manual testing over AI powered testing.

I'm currently a member of the CREST Maturity Assessment Tool & Pentest Buyers Guide Working Groups, and we will be rolling AI questions into these tools in the coming months, but as of right now, you need to be gaining clarity from your current or future pentest firm(s) and ensure they are not exposing you inadvertently to AI usage risks.

My team at Nexon have been pentesting for over 20 years, we have a very mature practice, and we include our AI disclosure statement in all pentests, we're upfront about how we use AI, and how we ensure our customers data is always protected as that's the number 1 priority to us. To sum up our usage it is very very limited, we prefer to utilise certified, experienced and knowledgeable testers to manually deliver all of our engagements, our AI usage is only 5%, with some higher percentages if we are leveraging Deepfakes in some social engineering campaigns.

To that note, if you are looking for your next pentest and an alternate firm for a different lens, please feel free to reach out, i'd love to hear from you!

#cybersecurity #pentesting #ai #aigovernance #infosec #cyberrisk #danweis #boardroomcyber