AI Pentesting: What Autonomous Systems Actually Deliver Today

AI pentesting is the industry's big promise: an agent that takes over the entire test fully autonomously. We put the claim to the proof ourselves, using the available fully autonomous systems and our own local inference. The balance is more sober than the advertising, and a good deal more useful.

In short: enumeration and parts of exploitation work well today. Fully autonomous, the systems are inconsistent and unreliable. The potential lies in assisting pentests and automating single, small sub-steps.

Autonomous systems put to the test

We tested Strix and PentAGI, two available implementations of autonomous pentest agents.

Strix finds weaknesses in code and on web servers, but "finds" means neither reliably nor consistently. The user interface is not production-ready: it contains bugs, crashes on unexpected model output, and delivers a single markdown file without a fixed structure as its only output. A test report becomes a collection of text that a human has to sort out again.

PentAGI appears more mature on the surface and organises tasks considerably better than Strix. In practice, flows partially pause when they are supposed to run in the background, and the results remain inconsistent. The agents often do more than the current task, duplicating a lot of work. Most critical are the scope violations: at times, IPs and ports outside the defined perimeter were tested. What is inefficient in the lab is a contractual problem at a client. A pentest without scope discipline is not a pentest.

The takeaway: assisted, not autonomous

The value of a pentest lies in traceability, adherence to scope, and a dependable rating. Those are exactly the three points autonomous agents do not deliver today. A model can, however, work reliably where sub-steps are reproducible: structuring attack surfaces, narrowing down code locations, pre-drafting raw report text, adding variety to wordlists and fuzzing tests. The tester remains the decision-maker, the model the multiplier.

Local inference instead of cloud APIs

Anyone sending pentest data through a cloud API hands attack surfaces, findings, and architecture details to a third party. Most clients rightly do not want that. Which is why we run inference locally.

The hardware choice fell on an NVIDIA RTX PRO 6000 (Blackwell): 96 GB of GDDR7 memory, NVFP4 hardware support for quantised models, and CUDA as a software ecosystem still clearly ahead of AMD. The card is available as a single PCIe card, unlike data centre accelerators such as the B200. Forgoing NVLINK is acceptable for our workflows. Alternatives reviewed: the RTX PRO 5000 Blackwell with slightly less performance and memory; the H100 NVL and H200 NVL with more memory bandwidth and NVLINK, but an older generation and higher price; the AMD Radeon AI PRO R9700, considerably cheaper but with far less VRAM and weaker software support; and the GB10 with shared, far slower memory and no option to expand with additional cards.

Inference stack and open-weight models

On the inference software: ollama is simple, with few configuration options. llama.cpp is the right choice for CPU offloading or inference purely on the CPU. For production we use vLLM, today's standard for full GPU inference, comfortable with several concurrent users and delivering the best performance on graphics cards.

Running fast on the card include Qwen3.8-27B, Qwen3.6-35B-A3B, Qwen3.5-122B-A10B, gpt-oss-120b, Mistral Small 4 and Gemma 4. Notable: Qwen3.8-27B is comparatively small and yet currently among the best and newest open-weight models. Open weights mean more than the absence of licence fees: full control over deployment, context length, and data custody.

The honest balance: AI does not replace the pentester. It replaces the boring parts of the job and speeds up the rest. Anyone buying a fully autonomous AI pentest today is buying a promising preview, not the present.