Shinobi's shortcut to AI-powered pentests in staging and dev envs

Shinobi Security is like leasing an elite ethical hacker—but this one's AI and never sleeps. 

Founded in 2023 by offensive security veteran Varun Uppal, Shinobi takes the red-team mindset and encodes it into an autonomous swarm of AI agents. These agents collaborate, escalate privileges, chain vulnerabilities, and think like real attackers, because their creators used to be ones.

Developers treat these agents just like they would security teammates. As soon as developers create a new version of their apps, they prompt the agent with a bit of context about the app and how it is supposed to work, and the agents get to work hacking at the new app and attempting to find vulnerabilities. 

Shinobi flips the script of most security tools: instead of drowning you in dashboards and warnings, it proves your vulnerabilities with working exploits. You don’t get a random alert. You get told, “You will get hacked unless you fix this, and here’s how.”

The roadblock: getting AI hackers into your apps (safely)

Customers didn’t want even Shinobi's AI hackers testing on production apps, so they often requested that Shinobi agents work within the confines of staging or dev environments. Shinobi’s product works wonderfully once it’s set up in a customer’s environment, but customer environments can be tricky.

Applications often required custom headers or authentication credentials just to honor requests from Shinobi's agent, which meant they would have to customize their solution for every nuance and oddity of those dev/staging environments. Varun says, "An out-of-the-box configuration just doesn't work for us."

From zero to 'we're in' in 15 minutes

After evaluating several options, Shinobi Security selected ngrok to create public endpoints for local machines within their customers' networks. ngrok's flexibility and ease of configuration were pivotal in their decision—Varun was able to get a PoC up and running in minutes. 

Varun says, "Our demo to trial conversion rate is like 95%, which is insanely high in the enterprise space. ngrok has been a blessing, cutting down our time to value insanely short." Implementing ngrok reduced the setup time to run penetration tests on their customers' apps to just 15 minutes.

Even through ngrok simplified how Shinobi accessed customer networks and shrunk the time-to-test, they sometimes ran into other hurdles. For example, some agents would need access to an app within a customer’s network that had internal DNS names or redirect flows that would break in an outbound tunnelled environment (e.g., 302 redirects to internal.corp.local). Shinobi wrote a unique policy unique traffic policy to intercept and rewrite 302 redirect headers coming from customer apps, preserving UX and agent behavior.

on_http_request:
  - actions:
      - type: add-headers
        config:
          headers:
            host: 127.0.0.1
      - type: forward-internal
        config:
          url: https://service-01.customer-abc.internal

on_http_response:
  - expressions:
      - "res.status_code == 302"
    actions:
      - type: set-vars
        config:
          vars:
            orig: "${url.parse(res.location).path}"
      - type: remove-headers
        config:
          headers:
            - Location
      - type: add-headers
        config:
          headers:
            Location: "$NGROK_DOMAIN/${vars.orig}"

Find the entire Shinobi policy in our examples gallery—along with other ready-to-ship setups using endpoints and Traffic Policy.

As they expand their customer base, Shinobi also imagines writing other policies for ensuring their AI agents can properly authenticate themselves even in dev/staging environments.

The payoff: faster tests, safer software

Companies choose Shinobi because they want truly intelligent vulnerability testing integrated into their development workflows. With ngrok, Shinobi can demonstrate value as quickly as possible, and agents can work asynchronously without interrupting developer workflows or simply failing to deliver on Shinobi’s promise.

As Varun puts it: “We pitch it not like any other tool. We say, you're going to lease a hacker from us. They'll become your first security tester or penetration tester—and whatever extra work you have, just farm it out to them. ... [Shinobi] just goes and does it thing and then saves the world one application at a time.”