ngrok Security Disclosure, May 2022

Last week, we fixed a multi-tenancy bug in the ngrok dashboard’s caching layer that unintentionally leaked data between a small subset of ngrok accounts when they viewed the ngrok dashboard.

This bug affected less than 5% of ngrok’s active users. We have contacted all accounts affected by the bug directly via email with instructions for remediation.

If you have not received an email, your account was not affected.

We have neither found evidence nor received reports of any malicious activity related to the bug.

What is the extent of information revealed by the bug?

• The personal ngrok Authtoken of affected users may have been viewed by another ngrok user. This is the most serious repercussion. An ngrok Authtoken is a scoped credential that has limited capabilities: it cannot be used to modify or access your account’s configuration data. However, it can be used to start new tunnels on your ngrok account.
• Account data, including your email address, online tunnels, reserved domains and addresses, and other configuration data may have been viewed by another ngrok user.

What are the limitations of information revealed by the bug?

• The failure was read-only. No affected account could modify another’s.
• No traffic across tunnels or endpoints was ever viewable to the other account.
• No passwords, API keys, SSH Keys, additional Authtokens, Credit Cards or Payment Data were viewable by another account.

What is the scope of affected accounts?

• This bug affected less than 5% of ngrok’s active accounts.
• The majority of accounts were only affected by a single request leak.
• The logs do not identify any malicious patterns of exploitation of the bug.
• This was not a security breach by a malicious attacker and ngrok’s systems have not been compromised.

All affected users have been contacted directly via email with the recommendation to rotate their Authtoken using the following steps:

1. Log into the ngrok Dashboard
2. Click the following link: Reset Your Authtoken and follow the instructions
3. The existing token will be revoked immediately and a new one will be generated
4. Update your ngrok.yml file either manually or by either running the command line operation on that page

Anatomy of the Bug

Ngrok’s internal RPC systems perform a type of caching called ‘request coalescing’. For certain annotated methods, if two RPCs occur simultaneously and they specify the same request parameters, the system only performs a single RPC and returns the same result to both callers.

This optimization was applied to the RPC that loads an ngrok account’s dashboard state. Unlike most requests in our system, the request object for this RPC does not include the ID of the account in the request parameters because that data is pulled from the dashboard user’s session state. This meant that requests by two different users for their dashboard state could be incorrectly collapsed into a single request.

For example, two simultaneous calls to GetAccount(id=10) and GetAccount(id=10) could be coalesced but GetAccount(id=9) and GetAccount(id=10) would not. The call to load the ngrok dashboard loads the account ID via session state, so it is expressed as LoadDashbard() and not LoadDashboard(id=10). This resulted in the bug where two simultaneous calls to LoadDashboard() for different accounts could be coalesced.

More plainly, if Alice and Bob both loaded the dashboard at the same time, and Bob's request reached the dashboard service while Alice's request was being processed, they would both receive the response meant for Alice's dashboard, letting Bob view Alice's dashboard state.

Discovery and Investigation

On February 2nd 2022, we deployed a change that enabled request coalescing to the ngrok Dashboard that could result in account information from one user account being displayed to both users.

On May 7th, we received a report from a user describing the unexpected behavior of seeing another user’s account information.

ngrok’s engineering team began investigating, eventually building a reproduction of the bug and crafting a fix.

On May 11th, the fix for the bug was deployed to production.

ngrok’s engineering team began performing forensics on log data to assess the impact of the bug across all accounts as well as to review the data for evidence of any malicious activity. We enumerated all instances where one account could have viewed information from another.

On May 17th, the Engineering and Customer Success teams contacted affected users via email with a notice and remediation steps.

How we’re improving

We’ve used this situation to start a comprehensive analysis of how we handle and process log data, detect patterns within it, and respond effectively. These lessons will help us detect suspicious activity sooner and at a larger scale than before.

With regards to this specific incident, we are:

• rearchitecting parts of the User Dashboard to prevent the use of caching layers and make more granular permission checks
• reconsidering the role and display of the Authtoken to use it safely while keeping our onboarding experience simple
• exposing more account events with Event Sources for customers to understand account activity in near real time

Please reach out with any questions to support-may2022@ngrok.com

‍