Webhook Security in the Real World

Webhooks are a simple and powerful way for services to notify each other that something interesting has happened. So much so that it became the standard for communicating events across different SaaS services, systems, and apps — with examples ranging from notifying when a Docusign contract is in effect, to when a GitHub PR gets submitted, to when a Stripe payment is processed.

While webhooks give us power and flexibility, they also require a live connection to the public internet. Without good security controls, third parties could explore webhooks for malicious activities such as sending fake events to taint your systems, obtain confidential information, corrupt your data, or spam users. Luckily, many webhook providers deliver capabilities to verify the sender, validate the data sent, and even mitigate more specific attacks.

At ngrok, we’ve developed a module to validate webhook signatures without requiring developers to write additional code making their apps more secure by default. After exploring 100 webhook providers and building support for over 50, we’ve collected the most common, interesting, and challenging patterns. In this article, I’ll dive into some of the pains and possibilities of webhook validation with real-life examples.

TL;DR

We studied 100 and integrated with over 50 webhook providers and found some definite patterns:

• Hash-based Message Authentication Code (HMAC) is, by far, the most popular webhook authentication method: HMAC is used by over two thirds of webhook services.
• The most popular HMAC implementations use SHA-256 as hash and sign only the message body using base64 or hex to encode the result, without additional controls for timestamp, versioning, and zero downtime rotation.
• Only 30% of the webhooks have controls to mitigate replay attacks. While there are a variety of approaches to improve security, adding a timestamp in webhook signature validations would be a strong start.
• Only 27% of implementations provide versioning and forward compatibility, giving providers and developers a reliable path to support both a current and previous implementation without risking downtime.
• Only 8.5% of implementations offer zero downtime secret rotation to allow developers to update applications and systems over time instead of coordinating a global system wide change all at once.

How this research is organized

This article is split into three parts:

In Findings, we summarize how the industry implements webhook security.

In Bits and Bytes, we dive into our findings deeper, showing in detail the good, weird, and maddening things we found. Because this section is deep, we organized it into different subsections:

1. Webhook Security: during Setup & Runtime
2. Webhook Authentication and Message Security

In Best Practices, we give ideas on how to both build and consume webhook providers and better secure webhook listeners.

If you're not familiar with webhooks, we recommend a quick read of our webhook primer before diving into the details.

A quick primer on webhooks

Already familiar with webhooks? Skip to the next section.

What are webhooks?

Webhooks are a language and framework agnostic way for a service to send updates to other applications as events happen. They're popularly used to notify third-party systems of important events and changes, such as a new incident tracked by PagerDuty, a recurring payment processed in Stripe, or an SMS message sent in Twilio. Much of the modern web depends on this distributed communication pattern.

1: Findings

As part of developing webhook validation in one line for the ngrok platform, we studied 100 different webhook providers and integrated the 50 most requested. During our journey, we saw thoughtful and challenging patterns. These are some of our findings:

‍

Hash-based Message Authentication Code (HMAC) is the most popular webhook authentication method

HMAC is the overwhelming favorite webhook authentication method in use by two-thirds of webhooks we've researched (65%) with no authentication being a distant second (16%).

While none of the other methods reached over 10% adoption, some key points:

• Shared Secrets — i.e. basic auth credentials and bearer tokens— are used 8% of the time.
• Asymmetric Key Encryption (RSA and ECDSA) — despite its non-repudiation controls — is not widely adopted. We believe this is likely due to the complexity of rotating keys.

A growing number of companies (5%) use OAuth 2.0, JSON Web Tokens (JWTs), and JSON Web Keys (JWKs) to secure webhooks, but with inconsistent implementations (more details on JWTs, JWKs, and OAuth 2.0)

‍

The most popular HMAC implementations

Out of the HMAC webhooks we built integrations, we found 2 patterns tied in 1st place with the only difference being the encoding method:

‍

As soon as additional controls are added, webhook validations have nothing else in common

Once you add additional controls to webhooks — such as timestamps, versioning, and zero downtime rotation — the similarities disappear. This is where webhook validation becomes both interesting and sometimes frustrating.

These variations range from the clear and simple — like timestamp format — to the more complex and subtle such as the ordering of fields used in the signature. Each of these change the verification process in a significant way.

As a result, of the webhook providers we integrated with at least one additional security control — timestamp, versioning, or zero downtime rotation — only 2 services used the same procedure for validating their signatures: Calendly and Mux.

‍

Webhook Documentation is key for Verification

Given both the variations and similarities between webhook implementations, good documentation is crucial for building reliable and secure webhook listeners. When the differences are significant and therefore clear, it’s easy to distinguish implementations. When the differences are more subtle and less obvious, the familiarity becomes a risk leading to complacency and missing those differences. The bare minimum documentation details the validation steps, but more complete documentation should include the webhook’s security controls and provide example payloads and sample verification code. Unfortunately, we found webhook documentation that makes knowledge assumptions, skips steps, and consistently delivers incomplete code snippets. About 10% of the webhooks we've implemented missed critical information in their docs — including lack of the encoding format (base64 or hex), the signature date format (Unix or RFC 3339), and hash signature payload construction instructions.

To overcome incomplete docs, we spent significant time performing guesswork, scraping the internet, and unit testing webhooks until finding something that made sense. While we were motivated to continue digging and researching to get the correct answer, many developers would have skipped verifying the webhooks out of frustration and risked having an insecure app.

‍

Almost 1 in 3 webhook implementations lack or present weak authentication or message validation mechanisms

With a weak or non-existent signature validation, a bad actor could generate malicious webhook requests indistinguishable from legitimate requests. Of the webhooks surveyed, 32% used weak authentication and message validation:

• 8% of webhooks use SHA1 to sign messages: SHA1 has been considered unsafe for use for over 15 years and subject to chosen-prefix collisions since 2020 (see SHA-1 Shambes).
• 24% of webhooks offered no authentication or rudimentary authentication controls — shared secrets, basic auth, bearer tokens, or verification token — and no message signature validation, which is even worse than SHA1.

‍

Over 80% of webhooks with HMAC use SHA-256 to sign messages, but lack complete security controls

Of the webhook providers using HMAC, 83% sign their webhook messages with SHA-256, which is considered a strong hashing algorithm for webhook authentication and message integrity. Unfortunately, the vast majority won't include features to mitigate other attacks:

• 83% sign webhooks with SHA-256
• Only 31.9% mitigate replay attacks by signing payloads with a request timestamp
• Only 27.7% implement versioning and forward compatibility controls
• Only 8.5% have the ability to rotate keys with zero downtime

These security controls are adopted sporadically. Within the webhook providers using HMAC, 17% implement none of them, 40.4% adopt only one of these best practices, and only 4.3% of providers apply all three additional controls.

2: Bits & Bytes

Now let’s dive into our findings at a deeper level, showing the good, weird, and maddening patterns we found. Because this section is deep, we organized it into different subsections:

1. Setup
2. Runtime
3. Authentication
4. Message Security

‍

Webhook Security: Setup

All providers provide some form of authentication and authorization to access the webhook setup pages and APIs. Some webhook providers like Twitter, Okta, and Microsoft OneDrive take an extra step, requiring a webhook listener verification during setup to ensure you own and control the app that will receive webhook calls.

These providers trigger the verification process with a special request containing a random string that must be echoed back as part of the response. For example:

GET https://listener.com/my-webhook?validationToken={randomString}

Content-Length: 0
HTTP/1.1 200 OK
Content-Type: text/plain

{randomString}

While echoing back a string seems like a trivial exercise, this verification requires admins to demonstrate they control the webhook listener before any notification with sensitive data is sent.

‍

Webhook Security: Runtime

Webhook providers implement security controls at runtime to give listeners ways to determine if notifications are legitimate, modified, or replayed.

In our research, 97% of the webhook providers implement security controls on runtime. However, at the time of our study, we couldn't find any documentation on Jira, Olark, and ActiveCampaign webhook security controls on runtime.

Security during runtime is the most astounding area of this article in both volume and findings.

Webhook Authentication and Message Security

Authentication and message security are where webhook providers include most security controls and introduce the most quirks in their implementations. Therefore, we will spend the most time, effort, and frustration here.

Webhook providers implement different methods for authenticating and securing webhook messages, with HMAC being the most popular choice by far with over 65% but let’s start from the simplest and build up to that approach.

‍

Shared Secret (Basic Auth, Shared credentials, Bearer Token, or verification token)

In our research, 8% of the webhook providers used some form of shared secret — Basic Authentication, shared credentials, bearer token or a verification token — for authentication. In this method, the webhook provider and listener share a common secret used exclusively to authenticate webhook requests. The secret is not used for signing the webhook payload:

1. On webhook requests, the provider sends a webhook notification containing the message plus a shared secret in a pre-defined header variable or the Authorization header using the Basic Auth format (Authorization: Basic <"username:password" in base64>) or the bearer token format (Authorization: Bearer ).
2. The webhook listener validates the header value versus the shared secret. Only requests with the correct secret are considered “valid” and should be accepted.

This security method addresses only the webhook service authentication and does not implement any control on message integrity. Even with HTTPS, the shared secret is processed in clear text, increasing the risk of the secret getting compromised.

Therefore, a shared secret-based approach is a weak choice by itself and should include compensatory controls outside the webhook implementation — such as IP Restrictions and callback requests — to mitigate risks.

Note: Some services offer this method as a quick option meant only for development/unit testing, alongside more robust security controls for production usage. DocuSign, for example, offers basic authentication, Request Signatures with HMAC, and Mutual TLS, and encourages the use of HMAC and mTLS in production use-cases. This is a reasonable approach to speed development without putting production at risk.

‍

Hash-based Message Authentication Code (HMAC)

HMAC is, by far, the most popular authentication and message security method used on webhook requests including 65% of the 100 webhooks we studied. In this method, the webhook provider and listener use a secret key to sign and validate webhook requests.

1. On webhook requests, the provider signs the webhook message using the secret key plus a hashing algorithm — typically HMAC-SHA256 — encodes the resulting signature in base64 or hex, and includes the encoded signature in the webhook request as a header.
2. The webhook listener receives the request and repeats the same steps — signs and encodes the webhook message using the secret key — and compares the resulting signature with the value sent in the request header. If the result matches, the request is considered legitimate.

HMAC offers some significant advantages versus Shared Secrets:

1. The secret key is never transmitted via the same channel: In HMAC, secret keys are not sent with the webhook request — only the signatures created with it — reducing the risk of stolen keys.
2. Authentication + message integrity: Assuming the secret key is known only by the webhook provider and the listener, the HMAC process verifies that the message comes from the webhook provider (authenticity) and its contents are the same as they were at the time of sending (integrity).

‍

HMAC is only as good as its implementation

Like any other security control, HMAC is only as good as its implementation. In our research, we saw many examples of webhook providers with unnecessary complexity, poor security, and lack of documentation that made their solutions tough to implement, manage, and keep safe. While we have more examples in findings, here are three of the most frustrating cases:

Unnecessary complexity: Many webhook providers have signature steps with unnecessary steps. One webhook provider adds line breaks within the webhook signature. This adds zero security but increases complexity, introduces problems on systems with different line breaks, and increases the odds of frustrating developers. Here's an example of steps from a webhook provider:

1. Take the method for the request
2. Add a line break
3. Take the URL path of the request
4. Add a line break
5. Take the list of signed headers comma-delimited from a specific header.
6. For each header, append the header name (in lower case) + &quot;:&quot; + the header value.
7. After all headers are collected, concatenate them separated by semicolon &quot;;&quot;
8. End the list of headers with
9. Add a line break
10. Add the request body5. Hash the string above using the secret key6. Base64 encode the resulting hash

‍Use of privileged credentials as secrets: One of the SHA 1 webhooks we implemented used an API key — effectively a credential for administrative operations — as the key to sign webhooks. Because the API key could be used — alongside an account id — to make REST API requests, it introduces additional risk and makes the webhook verifier code unnecessarily riskier to distribute. In practice, this shared secret should be a long, randomly generated string that has no relation to any account identifier or system credential.

Bad Documentation: Some providers skip steps, deliver incomplete code snippets, and assume people's familiarity with their webhook security implementation. Approx. 10% of the webhook providers we implemented missed critical information on how to generate their webhook signatures. One of the implementations documented their entire webhook security in one paragraph, missing key information such as the encoding format used in their signature. To overcome the bad docs, we spent extra calories performing guess work, scraping the internet, and unit testing webhooks until we found the correct incantation. While we were motivated to provide a complete solution, many developers would likely skip verification after dealing with this frustration.

‍

HMAC Key Rotation (with zero downtime)

HMAC keys are no different than other encryption keys and should be rotated periodically for better security. Some webhook providers — like Box, Brex, Docusign, and PagerDuty — implemented interesting controls within their webhooks for key rotation. PagerDuty, for example, can sign webhooks using multiple signatures and then add them all to the X-PagerDuty-Signature header (comma separated). This allows you to roll out a new secret key and gradually update your webhook listeners without breaking all listeners at once.

‍

HMAC Forward Compatibility

In addition to the key rotation, some providers implement versioning within their signatures, allowing for future upgrades in the hash algorithm and signature process. Hubspot is an interesting example of a provider that changed its webhook notification over the years. In its initial version, Hubspot signed only the webhook request body to validate calls. Over the years, Hubspot improved its webhooks to include timestamp validation, changed the webhook signature contents, and changed the encoding from hex to base64. With Hubspot providing the webhook signature version to the listener (via a header value or within the signed message), it ensures forward compatibility possible without breaking legacy listeners.

‍

Timestamp Validation and Replay prevention

Another interesting feature of HMAC is that signatures can be extended beyond the request body to include other pieces of information. By combining HMAC with timestamps, providers — such as Calendly, Slack, and Zendesk — offer a way to mitigate replay attacks. HMAC can be used with timestamps as follows:

On webhook requests, the provider:

• Concatenates the timestamp of when the request is created with the webhook message.
• Signs the concatenated value with the secret key
• Adds both the encoded signature and the timestamp to the webhook request as header variables.

The webhook listener receives the request and:

• Extracts the timestamp from the request header and validates if the timestamp is within an acceptable time frame (i.e., 3-5 minutes).
• If the timestamp is acceptable, repeat the same steps from the webhook provider to sign the timestamp+request body and compare the results with the signature sent via the request header. If the result matches, the request is considered legit. If not, the request is not authenticated and its contents or timestamp are tampered.

To ensure the timestamp validation works, you should keep your listener clock in sync with an NTP server.

‍

Asymmetric Key Encryption (ECDSA and RSA)

In our research, we found a couple of providers — PayPal and SendGrid — using asymmetric encryption for webhook security. In this method, the webhook provider uses a private key (only known to the provider) to sign requests, while the listener uses a public key with a verifier to validate webhook calls. At a conceptual level, the process works as follows:

1. On webhook requests, the provider signs the webhook message using its private key. The signature is included in the webhook request as a header.
2. The webhook listener receives the request and runs an ECDSA or RSA verifier with the public key, the request body, and the signed value from the provider. If the verifier returns positive, it means that the signature could only be created by the private key, and the request is considered legit.

The big difference between HMAC and asymmetric key encryption is in how the verification process is carried out. In HMAC, webhook listeners generate the same signature from the provider using the same key to validate the request. In asymmetric encryption, listeners cannot generate the same signature. Instead, they use a verifier with the public key to confirm if the request is legit.

This difference in verification logic is used to deliver non-repudiation. In HMAC, anyone with access to the private key can generate a webhook request. With asymmetric keys, only the webhook provider — in possession of the private key — can generate legit signatures.

Webhook implementations with asymmetric keys can also use the same techniques as HMAC providers to add forward compatibility and replay prevention. SendGrid, for example, implements a timestamp header (X-Twilio-Email-Event-Webhook-Timestamp) within the payload to mitigate replay attacks.

However, asymmetric encryption comes with drawbacks. The most compelling: asymmetric encryption is harder to implement than HMAC and will introduce many different methods for shipping and rotating public keys

PayPal sends the public key with the webhook request for verification (in the PAYPAL-CERT-URL header). SendGrid informs the public key in its settings page.

JWTs, JWKs, and OAuth 2.0

Webhook providers such as Sendgrid, Wix, and Brex use security methods and protocols typically found in REST APIs — such as OAuth 2.0, JSON Web Tokens (JWTs), and JSON Web Keys (JWKs) — in creative ways to protect their webhooks:

• Sendgrid uses OAuth as a client. Before sending a webhook message, Sendgrid authorizes itself against third-party OAuth Authorization Servers to issue access tokens. The OAuth tokens must be validated on the listener side to ensure the request is legit and authorized. The validation method is defined by the OAuth Authorization Server, its features, and configuration.
• Brex uses OAuth to release webhook keys. Brex uses HMAC with symmetric SHA256 keys for webhook validation. However, in order to obtain the shared key, webhook listeners must make a request to a REST API (GET /v1/webhooks/secrets) protected with OAuth. The call should happen at regular intervals to prevent downtime after a key rotation.
• Wix uses JWT in the webhook payload. Wix implements JWT in the webhook payload. Webhook listeners should follow the standard procedure in JWT to validate the webhook payload, using a public certificate provided by Wix in the webhook registration.
• Janrain (an Akamai product) uses JWKs and JWTs in webhook notifications. Janrain takes a similar approach to wix and sends a JWT in the webhook payload for validation. However, instead of providing public certificates with the webhook registration, they implement the JSON Web Key (JWK) standard and endpoint to obtain the public keys for webhook validation.
• Plaid uses JWTs and JWKs in webhook notifications, but not on the message. Plaid uses JWTs and JWKs like Janrain, but stores the JWT in the plaid-verification header and uses the webhook payload to send information.

Mutual TLS Authentication (mTLS)

The use of mTLS authentication ensures that traffic is secure and trusted in both directions between webhook services and listener apps. With mTLS, both webhook service and listener apps pass through a TLS handshake — in which each system is required to authenticate with their certificates — before the webhook message is sent. This method delivers a stronger authentication between the webhook service and listener, ensures communication confidentiality (the webhook message is sent only after both sides are authenticated), and both integrity and non-repudiation (when combined with webhook signatures).

However, mTLS is often difficult to configure. It also requires updates whenever TLS certificates are issued, renewed, or revoked both on the webhook service and the listener side. This additional burden is considered overkill for most use-cases.

3: Best Practices

In this section, we present some ideas on how to better secure webhooks for both providers and consumers.

Best Practices for Webhook Providers

Provide amazing documentation: in webhook communications, the authentication, message integrity validation, and replay prevention validation happen on the webhook listener. Therefore, webhook providers must simplify the work on developers by providing documentation with:

• Complete specs on your webhook security, including the authentication mode, ciphersuite, signature procedure, and best practices for security.
• A detailed step by step description to validate your webhook requests.
• Sample code (not a pseudo-code) developers can run to understand how the webhook listeners should be implemented.
• Instructions on how developers can test your webhooks against their code running on localhost.
• A mechanism to trigger, test, and replay webhooks calls with a close feedback loop.

Secret keys

• Use asymmetric keys for non-repudiation.
• Use individualized secret keys per listener.
• Implement a key rotation with zero downtime (i.e. signing webhook messages with multiple keys) so webhook listeners can rotate keys regularly.
• Automate key rotation by implementing APIs listeners can call to fetch new keys as needed.

Encryption and hashing algorithm

• Do not use old ciphers or hashing algorithms such as SHA-1 and MD5.
• For HMAC, use at least SHA-256.

Signature Payload

• Concatenate body, timestamp, and sensitive headers to form the payload.
• Don't add unnecessary steps to the signature, such as changing the payload capitalization and adding line breaks.

Replay Prevention

• Use timestamp to provide a simple way to validate if requests were sent recently, and reduce the risk of message replays.
• When adopting timestamps, consider using Unix timestamp as format and provide a recommendation of how long messages should be considered valid in your documentation.
• Include the timestamp in the signature payload to reduce the risk of tampering.

Versioning:

• Add a version header to your webhook messages or the hash signature to provide forward compatibility.

Compensatory controls:

• Whenever possible, list the source IPs for your webhook messages to allow developers to implement IP restrictions.
• Provide a convenient way for listeners to verify the validity of a webhook call, such as including a webhook receipt URL/API within the webhook message.

‍

Best Practices for Webhook Listeners

In addition to using a good webhook provider and implementation, you can take the following steps to improve your webhook security:

• Use HTTPS on the listener with a strong cipher suite: First and foremost, do not operate in clear text (seriously, don't!). Most webhooks don't provide any control to encrypt your messages. This recommendation is even more critical in webhooks with Basic Authentication and Shared Secret. Use HTTPS with a strong cipher suite to mitigate the man-in-the-middle.
• Review the webhook documentation and ensure you're implementing all the security features offered: in webhook communications, the authentication, message integrity validation, and replay prevention validation happen on the webhook listener. Therefore, you should read, understand, and implement the security controls made available by your provider.
• Restrict who can send you webhook requests based on the IP: Some services — i.e. GitHub — provide a public list of IPs used to send webhook requests. Use this list to implement network restrictions and ensure only the webhook service is sending the notification. Note that is not possible if the webhook service does not provide a list of their origin IPs.
• Storing secrets: Credentials and shared secrets should be treated as confidential information and protected just as you protect private keys or passwords.
• Segmenting secrets: Some services — like DocuSign — will let you adopt a unique secret for each webhook listener. This allows you to segment secrets and reduce the likelihood of compromise.
• Rotating secrets: In addition to segmentation, you can also change secrets at a regular intervals or after important events — i.e. when an important employee leaves the organization — to mitigate risk if the secret is compromised.
• Use strong signature algorithms: Use the strongest hashing algorithms available with the webhook signature. HMAC-SHA256 is the most popularly used hash signature.
• Call back the service: An interesting method to validate if a webhook request is legitimate is to call the service that sent the webook from a different channel such as their REST API to confirm said event happened. Some services — i.e. Atlassian Jira — will include an API endpoint in the webhook body to facilitate the validation. However, this approach impacts performance and time to value.

Source Webhooks:

• https://webhooks.fyi/docs/webhook-directory

‍