Getting Started with Webhooks
From the earliest webhooks at Twilio to every major SaaS platform, webhooks have become a defacto standard in web development. We use them for everything from kicking off workflows to changing the flow and operation of our apps. In terms of use, it makes sense because webhooks serve to send messages between two systems while remaining agnostic to the underlying tech stacks or networks they’re on. When we look at the implementation of webhooks, they make sense because they’re simple HTTP requests with a predictable structure.
What Problem do Webhooks Solve?
The concept of passing messages between systems or components didn’t begin with webhooks. A shared communications channel is a core part of many systems all the way down to the deepers layers of an operating system. When we’re working within the OS on a given machine or even within a given application inside a data center, we can implement that in a variety of ways.
Unfortunately, as we move to distributed applications - or worse, globally distributed applications in a variety of technologies - suddenly our implementation choices shrink. We have to shift our thinking from specific tools or vendors to the underlying principles. Webhooks serve as a technology and vendor-agnostic approach to passing messages between applications over a commonly understood protocol (HTTP).
The Anatomy of a Webhook
Every webhook has three parts: the URL requested, the headers, and the body or payload. Let’s break down each separately.
First, the URL is the destination where the webhook is sent. This is the URL for your application or development environment you configure with your webhook provider. Generally, this is static and we only change this as we move from development to production.
Next, we have the headers. This is where things get more powerful and fun. At minimum, most webhooks will have a Content-Type header specifying application/json as a payload. Beyond validating the payload, this isn’t useful. More advanced webhook providers will use headers to track transaction information, establish sender credentials, or provide additional context on the payload. We’ll dive into this further in the Webhook Security section.
Finally, we have the payload which is the data the provider is passing to our application. The data can be as simple as a few fields or an entire object from the provider’s API. It varies by provider and possibly by use case so always explore their documentation.
Webhook Security
Too many developers don’t think about security for their webhook applications. They set their code to receive a request, take action, and hope for security through obscurity. While it’s easy in the short term, the consequences can be catastrophic:
- A thief could spoof Stripe webhooks to mark their order as PAID to trigger a shipment
- A spammer could spoof a Twilio webhook to bounce text messages to victims frustrating them and damaging your brand
- A hacker could spoof a Github webhook to phish your development team with bad links
With this in mind, all of these providers have ways to validate, verify, and have confidence that a request comes from them. In general, it’s easiest to think about this in three layers of how, who, and what or in technical terms: transport, authentication, and payload.
On the how or transport layer, we need to ensure that the webhook is protected as it travels from the provider to our application. If we use simple http, the request can be logged, analyzed, or snooped. Therefore, we must use https. With the ease of systems like Let’s Encrypt, there’s no excuse for http connections in any application.
On the who or authentication layer, our options depend on the provider:
Some providers will publish the IP addresses which broadcast their webhooks. Both Stripe and PagerDuty provide their IP lists in both their docs and linked json files while Github provides an API endpoint for the same. Since IP addresses change constantly, having these easily retrievable is key. Unfortunately, some providers don’t publish their list, so this is not a complete solution.
More uniquely, some providers - like Dropbox or Okta - do not send any webhooks at all until they validate the receiving application via a specific one-time challenge/response handshake. Once complete, they start sending webhooks. To be clear, this only validates the receiver’s identity, not the provider, so it mostly mitigates against typos, misconfigured applications, and similar. This alone does not protect your application from the scenarios above.
To protect against those scenarios, we need to validate the identity of the provider which is where our headers come back into play. Most providers implement authentication within the headers directly. In some cases - like with VMWare Workspace One or mParticle - this is a HTTP Basic Authentication (username and password) approach. You configure it on the provider side and each request includes the header which you validate. It’s quick, simple, and to the point.
Many other providers - such as Twilio or Shopify - combine the “who” and the “what” or the authentication and payload layers into a single step with HMAC (a hashing approach) using both a shared secret and the payload and inserting the resulting hash into a header. When your application receives the request, it combines the shared secret with the payload, generates the hash, and compares it with the one from the header. If it matches, this proves that both the request is from the provider (via the shared secret) and that the payload has not been modified. It is the best approach by far. Unfortunately, it is also the most complex and easy to make a mistake, so providers often provide helper functions within their SDKs.
Using Webhooks in my App
The most common question we get at this point is “which options should I use for my webhook app?” and the short answer is “every option your provider gives you.” Use https, filter traffic by IP address, validate the sender’s identity, and verify your payloads.
In most application development, our industry trends towards too little security instead of too much. Don’t make it easy for attackers. The “silly application” shared among friends today could easily become a target tomorrow.
