2018/07/11 by Willem Stam.
Postmastery supports SendGrid clients with template for multiple webhook endpoints
Postmastery provides delivery monitoring and optimisation services on top of various email delivery environments. Sometimes we offer this service on top of on-premises MTA (like PowerMTA or EmailSuccess) and sometimes on top of cloud-based ESP’s (like Mailchimp or Smartfocus). A prerequisite of delivery monitoring and optimization services, is to get the right data points like SMTP-data, blacklist data, DMARC-data or FBL-data in place. In some cases it is difficult to get the SMTP data for our clients who use SendGrid. Not so much because they do not provide the data, but because SendGrid only supports one endpoint which is often already in use for retrieving bounce data by the users. We, Postmastery, need at least one other endpoint to retrieve similar data as well. For this reason, Postmastery has decided to develop a template to deploy a webhook forwarder on Google Cloud Platform.
Why Google Cloud platform?
Developers at Postmastery studied various webhook forwarder options. New ‘serverless’ cloud products are ideal for this. They rejected Amazon Lambda, because it can’t handle messages larger than 256K and it can’t connect to SQS to queue messages. By using Google Cloud Functions and Google Cloud Pub/Sub, Postmastery was quickly able to create a forwarder. It is documented on Github.
How does it work?
As mentioned, the template leverages Google Cloud Functions and Google Cloud Pub/Sub to provide scalability, reliability, and separation with minimal code.
The endpoint function receives HTTPS POST requests from a webhook and publishes the messages to a Pub/Sub topic. The subscriber function is subscribed to the Pub/Sub topic and forwards the messages to a specific URL. Multiple subscriber functions with different URLs can be deployed, which all get the same messages.
Through Pub/Sub the endpoint is decoupled from the subscribers. As soon as the message is saved in the queue, incoming messages are immediately acknowledged. If forwarding to a receiver fails, the message remains in the queue and the subscriber function is retried.