Fingerprinting Knock errors for Sentry in Elixir



At SwayDM, we use the fantastic Knock platform to define notification workflows, and deliver emails, and SMSs to our users.

Our codebase is written in Elixir, and we use Sentry for error reporting via sentry-elixir.  One thing you can do with sentry-elixir is report an event to Sentry every time your app logs at the error level.  You can also specify which metadata logger keys are forwarded to sentry as metadata attributes.

This works great in conjunction with Knock, but if you only log the error, all your Knock related issues in Sentry will be grouped as one, regardless of which identify, or workflow caused them.   This is where custom fingerprinting comes in.

Configure Sentry

First, configure sentry to define a module, and function to be invoked before the event is sent to Sentry.

You must have the Sentry LoggerBackend configured.  If you want to pass logger metadata to sentry, you must explicitly add it to this configuration.



Capture Logger Metadata

Capture details in the logger metadata before logging the error.  These functions are defined in a module that implements a Behavior named SwayDM.Notifications.Provider so we can easily stub or mock out notification calls.

We currently only make calls to two different knock resources.  If we were calling more, we could probably make a nice wrapper, or more generic function that could handle and log all knock responses.

Calls to Knock.Users.identify


Calls to Knock.Workflows.trigger




Create before_send Function

Next, create a before_send function in your new Sentry module.   We use pattern matching on function heads to determine which before_send function is invoked.   

The before_send function takes in a sentry event, and returns a sentry event.  You can make any modifications you see fit to the event.  This is great for when you want to grab more info out of an error, or log message, especially if it is supplied via logger metadata or buried in an error message. 

It is also a good method to redact sensitive information that may be accidentally sent to Sentry. I've seen cases where a match error gets inspected, and details of the matched data structures gets sent to sentry. This can compromise PII, or other secrets if care is not taken.

Creating the Fingerprint

In this case, we are supplying a custom event message, and fingerprint.  The fingerprint is just a list of strings.  The order needs to be consistent, as it is hashed by sentry to group events efficiently.

We are setting an initial namespace of ["elixir_app", "knock"].  This is just a safeguard to know our fingerprint will not accidentally interfere with other events (very unlikely, but a good practice).  Next, we supply [workflow, code], which are pulled from the metadata, and knock error.   

Now, rather than all Knock errors being grouped together, they are grouped by the combination of workflow and code.  You could group by other attributes though if desired.


How events how show up in Sentry issue list:

How the event details look:

You can see our custom message, and the nicely formatted error.  The message was overwritten in our before_send function, and the nicely formatted logger_metadata values are due to the Logger.error invocation, and Sentry.LoggerBackend configuration.


Sentry Fingerprint:

Finally, you can see the event fingerprint in the Event Grouping Information on the Sentry event.



Sentry and Knock have saved me huge amounts of development, and debugging time.  I highly recommend both platforms.  If you have any questions, just ask!

- Andy Glassman


Comments

Post a Comment

Popular posts from this blog

Phoenix LiveView: Async Assign Pattern

Image
I've been using LiveView for about two years now. It's a great framework that makes snappy and responsive pages. One anti-pattern I see fairly often is loading a lot of data in the initial page render. For the un-initiated, the mount/3 function is called twice.  Once for the initial 'dead' render, and again after the socket is connected.   Many times, for the sake of simple straight forward code, not much is done differently between these two renders. I haven't found any references (I'm sure they exist) on a best practice for managing the following flow: Set sensible, lightweight default values on initial render. Kick off one or more async processes to make longer running function calls. Receive the values in the LiveView, and update the assigns. As always in Elixir, the tools are powerful, and theres many ways to accomplish this. Spawn a linked, or unlinked process. Start a supervised, or unsupervised Task. Make a call to a GenServer, and have it send a mess
Read more

Write Admin Tools From Day One

Image
UPDATE:  I gave a talk expanding on this blog post at EMPEX NYC 2023 . The Problem Writing useful features for your users is key to a successful product.  It makes sense then that you should maximize your time writing features for those users.   This approach is very effective at the beginning of a product's life, but over time, you may find you are spending more time maintaining the product that developing it. Some of the biggest development time sucks can be: Tracking down the cause of unexpected behavior. Fixing data. Answering user, or team member questions. Running maintenance scripts, or other ad-hoc tasks. The "Oh #&@$" Solution These small support tasks start to add up, and there is no easy way to offload the work to product, or support.  You and your development team reach a breaking point where you must start prioritizing administrative tools. Unfortunately, you will slow down your new feature development even more in the short term.  It will take time for t
Read more

Custom Kaffy Styling

Image
I recently gave a talk at Empex NYC ( link to talk ) on why you should be writing admin tools from day one .  In my talk, one thing I highly suggest is to make it very obvious what environment you are in when providing a web admin UI for your team.   We've been using Kaffy at SwayDM since the beginning.  At the time, you couldn't easily overwrite the styling.  I maintain a private fork of Kaffy for changes I've made, which included the ability to change the background color. I recently merged the past few minor Kaffy releases back into my fork, and saw that Kaffy now has the ability to provide Extensions which allow you to inject custom html, css and javascript.  I spent a little time making a custom CSS file to change the color of the Kaffy nav, and add a title which notes which environment you are in. For each environment, I have a static file for CSS styling overrides. kaffy-ext-local.css kaffy-ext-staging.css kaffy-ext-production.css defmodule SwayDMWeb.Admin.Kaffy
Read more