Introducing: Redact

Introduction

Last night I was enjoying my last batch of Matte Leão, and my plan was to follow up on covering design patterns in Go as I previously did with the Decorator and the Strategy patterns. The idea was to write a post about the Pipeline Pattern. However, while I was looking for an example case I could use in the post, I found a project I started a while ago and never finished. Since Matte is a high-caffeine drink, and I love it and drink it in industrial quantities, I couldn’t sleep, got carried away and ended up completely refactoring the project using pipelines. While this project does not implement the pipeline pattern, as it doesn’t make use of connected channels and concurrency, it takes on the concept of pipelines with stages to redact sensitive information from logs generated by the log/slog package.

In this post, I’ll cover the implementation details of the redact package, and hopefully, you’ll find it useful for your projects, or at least get inspired by its design for your own ideas.

So, let’s get started! ☕🍃

Overview

The idea behind the redact package is to provide a way to redact sensitive information from logs generated by the log/slog package. It does so by creating a pipeline of redaction stages that can be configured to redact specific fields from the log records.

We’ll discuss the API in a moment, but the main idea is that each field we include will generate its own redaction stage (pipeline). The stages are executed sequentially, and at each stage the value—being the sensitive information we want to redact—is replaced by a predefined value, like a mask, in our case [REDACTED].

The result should look like this:

time=2025-02-08T19:49:23.826+02:00 level=INFO msg="Test log" user=rigoletto email=[REDACTED] password=[REDACTED]

Where the email and password fields we declared to be redacted.

For reference, the log record was generated by:

logger.Info("Test log", slog.String("user", "rigoletto"), slog.String("email", "foo@bar.qux"), slog.String("password", "abc123"))

There are a few considerable benefits of using the stage approach when working on a problem like this. In our case, a log message is a record that contains a set of fields with possible sensitive information we might want to obfuscate, where each field is processed on its own stage. This allows us to achieve better:

• Modularity: each stage in the pipeline encapsulates a specific redaction logic which makes the code more modular allowing better testability and maintainability as individual stages can me developed and debugged in isolation. In our code, an upcoming feature would be to allow custom redaction functions to be added to the pipeline.

• Flexibility and Extensibility: Pipelines allow for the dynamic addition, removal, or even reordering of stages without impacting the overall system. This flexibility facilitates easy updates and feature enhancements.

• Reusability: Individual pipeline stages can be reused across different pipelines or applications, promoting code reuse and reducing redundancy.

• Scalability: We could decide on run each stage concurrently with goroutines. This is not so interesting for our use case of log redaction, since we’re processing fields of a single record at the time, but it’s a good idea if for example we were fetching records from one place, doing the redaction, doing other kind of transformations, and then sending it to another place.

Design

The design of the redact package is quite simple. We start with a slog handler, we initialize a RedactionPipeline declaring the fields we want to redact and then we wrap everything in a RedactionHandler that will apply the redaction pipeline to the log records before passing them to the original handler.

package main

import (
	"log/slog"
	"os"

	"github.com/alesr/redact"
)

func main() {
	// Create a base handler
	handler := slog.NewTextHandler(os.Stdout, nil)

	// Create a redaction pipeline
	pipeline := redact.NewRedactionPipeline()
	pipeline.AddRedactField("email")
	pipeline.AddRedactField("password")

	// Create a RedactionHandler that wraps the original handler
	redactionHandler := redact.NewRedactionHandler(handler, pipeline)

	// Create a logger with the RedactionHandler
	logger := slog.New(redactionHandler)

	logger.Info("Test log", slog.String("user", "rigoletto"), slog.String("email", "foo@bar.qux"), slog.String("password", "abc123"))
}

You run the example here.

Implementation

I’ll focus on the most important parts of the implementation, but you can check the full code in the redact.

1. RedactionFunc and RedactionPipeline

// redactionFunc defines the signature for redaction functions.
type RedactionFunc func(ctx context.Context, r slog.Record) slog.Record

// RedactionPipeline manages a list of redaction stages.
type RedactionPipeline struct {
	mu     sync.RWMutex
	stages []RedactionFunc
}

The RedactionFunc type is a function that takes a context and a log record and returns a log record. This function is the building block of the redaction pipeline. The RedactionPipeline type is a struct that holds a list of redaction stages.

1. AddRedactField

I want to have a declarative way to add redaction stages to the pipeline that specifies the field to be redacted with the ability to add more fields later.

The important part here is that you need to be consistent with the field names you’re redacting. If you add a field to the pipeline that doesn’t exist in the log record, the redaction stage will not be applied.

We could go with a more dynamic approach, where we try to do some reflection or regex to look for the fields in the log record like anything looking like an email or password, but that would be a bit overkill for how I want to use this package.

To add a redaction stage for a specific field, we use the AddRedactField method. This method appends a new redaction stage to the pipeline that replaces the value of the specified field with a predefined value.

// AddRedactField adds a redaction stage for a specific field.
func (p *RedactionPipeline) AddRedactField(fieldName string) {
	p.mu.Lock()
	defer p.mu.Unlock()

	p.stages = append(p.stages, func(ctx context.Context, r slog.Record) slog.Record {
		var attrs []slog.Attr
		r.Attrs(func(a slog.Attr) bool {
			attrs = append(attrs, a)
			return true
		})

		newRecord := slog.NewRecord(r.Time, r.Level, r.Message, r.PC)

		for _, a := range attrs {
			if a.Key == fieldName {
				newRecord.AddAttrs(slog.String(fieldName, redactedValue))
			} else {
				newRecord.AddAttrs(a)
			}
		}
		return newRecord
	})
}

And the API is quite simple:

pipeline := redact.NewRedactionPipeline()
pipeline.AddRedactField("email")
pipeline.AddRedactField("password")

1. Process

The Process method simply applies all redaction stages to the log record.

// Process applies all redaction stages to the log record.
func (p *RedactionPipeline) Process(ctx context.Context, r slog.Record) slog.Record {
	p.mu.RLock()
	defer p.mu.RUnlock()
	for _, stage := range p.stages {
		r = stage(ctx, r)
	}
	return r
}

1. RedactionHandler

The last part is the RedactionHandler that wraps the original handler and applies the redaction pipeline to the log records before passing them to the original handler.

// RedactionHandler is a custom slog.Handler that applies a redaction pipeline.
type RedactionHandler struct {
	handler  slog.Handler
	pipeline *RedactionPipeline
}

Everything else you see related to the RedactionHandler is just the implementation of the slog.Handler interface.

Conclusion

I’m quite happy with how this project turned out. It’s a simple and lightweight package that can be used to redact sensitive information from logs generated by the slog package. The tests were easy to write, and the implementation was straightforward. I hope you find it useful for your projects, or at least get inspired by its design for your own ideas.

Today I ran out of Matte Leão, and I can’t find any more to buy here in Crete. I’m depending on my mom to send me a new shipment. If you happen to know a good dealer in Greece, or would like a few more features in the package, please let me know—I can trade you some code for some Matte. 😄