Tracer: Distributed system tracing

Tracer noun, /ˈtɹeɪsɚ/: A round of ammunition that contains a flammable substance that produces a visible trail when fired in the dark.

Tracer is a library that manages custom trace identifiers and carries them through distributed systems. Traditionally traces are transported as custom HTTP headers, usually generated by clients on the very first request. Traces are added to any subsequent requests and responses, especially to transitive dependencies. Having a consistent trace across different services in a system allows to correlate requests and responses beyond the traditional single client-server communication.

This library historically originates from a closed-source implementation called Flow-ID. The goal was to create a clean open source version in which we could get rid of all the drawbacks of the old implementation, e.g. strong-coupling to internal libraries, single hard-coded header and limited testability.

Status: Under development and used in production

Features

Tracing of HTTP requests and responses

Customization by having a pluggable trace format and lifecycle listeners for easy integration

Support for Servlet containers, Apache’s HTTP client, Hystrix, JUnit, AspectJ and (via its elegant API) several other frameworks

Convenient Spring Boot Auto Configuration

Sensible defaults

Dependencies

Java 8

Any build tool using Maven Central, or direct download

Servlet Container (optional)

Apache HTTP Client (optional)

Hystrix (optional)

AspectJ (optional)

JUnit (optional)

Spring Boot (optional)

Installation

Selectively add the following dependencies to your project:

org.zalando

tracer-core

${tracer.version}

org.zalando

tracer-servlet

${tracer.version}

org.zalando

tracer-httpclient

${tracer.version}

org.zalando

tracer-hystrix

${tracer.version}

org.zalando

tracer-aspectj

${tracer.version}

org.zalando

tracer-junit

${tracer.version}

test

org.zalando

tracer-spring-boot-starter

${tracer.version}

Usage

After adding the dependency, create a Tracer and specify the name of the traces you want to manage:

Tracer tracer = Tracer.create("X-Trace-ID");

If you need access to the current trace's value, call getValue() on it:

Trace trace = tracer.get("X-Trace-ID"); // this is a live-view that can be a shared as a singleton

entity.setLastModifiedBy(trace.getValue());

By default there can only be one trace active at a time. Sometimes it can be useful to stack traces:

Tracer tracer = Tracer.builder()

.stacked(true)

.trace("X-Trace-ID")

.build();

Generators

When starting a new trace, Tracer will create trace value by means of pre-configured generator. You can override this on a per-trace level by adding the following to your setup:

Tracer tracer = Tracer.builder()

.trace("X-Trace-ID", new CustomGenerator())

.build();

There are several generator implementations included.

UUIDGenerator

This is the default generator implementation. It creates 36 characters long, random-based UUID string as a trace value.

FlowIDGenerator

This generator was created for historical reasons. It basically renders a UUID as a base64-encoded byte array, e.g. REcCvlqMSReeo7adheiYFA. The length of generated value is 22 characters.

PhraseGenerator

Phrase generator provides over 10^9 different phrases like:

tender_goodall_likes_evil_panini

nostalgic_boyd_helps_agitated_noyce

pensive_allen_tells_fervent_einstein

The generated phrase is 22 to 61 character long.

Listeners

For some use cases, e.g. integration with other frameworks and libraries, it might be useful to register a listener that gets notified every time a trace is either started or stopped.

Tracer tracer = Tracer.builder()

.trace("X-Trace-ID")

.listener(new CustomTraceListener())

.build();

Tracer comes with a very useful listener by default, the MDCTraceListener:

Tracer tracer = Tracer.builder()

.trace("X-Trace-ID")

.listener(new MDCTraceListener())

.build();

It allows you to add the trace id to every log line:

Another built-in listener is the LoggingTraceListener which logs the start and end of every trace.

Servlet

On the server side is a single filter that you must be register in your filter chain. Make sure it runs very early — otherwise you might miss some crucial information when debugging.

You have to register the TracerFilter as a Filter in your filter chain:

context.addFilter("TracerFilter", new TracerFilter(tracer))

.addMappingForUrlPatterns(EnumSet.of(REQUEST, ASYNC, ERROR), true, "/*");

Apache HTTP Client

Many client-side HTTP libraries on the JVM use the Apache HTTPClient, which is why Tracer comes with a request interceptor:

DefaultHttpClient client = new DefaultHttpClient();

client.addRequestInterceptor(new TracerHttpRequestInterceptor(tracer));

Hystrix

Tracer comes with built-in Hystrix support in form of a custom HystrixConcurrencyStrategy:

final HystrixPlugins plugins = HystrixPlugins.getInstance();

final HystrixConcurrencyStrategy delegate = HystrixConcurrencyStrategyDefault.getInstance(); // or another

plugins.registerConcurrencyStrategy(new TracerConcurrencyStrategy(tracer, delegate));

AspectJ

For background jobs and tests, you can use the built-in aspect:

@Traced

public void performBackgroundJob() {

// do work

}

or you can manage the lifecycle yourself:

tracer.start();

try {

// do work

} finally {

tracer.stop();

}

JUnit

Tracer comes with a special TestRule that manages traces for every test run for you:

@Rule

public final TracerRule tracing = new TracerRule(tracer);

It also supports JSR-330-compliant dependency injection frameworks:

@Inject

@Rule

public final TracerRule tracing;

Spring Boot Starter

Tracer comes with a convenient auto configuration for Spring Boot users that sets up aspect, servlet filter and MDC support automatically with sensible defaults:

Configuration

Description

Default

tracer.stacked

Enables stacking of traces

false

tracer.aspect.enabled

true

tracer.async.enabled

Enables for asynchronous tasks, i.e. @Async

true

tracer.filter.enabled

true

tracer.logging.enabled

false

tracer.logging.category

Changes the category of the LoggingTraceListener

org.zalando.tracer.Tracer

tracer.mdc.enabled

true

tracer.scheduling.enabled

Enables support for Task Scheduling, i.e. @Scheduled

true

tracer.scheduling.pool-size

Configures the thread pool size, i.e. the number of scheduled tasks

# of CPUs

tracer.traces

Configures actual traces, mapping from name to generator type (uuid, flow-id or phrase)

tracer:

aspect.enabled: true

async.enabled: true

filter.enabled: true

logging:

enabled: false

category: org.zalando.tracer.Tracer

mdc.enabled: true

scheduling.enabled: true

traces:

X-Trace-ID: uuid

X-Flow-ID: flow-id

The TracerAutoConfiguration will automatically pick up any TraceListener bound in the application context.

Getting Help with Tracer

If you have questions, concerns, bug reports, etc., please file an issue in this repository's Issue Tracker.

Getting Involved/Contributing

To contribute, simply make a pull request and add a brief description (1-2 sentences) of your addition or change. For more details, check the contribution guidelines.

Alternatives

Tracer, by design, does not provide sampling, metrics or annotations. Neither does it use the semantics of spans as most of the following projects do. If you require any of these, you're highly encouraged to try them.