Magnus' blog


Debugging in production (Spring Boot)

I never really understood the debug log level until recently, when I discovered it could be toggled on a specific class instead of on the entire system. Effectively, you can change the log level directly in production for a specific class. This is incredibly useful when you have a bug that cannot easily be reproduced in other environments. It is also very simple in Spring with a built-in actuator.

Enabling the actuator

The only dependency needed is the actuator starter:

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-actuator")
}

The loggers endpoint is enabled by default, but it is not exposed over HTTP. Only health is exposed out of the box, so it has to be added explicitly:

management:
  endpoints:
    web:
      exposure:
        include: health,loggers

Since the endpoint can change the behaviour of a running system, it should not be reachable by just anyone. I lock it down with Spring Security, matching the endpoint with /actuator and only allowing authenticated users with a specific role to access it.

Changing the log level

With the endpoint exposed, the current level of any logger can be inspected with a GET on the fully qualified class name:

curl http://localhost/actuator/loggers/com.example.orders.OrderService
{
  "configuredLevel": null,
  "effectiveLevel": "INFO"
}

configuredLevel is null because nothing has been set on this specific class, so it inherits INFO from its parent. To turn on debug for just that class, POST the new level to the same URL:

curl -X POST http://localhost/actuator/loggers/com.example.orders.OrderService \
  -H "Content-Type: application/json" \
  -d '{"configuredLevel": "DEBUG"}'

It takes effect immediately, with no restart and no redeploy. Everything else keeps logging at INFO, so the log volume stays manageable while the one class I care about tells me everything. When I am done, the level is reset by posting null, which makes the class inherit from its parent again:

curl -X POST http://localhost/actuator/loggers/com.example.orders.OrderService \
  -H "Content-Type: application/json" \
  -d '{"configuredLevel": null}'

Two things are worth knowing. The change is only held in memory, so a restart puts the level back to whatever the configuration says. And the change only applies to the instance that received the request, so with multiple instances behind a load balancer, each one has to be called, or the request has to be routed to the specific instance where the bug is happening.

Grouping loggers

A bug rarely lives in a single class. Usually I want to see the whole flow through a feature, which spans a handful of classes across a few packages. Spring Boot has log groups for exactly this. A group is a named list of loggers that can be configured as one:

logging:
  group:
    orders: com.example.orders,com.example.payments,com.example.shipping

Now orders can be used anywhere a logger name is expected, both in the configuration and in the actuator:

curl -X POST http://localhost/actuator/loggers/orders \
  -H "Content-Type: application/json" \
  -d '{"configuredLevel": "DEBUG"}'

This sets every member of the group to DEBUG in one call, and posting null resets all of them at once. A GET on /actuator/loggers lists the groups along with their members, so it is easy to see what is currently turned on. Spring Boot also ships with two predefined groups, web and sql, which cover the web stack and the data-access stack, respectively. They are a good first thing to switch on when a request is behaving strangely and the cause is not yet known.

Takeaway

The debug level finally has a clear purpose for me: it is the detail that stays silent until a production bug needs it. I will be writing a lot more debug log statements going forward to ensure production can easily be debugged.

View next or previous post: