Load testing gotcha nr. 1: logging

This will be a first in, hopefully, a series of short but important things to keep in mind when load testing. So without further ado:

Load Testing Gotcha nr. 1

Logging is a good thing, but too much will kill you, your performance and ultimately your server. So use it wisely young Padawan.

Logging and certainly good contextual logging is always nice when things go wrong, but during load testing it should be turned off or as low as possible. The reasoning behind this is that logging, certainly on the more verbose levels will impose a hard to ignore load on your CPU/Memory and certainly on your disk(s).

This extra load will influence the request/response times of your tests. So when your log level is set too high and doesn't reflect your production settings, your load tests won't be representative and can't be reliably used too predict production server behavior under load.

As far as I can tell there are only two reasons to load test using a more verbose log level:

1. You're actually suspecting your logging of being a performance bottleneck and as such you want to do some load tests at different verbosity levels to study the impact of your logging.
2. Your application is failing at certain load levels and you need more detailed logging to home in on the cause of the failure.

So this is my (current) view on logging and load testing. I also think that this view isn't only applicable on load testing alone, but should also be kept in the back of your mind during application development. What I want to say with this is that you always have to think carefully about what to log on which logging level.

And one extra free piece of advice: never, ever, use XML as a logging format. I have encountered it once or twice in the wild as a consultant and it will use about 30% more disk space than non-XML logging (possibly even more depending on the length of your tags) and isn't exactly human-readable unless you're name is Neo and you're living in the Matrix.