Kristian Lyngstøl's Blog

Varnish best practices

Posted on 2010-01-26

A while ago I wrote about common Varnish issues, and I think it's time for an updated version. This time, I've decided to include a few somewhat uncommon issues that, if set, can be difficult to spot or track down. A sort of pitfall-avoidance, if you will. I'll add a little summary with parameters and such at the end.

1. Run Varnish on a 64 bit operating system

Varnish works on 32-bit, but was designed for 64bit. It's all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you're somewhat on your own. However, try to fit it within 2GB. I wouldn't recommend a cache larger than 1GB, and no more than a few hundred threads... (Why are you on 32bit again?)

2. Watch /var/log/syslog

Varnish is flexible, and has a relatively robust architecture. If a Varnish worker thread was to do something Bad and Varnish noticed, an assert would be triggered, Varnish would shut down and the management process would start it up again almost instantly. This is logged. If it wasn't, there's a decent chance you wouldn't notice, since the downtime is often sub-second. However, your cache is emptied. We've had several customers contact us about performance-issues, only to realize they're essentially restarting Varnish several times per minute.

This might make it sound like Varnish is unstable: It's not. But there are bugs, and I happen to see a lot of them, since that's my job.

An extra note: On Debian-based systems, /var/log/messages and /var/log/syslog is not the same. Varnish will log the restart in /var/log/messages but the actual assert error is only found in /var/log/syslog, so make sure you look there too.

The best way to deal with assert errors is to search our bug tracker for the relevant function-name.

3. Threads

The default values for threads is based on a philosophy I've since come to realize isn't optimal. The idea was to minimize the memory footprint of Varnish. So by default, Varnish uses 5 threads per thread pool. By default, that's 10 threads minimum. The maximum is far higher, but in reality, threads are fairly cheap. If you expect to handle 500 concurrent requests, tune Varnish for that.

A little clarification on the thread-parameters: thread_pool_min is the minimum number of threads for each thread pool. thread_pool_max is the maximum total number of threads. That means the values are not on the same scale. The thread_pools parameter can safely be ignored (tests have indicated that it doesn't matter as much as we thought), but ideally having one thread_pool for each cpu core is the rule of thumb, if you want to modify it.

You also do not want more than 5000 as the thread_pool_max. It's dangerous, though fixed in trunk. It's also more often than not an indication that something else is wrong. If you find yourself using 5000 threads, the solution is to find out why it's happening, not to increase the number of threads.

To reduce the startup time, you also want to reduce the thread_pool_add_delay parameter. '2' is a good value (as opposed to 20 which makes for a slow start).

4. Tune based on necessity

I often look at sites where someone has tried to tune Varnish to get the most out of it, but taken it a bit too far. After working with Varnish I've realized that you do not really need to tune Varnish much: The defaults are tuned. The only real exception I've found to this is number of threads and possibly work spaces.

Varnish is - by default - tuned for high performance on the vast majority of real-life production sites. And it scales well, in most directions. By default. Do yourself a favor and don't fix a problem which isn't there. Of all the issues I've dealt with on Varnish, the vast majority have been related to finding out the real problem and either using Varnish to work around it, or fix it on the related system. Off the top of my head, I can really only remember one or two cases where Varnish itself has been the problem with regards to performance.

To be more specific: - Do not modify lru_interval. I often see the value "3600". Which is a 180 000% (one hundred and eighty thousand percent) increase from the default. This is downright dangerous if you suddenly need the lru-list, and so far my tests haven't been able to prove any noticeable performance improvement. - Setting sess_timeout to a higher value increase your filedescriptor consumption. There's little to gain by doing it too. You risk running out of file descriptors. At least until we can get the fix into a released version.

So the rule of thumb is: Adjust your threads, then leave the rest until you see a reason to change it.

5. Pay attention to work spaces

To avoid locking, Varnish allocates a chump of memory to each thread, session and object. While keeping the object workspace small is a good thing to reduce the memory footprint (this has been improved vastly in trunk), sometimes the session workspace is a bit too small, specially when ESI is in use. The default sess_workspace is 16kB, but I know we have customers running with 5MB sess_workspace without trouble. We're obviously looking to fix this, but so far it seems that having some extra sess_workspace isn't that bad. The way to tell is by asserts (unfortunately), typically something related to "(p != NULL) Condition not true" (though there can obviously be other reasons for that). Look for it in our bug report, then try to increase the session workspace.

6. Keep your VCL simple

Most of your VCL-work should be focused around vcl_recv and vcl_fetch. That's where you define the majority of your caching policies. If that's where you do your work, you're fairly safe.

If you want to add extra headers, do it in vcl_deliver. Adding a header in vcl_hit is not safe. You can use the "obj.hits" variable in vcl_deliver to determine if it was a cache hit or not.

You should also review the default vcl, and if you can, let Varnish fall through to it. When you define your VCL, Varnish appends the default VCL, but if you terminate a function, the default is never run. This is an important detail in vcl_recv, where requests with cookies or Authroization-headers are passed if present. That's far safer than forcing a lookup. The default vcl_recv code also ensures that only GET and HEAD-requests go through the cache.

Focus on caching policy and remember that the default VCL is appended to your own VCL - and use it.

7. Choosing storage backend (malloc or file?)

If you can contain your cache in memory, use malloc. If you have 32GB of physical memory, using -smalloc,30G is a good choice. The size you specify is for the cache, and does not include session workspace and such, that's why you don't want to specify -smalloc,32G on a 32GB-system.

If you can not contain your cache in memory, first consider if you really need that big of a cache. Then consider buying more memory. Then sleep on it. Then, if you still think you need to use disk, use -sfile. On Linux, -sfile performs far better than -smalloc once you start hitting disk. We're talking pie-chart-material. You should also make sure the filesystem is mounted with noatime, though it shouldn't be necessary. On Linux, my cold-hit tests (a cold hit being a cache hit that has to be read from disk, as opposed to a hot hit which is read from memory) take about 6000 seconds to run on -smalloc, while it takes 4000 seconds on -sfile with the same hardware.  Consistently. However, your milage may vary with things such as kernel version, so test both anyway. My tests are easy enough: Run httperf through x-thousand urls in order. Then do it again in the same order.

Some of the most challenging setups we work with are disk-intensive setups, so try to avoid it. SSD is a relatively cheap way to buy yourself out of disk-issues though.

8. Use packages and supplied scripts

While it may seem easier to just write your own script and/or install from source, it rarely pays off in the long run. Varnish usually run on machines where downtime has to be planned, and you don't want a surprise when you upgrade it. Nor do you want to risk missing that little bug we realized was a problem on your distro but not others. If you do insist on running home-brew, make sure you at least get the ulimit-commands from the startup scripts.

This is really something you want regardless of what sort of software you run, though.

9. Firewall and sysctl-tuning

Do not set "tw_reuse" to 1 (sysctl). It will work perfectly fine for everyone. Except thousands of people behind various NAT-based firewalls. And it's a pain to track down. Unfortunately, this has been an advice in the past.

Avoid connection-tracking on the Varnish server too. If you need it, you'll need to tune it for high performance, but the best approach is simply to not do connection-tracking on a server with potentially thousands of new connections per second.

10. Service agreements

(Service agreements are partly responsible for my salary, so with that "conflict of interest" in mind....)

You do not need a service agreement to run Varnish. It's free software.

However, if you intend to run Varnish and your site is business critical, it's sound financial advice to invest some money in it. We are the best at finding potential problems with your Varnish-setup before they occur, and solving them fast when they do occur.

We typically start out by doing a quick sanity-test of your configuration. This is something we can do fast, both with regards to parameters, VCL and system configuration. Some of our customers only contact us when there's something horribly wrong, others more frequently to sanity-check their plans or check up on how to use varnisncsa for their particular logging tool and so on. It's all up to you.

We also have a public bug tracker anyone can access and submit to. We do not have a private bug tracker, though there are bugs that never hit the public bug tracker - but that's because we fix them immediately. Just like any other free software project, really. We have several public mailing lists, and we answer them to the best of our ability, but there is no guarantee and our time is far more limited. If you run into a bug, my work on other bugs will be postponed until your problems are solved. Better yet: if you run into something you don't know is a bug, we can track it down.

A service agreement gives you saftey. And your needs will get priority when we decide where we want to take Varnish in the future.

We also offer training on Varnish, if you prefer not to rely on outside competence.

Oh, and I get to eat. Yum.

Summary

Keep it simple and clean. Do not use connection tracking or tw_reuse. Try to fit your cache into memory on a 64-bit system.

Watch your logs.

Parameters:

thread_pool_add_delay=2
thread_pools = <Number of cpu cores>
thread_pool_min = <800/number of cpu cores>
thread_pool_max = 4000
session_linger = 50
sess_workspace = <16k to 5m>

So if you have a dual quad core CPU, you would have 8 cpu cores. This would make sense: thread_pools=8, thread_pool_min=100, thread_pool_max=4000. The number 800 is semi random: it seems to cover most use-cases. I addedd session_linger into the mix because it's a default in Varnish 2.0.5 and 2.0.6 but not in prior versions, and it makes good sense.