A quick reminder on how Varnish bans objects

Varnish bans objects by adding an expression to a list, and then when an object is "hit", it is tested against all expressions on the list since the last time it was hit. So only when an object is hit can it be evaluated.

You can ban on both req.* and obj.* items. When you ban in VCL, there are THREE contexts involved:

1. The VCL context that puts the ban on the ban-list.
2. The context of the next request that hits the object
3. The context of the object

This means that a purge for:

purge ("req.url == " req.url " && obj.http.magic == " req.http.purge-magic)

all three purges are used. The "req.url" outside the quotations marks (the second one) is in the context of the VCL that triggered this purge() function. The "req.url" inside the quotation marks is in the context of the request that will hit the object next - whatever that may be. The "obj.http.magic" variable will be in the context of the actual object on the cache, while the last "req.http.purge-magic" is in the current vcl-context.

If that was confusing, it's ok. If it wasn't confusing, I'm impressed.

Enter the ban-lurker

So the problem with Varnish bans is that not all objects are hit constantly. In fact, some objects are rarely ever hit. And they will linger even if they are banned. To solve this, the ban-lurker was created. Put simply, the ban-lurker will check older objects against the current ban-list to see if they are worth keeping around. This serves two functions: 1. Banned objects can be discarded. 2. The size ban-list can be reduced.

Now, the problem with the ban-lurker is that it doesn't stem from a request. In other words, it doesn't HAVE a "req." structure. It doesn't have a URL. So if your bans are for ("req.url == " req.url), the ban lurker can't help you, because it doesn't have a req.url to test against. It can ONLY test against obj.

As most bans are for req.url, this means that the ban-lurker isn't very useful out-of-the-box. Unless you adapt your approach to bans a bit.

Writing ban-lurker-friendly bans

To utilize the ban-lurker, there are two things you need to do:

1. Turn it on with "param.set ban_lurker_sleep 0.1" (for example. Any non-zero value enables it)
2. Do not use req.* in your bans.

Now, how to avoid using req? Well, the simples method is probably to store req.url on the object, and ban on that. Example:

sub vcl_fetch {
set obj.http.x-url = req.url;
}
sub vcl_deliver {
unset resp.http.x-url; # Optional
}
sub vcl_recv {
if (req.request == "PURGE") {
if (client.ip !~ purgers) {
error 401 "Not allowed";
}
purge("obj.http.x-url ~ " req.url); # Assumes req.url is a regex. This might be a bit too simple
}
}

The above code-snippet simply stores the URL on the object as a HTTP header called x-url, then uses that instead of req.url to ban. It also removes the x-url header before it's returned to the clients. It may not look as intuitive (... because "req.url == " req.url is clearly intuitive), but it allows you to take advantage of the ban-lurker.

Conclusion

We (Varnish community (http://www.varnish-cache.org)) don't know how useful the ban-lurker will prove to be, nor do we know what the best grace period for the ban_lurker_sleep parameter is. This is all fairly new, and not used much. We're confident in it's stability, but not the benefit.

However, it's clear that if you have a possibly large data-set and you do frequent purges, using the ban-lurker can help keep the size of the ban-list within manageable limits, as well as help you throw banned content out faster. We can't really see the downside of it.

However, using the ban-lurker without adjusting your approach to actual bans is pointless. It wont be able to do anything except wake up every second and discover it can't do anything.

I hope this helped. Happing banning!