Kristian Lyngstøl's Blog

Varnish Wishlist

Posted on 2015-09-19

I recently went back to working for Redpill Linpro, and thus started working with Varnish again, after being on the side lines for a few years.

I've been using Varnish since 2008. And a bit more than just using it too. There's been a lot of great change over time, but there are still things missing. I recently read http://kacper.blog.redpill-linpro.com/archives/743 and while I largely agree with Kacper, I think some of the bigger issues are missing from the list.

So here's my attempt to add to the debate.

TLS/SSL

Varnish needs TLS/SSL.

It's the elephant in the room that nobody wants to talk about.

The world is not the same as it was in 2006. Varnish is used for more and more sensitive sites. A larger percentage of Varnish installations now have some sort of TLS/SSL termination attached to it.

TLS/SSL has been a controversial issue in the history of Varnish Cache, with PHK (Principal architect of Varnish Cache - https://www.varnish-cache.org/docs/trunk/phk/ssl.html) being an outspoken opponent of adding TLS in Varnish. There are valid reasons, and heartbleed has most certainly proven many of PHK's grievances right. But what does that matter when we use TLS/SSL anyway? It's already in the stack, we're just closing our eyes to it.

Setting up nginx in front of Varnish to get TLS/SSL, then nginx behind Varnish to get TLS/SSL... That's just silly. Why not just use nginx to cache then? The lack of TLS/SSL in Varnish is a great advertisement for nginx.

There are a lot of things I dislike about TLS/SSL, but we need it anyway. There's the hitch project (http://hitch-tls.org), but it's not really enough. We also need TLS/SSL to the backends, and a tunnel-based solution isn't enough. How would you do smart load balancing through that? If we don't add TLS/SSL, we might as well just forget about backend directors all together. And it has to be an integral part of all backends.

We can't have a situation where some backend directors support TLS/SSL and some don't.

Varnish Software is already selling this through Varnish Cache Plus, their proprietary version of Varnish. That is obviously because it's a deal breaker in a lot of situations. The same goes for basically any serious commercial actor out there.

So we need TLS/SSL. And we need it ASAP.

Note

After speaking to PHK, let me clarify: He's not against adding support for TLS, but adding TLS itself. Varnish now supports the PROXY-protool which is added explicitly to improve support for TLS termination. Further such additions would likely be acceptable, always doing the TLS outside of Varnish.

Better procedures for VCL changes

With every Varnish version, VCL (The configuration language for Varnish) changes either a little bit, or a lot. Some of these changes are unavoidable due to internal Varnish changes. Some changes are to tweak the language to be more accurate (e.g. changing req.request to req.method, to reflect that it's the request method).

If Varnish is part of your day-to-day work, then this might not be a huge deal. You probably keep up-to-date on what's going on with Varnish anyway. But most users aren't there. We want Varnish to be a natural part of your stack, not a special thing that requires a "varnish-admin".

This isn't necessarily an easy problem to solve. We want to be able to improve VCL and get rid of old mistakes (e.g., changing req.request to req.method is a good thing for VCL). We've also changed the way to do error messages (or custom varnish-generated messages) numerous times. And how to create hitpass objects (a complicated aspect of any cache).

A few simple suggestions:

  • All VCL changes reviewed in public as a whole before the release process even starts. To avoid having to change it again two versions down the line.
  • Backward compatibility when possible. With warnings or even requiring an extra option to allow it. E.g.: req.request could easily still work, there's no conflict there. Never for forever, but perhaps to the end of a major version. Not everything will be backwards compatible, but some can.

I've had numerous complaints from highly skilled sysadmins who are frustrated by this aspect of Varnish. They just don't want to upgrade because they have to do what feels like arbitrary VCL changes every single time. Let's see if we can at least REDUCE that.

Documentation?

There's a lot of documentation for Varnish, but there's also a lot of bad documentation. Some issues:

  • People Google and end up on random versions on varnish-cache.org. No, telling people "but there's a version right there so it's your own fault!" is not an acceptable solution. Varnish Software them self recently had a link in their Varnish Book where they used a link to "trunk" instead of "4.0", whereupon the "here is a complete list of changes between Varnish 3 and Varnish 4" link was actually a link to changes betwen Varnish 4.0 and the next version of Varnish.

  • "user guide" and "tutorial" and "installation"? Kill at least two and leave the others for blog posts or whatever. Hard enough to maintain one with decent quality.

  • Generated documentation needs to be improved. Example:

    Prototype
            STRING fileread(PRIV_CALL, STRING)
    Description
            Reads a file and returns a string with the content. Please
            note that it is not recommended to send variables to this
            function the caching in the function doesn't take
            this into account. Also, files are not re-read.
    Example
            set beresp.http.served-by = std.fileread("/etc/hostname");
    

    PRIV_CALL should clearly not be exposed! Other examples are easy enough to find.

    In addition, the Description is a mixture of reference documentation style and elaboration. Reference documentation should be clearly separated from analysis of consequences so technical users don't have to reverse-engineer a sentence of "don't do this because X" to figure out what the code actually does.

    And where are the details? What happens if the file can't be opened? What are the memory constraints? It says it returns the content of the file as a string, but what happens with binary content? There's clearly some caching of the file, but how does that work? Per session? Per VCL? Does that cache persist when you do varnishadm stop; varnishadm start? That's completely left out.

  • Rants mixed in with documentation? Get rid of "doc/shpinx/phk" (https://www.varnish-cache.org/docs/4.0/phk/) and instead reference it somewhere else. Varnish-cache.org/doc should not be a weird blog-space. It clutters the documentation space. Varnish is not a small little project any more, it's grown past this.

VMOD packages

Varnish vmods are awesome. You can design some truly neat solutions using Open Source vmods, or proprietary ones.

But there are no even semi-official package repositories for the open source vmods. Varnish Software offers this to customers, but I really want it for the public too. Both for my own needs, and because it's important to improve Varnish and VMOD adaption.

Until you can do "apt-get install varnish-vmod-foo" or something like that, VMODS will not get the attention they deserve.

There are some projects in the works here, though, so stay tuned.

TLS/SSL

In case you missed it, I want TLS/SSL.

I want to be able to type https://<varnish host>

BTW: Regarding terminology, I decided to go with "TLS/SSL" instead of either "SSL" or "TLS" after some feedback. I suppose "TLS" is correct, but "SSL" is more recognized, whether we like it or not.