Playing with swap on Linux

Posted on 2009-02-22

Swap is a funny thing.

I've got 4GB of memory on my desktop at home, and I've been experimenting lately with different swap setups. For the longest time, I didn't really run any swap on desktop setups with a reasonable  amount of memory, but once I installed this one, I went with the default 1.5GB that Ubuntu gave me.

Most of the time, it's not in use. However, I ran in to a funny bug with MPlayer which, combined with my 800MB cache, resulted in 3x800MB mplayers lingering in the background. When you add Firefox and three virtual machines into the mix, you've got yourself a swapping machine.

So I got curious. I know that with my setup, most of the memory usage is inactive. We're probably talking about Firefox history/cache, virtual machines with free memory and so on, so I figure, why not use it for disk cache instead? Well, I tried. I happen to have two 37GB 10k raptors in my machine. One of them wasn't doing anything so I set it up as swap.

Now, to control swap-behavior under Linux, you use /proc/sys/vm/swappiness. There's been some debate as to what is a sane number, but it ranges from 0 to 100. Default was 60 on my Ubuntu machine. I first tried setting it to 100, causing maximum swapping. The result was nothing short of nasty. After I'd left the machine alone for a while (for half a day, for instance), it would seem that just about everything was swapped out, and I'd get nasty delays just opening yet an other terminal or even closing one. We're talking seconds. For everything. Clearly, this was not very nice.

To make a long story short, I discovered that Linux doesn't seem to be as good as I hoped. Even when I played large amounts of video, the disk cache didn't seem to go up, yet things were still getting cached. In fact, most of my memory seemed to be use for swap cache. Turning the swappiness down to 70 seemed to do little good, though it did improve the snappiness of the desktop considerably. However, I was not happy with the results, so I tried to turn off swap.

That's when the fun begun.

Apparently, Linux doesn't like it when you rip out 5GB of virtual memory that's in use.

My system essentially froze for 10 minutes while swapoff was running, and only woke up when oomkiller woke up to kill swapoff. To get my swap turned off, I first had to turn off firefox... Amazing.

At any rate, I've turned the swap back on and I'll be testing with swappiness of 30 for a while. Hopefully, that will make more sense.

Here are some possibly interesting munin graphs from the testing period: