I run BTRFS on my root filesystem (on Linux), mostly for the quick snapshot and restore functionality. Yesterday I ran into a common problem: my drive was suddenly full. I went from 4GB of free space on my system drive to 0 in an instant, causing all sorts of chaos on my system.
This problem happens to lots of people because BTRFS doesn’t have a linear relationship to “free space available”. There are a few concepts that get in the way:
- Compression: BTRFS supports compressing data as it writes. This obviously changes the amount of data that can be stored. – 50MB of text may take only 5MB “room” on the drive.
- Metadata: BTRFS stores your data separately from metadata. Both data and metadata occupy “space”.
- Chunk allocation: BTRFS allocates space for your data in chunks.
- Multiple devices: BTRFS supports multiple devices working together, RAID-style. That means there’s extra information to store for every file. For example, RAID-1 stores two copies of every file, so a 50MB file takes 100MB of space.
- Snapshots: BTRFS can store snapshots of your device, which really store more like a diff from the current state. How much data is in the diff depends on your current state… so the snapshot itself doesn’t have a consistent size.
- Nested volumes: BTRFS lets you divide the filesystem into “subvolumes” – each of which can (someday) have its own RAID configuration.
It’s easy to look at the drive and tell how many MiB of space has not been used yet. But it’s very hard to accurately say how much of your data you can write in that space. For this reason the amount of “free space” reported on BRFS volumes by system utilities like
df can jump a lot – like my disappearing 4GiB. Worse, the free space reported by general tools is misleading. BTRFS can run out of space while
df still thinks you have lots available.
Let’s walk through how BTRFS stores data, to understand the problem a bit better. Then we can solve it with some of BTRFS’ own tools.
How much free space do I have?
Rather than using general tools like
df to answer this question, it’s better to get more detail using the
btrfs CLI tool.
BTRFS starts out with a big pool of raw storage, and allocates as it goes. You can get a listing of all the devices in a block device like this:
1 2 3 4
In this case, I only have one physical device involved. You can see that it gives me a total number of bytes allocated, compared to the total size. In another filesystem this might be the number reported to
df. Not so with BTRFS! Let’s dig deeper.
1 2 3 4 5 6
The “total” values here are the breakdown of what the first command counts as “used”.
btrfs fi df shows us of the allocated space, how much is actually storing data, and how much is just empty allocation. In this case: on my 48GiB device, 47GiB is allocated. Of the allocation, 31GiB is actually storing data. Side note: if you’re in a multi-drive situation this command will take into account RAID metadata.
Here’s an easier view:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
This shows the breakdown of space allocated and used across all the devices in this block device. “Overall” is for the whole block device, and that “Free (estimated)” number is what gets reported to
This is a problem: most of my normal tools tell me I have 15GB free space. But if I write 1GiB more data, BTRFS will run out of space anyways. This issue is a pain in the ass and hard to diagnose. It’s even harder to fix, since most of the solutions require having some extra space on the device.
Converting unused allocation to free space
So, why does BTRFS allocate so much space to store such a small amount of data? Here I am storing 31GiB of data in 47GiB of allocation, the used/total ratio is 0.66! This is very inefficient. It’s an unfortunate consequence of being a copy-on-write filesystem – BTRFS starts every write in a freshly allocated chunk. But the chunksize is static, and files come in all sizes. So lots of the time, a chunk is incompletely filled. That’s the “allocated but not used” space we’re complaining about.
Fortunately there’s a way to address this problem: BTRFS has a tool to “rebalance” your filesystem. It was originally designed for balancing the data stored across multiple drives (hence the name). It is also useful in single drive configurations though, to rebalance how data is stored within the allocation.
balance will rewrite all the data on the disk. This is probably unnecessary. Chunks will be unevenly filled, but we saw above that the average should be about 66% used. So we’ll filter based on data (
-d) usage, and only rebalance chunks that are less than 66% used. That will leave any partially filled chunks which are more-filled than average.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
There’s a nice big differnce once it’s finished:
1 2 3 4 5
That’s 15GiB of space allocated for other use. My usage ratio is now 0.94. Huzzah! In some rare cases you may need to do this on the Metadata allocation (use
-musage instead of
If you’ve already run out of space
If you have already run out of space, you can’t run a
balance! In that caseyou have to get sneaky. Here are your options:
1) Free up space
This is harder than it sounds. If you just delete data, it will probably leave those chunks partially filled and therefore allocated. What you really need is unallocated space. The easiest place to get this is by deleting snapshots. Start from the oldest one, since it will be the biggest.
Once you have a little bit of wiggle room, rebalance a small segment, like Metadata. Then proceed with rebalancing data as described above.
2) Add some space
Don’t forget, a BTRFS volume can span multiple devices! I had to exercise this option recently. Add a device – a flash drive will do, but choose the fastest thing you can – and add it to the BTRFS volume.
1 2 3 4 5 6 7 8 9
Balance operations usually take a long time – more than an hour is not unusual. It will take even longer with slow flash media involved. For that reason, I use a very low balance filter (
-dusage=) in this example. We only need to free up a teensy bit of space to run balance again without the flash disk in the mix.
And this last option is how I saved my computer last night. I hope this helps someone out of a similar predicament someday.