Five ways to copy one file, and four of them changed its size

Copying a file is supposed to be the least surprising operation available. You end up with two files and they are the same size.

That holds until the file has holes in it. Copy one of those five different ways on macOS and you get results ranging from zero bytes of disk to a hundred megabytes, from a source that occupies sixteen kilobytes. Every result is byte-identical. Every one reports the same size to ls.

6,400×

Making a file with a hole in it

A sparse file has a range of zeroes that was never written to disk. The filesystem records that the range exists and is empty and allocates nothing for it. Reading the range returns zeroes, so nothing above the filesystem can tell.

Seek past the end of an empty file and write one byte:

python3 -c "f=open('sp.bin','wb'); f.seek(100*1024*1024-1); f.write(b'\0'); f.close()"

ls -l sp.bin
du -k sp.bin

-rw-r--r--  1 teja  staff  104857600 25 Aug 20:44 sp.bin
16	sp.bin

Both numbers are correct. ls reports how long the file is. du reports how much disk it consumes. Here they differ by a factor of 6,400.

16 KiB

A trap in stat on the way past

The obvious cross-check is stat, and read the obvious way it gives a wrong answer:

stat -f '%b blocks, %k blocksize' sp.bin

32 blocks, 4096 blocksize

Thirty-two blocks at the 4096 printed beside it comes to 131,072 bytes, or 128 KiB. That disagrees with du by a factor of eight.

du is right. %b is counted in 512-byte units, and %k is the preferred I/O block size for the filesystem, not the unit %b is expressed in. Two adjacent fields that look related and are not.

32 × 512  = 16,384 bytes  = 16 KiB   matches du
32 × 4096 = 131,072 bytes = 128 KiB  does not

Five copies

The same source, copied five ways, measured with du -k:

cp sp.bin       out_cp.bin
cat sp.bin    > out_cat.bin
ditto sp.bin    out_ditto.bin
rsync -a  sp.bin out_rsync.bin
rsync -aS sp.bin out_rsyncS.bin

Method	On disk	Ratio to source
Original	16 KiB	1×
cp	16 KiB	1×
rsync -aS	0 KiB	smaller than source
cat >	102,400 KiB	6,400×
ditto	102,400 KiB	6,400×
rsync -a	102,400 KiB	6,400×

Repeated three times from a fresh source, the numbers did not move:

run 1: cp 16 | cat 102400 | ditto 102400 | rsync-a 102400 | rsync-aS 0
run 2: cp 16 | cat 102400 | ditto 102400 | rsync-a 102400 | rsync-aS 0
run 3: cp 16 | cat 102400 | ditto 102400 | rsync-a 102400 | rsync-aS 0

Three of the five methods turned a 16 KiB file into a 100 MiB one. cmp reports no difference between any copy and the source, and ls -l reports 104,857,600 bytes for all of them.

The two results worth staring at

ditto expands the file. This is the one most likely to catch someone out, because ditto is Apple’s own tool and the one usually recommended for copying anything with macOS metadata attached. It preserves resource forks and extended attributes carefully and writes the data as a plain stream of bytes, holes included as real zeroes. Preserving metadata and preserving sparseness are separate problems, and ditto solves the first.

rsync -aS produces a copy smaller than the original. Zero blocks against the source’s sixteen.

0 KiB

That is not an error. Creating the source wrote one real byte at the 100 MiB mark, which allocated the block containing it. That byte happens to be a zero. -S tells rsync to detect runs of zeroes and write them as holes, so it treats the final block as part of the hole and allocates nothing whatever. Reading the result returns the same zeroes, and cmp confirms the two files are identical.

The source, in other words, was never maximally sparse. rsync -aS made it more sparse than the thing it copied.

Why the methods diverge

The dividing line is whether the tool is handed a file or a stream.

cp opens the source as a file. It can seek around it, find the runs of zeroes and reproduce the holes at the destination. macOS cp does this without being asked.

A shell redirect cannot. cat reads the file and receives bytes, and reading a hole returns real zeroes because that is precisely what a hole is for. cat has no way to distinguish a zero that was stored from one that was never written, so it emits 104,857,600 zeroes into the pipe and the redirect writes every one.

This predicts the rest of the table. ditto streams, so it expands. rsync -a transfers contents without looking for zero runs, so it expands. rsync -aS explicitly looks, so it does not. Sparseness is a property of files that does not survive being turned into a stream, and no amount of care in the receiving program can reconstruct what the sending side has already flattened.

One detail specific to current macOS

rsync --version | head -1

openrsync: protocol version 29

macOS no longer ships GNU rsync. It ships openrsync, a BSD-licensed reimplementation, and the version reported here is protocol 29. Anything you have read about rsync flags on Linux should be treated as a hypothesis on this machine rather than a fact, and -S behaving as it does above is measured here rather than assumed from the GNU manual.

Touching a hole costs more than the bytes you wrote

The arithmetic of partial writes is also not what most people would guess. Writing 1 MiB of real data into the front of the sparse file:

python3 -c "f=open('sp.bin','r+b'); f.write(b'x'*1024*1024); f.close()"
du -k sp.bin

1040

1,040 KiB, not 1,024. The extra sixteen is the block holding that final byte at the 100 MiB mark, allocated when the file was created and still there. The file now carries real data at both ends with a hole between them, and it costs the sum of the two ends rather than the span between them.

This is the mechanism behind a complaint people make about disk images: they grow and never shrink. Every region you write to once stays allocated afterwards. Deleting a file from inside a mounted disk image does not punch a hole in the image, it just marks space free within the filesystem the image contains, one layer above the layer where the allocation lives. The image keeps every block it has ever touched until something deliberately reclaims them.

That is also why the numbers in the copy table are worth knowing rather than merely interesting. A virtual machine disk nominally sized at 200 GB, holding 12 GB of real data, is a 12 GB file until something copies it badly. Move it with ditto as part of a backup script and it becomes a 200 GB file, on a destination chosen on the assumption that 12 GB was what needed moving. Nothing reports an error. The backup completes, the file verifies, and the disk fills.

Reclaiming space from a file that was already expanded

If a copy has already flattened a file, the holes are recoverable, because the zeroes are still zeroes. Copying it once more with a tool that looks for them will restore the sparseness:

rsync -aS bloated.bin recovered.bin && mv recovered.bin bloated.bin

On the expanded copies above this returned them to zero blocks, with cmp confirming the contents unchanged. The requirement is free space for the intermediate copy, which is awkward precisely when you need it most, on a disk that the expansion has already filled.

What to do about it

The check is two commands, and the rule is short:

ls -l file    # apparent size
du -k file    # what it actually costs

If those disagree by a wide margin, the file has holes, and the method you use to move it decides whether they survive. Use cp, or rsync -aS. Avoid a shell redirect, and know that ditto will expand it despite being the tool most often recommended for careful copying.

And if a copy that should have been small has just consumed a large amount of disk, this is the first thing to test, because the file responsible will look entirely correct in every listing you run against it.