Build Rsync faster from scratch in Go


For fast file transfer between two computers, SCP is a good program to use. For more complex, large, or regular backups, however, the go-to tool is rsync. It’s faster, more efficient, and usable in a wider range of circumstances. For all its advantages, [Michael Stapelberg] felt it had a major weakness: it’s a tool written in C. [Michael] is philosophically opposed to programs written in C, so he decided to implement rsync from scratch in Go instead.

[Michael]The road to the decision to tackle this project is complicated. His ISP recently upgraded his Internet connection to 25 Gbps, which meant his custom router was the bottleneck in his network. To solve this problem, he migrated his router to a PC with several 25 Gbit/s network cards. To take full advantage of the speed now theoretically available, he started using a tool called gokrazy, which turns apps written in Go into their own device. This means that instead of installing a full Linux distribution to handle specific tasks (like a router, for example), the only thing loaded on the computer is basically the Linux kernel, compiler, and Go libraries, and then the Go app itself.

With a new router with hardware capable of supporting those fast speeds and running only software written in Go, the final step was to finally build rsync to take over its tasks on its network. This meant that rsync itself had to be built from scratch in Go. Once [Michael] completed this last task, he found that his implementation of rsync is actually much faster than the version built in C, thanks to the modernization found in the Go language and the fact that his router doesn’t run all the associated cruft to a standard Linux distribution.

For a software project of this size, we find [Michael]The step-by-step process is worth considering for any problem any of us try to solve. Not only that, refactoring a foundational tool like rsync is a complex task in itself, let alone creating it simply to increase network speeds beyond what most of us would already consider blazingly fast. We’re leaving out a ton of detail on this release, so we highly recommend checking out his speech in the video below.

Thanks to [sarinkhan] for the tip!


About Author

Comments are closed.