Rakaly Status Report #12: next level compression banner

Rakaly Status Report #12: next level compression

Published on:

Optimizations found in how saves are uploaded, downloaded, and stored at rest resulted in a 2-3x reduction in bandwidth and up to a 2x reduction in time to parse shared saves. This benefits all Rakaly users as this made it easy to increase the save limit from 10 to 100. For those interested in the technical details: keep reading.

EU4 save files come in three forms:

Previously Rakaly was uploading the zip files as is while gzipping the plaintext. This method has some nice properties: mainly those uploading zips don’t need to preprocess the upload at all. But upon closer inspection, the generated zip files are poorly compressed and leave a lot to be desired. Since some saves are shared with many others, it would benefit all if the bandwidth needed to upload and download was reduced.

The solution is to use a modern compression algorithm called brotli. I sampled a few saves and found it compressed 2x better than gzip and 3x better than zips. Since brotli only performs compression, we needed an archival format where one can store files. That’s where tar comes into view. In short, when a user uploads a save, we extract the files from the zip and place them into a tar file before the entire payload is brotli compressed. Everything should be transparent without the user ever needing to know about this format.

This is a backwards compatible change. Previously uploaded saves remain in their original form and can continued to be parsed. 3rd party apps using the API can continue to transmit plain zips and gzips. The Rakaly CLI has been updated to take advantage of this new scheme, so it is recommended to upgrade for those that upload files through it.

To explain where the beforementioned parsing performance gains come from, browsers don’t natively understand zip files. To them, zip files are just a blob to be passed along to Rakaly which implements unzip. Contrast that with brotli, which browsers do natively understand. Browsers are able to decompress the save at the speed of native hardware instead of Rakaly needing to reimplement it inside the browser environment. Now Rakaly only needs to concern itself with parsing and not decompression, hence the performance improvement. Most won’t see a dramatic 2x improvement, but hopefully it is still tangible.

There are some discovered downsides, but they don’t seem too major:

There is potential for further improvement. If one cranks up the quality of the brotli compression to the max, we can achieve under 1 MB files. The tradeoff is that compressed output shouldn’t be expected in real time. A compromise would be for users to upload compressed enough saves and then have an offline system that compress them further.

Enjoy!


Feel free to get in contact via Discord or hi [(at)] rakaly.com