I run a Pi 4 powered NAS at home, and I religiously back my data up to a local disk once in awhile. In the case of disk failures, I always have the backup that can be used to restore my data, so I sleep well at night.
After I moved to California, there’s one thing I found out quickly about the state: wild fires are everywhere! Every year, they may be late, but will never be absent. A few years ago, I moved to the current location on a hill with a grove of trees, and there was a fire here this year. That concerns me. In the face of unforeseeable disaster (knock on wood), losing all data in a “puff of smoke” would really suck. Because of this, I decided to back up my data to the cloud in addition to my local storage. At the same time, I want to share my processes of backing up data so it may helpful to others, as well as discovering potential shortcomings.
Local Backup
Local backup is mostly used to recovery from disk failures. For this purpose, I compress all my files into a giant 7z zip file that’s sitting on a separate disk, and incrementally updates it using 7z’s update option. The initial creation can be time consuming, but each incremental update after that is much faster. I run the updates at night when no other tasks are running so the Pi 4 can dedicate all CPU powers to do the job. At the end, it creates a nice single file that is compressed. If I need to restore, I can simply uncompress it.
Cloud Backup
I picked S3 Glacier because of its ubiquitous presence, good documentations on SDKs, and relatively transparent and straightforward cost (ask me about my scary experiences using Google cloud). Because it’s hard (and pointless) to overwrite archives, and you are charged on each upload request, I upload the 7z file to Glacier every 3 months, and delete the older archives at the same time. I picked 3 months because archives older than 3 months can be deleted free of charge, and I’m ok with losing at most 3 months of memory.
Command Line Based Tools
I wrote a tool in Golang to handle the upload process to Glacier, you can find the source code or download the tool from my Github repository. On the repository’s README file, I described in more details on why I decided to make this tool instead of using existing solutions. But the tl;dr is 1. I’m trying to build something useful with Golang. 2. I want something to run on my Pi 4 with small memory footprint, and requires no additional software installed. The go language fits my use cases perfectly.
Please let me know your comments, and share with me if you approach your backups differently.
