Synology-based synchronization solution

When CrashPlan announced the end of its unlimited backup service, I had to actively seek a new solution asking myself the question of how to provide two services I was previously (more or less) getting from this provider:

1. Backups
2. Synchronization between several machines (several PC and several iPhone)

CrashPlan easily provided the backup feature despite a high volume of photo files (more than 2 TB as of today), but was relatively poor interms of synchronization. So, it was time to spend some energy defining the whole need/specification and finding a true solution.

Specification

For backup, my requirements have long been defined:

1. At least three backup copies
2. At least two different technologies (e.g; not to rely only on optical drives or hard drives)
3. At least one off-site location (to be protected against theft, fire hazards, etc., think off-site, or cloud, or external/bank safe box)

For synchronization, previous use of DropBox and Evernote (on small volumes) allowed me to confirm the precise needs I had:

1. Real-time sharing of some critical files with daily updates (as for Evernote note files, or the database for my password management application, etc.)
2. Encryption of all comms (I don’t want to leak my passwords or my bank account data).

Solution

I was clearly influenced by several factors in my choice, and I will try and make these explicit (your solution may not be strictly identical to mine). So, let’s start by what I chose to deploy.

A smal file server (a Synology DS413j) at my home. It hosts 4 hard drives of 3 TB (initially, four Western Digital RED) in SHR/RAID 5 mode for a total capacity of 6 TB.

The large capacity was defintely dictated by the the need to host a large multimedia server and a large quantity of digital photos (Today, we have altogether much more than 100.000 pictures on this server).

The choice of RAID 5 was a very early one because of its optimum balance between overall dedundancy (you can loose one hard drive with no data loss) and cost (you loose less data to redundancy than in other two-disk RAID configurations). The Synology SHR technology is even better because is allows disks with different capacities (the two smaller ones define the maximum capacity. So, you’d better have one 500 GB with three 4 TB bigger ones; It would be a waste) and it accepts (like all RAIDs) to replace one failed drive with a new one (during the rebuilding of RAID/SHR, you have no more data protection but this is temporairy – and you can keep working in paralle).

Going to Synology was even more justified by a single important item rarely listed in RAID sevrer comparisons: The software offering is specially attractive at Synology. DSM (Synology OS) hosts  “packages” for many applications. The complements I installed (see below) are provided as ready-to-use instalations and can be easy to install even if you have minimal understanding of the inner workings of a server.

So, I was able to deploy a two-stage solution:

1. PC-to-Synology synchronization (three PC via Cloud Station Server)
2. Synology-to-Amazon synchronization (Amazon Cloud Drive through Cloud Sync)

Cloud Station Server and Cloud Sync are two applications from Synology which are installed in about 5-6 minutes (including download time) from the  “Packages Center” of Synology DSM.

If we start looking at the synchronization between PC (or iPhone) and Synology, merely opening the Cloud Station Server in the list below (in the management interface of my Synology DS413j DSM) leads us directly to a menu allowing to download the Windows application (or iPhone app) to install before starting synchronizations between PC and Synology cloud server.

A few clicks and it’s done. Nothing could be simpler. a few minutes and synchronization is starting. It runs in background (it does not hinder normal PC activity) and you will only notice the new change to the file icons showing that it is already synched ot the cloud (see examples below) or that it is still waiting for synchronization (blue tag).

At this point, PC are synchronized, via the Synology server. You can consider that this is a backup since I chose to keep all most recent 32 versions of each file modifications and to not let Synology wipe deleted files (they are stored in the server dust-bin). So, this is a Time Machine feature on top of a backup.

Important notice: The synchronisation between PC and Synology cloud server is very fast and does not eat up resource even on a small/old PC (contrary to some backup programs I would not dare name, like Cobian Backup). this is re-assuring: The end-users can forget about the backup running in the background, stopping if the PC goes to sleep and starting again when it wakes up. This whole process has nearly no impact on the PC behaviour.

Similarly, the iPhone and iPad can use synchronization (of some files, I don’t want them overwhelmed) when on the move.

The second feature replicates these data to another location (here, I chose to synchronize with Amazon Cloud Drive aka ACD because its cost is only 70€/an which appears very reasonable and can be reduced if you have Amazon Prime and a lot of photos). I understood that transfer speed would be limited by my RAID server (the DS413j is quite old and has a very cheap ARM CPU, but it defintely did not break the bank) rather than my bandwidth (I have optical fiber broadban) or by Amazon (all users observe that Amazon is going fast).

Synology Cloud Sync provides this in a matter of a few clicks. Just provide your Amazon login and password. Synology will do it all for you. Synchronization will start in a minute at most (I tried it with Google Drive too. this is as simple and as fast – bandwidth is quite high, which is a good thing; I thought about trying c hubic from OVH, but Internet comments and face-to-face talks with past users seem to point to a product without the needed performance and reliability you want here).

Observations

A few remarks I may have to add after some real-life use:

• The DS413j has a very small CPU from ARM. the CPU workload stays very high (~100%) during all the synchronization phase.
• The first synchronization is lenghty (several days or several weeks depending on the data volume). I recommend to build the data set progressively (not to bunch all your data on the first day).
• Next synchros will generally be much faster (nearly instantaneous).
• Recently modified files tend to be synchronized first.
• I used Cloud Sync to link my server to Amazon, but there is also a solution to synchronize from Synology to Synology: Cloud Station Share Sync. I did not try it but it could be worht exploring if you already have to geographical locations (A second home? A company with two offices?) and two Synology servers. It seems to be as simple as synchronizing with Amazon Cloud Drive.

• In order to elevate the RAID server reliability, you’d better choose hard drives designed for 24/7 use. I would point to Western Digital RED (specailly designed for RAID servers) or Hitachi UltraStar (24/7 version of the DeskStar drives). Also, if you can avoid having all 4 or 5 identical drives, it will reduce the risk of seeing them all failing nearly simultaneously. See BackBlaze statistics on hard drives reliability.
• You may ask where are my “three backup copies”. I actually have the following:
  • The original files on a master PC
  • A manually-copied copy on a separate hard drive (at least one, sometimes two or three of them on spare drives)
  • A synchronzied copy on the Synology server
  • A synchronized copy on Amazon Cloud Drive
• While the Synology DS413j is no longer listed, you can stil purchase the DS416j ou DS418j.
• On the Synology NAS server, I ensured that the synchronized directory is not visible to Windows: A ransomware can’t see it, it would have to hack through the Synology software. this is protecting me against those ransomwares which try to encrypt netork drives too. (Anyway, I can still control my data through the web interface of DSM).
• If a ransomware tries to encrpyt my files (they may then be synchronized in encrypted form), I made sure that versioning is enabled on Synology (This way, I could travel back in time and avoid “paying the ransom”).