SOLVED: Bittorrent Sync performance over a LAN

I have multiple Linux and Windows 7 machines on my local LAN which I use Bittorrent Sync in order to keep certain files synchronized. Bittorrent Sync is a fantastic way of keeping files synchronized across multiple devices (either LAN or WAN) without there being a central server of some form (like Google Drive or DropBox, for example).

One often misunderstood aspect is the mechanism of file synchronisation is not simply a file copy. This is the Bittorrent protocol and can sync multiple files at once and/or uploads/downloads different sections of the file to different hosts simultaneously (who themselves also upload/download of course). Another incredibly useful thing that is often overlooked is the .SyncIgnore file (especially when dealing with Windows shares) which is placed in the root of the shared folder and applies to all subfolders as well and can be used to define which files should be ignored by the synchronization process, say thumbs.db or desktop.ini on Windows, for example (which are not sync'ed by default).

The default .SyncIgnore file is :

.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
Icon?
ehthumbs.db
desktop.ini
Thumbs.db

This is a big time saver for sync'ing large directory structures.

The net effect is a fantastically efficient mechanism, especially across the LAN.

The main usage is SO simple... on one host, add a folder and generate the public key for that folder, then on the other hosts add a folder and specify the main hosts public key. That's it. There are other options like the ability to set up the folder in read-only mode (or a key that only lasts 24 hours for example).

Another thing to watch out for if you're not getting the LAN performance you expect is to turn off the encryption of LAN traffic, by creating a btsync.conf (on linux) with JSON content of the form :

{
  "device_name": "My Sync Device",
  "listening_port" : 0,

  "storage_path" : "/home/user/.sync",

  "check_for_updates" : true,
  "use_upnp" : true,

  "download_limit" : 0,
  "upload_limit" : 0,

  "webui" :
  {
    "listen" : "0.0.0.0:8888",
    "login" : "admin",
    "password" : "password"
  },
  "lan_encrypt_data" : false
}

and then running btsync with the --config option, i.e.

$ ./btsync --config btsync.conf

On Windows you can set this parameter within the UI via Preferences -> Advanced and toggling lan_encrypt_data to false. On my system this made at least a 10x increase in LAN performance. See http://labs.bittorrent.com/experiments/sync/get-started.html#config-file for further details.

Sometimes, as well, I did have trouble sync'ing windows and linux clients. I'm not 100% sure why this would be, but one thing that significantly helped (and is probably a really good security idea anyway) is specifying pre-determined hosts, i.e. state that only 192.168.0.5:58639 (or whatever) can sync the folder.

Fun with the r8169 driver on Linux and a RTL8102E NIC

My son's netbook has a Realtek RTL8102E fast ethernet controller which (on linux kernels <= 3.9.7) was handled by the, technically incorrect but just about works, r8169 built-in kernel module. However, installing linux kernel 3.9.8, the ethernet connection (eth0 in this case) just had LOTS of entries of the following form in the output of dmesg (roughly 6 seconds apart) :

...
eth0 link up
...

I worked out the regression range of the kernel to the following kernel change which was backported to 3.9.8 : r8169: fix offloaded tx checksum for small packets.

So, it's a linux kernel regression yeah? Well, I think so, yeah, a non-technical user of Linux (after upgrading) would be at a loss (and i'll consider logging a kernel bug or mentioning it on #linux). However, technically, you could argue it's not a bug, since really the r8169 module is not intended for this NIC, the correct module (r8101) can be found at the official Realtek website.

So, after downloading the correct driver from the Realtek website (and extracting to a r8101 folder), I installed in the usual way :

$ sudo apt-get install build-essential
...
$ cd r8101
$ sudo ./autorun.sh

autorun.sh will rmmod the r8169 module and load the r8101 module. Also, just to make sure that the r8169 module was never loaded, I manually blacklisted r8169 by adding "blacklist r8169" to /etc/modprobe.d/blacklist.conf. The only downside now, though, is that every kernel update will have to rerun the driver installer, but hey ho, I'd rather have the benefits of latest kernels and drivers.

Note, that at time of writing, the Realtek driver does not compile against a 3.10 kernel, but is fine against 3.9.9 (the latest 3.9.x stable kernel at time of writing).