SOLVED: Bittorrent Sync performance over a LAN

I have multiple Linux and Windows 7 machines on my local LAN which I use Bittorrent Sync in order to keep certain files synchronized. Bittorrent Sync is a fantastic way of keeping files synchronized across multiple devices (either LAN or WAN) without there being a central server of some form (like Google Drive or DropBox, for example).

One often misunderstood aspect is the mechanism of file synchronisation is not simply a file copy. This is the Bittorrent protocol and can sync multiple files at once and/or uploads/downloads different sections of the file to different hosts simultaneously (who themselves also upload/download of course). Another incredibly useful thing that is often overlooked is the .SyncIgnore file (especially when dealing with Windows shares) which is placed in the root of the shared folder and applies to all subfolders as well and can be used to define which files should be ignored by the synchronization process, say thumbs.db or desktop.ini on Windows, for example (which are not sync'ed by default).

The default .SyncIgnore file is :

.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
Icon?
ehthumbs.db
desktop.ini
Thumbs.db

This is a big time saver for sync'ing large directory structures.

The net effect is a fantastically efficient mechanism, especially across the LAN.

The main usage is SO simple... on one host, add a folder and generate the public key for that folder, then on the other hosts add a folder and specify the main hosts public key. That's it. There are other options like the ability to set up the folder in read-only mode (or a key that only lasts 24 hours for example).

Another thing to watch out for if you're not getting the LAN performance you expect is to turn off the encryption of LAN traffic, by creating a btsync.conf (on linux) with JSON content of the form :

{
  "device_name": "My Sync Device",
  "listening_port" : 0,

  "storage_path" : "/home/user/.sync",

  "check_for_updates" : true,
  "use_upnp" : true,

  "download_limit" : 0,
  "upload_limit" : 0,

  "webui" :
  {
    "listen" : "0.0.0.0:8888",
    "login" : "admin",
    "password" : "password"
  },
  "lan_encrypt_data" : false
}

and then running btsync with the --config option, i.e.

$ ./btsync --config btsync.conf

On Windows you can set this parameter within the UI via Preferences -> Advanced and toggling lan_encrypt_data to false. On my system this made at least a 10x increase in LAN performance. See http://labs.bittorrent.com/experiments/sync/get-started.html#config-file for further details.

Sometimes, as well, I did have trouble sync'ing windows and linux clients. I'm not 100% sure why this would be, but one thing that significantly helped (and is probably a really good security idea anyway) is specifying pre-determined hosts, i.e. state that only 192.168.0.5:58639 (or whatever) can sync the folder.

Fun with the r8169 driver on Linux and a RTL8102E NIC

My son's netbook has a Realtek RTL8102E fast ethernet controller which (on linux kernels <= 3.9.7) was handled by the, technically incorrect but just about works, r8169 built-in kernel module. However, installing linux kernel 3.9.8, the ethernet connection (eth0 in this case) just had LOTS of entries of the following form in the output of dmesg (roughly 6 seconds apart) :

...
eth0 link up
...

I worked out the regression range of the kernel to the following kernel change which was backported to 3.9.8 : r8169: fix offloaded tx checksum for small packets.

So, it's a linux kernel regression yeah? Well, I think so, yeah, a non-technical user of Linux (after upgrading) would be at a loss (and i'll consider logging a kernel bug or mentioning it on #linux). However, technically, you could argue it's not a bug, since really the r8169 module is not intended for this NIC, the correct module (r8101) can be found at the official Realtek website.

So, after downloading the correct driver from the Realtek website (and extracting to a r8101 folder), I installed in the usual way :

$ sudo apt-get install build-essential
...
$ cd r8101
$ sudo ./autorun.sh

autorun.sh will rmmod the r8169 module and load the r8101 module. Also, just to make sure that the r8169 module was never loaded, I manually blacklisted r8169 by adding "blacklist r8169" to /etc/modprobe.d/blacklist.conf. The only downside now, though, is that every kernel update will have to rerun the driver installer, but hey ho, I'd rather have the benefits of latest kernels and drivers.

Note, that at time of writing, the Realtek driver does not compile against a 3.10 kernel, but is fine against 3.9.9 (the latest 3.9.x stable kernel at time of writing).

Lubuntu 12.10 and reboot issue on netbooks

Just thought I'd do a quick note about a reboot issue I recently looked into on a Packard Bell netbook ( dot.s model ).

Essentially, the problem was that the machine would power off (either via the GUI or via the "poweroff" command), but the restart/reboot option just left you at a black screen.

The fix was to add reboot=efi to the /etc/default/grub config file, since it appears that some netbooks don't actually have a keyboard controller (which is the default setting for the reboot option) and in this situation, the netbook won't restart...

Anyway, the fix (and other options for the reboot option in grub config) is detailed at http://www.inforbiro.com/blog-eng/ubuntu-netbook-restart-problem/.

DLNA on Lubuntu for PS3, iPad and Android

Here we go again, a quick write-up of my experiences setting up a DLNA server on Lubuntu 12.10 (on an eeePC).

So, what's DLNA? DLNA is a HTTP-based XML format (via uPNP) for providing and delivering streamed media over a network. So, it's the usual broadcast story of uPNP and request/response over web services.

So, you obviously need a DLNA server and for this I chose miniDLNA simply because the DLNA server was going to be run on an eeePC netbook, so I needed as lightweight implementation as possible. The fact that it is command line only also helped (I'm a command-line freak).

Installing miniDLNA is as simple as usual when using a Debian-style OS :

$ sudo apt-get install minidlna

Once installed, the config file is in /etc/minidlna.conf and has the usual entries such as default listening port (8200) and the media folders to actually share your stuff from, defaulting to /opt but it should really be changed to something other than this, say /home/user/media.

Now, assuming that all your firewall/iptables setup allows the uPNP and port 8200 traffic, then you need to start the DLNA server. There are a couple of ways to do this, the simplest being :

$ minidlna

which will run as a daemon. All log files will go to /var/log (unless specified otherwise in the config file).

You can, however, run it non-daemonized (in the current shell process) in debug mode which allows you to see the error / log messages via stdout (useful to see what clients are actually sending in the HTTP POST requests etc.), i.e.

$ minidlna -d

Once this is done, then all DLNA clients should see the DLNA server, certainly the iPad3 and PS3 did successfully. You do, of course, then have problem of media compatibility between the clients, iPADs only support MP4 and MOV by default etc. etc., but that's another story...

Setting up an AirPrint server on Linux

I recently had a need to set up printing from an iPad3 and iPhone and hoped to use my existing Lexmark x4650 which serves my home network.

Now, there are a few things to remember about the architecture of iOS printing. iOS devices require that a printer has to support AirPrint which uses Multicast DNS (mDNS). mDNS is a broadcast protocol (by default on port 5353 and using UDP in the same way that DNS does) that the airprint service uses to advertise that the printer is available (and on what port the service is listening on).

Unfortunately for me, the Lexmark x4650 does not support AirPrint, so an alternative mechanism needs to be used to deliver this requirement. Now, since I'm a linux user (Lubuntu is my distro of choice at the moment), Linux printing uses CUPS as it's print manager architecture which listens on port 631, by default, so you need some mechanism of implementing the listener such that it accepts AirPrint traffic whilst interacting with CUPS. So, in steps avahi, which is the standard way of implementing this listener.

So, firstly, I got the printer setup on Linux. Whilst it's not supported out of the box (this is handled by the openPrinting.org initiative), fortunately, Lexmark do offer Linux drivers.

Once the printer was setup (and hence CUPS was setup), then all that was needed was to configure avahi. So, a quick peruse of /etc/avahi/services shows that you need to define a .service file, which I manually created with the following XML :

<?xml version="1.0" ?>
<!DOCTYPE service-group  SYSTEM 'avahi-service.dtd'>
<service-group>
  <name replace-wildcards="yes">
AirPrint Lexmark-3600-4600-Series @ %h</name>
  <service>
    <type>_ipp._tcp</type>
    <subtype>_universal._sub._ipp._tcp</subtype>
    <port>631</port>
    <txt-record>txtvers=1</txt-record>
    <txt-record>qtotal=1</txt-record>
    <txt-record>Transparent=T</txt-record>
    <txt-record>URF=none</txt-record>
    <txt-record>rp=printers/Lexmark-3600-4600-Series</txt-record>   
    <txt-record>note=Lexmark 3600-4600 Series</txt-record>
    <txt-record>product=(GPL Ghostscript)</txt-record>
    <txt-record>printer-state=3</txt-record>
    <txt-record>printer-type=0x2d00c</txt-record>
    <txt-record>pdl=image/urf,application/octet-stream,
application/pdf,application/postscript,application/vnd.cups-raster,
image/gif,image/jpeg,image/png,image/tiff,
text/html,text/plain,application/vnd.adobe-reader-postscript,
application/vnd.cups-pdf</txt-record>
 </service>
</service-group>

Notice, that I had to add the image/urf MIME type to the PDL txt-record entry.

Then, you have to tell CUPS to map the image/urf MIME type to the same as a PDF entry. You do this via creating two files in /usr/share/cups/mime. I called mine apple.types which contained the following :

image/urf urf (0,UNIRAST)

and then a local.convs file which contains the following :

image/urf application/vnd.cups-postscript 66 pdftops

Once this was done then the iPhone/iPad successfully detected the printer (when scanning for an AirPrint printer) and traffic sent to the AirPrint service was placed onto the relevant CUPS print queue for the x4650.

There were, of course, a few false starts and for future reference it is important to note that CUPS error logs are contained in /var/log/cups and it is also important to realise that since mDNS and CUPS are in play, you need to ensure that your firewall / iptables setup allows port 5353 and 631 open for this to work as well.

iptables on lubuntu 12.10

I recently had the requirement to really lock down a Lubuntu 12.10 VM (actually running a reverse proxy and a small httpd daemon) and one of the tools I used to do this with iptables. Instead of posting my entire table/chain structure (which is obviously no one else's business except my own! ;-)), I thought I'd give an example of what I did and why.

From a security perspective, the problem with a lot of modern Linux distributions is that the default table/chain setup is to essentially allow all traffic on all ports. The default filter table has both INPUT, FORWARD and OUTPUT chains and each of these has the policy set to ACCEPT. From an end users point of view (especially on user-friendly distros such as Ubuntu) this makes quite a bit of sense, you don't really want non-techie users of apps like Skype to have to modify iptables chains, they simply will move on to other distros. However, from a system admin point of view, it's absolutely required that servers are locked down so that users can only access what you intend them to access, at least at the TCP level anyway.

So, let's assume we have a http server running solely on port 80. The absolute minimum is to configure iptables to prevent access to all other ports except for port 80.

So, firstly, let's change the policy of all chains to be DROP rather than ACCEPT, which means that all traffic will be blocked unless explicitly allowed :

# flush all rules first
$ sudo iptables -F

# Now, change INPUT, FORWARD and OUTPUT chains to be a default of DROP
$ sudo iptables -P INPUT DROP
$ sudo iptables -P FORWARD DROP
$ sudo iptables -P OUTPUT DROP

# Now, allow all port 80 traffic on both INPUT and OUTPUT
$ sudo iptables -A INPUT -p tcp --sport 80 -j ACCEPT
$ sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT

Note, that this will block EVERYTHING except port 80 traffic. iptables can be used to setup really complex (or simple) rule chains which allow you to lockdown any machine. For example, the machine I'm typing on now has a similar rule chain as the above, but allowing port 80 (HTTP), port 443 (HTTPS), port 53 (DNS) and port 6697 (for SSL/TLS IRC connections).

The only downside with this is that rule chains can become complex unless you ensure that they are as simple as you can make them. Once you get used to doing iptables chains you find they are easy to setup and you can be sure that you've added yet another powerful security barrier to your system.

IRC via irssi

Haven't blogged for quite a while, been really busy with various things. So, just thought I'd have a little techie post about IRC.

A lot of people have forgotten IRC as a chat facility, but it really is the granddaddy of them all and it is certainly still very popular, especially in the techie world. A lot of techie and developer discussions about subjects as Linux, Firefox, Oracle and the hacker fraternity all meet in IRC, in fact, it's probably fair to say that IRC is the primary communication facility for developers of certainly the first two technologies in that list. There's are many advantages with using IRC as a chat facility and that is its simplicity of basic use, anyone can use it, it doesn't take long to get to grips with the basics. For the advanced user, you can script to your heart's content, or even run your own server for your own purposes (even embedding the technology in your own product, just check out how the chat facility in the python-based MMORPG "Eve Online" works!

There are many IRC client and server software available, whether you like GUIs or command line, the choice is yours. I tend to prefer command-line software for most things, so this blog post will concentrate on my favourite client, irssi using a Lubuntu Linux OS.

So, the first step is to install the client, which as with most things on Debian-based distros, is quite easy :

$ sudo apt-get install irssi

Once the install has completed, you then have to modify your settings. All settings can be done through the command-line UI, and for those commands I will direct you to the irssi documentation. However, all settings are stored in $HOME/.irssi/config and is probably quicker to manually change the file (vim is your friend!). The only thing to point out is to issue a /save command if you do change the settings via the front-end otherwise this file won't be updated.

So, most of the settings can be left as-is, all settings such as default and custom aliases are defined in here. However, the important entries are the hostname, real_name and nick(name). These should be changed to whatever you require.

Once this is done, that's pretty much it. You can then connect to an IRC server by either using the /connect IRC command or via the command-line as :

$ irssi -c irc.undernet.org

Once you are connected, you can then issue any of the IRC commands that you might know, such as /join, /msg, /alias etc. This blog entry will not go into them, a quick Google for "IRC commands" or the irssi documentation will give you plenty of resources.

If you are connected to multiple channels, then you can use ALT- to switch between contexts, i.e. ALT-2 might be #oracle, ALT-3 might be #python etc. You can connect to multiple servers via the /connect command, and then Ctrl-X will switch between connections.

If anyone is interested and wants to chat (about linux/oracle/python), then, I use the nick "tackd" and am generally found on #oracle #linux and/or #python on DALnet, Undernet or EFnet (the major IRC networks really).