SOLVED: Bittorrent Sync performance over a LAN

I have multiple Linux and Windows 7 machines on my local LAN which I use Bittorrent Sync in order to keep certain files synchronized. Bittorrent Sync is a fantastic way of keeping files synchronized across multiple devices (either LAN or WAN) without there being a central server of some form (like Google Drive or DropBox, for example).

One often misunderstood aspect is the mechanism of file synchronisation is not simply a file copy. This is the Bittorrent protocol and can sync multiple files at once and/or uploads/downloads different sections of the file to different hosts simultaneously (who themselves also upload/download of course). Another incredibly useful thing that is often overlooked is the .SyncIgnore file (especially when dealing with Windows shares) which is placed in the root of the shared folder and applies to all subfolders as well and can be used to define which files should be ignored by the synchronization process, say thumbs.db or desktop.ini on Windows, for example (which are not sync'ed by default).

The default .SyncIgnore file is :

.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
Icon?
ehthumbs.db
desktop.ini
Thumbs.db

This is a big time saver for sync'ing large directory structures.

The net effect is a fantastically efficient mechanism, especially across the LAN.

The main usage is SO simple... on one host, add a folder and generate the public key for that folder, then on the other hosts add a folder and specify the main hosts public key. That's it. There are other options like the ability to set up the folder in read-only mode (or a key that only lasts 24 hours for example).

Another thing to watch out for if you're not getting the LAN performance you expect is to turn off the encryption of LAN traffic, by creating a btsync.conf (on linux) with JSON content of the form :

{
  "device_name": "My Sync Device",
  "listening_port" : 0,

  "storage_path" : "/home/user/.sync",

  "check_for_updates" : true,
  "use_upnp" : true,

  "download_limit" : 0,
  "upload_limit" : 0,

  "webui" :
  {
    "listen" : "0.0.0.0:8888",
    "login" : "admin",
    "password" : "password"
  },
  "lan_encrypt_data" : false
}

and then running btsync with the --config option, i.e.

$ ./btsync --config btsync.conf

On Windows you can set this parameter within the UI via Preferences -> Advanced and toggling lan_encrypt_data to false. On my system this made at least a 10x increase in LAN performance. See http://labs.bittorrent.com/experiments/sync/get-started.html#config-file for further details.

Sometimes, as well, I did have trouble sync'ing windows and linux clients. I'm not 100% sure why this would be, but one thing that significantly helped (and is probably a really good security idea anyway) is specifying pre-determined hosts, i.e. state that only 192.168.0.5:58639 (or whatever) can sync the folder.

Fun with the r8169 driver on Linux and a RTL8102E NIC

My son's netbook has a Realtek RTL8102E fast ethernet controller which (on linux kernels <= 3.9.7) was handled by the, technically incorrect but just about works, r8169 built-in kernel module. However, installing linux kernel 3.9.8, the ethernet connection (eth0 in this case) just had LOTS of entries of the following form in the output of dmesg (roughly 6 seconds apart) :

...
eth0 link up
...

I worked out the regression range of the kernel to the following kernel change which was backported to 3.9.8 : r8169: fix offloaded tx checksum for small packets.

So, it's a linux kernel regression yeah? Well, I think so, yeah, a non-technical user of Linux (after upgrading) would be at a loss (and i'll consider logging a kernel bug or mentioning it on #linux). However, technically, you could argue it's not a bug, since really the r8169 module is not intended for this NIC, the correct module (r8101) can be found at the official Realtek website.

So, after downloading the correct driver from the Realtek website (and extracting to a r8101 folder), I installed in the usual way :

$ sudo apt-get install build-essential
...
$ cd r8101
$ sudo ./autorun.sh

autorun.sh will rmmod the r8169 module and load the r8101 module. Also, just to make sure that the r8169 module was never loaded, I manually blacklisted r8169 by adding "blacklist r8169" to /etc/modprobe.d/blacklist.conf. The only downside now, though, is that every kernel update will have to rerun the driver installer, but hey ho, I'd rather have the benefits of latest kernels and drivers.

Note, that at time of writing, the Realtek driver does not compile against a 3.10 kernel, but is fine against 3.9.9 (the latest 3.9.x stable kernel at time of writing).

Lubuntu 12.10 and reboot issue on netbooks

Just thought I'd do a quick note about a reboot issue I recently looked into on a Packard Bell netbook ( dot.s model ).

Essentially, the problem was that the machine would power off (either via the GUI or via the "poweroff" command), but the restart/reboot option just left you at a black screen.

The fix was to add reboot=efi to the /etc/default/grub config file, since it appears that some netbooks don't actually have a keyboard controller (which is the default setting for the reboot option) and in this situation, the netbook won't restart...

Anyway, the fix (and other options for the reboot option in grub config) is detailed at http://www.inforbiro.com/blog-eng/ubuntu-netbook-restart-problem/.

DLNA on Lubuntu for PS3, iPad and Android

Here we go again, a quick write-up of my experiences setting up a DLNA server on Lubuntu 12.10 (on an eeePC).

So, what's DLNA? DLNA is a HTTP-based XML format (via uPNP) for providing and delivering streamed media over a network. So, it's the usual broadcast story of uPNP and request/response over web services.

So, you obviously need a DLNA server and for this I chose miniDLNA simply because the DLNA server was going to be run on an eeePC netbook, so I needed as lightweight implementation as possible. The fact that it is command line only also helped (I'm a command-line freak).

Installing miniDLNA is as simple as usual when using a Debian-style OS :

$ sudo apt-get install minidlna

Once installed, the config file is in /etc/minidlna.conf and has the usual entries such as default listening port (8200) and the media folders to actually share your stuff from, defaulting to /opt but it should really be changed to something other than this, say /home/user/media.

Now, assuming that all your firewall/iptables setup allows the uPNP and port 8200 traffic, then you need to start the DLNA server. There are a couple of ways to do this, the simplest being :

$ minidlna

which will run as a daemon. All log files will go to /var/log (unless specified otherwise in the config file).

You can, however, run it non-daemonized (in the current shell process) in debug mode which allows you to see the error / log messages via stdout (useful to see what clients are actually sending in the HTTP POST requests etc.), i.e.

$ minidlna -d

Once this is done, then all DLNA clients should see the DLNA server, certainly the iPad3 and PS3 did successfully. You do, of course, then have problem of media compatibility between the clients, iPADs only support MP4 and MOV by default etc. etc., but that's another story...

Setting up an AirPrint server on Linux

I recently had a need to set up printing from an iPad3 and iPhone and hoped to use my existing Lexmark x4650 which serves my home network.

Now, there are a few things to remember about the architecture of iOS printing. iOS devices require that a printer has to support AirPrint which uses Multicast DNS (mDNS). mDNS is a broadcast protocol (by default on port 5353 and using UDP in the same way that DNS does) that the airprint service uses to advertise that the printer is available (and on what port the service is listening on).

Unfortunately for me, the Lexmark x4650 does not support AirPrint, so an alternative mechanism needs to be used to deliver this requirement. Now, since I'm a linux user (Lubuntu is my distro of choice at the moment), Linux printing uses CUPS as it's print manager architecture which listens on port 631, by default, so you need some mechanism of implementing the listener such that it accepts AirPrint traffic whilst interacting with CUPS. So, in steps avahi, which is the standard way of implementing this listener.

So, firstly, I got the printer setup on Linux. Whilst it's not supported out of the box (this is handled by the openPrinting.org initiative), fortunately, Lexmark do offer Linux drivers.

Once the printer was setup (and hence CUPS was setup), then all that was needed was to configure avahi. So, a quick peruse of /etc/avahi/services shows that you need to define a .service file, which I manually created with the following XML :

<?xml version="1.0" ?>
<!DOCTYPE service-group  SYSTEM 'avahi-service.dtd'>
<service-group>
  <name replace-wildcards="yes">
AirPrint Lexmark-3600-4600-Series @ %h</name>
  <service>
    <type>_ipp._tcp</type>
    <subtype>_universal._sub._ipp._tcp</subtype>
    <port>631</port>
    <txt-record>txtvers=1</txt-record>
    <txt-record>qtotal=1</txt-record>
    <txt-record>Transparent=T</txt-record>
    <txt-record>URF=none</txt-record>
    <txt-record>rp=printers/Lexmark-3600-4600-Series</txt-record>   
    <txt-record>note=Lexmark 3600-4600 Series</txt-record>
    <txt-record>product=(GPL Ghostscript)</txt-record>
    <txt-record>printer-state=3</txt-record>
    <txt-record>printer-type=0x2d00c</txt-record>
    <txt-record>pdl=image/urf,application/octet-stream,
application/pdf,application/postscript,application/vnd.cups-raster,
image/gif,image/jpeg,image/png,image/tiff,
text/html,text/plain,application/vnd.adobe-reader-postscript,
application/vnd.cups-pdf</txt-record>
 </service>
</service-group>

Notice, that I had to add the image/urf MIME type to the PDL txt-record entry.

Then, you have to tell CUPS to map the image/urf MIME type to the same as a PDF entry. You do this via creating two files in /usr/share/cups/mime. I called mine apple.types which contained the following :

image/urf urf (0,UNIRAST)

and then a local.convs file which contains the following :

image/urf application/vnd.cups-postscript 66 pdftops

Once this was done then the iPhone/iPad successfully detected the printer (when scanning for an AirPrint printer) and traffic sent to the AirPrint service was placed onto the relevant CUPS print queue for the x4650.

There were, of course, a few false starts and for future reference it is important to note that CUPS error logs are contained in /var/log/cups and it is also important to realise that since mDNS and CUPS are in play, you need to ensure that your firewall / iptables setup allows port 5353 and 631 open for this to work as well.

iptables on lubuntu 12.10

I recently had the requirement to really lock down a Lubuntu 12.10 VM (actually running a reverse proxy and a small httpd daemon) and one of the tools I used to do this with iptables. Instead of posting my entire table/chain structure (which is obviously no one else's business except my own! ;-)), I thought I'd give an example of what I did and why.

From a security perspective, the problem with a lot of modern Linux distributions is that the default table/chain setup is to essentially allow all traffic on all ports. The default filter table has both INPUT, FORWARD and OUTPUT chains and each of these has the policy set to ACCEPT. From an end users point of view (especially on user-friendly distros such as Ubuntu) this makes quite a bit of sense, you don't really want non-techie users of apps like Skype to have to modify iptables chains, they simply will move on to other distros. However, from a system admin point of view, it's absolutely required that servers are locked down so that users can only access what you intend them to access, at least at the TCP level anyway.

So, let's assume we have a http server running solely on port 80. The absolute minimum is to configure iptables to prevent access to all other ports except for port 80.

So, firstly, let's change the policy of all chains to be DROP rather than ACCEPT, which means that all traffic will be blocked unless explicitly allowed :

# flush all rules first
$ sudo iptables -F

# Now, change INPUT, FORWARD and OUTPUT chains to be a default of DROP
$ sudo iptables -P INPUT DROP
$ sudo iptables -P FORWARD DROP
$ sudo iptables -P OUTPUT DROP

# Now, allow all port 80 traffic on both INPUT and OUTPUT
$ sudo iptables -A INPUT -p tcp --sport 80 -j ACCEPT
$ sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT

Note, that this will block EVERYTHING except port 80 traffic. iptables can be used to setup really complex (or simple) rule chains which allow you to lockdown any machine. For example, the machine I'm typing on now has a similar rule chain as the above, but allowing port 80 (HTTP), port 443 (HTTPS), port 53 (DNS) and port 6697 (for SSL/TLS IRC connections).

The only downside with this is that rule chains can become complex unless you ensure that they are as simple as you can make them. Once you get used to doing iptables chains you find they are easy to setup and you can be sure that you've added yet another powerful security barrier to your system.

IRC via irssi

Haven't blogged for quite a while, been really busy with various things. So, just thought I'd have a little techie post about IRC.

A lot of people have forgotten IRC as a chat facility, but it really is the granddaddy of them all and it is certainly still very popular, especially in the techie world. A lot of techie and developer discussions about subjects as Linux, Firefox, Oracle and the hacker fraternity all meet in IRC, in fact, it's probably fair to say that IRC is the primary communication facility for developers of certainly the first two technologies in that list. There's are many advantages with using IRC as a chat facility and that is its simplicity of basic use, anyone can use it, it doesn't take long to get to grips with the basics. For the advanced user, you can script to your heart's content, or even run your own server for your own purposes (even embedding the technology in your own product, just check out how the chat facility in the python-based MMORPG "Eve Online" works!

There are many IRC client and server software available, whether you like GUIs or command line, the choice is yours. I tend to prefer command-line software for most things, so this blog post will concentrate on my favourite client, irssi using a Lubuntu Linux OS.

So, the first step is to install the client, which as with most things on Debian-based distros, is quite easy :

$ sudo apt-get install irssi

Once the install has completed, you then have to modify your settings. All settings can be done through the command-line UI, and for those commands I will direct you to the irssi documentation. However, all settings are stored in $HOME/.irssi/config and is probably quicker to manually change the file (vim is your friend!). The only thing to point out is to issue a /save command if you do change the settings via the front-end otherwise this file won't be updated.

So, most of the settings can be left as-is, all settings such as default and custom aliases are defined in here. However, the important entries are the hostname, real_name and nick(name). These should be changed to whatever you require.

Once this is done, that's pretty much it. You can then connect to an IRC server by either using the /connect IRC command or via the command-line as :

$ irssi -c irc.undernet.org

Once you are connected, you can then issue any of the IRC commands that you might know, such as /join, /msg, /alias etc. This blog entry will not go into them, a quick Google for "IRC commands" or the irssi documentation will give you plenty of resources.

If you are connected to multiple channels, then you can use ALT- to switch between contexts, i.e. ALT-2 might be #oracle, ALT-3 might be #python etc. You can connect to multiple servers via the /connect command, and then Ctrl-X will switch between connections.

If anyone is interested and wants to chat (about linux/oracle/python), then, I use the nick "tackd" and am generally found on #oracle #linux and/or #python on DALnet, Undernet or EFnet (the major IRC networks really).

Quick and easy (tm) Ubuntu Netbook Remix

I've blogged about UNR before, but just thought I'd email just how easy it is to get up and running with this great version of Ubuntu specifically for Netbooks.

1. Download the ISO from http://www.ubuntu.com/GetUbuntu/download-netbook
2. Use the USB Startup Disk Creator to burn it to a USB stick (or burn it to a CD, but you'd need an external CD drive for that, USB sticks are MUCH easier)
3. Boot from the USB stick and install Ubuntu via the installer

That's the main bit. I've just done it in 10 minutes from start to finish.

As additional extras, if you enable the "partner" repositories in Software Sources, then install the "ubuntu-restricted-extras" and "sun-java6-plugin" packages, you'll have all the common software such as flash and java all pre-configured.

Wonderful.

DNS caching in Ubuntu

DNS caching in Ubuntu is really straightforward and can make a big difference to the speed of your general network activities, including browsing. Why? Well, on an average home network, a DNS request can take 40ms+, and given that (unless your on a webpage using direct IP addresses of course) pretty much every actual HTTP request, i.e. one that doesn't result in a local cache get, results in an initial DNS lookup. So, multiply every non-cached object on the webpage by your DNS lookup time, and you can have a big overhead.

Windows has had a local DNS cache for ages now, certainly since XP (and probably before that), but Linux distros have not, for various reasons. In fact, as far as I'm aware, only openSUSE has had a DNS cache enabled by default (the excellent ncsd).

There are many different DNS caching daemons available, but my favourite is dnsmasq. You can find more information about the advanced features of this software here, i.e. you can run it as a fully-fledged DNS and DHCP server for your network. Here I'll explain how to do simple DNS caching with it.

Firstly, install it (obviously). The repository for your distro will almost certainly have it, here I'm using Ubuntu 9.10, and the standard repository.

$ sudo apt-get install dnsmasq

Once it's installed, modify the /etc/dnsmasq.conf, and specify the listening IP address. Here, we're obviously going to use the local host address, so uncomment the following line :

listen-address=127.0.0.1

Now, you need to modify your DHCP configuration to use the local DNS cache first, if there isn't one, then fall back on your main DNS server. So,

$ sudo vi /etc/dhcp3/dhclient.conf

include the "prepend domain-name-servers 127.0.0.1" line

Note, that "prepend" puts the dns server ABOVE everything else, so ensure that the prepend statement above is LAST in the list.

The following is my setup, note I use OpenDNS, hence the IPs 208.67.222.222 and 208.67.220.220.

prepend domain-name-servers 208.67.222.222;
prepend domain-name-servers 208.67.220.220;
prepend domain-name-servers 127.0.0.1;

Now, when you DHCP lease expires the order of DNS servers should be 127.0.0.1 and then your existing DNS setup.

If you don't want to wait until your DHCP lease expires, you need to manually modify the /etc/resolv.conf file (which gets overwritten when the lease expires) :

$ sudo vi /etc/resolv.conf

And add

nameserver 127.0.0.1

to the top of the nameserver lists already in there.

Once this is done, restart your dnsmasq daemon, i.e.

$ sudo /etc/init.d/dnsmasq restart

So, how can you test that all this has had a positive effect? Well, you use the wonderful
dig command. Here's the example from my local machine :

$ dig chaddersblog.blogspot.com

; <<>> DiG 9.6.1-P2 <<>> chaddersblog.blogspot.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23451
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;chaddersblog.blogspot.com. IN A

;; ANSWER SECTION:
chaddersblog.blogspot.com. 2886 IN CNAME blogspot.l.google.com.
blogspot.l.google.com. 294 IN A 216.239.59.191

;; Query time: 31 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Jan 23 10:33:56 2010
;; MSG SIZE  rcvd: 91

$ dig chaddersblog.blogspot.com

; <<>> DiG 9.6.1-P2 <<>> chaddersblog.blogspot.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27521
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;chaddersblog.blogspot.com. IN A

;; ANSWER SECTION:
chaddersblog.blogspot.com. 2884 IN CNAME blogspot.l.google.com.
blogspot.l.google.com. 292 IN A 216.239.59.191

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Jan 23 10:33:58 2010
;; MSG SIZE  rcvd: 94

You can see from the bold sections that the first time I ran the command, the "Query time" was 31 msec (because it didn't have the DNS entry for that address), but the second time it did, so the Query time was instantaneous.

Grub

Haven't posted for a while, so thought I'd go a bit geeky and give some information on the usage of various technologies (since I get asked these types of questions a lot on various IRC / IM channels). My first post will be on the architecture and usage of GRUB (note, the latest versions of Ubuntu etc using GRUB2 which is a complete rewrite of GRUB. I'll try and highlight some of the differences as I'm going along, but I'm still getting my head around GRUB2. For a complete breakdown of the differences between GRUB and GRUB2, then look here), then (at some point in the future) I'll blog about Xorg. These are the two most common queries and, in my opinion, the two most unappreciated pieces of software in existence.

So, what is GRUB? Well, simply put, it's a bootloader. However, that really is simplifying it rather too far. For one, it's the reference implementation of the Multiboot specification on the GNU OS. What this means is that it pretty much defines the de-facto standard for bootloaders. Without GRUB, you wouldn't have UNIX, Linux or even Solaris (since v10).

So, what's a bootloader? Well, generally speaking, when a computer is powered on, the BIOS reads the first 512 bytes of the hard disk (known as the Master Boot Record (MBR)). The MBR contains code known as the "bootstrap" program. Control is then passed to the bootstrap program. This is known as "Stage 1".

Stage 1 generally does little more than just load and execute code located at some other point in some partition on the hard disk, known as "Stage 2". (Actually, there is a situation where the GRUB MBR, for example, reads code from the 30K immediately following the MBR. This is generally termed "Stage 1.5". GRUB2, however, has eliminated this stage completely.).

Stage 2 generally presents an interface to the user (dependent on settings) to decide which OS to load. GRUB2 doesn't do this if there's only one OS installed, although you can get to the menu if you press the shift key. In situations where the OS installs a completely different bootloader into the MBR (such as Microsoft Windows), GRUB handles this by "chainloading", which creates a copy of the alternative MBR, and then executes that as though is was execute by the BIOS. It is this which allows GRUB to boot Windows (and even allow multiple versions of Windows).

So, how do I install GRUB? Well, it's obviously the default on most Linux distributions, but if the question really is "how do I install GRUB when it's not already installed?", the answer is to use a Linux distribution (any really small one will generally do) and use the "grub-install" routine. This will then install stage1 into the MBR, and the stage2 code into a usable partition, for example :

# grub-install /dev/hda

Installs GRUB stage1 onto the MBR of /dev/hda which is, typically, the first IDE disk. See below for typical device mappings.

# grub-install --root-directory=/boot /dev/hda

Installs GRUB stage1 onto the first disk, with the boot manager code in /boot.

So, how do I configure GRUB stage 2? Well, this is the power of GRUB really. It's all done by a single config file on the "usable" partition defined when using grub-install. In Debian, the file is /boot/grub/menu.lst, whereas in Fedora (and Gentoo) it's in /boot/grub/grub.conf. In GRUB2, it's been replaced by /boot/grub/grub.cfg (although, this is not meant to be edited, not even by root, GRUB2 provides /etc/default/grub which is merged into /boot/grub/grub.cfg when the update-grub command is used).

In terms of GRUB terminology of how your hard drive partition(s) are denoted, they follow the layout of "hd" followed by a number, a comma followed by the partition number (starting at 0 in GRUB, 1 in GRUB2), i.e. :

Primary Master=hd0,0
Primary slave=hd1,0
Secondary Master=hd2,0
Secondary slave=hd3,0

Therefore, in GRUB, hd0,0 denotes the first partition on the primary master, hd0,1 the second, and so on. In GRUB2, these would be hd0,1 and hd0,2 respectively.

A typical GRUB menu.lst looks something like this :

#
# GRUB's menu.lst config file, with examples for various OSes
#

# Default: boot the first entry.  Problems? boot the second entry.
default 0
fallback 1

title  Ubuntu hardy (development branch), kernel 2.6.24-12-386
root  (hd0,0)
kernel  /boot/vmlinuz-2.6.24-12-386 root=UUID=d1562435-e820-4486-99b8-eaad92e502cd ro clock=pit acpi=off noapic nolapic
initrd  /boot/initrd.img-2.6.24-12-386
quiet

title  Ubuntu hardy (development branch), kernel 2.6.24-12-386 (recovery mode)
root  (hd0,0)
kernel  /boot/vmlinuz-2.6.24-12-386 root=UUID=d1562435-e820-4486-99b8-eaad92e502cd ro clock=pit single
initrd  /boot/initrd.img-2.6.24-12-386

title        Microsoft Windows XP
root        (hd0,1)
rootnoverify (hd0,1)
makeactive
chainloader    +1

title Linux from second HDD
  kernel (hd1,0)/vmlinuz root=/dev/hdb1

Here showing the first partition containing an installation of Ubuntu (Hardy Heron = 8.04) on the first partition (hd0,0) and an installation of Microsoft Windows XP on the second partition (hd0,1), as well as others. The "chainloader +1" command, specifies that (as I mentioned above) GRUB should pass control over to the subsequent boot manager specified in the first sector of the relevant partition.

The version of GRUB installed can be found easily (here on my Ubuntu 9.10 installation, which uses GRUB2 by default when newly installed)

chadders-desktop:/boot/grub$ grub-install -v
grub-install (GNU GRUB 1.97~beta4)

GRUB (now termed "GRUB legacy") would have a version of 0.97 or similar.

Obviously, GRUB (and certainly GRUB2) is an extremely complex beast. A full breakdown of all commands etc is explained at https://help.ubuntu.com/community/Grub2.

More musings on Xubuntu

I'd been getting more and more annoyed with some bugs in Xubuntu 8.04 ( Hardy Heron ), particularly with USB drives being detected twice and my DVD+RW drive not working correctly. I also had occasional problems with wireless not detecting the access point, or if it did, it didn't connect properly using WPA (although, WEP was usually fine).

They weren't major bugs, but they were annoying. I'd also realised that I hadn't used my Ubuntu 8.04 partition since I installed it. So, a week ago, I made the jump to Xubuntu 8.10 ( Intrepid Ibex ), completely repartitioning the drive and having solely 8.10.

To say I'm impressed would be an understatement. I'm also glad that I upgraded my Nvidia card a while ago, since previous nvidia cards aren't supported under the 8.10 kernel (2.7.1.27), so they regress to the "nv" driver which doesn't support 3D acceleration. Fortunately, I don't have any of those problems.

So, let's run through the checklist :

1. USB drive(s) detected correctly? check.
2. DVD drive(s) detected correctly? check.
3. Wireless functionality working? check.
3. Anything else that was annoying me with 8.04 fixed? check.

Astonishing. The speed increase of 8.10 is also breathtaking. It won't be long, and more and more people will be on Linux, and at that point, the world will be a better place.

Not much to say...

Wow... how long has it been since I posted anything? Frankly, this post hasn't got much to say either. However, I just wanted to post a link to possibly the most beautiful landscape picture I have ever seen...
http://flickr.com/photos/dotdoubledot/1281864495/sizes/l/
Astonishing.

Hardy Heron and NVidia

Well... I've finally taken the leap from Feisty to Hardy, and wow... I'm glad I did. The performance difference alone is worth the money. I've currently got Xubuntu and Ubuntu dual booting (lovely tool GRUB!).

Most stuff worked straight out of the box, apart from one thing. The nvidia drivers had a problem initialising, the "nv" drivers worked ok, but not the nvidia ones, and, unfortunately, you need the nvidia ones for 3D acceleration to work, and, of course, you need 3D acceleration to do Second Life etc. etc.

The problem I was having was that every time I installed the nvidia 96.43 drivers (I have a GeForce4 440MX), the display looked rubbish (to the point where I couldn't even read the text). So, a quick look at /var/log/Xorg.0.log showed the following error :

Failed to initialize GLX extension (Compatible NVIDIA X driver not found)

It all basically (after about two days of investigation) found that there's a bug in Hardy which doesn't detect the monitor correctly, so you basically have some kind of basic monitor which doesn't have the correct resolutions, and this throws the driver completely. Anyway, a quick addition of :

SubSection "Display"
Viewport 0 0
Depth 24
Modes "1024x768" "800x600" "640x480"
EndSubSection

to the xorg.conf file and it sorted it.

Other than that, no problems, and more than happy.

Data breaches

I blog quite a bit about the amazing ineptitude of various government departments with regards to data security. On the subject, FlowingData has published a picture of the 10 largest data breaches since 2000. It's fascinating reading.

Virtualisation (take 3)

Okay, one minor hiccup overcome. I attempted to RDP into one of my VirtualBox VMs yesterday, and found that I couldn't, for one reason and one reason only. By default, VirtualBox (on Linux, anyway) sets up the VM running via a NAT connection, and for those in the know, this basically piggybacks on the current IP address, rather than getting it's own MAC / IP address. This is fine for VMs where you only ever want to communicate "out", i.e. surf the web from the VM etc., but when you want to use them as "servers", i.e. to communicate "to" them, it's no good. And this is precisely what I pretty much exclusively use them for.

Well, it's a bit bizarre to set up a bridging network in Ubuntu, but it is possible, and actually works really well, it's just a little bit more fiddly than VMware, which does it by default.

I'm not going to go into too much detail (if anybody wants the detail, then by all means email me), but you basically have to set up some custom bridging network adapters, (via the bridge-utils package), assign the bridged adapter an IP address, and then associate the VM with the adapter. Now, I've currently only got it working via DHCP, and, unfortunately, my router's not fantastic at this, so, in the past, I've tended to manually assign the IP addresses to my network nodes, but it seems to be working at the moment, fingers crossed.

I suppose a downside, and it's whether you consider it a downside or not, is that, using this approach, you have to have a seperate bridge adapter for each VM, since VirtualBox can't even start a VM which is sharing an adapter, but I can live with this at the moment, even though I have 5 adapters defined. Fortunately, a quick

$ ifconfig  down

disables the currently unused adapter.

So, what's the state of play?

Well, I currently have the following VMware VMs :

1. MediaWiki running via a JumpBox
2. Oracle Enterprise Linux 5
3. Minix

And the following VirtualBox VMs :

1. Ubuntu 7.10
2. Ubuntu 7.10 JeOS
3. Google gOS
4. Damn Small Linux 4.2.2

Sad, moi? ;-)

Virtualisation (take 2)

Well, hope everyone had a nice christmas and are looking forward to a great new year. I certainly am.

Anyway, just thought i'd blog about a development in my quest for the perfect virtualisation approach. People who know me know that I am passionate about the use of Virtual Machines (VMs) for practically everything I do. I just love the idea of having my main OS (currently Ubuntu 7.04), but being able to run Ubuntu / OpenSUSE / Damn Small Linux (DSL), or, indeed, any other OS that has the feature that I require (such as my 11g instance on Oracle Enterprise Linux, or my Ubuntu 10g XE database, again all running as RDP'able VMs).

In the past, I've always used VMWare Player, and got my head around the creation of VMDK and VMX files by the use of QEMU. However, recently, I've started to get my head around VirtualBox, an open-source project for managing your VM server, and, I must say, how impressed am I? The software is an absolute breeze to install / run, and the creation of the VMs through the use of wizards is a bit funny to start off with, but once you understand it, it's by far the easiest way to create all the fiddly configuration settings for a given VM, such as hard disk size, memory usage etc.

One problem with simply using VMWare player, as opposed to VMWare Workstation (or similar), is that you have to basically manage your VM server like you're managing file(s) and window(s), i.e. through a file manager. However, what I like about VirtualBox is that the software is
full enterprise-class software, having a VM manager, the full works! And, the fact that it's one of the only software which fully supports Remote Desktop Protocol (RDP) (although, you can do this with other VM products, they don't fully support it... not yet anyway). For a little bit more information on this, see here.

Of course, it's OSS as well, and extremely light-weight.

So, as I'm writing this, Ubuntu 7.10 is being installed. I'll let you know how I get on with my future VM projects in future blog posts.

Have a happy new year!

Can you adam and eve it?

I just DO NOT believe this... maybe it's just that things are getting put in to the public domain more because of recent events, and granted that this is the Royal Mail, and not, strictly speaking, some inept government department, but how long do we have to live in a country with such a woefully and grotesquely inept attitude towards data security awareness? How can things just go so badly wrong all the time (as seems to be the case)?

Mr Brown is probably thinking "why did I ever take this job?". The rest of us are thinking exactly the same thing.....

Here we go again

Almost as a response to my previous blog, here we go again. Now, nine NHS trusts have "lost" data (one, according to the Sunday Mirror, involved the details of 160,000 children). Utterly unbelievable. The stupidity, ignorance and ineptitude staggers belief!

Data security (or lack thereof)

My previous post sparked a minor flurry of comments (cheers Tim) about top-n lists, but I was shocked tonight (literally about half an hour ago) to find ANOTHER piece on the newswire about a government department / local authority misplacing yet another set of valuable documents and information in some form of internal postal system. Granted, with this one there was not any way that innocent individuals information could be gleaned, however, this is another example of this systemic ineptitude and total ignorance of basic data security principles when dealing with the important issue of personal data.

As somebody who has made a career in the data management industry, with experience in large scale data warehousing and business intelligence projects (even presenting a series of talks of data security principles), this is something I feel I can comment on with some authority. In a sense, Gordon Brown has been unlucky. He took over just as two major data security violations took place in government departments, namely the HM Revenues and Customs (HMRC) and the Driver and Vehicle Licensing Agency (DVLA), and it would be naive to think that government departments and local authorities all work in some kind of revolutionary way such that the Prime Minister knows EVERYTHING that goes on, of course not, he relies on his junior ministers / secretaries of whatever stupid government department's turn it happens to be this time, nor would it have been any different under any other government party, since it's still the same employees who work in the HRMC or DVLA, but just as the managing director of a company is liable for corporate mistakes by employees (generally), I'm afraid the buck stops with Mr. Brown. This has certainly not been a good few months for the Labour party, especially with the other things such as questions over funding rear their ugly head.

I'm afraid, though, as much as I understand about the way that this country is structured, I have no sympathy for any of them. It, quite simply, staggers belief the sheer scale of some of the data security violations (most of which definitely contravene the Data Protection Act 1998).

So, here's the top 10 list of government ineptitude, in ascending order of total inept ignorance and each showing a total lack of respect for the important of security of personal information :

10. Hundreds of HMRC documents (including VAT returns and personal details) found by a BBC reporter on a Nottingham street

9. An ex-employee for the Department for Work and Pensions (DWP) having thousands of benefit claimant details (fortunately, non-financial) on 2 CDs for over a year after leaving

8. The DVLA sending 1,215 questionnaires with personal details out and 100 going to the wrong addresses.

7. The Scottish Government losing pension statements for 200 people (fortunately, found again a day later)

6. HMRC (again) involved in a postal mix-up regarding the personal details of 50 people being accidentally sent to a training company in Dundee.

5. Domestic violence victim details (names and addresses) faxed by mistake to local shops in Glasgow by Strathclyde Police

4. 15,000 Standard Life customer details "lost" in transit from HMRC (again)

3. Ruth Kelly admitting that the details of three million learner drivers were "lost" by a contracted company based in Iowa, contracted to the DVLA.

2. The Department of Work and Pensions (DWP) sending the pension details of 26,000 pensioners to the wrong addresses

and, of course.....

1. By far the most staggering, 25 million child benefit records from HMRC (yes, again!) to the National Audit Office (NAO) on 2 CDs being lost in internal mail.

As somebody who has personally been affected by number 1, and also aware that their financial and personal details on a (get this!) "password-protected" set of 2 CDs generated by a junior official and sent by UNREGISTERED courier, seemingly with the full knowledge of much more senior managers (even though the chanceller, Alistair Darling, blamed the fiasco solely on this "junior official"), this shows another absolutely astounding lapse of data security by a government. In any other context, senior heads would roll for this, certainly if this country was a private company, no one would touch us with a ten-metre cattle prod. To add insult to injury, the Information Commissioner, Richard Thomas, has stated that the government did break the Data Protection Act 1998 by not safeguarding sensitive data. However, the Information Commissioner's Office (ICO) lacks any kind of teeth, since the maximum fine that a Data Protection Act 1998 violation can incur is, wait for it...., £5,000. Yeah, that's right, only five grand, although, to be fair, Mr. Darling has promised that the ICO will get more powers of prosecution for future data protection violations. What this will mean for us citizens / consumers, we'll have to wait and see.

Rectification and early 80s animation

I haven't blogged for a while (that's an usual line for me.. not), so thought best had.

Firstly, I'd like to answer a little bit of criticism about one of my previous posts on top 10s. Basically, my mate Tim Hall decided to have a little pop at me about it being quite sad that I posted my top-10 software apps, which is rich really considering some of his posts have been about his top 30 lyrics from songs that noone (apart from 23 people in Crewe Limelight) have ever heard of! ;-)

However, I think it's fair to say that he is probably right, as much as it pains me to say it, and, as penance, will obviously be posting an updated top-10 list of apps, but this time concentrating on linux, again in descending order to importance. Can't wait for the next post on Tim's blog, i.e. the top-10 signal buffers on class-5 rolling steam engines... ;-)

So, to continue, and since my decisions have changed dramatically over the last few months, not least of which is because of my continuing involvement in virtualisation products

• 10. Skype - Now that video is part of the linux client


• 9. Google Earth


• 8. Second Life - This has fallen somewhat, since I go on this far less than I used to, and I wish Linden would get around to sorting out the performance problems that the linux client has


• 7. Envy - A Python / PyGTK app for automating the installation of NVidia drivers for Debian-based distros


• 6. Evolution mail


• 5. Ubuntu Terminal (where would we be without the command line?)


• 4. LifeRea


• 3. VMWare


• 2. Qemu


• 1. Firefox

So, what else have we been up to? Well... of course, it's coming up to Christmas, and the kids are so excited it's unreal. In fact, just watched The Snowman, one of the most beautifully animated short films of all time. The amazing thing about this film is that it never looks or feels dated (despite being made in 1982), and it's a testament to both the author, Raymond Briggs, and the animators that, even now, kids such as mine will watch it and still be enthralled (as I was at their age).