Odds & sods

I've blogged building my own NAS/home server before, see here, here, here and here.

After a few months, I think it might be time for a little interim update.

Unfortunately in its original incarnation, the server wasn't as stable as it should have been given my previous experience of FreeBSD. For some reason, it would crash every few weeks and sometimes even hang on reboot. Not good, especially as it happened a few times while I wasn't home. I guess I should have heeded the warning about the zfs integration being experimental...Things got worse when I added a wireless card and retired my access point. Roughly around this point in time I got fed up with this enough to go back and start building an OpenSolaris VM to try out a mail server setup similar to the one I'm running on FreeBSD.

Before I got anywhere with this, FreeBSD 8.0 came out, so I upgraded. Zfs had be promoted from experimental, the wireless stack has been overhauled, etc pp. The stability problems disappeared and the machine has been utterly reliable since then. Where before, trying to use Time Machine from to back up my MacBook via the wireless network was a good way to a 50% chance to crash the server, it now "just works". This is where I wanted to get and I've now got there. Performance also seems to have improved - copying large files from the server to my Windows 7 machine sees a reliable 78MB/s via my Gigabit network now.

I've still got a couple of small changes I want to make to the machine - for example, I've got 4GB of RAM that I want to put into the machine. This should enable zfs readahead which should give me further performance improvements. I also plan to add two more fans to blow cold air over the hard drives to keep them happy and working for longer.

If I built another machine, I would probably get another motherboard. Not that there is anything particularly wrong with the one in the machine, but the CPU fan speed control only goes down to 70%, so no wonder that the CPU fan is noisier than it should be.

Was it worth it? Overall I'd say yes, although I probably should have stuck to tried and tested technology (either using FreeBSD's built-in RAID5 or use OpenSolaris with zfs). This caused unnecessary problems at the beginning and pushed up the cost as I was dithering between either. Next time I probably set up the server on OpenSolaris and run the mail server on FreeBSD in a VM running on OpenSolaris. Given that the current configuration is working, I leave it alone for the time being though.

A couple of links to other people's interesting posts I've come across in the last few days.

Raymond Chen on "There is no law that says meetings can't end early". I wish more people would take this advice to heart, but that's been on my Christmas wishlist right next to "create an agenda for a meeting and stick to it".

Interesting blog post regarding the difference between technology for technology's sake and using technology to create something. For the record, I tend to be more interested in doing something with my technology "toys" rather than having them purely for having them. That's probably one of the reasons that I tend not to have the latest, greatest and fastest machines (or phones, cameras etc) but rather buy quality kit and use it for a few years instead.

Do programmers still buy printed books? I know I do, but I'm getting a little more choosy. I've bought a couple of books recently that I'd classify more as a waste of a perfectly good tree. That's unfortunately not unusual for tech books, there are a couple of brilliant ones out there and an awful lot of, how shall I call it, rubbish. I'm also one of those throwbacks to an earlier age who prefer to hold a physical book in their hands rather than stare at a screen. The reason Antonio mentions - switching between book reading mode and computing mode - certainly has something to do with it; after all, if you're immersed in a book you probably don't check your email every thirty seconds whereas I'd be tempted to do that if I were to read the book on a computer. Reading on the iPhone is something I'd rather not do, my eyesight is bad enough as it is...

Antonio's book list his previous blog post also looks very much like I should update my Amazon wish list.

If you're conducting an interview using your mobile phone (that's a cell phone to the readers on the other side of the pond), at least go somewhere with good reception so the interviewer can hear what you're saying. If you don't you're probably not going to make a very good impression if the interviewer has a hard time understanding half of what you say...

I've done some more performance testing and while I'm not 100% happy with the results, I decided to keep using FreeBSD with zfs on the server for the time being. Pretty much all measurements on Linux (both a recent Ubuntu Server and CentOS 5.3) showed lower performance and while OpenSolaris is a lot faster when it comes to Disk I/O and thus would have been my first choice for a pure NAS, the effort in porting my current mail server configuration would have resulted in the server being ready sometime in 2010...

I might be exaggerating this a little but the effort of building amavisd-new is non-trivial due to its large number of dependencies. If packages or at least some sort of package build system like the FreeBSD ports is available but building it from scratch and ensuring that all the necessary small Perl components are present isn't trivial. So FreeBSD it is for the time being. I will still be looking into rolling a few packages for OpenSolaris for at least some of the tools I'd like to see on there as the thought of running my home server on what is essentially an enterprise-class OS is very appealing to me.

Unfortunately the excitement from seeing OpenSolaris's disk performance died down pretty quickly when I noticed that putting some decent load on the network interface resulted in the network card locking up after a little while. I guess that's what I get for using the on-board Realtek instead of opening the wallet a little further and buy an Intel PCI-E network card. That said, the lock-up was specific to OpenSolaris - neither Ubuntu nor FreeBSD exhibited this sort of behaviour. I could get OpenSolaris to lock up the network interface reproduceably while rsyncing from my old server.

This gave me the opportunity to try the third option I was considering, Ubuntu server. Again, this is the latest release with the latest kernel update installed. The four 1TB drives were configured as a RAID5 array using mdadm. Once the array had been rebuilt, I ran the same iozone test on it (basically just iozone -a with Excel-compatible output). To my surprise this was even slower than FreeBSD/zfs even thought the rsync felt faster. Odd that.

Here a few pretty graphs that show the results of the iozone write test - reading data was faster, as expected, but the critical bit for me is writing as the server does host all the backups for the various other PCs around here and also gets to serve rather large files.

First, we have OpenSolaris's write performance

FreeBSD with zfs is noticeably slower, but still an improvement over my existing server - that one only has two drives in a mirrored configuration and oddly enough its write speed is about half of the one on the new server:

FreeBSD with the geom-based raid is slower but I believe that this is due to the smaller number of disks. With this implementation you need to use an odd number of disks so the fourth disk wasn't doing anything during those tests. Not surprising that the overall transfer rate came in at roughly 3/4 of the zfs one.

Ubuntu with an mdadm raid array unfortunately brings up the rear. I was really surprised by this as it comes in below the performance of the 3 disk array under FreeBSD:

One thing I noticed is that while the four Samsung drives are SATA-300 drives, the disk I reused as a system drive is not. I'm not sure if that does make any difference but I'll try to source a SATA-300 disk for use as the system disk to check if that makes any difference at all. I'm also not sure if FreeBSD and Linux automatically use the drives command queueing ability. On these comparatively slow but large drives that can make a massive difference. Other than that I'm a little stumped; I'm considering the purchase of the aforementioned Intel network card as that would hopefully allow me to run OpenSolaris but other than that I'm open to suggestions.

The good news is that the hardware seems to be behaving it for a while now and everything appears to Just Work. FreeBSD makes things easy for me in this case as I'm very familiar with it so I only spent a few hours pretty much getting everything set up. So far, so good.

I've rsynced most of the data off the old server and while doing that I already had this nagging feeling that the transfer seemed to be well, a little slow. Yes, I transferred several hundred Gigabytes but nevertheless I thought it would take less time, so this morning I did a few performance tests using Samba 3.3 on the new server. Turns out the performance was around 12-25 MB/s and that's a lot less than I expected. In fact, that's a little too close to 100Mb network performance for me liking - even without using jumbo frames I would think that a straight data transfer would be faster than this. Unfortunately I had introduced a few unknowns into the equation, including using the FreeBSD implementation of zfs. I am not saying that it is the culprit but I'll start with server disk performance measurements before I do anything else.

Ah well, looks like the old server needs to do its job a little longer then.

Update: Running iozone on the machine itself suggests that the write speed of 12MB/s-13MB/s I was seeing via Samba are more or less the actual transfer rate to disk on the zfs software raid array. I'm not sure if that's a limitation on my hardware or a problem with the (still experimental) zfs implementation on FreeBSD. Nevertheless this is a little on the slow side to put it mildly (the four disks should easily saturate a gigabit network link) so I guess it's time to wipe the zfs config and use FreeBSD's native RAID5 instead for a check if that hypothesis is holding water. If it's not then the bottleneck is somewhere else.

Update II:FreeBSD's graid3 unfortunately needs an odd number of drives which left one of the 4 Samsung idle. Same tests suggest that the performance was 1/4 less than FreeBSD zfs with four disks so it seems that it was driving it as fast as it can. Hmm. Just to get a comparable figure I threw OpenSolaris on the box, again with the four 1TB drives as a single zfs tank with a single directory structure on it. With the default settings, the same iozone tests are nudging write speeds over 35MB/s. Not as fast as I expected by a long shot but noticeably better at the expense of a more painful configuration. Something to think about but I think I need to test a few other configurations before I can really make up my mind as to which way I want to go.

Up to now I've mostly been using recycled workstations as my home mail, SVN and storage server. Nothing really wrong with that as most workstations are fast enough but I'm running into diskspace issues again after I started backing up all the important machines onto my server. That's especially annoying as I started using Time Machine on my iMac and now haven't got enough space left on the server to also back up the MacBook. Time Machine is great as a backup solution simply because it is so unobtrusive and it appears to just work.

Inspired by an article in the German magazine c't, I decided that instead of finding another used desktop machine to recycle, I was going to build a proper home server with a few additional bells and whistles. I'm still using mostly desktop parts as I can't really justify the expense and noise of "proper" server components but I tried to select decent quality parts. Here's the hardware list:

Yes, this is another one of these "put it on my blog lest I forget what I did" posts.

There are quite a lot of blog posts about this, pretty much all of them centred around the required change to the system preferences that you need to make in order to use the unsupported devices (ie, not a Time Capsule or another Mac). I'm currently backing up onto a FreeBSD server via Samba after following the instructions here. The command line needed to create the sparse bundle used to hold the backup didn't work for me on Mac OS X 10.5.6 but with the help of the manpage, I created the following one that worked for me:

hdiutil create -size $SIZE -fs HFS+J -type SPARSEBUNDLE -volname $MACHINE_MAC_ADDRESS.sparsebundle $MACHINE_MAC_ADDRESS.sparsebundle

And yes, I realise that it probably wasn't necessary to give the volume and the sparse bundle directory the same name...

I've mentioned this in a couple of posts recently - basically posting to this serendipity blog via XML-RPC strips out the '<' and '>', which in turn renders all HTML formatting that I post from external clients pretty much useless. It turns out that I triggered the whole problem myself when I recently upgraded libxml2 and PHP on this server and ran straight into a known PHP/libxml2 interaction bug. Took me a few hours wandering around the serendipity forums to find the hint but I got there in the end.

I guess I'll have to wait until an upgraded PHP is available for the server to fix the problem.

I just came across this article, which was mentioned in the weekly ACM CareerNews email: The Best IT Job Boards.

Obviously given the source of the article and the location of ACM, it is a tad US-centric, but the general advice seems to be rather useful. Now if someone had a list similar to this for the UK and other European countries, that would be really useful.

Actually, if you are an ACM member and don't get the CareerNews email, get yourself on the subscription list. I found it a very useful resource.

It seems that I have jumped on the tail end of yet another bandwagon, in this case, replacing a not-so-perfectly running Wintel laptop with a MacBook. While I do like to play with various operation systems, my machines are usually dual-boot Windows XP/Linux or Solaris machines as I'm pretty comfortable with those OSs, have all my tools for them or know where I can get them and they work. Well, most of the time they do. I was just getting fed up with the oddities of the various systems that were mostly but not completely right. And IMHO it's those little things that tend to annoy me when I've got five minutes to whip out the laptop, check my mail and hopefully write a reply or two before wandering off to the next conference session. A good friend of mine - himself a diehard Linux hacker - had made the Mac OS X switch for his desktops and laptops a while ago and was happy to answer my questions. So a few days before I flew out to SD West 2008, I drove to the Apple store and returned with this:

I obviously had to drop all my usual tools on there including various emacsen. I knew the system already came with a textmode version of a recent GNU Emacs, but given that I use the Emacs to do a lot of other things besides writing code (like getting the skeleton of my blog posts out or reading/posting to newsgroups using Gnus), I looked at the various GUI versions available. While Aquamacs appears to have the better integration into the Mac GUI, I so far prefer Carbon Emacs for the simple reason that it seems to behave more like the Unix Emacsen I'm used to. Aquamacs is fine and does have a little more visual appeal, but I didn't like the way it was popping up windows all over the place in Gnus. So, Carbon Emacs it is for the time.

With the basic install complete, I took off for the conference and the MacBook just did what it said on the tin. For the first week, I only ever put it to sleep, it happily switched between networks where necessary (and without too much user input). So top marks all around, even though I must admit that I did a very non-geeky thing and bought the Mac OS X Leopard edition of the 'missing manuals' series. Given that the last Mac I had was an SE/30 I used as a mail- and newsreader, I had a few generations of MacOS to catch up with.

Oddly enough there has been a number of people who recently switched from various other platforms to Macs, so I guess that Apple is doing something right with their combination of a well thought-out GUI and UNIX underpinnings.

In fact I did like working on the MacBook so much - with the exception of the small screen, but that was intentional as I do use my laptops on trains, planes and just about everywhere else - so I went out and bought an iMac as a desktop development machine instead of the bigger monitor I was planning to purchase anyway.

... you might not be a redneck, but you probably want to find a new job instead.

Have a look a Joel Spolsky's 'checklist' for a doomed project.

Anyway I had a good chuckle reading it and it does contain some rather useful advice...