In April this year I attended the zSeries Special Interest Group miniconf[1], part of the greater Independent Oracle Users Group (IOUG) event COLLABORATE 11.  I was amazed to discover that there are actually Oracle employees whose job it is to work on IBM technologies — just like there are IBM employees dedicated to selling and supporting the Oracle stack.  Never have I seen (close-up) a better example of the term “coopetition”.

On my return from the zSeries SIG and IOUG, I’ve become the local Oracle expert.  However, I’ve had no more training than the two days of workshops run at the conference!  The workshops were excellent (held at the Epcot Center at Walt Disney World, no less!) but they could not an expert make.  So I’ve been trying to build some systems and teach myself more about running Oracle.  I thought I’d gotten off to a good start too — I’d installed a standalone system, then went on to build a two-node RAC.  I communicated my success to one of my sales colleagues:

“I’ve got a two-node RAC setup running on the z9 in Brisbane!”

“Great!  Good work,” he said.  ”So the two nodes are running in different LPARs, so we can demonstrate high-availability?”

” . . . ”

In my haste I’d built both virtual machines in the same LPAR.  Whoops.  (I’ve fixed that now, by the way.  The two RAC nodes are in different LPARs and seem to be performing better for it.)

Over the coming weeks, I’ll write up some of the things that have caught me out.  I still don’t really know how all this stuff works, but I’m getting better!

Links:

IBM System z: www.ibm.com/systems/z or www.ibm.com/systems/au/z

Linux on System z: www.ibm.com/systems/z/os/linux/index.html

Oracle zSeries SIG: www.zseriesoraclesig.org

Oracle Database: www.oracle.com/us/products/database/index.html

[1] Miniconf is a term I picked up from linux.conf.au — the zSeries SIG didn’t advertise its event as a miniconf, but as a convenient name for a “conference-in-a-conference” I’m using the term here.

After I got the Fedora system working, I thought it would be a good idea to have other systems in the complex using SSL/TLS also.  The process was moderately painless on a SLES 10 system, but on the first SLES 11 system I went to YaST froze while saving the changes.  I (foolishly) rebooted the image, and it hung during boot.  Not fun.

After a couple of attempts to fix up what I thought were the obvious problems (each attempt involving logging off the guest, connecting its disk to another guest, mounting the filesystem, making a change, unmounting and disconnecting, and re-IPLing) with no success, I went into /etc/nsswitch.conf and turned off LDAP for everything I could find.  This finally allowed the guest to complete its boot — but I had no LDAP now.  I did a test using ldapsearch, which reported it couldn’t reach the LDAP server.  I tried to ping the LDAP server by address, which worked.  I tried to lookup the hostname of the LDAP server, and name resolution failed with the traditional “no servers could be reached” message.  This was odd, as I knew I’d changed it since it was pointing to the wrong DNS server before…  I could ping the DNS by address, and another system resolved fine.

I thought it might have been a configuration problem — I had earlier had trouble with systems not being able to do recursive DNS lookups through my DNS server.  I went to YaST to configure the DNS Server, and it told me that I had to install the package “bind”.  WHAT?!?!?  How did the BIND package get uninstalled from the system…

Unless…  It’s the wrong system…

I checked /etc/resolv.conf on a working system and sure enough I had the IP address wrong.  I was pointing at a server that was NOT my DNS server.  Presumably the inability to resolve the name of the LDAP server I was trying to reach is what made the first attempt to enable TLS for LDAP fail in YaST, and whatever preload magic SLES uses to enable LDAP authentication got broken by the failure.  Setting the right DNS and re-running the LDAP Client module in YaST not only got LDAP authentication working but got me a bootable system again.

A simple fix in the end, but I’d forgotten the power of the resolver to cause untold and unpredictable havoc.  Now, pardon me while I lie in wait for the YaST-haters who will no doubt come out and sledge me…  :-)

>>> Unpacking source…
>>> Unpacking traceroute-2.0.15.tar.gz to /var/tmp/portage/net-analyzer/traceroute-2.0.15/work
touch: setting times of `/var/tmp/portage/net-analyzer/traceroute-2.0.15/.unpacked’: No such file or directory

…and the emerge error output.  Took me a little while to get the answer, but it was (of course) caused by a new version of something that came in with the system update.  This bug comment had the crude hack I needed to get back working again, but longer-term I obviously need to fix the mismatch between the version of linux-headers and the kernel version my VPS is using (it’s Xen on RHEL5).

One of the problems I needed to fix was the service check for IAX connections into my Asterisk box.  The script (the standard check_asterisk.pl from the Nagios Plugins package) was set up correctly, but it would fail with a “Got no reply” message.

I started doing traces and “iax2 debug” in Asterisk, but got nowhere — Asterisk was rejecting the packet from the check script.  Finally I decided to JFGI, and eventually I found this page with the explanation and the fix.  Basically, sometime in the 1.6 stream Asterisk toughened up security on the control message the Nagios service check used to use.  Thankfully, at the same time a new control message specifically designed for availability checking was implemented, and the fix is to update the script to use the new control message.  Easy!

BTW, while on Nagios, I got burned by the so-called “vconfig patch” which broke the check_ping script.  I’ve had to mask version 1.4.14-r2 and above of the nagios-plugins package until the issue is fixed.

I had always resisted runing Linux on the PowerMac, thinking that the last thing I needed was yet another Linux box in the house. I had tried a couple of times, but it was in the early days of support for the liquid cooling system in the dual-2.5Ghz model and those attempts failed dismally. I figured that by now those issues would be resolved and I would have a much better time.

I assumed that Yellow Dog was still the ‘benchmark’ PPC Linux distro, so I went to their site. I saw a lot of data there about PS3 and Cell; it seems that YDL is transitioning to the cluster and/or research market by focussing on Cell.

The next thing I discovered is the lack of distributions that have a PPC version, even as a secondary platform. My old standby Gentoo still supports PPC, as does Fedora (I think: I saw a reference to downloading a PPC install disk, bit didn’t follow it), but every other major distro has dropped it — openSUSE, for example, with their very latest release (their download page still has a picture of a disc labelled “ppc”, but no such download exists, oops). I guess that since the major producer of desktop PPC systems stopped doing so, the distros saw their potential install base disappear. Unfortunately for those distros, I can see the reverse happening: now that Apple has fully left PPC behind, plenty of folks like me who have moderately recent G4 and G5 hardware and who still want to run a current OS will come to Linux looking for an alternative… I guess time will tell who is right on this one.

So I went to install Gentoo, and to cut a long story short I had exactly the same problem as before: critical temperature condition leading to emergency system power-off. I found that if I capped the CPU speed to 2Ghz I could stay up long enough to get things built, but then the system refused to boot because it couldn’t find the root filesystem. Probably something to do with yaboot, SATA drives and OpenFirmware. So again I’m putting it aside.

My next plan was to treat it as a file server. Surely a BSD would support my G5 hardware: after all, Mac OS X is BSD at heart… Well, no. FreeBSD has no support for SATA on ppc, OpenBSD specifically mentioned liquid-cooled G5s as having no support, and I don’t think I saw any ppc support on NetBSD more recent than G3 [1].

This is one of the things that annoys me about the computer industry: that somehow it’s okay to so completely disregard your older releases. What if the automotive industry worked that way?

So I may yet try Fedora, or give the game away for another year or so and see what the situation looks like then.

[1] I may have mixed up a couple of these details.

Edit: Gentoo’s yaboot has managed to make it so that I can’t boot Mac OS X on the machine any more.  Oh dear.

(The other method of combining VLAN-tagging and bridging is to bridge the physical interface first, then create VLANs on the bridge.  I couldn’t work out how to get VLAN-unaware guests attached to this kind of setup, and it didn’t work for me even to give IP access to the host using a br0.100 for example.  Still, it must work for someone as it’s written about a lot…)

I realised that from particular virtual machines I needed to get access to the VLAN tags — I needed VLAN-awareness.  Now I knew up-front that the way I could do this was to just throw another NIC into the machine and either dedicate it to the virtual guest or set up a bridge with VLAN tags intact.  I really wanted to exhaust all possible avenues to solve the problem without throwing hardware around (as I’ve been doing a bit of that recently, I have to admit).

First, I tried to use standard Linux bridges as a solution, but discovered that an interface can’t belong to more than one bridge at a time, which put paid to my plan to have one or more VLAN-untagging bridges and a VLAN-tagged bridge.  I figured it could be done with bridges, but I envisaged a stacked mess of bridge-to-tap-to-bridge-to-tap-to-guest connections and decided that wasn’t the way to go.

Next I checked out VDE, which I had first seen a couple of years ago — but something gave me the impression that VDE either wasn’t really going to give me anything more than bridging would, or was not flexible enough to do what I needed.  I like the distributed aspect of VDE (the D in the name) but I’d rarely use that capability so it wasn’t a big drawcard.  I widened my search, and found two interesting projects — one that eventually became my solution, and another that I think is quite incredible in its scope and capability.

First, the amazing one: ns-3, “a great network simulator for research and education”.  As the name suggests, it simulates networks.  It is completely programmable (in fact your network “scripts” are actually C++ code using the product’s libraries and functions) and can be used to accurately model the behaviour of a real network when faced with network traffic.  The project states that ns-3 models of real networks have produced libpcap traces that are almost indistinguishable from the traces of the real networks being modelled…  I’ll take their word for that, but when you get to configure the propogation delay between nodes in your simulated network it seems to me it’s pretty thorough.  Although the way that I found ns-3 was via a forum posting from someone who claimed to have used it to solve a similar situation as me, and ns-3 does provide a way to “bridge” between the simulated network and real networks, the simulation aspect of ns-3 seems to be more complexity than I’m looking for in this instance.  It does look like a fascinating tool however, and one I’ll definitely be keeping at least half-an-eye on.

To my eventual solution, then: Open vSwitch.  Designed with exactly my scenario in mind–network connection for virtualisation–it has at least two functions that make it ideal for me:

• a Linux-bridging compatibility mode, allowing the brctl command to still function
• IEEE 802.1Q VLAN support (innovatively at that)

The Open vSwitch capability can be built as a kernel module (there’s a second module that supports the brctl compatibility mode), or very recent versions have the ability to be run in user-space (with a corresponding performance drop).

On the surface, configuring an OvS bridge does seem to result in something that looks exactly like a brctl bridge (especially if you use brctl and the OvS bridging compatibility feature to configure it), but its native support for VLANs really brings it into its own for me.  In summary, for each “real” bridge you configure in OvS, you can configure a “fake” bridge that passes through packets for a single VLAN from the real bridge (the “parent” bridge).  This is exactly what I needed!

For the guest interfaces that needed full VLAN-awareness, I simply provided the name of my OvS bridge as the name of the bridge for libvirt to connect the guest to–OvS bridge-compatibility mode took care of the brctl commands issued in the background by libvirt.  The VLAN-unaware guest interfaces presented a bit of a challenge–the OvS “fake” bridge does not present itself like a Linux bridge, so it doesn’t work with libvirt’s bridge interface support.  This ended up being moderately easy to overcome as well, thanks to libvirt’s ability to set up an interface configured by an arbitrary script–I hacked the supplied /etc/qemu-ifup script and made a version that adds the tap interface created by libvirt to the OvS fake bridge.

The only thing I might want from this now is an ability for an OvS bridge to have visibility over a subset of the VLANs presented on the physical NIC.  The OvS website talks about extensive filtering capability though, so I’ve little doubt that the capability is there and I’m just yet to find it.  From a functionality aspect, OvS is packed to the gills with support for various open management protocols, including something called OpenFlow that I’d never heard of before (but I hope that some certain folks in upstate New York have!) but is apparently an open standard that enables secure centralised management of switches.

Detail of exactly how I pulled this all together will come in a page on this site; I’ll make a bunch of pages that describe all the mucky details of my KVM adventures and update this post with a link, so stay tuned!

N’s school recently decided to distribute the weekly school newsletter via e-mail, and had allowance for one e-mail address per family.  Not wanting the additional overhead of having to have either S or me receive it and then having to forward it to the other, I thought it would be neat to have a single common address that, when items arrived, distributed the mail to multiple boxes.  Of course I took the stupid path of providing the school with a yet-to-be-created e-mail address, foolishly trusting my ability to set the system up before they tried to send anything to it…  but in the end it was not so foolish after all, as unbeknown to me I already had everything I needed to achieve my objective.

Unfortunately the first thing I did was assume that I needed mailing list software.  I installed Mailman, and started to read-up on the process to get it working.  I did this on my yet-to-be-commissioned KVM-hosted mail server (a blog post for another day), and started trying to diagnose why mail wasn’t getting delivered.  I had set up Postfix on this mail server to point to my existing LDAP to test, and thought that there was a problem there (but also started to work out if there was a way to use the LDAP server to manage the Mailman aliases).  I re-found the Postfix LDAP HOWTO, and stumbled over the section entitled “Example: expanding LDAP groups”.  Et voila: multidrop incoming mail without the need for a mailing list manager!

I had always assumed that e-mail aliases were a one-to-one mapping of alias address to real destination.  Not the case: an alias can have multiple destinations.  It doesn’t just apply to LDAP alias support, either: as per the “aliases” man page you can do

name: value1, value2, ...

In my LDAP situation, all I need to do is list the alias in the “mailLocalAddress” attribute of which ever users need to receive mail for that alias.  Done!

I may have to keep Mailman, however.  Shortly after this success, I wondered how cool it would be to have the notification SMS messages for voicemail received at home, that currently go only to S, come to me as well.  I’m using a hosted email-to-SMS gateway service for this, so the “alias” would have to expand to multiple external e-mail addresses.  I’m not sure if you can alias mail addresses that are not in your domain…  I’ll have to try and see–might be easier to do that than subscribing to a Mailman list via SMS-to-email! 

I figured this would be an ideal way to make use of an old Nokia 6230 with a broken speaker.  Somewhat foolishly, on the assumption that it would Just Work (and that all the troubles experienced by others would not beset me) I went and bought a two-pack of prepaid mobile SIM cards and went through the adventure of activating them.  One of these SIMs I threw into the 6230, the other I kept on hand for after I got everything working.  The plan, you see, was to be able to take advantage of free calls between the two accounts by taking one of the phones with me when travelling and leaving the other strapped to Asterisk at home.

I think it’s probably fair to say that I’ve had more success with it than a lot of other folk have.  The process of configuring Asterisk to use the Bluetooth dongle is quite straightforward, and it’s even quite easy to configure the chan_mobile driver to have calls enter your Asterisk system in a routable way.  When I dialled the “tethered” mobile from another phone, I was rewarded with the ringing of my desk phone–and at this point, I think I gave myself the kiss-of-death.  “Wow, that was easy,” I thought…

When I picked up the desk phone, I was rewarded with silence.  Not just the silence of the phone not ringing any more, but also the silence of no audio being passed either way over the call path.  Nothing put the pure, desolate sound of FAIL.

Things actually went downhill from there, believe it or not.  I have tried a total of four different Bluetooth dongles, with results ranging from the aforementioned signalling-but-no-audio to why-the-@#%$-won’t-this-thing-pair.  The three different phones I’ve tried elicited a similar spectrum of results.  “Make sure your dongle has a Cambridge Silicon radio, they definitely work” say the forum experts…  Sorry guys, one of the biggest failures I had–failure of Asterisk to pick up the call–was on the last dongle I tried and, yes, it was a CSR.  I’ve even had two different versions of the bluez stack and (I think) two different asterisk-addons versions.

The one thing that I’ve distilled from all the experiences I read through is that there is a ridiculously high level of sensitivity to particular phone and dongle features.  For example, great success has been reported with the Nokia 6230i.  I figured that I was lucky and that a 6230 would be close enough…  Doesn’t look like it.  There is one model of D-Link Bluetooth device–no longer in production, by the way–generally reported to give the most success.  Tweaking the device class reported by the bluez stack in the Linux host is said to give success too, but led to me being unable to get a connection to Asterisk.  Unfortunately, I have neither the time nor the patience to spend too much time trying to go through the motions of getting it working.  I tell you, if it really is that difficult to get two Bluetooth devices to talk to each other it’s no wonder that the majority of folks still use wired headsets!

Luckily all this little experiment has cost me so far is time.  The two-pack of SIM cards cost me the grand total of $2, and they had enough start-up credit on them to allow me to receive calls without a top-up.  The handsets are from that ever-growing pile of GSM hardware that just about every modern household is accumulating now (well, at least the ones that house a gadget-freak who can’t even bear to part with a broken one).  The kernel version I’m running on the system could be an issue, since I get ugly error messages from the btusb module when I take a call, so a kernel update might help.  After that though it’s likely to cost real money–buying a new/different Bluetooth dongle, for example.

If anyone out there has suggestions on something else to try, I’m listening (reading? watching?).  I don’t mean to complain, after all I am one that usually subscribes to the “it’s Open Source, it’s the hard work and dedication of others, you got it for nothing, you’ve got no right to complain” philosophy.  It is really frustrating to come away from a couple of days’ effort with nothing to show for it, though.

I’m implementing a router box on an old low(-ish)-power PC that will be backed up by a virtual machine on my main virt-box.  I’ve already done most of the preparation of using keepalived to implement VRRP, and a colleague has given me some pointers in using the Linux-HA tools like Heartbeat and DRBD to make services like e-mail and Samba redundant.

I’ve had a soft spot for LDAP for ages; I’ve always thought that putting as much backend data into LDAP as you can would be a really good way to get failover and redundancy.  Instead of having to deal with every single server’s different way of doing replication and failover, just bung everything into LDAP and get that replicating.  Sounds good in theory, but in a nutshell it’s not working out that way for the two least-celebrated but most important components of my (arguably any) network: DNS and DHCP.

There are a number of LDAP-backed DNS projects out there.  If I’m willing to go to the bleeding edge with BIND on my Gentoo build I can get access to the two most talked-about ones (bind9-sdb-ldap and the BIND DLZ LDAP driver), and other solutions like PowerDNS and ldapdns are available.  But none of them offer integration with DHCP, and I’m currently using dhcpd’s “interim DDNS update method” to make sure that hostnames are seen in my DNS when a lease is granted (okay, there’s a Perl daemon that goes with bind9-sdb-ldap, but it seems like a sort-of clunky afterthought).

Speaking of DHCP, LDAP backends for it are virtually non-existent.  The only LDAP-enablement I’ve found for ISC DHCP involves putting the config file into LDAP, not the leases…  I actually used that for a few days a while ago and pulled it out because it was actually more work to do it that way (and for no benefit in failover).

It seems to me it would be a project ripe for the picking: take an integrated DNS/DHCP server like dnsmasq and make it write into LDAP instead of to a file.  If I had more free time I’d probably have a go at it, except for the fact that no-one really seems to be that interested in storing DNS and DHCP in LDAP: that it hasn’t been done says to me that there’s no demand for it, and it’d end up being a big waste of time and effort.

Over to you, lazyweb…  Is this a yawning chasm of unfulfilled networking dreams, or a case of me trying to make something more complex than it needs to be?  After all, the rest of the world gets by with DNS master-slave and DHCP failover, they should be good enough for me too, right? 

I’d recently updated the machine that provides the transparent web proxy function for the network; one of the updates took Squid up to version 3.0 (from 2.6). This was the first thing I was suspicious of.

There’s an option in Squid that controls how it handles an “If-Modified-Since” request from a client. The default is for Squid to respond based on the age of the item in the cache, not based on the real item on the source web page. The comments in the Squid config file indicate that some clients use an IMS when requesting a reload — looks like APT is one of those clients.

Setting this option to “on” (from the default of “off”) in squid.conf fixed the issue for me:

refresh_all_ims on

I've turned off the comment capability, until I can get something in place to bring the rubbish under control (a recent update to PolarBlog helped a bit, in that the crap doesn't display on the site any more, but when I log on I get to see the mess). I'm thinking of a new site, where I can discuss technical stuff a bit more and thoroughly while keeping the private stuff separate if I need to.

The site has had a bit of downtime recently, due to my non-existent monitoring of what's happening on my hosted server. This will change shortly, and I'm looking forward to things returning to the stability they had when I was self-hosting.

S's and my respective creative sides are being adequately satisfied by the iLife suite on the Mac, but there are times when we need to get the pictures out of the silver tower and onto other media—on this occasion paper, for albums and so on. A large retailer here has part of their floor space in each store set aside for those photo printing kiosks, and I introduced S to the art of putting photos onto a USB stick so that she could print some photos when next she went there…

On her return from the shop, she reported that we hadn't successfully put the photos she wanted onto the stick. When she'd plugged the stick in, she'd found only less than half of the photos we'd stored there. Sure enough, when I plugged the stick in all the files were there safe and sound. Strange thing was I could find nothing in common about the files (uppercase/mixedcase filename, long or 8.3 filename, datestamp, etc) that would have yielded the number of photos that the kiosk had found on it.

Annoying, but life is too short to worry about it. After all, this same retailer was plastering adverts of their new web-based photo printing service… S could submit the photos online for printing and pick them up from the store later.

<sarcasm>This is where the fun really started.</sarcasm>

Their app is Flash-based but seems to have some Java involved as well. While it loaded quickly enough, the app portion of the web page had an incongruous grey background that just looked dodgy. S had to create an account and sign onto the site just to get this far though, which was a bit annoying.

The workflow seemed to be to create an album, upload pictures to the album, then select photos from the album for processing. Creating the album went fine, but when the upload function was selected there were no action buttons visible to complete the operation! S was using Safari, but Firefox made no difference.

Then I suggested she use her laptop, which runs Ubuntu 8.04. The situation actually seemed a bit better to start with, as instead of the upload function showing an embedded file selection dialog like it did on the Mac we got a "normal" GNOME file dialog box. However, only some of the photos showed again: this time, it was because they had hard-coded a non-modifiable filename filter for the dialog that was only picking lower-case file extensions!

Trying to work around this, I mounted the stick manually with different mount options. I succeeded in getting all but one of the files showing with a lowercase name, and a rename fixed that one. Back in the web page however, it still didn't like us: any file chosen from the dialog box resulted in a nonsensical error message followed by a "You have selected no files to upload" dialog.

S was beyond caring by this stage (she has a very low threshold for being stuffed around by technology). She went to Snapfish after a friend's recommendation, and found a well-designed and easy to use WEB site that required no downloads or other junk.

So why did this wind me up to the point of spending all this time blogging it? Because nowhere on Big-W's site is there any mention of browser or operating system compatibility. Not even a "we've tested only on Windows, Mac users may experience difficulty"[1]. Not a blessed thing. Their Help page has a single paragraph about trouble uploading, blaming "your IT Department" for "setting certain network properties that inhibit the upload tool from working".

I wonder if the developers of the app were just so blind to believe that their gunk would just work wherever it was run, or whether they really think that it's a Windows world. Of the two I hope it's the former.

So Snapfish gets a recommendation for being not just an application hosted on the web but a web application. They do good photos too!

[1] I never expect to see Linux mentioned on these things and get pleasantly surprised on the occasions it is; even if it says "Linux is not supported", someone there at least knows enough to mention it.

The problem is that Synergy and the VMware console don’t play well together (I could have sworn that when I first started using Synergy I had no trouble with it, but there are a few hits around that describe problems like I’ve now hit). The problems people are reporting are that keys like Shift and Ctrl are not passed to the VM (some described here and here).

My problem is slightly different: the screen of my Synergy client (the one that’s running VMware) locked while a VMware guest had focus. Now, the Shift and Ctrl keys are not picked up by gnome-screensaver to unlock the screen. Even the real keyboard attached directly via USB doesn’t work. Big problem, for the following reasons:

* Thanks to password strength rules enforced on the Linux build I use, my password now has a Shift-obtained punctuation character.
* I can’t switch to a virtual console, since that requires Ctrl (e.g. Ctrl-Alt-F1).

Okay, so the keyboard doesn’t work. This client machine just happens to be a tablet PC, and I had hacked gnome-screensaver (to display the onscreen keyboard to allow the screen to be unlocked in tablet mode). I grabbed the pen and tapped out my password, but it *still* didn’t work: even the output of the virtual keyboard gets the Shift modifier dropped. Hmm… Starting to fume now.

Never mind, I’ll connect via the network…

* Fedora does not start SSH by default (okay, yes, and I didn’t make sure it gets started after I’d finished the install).
* There is no remote desktop (VNC server, XDMCP) configured.
* The shiny web-based management interface on VMware Server 2.0 only listens on 127.0.0.1 (or is being blocked by the Fedora firewall).

So with no way to get access to the machine to try and fix it, a power-off is the only solution. Some readers are probably thinking “boo-hoo, diddums had to kill-switch his widdle poota, how tewwible,” but I hate having to do that; not because the system doesn’t recover, but it’s “problem resolution, Windows-style”.

Even though the real problem was between Synergy and VMware, I’m blaming the (perceived) need for security since without that I wouldn’t have a cryptic password that I can’t enter without Shift and a system I can’t administer over the network. Red Hat and Fedora doing everything in their power to ensure I don’t fall prey to nasty Internet fiends (rich analogies to governmental nannying, but that’s probably over-thinking things).

So in summary: Synergy is great, just as long as you’re not using VMware console and have a password with punctuation or uppercase… Remember to have your SSH or other network access enabled before you play!

The first was an update to the spine resource poller, part of the Cacti project but installed separately (it used to be called cactid). Turns out that somewhere between 0.8.7a and 0.8.7b, bugs were introduced that made spine unreliable on 64-bit systems. The update brought in a SVN version of spine which, while still labelled 0.8.7a, must have been somewhere after one or more of the bugs came in. The symptom was that every data value obtained via SNMP was garbage and ignored.

The second issue was strange — graphs were getting generated (even those for which there was no data) but there was no text on them! Titles, margins, legend, axes, all were blank. Some posts pointed to a problem accessing the TTF font file provided with rrdtool, but the actual problem turned out to be the upgrade to rrdtool 1.2.28 which introduced different parameters for the configuration of text attributes in graphs — and a corresponding “feature” that suppressed any text output if the new parameters were missing.

So what does “~” have to do with this? The software on your system is built according to the architecture of your machine. In Gentoo, this is called your “arch” (for architecture) and is usually “x86″ or “amd64″. Gentoo implements a “testing branch” in an arch which starts with “~”; if a pre-release version of a package exists in portage you can bring it in with the “~x86″ keyword. The nice thing about this is that you don’t have to enable a testing repository across your whole system — you can enable the ~ keyword for specific packages on your system, and everything else stays stable.

Unfortunately, this flexibility has a cost. The “amd64″ arch seems to lag a bit behind “x86″ in terms of packages being marked stable or just simply having packages available. This means that just to get things installed, it’s necessary to flag packages with “x86″, “~amd64″ or even “~x86″. This flagging is easily done — almost too easy in fact, as it creates a problem later on when the package you actually set the keyword for eventually becomes stable and you don’t need the keyword set any more. It’s a manual process to revisit the keywords you’ve set and verify that they are still needed (and you know how well manual processes work).

Some time ago I started adding comments to the Portage config file where keywords are set, trying to explain why I set the flag: “to bring in version 1.2.34″ for example. That way, if I ever do get around to manually auditing the package.keywords file, I’ll be able to check if some of the keywords are still needed. Still a manual review though.

So in the case of rrdtool and spine, I had set the “~” keyword some time in the past for some reason, possibly to get early access to a bug-fix ebuild. With no established method to revisit the keywords, I continued to pull in unstable versions of packages long after the packages I really needed had been marked stable. Eventually, it bit me.

The pre- and post-upgrade chacklist grows some more…  :)

I’ve got a system here that is a host for a virtualisation environment I run. I dedicated a couple of network cables to the adapters owned by the virtualised system, and a third one was attached to the host’s IP stack. To get connectivity for another system, I had to steal the host’s cable though–which wasn’t a problem as the operation of the system works more-or-less entirely from the console rather than over the network. Just for grins, however, I decided to set up connectivity to the host by routing through the virtualised environment it hosts.

Having established the tunnel connection between the virtualiser and the host stack, I set about configuring the special details required to support routing through this system. After a few tries at getting it right, I was rewarded with successful pings between the systems on my LAN and the hosts system on its routed connection. So I jumped onto the console of the machine and light up Firefox, but got an error page. I realised I hadn’t set DNS resolution–on the LAN, the machine was having resolv.conf configured by DHCP, so now I had to do it manually.

Okay, so DNS resolver now correctly set, let’s see Firefox WIN! Oh. Fail.

When I hit Try Again or Reload, the page would instantly refresh. This was starting to look like no routing problem. I used dig to test name resolution, and it told me it was being rejected. I looked at my dns.conf… Nope, so subnet restrictions coded there…

So I hit the lazyweb, and it didn’t take too long before I found a forum post that led me to this. In BIND 9.4.1-P1, ISC basically changed the default behaviour of a couple of query filtering settings. This had the effect of rejecting some requests that were previously accepted, such as those from non-local subnets. A reconfiguration of my DNS server gave me success at last.

Hooray for persistence! Now, someone hand me some Cat-5 so I can make a cable and plug this thing back in properly.

The onboard wireless on this thing is an ipw2100, hence only 802.11b, and I had a PCMCIA 802.11g NIC lying around (actually it came from the Lifebook, liberated from there after I bought it a Mini-PCI 802.11g card on eBay). On Gutsy, I used the hardware kill-switch to disable the onboard adapter to make double-sure that it wouldn’t try and drag the network down to 11Mbps.

This laptop was the last machine I upgraded to Hardy, and I was playing with KDE 4 on it so I was looking forward to seeing what KDE4-ness made it into Hardy. While the upgrade was taking place the wi-fi connection dropped out, but I didn’t think anything of it since Ubuntu upgrades try and restart the new versions of things and I figured NetworkManager had fallen and couldn’t get up. After the reboot, however, KNetworkManager (still the KDE3 version, don’t get me started there) could find no networks — could find no adapters, in fact.

I logged back into KDE3 and poked. Still no wireless (as if the desktop would make a difference, but I had to make *some* start on pruning the fault tree). The Hardware Drivers Manager was reporting that the Atheros driver was active (for the PCMCIA card), and an unplug-plug cycle generated all kinds of good kernel messages.

On a whim, I flicked the hardware kill-switch for the onboard wifi[1]. Almost instantly, KNetworkManager prompted to get my wallet unlocked — it had found my network and wanted the WPA passphrase. I provided it, and got a connection: via the PCMCIA NIC.

“That’s odd”, I thought, and flicked the switch. A few seconds passed, and the link dropped. Flicked the switch on, link came back. Flicked the switch off again: this time a few minutes went past, but again the link failed. Tried it several times again, and the same thing happened. The state of the kill-switch for the onboard NIC was influencing the other NIC too!

It seems that this is altered behaviour in NetworkManager, applying the state of the hardware switch to all wi-fi adapters. If it annoys me significantly I’d like to think I’ll trawl changelogs, or even better lodge something on Launchpad… more likely though I’ll forget all about it having found a kludgy workaround.

I’ve now added ipw2100 to the module blacklist and things work okay (presumably because the state of the onboard switch can’t be reported any more). I’ll also have a think about whether a few dollars for another g-capable Mini-PCI NIC will be throwing good money after bad, as this laptop really is quite long-in-the-tooth.

Oh yes, that’s right… KDE 4. Next time perhaps.

[1] I can’t think why I did this. I knew that I’d disabled 802.11b in my access point, to make triple-sure an 802.11b device wouldn’t slow my network down… The onboard 802.11b NIC would never successfully get a connection.

The fact that it hasn’t taken centre-stage is possibly as much to do with VMware’s bogus clock-drift problems as anything, as I haven’t dedicated hardware to my Zeroshell instance yet (I could keep it running virtual, but some of the things I want to do with it will make more sense if it’s a separate machine). VMware Server takes another barb for its handling of VLAN tagging (but to be fair that might be the Linux 8021q module works). It seems that if you have any VLAN definitions on a network card, VMware won’t get to see any VLAN tags on that NIC. You can get a guest attached to a bridged interface to see the real VLAN tags, but only if Linux has not got any VLAN awareness over that NIC.

Alright, so enough ragging on VMware. I have Zeroshell attached to the networks it needs and all is fine. Except that I can’t actually change anything! The web interface that I spoke so highly of originally is actually very restricted in some areas. One of these is in the RADIUS server, and it bit me badly when I decided I’d use Zeroshell’s RADIUS server to authenticate access to the Web interface of my Linksys switch. Turns out that the Linksys firmware expects a particular attribute to appear in the response from the RADIUS server.

The fact that Linksys don’t document this anywhere is not Zeroshell’s fault, but that there is no interface allowing me to do updates to the records above what Zeroshell uses for its own applications is a bit of an issue. It means that instead of a Zeroshell box potentially becoming the hub of administration functions, it is in danger of becoming just another little vertical application server that doesn’t integrate.

Having said that, the backend for most (all?) authentication data is LDAP so a tool like PHPLDAPAdmin might be usable to extend the base records. But, arguably, I shouldn’t have to do that! It is still beta software though, so improvements and enhancements will be made.

The other area that it’s a bit lacking in is monitoring/graphing. Okay sure, I’d probably integrate Zeroshell into the rest of my Cacti setup, but it would be nice if Zeroshell did like other router distos and had a pre-built statistics/graphing page.

Zeroshell is still my pick (I revisited pfSense and fixed the problem updating, but to me it doesn’t have enough function to justify running its own hardware), but it’s just not quite the bees-knees it was when I first saw it.

I’ve become fairly complacent about system updates. All the distros I use now have got excellent tools for keeping everything up-to-date, and for making sure that things don’t go wrong in the process. It’s all just software, however, and it’s all too easy for something to get missed or for a bug to creep in. One such bug that did exactly that is this one. Unreported at the time I did my update, it rendered my Slug unbootable after the update I gave it.

It took me a day to realise that the Slug was off the network. The failure of the nightly backups was my first clue. Next was the inability to stream any of the media files stored on it. For the next week, on-and-off, I tried a dozen things in an attempt to get it working again. I finally arrived at a process that used the Debian Installer firmware image as a way to get a running system onto the device, allowing me to then access the hard disk and try and reflash earlier kernel and initrd images to it.

I started trying to work on the boot disk, but I couldn’t see it for some reason. Then I discovered that the power supply of the USB2 disk enclosure that holds it was playing up! Now, I had two problems–was one related to the other? Was my boot problem just a hard disk problem all along? Turns out that the power supply failure was a coincidence–replacing the power supply got the disk working again but made no improvement in the bootup scenario.

The NSLU2 boots differently to a PC. On a PC, the BIOS locates some boot code on a storage device and executes that, which usually is a program like LILO or GRUB that has more intelligence and (in the case of GRUB) a way to interact with it. These boot loader programs then load in the kernel and start executing it. With the NSLU2, however, the kernel and the “initial root device” are written into the flash memory of the device–they more-or-less are the BIOS.

On a PC, if there’s a problem with the kernel or initrd you can generally select another one from a list. Worst-case would have you installing the hard-disk in a different PC and fixing the problem from there. On a NSLU2, however, any problem with the kernel or initrd can’t be fixed by changing the hard disk because the kernel and initrd aren’t read from the hard disk but from the flash memory instead. There’s also no option for selecting another kernel, since the NSLU2 is a “headless” device with no console (besides, there’d be no room in the flash memory for two copies of kernel and initrd).

Once I’d been able to get my Slug booting (by writing out a previous version of a kernel and initrd) I was going to leave it alone… but curiosity got the better of me. I’d suspected a bad update to the utility that generates the initrd, and sure enough an “apt-get update && apt-get upgrade” revealed a pending update to the initramfs-tools package. Google led me then to the above bug report. With fingers crossed I did the update, reflashed, and rebooted… successfully!

The Slug is now back in its usual place, quietly going about its business of entertaining us and keeping critical data safe. I might at least think twice before doing a kernel update on the poor beast in future though!

First I found pfSense, a FreeBSD-based distro that seemed to fit the bill–indeed the very first question the Live-CD asked me on bootup was “do you want to use VLANs?”. It also promised a very extensive set of additional packages that extend it’s capability into areas like file/print, WWW proxying, and a host of other features. However, even though it has a very nice web-based configuration facility, due to what looks like a problem on their web site I was unable to even look at what packages are available. Since some of the basic function I would like is provided by these packages, I’ve had to move on–but pfSense gets an honourable mention because of its easy installation and excellent configuration interface.

I looked again at Smoothwall, but soon remembered why I discounted it at the time I chose IPCop. For me, the level of function I think I’d use is a bit too close to the threshold of function in the “community” (read, “free”) version. Astaro would go in this category too, except that I was too dense to be able to even find much clear information about the level of function you get in their community version. So no recommendation on either of these, as I’ve never used either–I do work with a fellow who happily uses Smoothwall though.

Then, I came across Zeroshell. The lead developer describes it as “a small Linux distribution for servers and embedded devices aimed at providing the main network services a LAN requires”. And does it ever! It’s a veritable Alladin’s Cave of features and functions. It certainly does everything I was looking for, from VLAN tagging through QoS to VPNs, from an SPI firewall to multi-zone DNS and multi-subnet DHCP servers, but also has Certificate Management (using a self-signed CA certificate or one you import), a RADIUS server, WiFi access-point capability with multiple SSID and VLAN mapping, captive portal or “normal” HTTP proxying, 802.1d bridging, clients for Dynamic DNS, a Kerberos 5 server, plus a raft of other capabilities. Zeroshell–named because the author wanted to provide a system that was extremely flexible and powerful yet did not require users to access a shell prompt–is remarkably feature rich, and yet the download for the ISO image is only around 100MB (a bit beefier than pfSense, admittedly, which weighed in at around 60MB).

There are a couple of downsides, however. Until very recently, installing to a hard disk was not supported. The distro is designed to boot from a CD only, but can use an installed hard disk (if available) for what it calls “databases”, where configuration and other data is kept. With the latest release, however, the developers have created a “1GB USB drive” download (the size of the download isn’t 1GB), which is designed to be copied to a USB pendrive or hard disk.

The other downside (and it’s not fair to say that, as will become clear) is the web interface. Not because it’s ugly or not functional: it is neither of those. It’s clean and well laid out, and fairly consistent. It’s very technical, however. Where other distros tackle the “SOHO divide” by hiding details such as protocol numbers or port ranges, Zeroshell uncovers all this stuff in its gory detail. So it’s great for someone like me, who looks at the interfaces on other systems and pines for the knobs I can’t fiddle with, but it’s not for newcomers.

It looks to be a fairly new project (current release is 1.0beta9), but the forums look good and there does seem to be a bit of activity around it. I’m running Zeroshell in a VMware guest at the moment while I kick the tyres–the VMware download is also available from the project’s mirrors–but I reckon this one will be a keeper!

Let me get something straight: Shepherd is the bees-knees of EPG grabbers for Australian MythTV users. If you’re a MythTV user in .au and not running Shepherd, stop reading this right now and go and update your system to use it–you’ll be glad you did. If I had just looked at some of the output it has been generating since OzTivo announced it’s changes, most of the agro I’ve suffered the last few hours would have been avoided.

In a nutshell, Shepherd is a “meta-grabber”. It includes code that can get program data from a dozen or so sources, and keeps looking up sources until it fills your listings with data goodness. It automatically updates these individual source grabbers as well, so you should never need to worry about its up-to-date-ness (more on that later though). It also fetches extra program data from IMDB and TVDB, and can even automatically grab station icons for you. Highly, highly recommended.

I could see that some of my EPG data was coming from OzTivo because I had seen the notes that they had put in the program data advising of the API change. The weird thing I saw was that for a program I was recording in the same timeslot each day, sometimes the message would be there and other times not. While I thought that this was a little strange, I figured that the OzTivo folks were just being overly cautious and trusted Shepherd to do all the updates it needed.

Then, ever since Sunday morning when the southern states *didn’t* switch back from DST, I’ve had recording times out by an hour–programs trying to record an hour early. So as I mentioned, I had ye-olde timezone data on the backend, which can’t have helped depending on the data source (although I’m trying to work out if this actually is a contributor as I would have thought it would send the recordings an hour late… plus, others who have confirmed their timezone data have had the same problems). For a couple of programs, I actually had double entries: one an hour too early, then a second one at the right time. This was weird, and I still can’t explain it!

A manual run of mythfilldatabase showed why I was getting the repeated OzTivo API messages. Shepherd had downloaded the updated grabber alright, but the new version has a Perl dependency that wasn’t satisfied and it couldn’t run. Rather than bail out, Shepherd elected just to keep running with the old grabber. Given the circumstances, I’m still deciding how I feel about that. :-

So once I was confident that the grabbers were working okay again, I decided to get the EPG straight. I remembered that mythfilldatabase will not replace any existing data it thinks is valid, which is why only data post-April-5-or-so looked nice again. So, with a mailing list post or two as encouragement, I truncated (database-admin-speak for “deleted all the data from”) the “program” and “programrating” fields in the mythconverg database and ran mythfilldatabase. After about 20 minutes, voila, fixed guide data!

So now I’m thinking of how I can alert myself to a problem with Shepherd. I used to just check the result of the last mythfilldatabase run through Information Centre or mythweb, but since Shepherd ends cleanly so does mythfilldatabase. Looks like I might have to come up with something hackish to look for Perl runtime errors in the mythfilldatabase log and do a Nagios passive service check or something… Sigh, as if I needed another little project to keep me busy…

Some things aren’t starting because of missing binaries in /usr, frustrating but probably recoverable. The network startup is totally clagged though, and I can’t even begin to work out how what happened… happened.

During bootup, at the time it tries to configure the network interface, I get streams of error messages about problems running the “ip” command. The error text is full of garbage that the init script is trying to parse as text configuration–it looks like a corrupted filesystem or a binary file.

I manually configured the network (not a trivial task in s390x, it must be said), and started to poke around. I got this when I logged in as root:

Last login: Sat Mar 15 12:48:36 2008
/usr/X11R6/bin/xauth: error while loading shared libraries: libXau.so.6: cannot open shared object file: No such file or directory
-bash: read: read error: 0: Is a directory
lxs0za01:~ #

Okay, so I won’t get funky X-based YaST.  No problem, I’ve spent more time in the ncurses-mode YaST anyway…

lxs0za01:~ # yast
warning: the ncurses frontend is installed but does not work
You need to install yast2-ncurses to use the YaST2 text mode interface
lxs0za01:~ #

WHAT!!! What the @#!$ happened there?!?!?

Okay, so I’ve calmed down about that, so I go looking for the problem with the network initialisation…

lxs0za01:~ # cd /etc/sysconfig/network
lxs0za01:/etc/sysconfig/network # ls -go ifcfg*
-rw-r–r– 1   141 2006-06-17 07:30 ifcfg-lo
lrwxrwxrwx 1    16 2008-03-15 02:23 ifcfg-qeth-bus-ccw-0.0.0f00 -> /lib64/ld-2.4.so
-rw-r–r– 1 27470 2006-06-17 07:30 ifcfg.template
lxs0za01:/etc/sysconfig/network #

Priceless. You can’t make this stuff up. I cannot for the life of me work out how this could possibly have happened. I guess I just blame it on a whacked-out filesystem and move along.

Okay, so both of these issues probably have extenuating circumstances unrelated to SLES or YaST… but it’s nice to have a vent now and then. I’ll write up something a bit fairer once I fix this b0rkedness.

I run Gentoo on both my “servers” at home. At the time I got my dual-Opteron, Gentoo was the only “free” distro around that had a x86_64 version ready to roll. When it came time to build my phone-and-TV server, it got Gentoo as well because it was the only way I could get the right combination of all the versions of code (Apache, PHP, Asterisk, MySQL, MythTV, ccxstream, etc) that I needed and have them all maintained in the distribution’s package management system (Debian has no ccxstream package, for instance). I don’t run Gentoo because I’m a ricer. Portage has the right package mix for me, and its ability to control the configuration of packages through USE flags gives me an opportunity to control the options that are enabled in the packages I install.

I have blogged previously about some hardware I bought that I haven’t been able to put to good use. I decided to give it another try by building a Gentoo system on it, because an ebuild for the bleeding-edge ATI driver that is supposed to support the graphics chipset in this clunker is in Portage.

Let me say, it’s been a while since I built a Gentoo system from scratch. You don’t even do it truly from scratch anymore either — the days of starting with a stage-1 tarball are over apparently, and stage-3 is always the way to go. Even so, this system took a whole weekend to get to the stage where I could log on and get a KDE desktop (to be fair though, there was a lot of kicking off an emerge, coming back to it a couple of hours later to find it had died ten minutes in, fixing the issue and restarting… so it wasn’t 48 hours solid time spent).

Unfortunately the ATI driver still doesn’t support XVideo on this chipset, so I still can’t use this board for its intended use as a MythTV frontend (I do have an old PCI nVidia 5200 card that, even though it’s at least three years old, I’m sure will run rings around this stinking ATI 1250). So the point of the whole exercise was, unfortunately, lost. But I did get a refresher in the amount of effort a Gentoo build would take.

After that weekend’s effort, I was a bit put off by the thought of building up an entire desktop system from scratch. When I thought about it though, my concerns were for nothing. The compiling? The kind of systems I’m building on (modern dual-core chips) will chew through compiling most software in a snap — heck, for simple packages I can install on a Gentoo system quicker than yumex can initialise its repositories. I’ve got running systems I can use as a model to get USE flags right, and my NFS-shared Portage tree means that I sync once and use everywhere (even downloading source packages happens only once).

Plus, now, I know Gentoo. Sure, APT on a Debian-based distro is nice, but I still am lost when it comes to the right dpkg command to locate what package provides a certain file, for instance. I get frustrated when something fails to build on Gentoo because some other package wasn’t built with the right USE flag, but I know how to fix that, and its fixed in a flash. Likewise for rebuilding some system library that causes a bunch of other packages to fail without warning, and likewise for the strange b0rkedness that happens in Portage sometimes when packages change versions (gnupg is a recent example). I know how to fix Gentoo when it breaks — I can’t say that with much confidence for other distros.

Some might say “use a distro that doesn’t break in the first place”, which is a fair comment. But if I have to choose between an occasional hiccup and missing functionality, then hand me the Eno (Pepto-Bismol, Tums, etc)…

Which brings me to my dilemma — apart from the fact that I have crappy unaccelerated non-video graphics and I haven’t been able to run Compiz for ages (a problem that Gentoo wouldn’t solve for me anyway), Ubuntu isn’t really broken for me. There’s not a compelling reason for me to throw Gutsy out, and with Hardy around the corner there’s even less reason to switch right now.

So, I’ll wait. And watch. Having to work on more Red Hat systems at work is reacquainting me with their particular mojo, perhaps even enough to try Fedora. Also, I’ve just scraped together some parts to make an openSUSE 10.3 build for something work-related so I’ll catch up with things there (since I haven’t really seen a SUSE system as a desktop since SuSE Linux 7).

I love this about Linux — freedom to choose!

I’ve been playing the game all night since I found it on Monday afternoon. Sleep seems a distant priority compared to making sure I snag the subsidy for a passenger service from Podlondlington to Nunmubhattan…

It’s easy to install on the Ubuntus, but you do need to obtain the data files from the original CD — the Full Circle article contains instructions on how to do that (or I’m sure the website tells you).

Sure, the graphics don’t measure up to today’s insane system-melting specifications and the isometric view, while state-of-the-art in its day, is at times frustrating (I’m sure there was a control you could use to hide the buildings so you could see behind things… maybe I’m thinking of Lincity). Still, it’s both a great bit of entertainment and a trip down memory lane at the same time. If you’re like me and played with the Tycoon games as a kid, or if you’re a bit of a retrogamer, I encourage you to check it out. Don’t expect to see much of your family for a while though…

I tried an early Beta of the KDE 4.0 Live CD, but it was still using the KDE3 Kicker and was also a bit unstable. I wasn’t sure if it was the fact I was running in a virtual machine that made the graphics a bit flaky or whether it really was beta-quality code making things a bit funny. The KDE team put a lot of effort into bug-swatting in the weeks leading up to 4.0 being tagged, and it’s a lot better now!

This announcement from the Kubuntu folk shows how to get the KDE 4 packages installed on Gutsy. KDE 4 installs in a different path to KDE 3, so you can try out KDE 4 without affecting your existing environment.

I did have a bit of a heart-starter with this though, as apt-get wanted to remove a package called “kdebase-bin-kde3″, which looked risky! It’s okay though, as equivalent binaries are provided by “kdebase-bin-kde4″. In fact, if you follow Kubuntu’s instructions exactly, you should not see the issue: it happened to me because I did a system update after adding the Kubuntu PPA repository but before installing KDE 4. The system update brought a bunch of updated KDE 3 packages out of the PPA, one of which was to replace the standard “kdebase-bin” package with a “kdebase-bin-kde3″.

First impressions are that Oxygen (the new artwork for 4.0) looks great — it’s a very modern look. Some might think it borrows from Vista, but to me it’s got as much of Mac OS X’s appearance as that of Aero. Plasma (the desktop shell) does some interesting things, like turning desktop icons into widgets, but I’m yet to spend enough time with it to experience the other improvements it brings.
The biggest thing I’m looking to trying out is the compositing built into the window manager, KWin. Unfortunately the laptop is a bit old for this to work well (or at all in fact), so I’ll either have to find some magic Xorg setting or get the KDE 4 packages on the desktop machine. I’ve had trouble running Beryl and Compiz thanks to something about the terminal program Yakuake tickling a long-lived bug in X11 (I think part of the reason it’s long-lived is that the X11 folks don’t accept it as a bug but rather a fringe case that Yakuake shouldn’t be exercising, hence a stand-off) so it will be interesting to see if KWin has the same kind of issue.

As for bugs, well there look like plenty. As I’m keying this, Konqueror is chewing 100% CPU and the characters are delayed by a couple of seconds (and of course, now that I observe this, it stops doing it). Still with Konqueror, this is about the third time I’ve tried to post this thanks to Konqueror segfaulting for strange reasons. Also, the Alt-F2 program launcher reports that it was unable to launch whatever you told it to, even though it does so successfully.

There has been plenty written by the KDE folks about the “1.0.0 release of KDE 4″, and they’re copping a fair amount of stick from people who think they’ve done the wrong thing by releasing as 4.0.0. I’m on KDE’s side. Although many KDE folks have used their KDE 4 builds as their daily desktop for months, I haven’t seen anyone who wears a KDE hat recommending that others do so. The term “will eat your children” has been used to describe KDE 4 by folks from the KDE team, so there has never been any pretense that KDE 4.0.0 would be a daily desktop for the majority of users. I’ve never really participated in large-scale software development, but I can see their motivation for releasing what they had as 4.0.0 — I’m proof of it. As long as it was a beta I was not really all that fussed about trying it out; even after there were release candidates I wasn’t all that keen. As soon as you call it a release, however, your early-adopters rush in and kick the tyres and your real testing can start.

By being open about 4.0.0′s status (and I don’t think you can get more open than “will eat your children”), they can make sure that subsequent releases are a lot better than they would be if they dragged on in perpetual beta — the model that Google and the Web 2.0 fraternity seem to insist is better, plodding on for months hiding behind beta status and its implicit “get out of jail free” card.

Instead, KDE has shown the courage to take their code, along with its bugs, and hold it up as something they are proud to give to the world. It’s the foundation not only for future releases of KDE, but possibly the start of new ways that people work with their computers. By working with the community, instead of closeted away from it, I believe the KDE team will succeed.

Okay, so that finished a bit more ra-ra than I planned! Seriously, give KDE 4.0 a try… but if you aren’t happy to suffer a few bugs then by all means wait until 4.0.1 or even 4.1. Oh, and be free.

You see, it wasn’t until I read the how-tos that it became clear how it works. The trick is that Google doesn’t run Jabber transports on their own servers, so you therefore need to take advantage of various “open” Jabber servers that do (“open” in this context refers to a server that lets you use its transports without necessarily being a registered user there).

Seeing there didn’t seem to be any restrictions on the servers that could be used, I figured that I could use my own server. Sure enough, after the right incantations to expose the service on the ‘net, I could connect my Google Talk ID through the Jabber-MSN transport on my server to my MSN account. Yay, right? Well, not really — each little test message I sent in either direction incurred three trips over my Internet connection! Yes, three: one to go from my Google Talk client to Google, one back from Google to the transport on my Jabber server, then a third from the transport to MSN. Obviously the same happens in reverse as well (for incoming messages from MSN).

Seeing this as a less than optimum setup, and also being wary of getting listed as a Google Talk-friendly Jabber transport provider, I lopped the transport’s external visibility and went back to using my own JID for transport access. It’s a bit of a shame too; since fring (mentioned briefly in my last post) doesn’t let me connect to an arbitrary Jabber server, to keep connected to everything I’d need two mobile chat programs running.

It’s not like I do that much IM that I need to keep all this running, but it is at least a little bit interesting…

I run a Jabber server for internal things. I wanted to have a secure, private chat facility to use over VPN with my nephews; I want to someday migrate my Nagios IRC bot to Jabber; and I use transports to link into MSN and Yahoo! to reach friends on those networks. The last point is great: I really like the fact that now, from whatever Jabber client I use (even the mobile ones I’ve played with) that I merely connect to my Jabber server and I’m online on MSN and Yahoo! as well.

Google Talk, though, has proven to be a bit of a challenge. It’s actually working like a tower, even though it’s based on (arguably) the most open of the IM platforms! You see I more-or-less took for granted that “transport” way of doing things, using my Jabber server to bridge to other networks. There’s no Jabber transport for Jabber though!

What I want to do kind-of flies in the face of how Jabber is designed. Ideally, you’re supposed to only have one Jabber ID (JID) — Jabber creates an open network with servers establishing connections when needed, very much like e-mail, and you only need an ID on one server to be able to chat with anyone on any other server. So what I wanted to do, which was connect to one Jabber server and have it “relay” messages to an ID on a different server is just not necessary with Jabber. Nor should it be necessary for Google Talk users to send messages to me using my Google Talk ID only — they can send straight to my JID on my Jabber server.

In the early days of Google Talk, Google had not enabled the “server-to-server” functionality that allowed this kind of communication to happen. Google Talk worked just like MSN, Yahoo! or AIM — you had to have a Google Talk account to chat with anyone on Google Talk. While this was the case, folks were looking making a Jabber-Jabber transport for connecting Jabber servers to Google Talk. At some point, though, Google opened the connectivity paths that allowed Google Talk to exist on the open Jabber network (I’ve tested this for myself). Once this happened, the need for a  ”Google Talk Transport” for Jabber evaporated in most people’s minds.

The solution nowadays is to use a client that supports multiple connections, and connect to your Jabber and Google Talk accounts at the same time. It works of course, but you don’t get the nice benefits that a transport provides — the main one being access to all your IM services and accounts from a single server connection.

So now, having resigned myself to not being able to bring my home JID and Google Talk ID together, the question arose: do I still need my own Jabber server? My current fave mobile IM client only connects to Google Talk… Could I get by just using the Google Talk service? Find out in Part two!

Nothing was being logged either, as I was trying to find out what was going on and why the processes weren’t starting. It was as if it was suddenly ignoring all my configuration files!

Careful inspection of some output from eix showed the problem: Jabberd 2 has been moved to its own ebuild (jabberd2), and the highest version in the jabberd ebuild is now a 1.4.4-something. Not only that, they’ve hard-masked jabberd2:

# Krzysiek Pawlik  (08 Oct 2007)
# Masked untill the split from net-im/jabberd is complete.
# See bug #178055 and bug #195091
net-im/jabberd2

Looks like the last time I emerged I downgraded my Jabberd 2 to 1.4. No wonder the thing was not responding to me.

This is the kind of thing that happens on Gentoo from time-to-time. It’s why I started a regular sync of portage and email-output-of-emerge-pretend-world process: so that I didn’t get too far behind and have a heap of these things to sort out. This one got me off guard though.

Note to self: pay closer attention to emerge output in future!

Multiple iterations of db_recover, attempts to re-index, dump-and-restores of the raw Berkely DB files… Nothing helped. In the end, all that was left was the slapcat-delete-slapadd dance.

(You know that your OpenLDAP is especially sick when commands like slapcat generate glibc backtraces. )

So with what was left of my LDAP data, I started to compare against my replicated LDAP server. The first thing I noticed was that a number of records that I expected to have been replicated were not. I figured that records in the master directory that were lost to database corruption and not to an LDAP operation (a modify or delete) should have been present on the replicated copy. This was not the case, which makes me think that replication only takes effect after the master directory’s backend is updated, and if something like a corrupted database prevents the master from being updated then the replication doesn’t take place. As Zaphod might say, ten points for directory consistency but minus several million for data preservation…

(As I think about this though, the more it doesn’t make sense. If slapd had been unable to update the backend, and hence the replication didn’t take place, surely that would have been returned to me as an update error? I know for a fact that the data I lost made it to the database because I tested an app using the data. It’s unreasonable to me to think that BDB would have returned success on a write operation unless it had actually done so, but I suppose write-caching might create an opportunity for that to occur… No, I suspect a different problem, maybe just replication being suspended at the time, as the real reason that some data was missing from the replica.)

Next I found, despite what I thought was happening based on the lost records, there were quite a few records that were on the replica. This makes me think I’ve had multiple failures, apparently at different times, that have impaired my master directory — one that caused new updates to be lost, the other resulting in loss of existing data.

I’ve added a step to my Bacula processing that performs a slapcat and backs up the resulting LDIF, so if anything happens in the future I have a bit of a chance of running through old files and restoring. The other thing that I’ll kick off is a process to verify the accuracy or integrity of the replica — this might tip me off to a problem sooner rather than later.

My theory on what the cause of this hassle was? Well a while ago I was having a bit of trouble with partitions filling. At a guess I’d say that OpenLDAP was trying to do something (update a transaction log maybe) at a time when the partition its data lives on was full, and got twisted. Soon I’m going to write a separate post with my (updated) thoughts about isolation of failure domains…

For those that haven’t seen it, here’s the process I used to get things back:

# cd /var/lib
# slapcat > whatsleft.ldif
# /etc/init.d/slapd stop
# mv openldap-data openldap-data-old
# mkdir openldap-data
# chown ldap:ldap openldap-data
# cp -a openldap-data-old/DB_CONFIG openldap-data/
# cd openldap-data
# slapadd < ../whatsleft.ldif
# chown ldap:ldap *
# /etc/init.d/slapd start

Obviously if you find yourself in the unfortunate position of having to use this process, substitute your distribution's values for the path to the OpenLDAP data directory and the user/group that LDAP runs under.

On my Gutsy system it was a simple apt-get install libpam-ssh to install (it’s in Universe). Once installed, you need to enable it by changing your PAM configuration. The package provided a couple of sample files in the PAM configuration directory, pam-ssh-auth and pam-ssh-session. You can take these files and copy them into the relevant part of your config.

For the “auth” section I elected just to add the right line to config-auth. What is the right line for you depends on whether you want to still prompt for the Unix password or not (some folks seem to want to use pam_ssh as a supplementary login, a bit like two-factor authentication). For me I wanted to be able to log on with either SSH passphrase or Unix password, so I added a new line like this:

auth sufficient pam_ssh.so

and changed the “pam_unix” line to look like this:

auth sufficient pam_unix.so try_first_pass nullok_secure

The important change is to make sure that “required” on the pam_unix line is change to “sufficient” — this ensures that login can proceed with either the correct passphrase or the correct password.

For the “session” section I simply added the line as suggested by the package maintainers:

session optional pam_ssh.so

With PAM modified I jumped to a virtual console to try a TTY login:

Ubuntu 7.10 columbia tty1

columbia login: vicc
SSH Passphrase:

I entered my passphrase, and was warmly greeted with a command prompt. Hooray! The further test was yet to come…but it passed with flying colours. The ssh-agent that PAM started for me seemed to work a treat, as I was able to login to my servers without password or passphrase prompts.

Now to try a GDM login. I was expecting that there would be a problem with the ssh-agent automatically started by the GNOME login process. However, I was pleasantly surprised to find that GNOME didn’t start another ssh-agent, and my SSH logins from gnome-terminal were again password-prompt-free!

I tested to make sure that logins worked with either passphrase or password, and they do — the only difference is that (obviously) the SSH key is not added to the running ssh-agent when the password is used to log on instead of the passphrase.

I made one minor change from my first trials. I found that the “SSH Passphrase” prompt I saw on the console login also appeared on the GDM login panel. I looked cool to me, but since the machine I was using to test is my wife’s laptop I figured that I had better make the change as transparent as possible. So, I reversed the order of the “auth” lines to make pam_unix.so appear first. I had placed pam_ssh.so first originally in order to make sure that SSH was tried first, but it doesn’t make any difference to the results… and since PAM displays the prompt text from the first module in the order[1], putting pam_unix.so back to the top of the list gives a “normal” login box.

Once you’re logged on, if you made the PAM change as shown to the common config files (on Ubuntu, the common-* files, on others it may be system-auth for example) you can continue to use either password or passphrase for things like sudo prompts and screen saver unlocks. If you included the change in only a specific PAM config, your ability to use the SSH passphrase applies only to that application[2].

I’m keen to try this out on other systems, if only to eliminate the additional “ssh-add” run I currently have to do to get my key into an agent. I’m also keen to try it with KDE to see if it also doesn’t start its own ssh-agent.

While this module seems to work really well, you may have to check a couple of things for yourself before using it. One is that the project pages on Sourceforge appear to be dead: the last released version was in 2004 and there has been no developer response on the open bugs in the tracker. Secondly, while I had no trouble finding Ubuntu packages, a Gentoo ebuild, and documents about its inclusion in OpenSUSE/SLES, if you use RHEL I don’t believe it’s in a standard repository (in Fedora it’s in Fedora Extras which I think doesn’t bode well for RHEL).

Lastly, and probably the most serious point, is that if you decide to use pam_ssh and your current SSH key has a blank passphrase you should think VERY seriously about setting one. If you use your blank-passphrase key to log on to your computer, you are giving free access to anyone who walks up to your computer and enters your username — not only to that computer, but to every other computer that your SSH key provides access to. Admins of multi-user systems that use pam_ssh should make sure that “nullok” or “nullok_secure” doesn’t appear on the module line that prompts for the password, to ensure their system doesn’t create the opening for an exploit. Also, organisations that use SSH keys for authentication need to give some thought to either centralised generation of SSH keys for staff, or testing/verification of passphrase strength.

Despite the negatives, I’m impressed by pam_ssh. It’s easy to set up — it’s taken me longer to write this blog post than get the module working! — and it works well.

[1] When use_first_pass is coded on second and subsequent “auth” lines, those modules will never prompt for a password themselves. With try_first_pass, a subsequent module might generate a prompt if the password that filters down is not correct and the module wants to get a new password to try.

[2] This page has an example of using pam_ssh for XDM only, by editing the PAM config for XDM only.

Edit: Turns out that by using “sufficient” in the PAM “auth” config things break. I was using sufficient as a short-circuit to avoid having to enter both SSH passphrase and normal password (for the case where the passphrase is different from the password). I ASSuMEd that all the PAM applications had no additional auth settings after the inclusion of config-auth, but I was wrong: the PAM setup for GDM has the config that brings in pam_gnome_keyring (the module that automatically unlocks the GNOME login keyring). By using “sufficient” in config-auth for the pam_ssh and pam_unix modules, the pam_gnome_keyring module doesn’t get invoked.

I could change “sufficient” to “required” again, but this will break (preventing an otherwise valid login) in the case where the user has no SSH key. Forcing all users to create an SSH key would get around this; possibly not hard to automate at userid creation time, but tedious for those with existing IDs.

What I think I need is something like the difference between “required” and “requisite”. Both “required” and “requisite” modules have to return ok in order for the entire stack to return ok, but if the module returns not-ok “required” allows subsequent modules to be executed whereas “requisite” exists immediately. “sufficient” seems to be like a kind-of affirmative “requisite”; I need an equivalent to “sufficient” that will pass control to later modules but not consider the failure of a later module as failure of the stack. I’m pretty sure I can do this with the extended control syntax but it might take some mucking about.

I also feel the need to restate something I mentioned in the original post about centralised generation of SSH keys. Much of the advice about using SSH keys for logging on to systems is written from the user perspective, and goes along the lines of “generate your own key, so that only you have the private key and no-one else has a copy”. I agree with this (and would fight for it) as a user of SSH, but as a system admin for systems that allow login via SSH key I tremble at the thought that my users might have SSH keys with no passphrase.

This is one of those areas where trust and responsibility are required on either side: sysadmins must trust their users not to do stupid things, and users must not abuse that trust by acting irresponsibly when creating keys. In my opinion as a sysadmin (and part-time commentator on things security), the convenience of interactive login using SSH keys is a privilege rather than a right, to be withdrawn if necessary to maintain the integrity of the system and the network.

I accept that SSH keys with no passphrase are a necessity for some system automation tasks. That’s why I said “interactive” in my previous statement. Enforcement of a good passphrase for a key used for interactive logins though is tricky. A modification to pam_ssh, to reject the use of an SSH key with a weak or empty passphrase — kind of a built-in pam_cracklib — might be one way. Regular sweeps of user home directories to look for keysets with no passphrase might be another (be careful to update any Acceptable Usage statement or equivalent, to ensure your users know upfront that they’d be subject to this however). It might just be as simple as making sure “nullok” is not coded for the pam_ssh module, as I mentioned originally.

Knowing that generic error can apply to a missing file that the program is trying to execute, I checked what type of file I was looking at: file reported a dynamically linked program.  Great, run ldd to find out what it wants: ldd reports “not a dynamic executable”.  Oh dear.  It was starting to look like a long night was ahead.

I jumped on the Googleweb and discovered that others had encountered the problem I was seeing, but the hits were all a couple of years old.  Their problems seemed to be caused by missing 32-bit libraries on a 64-bit system.  How could this happen?  In older Gentoo releases you had to choose multilib, but according to most of the doco all profiles are multilib unless you choose a “non-multilib” profile (this explained the fact there were few-to-no recent hits for the issue).

Recently I had switched to the hardened profile…  I had a look, and there is a separate “multilib” profile in hardened.  So is the doco wrong: are all profiles multilib except ones called “non-multilib” AND except hardened because they have a different rule?

I had two choices then, try out the hardened multilib profile, or switch back to the previous profile I used.  Considering I hadn’t enabled any Hardened features and don’t really have time to figure it all out at the moment any (I only did it to get rid of the “unsupported profile” warning I get every time I merge a package), I copped out and switched back to the old profile.

Then I had the next issue: I couldn’t use the non-multilib gcc and glibc to build multilib versions of gcc and glibc.  The gcc build complained about a missing 32-bit header (should have been part of glibc) and the glibc build complained that cpp failed sanity test.  Again the Googleweb came to the rescue, pointing me to a Gentoo repository containing binary packages of gcc and glibc that I could apply.  They allowed me to rebuild my own gcc and glibc.

At this point I found that the vmware-config.pl script could run again.  I was BACK!  I started VMware services, ran the managment console, and started my VMs.

I think I get a bit complacent with my home gear sometimes; switching profile to hardened was something I almost did on a whim, and it’s bitten me fairly badly.  Lesson learned.

I broke out Wireshark and did a capture.  DNS request and response normal, TCP three-way handshake fine, SSL Client Hello…  Hmm, TLS handshake failure.  Strange.  I traced a Firefox connection, and (obviously) after the SSL Client Hello there is a Server Hello in response, and the connection establishes okay.

What I found is that the cipher suites presented by Konqueror and Firefox differ: Firefox offers a couple that Konqueror doesn’t, and vice-versa.  More importantly, the one that is presented in the Server Hello on the Firefox connection is labelled by Wireshark as “Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)” and is one of the ones missing from Konqueror’s Client Hello.

So there are two issues here.  Firstly, Konqueror is missing some TLS cipher suites (or at least Ubuntu’s build of Konqueror is).  Secondly, Konqueror’s reporting of the problem is not helpful — stating it was a “security negotiation failure” would be a lot more helpful than just saying “could not connect”.

/me goes looking for KDE’s bug reporting system…

Recently though, I found that FreePBX does indeed have an alternate programming method that matches up with my original intended use.  The default method is called “Extensions” mode, while the different method is called “Device-and-User”.  The extensions mode, in effect, creates a user for every device defined, and calls it an extension.  The device-and-user mode however allows you to configure each separately.  Your device configurations are simply end-points for your handsets (SIP definitions for example) and users are the entities you actually want to reach (i.e. people).

A device can be either “Fixed”, where it is always associated with a particular user, or it can be “Ad-hoc”.  An ad-hoc device allows a user to log on to the device and receive their calls at that device.  A user can be logged on to multiple devices at once, or even a mixture of fixed and ad-hoc devices.

I was really excited by this, as it seemed that I could replace everything I had set up with my extra extensions and associated Ring Groups by just switching to device-and-user.  There is a little snag though — even though devices still have to have a numeric name that looks just like an extension, it is not available to the dialplan in its own right.  If I have configured my ATA-attached cordless phone as device 852, I cannot dial 852 and make it ring.  I can only dial whatever user number the device is associated with, which in turn means that if no-one is logged-in to an ad-hoc device there is no way to make it ring.  Also, a device can only be associated with one user at a time.

I have auto-answer SIP presences on all the handsets that support it, which I use as a two-way intercom system.  This supplements FreePBX’s Paging facility which I use for broadcast, one-way announcements to all (such as “dinner is on the table!).  I couldn’t switch to device-and-user mode completely, as I would lose the ability to selectively dial devices such as the intercom lines that would not be associated with a user (or would need to be associated with more than one user to support both paging and intercom).

So for now I’m sticking with what I’ve got.  I like device-and-user, but by not making the device’s number addressable in the dialplan they’re eliminating a lot of flexibility and possible functionality.  When we moved into our current home I ripped out much of the builder’s phone wiring and replaced it because I didn’t want all my phones in parallel… that’s what device-and-user feels like right now: everything in parallel.  I’ll keep an eye on it though…

I still have the Gentoo bug, so the idea of a Gentoo system where someone else has done the work of gluing all the packages together is very appealing.  It installs nicely — like many distros they use the “Live CD” approach (where you boot from a CD or DVD into a running system that you can try-before you buy, so to speak, before deciding if you want to commit it to your hard drive), and use Red Hat’s Anaconda installer to do the work of getting the system onto your disk.

It is very much a gamers system.  Some popular Linux games are pre-installed, including Battle of Wesnoth, which has already cost me a decent amount of leisure time.   They include things like the NVidia and ATI binary drivers, AIGLX or XGL for desktop effects thanks to Beryl, and other things like Google Earth and Kerry Beagle preinstalled.  They also have support for Xen and KVM/QEMU virtualisation, and the virt-manager tool for managing virtual machines.

I started to wonder though: do they maintain a separate Portage infrastructure for their own stuff, or is it vanilla Gentoo?  The answer is that for the most part it’s vanilla Gentoo.  They use Layman to track their own Portage overlay, but behind it is vanilla Gentoo, which means that at a time like this, where a new release of Sabayon is just around the corner, doing an emerge world is much more likely to get stuff from Gentoo Portage than the Sabayon overlay — especially since the testing “~” keyword is set by default.

I really like what they’ve done, but to me having it backed by Gentoo Portage means that you’re back to riding the knife edge that is the Gentoo “rolling release” strategy.  And after a couple of updates, you’re running Sabayon Linux in name only.

This is all based on very early experiences mind you.  I haven’t had a proper chance yet to settle in and see how some of this stuff works in real life.  I think I might sit tight until the 3.4 release goes out (should be soon), and get stuck into the doco then.

For the record, I’m talking about Linux kernel-based virtualisation.  It wins my award for the Technology With The Worst Name Ever because if, like me, you go looking on your favourite search engine for issues with Linux kernel-based virtualisation, all you find is issues about keyboard-video-mouse switches.  This is because for the last fifteen years (at least), in the computer industry KVM has stood for keyboard-video-mouse (switch).

This is not your common-or-garden-variety case of acronym overloading, either, because many folk (myself included) still use a KVM.  Usually acronym overloading occurs when the acronym being overloaded has fallen from use and a new technology takes its place, or when an acronym is used to reference a little-known technology and a new usage of the acronym is unaware of the previous usage[1].

Here, KVM was already in heavy use, and I’m sure that none of the Linux kernel hackers could claim to being unaware of the term.  Yet they used it anyway.  And nothing in /usr/src/linux/Documentation explains why.

I’m sure there’ll be something out there about why they chose the name…  but in the meantime I’m left to find other reasons why KVM doesn’t seem to work on my system.  Which KVM do I mean?  Ah, that’s for me to know…  ;)

[1] RTP is an excellent example of this — it was already in use as Rapid Transit Protocol (a part of the APPN-HPR suite), but the VoIP folk never heard of APPN and used RTP for their Realtime Transfer Protocol.

Firstly, a rant.  Why the #@&*%$ should it be so hard to get TV guide data in electronic form?  The holders of this info charge money for it, and have clamped-down in the past on those that distribute it freely (I daresay that the reason the grabber failed is that the mob that was making it available, who were doing so as their contribution to the Open Source community in return for basing their commercial product on FOSS, were told to stop).  As far as I am concerned, Free-To-Air Television should be exactly that, free — I should not have to pay to find out what’s on and when.  All that they are doing is forcing folks on to Bittorrent; by making it more inconvenient and less reliable for viewers to watch when they want (restricted guide data, shows that run over-time and push the schedule out), they ensure that viewers NEVER see their advertisers’ content.

Right, rant off.

So I noticed that I was getting errors from the nightly mythfilldatabase run.  mythfilldatabase was running, but not adding any guide data.  After the problems I’d had in the past with the tv_grab_au script and D1′s data (mysterious timezone shifts, missing data), I immediately thought the worst and renewed my search for an alternate grabber.

One of my work colleagues mentioned Shepherd a while ago, and Google regarded it highly, so I gave it a run.  It seems to be an agreggator of a number of different grabber scripts that each pull data from a different resource, by the looks of things, it tries different grabbers (with a kind-of internal quality rating) and keeps going until it’s filled all the gaps in your guide data.  Neat.

Of course migrating from D1′s data to the Shepherd data was painful, because they use different XMLTV IDs for the stations.  Took me a number of channel scan/configure/mythfilldatabase cycles to get things straight, but it all seems to be good now.

Update: MythTV users at work mentioned that they lost some data for a few days, but it seems to be back.  Oh well.  :)

I’ve had a tuner card for some time, but my first attempts at getting it to work with MythTV under Gentoo a couple of years ago were less than successful.  The final prompter to do something was discovering KnoppMyth, a purpose-built Linux distro based originally on the Knoppix “live-CD” distro.  KnoppMyth has a customised installer that sets up a Linux system with MythTV and a host of other plugins and extensions that take you from zero-to-PVR with a minimum of fuss.

I downloaded the CD of the latest KnoppMyth, threw the tuner card into a PC reclaimed from a defunct project of mine, and sure enough it worked beautifully.

There were a couple of things that didn’t work out-of-the-box though.  I didn’t understand how the setup process configured the “grabber” — MythTV’s name for the process that obtains program listings.  The MythTV setup program runs in its Qt interface, but the configuration of the grabber (at least, the one for Australia) runs in the text-mode window that the Qt program is launched from.  This little misunderstanding cost me some time in getting it set up properly.  Then, for some reason the channels scanned on the tuner card did not match up with the listings obtained by the grabber, so I had to go through each channel entry and manually add the XMLTV URI for the program guide data.  But now that’s done, woo-hoo!  It works a treat.  Even the IR remote control was detected and set up by KnoppMyth.

Buoyed by my success, I went and found a second tuner card for the box (you can’t have a MythTV backend box with just one tuner, you just can’t).  Here, I committed the cardinal sin of Linux — I bought a brand-new piece of kit and expected it to work.  My sole reasonig was “well it’s the same brand as the one I’ve already got, so it must work”.  Idiot.

So the KnoppMyth box is the backend (and a frontend), and I’ve found the XBMCMythTV scripts to use XBMC (on the XBox) as a frontend as well.  MythTV’s functionality in this regard is fantastic — although it is frustrating that every MythTV backend and frontend must be running exactly the same version in order to work together.  KnoppMyth is currently based on MythTV 0.19, and Ubuntu Edgy (which I’m running on the laptops) is at 0.20a…  Probably not too much of a drama, as the laptops are only on 802.11b wireless so running a frontend on them probably wouldn’t work well anyway.

Apart from getting a frontend on the laptops, and the tuner card that doesn’t yet work, the other remaining concern is that it’s just one more box running in the house.  This can be solved however, with a couple of tricks based on Wake-On-LAN.  The MythTV frontend can already be configured to send the “magic packet” to the backend to wake it up if powered off, and there’s a couple of ways to have a backend system wake up to make a scheduled recording (one involves poking the PC’s BIOS to set the timed power-on function, the other is an ingenious method involving a second PC — ideally a low power device like a Linksys NSLU2 or WRT54 — sending the WOL magic packet at the scheduled time).

The announcement of Ubuntu 6.10 (Edgy Eft) gave me an additional prod.  I had a DVD of Ubuntu 6.06 LTS (Dapper Drake) that I figured I could install and upgrade to Edgy, and that process went alarmingly well.  Even installing kubuntu-desktop was painless.  It looks like a really well-integrated distro with just the right amount of knobs and dials to keep me running.

Or so I thought, until it came time to get wireless working.  I run WPA, and the network config tools in Dapper don’t grok it.  I figured that Edgy would be an improvement, but alas not.  I’ve tried just about every network config tool available, in both GNOME and KDE, with no luck.

About the closest I’ve managed to get was using kwlan, but it seemed to get confused in trying to save the configuration and activate the link.  Start wpa_supplicant prior to configure, and things seem to save but nothing activates.  With wpa_supplicant stopped, I cannot save a profile.

I’ve seen forum notes that recommend downloading and building CVS versions of NetworkManager and wpa_supplicant — seems to go against the Ubuntu ethos a bit in my mind (if I’ve got to build stuff from source, I might as well be running Gentoo on it).

So I’m wired, but not for sound.  I like (K)Ubuntu though, so much so I’m downloading a Xubuntu install CD to try it out on a low-spec laptop I am trying to make use of.  Time will tell if the Edgy Eft is just visiting or gets to say a while.

Ever since I moved the main server off Red Hat, I’ve not really had an effective VPN (I know, slack me).  So when it comes to things like e-mail access, I’ve been using various webmail solutions.  At the moment, I even find myself without one of those!  So when Zimbra with all it’s Ajax goodness popped off the magazine pages at me, I jumped.

The trouble is that I have been at somewhat of a crossroads of late when it comes to the Crossed Wires system environment.  Additional complexity is definitely not desirable, unless it comes with a really good ROI or leads to a possible future simplification.  To that end, there are a couple of negatives to Zimbra: it appears to have its own mail infrastructure and its own LDAP, both of which we already have working stably, so unless it can work with existing instances I’d have to work out how to hook Zimbra’s into my own.

I should be fair and say that I got this impression after downloading and starting Zimbra’s pre-built VMware Appliance of the ZCS community build, based on FC4.  So it might be that a standalone Zimbra install just uses your pre-existing daemons.  Which leads me to the next point — Zimbra is not in Portage, and the only install doco I’ve found for it discusses setting up a Debian or Ubuntu chroot gaol and running Zimbra in that.  Not a small amount of management overhead…

So if Zimbra does stay around, it will be as a virtual server using one of its supported distros rather than running natively somewhere (I suppose it could run natively on the CentOS Trixbox/VMware server, but it’s got enough to do).

On Zimbra, well, it’s sweet — as soon as I saw it I thought “I have to have this”.  The Ajax interface looks really nice, and it seems to have all the function of a Thunderbird or Outlook Express killer.  The fact that the same interface gets used regardless of hardware, OS or location is really nice.

Otherwise, I just get my digit out and do a VPN…

Before you say ANYTHING, this is not an indication that the Crossed Wires campus is switching to the evil side.  Any experienced Linux sysadmin will tell you that working with Windows systems can’t be avoided — and in some cases, welcomed (after all it’s better to have one or two Linux boxes in a sea of Windows than no Linux boxes at all).  My main customer at work is essentially a Windows shop, but their main file servers are Linux on zSeries, which means that me as a Linux guy needs to know more than I thought I would want to know about bringing Linux and Windows together.

So they are doing a migration to Microsoft Active Directory, and the Linux systems need to be integrated into the AD setup.  To our architects, Linux Windows integration equals Samba — they never bothered to look at making use of AD’s LDAP component to create a model that Linux can handle natively, instead of the (to me) less-than-optimal Winbind (don’t get me wrong, Winbind works, it just imposes some operational issues that I’d sooner do without, like SID-[UG]ID mapping, for instance).

So I proposed that the solution be updated to utilise LDAP, through the use of Microsoft’s own Services for Unix (SFU).  I was told “yeah, dunno why it wasn’t designed that way, would be the best way to do it, but no”.  Sigh.

So I decided to stick to my guns and set up something to show that it would work exactly as I said it would.  And I have!  I’ve worked around some inaccurate information on the ‘Net, some incomplete documentation from Microsoft, and some finger-checks on my part, to be able to show The Right Way to anyone who cares…  Yep, sometimes the useless thing is just worth doing.  :)

In Australia, there is no evangelism of zSeries.  There’s an attitude bordering on arrogance that seems to say “we’re not going to explain zSeries to you; if you don’t know you want it already then you’re not worth it”.  At least that’s what it looks like to me.

I’m surrounded by people who think that all problems can be solved by installing an xSeries or pSeries machine.  Maybe some can be, but IMHO they’ll be replacing one set of problems with another (possibly greater) set.

Anyway, it’s nice to hear different stories — like a company whose IT costs went from 1.7% to 0.9% of sales by migrating their ENTIRE server farm (including about a dozen p690s) to a z990 running Linux.  Like a company that has placed 250 Linux server guests onto z/VM inside a year, freezing acquisition of new discrete servers.

Then, Linus (Torvalds, for those readers who don’t know Linux and therefore don’t know who Linus is) posts a message something like “err, guys, I think it might be this, here’s a patch which is totally untested and might not even compile so someone should check it”.  Fixes the problem — it’s a workaround to an actual problem in the AMD64 chips (the patch shuts off the TLB flush filter, a component of the chip that seems to behave a little oddly under heavy load and for which AMD have issued at least two errata).

Andi Kleen challenged Linus over his patch, implying that shutting off the TLB flush filter is too heavy-handed (my words, not Andi’s).  Linus then responded by saying “considering the pain this has caused for us, if I get even a single report that it fixes the problem, I _am_ going to commit that fix without any further questions”.

That’s the Linus we know and love!  He lurked away watching the discussion, dropped the fix on there when it became clear that there was little traction, and then dealt out some “Diplomacy: Torvalds-style”.  I’m a fan!

First a bit of history: this machine is a dual-processor Opteron system, and as far as free (as-in beer) Linux distros Gentoo was about the only one that had a x64_64 port available at the time.  Over time it’s grown to have a lot of stuff on it (applications, not just the data), so changing to a different distro will be FAR from trivial.  I know that Gentoo isn’t really a server distro, but this install has a lot of momentum behind it now…

Where was I?  That’s right: VNC.  Something else I really like is VNC.  I had a neat setup on my box that worked like a terminal server: you connect using your VNC client, get a login window from X, do some work, then log out when you’re done.  No having to set up a permanently-running X desktop for every user that might want to connect!  This was set up and working really well, until I just went to use it (after having not used it for a while) and found it broken.  Seems that some other change I’d made since last using it caused the Xvnc process to start segfaulting.  Rebuilding it made no difference.

This led me on a wild ride through Google searches, fora and mailing list archives (with a detour throug the LKML, which I’ll meniton later) to discover that in current versions TightVNC doesn’t play well on 64-bit distributions and that it’s been a known problem for months with no real end in sight.  On someone’s recommendation I removed TightVNC and switched to the RealVNC package, and things started working again (once I fixed a different problem in KDM caused by Gentoo’s configuration file management).

I’m finding more and more that I have less and less time to frig around with this stuff.  I need this kit to JUST WORK, and a bleeding edge distro like Gentoo isn’t helping me.  Perhaps I need to change to using the Gentoo Reference Platform (GRP), which is a pre-built-binary version of Gentoo.  But with the GRP, much of the advantage of Gentoo (custom-built packages, flexibility) is lost.

I guess I’ve been wanting to have my cake and eat it too — I want nicely-tuned custom-built packages, but I want stability and proven integration as well!  I’m going to have to give something up, and I think that stability is going to win.

I’m attracted to CentOS, the respin of Red Hat Enterprise Linux.  I guess I could have a play with that on some other kit and see how it goes…

Coinciding with the relocation of the prime server from Rubicon DC to Ellendale DC, we’ve implemented LDAP authentication for Linux and Mac OS X clients.  There’s also automounted home directories to boot!  It went quite smoothly, all things considered.

Now will come the dreaded data reorganisation (there’s about 1.5TB of storage across all the Crossed Wires machines).  Also, I’ve been running Samba 3 for a while so there’s probably not much reason to keep putting off integrating the Windows boxes into LDAP as well…

I’ve been holding off on buying a TV card for years, partly because of my perception of poor support in Linux, but also because of the state-of-flux that TV in Australia is in (probably in other parts of the world too) regarding Digital TV.  But, XBMC made me take the plunge and I’m thinking it was not a good move.

I wanted to get TV going via XBMC.  There’s built-in support for ReplayTV, but since we don’t have and can’t get that over here I went a-browsing.  There’s a Python XBMC script that hooks XBMC into MythTV — cool, I thought.  I bought a card (supported by drivers written by a local guy, so I figured it would work well and be proven in the local area).

Getting the drivers to build and install was a trial.  Running a 2.6 kernel as I am, the way this guy’s package builds it basically toasts a large portion of my kernel tree and replaces it with symlinks to his code (which looks like a snapshot of the LinuxTV code).  There’s probably a lot of folk out there that don’t know any better, but IMHO you just don’t do that!  On account of how I don’t know a better way to do it though, I’ll shut up.

So I finally got the drivers built.  Where are the dvb-utils?  Ah, good question…  You see on Gentoo, the dvb-utils package builds NOTHING.  The maintainers of this package realise that if you are running a 2.6 kernel you’ll have fairly complete LinuxTV support in your kernel, so they tell you that they are doing nothing.  But what about the user-space utilites?  scan?  tzap?  Grrr…  Watch out for my flaming bug report…

So in the background while a lot of this was happening I was building MythTV.  When it finished, I was itching to watch TV!  BANG — nothing but an error from mythsetup saying that I had to add dvb support if I want to use a DVB card…  Seems there’s a USE flag I was missing.  I like Gentoo, but sometimes…  So I remerge MythTV…

So now I have a working MythTV, and a working card and drivers.  But how do I tune it in?  As they say, nothing worth having is easy to obtain.  This part of MythTV is probably one of its poorest-documented — how to find out the magic stuff you need to plug in to get going with DVB.  After two days of fiddling, I finally Googled a moderately simple method, but it still requires me to key in arcane little numbers and setting for every stream (not just each channel).

So finally I get to watch some TV!  But don’t make it high definition, oh no…  Seems that something in the chain (driver or MythTV) can’t do HDTV on this HDTV-capable card.  And if one of the arcane little numbers is not quite right, the whole things locks up.  Still, it could be worse, I’ve only had a couple of kernel freezes on what was previously a rock-solid machine — I’m just glad that I’m using a desktop machine to test with rather than going with my original plan, which was to put the TV card straight into my server.

It’s no wonder that “people” don’t like Linux.  Almost every endeavour I have undertaken using Linux has required that I undertake an exhaustive course of self-instruction to become aware of almost every facet of the mechanics of what I’m trying to do.  It happened with telephony, with DVD authoring, with TV.  How many people that run DVB cards on Windows would even know what PIDs are?  I have to know, though, because I run Linux.

Lucky I’m such a fucking geek.