Mark isgetting an internal server error on http://tinyurl.com/n5fgd9.  I think I amgetting an internal server error too.

Unless, of course, I don’t get it.

(The first result got fixed, this version (courtesy Google’s cache) has the goodness.)

I’ve been extremely slack at blogging recently. It’s ironic that what has been keeping me from blogging has probably been the very thing that I could be blogging plenty about: the constant travel to Canberra. I’m planning on writing a bit more about my experiences of and in our nation’s capital, so for now here is the first edition.

I’ve been doing the three-day-per-week thing now for nearly two months and it’s fair to say that I’m over it. It’s not the weather so much: okay, it’s cold, but at the moment I’m breezing in and out so I don’t have to live with it. It’s not the loneliness; I’ve been working at home (and away) for long enough that solitude doesn’t bother me at all. I have a bit of an idea of things it might be, though…

* The inconsistency of accommodation: one week I’ll be staying five-plus stars, the next I’ll be in a little fleabag place with barely adequate heating and hundred-year-old floors (which is quite a feat when the city itself has only been there eighty years[1]). One week I’ll have a kitchen and be making myself healthy meals, the next I’ll be eating lukewarm takeaway noodles using toothbrush handles as chopsticks.

* Trying to do the right thing by work by saving money: this has me riding my motorbike to the airport at 5am to save a night’s accommodation and a couple of taxi fares. I’m DEFINITELY over that.

* Separation from N: Although he’s coping really well now (much better than me, I think), it worries me that I’ll be one of those absent fathers that ends up having only a passing influence on the lives of their children. A bit dramatic, sure, as this is not meant to last for long, but how long is too long?  How long is long enough to do damage? It seems like I do little else but spend time with N when I get back, which is probably making things worse.

* Separation from S: I think we’ve changed. We used to do the living-apart thing really well (when I was working in Melbourne and Auckland, we really seemed to be living proof that “absence makes the heart grow fonder”), but now things are different. She is looking after N, and is in full-time work, so she has a lot more on her plate than before, and I barely get wound-down from one trip than I’m having to gear up for the next.

* Not being able to “connect” with the place: When I have worked away from home in the past, I went to special effort to do things that would familiarise me with the locality. In Melbourne I went on tram rides “just because”, and in Auckland I’d ride my motorbike here and there. Day trips, intended to not only help me know my way around but to give me a feel for where my career had sent me. With Canberra, however, these work trips have never been across weekends and I’ve not had any opportunity to do my “connecting”. If not for a colleague of mine who was living in Canberra about eight years ago, and whom I visited one weekend while I was working in Melbourne, I would never have visited places like the Australian War Memorial, Parliament House, Mt Stromlo Observatory, and the comms tower on Black Mountain. After all the visits I’ve made I’ve never seen the National Gallery, The Royal Australian Mint, and the dozens of other things worth visiting.

However, probably the number-one thing that’s getting to me:

* Having little (nothing) to do when I get there: I’m a technical person, and I feel most useful doing technical things. From that perspective, there is no need for me to be in Canberra to do what I’m doing. My presence there at the moment is merely a perception thing, building trust and a relationship; intangible things that probably are tremendously significant to the client and their perceptions, but don’t register with me at all.

I mentioned how well N is coping with me being away: he’s doing so well that I’m in awe. It seems like he’s keeping me sane. He got upset the first couple of times, but now he almost packs my bag for me! I took him to preschool the other day and this is the conversation that took place:

I said “now, Mummy’s going to come and get you this afternoon, mate.”

“Why?” said N.

Oh great, I thought, I almost made it without mentioning anything. “Well I’m going on the plane this afternoon,” I said.

“Are you going to Sydney, or Canberra?” he asked.  I was taken-aback: no tears, no don’t-go-daddy-I-need-you…

“Canberra,” I told him, “back to Canberra.”

“Oh, okay Daddy,” said N, “have a good trip. Bye!”, and off he ran to play with his friends.

Four. Years. Old.

So on Canberra… I think I’ve worked out which are the roads not to be on when I need to get from place to place (not having to spend much time in, near, or passing through Civic helps in that regard). I used to take a drive around after work at least once a trip (the closest I’ve been able to come to my “connecting”), but I’ve stopped doing that now as I think I’ve been to most of the districts. I have a bit of an idea what are the good food places, and which restaurants one can go to (as a male person dining alone) without ending up feeling like a total outcast.

Next update I might talk a bit more about the places I work and stay. Don’t expect anything too soon, though, as I have about 18 more months in which to do it…

[1] Okay, my little joke. The floors obviously aren’t a hundred years old. They do, however, sound like they belong on the sound-stage of a cinematic re-enactment of the voyage of the First Fleet.

It doesn’t take much to see just how rubbish I am at this blogging thing. The last few months have been full of interesting happenings, and the blog has been silent. I’m not going to embarrass myself by saying “I’ll do more frequent updates, really I will” because I know it won’t happen. Oh well, as appears in the gospel according to Dirty Harry, chapter Magnum Force, “A man’s got to know his limitations”.   

Okay, I’m sure it’s not really like that. I’m ashamed to admit this one bit me, though: ORDB, which 16 months ago called it quits but has still been online and more-or-less functional, decided last week to false-positive all queries (a mailing list thread on the issue can be seen here. Harsh, but fair–I do stuff-all email through this site but it still has been costing someone bandwidth to bounce my silly queries back to me. Multiply that up by all the part-time-sysadmins like me that don’t pay close-enough attention, follow old wiki articles, or get bad advice, and pretty soon you’re talking about a HEAP of bandwidth wasted to NOT provide a service any more. Shame it had to end like this though.

Just wended my way through another SLES 10 install on s390x. It’s b0rked though, and I’ll probably have to redo it. I had some kind of I/O error during the install which seems to have resulted in a couple of the filesystems being remounted read-only. Not too much trouble you’d think…

Some things aren’t starting because of missing binaries in /usr, frustrating but probably recoverable. The network startup is totally clagged though, and I can’t even begin to work out how what happened… happened.

During bootup, at the time it tries to configure the network interface, I get streams of error messages about problems running the “ip” command. The error text is full of garbage that the init script is trying to parse as text configuration–it looks like a corrupted filesystem or a binary file.

I manually configured the network (not a trivial task in s390x, it must be said), and started to poke around. I got this when I logged in as root:

Last login: Sat Mar 15 12:48:36 2008
/usr/X11R6/bin/xauth: error while loading shared libraries: libXau.so.6: cannot open shared object file: No such file or directory
-bash: read: read error: 0: Is a directory
lxs0za01:~ #

Okay, so I won’t get funky X-based YaST.  No problem, I’ve spent more time in the ncurses-mode YaST anyway…

lxs0za01:~ # yast
warning: the ncurses frontend is installed but does not work
You need to install yast2-ncurses to use the YaST2 text mode interface
lxs0za01:~ #

WHAT!!! What the @#!$ happened there?!?!?

Okay, so I’ve calmed down about that, so I go looking for the problem with the network initialisation…

lxs0za01:~ # cd /etc/sysconfig/network
lxs0za01:/etc/sysconfig/network # ls -go ifcfg*
-rw-r–r– 1   141 2006-06-17 07:30 ifcfg-lo
lrwxrwxrwx 1    16 2008-03-15 02:23 ifcfg-qeth-bus-ccw-0.0.0f00 -> /lib64/ld-2.4.so
-rw-r–r– 1 27470 2006-06-17 07:30 ifcfg.template
lxs0za01:/etc/sysconfig/network #

Priceless. You can’t make this stuff up. I cannot for the life of me work out how this could possibly have happened. I guess I just blame it on a whacked-out filesystem and move along.

Okay, so both of these issues probably have extenuating circumstances unrelated to SLES or YaST… but it’s nice to have a vent now and then. I’ll write up something a bit fairer once I fix this b0rkedness.

“Once upon a time, in a land far from here, there was a clothing factory. One part of this clothing factory made uniforms. The uniforms came in many shapes and sizes: some were sporting team outfits, some were “corporate wardrobe”, others were emergency services uniforms… but they all looked alike. Few of the people working in the factory spent any time actually making uniforms — more time was spent on things like adding or removing badges and insignia and players’ names from the uniforms, or completing reports to state that the holes in customers’ uniforms were appropriately mended (whether or not any such holes actually existed)…

“Sometimes the sleeves would fall off, and the pockets would be on the back instead of the front (or on the front instead of the back), and when this would happen and the customers complained the managers of the factory would make lots of noise and implement  more procedures and promise the customers it wouldn’t happen again. The factory workers would have to complete more reports, spending more time on the reports and still less time making uniforms.

“Many of the factory workers were unhappy, and dreamed of getting back to making uniforms instead of writing reports and sewing badges. Others dreamed of designing uniforms. Still others dreamed of moving to a different part of the factory, where suits or pyjamas or swimming togs or wedding dresses were made without the need for any of the strict rules that the managers said were required when making uniforms…”

Part Two coming soon…

Since I watch Planet KDE it was easy to get caught up in the excitement around the launch of the new version of KDE (the announcement is here). I was unable to resist giving it a try on the laptop! So this post is coming from Konqueror 4.0.0.

I tried an early Beta of the KDE 4.0 Live CD, but it was still using the KDE3 Kicker and was also a bit unstable. I wasn’t sure if it was the fact I was running in a virtual machine that made the graphics a bit flaky or whether it really was beta-quality code making things a bit funny. The KDE team put a lot of effort into bug-swatting in the weeks leading up to 4.0 being tagged, and it’s a lot better now!

This announcement from the Kubuntu folk shows how to get the KDE 4 packages installed on Gutsy. KDE 4 installs in a different path to KDE 3, so you can try out KDE 4 without affecting your existing environment.

I did have a bit of a heart-starter with this though, as apt-get wanted to remove a package called “kdebase-bin-kde3″, which looked risky! It’s okay though, as equivalent binaries are provided by “kdebase-bin-kde4″. In fact, if you follow Kubuntu’s instructions exactly, you should not see the issue: it happened to me because I did a system update after adding the Kubuntu PPA repository but before installing KDE 4. The system update brought a bunch of updated KDE 3 packages out of the PPA, one of which was to replace the standard “kdebase-bin” package with a “kdebase-bin-kde3″.

First impressions are that Oxygen (the new artwork for 4.0) looks great — it’s a very modern look. Some might think it borrows from Vista, but to me it’s got as much of Mac OS X’s appearance as that of Aero. Plasma (the desktop shell) does some interesting things, like turning desktop icons into widgets, but I’m yet to spend enough time with it to experience the other improvements it brings.
The biggest thing I’m looking to trying out is the compositing built into the window manager, KWin. Unfortunately the laptop is a bit old for this to work well (or at all in fact), so I’ll either have to find some magic Xorg setting or get the KDE 4 packages on the desktop machine. I’ve had trouble running Beryl and Compiz thanks to something about the terminal program Yakuake tickling a long-lived bug in X11 (I think part of the reason it’s long-lived is that the X11 folks don’t accept it as a bug but rather a fringe case that Yakuake shouldn’t be exercising, hence a stand-off) so it will be interesting to see if KWin has the same kind of issue.

As for bugs, well there look like plenty. As I’m keying this, Konqueror is chewing 100% CPU and the characters are delayed by a couple of seconds (and of course, now that I observe this, it stops doing it). Still with Konqueror, this is about the third time I’ve tried to post this thanks to Konqueror segfaulting for strange reasons. Also, the Alt-F2 program launcher reports that it was unable to launch whatever you told it to, even though it does so successfully.

There has been plenty written by the KDE folks about the “1.0.0 release of KDE 4″, and they’re copping a fair amount of stick from people who think they’ve done the wrong thing by releasing as 4.0.0. I’m on KDE’s side. Although many KDE folks have used their KDE 4 builds as their daily desktop for months, I haven’t seen anyone who wears a KDE hat recommending that others do so. The term “will eat your children” has been used to describe KDE 4 by folks from the KDE team, so there has never been any pretense that KDE 4.0.0 would be a daily desktop for the majority of users. I’ve never really participated in large-scale software development, but I can see their motivation for releasing what they had as 4.0.0 — I’m proof of it. As long as it was a beta I was not really all that fussed about trying it out; even after there were release candidates I wasn’t all that keen. As soon as you call it a release, however, your early-adopters rush in and kick the tyres and your real testing can start.

By being open about 4.0.0′s status (and I don’t think you can get more open than “will eat your children”), they can make sure that subsequent releases are a lot better than they would be if they dragged on in perpetual beta — the model that Google and the Web 2.0 fraternity seem to insist is better, plodding on for months hiding behind beta status and its implicit “get out of jail free” card.

Instead, KDE has shown the courage to take their code, along with its bugs, and hold it up as something they are proud to give to the world. It’s the foundation not only for future releases of KDE, but possibly the start of new ways that people work with their computers. By working with the community, instead of closeted away from it, I believe the KDE team will succeed.

Okay, so that finished a bit more ra-ra than I planned! Seriously, give KDE 4.0 a try… but if you aren’t happy to suffer a few bugs then by all means wait until 4.0.1 or even 4.1. Oh, and be free.

In part one I mentioned how I was considering using Google Talk as my main chat ID. As it turns out, I talked myself out of it pretty quickly after I delved into using Google Talk to connect to MSN and other services as I do now with my own Jabber server. While there are a lot of links around for using Jabber transports to hook your Google Talk ID to other services, there’s a tiny catch… well, actually, I think it’s a bloody great huge catch personally.

You see, it wasn’t until I read the how-tos that it became clear how it works. The trick is that Google doesn’t run Jabber transports on their own servers, so you therefore need to take advantage of various “open” Jabber servers that do (“open” in this context refers to a server that lets you use its transports without necessarily being a registered user there).

Seeing there didn’t seem to be any restrictions on the servers that could be used, I figured that I could use my own server. Sure enough, after the right incantations to expose the service on the ‘net, I could connect my Google Talk ID through the Jabber-MSN transport on my server to my MSN account. Yay, right? Well, not really — each little test message I sent in either direction incurred three trips over my Internet connection! Yes, three: one to go from my Google Talk client to Google, one back from Google to the transport on my Jabber server, then a third from the transport to MSN. Obviously the same happens in reverse as well (for incoming messages from MSN).

Seeing this as a less than optimum setup, and also being wary of getting listed as a Google Talk-friendly Jabber transport provider, I lopped the transport’s external visibility and went back to using my own JID for transport access. It’s a bit of a shame too; since fring (mentioned briefly in my last post) doesn’t let me connect to an arbitrary Jabber server, to keep connected to everything I’d need two mobile chat programs running.

It’s not like I do that much IM that I need to keep all this running, but it is at least a little bit interesting…

I reactivated an idle Google account the other day. A friend of mine from the Netherlands invited me ages ago but I never really did anything with it until I discovered that a Google Mail account can be used for other Google stuff as well, including Google Talk. I read that Google Talk is based on Jabber and works with any Jabber client, so I flicked over to Kopete and plugged in the details. Sure enough it worked… but then it got interesting.

I run a Jabber server for internal things. I wanted to have a secure, private chat facility to use over VPN with my nephews; I want to someday migrate my Nagios IRC bot to Jabber; and I use transports to link into MSN and Yahoo! to reach friends on those networks. The last point is great: I really like the fact that now, from whatever Jabber client I use (even the mobile ones I’ve played with) that I merely connect to my Jabber server and I’m online on MSN and Yahoo! as well.

Google Talk, though, has proven to be a bit of a challenge. It’s actually working like a tower, even though it’s based on (arguably) the most open of the IM platforms! You see I more-or-less took for granted that “transport” way of doing things, using my Jabber server to bridge to other networks. There’s no Jabber transport for Jabber though!

What I want to do kind-of flies in the face of how Jabber is designed. Ideally, you’re supposed to only have one Jabber ID (JID) — Jabber creates an open network with servers establishing connections when needed, very much like e-mail, and you only need an ID on one server to be able to chat with anyone on any other server. So what I wanted to do, which was connect to one Jabber server and have it “relay” messages to an ID on a different server is just not necessary with Jabber. Nor should it be necessary for Google Talk users to send messages to me using my Google Talk ID only — they can send straight to my JID on my Jabber server.

In the early days of Google Talk, Google had not enabled the “server-to-server” functionality that allowed this kind of communication to happen. Google Talk worked just like MSN, Yahoo! or AIM — you had to have a Google Talk account to chat with anyone on Google Talk. While this was the case, folks were looking making a Jabber-Jabber transport for connecting Jabber servers to Google Talk. At some point, though, Google opened the connectivity paths that allowed Google Talk to exist on the open Jabber network (I’ve tested this for myself). Once this happened, the need for a  ”Google Talk Transport” for Jabber evaporated in most people’s minds.

The solution nowadays is to use a client that supports multiple connections, and connect to your Jabber and Google Talk accounts at the same time. It works of course, but you don’t get the nice benefits that a transport provides — the main one being access to all your IM services and accounts from a single server connection.

So now, having resigned myself to not being able to bring my home JID and Google Talk ID together, the question arose: do I still need my own Jabber server? My current fave mobile IM client only connects to Google Talk… Could I get by just using the Google Talk service? Find out in Part two!

Something ugly happened to my LDAP database a while back, and I never noticed. I saw it had lost a bunch of records, but I’d put it down to some replication problem and never investigated. It wasn’t until I tried to replace one of the lost records, and got an error from LDAP telling me the non-existent record already existed, that I figured something was really wrong.

Multiple iterations of db_recover, attempts to re-index, dump-and-restores of the raw Berkely DB files… Nothing helped. In the end, all that was left was the slapcat-delete-slapadd dance.

(You know that your OpenLDAP is especially sick when commands like slapcat generate glibc backtraces. )

So with what was left of my LDAP data, I started to compare against my replicated LDAP server. The first thing I noticed was that a number of records that I expected to have been replicated were not. I figured that records in the master directory that were lost to database corruption and not to an LDAP operation (a modify or delete) should have been present on the replicated copy. This was not the case, which makes me think that replication only takes effect after the master directory’s backend is updated, and if something like a corrupted database prevents the master from being updated then the replication doesn’t take place. As Zaphod might say, ten points for directory consistency but minus several million for data preservation…

(As I think about this though, the more it doesn’t make sense. If slapd had been unable to update the backend, and hence the replication didn’t take place, surely that would have been returned to me as an update error? I know for a fact that the data I lost made it to the database because I tested an app using the data. It’s unreasonable to me to think that BDB would have returned success on a write operation unless it had actually done so, but I suppose write-caching might create an opportunity for that to occur… No, I suspect a different problem, maybe just replication being suspended at the time, as the real reason that some data was missing from the replica.)

Next I found, despite what I thought was happening based on the lost records, there were quite a few records that were on the replica. This makes me think I’ve had multiple failures, apparently at different times, that have impaired my master directory — one that caused new updates to be lost, the other resulting in loss of existing data.

I’ve added a step to my Bacula processing that performs a slapcat and backs up the resulting LDIF, so if anything happens in the future I have a bit of a chance of running through old files and restoring. The other thing that I’ll kick off is a process to verify the accuracy or integrity of the replica — this might tip me off to a problem sooner rather than later.

My theory on what the cause of this hassle was? Well a while ago I was having a bit of trouble with partitions filling. At a guess I’d say that OpenLDAP was trying to do something (update a transaction log maybe) at a time when the partition its data lives on was full, and got twisted. Soon I’m going to write a separate post with my (updated) thoughts about isolation of failure domains…

For those that haven’t seen it, here’s the process I used to get things back:

# cd /var/lib
# slapcat > whatsleft.ldif
# /etc/init.d/slapd stop
# mv openldap-data openldap-data-old
# mkdir openldap-data
# chown ldap:ldap openldap-data
# cp -a openldap-data-old/DB_CONFIG openldap-data/
# cd openldap-data
# slapadd < ../whatsleft.ldif
# chown ldap:ldap *
# /etc/init.d/slapd start

Obviously if you find yourself in the unfortunate position of having to use this process, substitute your distribution's values for the path to the OpenLDAP data directory and the user/group that LDAP runs under.