It’s been ages since I did an update on the main network machine here, and I bit the bullet over the weekend. 250+ packages emerged with surprisingly little trouble, and all I was left to do was build the updated kernel and reboot.
I usually end up with something that doesn’t restart after the reboot, usually because of a kernel module that needs to be rebuilt after the kernel (because I forget to remerge the package before the reboot, oops). This time the culprit was Asterisk (the phone system), which I also often have trouble with after an update due to a couple of codec modules external to the Asterisk build. This time however the problem ended up being due to the Asterisk CAPI channel driver failing.
Thinking it was the usual didn’t-rebuild-the-module problem, I went looking for the package I had to rebuild… only to find it was masked. Turns out the driver for the ISDN card in the box, a FritzCard PCI, is no longer maintained and doesn’t build on modern kernels, which has resulted in the Gentoo folks hard-masking the entire set of AVM’s out-of-tree drivers.
Help was at hand in the form of a Patton SmartNode 4552 ISDN VoIP router I’d bought months ago to replace the Fritz card. Even though there isn’t much information about how to configure the SmartNode for Asterisk around, I managed to get the setup working in only a couple of hours. I even managed to get the outgoing routing for the work line set up right!
Eventually I’ll get something posted here that goes into a bit more detail about the configuration. Let me know in a comment if you need to hurry me up! 
Archive for category Linux
Network virtualisation
Feb 21
I’ve been doing a lot of mucking around with KVM with libvirt (I keep promising an update here, don’t I). In my desktop virtualisation requirements I had a need for presenting VLAN traffic to guests: simple enough, and I’ve done it before. You can do what I usually do, and configure all your VLANs against the physical interface then create a bridge for each VLAN you want to present to a guest. The guest then attaches to the bridge appropriate to the VLAN it wants access to, with no need to configure 8021q.
(The other method of combining VLAN-tagging and bridging is to bridge the physical interface first, then create VLANs on the bridge. I couldn’t work out how to get VLAN-unaware guests attached to this kind of setup, and it didn’t work for me even to give IP access to the host using a br0.100 for example. Still, it must work for someone as it’s written about a lot…)
I realised that from particular virtual machines I needed to get access to the VLAN tags — I needed VLAN-awareness. Now I knew up-front that the way I could do this was to just throw another NIC into the machine and either dedicate it to the virtual guest or set up a bridge with VLAN tags intact. I really wanted to exhaust all possible avenues to solve the problem without throwing hardware around (as I’ve been doing a bit of that recently, I have to admit).
First, I tried to use standard Linux bridges as a solution, but discovered that an interface can’t belong to more than one bridge at a time, which put paid to my plan to have one or more VLAN-untagging bridges and a VLAN-tagged bridge. I figured it could be done with bridges, but I envisaged a stacked mess of bridge-to-tap-to-bridge-to-tap-to-guest connections and decided that wasn’t the way to go.
Next I checked out VDE, which I had first seen a couple of years ago — but something gave me the impression that VDE either wasn’t really going to give me anything more than bridging would, or was not flexible enough to do what I needed. I like the distributed aspect of VDE (the D in the name) but I’d rarely use that capability so it wasn’t a big drawcard. I widened my search, and found two interesting projects — one that eventually became my solution, and another that I think is quite incredible in its scope and capability.
First, the amazing one: ns-3, “a great network simulator for research and education”. As the name suggests, it simulates networks. It is completely programmable (in fact your network “scripts” are actually C++ code using the product’s libraries and functions) and can be used to accurately model the behaviour of a real network when faced with network traffic. The project states that ns-3 models of real networks have produced libpcap traces that are almost indistinguishable from the traces of the real networks being modelled… I’ll take their word for that, but when you get to configure the propogation delay between nodes in your simulated network it seems to me it’s pretty thorough. Although the way that I found ns-3 was via a forum posting from someone who claimed to have used it to solve a similar situation as me, and ns-3 does provide a way to “bridge” between the simulated network and real networks, the simulation aspect of ns-3 seems to be more complexity than I’m looking for in this instance. It does look like a fascinating tool however, and one I’ll definitely be keeping at least half-an-eye on.
To my eventual solution, then: Open vSwitch. Designed with exactly my scenario in mind–network connection for virtualisation–it has at least two functions that make it ideal for me:
- a Linux-bridging compatibility mode, allowing the brctl command to still function
- IEEE 802.1Q VLAN support (innovatively at that)
The Open vSwitch capability can be built as a kernel module (there’s a second module that supports the brctl compatibility mode), or very recent versions have the ability to be run in user-space (with a corresponding performance drop).
On the surface, configuring an OvS bridge does seem to result in something that looks exactly like a brctl bridge (especially if you use brctl and the OvS bridging compatibility feature to configure it), but its native support for VLANs really brings it into its own for me. In summary, for each “real” bridge you configure in OvS, you can configure a “fake” bridge that passes through packets for a single VLAN from the real bridge (the “parent” bridge). This is exactly what I needed!
For the guest interfaces that needed full VLAN-awareness, I simply provided the name of my OvS bridge as the name of the bridge for libvirt to connect the guest to–OvS bridge-compatibility mode took care of the brctl commands issued in the background by libvirt. The VLAN-unaware guest interfaces presented a bit of a challenge–the OvS “fake” bridge does not present itself like a Linux bridge, so it doesn’t work with libvirt’s bridge interface support. This ended up being moderately easy to overcome as well, thanks to libvirt’s ability to set up an interface configured by an arbitrary script–I hacked the supplied /etc/qemu-ifup script and made a version that adds the tap interface created by libvirt to the OvS fake bridge.
The only thing I might want from this now is an ability for an OvS bridge to have visibility over a subset of the VLANs presented on the physical NIC. The OvS website talks about extensive filtering capability though, so I’ve little doubt that the capability is there and I’m just yet to find it. From a functionality aspect, OvS is packed to the gills with support for various open management protocols, including something called OpenFlow that I’d never heard of before (but I hope that some certain folks in upstate New York have!) but is apparently an open standard that enables secure centralised management of switches.
Detail of exactly how I pulled this all together will come in a page on this site; I’ll make a bunch of pages that describe all the mucky details of my KVM adventures and update this post with a link, so stay tuned!
LDAP groups in Postfix
Feb 10
For a long time I’ve been managing virtual e-mail addresses (the ones you create when you sign up to a web service, so that you know where your spam is originating) using Postfix’s LDAP alias capability. At the time I was still putting every bit of configuration I could into LDAP–particularly if it was user-id related–and I’ve never had a need to change what was working really well.
N’s school recently decided to distribute the weekly school newsletter via e-mail, and had allowance for one e-mail address per family. Not wanting the additional overhead of having to have either S or me receive it and then having to forward it to the other, I thought it would be neat to have a single common address that, when items arrived, distributed the mail to multiple boxes. Of course I took the stupid path of providing the school with a yet-to-be-created e-mail address, foolishly trusting my ability to set the system up before they tried to send anything to it… but in the end it was not so foolish after all, as unbeknown to me I already had everything I needed to achieve my objective.
Unfortunately the first thing I did was assume that I needed mailing list software. I installed Mailman, and started to read-up on the process to get it working. I did this on my yet-to-be-commissioned KVM-hosted mail server (a blog post for another day), and started trying to diagnose why mail wasn’t getting delivered. I had set up Postfix on this mail server to point to my existing LDAP to test, and thought that there was a problem there (but also started to work out if there was a way to use the LDAP server to manage the Mailman aliases). I re-found the Postfix LDAP HOWTO, and stumbled over the section entitled “Example: expanding LDAP groups”. Et voila: multidrop incoming mail without the need for a mailing list manager!
I had always assumed that e-mail aliases were a one-to-one mapping of alias address to real destination. Not the case: an alias can have multiple destinations. It doesn’t just apply to LDAP alias support, either: as per the “aliases” man page you can do
name: value1, value2, ...
In my LDAP situation, all I need to do is list the alias in the “mailLocalAddress” attribute of which ever users need to receive mail for that alias. Done!
I may have to keep Mailman, however. Shortly after this success, I wondered how cool it would be to have the notification SMS messages for voicemail received at home, that currently go only to S, come to me as well. I’m using a hosted email-to-SMS gateway service for this, so the “alias” would have to expand to multiple external e-mail addresses. I’m not sure if you can alias mail addresses that are not in your domain… I’ll have to try and see–might be easier to do that than subscribing to a Mailman list via SMS-to-email!
Asterisk chan_mobile fail
Jul 23
I’ve been struggling with setting up chan_mobile on my Asterisk system. For those fortunate enough to actually get it working, chan_mobile provides an interface for Asterisk to treat a mobile phone like a PSTN or VoIP trunk–when someone calls your mobile phone it can ring your desk phone or softphone, or you can use your normal handset to make an outgoing call on your mobile. It works by making the Asterisk system look like a Bluetooth headset or handsfree to the phone. You can even connect Bluetooth headsets to Asterisk using chan_mobile and have them appear like an extension in your dialplan (although that capability doesn’t seem to be covered very much).
I figured this would be an ideal way to make use of an old Nokia 6230 with a broken speaker. Somewhat foolishly, on the assumption that it would Just Work (and that all the troubles experienced by others would not beset me) I went and bought a two-pack of prepaid mobile SIM cards and went through the adventure of activating them. One of these SIMs I threw into the 6230, the other I kept on hand for after I got everything working. The plan, you see, was to be able to take advantage of free calls between the two accounts by taking one of the phones with me when travelling and leaving the other strapped to Asterisk at home.
I think it’s probably fair to say that I’ve had more success with it than a lot of other folk have. The process of configuring Asterisk to use the Bluetooth dongle is quite straightforward, and it’s even quite easy to configure the chan_mobile driver to have calls enter your Asterisk system in a routable way. When I dialled the “tethered” mobile from another phone, I was rewarded with the ringing of my desk phone–and at this point, I think I gave myself the kiss-of-death. “Wow, that was easy,” I thought…
When I picked up the desk phone, I was rewarded with silence. Not just the silence of the phone not ringing any more, but also the silence of no audio being passed either way over the call path. Nothing put the pure, desolate sound of FAIL.
Things actually went downhill from there, believe it or not. I have tried a total of four different Bluetooth dongles, with results ranging from the aforementioned signalling-but-no-audio to why-the-@#%$-won’t-this-thing-pair. The three different phones I’ve tried elicited a similar spectrum of results. “Make sure your dongle has a Cambridge Silicon radio, they definitely work” say the forum experts… Sorry guys, one of the biggest failures I had–failure of Asterisk to pick up the call–was on the last dongle I tried and, yes, it was a CSR. I’ve even had two different versions of the bluez stack and (I think) two different asterisk-addons versions.
The one thing that I’ve distilled from all the experiences I read through is that there is a ridiculously high level of sensitivity to particular phone and dongle features. For example, great success has been reported with the Nokia 6230i. I figured that I was lucky and that a 6230 would be close enough… Doesn’t look like it. There is one model of D-Link Bluetooth device–no longer in production, by the way–generally reported to give the most success. Tweaking the device class reported by the bluez stack in the Linux host is said to give success too, but led to me being unable to get a connection to Asterisk. Unfortunately, I have neither the time nor the patience to spend too much time trying to go through the motions of getting it working. I tell you, if it really is that difficult to get two Bluetooth devices to talk to each other it’s no wonder that the majority of folks still use wired headsets!
Luckily all this little experiment has cost me so far is time. The two-pack of SIM cards cost me the grand total of $2, and they had enough start-up credit on them to allow me to receive calls without a top-up. The handsets are from that ever-growing pile of GSM hardware that just about every modern household is accumulating now (well, at least the ones that house a gadget-freak who can’t even bear to part with a broken one). The kernel version I’m running on the system could be an issue, since I get ugly error messages from the btusb module when I take a call, so a kernel update might help. After that though it’s likely to cost real money–buying a new/different Bluetooth dongle, for example.
If anyone out there has suggestions on something else to try, I’m listening (reading? watching?). I don’t mean to complain, after all I am one that usually subscribes to the “it’s Open Source, it’s the hard work and dedication of others, you got it for nothing, you’ve got no right to complain” philosophy. It is really frustrating to come away from a couple of days’ effort with nothing to show for it, though.
LDAP-backed DNS and DHCP…?
Jul 22
I’m having a bit of an infrastructure redesign here at the Crossed Wires campus. Each time I have an outage (the last one was caused by a power failure) I learn a little more about the holes in my current setup and what I can do better.
I’m implementing a router box on an old low(-ish)-power PC that will be backed up by a virtual machine on my main virt-box. I’ve already done most of the preparation of using keepalived to implement VRRP, and a colleague has given me some pointers in using the Linux-HA tools like Heartbeat and DRBD to make services like e-mail and Samba redundant.
I’ve had a soft spot for LDAP for ages; I’ve always thought that putting as much backend data into LDAP as you can would be a really good way to get failover and redundancy. Instead of having to deal with every single server’s different way of doing replication and failover, just bung everything into LDAP and get that replicating. Sounds good in theory, but in a nutshell it’s not working out that way for the two least-celebrated but most important components of my (arguably any) network: DNS and DHCP.
There are a number of LDAP-backed DNS projects out there. If I’m willing to go to the bleeding edge with BIND on my Gentoo build I can get access to the two most talked-about ones (bind9-sdb-ldap and the BIND DLZ LDAP driver), and other solutions like PowerDNS and ldapdns are available. But none of them offer integration with DHCP, and I’m currently using dhcpd’s “interim DDNS update method” to make sure that hostnames are seen in my DNS when a lease is granted (okay, there’s a Perl daemon that goes with bind9-sdb-ldap, but it seems like a sort-of clunky afterthought).
Speaking of DHCP, LDAP backends for it are virtually non-existent. The only LDAP-enablement I’ve found for ISC DHCP involves putting the config file into LDAP, not the leases… I actually used that for a few days a while ago and pulled it out because it was actually more work to do it that way (and for no benefit in failover).
It seems to me it would be a project ripe for the picking: take an integrated DNS/DHCP server like dnsmasq and make it write into LDAP instead of to a file. If I had more free time I’d probably have a go at it, except for the fact that no-one really seems to be that interested in storing DNS and DHCP in LDAP: that it hasn’t been done says to me that there’s no demand for it, and it’d end up being a big waste of time and effort.
Over to you, lazyweb… Is this a yawning chasm of unfulfilled networking dreams, or a case of me trying to make something more complex than it needs to be? After all, the rest of the world gets by with DNS master-slave and DHCP failover, they should be good enough for me too, right? 
I recently started having trouble with APT transactions on my Kubuntu desktop. “apt-get update” would fail for some source entries with the error “The HTTP server sent an invalid reply header”. I thought it was something specific to (K)Ubuntu, but when I had the exact problem on my NSLU2 running Debian I figured the problem must be elsewhere…
I’d recently updated the machine that provides the transparent web proxy function for the network; one of the updates took Squid up to version 3.0 (from 2.6). This was the first thing I was suspicious of.
There’s an option in Squid that controls how it handles an “If-Modified-Since” request from a client. The default is for Squid to respond based on the age of the item in the cache, not based on the real item on the source web page. The comments in the Squid config file indicate that some clients use an IMS when requesting a reload — looks like APT is one of those clients.
Setting this option to “on” (from the default of “off”) in squid.conf fixed the issue for me:
refresh_all_ims on
Comments and Downtime
Mar 15
Observant readers will notice that they are no longer able to respond to posts. The blog-spammers have won the battle but, as they say in the classics, they will not win the war…
I've turned off the comment capability, until I can get something in place to bring the rubbish under control (a recent update to PolarBlog helped a bit, in that the crap doesn't display on the site any more, but when I log on I get to see the mess). I'm thinking of a new site, where I can discuss technical stuff a bit more and thoroughly while keeping the private stuff separate if I need to.
The site has had a bit of downtime recently, due to my non-existent monitoring of what's happening on my hosted server. This will change shortly, and I'm looking forward to things returning to the stability they had when I was self-hosting.
Photo printing pain
Dec 16
S went to print some photos the other day, and what was supposed to have been a simple exercise turned out to be a very frustrating one for both of us. I was utterly amazed to discover that even on the eve of 2009 there are web sites that think the world is only viewed through Windows…
S's and my respective creative sides are being adequately satisfied by the iLife suite on the Mac, but there are times when we need to get the pictures out of the silver tower and onto other media—on this occasion paper, for albums and so on. A large retailer here has part of their floor space in each store set aside for those photo printing kiosks, and I introduced S to the art of putting photos onto a USB stick so that she could print some photos when next she went there…
On her return from the shop, she reported that we hadn't successfully put the photos she wanted onto the stick. When she'd plugged the stick in, she'd found only less than half of the photos we'd stored there. Sure enough, when I plugged the stick in all the files were there safe and sound. Strange thing was I could find nothing in common about the files (uppercase/mixedcase filename, long or 8.3 filename, datestamp, etc) that would have yielded the number of photos that the kiosk had found on it.
Annoying, but life is too short to worry about it. After all, this same retailer was plastering adverts of their new web-based photo printing service… S could submit the photos online for printing and pick them up from the store later.
<sarcasm>This is where the fun really started.</sarcasm>
Their app is Flash-based but seems to have some Java involved as well. While it loaded quickly enough, the app portion of the web page had an incongruous grey background that just looked dodgy. S had to create an account and sign onto the site just to get this far though, which was a bit annoying.
The workflow seemed to be to create an album, upload pictures to the album, then select photos from the album for processing. Creating the album went fine, but when the upload function was selected there were no action buttons visible to complete the operation! S was using Safari, but Firefox made no difference.
Then I suggested she use her laptop, which runs Ubuntu 8.04. The situation actually seemed a bit better to start with, as instead of the upload function showing an embedded file selection dialog like it did on the Mac we got a "normal" GNOME file dialog box. However, only some of the photos showed again: this time, it was because they had hard-coded a non-modifiable filename filter for the dialog that was only picking lower-case file extensions!
Trying to work around this, I mounted the stick manually with different mount options. I succeeded in getting all but one of the files showing with a lowercase name, and a rename fixed that one. Back in the web page however, it still didn't like us: any file chosen from the dialog box resulted in a nonsensical error message followed by a "You have selected no files to upload" dialog.
S was beyond caring by this stage (she has a very low threshold for being stuffed around by technology). She went to Snapfish after a friend's recommendation, and found a well-designed and easy to use WEB site that required no downloads or other junk.
So why did this wind me up to the point of spending all this time blogging it? Because nowhere on Big-W's site is there any mention of browser or operating system compatibility. Not even a "we've tested only on Windows, Mac users may experience difficulty"[1]. Not a blessed thing. Their Help page has a single paragraph about trouble uploading, blaming "your IT Department" for "setting certain network properties that inhibit the upload tool from working".
I wonder if the developers of the app were just so blind to believe that their gunk would just work wherever it was run, or whether they really think that it's a Windows world. Of the two I hope it's the former.
So Snapfish gets a recommendation for being not just an application hosted on the web but a web application. They do good photos too!
[1] I never expect to see Linux mentioned on these things and get pleasantly surprised on the occasions it is; even if it says "Linux is not supported", someone there at least knows enough to mention it.
Newcomers to UNIX-like operating systems are often confused by the difference between the shell operations pipe and redirection. The difference is easily explained with an example, in the context of web development. The shell command echo "st=1" | ./lifeswork.pl shows how a pipe is used to supply command line input to a script usually invoked via CGI in a web server. This allows the script to be more easily debugged by testing at the command line. The shell command echo "st=1" > ./lifeswork.pl shows how redirection uses command line input to overwrite a script file, destroying the file and the web developer's sanity. Hopefully this example illustrates the difference between pipe and redirect, and helps you avoid the idiotic mistake I just made.
DandyID
Delicious
Disqus
Dopplr
Facebook
Google Reader
last.fm
Linkedin
Twitter
1113