In April this year I attended the zSeries Special Interest Group miniconf[1], part of the greater Independent Oracle Users Group (IOUG) event COLLABORATE 11.  I was amazed to discover that there are actually Oracle employees whose job it is to work on IBM technologies — just like there are IBM employees dedicated to selling and supporting the Oracle stack.  Never have I seen (close-up) a better example of the term “coopetition”.

On my return from the zSeries SIG and IOUG, I’ve become the local Oracle expert.  However, I’ve had no more training than the two days of workshops run at the conference!  The workshops were excellent (held at the Epcot Center at Walt Disney World, no less!) but they could not an expert make.  So I’ve been trying to build some systems and teach myself more about running Oracle.  I thought I’d gotten off to a good start too — I’d installed a standalone system, then went on to build a two-node RAC.  I communicated my success to one of my sales colleagues:

“I’ve got a two-node RAC setup running on the z9 in Brisbane!”

“Great!  Good work,” he said.  ”So the two nodes are running in different LPARs, so we can demonstrate high-availability?”

” . . . ”

In my haste I’d built both virtual machines in the same LPAR.  Whoops.  (I’ve fixed that now, by the way.  The two RAC nodes are in different LPARs and seem to be performing better for it.)

Over the coming weeks, I’ll write up some of the things that have caught me out.  I still don’t really know how all this stuff works, but I’m getting better!

Links:

IBM System z: www.ibm.com/systems/z or www.ibm.com/systems/au/z

Linux on System z: www.ibm.com/systems/z/os/linux/index.html

Oracle zSeries SIG: www.zseriesoraclesig.org

Oracle Database: www.oracle.com/us/products/database/index.html

[1] Miniconf is a term I picked up from linux.conf.au — the zSeries SIG didn’t advertise its event as a miniconf, but as a convenient name for a “conference-in-a-conference” I’m using the term here.

After I got the Fedora system working, I thought it would be a good idea to have other systems in the complex using SSL/TLS also.  The process was moderately painless on a SLES 10 system, but on the first SLES 11 system I went to YaST froze while saving the changes.  I (foolishly) rebooted the image, and it hung during boot.  Not fun.

After a couple of attempts to fix up what I thought were the obvious problems (each attempt involving logging off the guest, connecting its disk to another guest, mounting the filesystem, making a change, unmounting and disconnecting, and re-IPLing) with no success, I went into /etc/nsswitch.conf and turned off LDAP for everything I could find.  This finally allowed the guest to complete its boot — but I had no LDAP now.  I did a test using ldapsearch, which reported it couldn’t reach the LDAP server.  I tried to ping the LDAP server by address, which worked.  I tried to lookup the hostname of the LDAP server, and name resolution failed with the traditional “no servers could be reached” message.  This was odd, as I knew I’d changed it since it was pointing to the wrong DNS server before…  I could ping the DNS by address, and another system resolved fine.

I thought it might have been a configuration problem — I had earlier had trouble with systems not being able to do recursive DNS lookups through my DNS server.  I went to YaST to configure the DNS Server, and it told me that I had to install the package “bind”.  WHAT?!?!?  How did the BIND package get uninstalled from the system…

Unless…  It’s the wrong system…

I checked /etc/resolv.conf on a working system and sure enough I had the IP address wrong.  I was pointing at a server that was NOT my DNS server.  Presumably the inability to resolve the name of the LDAP server I was trying to reach is what made the first attempt to enable TLS for LDAP fail in YaST, and whatever preload magic SLES uses to enable LDAP authentication got broken by the failure.  Setting the right DNS and re-running the LDAP Client module in YaST not only got LDAP authentication working but got me a bootable system again.

A simple fix in the end, but I’d forgotten the power of the resolver to cause untold and unpredictable havoc.  Now, pardon me while I lie in wait for the YaST-haters who will no doubt come out and sledge me…  :-)

I’ve come back to implement some of what I’d put together then, and unfortunately found…  not errors as such, but things I perhaps could have discussed in a little more detail.  I’ve been using the z/VM LDAP Server on a couple of systems in my lab but had not enabled RACF.  I realised I need to "eat my own cooking" though, so decided to implement RACF and enable the SDBM backend as well as switch to using Native Authentication in the LDBM backend.

Native Authentication provides a way for security administrators to present a standard RFC 2307 (or equivalent) directory structure to clients while at the same time taking advantage of RACF as a password or pass phrase store.  Have a look in our Redbook for more detail, but basically the usual schema is loaded into LDAP and records are created using the usual object classes like inetOrgPerson, but the records do not contain the userPassword attribute.  Instead of comparing a presented password against the field contained in LDAP, the z/VM LDAP Server (when Native Authentication is enabled) issues a RACROUTE call to RACF to have it check the password.

In my existing LDAP database, I had user records that were working quite successfully to authenticate logons to Linux.  My plan was simply to enable RACF, creating users in RACF with the same userid as the uid field in LDAP (I have access to a userid convention that fits RACF’s 8-character restriction, so no need to change it).  After going through the steps in the RACF program directory, and various follow-up tasks to make sure that various service machines would work correctly, I did the LDAP reconfiguration to get Native Authentication.

At this point I probably need to clarify my userid plan.  The documentation for Native Authentication in the TCP/IP Planning and Administration manual says that the LDAP server needs to be able to work out which RACF userid corresponds to the user record in LDAP to be able to validate the password.  It does this by either having the RACF userid explicitly specified using the ibm-nativeId attribute (the object class ibm-NativeAuthentication has to be added to the user object), or by matching the existing uid attribute with RACF.  This is what I hoped to be able to do; by using the same ID in RACF as I was already using in LDAP, I planned to not require the extra object class and attribute.  In the Redbook, because my RACF ID was different from the LDAP one I went straight to using the ibm-nativeId attribute and didn’t go back and test the uid method.

So, I gave it a try.  I had to disable SSH public-key authentication so that my password would actually get used, and once I did that I found that I couldn’t log on.  It didn’t matter whether I tried with my password or pass phrase, neither was successful.  I read and re-read all the LDAP setup tasks and checked the setup, but it all looked fine.  In one of those "let’s just see" moments, I decided to see if it worked with the ibm-nativeId attribute specified in uppercase…  and it did!

Okay, so it appeared that the testing of uid against a RACF id was case-sensitive.  I decided to try creating a different ID, with an uppercase uid, in LDAP to double-check.  Since phpLDAPadmin wouldn’t let me create an uppercase version of my own userid (since that would be non-unique), I created a different LDAP id to test:

[viccross@laptop ~]$ ssh MAINT@zlinux1
Password:
Could not chdir to home directory /home/MAINT: No such file or directory
/usr/X11R6/bin/xauth:  error in locking authority file /home/MAINT/.Xauthority
MAINT@zlinux1:/>

My MAINT user in LDAP has no ibm-nativeId attribute, so the only operational difference is the uppercase uid (the error messages are caused by the LDAP userid not having a home directory; I use a NFS shared home directory had I hadn’t bothered setting up the homedir for a test userid).

The final test was to change the contents of the ibm-nativeId attribute in my LDAP user record to lower-case — and it broke my login.  So that would seem to indicate that the user check against RACF is case sensitive wherever LDAP gets the userid from.  I’m going to have a look through documentation to see if there’s something I need to change, but this looks like something to be aware of when using Native Authentication.

I also noticed that I didn’t describe the LDAP Server SSL/TLS support in the Redbook, but that’s a post for another day…

… and the question was, does the CPACF run at the speed of the CP (i.e. it runs sub-capacity if the CP is sub-capacity) or does it run at full speed like an IFL, zIIP or zAAP.  If the latter, the result after the upgrade should be the same as before — that would indicate the speed of crypto operations does not change with the CP capacity, and that CPACF is always full speed.  If the former, we should see an improvement between pre- and post-upgrade, indicating that the speed of CPACF follows the speed of the CP.

Place your bets…  Okay, no more bets…  Here’s the chart:

 
The graph compares the results from the first chart in blue (when the machine was at capacity setting F01) with the full-speed (capacity setting Z01) results in red.

Okay, so did you get it right?  If you know your z/Architecture you would have!  As the name suggests, the Central Processor Assist for Cryptographic Function (or CPACF) is pretty-much an adjunct to each CP, just like any standard execution unit (like the floating point unit, say).  It is not like the Crypto Express cards, which are actually an I/O device and totally separate from the CP.  Because it is directly associated with each CP, for sub-capacity CPs its CPACF is bound to the speed of that CP.

If you look closer, further evidence that CPACF performance scales with capacity setting can be seen in the respective growth rates of each set of data points.  To see this a little clearer (because I don’t know the right mathematical terms to describe the shape of the curve, so I’ll just show you) I drew a couple more graphs:

Looking at the left graph (which is the same as the bar graph above, just drawn in lines) you can see that in both the software and the CPACF case the lines for before and after the upgrade follow the same trend with respect to the block size.  If these lines followed different trends — for example if the Z01 CPACF line was flat across the block size range instead of a gently falling slope like the F01 line — I’d suspect something else was affecting the result.  Looked at a different way, the right-hand graph above shows the "times-X" improvement between software and CPACF.  You can see that the performance multiplier (i.e. the relative performance improvement between software and hardware; CPACF speed is 16x software at 8192 byte blocks) was the same for each block size.

Now, just to confuse things…  Although I’ve used OpenSSL on Linux as the testing platform for this experiment, most Linux customers will never see the effects I’ve demonstrated here.  Why?  Because Linux is usually run on IFLs, and the IFL always runs at full speed!  Even if there are sub-capacity CPs installed in a machine with IFLs, the IFLs run at full speed and so to does the CPACF associated with the IFLs.  I’ll say again: CPACF follows the speed of the associated CP, so if you’re running Linux on IFLs the CPACF on those IFLs will be full capacity just like the IFLs themselves.  If you have sub-capacity CPs for z/OS workload on the same machine as IFLs, the CPACF on the CPs will appear slower than CPACF on the IFLs.

As far as the actual peak number is concerned, it looks like a big number!  If I understand it right, 250MB/sec would be more than enough speed to have a server doing SSL/TLS traffic driving a Gigabit Ethernet at line speed (traffic over connected sessions, NOT the certificate exchange for connection establishment; the public key crypto for certificate verification takes more hardware than just CPACF, at least on the z9 anyway).  And that’s just one CP!  Enabling more CPs (or IFLs, of course) gives you that much more CPACF capacity again.  Keep in mind that these results are using hardware that is two generations old — I would expect z10 and z196 hardware to get higher results on any of these tests.  Regardless, these are not formal, official measurements and should not be treated as such — do NOT use any of these figures as input to system sizing estimates or other important business measurements!  Always engage IBM to work with you for sizing or performance evaluations.

The weather here is, obviously, hot and humid. It’s been far from unbearable though, in fact I’d almost say it’s comfortable (which is quite something from someone who usually can’t stand hot weather). I’ve rediscovered the transport system, the excellent MRT train system with its regular services and its cheap fares, and I’m using it to go to and from work.

I’ll make some further notes as the week goes on. Wish me luck with the training!

I can’t stress enough what a great job David, Neale (Aussie, Aussie, Aussie!), Adam and everyone at SNA have done.  Networking is not there yet, but I trust it’s not far (need a hand fellas? (: ).  It must have been a hard slog, and for some (particularly Neale) perhaps brought some unpleasant memories (anyone remember Bigfoot?).  Congratulations are deserved.  I can see the lolcat now: I IS SUN. IM IN UR MANEFRAYM, KIKIN OUT YR PENGUINZ.  YA RLY!  Only joking!

The catch is, ironically, the aspect of the port that makes it most useful in the “real” world.  The guys have made the port dependent on z/VM.  Don’t get me wrong, it’s the right thing to do — without z/VM, you can’t play to the strengths of the System z platform and it’s capabilities for massive resource sharing in a virtualisation environment.  Many believe that Linux on System z should have been taken in the same direction, as other platforms (like System p) do big-single-Linux-footprint better than what System z does.

The twist is that by tying the OpenSolaris port to z/VM, they’ve eliminated a set of would-be hackers from contributing to the effort.  Those with motivation, time, skill, and a big Intel box who can get a couple of hundred MIPS out of Hercules.

There are, rightly or wrongly, a lot of people who think that Solaris is a good platform.  These are the kind of people I’m thinking of — maybe folks who have always derided the mainframe, but perhaps are now thinking “gee, well if it runs Solaris now, it can’t be all bad.  Maybe I’ll check it out”.

Obviously I can’t speak for Sun (nor for IBM or SNA), but I’m sure I read that one of the objectives of OpenSolaris was to get Solaris into more hands and to try and benefit from the “millions of set of eyes” effect that Linux enjoys.  It seems ironic then that the first “non-Sun” platform to which OpenSolaris has been ported is one that doesn’t contribute to that goal.

Not to worry.  David at SNA has stated that they are committed to releasing their work to the community.  This will be the point at which an interested party could look at the code and potentially rip out or rewrite the z/VM-specific bits and replace them.  It wouldn’t be impossible — even CMS was able to IPL standalone once upon a time — but it would be a huge piece of work (no doubt part of SNA’s reasoning was to let z/VM do a lot of heavy lifting for I/O and such tasks; that would have to be written for OpenSolaris).  Bags I not-it.  Likewise, our potential interested party would be very likely to turn away to Linux… or even away from System z entirely.

Meh, enough doom-talk.  I’ve downloaded three different flavours of OpenSolaris for x86 (NexentaOS which I had a brief look at previously, Solaris Express Developer Edition, and something that called itself the “Indiana Preview”) and I’m running them in VMware to have a poke around (but not all at the same time, they need a heap of memory).

I’ll be following this as close as I can (or as close as I’m allowed).  I think it will be really interesting to see how this progresses.  Good luck to all involved (and if you need a hand guys…

In Australia, there is no evangelism of zSeries.  There’s an attitude bordering on arrogance that seems to say “we’re not going to explain zSeries to you; if you don’t know you want it already then you’re not worth it”.  At least that’s what it looks like to me.

I’m surrounded by people who think that all problems can be solved by installing an xSeries or pSeries machine.  Maybe some can be, but IMHO they’ll be replacing one set of problems with another (possibly greater) set.

Anyway, it’s nice to hear different stories — like a company whose IT costs went from 1.7% to 0.9% of sales by migrating their ENTIRE server farm (including about a dozen p690s) to a z990 running Linux.  Like a company that has placed 250 Linux server guests onto z/VM inside a year, freezing acquisition of new discrete servers.