Process substitution

In addition to widely used shell features like file globing and command substitution, modern shells like bash or zsh support something that is called process substitution. The goal of process substitution is to enable syntax which will allow us to get some commands marked within special characters behave as they were regular files. Common special characters for both mentioned shells are:

<(command list)
>(command list)

In addition to these, zsh adds additional syntax in form of:

=(command list)

Lets start with <() syntax. In it's simplest form, it can behave just as a regular pipe. For example, this is completely equivalent syntax:

ls -l | sort

or

sort <(ls -l)

What actually happens under the hood is that output of ls -l is connected to sort command through the mechanism of named pipes (FIFOs) or by using a file in /dev/fd

Off course, this was a simple usage pattern and nobody would use process substitution in this way instead of classic pipe scenario. But suppose a situation where you want a command to operate on results of more than one command. For example, you want to compare content of two directories (let's forget on dircmp command for a moment):

diff <(ls dir1) <(ls dir2) 

Here the command acts just like you have saved output of both ls commands to separate files, and then ran diff on those two files. Or for example, you could use something like:

paste <(cut -f1 file1) <(cut -f2 file2)

Or maybe you want to edit the output of some complex command directly into your favorite text editor - no problem, just use syntax similar to this:

vi <(ls -l /etc)

Now when you know how to use <(), usage pattern for >() is straightforward. You can send the standard output of a command to more than one command. For example (just make sure that MULTIOS option is set in your shell before running this, ie setopt multios for zsh):

ls > >(sort) > >(sort -r)

If you are using zsh (and you should! :), you can also use =() snytax. The only difference between this syntax and <() or >() is that =() actually creates a file on filesystem instead of using FIFO. This comes handy if command you are substituting process to is using system calls like lseek(2).

Going mobile

Until now I've always had a regular cell phone, i.e. not a smart phone or any similar kind of "smart" embedded device. That was fine with me as I've always used a cell phone only for making calls and sending text messages. After trying several vendors and models, I've finally decided to try something new and buy a smart-phone. As I was a starter in that area, I decided not to buy expensive gadget like Nokia N96 (although I was considering N95 for quite a bit), so at the end I bought a Nokia E66. Minimal set of features I required at the start was including wireless and GPS, and as I was biased to the Symbian based model (spoiled with Nokia's regular cell phones) I decided for E66.

Wireless is an enormous gain on a cell phone, and you won't know what you're missing once you don't start using it. Now I don't even bother turning my desktop or laptop PC on just to check my email (or Facebook :), I can do it with my cell phone. Besides that, there are many good web sites optimized for small screens found on cell phones. Be sure to try Wapedia (Wikipedia), BBC, Slashdot, Facebook and dozens of other sites optimized for a mobile web browser. It's not even a problem to find an open wireless access point, at least in the big cities, but if you live in a smaller city you'll most probably use GPRS/EDGE/HSDPA (which is btw extremely expensive here in Croatia). I assume you have a wireless connection at home, so this won't be a "show stopper" at all.

The next very useful thing to have on a cell phone is a GPS. Last month I was in London for ten days, and it saved me a lot of time to find locations I was interested in (and to get back on a familiar place once I got lost :). I suggest you to download maps for a location (country) you are traveling to on your PC at home (or directly via mobile browser and wireless), because you'll get quite a big bill to pay if you download it ad hoc via GPRS or EDGE (especially in roaming). Map of England for Nokia Maps was around 35 MB, so I assume you'll find similar sizes on all other GPS software. So bad that Google Maps (Symbian version) doesn't support such an option. Also, be sure to turn off a option to find GPS satellites over the Internet (AGPS), especially when in roaming, for a obvious reason. It will make your GPS device somewhat slower in finding satellites for the first time, but later you won't notice a difference (if you are using it regularly).

The latest great thing I "discovered", thanks to my colleague who had already been using it, was a great application called Nimbuzz. It can be used as chat program (MSN, Jabber, GTalk, ICQ...), and what's even better, it can be used for making Skype calls (or to some other SIP based applications). I've tested it and it works perfectly, now I can talk for hours without using my PC or wearing those annoying headphones with microphone. You can also take a picture or record a sound clip and it will automatically send a link to that content (uploaded to the Nimbuzz server) to the recipient over the selected instant messaging protocol. The best part of the story is that this application is totally free (as in "you don't need to pay money for it") and there are versions for Symbian, Windows Mobile etc. Give it a try, you want be disappointed.

There are far too many benefits to mention them all here (for example motion sensor, great Symbian applications etc), but the bottom line is that smart phone is really worth of money you give for it, if you need such functionality. Knowing myself, I won't by a regular phone again for sure.

Fixing fibre channel connectivity problems on AIX

Unix and storage administration always brings interesting and new problems ready to be solved. Few days ago I've encountered one of those problems. Task was to configure two AIX powered VIO servers to access LUNs on CLARiiON storage, i.e. to make high availability environment with several AIX LPARs accessing storage through VIO servers.

After wiring the environment, i.e. hell of finding which port is right by using only lsslot and instructing technician located in data center 80 km far away, next step was to zone the SAN switches. But something was wrong. Every VIO had one dual port FC HBA and every port was connected to a different SAN switch. But instead of seeing two new WWNs (i.e F-ports) on switch, there were two L-ports.

SW1> switchShow
Area Port Media Speed State Proto
=====================================
...
6 6 id N4 Online L-Port 1 public
7 7 id N4 Online L-Port 1 public
...

Situation on the other switch was OK - connected ports were reported as F-ports. For some reason one port of every HBA was L-port (loop port), and the other was F-port. After trying several things, I've managed to get things working as expected.

Apparently this is quite usual problem with AIX, so here is a cookbook how to fix it. You need to unconfigure FC devices and FC SCSI I/O controller protocol, and then set the link initialization protocol to Arbitrated Loop and Switch (AL).

First, look for FC devices:

# lsdev -H -Ccdriver -F "parent name description"
parent name description
fcs0 fcnet0 Fibre Channel Network Protocol Device
fcs1 fcnet1 Fibre Channel Network Protocol Device
fcs2 fcnet2 Fibre Channel Network Protocol Device
fcs3 fcnet3 Fibre Channel Network Protocol Device
fcs0 fscsi0 FC SCSI I/O Controller Protocol Device
fcs1 fscsi1 FC SCSI I/O Controller Protocol Device
fcs2 fscsi2 FC SCSI I/O Controller Protocol Device
fcs3 fscsi3 FC SCSI I/O Controller Protocol Device
iscsi0 iSCSI Protocol Device
powerpath0 PowerPath Control Device
sissas0 sas0 Controller SAS Protocol
sissas0 sata0 Controller SATA Protocol

From here we can see that FC HBAs are fcsX and FC SCSI I/O protocol devices are fcsciX, where X is in range from 0 to 3. Now, deconfigure devices and set the appropriated link initializaion mode. Don't forget to run cfgmgr at the end.

for i in 0 1 2 3
do
rmdev -Rdl fscsi$i
rmdev -Rl fcs$
chdev -l fcs$i -a init_link=al
cfgmgr -vl fcs$i
done

Now you should see both fabric logins and ports listed as F-ports:

SW1> switchShow
Area Port Media Speed State Proto
=====================================
...
6 6 id N4 Online F-Port 10:00:00:00:c9:76:07:33
7 7 id N4 Online F-Port 10:00:00:00:c9:76:0c:2b
...

Linux I/O accounting

One feature one would expect from an enterprise grade operating system is definitely a good I/O scheduler, but also some tools for monitoring it's performance. Starting from 2.6.x series, Linux has selectable I/O schedulers, actually four of them: no-op, anticipatory, deadline and complete fair queuing (CFQ) scheduler. Comparing to the older kernels, it was a great step ahead, but we were still lacking tools for I/O monitoring. Why was it so?

Up to the version 2.6.20, there was no I/O accounting support in the Linux kernel. Although process accounting was included in the kernel a long time ago, obtaining informations about per process I/O utilization was a very difficult task. Information you could get, for example with iostat command, was telling you that something is causing a heavy I/O load, but it was up to you to find out what process is actually causing it. Finding out that process was not an easy task, especially if you were running system with hundreds of processes, for example with ten Oracle instances.

Finally, Linux kernel 2.6.20 and newer included I/O accounting. Now every running process has associated I/O information in /proc/self/io. To enable this, besides having a recent Linux kernel, you must enable the following options in your kernel build configuration:


CONFIG_TASKSTATS=y
CONFIG_TASK_IO_ACCOUNTING=y


If you are running any recent Linux distribution, you probably already have this option, so there is no need for recompiling. Unfortunately, most "stable" Linux distributions you would see on production environments, like Debian or RHEL, use 2.6.18 at this moment, so you'll have a bit of work to do here.

Tool for per process I/O usage monitoring I use is iotop. It behaves much like widely used top or htop. i.e. it displays table of processes and threads on the system along with their I/O usage. Displayed data includes read and write I/O bandwidth in sampled period, and also percentage of time the thread spent while swapping in or waiting for I/O. iotop is actually a Python script, and if your favorite distribution doesn't already have it in packages, you can get it here.

I would strongly recommend trying iotop to every professional Linux administrator and even to beginning Linux users, because it helps a lot and gives you very usable information when doing a full system diagnostics.

(K)QEMU speed tests

QEMU without additional accelerators is very slow, and that is a well known fact. If you have a modern processor with virtualization options you will most probably want to use KVM. You can find out if your system can run KVM by checking the flags line in /proc/cpuinfo for vmx (Intel) or svm (AMD).

$ grep -E '^flags.*(vmx|svm)' /proc/cpu

Unfortunately my CPU doesn't have such options. I have one of those Core2Duo processors that lacks suck features (T5500, commonly used on laptops). So if you're in situation like me, you're stuck with KQEMU to speed the things up. Up to yesterday, I was not fully aware what is the performance penalty of not using one of the accelerators. I've made some tests of my own to see how virtual machine behave with and without accelerator module.

Tests were run on Debian GNU Linux (lenny), kernel 2.6.25 and with QEMU version 0.9.1. Underlying hardware was IBM Thinkpad R60e, Core2Duo 2x1.66 GHz with 1 GB of RAM. For testing purposes I've used guest with Red Hat Enterprise Linux 5.1. Tests were focused on three areas I was most interested in:

1. Processing speed, calculate as many RSA public/private keys as possible in 10 sec (openssl speed rsa4096)
2. Network speed, download 10 MB file with wget (wget URL)
   
3. Disk speed (hdparm -tT)

First, each of tests was run on host OS (Debian), then in virtual machine without KQEMU and finally in virtual machine with KQEMU module loaded. The results are:

Calculation of RSA public keys (10 sec)
Host OS: 7202
KQEMU: 6188
QEMU: 372

Calculation of RSA private keys (10 sec)
Host OS: 98
KQEMU: 85
QEMU: 6

Buffered disk I/O
Host OS: 33.51
KQEMU: 24.51
QEMU: 25.47

Cached disk I/O
Host OS: 918.97
KQEMU: 442.71
QEMU: 445

Network I/O (download of 10 MB file)
Host OS: 52
KQEMU: 53
QEMU: 51

While virtual machine I/O performance was pretty much the same in both cases (slightly better without accelerator module), number crunching tasks like RSA key pairs generation is where KQEMU shows up. KQEMU accelerated guest achieves at most 90% of host speed, but without accelerator only about 10% which is very slow. Tests also show that disk performance on guests is around 50% of that on host.

Next thing I want to check is VirtualBox performance. I expect it to be slightly better of that of KQEMU. Unfortunately there is no Xen enabled kernel in Debian packages for lenny so I can test Xen performance.