Daily nitpicking by ipozgaj

Python for Perl programmers #1 - fileinput module

2011-05-07T14:59:00.000+02:00

Python for Perl programmers is a set of short posts showing how some of the most common Perl idioms can easily be implemented in Python. This time I'll show you how to translate code chunks using Perl's diamond operator to elegant Python code.

while (<>) {
    ...
}

This loop written in Perl will iterate over all of the lines from standard input, whether it is pipe from another program or the list of the files specified as the program's arguments (argv). The same functionality can be done in Python by using the fileinput module, so the previous code snippet becomes:

import fileinput

for line in fileinput.input():
    ...

The default order of processing is:

sys.argv[1:]
sys.stdin

If file name is specified as "-", the standard input is used instead. You can get more information on fileinput module in official Python documentation.

Managed attributes in Python

2010-10-20T21:55:00.000+02:00

Attribute interception is a common pattern that occurs in almost every object oriented language. In addition to only changing the value of some attribute in object, in most cases you will need to implement some additional logic that deals with the changed value. Most straightforward solution for this by using setters and getters. This way the user of the object won't access the attribute directly - he will call the helper functions that hold all the logic needed for fetching and changing the attribute value.

To be honest, there is nothing wrong with this kind of approach, but it just isn't pythonic and elegant. The purpose of this article is to give a brief introduction on more advanced Python topics dealing with this problem. In most cases, you will find them to be just a syntactic sugar around setters and getters, but knowing more about these techniques will help you to read and understand code from more experienced Python programmers, and writing more effective and elegant code of your own.

setters and getters

As we have already mentioned, the most simple approach is to use setter and getter functions. For demonstrative purposes, we will use an example class Person with attribute name that verifies the name is always properly capitalized.

class Person:
    def setName(self, name):
        self._name = " ".join([e.capitalize() for e in name.split()])
    def getName(self):
        return self._name

>>> p = Person()
>>> p.setName("joe smith")
>>> p.getName()
'Joe Smith'

As shown, this just works, but we need to call functions instead using the attribute name directly. The rest of the examples will show how to avoid this kind of programming.

__setattr__/__getattr__and __setattribute__/__getattribute__

Next, we will show how to intercept the access to the attribute, so we can add whatever logic we need to process before the actual access has been made. We can do that by using either the __setattr__ or __setattribute__. Of course, if your goal is to implement something inside getter, and not setter, you would use the __getattr__ and __getattribute__ methods instead. Let's continue with our example - first we will show how to use __setattr__.

class Person:
    def __setattr__(self, name, value):
        if name == "name":
            value = " ".join([e.capitalize() for e in value.split()])
            self.__dict__[name] = value

Note that simply assigning value to the attribute wouldn't work, as it would loop in the __setattr__ call indefinitely, so we must add it to the __dict__ explicitly.

Approach with __setattribute__ is very similar, but there is one notable difference. While __setattr__ and __getattr__ will be called only for those attributes that are undefined, i.e. not on an instance and not inherited from a class, __setattribute__ and __getattribute__ will be called every time you try to access the attribute. In this matter, __setattribute__ and __getattribute__ are more generic as they will be called every time. Again, you need to be careful to avoid loops. While implementing setter with __setattribute__ will be the same as with __setattr__, there is a notable difference while implementing the __getattribute__ method. Here we can't use __dict__[name] as calling it would trigger the __getattribute__ again, causing a loop. To avoid this, you would use the object.__getattribute__ instead of self.__getattribute__.

Properties

Properties allow us to reroute the attribute's set, get or even delete operations to the functions of our choice. To do this we will use the property built-in. The basic syntax that can describe the use of properties is (all of the arguments are defaulting to None, and we will silently ignore the last two arguments for now):

attribute = property(get, set, del, doc)

How to use this? Well, it's pretty straightforward. Our example from previous sections rewritten to use properties looks like this:

class Person(object):
    def setName(self, name):
        self._name = " ".join([e.capitalize() for e in name.split()])
    def getName(self):
        return self._name
    name = property(getName, setName)

Note that we have derived our Person object from object, stating that we want to use is it as the new style class. This is required for properties to function normally (of course, if you're using Python 3.0+ you won't need this as new style classes are implied).

Properties can also be coded with decorators. In this case, base decorator @property is used for getter, and name.setter is used for setter. Again, the following example is merely syntactic sugar for the previous example:

class Person(object):
    @property
    def name(self):
        return self._name
    @name.setter
    def name(self, name):
        self._name = " ".join([e.capitalize() for e in name.split()])

In contrast with the previous methods, properties manage only a single attribute, and therefore are easier to write as you don't need to remember all the gotchas of __getattr__, __getattribute__ etc.

Descriptors

The final method we will examine are descriptors. In their essence, descriptors are superset of properties, and property built-in we have used before is only a specific case of descriptors. Descriptors allow us to reroute attribute's get and set (or even delete) operations to methods of another class.

The template code for coding descriptors is:

class Descriptor:
    def __get__(self, instance, owner):
        ...
    def __set__(self, instance, value):
        ...
    def __delete__(self, instance):
        ...

We will again ignore attribute deletion for now. A class containing any of these three methods is descriptor. Omitting the __get__ or the __delete__ method will cause that particular type of access as unsupported, however omitting the __set__ method will not render the attribute as read only, it will only allow its name to be redefined (and therefore descriptor will be hidden). If you want to make your attribute read only, you should implement __set__ with pass statement or raise an exception on assignment.

Argument names for descriptor methods are somewhat strange. Let's start with __get__ method. The owner argument is the name of the class to which the descriptor instance is attached, and the instance argument is the instance of the object through which the attribute was accessed. Meaning of the instance argument in the __set__ method is the same, and the value argument holds the value to be assigned to the attribute.

How do we use a descriptor? We just assign it to the attribute name. Our example, this time with descriptors, looks like this:

class Descriptor(object):
    def __get__(self, instance, owner):
        return instance._name
    def __set__(self, instance, value):
        instance._name = " ".join([e.capitalize() for e in value.split()])
 
class Person(object):
    name = Descriptor()

Again, we must use new style classes if we're not using Python newer or equal to Python 3.0.

An interesting thing with descriptors is that they can hold state. OK, technically speaking every of the previously described methods can also hold state, at least in the form of global variables, but by using descriptors you can use them elegantly in their own namespace - the descriptor class and instance.

Conclusion

This is by far not a complete tutorial on using any of techniques described here, as there are many more things to be described, but they are simply out of the scope of this article. You should read the official Python documentation or some of the better books on Python programming (I would strongly recommend Learning Python, 4th ed. from O'Reilly) to get the complete overview of managed attributes possibilities.

Back on the road

2010-07-22T18:57:00.002+02:00

After almost a year of pause, I've decided to revive this blog. Much has changed in the meantime, so I'm full of ideas and themes to write about. Some of them are:

cloud computing: I've almost completely abandoned all of the applications I've been using before (mail client, feed reader, office applications, personal information management applications etc), and switched to online and cloud versions (primarily Google's)
Android platform: more like continuation of the previous point - I can now access all of the cloud services I use from literally anywhere, thanks to this great mobile platform
development: migration from subversion to git, several themes on Perl, Python and shell scripting
systems administration: several capacity planing, storage optimizations and general system tweaking themes

So, stay tuned! :)

Windows 7 hype

2009-10-24T12:09:00.003+02:00

Sometimes it's really interesting to see what good marketing can do for a mediocre or even good product, even if the product itself doesn't offer anything revolutionary.. These days all you can find on RSS, IT news portals, Twitter etc can be summarized into one term - "Windows 7". First of all, I must admit - although I'm a long time Linux user/admin and that's the main thing I do for living, I got my hands on Windows 7 RTM. So here are my two cents on the new Microsoft's product and hype around it after using it for a little longer than a month.

Undoubtedly, Windows 7 is the most advanced desktop OS ever released from Microsoft. I wouldn't say it is the best desktop OS currently available, but it is probably the most consistent one. As I've never used Windows Vista, it was quite a surprise to log into this new OS, and to find dozens of new features. On the first look, it is far more visually appealing than XP. Much of concepts that I've been missing in the previous versions of Windows are now here, for example media and document libraries, better network manager, network profiles (home, domain, work, public etc). Advanced users (like myself :)), will be happy to hear that PowerShell 2.0 is included in the OS by default, and you event get a simple IDE for it (PowerShell ISE). As I can hear, it even runs much faster and uses much less resources than Vista, but I can't confirm this as I've never used Vista.

Of course, there's no such thing as perfect software product, and some things must be criticized. First of all, 10 GB for basic OS installation is simply silly. With some other operating systems (most notably Linux), you get OS and almost all other software you need in much less, like 3-4 gigs at most. Memory usage is also pretty high for an OS kernel and several core services, and I don't accept "RAM is cheap nowadays" as an argument. OS should use much less memory resources than it is the case with Windows 7. Period. I've also noticed a constant disk activity even when I'm away from my computer, and this is really annoying if your hard disk is not among the most silent ones. Finally, there is a nasty bug I've encountered (and some of my colleagues, and thousands of people as I can see on the Internet) - problem with crashing after locking the screen (either with win+L or after an idle period) while being logged on domain network. I've temporary "fixed" this problem with disabling screen locking in the registry, but there is still no fix from Microsoft. My colleagues have even reported this problem to Premier support, but we are still waiting for the solution.

But let's return to the main thing that made me start writing this article, marketing and hype about the Windows 7. Microsoft announces features like Aero shake, snap and peek as revolutionary, and that's what makes me laugh. First of all, you could have used such features years before Windows 7 on Linux (Beryl and Compiz). And seconds, silly GUI effects like those previously mentioned can hardly be described as something revolutionary (they could have at least "invent" virtual desktops... but no). In my opinion Mac OS or Linux are far ahead of Windows if you're main goal is to have nice desktop with eye candies. I mean, it's nice to have such things, but you can live without them.

To summarize, if you're a long time Windows user, you should switch to Windows 7. If you're Linux of MacOS user, you should probably stick to your OS. Windows 7 is polished, OS somewhat faster, and with some new features, but it's not nearly so "unique and revolutionary" as Microsoft announces it.

"Secret" Perl operators

2009-09-02T20:27:00.007+02:00

Here is a list of Perl operators you might not have seen so often as usual operators. Strictly speaking, the operators we'll show are not operators at all, yet they are ingenuous little hacks that can save you some time and lines of code in some common circumstances. On the other hand, guys who will debug or maintain your code might not be so happy about it, as these operators are commonly used to obfuscate Perl code. Here is the list

    symbol    nickname                Role
    ------    --------                ----
    <=>       spaceship               documented operator
    0+        venus                   numification
    }{        eskimo greeting         END{} in one-liners
    =()=      goatse                  lenght of array
    ~-        inchworm on a stick     high-precedence numification
    ~~        inchworm                scalar
    "@{[]}"                           join $", ...
    -+-       spacestation            high-precedence numification
    ~-        cotton candy
    ~~         caterpillar
    "@{[]}"    cyclops
    -+-        cross-hair
    ]->[       Frowning Sam

Don't abuse your compiler

2009-07-21T21:04:00.002+02:00

Today I was asked to take a look at some broken code to find out what was going wrong. Code snippet that caused a process to SEGFAULT was something like this:

char *tempfile_template = "tempXXXXXX";
mkstemp(tempfile_template);

Of course, code was wrong as mkstemp() will try to change XXXXXX in variable tempfile_template with random identifier, and char *tempfile_template declares a pointer to data in .rodata section that is read only.

$ objdump -s a.out
...
Contents of section .rodata:
80484b8 03000000 01000200 74656d70 58585858  ........tempXXXX
80484c8 585800                               XX.

$ objdump -h a.out
...
14 .rodata       00000013  080484b8  080484b8  000004b8  2**2
                CONTENTS, ALLOC, LOAD, READONLY, DATA

However, code was working before, and quick look into old Makefile revealed -fwriteable-strings option, which caused string "constants" to be placed into writeable sections. The option itself was abandoned somewhere about gcc 4.0, and version of gcc used was 4.3.

Clearly, this was an example of bad programming habits and poorly written code. You should always avoid using obscure options such as -fwriteable-strings, as someone who will eventually read and debug your code may not be even aware that such an option exist.

Package selection with aptitude

2009-07-12T17:07:00.005+02:00

Most of the Debian-like distributions users still use apt-* commands for package management and as interface to core packaging libraries. Even worse, they use Synaptic or similar GUI ;). Starting from Debian 4.0 (etch) aptitude is the prefered package management software, but it seems that it still has quite small user base compared to other ways of package management.

Although I could spend hours writing arguments on why you should use aptitude, this time I'll focus only on describing techniques for package selection in aptitude. In past, if you wanted to select specific packages to perform some action on a package set, you could use dpkg or apt-cache. Aptitude unifies functionalities of those two programs (and many more) and what is more important it brings new syntax that enables writing of more complex selection rules than before.

Let's start with basics. Search patterns that we'll be discussing can be used with any aptitude command like search, remove, purge, install, hold etc. Aptitude search patterns enable you to make searches based on specific package properties like package name, description, author etc. Almost every search pattern can be written in two forms: short and long. You will recognise short patterns as tilda (~) character followed by a single character (search category) and a search term, and you will recognise long search pattern as a question mark followed by a search category and a search term inside of brackets.

Before we start, just a short notice: all commands will be written as they are used in bash shell. If you're using some other shell (most notably zsh) you will have problems because it probably uses different globing and escaping syntax than bash. For example, if you're using zsh you'll have to quote search patterns (write them inside of quotation marks) or/and escape special characters like tilda (~) or question mark (?).

Simple searches

If you write a single word after an aptitude command, it will perform search based on package name. So, the following command will search all packages with word "python" in their name:


aptitude search python

Equivalent (but superfluous) syntax for this is:


aptitude search ~npython
aptitude search '?name(python)'

Take a note, I'll write both short and long syntax for all commands. But, let's suppose we want all packages that have word Python in their description. Then we'll use:


aptitude search ~dpython
aptitude search '?description(python)'

There are many search patterns that you could use for daily package management. For example, you may wonder how many packages you can upgrade after you have refreshed your package list with aptitude update. Just run:


aptitude search ~U
aptitude search '?upgradable'

Before I've started using aptitude, I've used the following command to purge packages that have left their configuration files on my system, but the owning packages have been removed:


apt-get --purge remove $(dpkg -l | awk '/^rc/{print $1}')

It's much more elegant to do this in aptitude:


aptitude purge ~c
aptitude purge '?config-files'

Let's find out packages that are maintained by packager "Josip Rodin":


aptitude search ~m'Josip Rodin'
aptitude search '?author(Josip Rodin)'

For list of all possible search patterns, take a look at aptitude reference manual.

Advanced searching

You may combine as many simple search terms as you want by using logical operators and/or/not, and you can write even more complex searches by using ?for keyword. Let's start with examples. We'll search for all packages from development section that provide C compiler and their name is not gcc:


aptitude search '~sdevel ~Pc-compiler !~ngcc'
aptitude search '?and(?and(?section(devel),?provides(c-compiler)), ?not(?name(gcc)))'

Or, let's find out what currently installed packages depend on python:


aptitude search '~i ~Dpython'
aptitude search '?and(?installed, ?depends(python))'

And finally, there is a king of all complex searches with aptitude and dream of every Debian system administrator - ?for :). First we will examine the simplest possbile pattern:


aptitude search '?for x: ?conflict(?=x)'

This will list every package that is in conflict with itself. I personally don't see why such packages even exist, but as you can find out if you run this command, they really do! :)

Next, let's find out packages with cirrcular dependencies:


aptitude search '?for x: ?depends(?for y: ?depends(?=x))'

As you may have noticed, ?for search pattern doesn't have short form. If the command above is not clear on the first look, you can interpret it like this: show me every package x such that x is dependent on package y that is in turn also dependent on x.

Let's examine next situation from aptitude documentation. We'll list all packages that depend on package(s) that they recommend:


aptitude search '?for x: ?depends( ?for y: ?x:recommends( ?=y ) ) '

This is just a short intro in countless possibilities of aptitude. Of course, a deeper look into aptitude documentation is logical next step.

Process substitution

2009-05-13T19:22:00.009+02:00

In addition to widely used shell features like file globing and command substitution, modern shells like bash or zsh support something that is called process substitution. The goal of process substitution is to enable syntax which will allow us to get some commands marked within special characters behave as they were regular files. Common special characters for both mentioned shells are:

<(command list)
>(command list)

In addition to these, zsh adds additional syntax in form of:

=(command list)

Lets start with <() syntax. In it's simplest form, it can behave just as a regular pipe. For example, this is completely equivalent syntax:

ls -l | sort

sort <(ls -l)

What actually happens under the hood is that output of ls -l is connected to sort command through the mechanism of named pipes (FIFOs) or by using a file in /dev/fd

Off course, this was a simple usage pattern and nobody would use process substitution in this way instead of classic pipe scenario. But suppose a situation where you want a command to operate on results of more than one command. For example, you want to compare content of two directories (let's forget on dircmp command for a moment):

diff <(ls dir1) <(ls dir2)

Here the command acts just like you have saved output of both ls commands to separate files, and then ran diff on those two files. Or for example, you could use something like:

paste <(cut -f1 file1) <(cut -f2 file2)

Or maybe you want to edit the output of some complex command directly into your favorite text editor - no problem, just use syntax similar to this:

vi <(ls -l /etc)

Now when you know how to use <(), usage pattern for >() is straightforward. You can send the standard output of a command to more than one command. For example (just make sure that MULTIOS option is set in your shell before running this, ie setopt multios for zsh):

ls > >(sort) > >(sort -r)

If you are using zsh (and you should! :), you can also use =() snytax. The only difference between this syntax and <() or >() is that =() actually creates a file on filesystem instead of using FIFO. This comes handy if command you are substituting process to is using system calls like lseek(2).

Going mobile

2008-12-21T22:41:00.002+01:00

Until now I've always had a regular cell phone, i.e. not a smart phone or any similar kind of "smart" embedded device. That was fine with me as I've always used a cell phone only for making calls and sending text messages. After trying several vendors and models, I've finally decided to try something new and buy a smart-phone. As I was a starter in that area, I decided not to buy expensive gadget like Nokia N96 (although I was considering N95 for quite a bit), so at the end I bought a Nokia E66. Minimal set of features I required at the start was including wireless and GPS, and as I was biased to the Symbian based model (spoiled with Nokia's regular cell phones) I decided for E66.

Wireless is an enormous gain on a cell phone, and you won't know what you're missing once you don't start using it. Now I don't even bother turning my desktop or laptop PC on just to check my email (or Facebook :), I can do it with my cell phone. Besides that, there are many good web sites optimized for small screens found on cell phones. Be sure to try Wapedia (Wikipedia), BBC, Slashdot, Facebook and dozens of other sites optimized for a mobile web browser. It's not even a problem to find an open wireless access point, at least in the big cities, but if you live in a smaller city you'll most probably use GPRS/EDGE/HSDPA (which is btw extremely expensive here in Croatia). I assume you have a wireless connection at home, so this won't be a "show stopper" at all.

The next very useful thing to have on a cell phone is a GPS. Last month I was in London for ten days, and it saved me a lot of time to find locations I was interested in (and to get back on a familiar place once I got lost :). I suggest you to download maps for a location (country) you are traveling to on your PC at home (or directly via mobile browser and wireless), because you'll get quite a big bill to pay if you download it ad hoc via GPRS or EDGE (especially in roaming). Map of England for Nokia Maps was around 35 MB, so I assume you'll find similar sizes on all other GPS software. So bad that Google Maps (Symbian version) doesn't support such an option. Also, be sure to turn off a option to find GPS satellites over the Internet (AGPS), especially when in roaming, for a obvious reason. It will make your GPS device somewhat slower in finding satellites for the first time, but later you won't notice a difference (if you are using it regularly).

The latest great thing I "discovered", thanks to my colleague who had already been using it, was a great application called Nimbuzz. It can be used as chat program (MSN, Jabber, GTalk, ICQ...), and what's even better, it can be used for making Skype calls (or to some other SIP based applications). I've tested it and it works perfectly, now I can talk for hours without using my PC or wearing those annoying headphones with microphone. You can also take a picture or record a sound clip and it will automatically send a link to that content (uploaded to the Nimbuzz server) to the recipient over the selected instant messaging protocol. The best part of the story is that this application is totally free (as in "you don't need to pay money for it") and there are versions for Symbian, Windows Mobile etc. Give it a try, you want be disappointed.

There are far too many benefits to mention them all here (for example motion sensor, great Symbian applications etc), but the bottom line is that smart phone is really worth of money you give for it, if you need such functionality. Knowing myself, I won't by a regular phone again for sure.

Fixing fibre channel connectivity problems on AIX

2008-10-25T20:43:00.004+02:00

Unix and storage administration always brings interesting and new problems ready to be solved. Few days ago I've encountered one of those problems. Task was to configure two AIX powered VIO servers to access LUNs on CLARiiON storage, i.e. to make high availability environment with several AIX LPARs accessing storage through VIO servers.

After wiring the environment, i.e. hell of finding which port is right by using only lsslot and instructing technician located in data center 80 km far away, next step was to zone the SAN switches. But something was wrong. Every VIO had one dual port FC HBA and every port was connected to a different SAN switch. But instead of seeing two new WWNs (i.e F-ports) on switch, there were two L-ports.


SW1> switchShow
Area Port Media Speed State     Proto
=====================================
...
6   6   id    N4   Online           L-Port  1 public
7   7   id    N4   Online           L-Port  1 public
...

Situation on the other switch was OK - connected ports were reported as F-ports. For some reason one port of every HBA was L-port (loop port), and the other was F-port. After trying several things, I've managed to get things working as expected.

Apparently this is quite usual problem with AIX, so here is a cookbook how to fix it. You need to unconfigure FC devices and FC SCSI I/O controller protocol, and then set the link initialization protocol to Arbitrated Loop and Switch (AL).

First, look for FC devices:


# lsdev -H -Ccdriver -F "parent name description"
parent  name       description
fcs0    fcnet0     Fibre Channel Network Protocol Device
fcs1    fcnet1     Fibre Channel Network Protocol Device
fcs2    fcnet2     Fibre Channel Network Protocol Device
fcs3    fcnet3     Fibre Channel Network Protocol Device
fcs0    fscsi0     FC SCSI I/O Controller Protocol Device
fcs1    fscsi1     FC SCSI I/O Controller Protocol Device
fcs2    fscsi2     FC SCSI I/O Controller Protocol Device
fcs3    fscsi3     FC SCSI I/O Controller Protocol Device
    iscsi0     iSCSI Protocol Device
    powerpath0 PowerPath Control Device
sissas0 sas0       Controller SAS Protocol
sissas0 sata0      Controller SATA Protocol

From here we can see that FC HBAs are fcsX and FC SCSI I/O protocol devices are fcsciX, where X is in range from 0 to 3. Now, deconfigure devices and set the appropriated link initializaion mode. Don't forget to run cfgmgr at the end.


for i in 0 1 2 3
do
    rmdev -Rdl fscsi$i
    rmdev -Rl fcs$
    chdev -l fcs$i -a init_link=al
    cfgmgr -vl fcs$i
done

Now you should see both fabric logins and ports listed as F-ports:


SW1> switchShow
Area Port Media Speed State     Proto
=====================================
...
6   6   id    N4   Online           F-Port  10:00:00:00:c9:76:07:33
7   7   id    N4   Online           F-Port  10:00:00:00:c9:76:0c:2b
...

Linux I/O accounting

2008-09-06T17:31:00.003+02:00

One feature one would expect from an enterprise grade operating system is definitely a good I/O scheduler, but also some tools for monitoring it's performance. Starting from 2.6.x series, Linux has selectable I/O schedulers, actually four of them: no-op, anticipatory, deadline and complete fair queuing (CFQ) scheduler. Comparing to the older kernels, it was a great step ahead, but we were still lacking tools for I/O monitoring. Why was it so?

Up to the version 2.6.20, there was no I/O accounting support in the Linux kernel. Although process accounting was included in the kernel a long time ago, obtaining informations about per process I/O utilization was a very difficult task. Information you could get, for example with iostat command, was telling you that something is causing a heavy I/O load, but it was up to you to find out what process is actually causing it. Finding out that process was not an easy task, especially if you were running system with hundreds of processes, for example with ten Oracle instances.

Finally, Linux kernel 2.6.20 and newer included I/O accounting. Now every running process has associated I/O information in /proc/self/io. To enable this, besides having a recent Linux kernel, you must enable the following options in your kernel build configuration:


CONFIG_TASKSTATS=y
CONFIG_TASK_IO_ACCOUNTING=y

If you are running any recent Linux distribution, you probably already have this option, so there is no need for recompiling. Unfortunately, most "stable" Linux distributions you would see on production environments, like Debian or RHEL, use 2.6.18 at this moment, so you'll have a bit of work to do here.

Tool for per process I/O usage monitoring I use is iotop. It behaves much like widely used top or htop. i.e. it displays table of processes and threads on the system along with their I/O usage. Displayed data includes read and write I/O bandwidth in sampled period, and also percentage of time the thread spent while swapping in or waiting for I/O. iotop is actually a Python script, and if your favorite distribution doesn't already have it in packages, you can get it here.

I would strongly recommend trying iotop to every professional Linux administrator and even to beginning Linux users, because it helps a lot and gives you very usable information when doing a full system diagnostics.

(K)QEMU speed tests

2008-07-26T20:26:00.004+02:00

QEMU without additional accelerators is very slow, and that is a well known fact. If you have a modern processor with virtualization options you will most probably want to use KVM. You can find out if your system can run KVM by checking the flags line in /proc/cpuinfo for vmx (Intel) or svm (AMD).

$ grep -E '^flags.*(vmx|svm)' /proc/cpu

Unfortunately my CPU doesn't have such options. I have one of those Core2Duo processors that lacks suck features (T5500, commonly used on laptops). So if you're in situation like me, you're stuck with KQEMU to speed the things up. Up to yesterday, I was not fully aware what is the performance penalty of not using one of the accelerators. I've made some tests of my own to see how virtual machine behave with and without accelerator module.

Tests were run on Debian GNU Linux (lenny), kernel 2.6.25 and with QEMU version 0.9.1. Underlying hardware was IBM Thinkpad R60e, Core2Duo 2x1.66 GHz with 1 GB of RAM. For testing purposes I've used guest with Red Hat Enterprise Linux 5.1. Tests were focused on three areas I was most interested in:

Processing speed, calculate as many RSA public/private keys as possible in 10 sec (openssl speed rsa4096)
Network speed, download 10 MB file with wget (wget URL)
Disk speed (hdparm -tT)

First, each of tests was run on host OS (Debian), then in virtual machine without KQEMU and finally in virtual machine with KQEMU module loaded. The results are:

Calculation of RSA public keys (10 sec)
Host OS: 7202
KQEMU: 6188
QEMU: 372

Calculation of RSA private keys (10 sec)
Host OS: 98
KQEMU: 85
QEMU: 6

Buffered disk I/O
Host OS: 33.51
KQEMU: 24.51
QEMU: 25.47

Cached disk I/O
Host OS: 918.97
KQEMU: 442.71
QEMU: 445

Network I/O (download of 10 MB file)
Host OS: 52
KQEMU: 53
QEMU: 51

While virtual machine I/O performance was pretty much the same in both cases (slightly better without accelerator module), number crunching tasks like RSA key pairs generation is where KQEMU shows up. KQEMU accelerated guest achieves at most 90% of host speed, but without accelerator only about 10% which is very slow. Tests also show that disk performance on guests is around 50% of that on host.

Next thing I want to check is VirtualBox performance. I expect it to be slightly better of that of KQEMU. Unfortunately there is no Xen enabled kernel in Debian packages for lenny so I can test Xen performance.

SQLite Manager

2008-05-22T12:49:00.003+02:00

Playing with databases doesn't always require installation of full-featured RDBMS. You have probably already come to situation where you need only a small subset of those features. For example, you may only need ability to store your data in relational manner and to use SQL to extract or manipulate that data. Of course, natural choice for this is SQLite - small RDBMS with footprint of only 800 kb. SQLite comes very handy for small databases (e.g. personal collections) where you don't really care about transactions, isolations or other features (although SQLite is ACID compliant). Personally, I use it for my backup management application, in combination with Python. You probably use SQLite every day, even if you are completely unaware of that (for example in Skype, iPhone, Mac OS etc). If you are using Firefox, you are in fact using it right now.

SQLite is an awesome piece of software, but it would be good if you could have some tools to manage your data on the fly, i.e. without using some glue (e.g. programming language) to access and manipulate your data. Fortunately, there is such a software, and you are just few clicks away from using it. Having SQLite embedded in Firefox browser is a great thing. You only need to install a Firefox add-on named SQLite Manager. This add-on allows you to manage any SQLite database on your computer directly through intuitive GUI in Firefox. It also includes features like data import/export (csv and XML), data browsing, ability to create indexes and triggers, DML, DDL, SQL syntax help etc.

You can get this add-on Firefox add-ons page (it works on Firefox 2.0 - 3.0). You can start it by selecting SQLite Manager entry from Tools menu.

Python decorators demystified

2008-03-25T21:23:00.003+01:00

One of the things you may have noticed if you took a look at some of the newer Python programs is extensive use of decorators. Decorators are relatively new concept in Python world, and they first appeared in version 2.4. Decorators can help you to avoid unnecessary code duplication and to write elegant code in many situations which could otherwise lead to ugly code. Decorators are way to do metaprogramming in Python, ie. to enhance functionality of an existing (decorated) function without modifying it's body.

What precisely is a decorator? Let's first give a more or less formal definition in context of Python programming. Decorator is an object that acts as syntactic glue and is used to modify behavior of a method or a function. You can recognize a decorator by the following syntax:

@<decorator_name>
def <function_name>:
   <function_body>

What Python interpreter actually does under the hood is translation of previous code to:

<function_name> = <decorator_name>(<function_name>)

Here decorator acts simply as a wrapper for an existing function, but as we will show later, it can add a new functionality to that function.

Although previous syntax indicates that a function may have only one decorator, that is not true - a function may have more that one decorator. By using the multiple decorators, code is translated from this syntax:

@<decorator1>
@<decorator2>
def <function_name>:
   <function_body>

to this:

<function_name> = <decorator1>(<decorator2>(<function_name>))

In all of the examples we have assumed that decorators are already defined, but in your programs you will have to write a decorator before you can use it.

Let's start with the most basic example. We will modify a function that makes a sum of two integers so it will return a result in hexadecimal notation instead of default decimal notation.

def sum(a, b):
   return a + b

Instead of modifying the function itself, we write a decorator function. Decorator function takes a function as an argument and it returns a new function which will replace the old one.

def hex(fn):
 def new_fn(*args):
    return "%x" % fn(*args)
 return new_fn

In the body of decorator, we have defined a new function that reads all arguments into a tupple (*args syntax), and then uses a decorated function (fn) to calculate result. The result is then returned in hexadecimal. At the end of decorator body, new function is returned. Now, we can simply modify any function that returns a single integer with this decorator. Just add "@hex" in front of function definition, and it will do it's job.

>>> @hex
... def sum(a, b):
...     return a + b
...
>>> sum(10,5)
'f'

Decorators, like everything else in Python, are objects, so we can do more interesting things with them. For example we can pass arguments inside decorator to specify our output more precisely.

What are the possible real world situations where you might use decorators? Let's suppose you're developing a multithreaded application and you need an elegant way to mutually exclude the execution of some functions that run in separate threads. What you want to achieve is basically serialization of two or more functions, i.e. we must prevent starting of other functions when any other is already running. To accomplish this, you can use Python's built in reentrant lock object (threading.RLock) inside of decorator to prevent concurrent execution.and queue object (collections.Queue) to remember what functions (or transactions) must be executed after lock is released.

Good starting point to learn more about decorators is PEP 318.

irqbalance

2008-03-15T12:20:00.002+01:00

Distributing interrupt requests (IRQs) to multiple cores/processors in symmetric multiprocessing system (SMP) may be of great value in high-load situations. You may notice that your Linux SMP kernel acts somewhat strange regarding the interrupts if you look at file /proc/interrupts. Here is a situation on my laptop (dual core T5500):

           CPU0       CPU1     
 0:     686825          7   IO-APIC-edge      timer
 1:       4884          0   IO-APIC-edge      i8042
 8:          0          0   IO-APIC-edge      rtc
 9:       4451          0   IO-APIC-fasteoi   acpi
12:     221600          0   IO-APIC-edge      i8042
14:      50215          0   IO-APIC-edge      ide0
20:          1          0   IO-APIC-fasteoi   uhci_hcd:usb1, yenta, i915@pci:0000:00:02.0
21:     272763          4   IO-APIC-fasteoi   uhci_hcd:usb2, HDA Intel, ipw3945
22:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
23:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, ehci_hcd:usb5
218:          2          0   PCI-MSI-edge      eth0
219:      34668          0   PCI-MSI-edge      ahci
NMI:          0          0
LOC:      69271     348478
ERR:          0
MIS:          0

As you can see, all interrupts are being handled by only one core. To distribute IRQs evenly on all cores, we can use userspace daemon irqbalance. After I've installed irqbalance, 10 minutes later I've checked /proc/interrupts again, and here are the results:


           CPU0       CPU1      
  0:     956615          7   IO-APIC-edge      timer
  1:       7535       1560   IO-APIC-edge      i8042
  8:          0          0   IO-APIC-edge      rtc
  9:       5835          0   IO-APIC-fasteoi   acpi
 12:     294198       8756   IO-APIC-edge      i8042
 14:      68017          0   IO-APIC-edge      ide0
 20:          1          0   IO-APIC-fasteoi   uhci_hcd:usb1, yenta, i915@pci:0000:00:02.0
 21:     343158          4   IO-APIC-fasteoi   uhci_hcd:usb2, HDA Intel, ipw3945
 22:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 23:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, ehci_hcd:usb5
218:          2          0   PCI-MSI-edge      eth0
219:      38105       3852   PCI-MSI-edge      ahci
NMI:          0          0
LOC:      86112     469976
ERR:          0
MIS:          0

Obviously it works ;). Now just to find some methods to do some measurements... maybe to run openssl speed to get one core to 100% and then try to copy some large files...?

File sharing using Python

2008-02-28T19:38:00.004+01:00

Suppose this very common situation: you have two machines, A and B. You have some files on A and you want to transfer those files to B. Let's assume you don't have ftp server, samba, nfs or anything similar which could make this operation trivial. If talking about home/desktop computers, this is a most common situation.

Although it's possible to transfer files by using scp/sftp, netcat or something else, today I saw a very cool trick which comes handy in such situations. All you need is Python.

First, change your current working directory to the directory that holds files you want to share. Then, just run this command:

python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"

This will run web server or port 8000 which you can use to browse and download files, on a similar way like using file://directory in your favorite browser or viewing web pages without index.html. When you are done, just press CTRL+C.

Python completion

2008-02-26T21:21:00.002+01:00

If you want completion in Python console, just execute these two lines:

import readline, rlcompleter
readline.parse_and_bind("tab: complete")

Now completion works just like in bash. You could also set any other key than tab, but most will want exactly this.

If you want this to run automatically every time you run the python console, add those two lines to any file, for example into .pythonrc in your $HOME, and then set environment variable PYTHONSTARTUP in shell startup scripts (~/.bashrc, ~/.zshrc etc) to point to that file:

export PYTHONSTARTUP=$HOME/.pythonrc"

Linux software that sux

2008-02-24T10:49:00.004+01:00

1. Evolution

If you have used it long enough, chances that you haven't seen "xyz does not match even after sync" message is minimal. It's big, bloated and slow application full of bugs. Sadly, there is no graphical MUA on Linux that satisfies me. OK, Sylpheed is close enough, but some of it's interface and functionality is awkward and broken.

2. OpenOffice

Good for writing small documents, 50 pages at most. I wrote my diploma thesis in OOo and it was a painful experience. The funniest thing was when I tried to turn the changes tracking on document with 70 pages. After I have made some changes, I noticed that disk I/O indicator on my laptop is constantly turned on. A minute later, the only thing I could have done was to press the power switch. Apparently, OOo ate all of my memory (1 GB of physical RAM + 1 GB of swap), just to track few changes. I usually don't have praises for Microsoft's products, but Microsoft Office is miles in front of OOo, and it will probably stay so for a long time.

3. Dia

Related to previous rant on OOo, there is Dia. There are simply no words to describe the difference between Dia and Visio, Visio is better for several orders of magnitude. Dia's functionality is broken and it's symbols are just ugly. Many common symbols can't even be found in Dia.

4. KDE

I really don't see what is so cool with KDE. It's slow, most of it's applications are pointless, uses too many libraries so when I use apt-get to update the system I always need some new KDE libraries, it looks too baroque, uses stupid K* names for applications, imitates features from other GUIs instead of focusing on developing new ones etc. There are dozens more reasons not to use KDE. I stick to lesser evil (GNOME) simply because I'm too lazy and don't have time to configure fvwm. Hopefully I will do that one day and get rid of KDE/GNOME junk forever.

5. Rhythmbox, Amarok, etc.

What's the point of having a music player which uses RDBMS?!? I have 20 GB of mp3 songs and I still don't have need to use a DBMS to organize my music or find a specific song! For that purpose I systematically save my music using the filesystem hierarchy, and that's enough!. Both Amarok and Rhythmbox use idiotic interface, use fat too much memory and CPU and try to be smarter than it's user (automatic album cover downloading, song rating... come on!). I'm fan of Winamp/XMMS style of programs, but unfortunately it seems that XMMS has stuck in development (still using ugly GTK 1.x). Recently I've discovered audacious, the XMMS fork - it looks nice and has nice plugins. I'll give it a chance, it looks promising.

IBM ACPI

2008-01-10T18:46:00.000+01:00

Nice thing with IBM Thinkpad (now Lenovo) laptops is ibm-acpi linux kernel module. You can access many sensors and laptop properties by simply reading and writing files in /proc/acpi/ibm. For example, you can turn the thinklight on the top of the laptop on/off with:

echo on > /proc/acpi/ibm/light
echo off > /proc/acpi/ibm/light

For commands that specific file accepts, just display it with cat. Very interesting file is thermal, which displays temperatures read from sensors scattered all over your laptop. You can also see/set fan speed with file fan. Here is a perl script I wrote for quickly displaying laptop temperatures and fan speed. Please note that number of temperatures displayed in thermal file changed in linux kernel 2.6.18, so before running this script make sure you have the newest kernel and loaded ibm-acpi module.


#!/usr/bin/perl
# (C) 2007 Igor Pozgaj

use strict;
use warnings;

open (IF, "</proc/acpi/ibm/thermal") or die "No ibm-acpi module found";
close(IF);
my %t = (
   "CPU"  => $values[1],   # CPU
   "PCI"  => $values[2],   # Mini-PCI
   "HDD"  => $values[3],   # Internal hard disk
   "BAT1" => $values[5],   # main battery, primary sensor (under F5)
   "BAT2" => $values[7],   # main battery, secondary sensor (under Home)
   "MCH"  => $values[9],   # northbridge, to DRAM bus (above track point)
   "ICH"  => $values[10],  # southbridge (under touchpad)
   "PWR"  => $values[11],  # power regulator (under F2 key)
);


open(IF, "</proc/acpi/ibm/fan") or die "No ibm-acpi module found";
my $fan = (split(/:\s+/, $line[0]))[1];
close(IF);

write;

format STDOUT_TOP=
                         Thermal data for IBM

 CPU     PCI     HDD    BAT1    BAT2     MCH     ICH     PWR      FAN
--------------------------------------------------------------------------
.

format STDOUT=
@## °C  @## °C  @## °C  @## °C  @## °C  @## °C  @## °C  @## °C  @#### rpm
$t{"CPU"}, $t{"PCI"}, $t{"HDD"}, $t{"BAT1"}, $t{"BAT2"}, $t{"MCH"}, $t{"ICH"}, $t{"PWR"}, $fan
.

Queuing in PostgreSQL

2008-01-08T19:41:00.000+01:00

PostgreSQL is a great DBMS and I use it almost exclusively for all of my projects where I need database. But, let's face the facts, it still lacks many features to compete with big players in enterprise scale applications area, such as Oracle and DB2. On my current job, my role is Unix administration and database administration, so I'm more or less familiar with Oracle which I use the most. There are far too many features in Oracle that PostgreSQL doesn't have. Notable examples are RMAN (backup and restore tool), ASM (automatic storage management), AWR (automatic workload repository), EM (enterprise manager) etc.

But let's return to the theme noted in article title. As part of my diploma thesis I'm developing application which needs some sort of queuing in database. I have a table "events" and I would like to have another table which would serve as a queue for application (if you are curious, it's event table from Snort database). Queue table should be populated at the same time as events table, but it should return rows in FIFO (First In First Out) principle and remove rows immediately after they have been fetched. Oracle has such functionality by default, it's AQ (Advanced Queuing) package, but in PostgreSQL you must implement it by yourself. As this is very common feature needed by developers, it would be very nice to see it in next major release of PostgreSQL.

Obvious access is to use triggers with an additional table. So, we have two tables, event and event_queue. We define queue table like this:

CREATE TABLE event_queue
(
   id        SERIAL PRIMARY KEY,
   signature INTEGER NOT NULL
);

Now, we must insert data into this table every time when base table (event) gets new rows. We use trigger and trigger function:

CREATE FUNCTION put_event() RETURNS TRIGGER AS $$
BEGIN
   INSERT INTO event_queue VALUES (NEW.signature);
   RETURN NULL;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER insert_event AFTER INSERT ON event
   FOR EACH ROW EXECUTE PROCEDURE put_event();

Now, there are two solutions for reading from queue. You could either write select by yourself (with ORDER BY id and LIMIT 1) and write another trigger which will delete row after it has been read, or you could write specialized function like this:

CREATE FUNCTION get_event() RETURNS INTEGER AS $$
DECLARE
   v_sig INTEGER;
BEGIN
   SELECT sig INTO v_sig FROM event_queue ORDER BY id LIMIT 1;
   DELETE FROM event_queue WHERE sig = v_sig;
   RETURN v_sig;
END;
$$ LANGUAGE plpgsql;

10 cool vim tips

2007-12-27T21:42:00.000+01:00

1. Search for a word under the cursor

Very usefull command, but most of the people use "/" to search and then search next with "n". It's enough just to position to a word and use "*" to search forward and "#" to search backward.

2. ROT13 whole file

Type "ggg?G"

3. Increment/decrement number under the cursor

Use the CTRL-a and CTRL-x key combinations.

4. Remote file editing

You can start vim like this: vim scp://username@host//path/to/file

Beware, paths are absolute!

5. Open file under the cursor

Type "gf". This will open the file with filename as word under the cursor.

6. Upercase/lowercase whole line

Use "ggU" for uppercase and "ggu" for lowercase conversion.

7. Macros

Type "qx" where "x" is an arbitrary letter (register). Use editor as you usually do and then type "q" again to finish recording. You can call your recorded macro with "@x", or you can repeat last called macro with "@@".

8. ASCII code of character under the cursor

Type "ga"

9. File specific settings

It's possible to give vim command inside of comments in a file. For examples, to set tab stop to four characters in a C++ source file, you could enter // vim: set ts=4. Next time you open the file, tab stop will be four characters no matter on contents of local or global vimrc.

10. Paste output of a command

Most of the people use :r!command, but it is much easier to use !!command. For example, !!ls

VMware 2.0

2007-12-04T20:15:00.000+01:00

I've completely abandoned VMware as a virtualization solution on my PC and switched to QEMU. QEMU has much more features than VMware, it can virtualize many more platforms than only IA32 and IA64, has better networking options, user mode networking, port forwarding, can have console output on host OS tty, runs pretty fast with kqemu module loaded into kernel etc. There is also nice GUI for QEMU, called qemu-launcher.

Two weeks ago VMware 2.0 beta was released and I was really disappointed. Good old VMware console is apparently completely abandoned, and instead of it you are forced to use completely idiotic and slow web based administration console. Just to mention, the interface uses Apache and Tomcat, and it takes much of RAM that could be used better by guests. OK, it now supports up to 8 GB of RAM per guest OS, it has USB 2.0 support, full 64 bit support, you can run Vista as guest OS :), but for average home user, it's too bloated and slow. With other virtualization products such as QEMU, Xen and OpenVZ, I won't miss VMware.

Disk Archive (dar)

2007-10-10T20:56:00.000+02:00

The best GNU program I've seen in the last month, maybe even in the last year is definitely dar. I found out for this masterpiece on Dinko's blog, and after spending an hour on playing with it I was really impressed.

Dar maybe lacks fancy GUI, database repository or live boot image like popular bacula or mondo, but who wants such applications for everyday home use? Dar is simple but powerful CLI tool for backup. It doesn't use dozen of other tools and libraries, apt-get said I need only two packages - the dar itself and one additional library. No database is needed, but if you need some kind of backup repository dar can make one in a single (binary) file which is managed by program dar_manager. Possibilities of this tool are enormous, just check the man page or features list!

I use dar for regular full and incremental backups of /home and /etc directories. Following are the commands I use for backup and saving the information to dar repository.

Full backup of /home:


dar -c /opt/backups/archives/home-full-`date +'%d%m%Y'` -z -R /home/ipozgaj -D -P .liferea/cache -P .mozilla/firefox/h48ywliv.default/Cache -P .pan2/article-cache -P .azureus/Documents

Incremental backup of /home:


dar -A /opt/backups/archives/home-full-07102007.1.dar -c /opt/backups/archives/home-diff-`date +'%d%m%Y'` -z -R /home/ipozgaj -D -P .liferea/cache -P .mozilla/firefox/h48ywliv.default/Cache -P .pan2/article-cache -P .azureus/Documents

If you're planning to use repository, you can initialize it like this:


dar_manager -C /opt/backups/repository

Adding archive to repository is simple:


dar_manager -A /opt/archives/home-full-07102007.1.dar -B /opt/backups/repository

You can list the contents of repository with:


dar_manager -l -B /opt/backups/repository

or you can start the interactive session with:


dar_manager -B /opt/backups/repository -i

Oracle XE

2007-09-04T18:57:00.000+02:00

This month I'm attending Oracle seminars (preparatory DBA I and DBA II courses for OCA and OCP certificates, and Oracle Application Server Administration Workshop I and II) so I decided to try Oracle XE at home. I was familiar that installing Oracle can be real pain (I had some bad experience with installing Oracle 9i on SLES :), so I was prepared for a long battle :). Only 30 minutes later, including the time for installing the virtual machine, I had fully functional Oracle instance. Installing Oracle XE on Linux is really a simple task. Here we go:

Host machine:
Intel Core 2 Duo 2x1.67 GHz, 2 MB cache
1 GB of physical memory
Debian GNU/Linux (lenny)

Virtual machine (Vmware):
512 MB of physical memory
4 GB virtual disk (1 GB for swap)
Fedora Core 6 (Zod), minimalistic install

You should download oracle-xe-universal rpm package from here. Before proceeding with installing, you must install libaio, that's only lib Oracle depends on. Also, you must have at least 700MB of swap and around 1.5GB of disk space to successfully end the installation.

$ rpm -ivh oracle-xe-univ-10.2.0.1-1.0.i386.rpm

After that, you can proceed with instalation

$  rpm -ivh oracle-xe-univ-10.2.0.1-1.0.i386.rpm

System is automatically configured to run Oracle instance. One thing you must not forget is to run configure script as root:

# /etc/init.d/oracle-xe configure

Just accept default values and remember that web interface can be accessed on http://hostaddr:8080/apex, and remember what you have entered for system password. Now it's time to configure environment. Installation creates "oracle" user with home directory in /usr/lib/oracle/xe. If you are familiar with Oracle, first thing you will probably do is to make this directory look more like following OFA guidelines:

# ln -s /usr/lib/oracle/xe /u01

Then add the following in your .bashrc

export ORACLE_SID=xe
export ORACLE_BASE=/u01
export ORACLE_HOME=$ORACLE_BASE/app/oracle/product/10.2.0/server
export TNS_ADMIN=$ORACLE_HOME/network/admin
export $PATH=$PATH:$ORACLE_HOME/bin

Now logout and login as oracle user and verify that you can connect to instance:

$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 4 21:26:56 2007

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to an idle instance.
SQL>

and try to run web interface. It should look like this:

Glade-3

2007-08-21T09:33:00.001+02:00

As Debian packagers are quite careful (some would say too careful) when adopting new versions of programs, I had a chance to work with new Glade just a few weeks ago. I must say that I'm not too impressed. Interface is even worse than in Glade-2. Single window interface is somewhat awkward when you're working on low resolution (1024x768) and with forms in size over 640x480. Previous versions were better as working form was floating over other windows, so you always had a view on whole form. Other thing that annoys me is that working form has no window decorations on it (i.e. there is no window frame, minimize, maximize and close buttons etc). You can't even manually resize working form - you must resize it by entering it's size in pixels in object inspector.

Few new top level dialogs like "recent chooser" and "assistant", and new widgets are all I see better than in previous versions. Also I like that palette icons can now be reduced to text only.

Altogether, there is no significant improvement in the new version of Glade. As I read on Senko's blog and his Guadec report, libglade is to be abandoned and GtkBuilder comes to scene. I don't know if there will be some new tools for GUI design or GtkBuilder support will be added to Glade-3, but there is much space for improvement when you take a look at the current state of Glade-3. And hopefully, Debian packagers will introduce it in lenny very soon so I can test it :-).

Playing with chattr

2007-05-27T22:01:00.000+02:00

One thing I've noticed among my fellow Linux colleagues is that although most of them frequently use chmod to set file access permissions, only a few of them use chattr to set up advanced file and directory parameters. chattr is a powerful command, used to set up additional parameters to file system objects in order to optimize file access speed, security, disk space utilization etc. Here is only a brief description of how you can use some of it's options to fine tune your file system.

1. Hashed trees indexing

You've been using this option for sure, but you probably didn't know. You can't set it by yourself, but system applies this option automatically for every directory which contains 255 items or more. What does it do? By default, items in directory are indexed by using trees, but this technique becomes inefficient when directory contains large number of files. So, when you create a 255th item in directory, the 'I' option is applied which instructs the kernel to index directory by using the hashed trees which speeds the things up. Example:


% mkdir test
% lsattr -d test
------------------ test
% for i in `seq 1 254`; do touch test/file$i; done
% lsattr -d test
------------------ test
% touch test/file255
% lsattr -d test
--------------I--- test

2. Immutable files

You can use this option to lock files for deleting, editing, appending, renaming or being linked. You can even protect your files so root can't delete them, for example if you hold some of your files in /tmp because of quotas on your home directory. You can also use it to lock files for which you are sure that will never change, for example some files from the /etc directory. Using this you can harden your system and protect it from malicious processes that want to modify system files. Note that all this doesn't really prevents root (more precisely, any user possessing the CAP_LINUX_IMMUTABLE option) from removing the 'i' option from file system object and then modifying/deleting it, but it will protect you from evil rm -rf commands and it will come handy when you want to prevent creation of new files in a directory. Example:


# whoami
root
# id -gn
root
# touch test
# chattr +i test
# rm -f test
rm: cannot remove `test': Operation not permitted
# chattr -i test
# rm test
rm: remove regular empty file `test'?
# rm -f test


# mkdir test
# chattr +i test
# touch test/file
touch: cannot touch `test/file': Permission denied

3. File compression

When your disk space is low, it's time for compression or deleting. Instead of manually compressing and decompressing your files, you can instruct kernel to do it for use. Compressing and uncompressing is fully transparent, you can work with your files as they are not compressed at all. Just apply the 'c' option to file or directory and watch your free space. Note that this is not yet implemented in ext2 and ext file systems.

You can find more details and other interesting options in chattr(1) and lsattr(1).

Speed up your build with multi core CPUs

2007-05-13T23:19:00.000+02:00

Multi core CPUs are becomming de facto standard today, but unfortunately most people don't know how to take advantage of multiple cores on even the most basic everyday operations. For example, dual core can speed up your code build process a lot if you pass the -j option to make, followed by the number of parallel jobs to run. I tested this option on a larger project that usually compiles in 5 minutes on my dual core T5500. I got best results with -j 4, that is four build jobs in parallel. Code now compiles in less than 2 minutes. Ofcourse, if your makefile is written in a way that no sources can be built in parallel, you won't see any improvement by using -j option, but this is less likey on any larger project.

MS Office COM components

2007-04-01T21:35:00.001+02:00

This article may help you if you get into the same problems I had this week... I had to develop a .NET program which will load some Microsoft Word Documents, change some text in that documents, and then save them into PDF. As Word 2007 has "Save As PDF" option, the obvious way to do this was to use "Microsoft Word 12" COM component (for some reason, MS Office .NET components are less powerful than COM components, and everyone suggests using the COM).

Then, the problems began... When I tried to run the program at home, all worked perfectly. But, when I tried the same on machine on work, program mysteriously crashed with generic COMException or SOHException when I tried to run SearchAndReplace command. After spending few hours on Google trying to find what's wrong, I finally found out what's happening.

This problem appears when you have multiple versions of Microsoft Office on your computer, or when you had an older version and then installed a new one. Some objects from Word 2007 have same name as objects from, for example, Excel 2003. So, solution for this problem is: you may either reregister components you use when you start the program (not recomended), or you can replace the following code:


public void SearchReplace(string originalText, string replacementText)
{
  object replaceAll = Word.WdReplace.wdReplaceAll;

  WordApplication.Selection.Find.ClearFormatting();
  WordApplication.Selection.Find.Text = originalText;

  WordApplication.Selection.Find.Replacement.ClearFormatting();
  WordApplication.Selection.Find.Replacement.Text = replacementText;

  WordApplication.Selection.Find.Execute(ref missing, ref missing,
      ref missing, ref missing, ref missing, ref missing, ref missing,
      ref missing, ref missing, ref missing, ref replaceAll, ref missing,
      ref missing, ref missing, ref missing);
}

with a variant that uses the late binding technique:


public void SearchAndReplace(string originalText, string replacementText)
{
  object srcText = originalText;
  object dstText = replacementText;
  object[] parameters = new object[15];
  object findObject = WordApplication.Selection.Find;
   
  parameters[0] = srcText;
  parameters[1] = missing;
  parameters[2] = missing;
  parameters[3] = missing;
  parameters[4] = missing;
  parameters[5] = missing;
  parameters[6] = missing;
  parameters[7] = missing;
  parameters[8] = missing;
  parameters[9] = dstText;
  parameters[10] = Word.WdReplace.wdReplaceAll;
  parameters[11] = missing;
  parameters[12] = missing;
  parameters[13] = missing;
  parameters[14] = missing;

  findObject.GetType().InvokeMember("Execute", BindingFlags.InvokeMethod,
      null, findObject, parameters);
}

Virtual PC 2007 sucks

2007-03-01T23:51:00.000+01:00

I've been playing with Microsoft Virtual PC 2oo7 since it was first published, and today I finally decided to get rid of that crap. Compared to other virtualization software that I tried, like VMware or XEN, Virtual PC is just a toy, not suitable for anything more than the simplest tasks. Virtual PC on production servers - oh give me a break :). Although it didn't crash a single time, the performance was simply awful. All that culminated today. I only needed a simple Ubuntu installation so I could test SELinux before I could deploy it on real server, and that simple task became a nightmare. Installation took me more than an hour (lately I did exactly the same installation on VMware - for only 15 minutes!). System was operating very slow, and every two or three minutes it was totally blocked for few seconds. High I/O systems are area where Virtual PC fails on it's knees - don't even bother trying this.

After I've lost my nerves with Virtual PC (dang, it's enough that I must do this on Windows instead of Linux, and now this!) , I returned to good old VMWare, where everything works as one would expect. It's hard to believe that a virtualization software can run so slowly on a brand new hardware (Core2Duo 2x1.7 GHz, 1GB DDR2 etc). When you look at Microsoft's latest products like Vista or Virtual PC 2007, you suddenly start to appreciate the fact that there are good alternatives to that crippleware - software that turns your new hardware to useless heap of plastic and metal. Avoid it if you can, it saves you money, and what is more important, it saves your nerves - priceless.

IBM - works as expected

2007-02-03T23:29:00.000+01:00

After a long time I've finally decided to replace my old computer with a new one. My old computer was outdated a long time ago (PII @ 366 MHz), but it is a quite sentimental value for me as it is the first computer on which I ran Linux and learned most things I know today :).

This time I made up my mind for a notebook. All I wanted was a robust machine for work, not a top model with ultra expensive video card fingerprint reader or other fancy stuff like that :). As Linux is my primary OS, I decided that it would be the best to get a HP or IBM model, as they have the best "Linux support". Ultimately, I decided for IBM as it has a reputation of being unbreakable, most of hardware has an open source driver, and I have previously worked on some IBM models so I knew what I could expect.

The final choice was IBM ThinkPad R60e with Intel Core 2 Duo T5500 1.66 GHz 2MB cache, 1 GB DDR2, 100GB SATA, etc. Most of things worked automatically on Debian Etch, only a minor tweaking was required. Things I needed to configure manually were sound card (Intel HDA), and wireless card (Intel 3945). One more thing that didn't work, and which took me the most time after the installation was a video car (Intel GMA 950). I had to manually edit xorg.conf to use i810 driver and to turn on composite extensions with:

Section "Extensions"
Option "Composite" "Enable"
EndSection

Then, direct rendering didn't work. Glxinfo said:

# glxinfo | grep direct
direct rendering: No

After that I installed package libgl1-mesa-dri, and everything worked fine. Strange, it haven't been installed automatically, and deborphan still says that package is not being used by any other.

I still haven't played with this machine's native virtualization support, so I guess this will be the next thing I'll be working on. So far, I'm very pleased with this notebook - at least I didn't have problems like Ante :).

Pydoc, gdb...

2006-12-19T12:32:00.000+01:00

It's really amazing how you can use something for ages and yet miss some of the basic things. That happened to me today with pydoc. Although I've been programming in Python for 4+ years I've never used pydoc. Today I needed a fast way to generate documentation from several files where I had a lot of functions and classes documented by docstrings. I was familiar with pydoc (I knew for that), but what I didn't know was pydoc -p. You run:

pydoc -p 1234

in terminal which runs the web server and then open URL http://localhost/1234 in your favorite web browser. Fancy!

This is not the first time I miss something crucial. For example, I've been using gdb for 8 years, but it took me almost 5 years to find out for TUI. Run

gdb -tui

and see if you're like me :).

Graphviz

2006-12-12T11:39:00.000+01:00

Graphviz is open source graph visualization software. It can display simple graphs, digraphs, finite state machines (take a look on some examples), write output to jpg, pdf, postscript etc. Program takes descriptions of graphs in a simple manner of text files, but wrapper libraries for several languages are also available. I tried only a Python interface, pydot, and it's amazing. This comes handy to me as I've been developing a Python graph library for a long time, and now I finally have a simple visualization tool.

Changes, changes...

2006-11-14T12:34:00.000+01:00

First of all, sorry for RSS problems during the last few days. As I've switched to the new blogger (beta), I was given an opportunity to give labels to the old posts, which I did and hence the problems.

Mono now supports WinForms and .NET 2.0. Who knows what will happen on Linux desktop now - maybe we'll use dozens of good (GUI) programs from Windows that have no good alternative on Linux. As Novell and Microsoft continue their cooperation I suspect that in the near future we can expect Microsoft's products in all major Linux distributions.

I've heard a rumor that Vista would be Microsoft's last major OS upgrade, and that they will base their complete business on SOA, web services and business applications. Many interesting predictions come to my mind immediately. If these rumors are true, if Microsoft and Novell would continue their cooperation, if Novell would continue it's active support of Mono project, then in future we can expect many great (wierd?) things. As virtualization techniques get better and better, I suspect that sooner or later we will see an OS with Linux kernel, which uses virtualization (Xen, Vmware, Virtual PC...) and virtual machines (Mono) to run Microsoft's applications.

Java is now also an open source product, released under the terms of the GPL. Only time will show how will this reflect to future of Java. My personal opinion is that Java is heavily bloated and broken, so I don't see how this will help to fix that . There is no point in fixing totally screwed things :).

BTW, take a look at the personalized google homepage. You can make tabs, customize your search results or add various widgets (weather forecast, clock, TODO list, calendar, inbox, news, sudoku :) and many other good stuff.

MIT OpenCourseWare

2006-11-05T22:10:00.000+01:00

Have you ever dreamed of being an MIT student? Well, now is your chance! :)
If you visit this link, you will get a full access to undergraduate and graduate courses on MIT. What you will get there are all complete lecture notes, exams, homeworks and other materials, and what you won't get are the certificates and degrees.

PostgreSQL? Or maybe MySQL?

2006-11-05T15:20:00.000+01:00

About three years ago when I had to do my first bigger job regarding the databases I faced a problem of selecting the proper RDBMS. At that time, MySQL was miles far from features that PostgreSQL had, so PostgreSQL was a logical choice. At that time, MySQL didn't have even most fundamental things one would expect from a serious DBMS, like subqueries, triggers, views, stored procedures, cursors, foreign keys, unions, full joins, constraints, transactions (excluding the InnoDB) etc. Only notable advantage of MySQL over PostgreSQL was it's speed.

When you compare that with today's situation (MySQL version 5.1, PostgreSQL version 8.2), much has improved. MySQL now has nearly all the features it lacked and it can be used for making real databases, not only some simple storage systems in some backend web applications. If you want to see how these two open source applications look compared to a big enterprise scale player like Oracle, you can see a detailed analysis here.

Now it seems that MySQL developers are implementing things much more faster that PostgreSQL guys. There is no big difference between these two systems anymore. Next time when I'll need a DBMS, I'll most certainly give a chance to MySQL.

C# 3.0 preview

2006-10-15T15:31:00.000+02:00

Two days ago I was at presentation of new features in C# 3.0. Well, I'm not impressed at all. Most of "new" features are "borrowed" from other languages, most notably from Python. Here is a short list of new stuff:

var keyword
anonymous types
object initializers
extension methods
lambda expressions
expression trees
query expressions

1. var keyword

This keyword is used to declare a variable whose type is inferred from initializing expression. For example, this is the old way:

int i = 1

and this is the new way:


var i = 1

Obviously there are many requests to make C# look more like weakly typed language, but it nevertheless stays strongly typed, so this new syntax can lead to ridiculous and stupid syntax like:

var array = new[]{1, 2, 3, 4}

2. anonymous types and object initializers

Nothing that we haven't seen before. You can now execute code like this:

var point = new {x = 1, y = 2}

Are you familiar with this syntax from somewhere? Maybe from Python's dictionaries with named parameters? :)

3. lambda expressions

In Croatia there is a proverb "Vidjela žaba da se konj potkiva pa i ona digla nogu", that in free translation means "A frog saw that a horse is beeing calked, so she also lifted her leg". Enough said :). Syntax is also an excellent example of borrowing syntax from other languages which resulted with this silly syntax:


func = () => Console.WriteLine("Something stinks from Redmond")

4. Expression trees

This is a new type that allows expressions to be treated as data at runtime. System.Expressions.Expression is simply an in-memory representation of a lambda expression.

5. Query methods

This is IMHO the only significant improvement in this version of C#. That is a new feature called LINQ (Language Integrated Query) which syntax is somewhat similar to SQL syntax. For example:


public class Point { public int x, y;}

List<points> points = ...

var result = from p in points where p.x == 1 select (p.x, p.y)

The simplest wiki

2006-09-27T09:17:00.000+02:00

I've found a link to tiddlywiki on Senko's homepage and after I've tried it I must admit: it's awesome! Unlike other wiki systems, this one doesn't require any server side technology like PHP and MySQL. Tiddlywiki runs completely on client's computer - it includes HTML, CSS and JavaScript in only one file, so you can edit it offline. The only negative aspect of this is that empty template is around 180 kb (not so large when you look at all the features available), and all content goes in that file which can grow rather large over the time. A good starting point for using the tiddlywiki is this tutorial.

Optical illusions

2006-09-19T16:33:00.000+02:00

Check out this page with 66 various optical illusions.

POSIX threads and manual pages rant

2006-08-16T10:59:00.000+02:00

Sometimes it's very difficult to find out what's wrong with your code using only man pages. For example, look at this demonstrative piece of code:

#include <pthread.h>

pthread_mutex_t mutex;

void *thread_func(void *param)
{
pthread_mutex_lock(&mutex);
pthread_mutex_lock(&mutex);

return NULL;
}

int main()
{
pthread_t th;
pthread_mutexattr_t attr;

pthread_mutexattr_init(&attr);
pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK);
pthread_mutex_init(&mutex, &amp;amp;amp;attr);
pthread_mutexattr_destroy(&attr);

pthread_create(&th, NULL, thread_func, NULL);
pthread_join(th, NULL);

return 0;
}

Error checking type of mutex is used to prevent deadlock (I just wanted to see if it works :). But when you try to compile the code you will probably get something like this:

% gcc -g -W -Wall -lpthread foo.c                  
foo.c:5: warning: unused parameter 'param'
foo.c: In function 'main':
foo.c:19: warning: implicit declaration of function 'pthread_mutexattr_settype'
foo.c:19: error: 'PTHREAD_MUTEX_ERRORCHECK' undeclared (first use in this function)
foo.c:19: error: (Each undeclared identifier is reported only once
foo.c:19: error: for each function it appears in.)

After spending almost an hour on reading manual pages for POSIX threads, I was confused even more! This code should have worked! Then, I found an old Usenet post with solution for this problem. It seems that function pthread_mutexattr_settype is relatively new addition to the Single Unix specification, so you need to compile the code with -D_XOPEN_SOURCE=500 which enables compatibility with UNIX98, or with -D_GNU_SOURCE which enables all extensions. Now everything worked as one would expect:

% gcc -D_XOPEN_SOURCE=500 -g -W -Wall -lpthread foo.c
foo.c:5: warning: unused parameter 'param'
% gcc -D_GNU_SOURCE -g -W -Wall -lpthread foo.c  
foo.c:5: warning: unused parameter 'param'

Where can I find that information? Man pages don't even say that it's UNIX98 extension!

Dictionaries again

2006-08-08T20:39:00.000+02:00

Last time I wrote about two favelets for translating from Croatian to English and vice versa. A friend of mine went a step further and wrote a Firefox extension which basically makes the same, but now you can call the Eudict dictionary from a popup menu. You can download the extension here

BTW, DeviantArt has released a new design, but most likely you won't see it today (heavy load).

Favelets

2006-08-07T11:51:00.000+02:00

Most of you probably have large number of bookmarks in your favourite web browser. Probably 99.9% of those bookmarks are simply links to some web pages. Most of people are not aware that bookmarks can hold much more than just links.

What is favelet? Favelet is JavaScript code prefixed with javascript: and saved as bookmark. What can it be used for? Well, suppose you are web designer, so you need to see how some page looks at 800x600. If you add this code to your bookmarks, you can view how it looks with only one click:

javascript:resizeTo(800, 600)

Today I made my life much simplier with two cool favelets :). I'm native croatian speaker, and I often need to look at online dictionary (I use eudict) for some english word. With this two favelts I automated that process to simply selecting the word and clicking on wanted favelet (I broke lines, but this must be a single line):

From English to Croatian:

javascript:x=escape(getSelection());if(!x)%7Bvoid(x=location.href)%7D;
location.href='http://www.eudict.com/?lange=engcro&word='+x+'';

From Croatian to English:

javascript:x=escape(getSelection());if(!x)%7Bvoid(x=location.href)%7D;
location.href='http://www.eudict.com/?lange=croeng&word='+x+'';

Eudict supports much more languages so you can customize favelets as you want.

Digital watermarking

2006-08-05T21:07:00.000+02:00

As I mentioned before in one of my previous posts, lately I'm playing a lot with GIMP. It's amazing what can be done with that piece of software. Today I came to a wild idea to add a digital watermark to some of my best works. Most notable example of watermarking system is that of Digimarc that may be found in Adobe's Photoshop. After spending almost an hour on searching for an open source watermarking system, I finally gave up. GIMP, ImageMagick and all other open source tools for manipulating images lack such a feature. Well, ImageMagick actually has some kind of watermarking system, but it only handles a visible watermark, not a true digital one.

Digital watermarking is a technique somewhat similar to steganography, but again much different. Steganography is basically technique of hiding confidential data in carrier. Carrier may be any kind of binary data, but usually it's audio or image file. Digital watermarking also adds additional data to carrier, but it differs from steganography in several points:

in steganography, carrier is irrelevant, only hidden data is important
in digital watermarking, both the carrier and hidden data are important
in digital watermarking, hidden information is used to authenticate the owner of carrier
digital watermarks are immune to cuts

Last point is especially important for watermarking. If you watermark a picture, and someone selects and uses only a part of it, you can still prove that part of image is yours because watermark can be found in every part of the picture. That is the main reason why steganography can't be used for digital watermarking - it doesn't add information to every part of the carrier.

I'm aware that digital watermarking systems are pretty sophisticated (and expensive!) technology and that many companies base their existence on such systems, but one would expect that there is at least one open source solution for that. At least, a most primitive one.

Design patterns in Python #1, Proxy

2006-08-02T18:07:00.000+02:00

Back to programming related topics :) . This is the first of several articles where I'll try to explain some of design patterns using the Python programming language. Design patterns are standard approaches and solutions for some kinds of problems in software engineering. They are often being classified into several classes like architectural, structural, behavioral etc.

In this article, we'll be dealing with fundamental architectural design pattern - Proxy. Proxy is a class functioning as interface to another class. Most known application for Proxy design pattern is reference counting, but it can be used in numerous other situations like defining an interface for class or hiding some parts of the base class. As object oriented model in Python does not allow to give access rights to class methods, Proxy comes very handy when you need to prevent lazy user to use your class in a wrong way.

We'll build Proxy for simple class which has only two methods. Methods just print some strings.

class Foo:
 def first(self):
    print 'First method'

 def second(self):
    print 'Second method'

Obviously from Proxy definition, it must pass method calls to base class. We'll initialize Proxy to point to base class in Proxy's __init__ method, and pass the method calls by using __getattr__ builtin method. This is the common code for all Proxy classes:

class Proxy:
 def __init__(self, target):
    self.target = target

 def __getattr__(self, attr):
    return getattr(self.target, attr)

Next, we need to adapt this generic class to our Foo class. Let's assume that we want to hide the method second. The way to do it in Python is to inherit the Proxy class and override the base method with empty (pass) statement:

class FooProxy(Proxy):
  def second(self):
     pass

And finally, we'll test our new FooProxy class:

>>> f = Foo() 
>>> f.first()
First method
>>> f.second()
Second method
>>> fp = FooProxy(f)
>>> fp.first()
First method
>>> fp.second()
>>>

You may wonder why we didn't just inherit the Foo class and override the second method. Well, that way we would need to make some kind of copy constructor to copy data from base class and then we would have two nearly identical objects in memory, and that is not what we wanted in the first place. Proxy must serve just as mediator to base class.

Healing brush soon to be found in Gimp!

2006-07-31T17:19:00.000+02:00

One of the key goals that OSS must achieve to attract desktop users to Linux is to have proper substitutions for widely used application suites like Microsoft Office, Adobe Photoshop and Macromedia Dreamweaver. While OpenOffice.org was already pretty good set of applications, more usable than that of Microsoft if you ask me, Gimp was still far behind Photoshop and there was nothing near as good as Dreamweaver.

Recently, I've been playing a lot with Gimp. For sure, Gimp can now be used for most of everyday image manipulations that you can do with Photoshop. Some day ago, I saw that one of the Google Summer of Code projects is titled "Healing Brush for Gimp". Great! Lot of users who used Photoshop missed that feature in Gimp. If that is to be implemented, Gimp could really become "PS killer". As Google Summer of Code is coming to end, I am eagerly waiting to see those new Gimp features!

XMLTV

2006-07-27T20:18:00.000+02:00

If you have a cable or satellite TV set, your biggest problem is probably to find something to watch. Finding out what's on by using only TV magazines and web pages generally sucks. Luckily, most of TV stations provide XMLTV data. XMLTV is XML like language, a common format used for tracking TV channels listings. Like all XML data, XMLTV is easily parsable and has clean syntax. For example, this is the XMLTV listing for first programme of Croatian national television).

One of the most advanced applications for tracking XMLTV is Phazer, great work of Croatian author Kowac. Phazer is based completely on .NET and is available as standalone application, web application and web service. It includes many additional features like alarms. Try it out, you may like it.

Downfall of Yahoo?

2006-07-21T10:15:00.000+02:00

Who of you still use Yahoo as primary search engine? Once Yahoo was probably the most advanced and most used search engine, but now little is left from the former fame. Suffering major market losses because of lack of innovations and incompetence to confront Google at any area, yesterday Yahoo suffered yet another hard hit. Company's shares are down for almost 22 percent and estimated loss for shareholders is round 10.4 billion dollars. This was started by announcements that net income is down by 78 percent for the second quarter of last year and that Yahoo's new ad system won't be out till end of year.

Surely nobody will make big investments in Yahoo in the near future. The time is coming where Yahoo will need to fight for it's own survival.

VIM's key features

2006-07-18T12:41:00.000+02:00

This is probably the most complete vim's command list I've ever seen. It includes both basic key commands and the most common advanced operations, so that even the most advanced vim gurus could find some features they didn't know. Assuming you don't use that damned "Esc Meta Alt Control Shi(f)t" editor, you'll eventually have use of that page. Also, you may take a look at the vim's cheatsheet (1024 x 768 gif, nice if you want to print it and hold it near your monitor) .

Easter eggs

2006-07-14T18:35:00.000+02:00

I found interesting site with Easter Eggs collection. Here are some of my favorites :)


$ apt-get moo
      (__)
      (oo)
/------\/
/ |    ||
*  /\---/  ~~   ~~
...."Have you mooed today?"...

$ aptitude -v moo
There really are no Easter Eggs in this program.
$ aptitude -vv moo
Didn't I already tell you that there are no Easter Eggs in this program?
$ aptitude -vvv moo
Stop it!
$ aptitude -vvvv moo
Okay, okay, if I give you an Easter Egg, will you go away?
$ aptitude -vvvvv moo
All right, you win.

                            /----                     -------/                          /                                  /                |
-----------------/                  -------- ----------------------------------------------

Happy?
$ aptitude -vvvvvv moo
What is it?  It's an elephant being eaten by a snake, of course.

Google strikes back

2006-07-12T18:02:00.000+02:00

It seems that rants on Google in my last post were a little bit harsh. I finally received the invitation and now I'm simply delighted with quality of Google Analytics - it's unbelievable what they're giving for free!

In the meantime, I've decided to use Google Calendar. It comes handy if you need to share your calendar with other people and you want it to be available anywhere and anytime. You don't need any additional software besides your favorite web browser.

At last, we come to Google Trends which displays search volume for specific keyword over the time. It's interesting to see graphs for some keywords, for example this (Java) or this (Ruby on Rails).

First disappointment with Google

2006-06-24T22:48:00.000+02:00

So far I'm very pleased with quality of Google services. I use Google Maps, Gmail, Google News and dozen more of Google's services. Lately I've been writing some scripts in PHP to track visits to my home page. Some day ago a friend of mine suggested me to try Google Analytics which in essence does the same, but also displays plenty of other statistics. Unfortunately, this service requires that you wait for invitation. After I applied for the invitation, I've been waiting for that invitation for almost a week! Not a kind of behavior I would expect from Google...

Digital maps

2006-05-30T23:19:00.000+02:00

If you're up to Croatia (or you live there and travel a lot), you may find this link very useful.

Arrays features

2006-05-25T14:45:00.000+02:00

One of the nice features that may be found in .NET CLS is CreateInstance method from Array class. Tired of arrays which starting index is 0 or 1? You can now make arrays with arbitrary starting index. Take the following example:

using System;

class Test
{
 public static void Main()
 {
   int[] bounds = {2000, 10};
   int[] sizes = {6, 10};
   int[,] array = (int[,])
       Array.CreateInstance(typeof(int), bounds, sizes);

   array[0,0] = 10;
 }
}

This creates two dimensional array which indexes are between 2000 - 2005 for first dimension, and 10-19 for second dimension. So, this example throws IndexOutOfRangeException when we try to access array element with index [0,0]. Change the index to [2002, 14] to get a proof that this works :)

Opera bye bye

2006-05-06T14:42:00.000+02:00

After 3+ years using the Opera web browser, today I've stopped using it. There are several reasons why I did that. First of all, Opera renders tables very bad - almost every day I see broken tables in Opera or they are not being displayed at all. Then the JavaScript support in Opera. When I compare all of the leading web browsers available (IE, Firefox and Opera), I think that JavaScript engine in Opera is most broken. You are lucky if author included only some minor scripts, but if the page contains some more complex scripts you will probably see broken page. Most obvious example for this are the pages which contain some Ajax functionality - most of them are not displayed in Opera at all. There are plenty more rants on Opera - it cannot integrate in my Linux system as much as Firefox (I prefer GTK over Qt), I don't want integrated email and Usenet client (Opera with -nomail option is not what I mean), I want the source to be opened... For all that, Opera is purged from my machine.

Back on the road

2006-04-17T00:29:00.000+02:00

After almost two months without posting I've finally returned to blogging. Much changed in the meantime - most notable change is that I've finally switched from old 56k modem to broadband 512k DSL connection so I guess I'll be writing more often now :)

Automated testing in Python, part 1

2006-02-18T14:41:00.000+01:00

As my work on IKEv2 project continues, I made a little investigation on automated testing facilities built into the Python programming language. My job on IKEv2 was to built automated test environment for the IKE daemon. Testing scripts were written in Perl (because I didn't write it from scratch by myself, much of the code was already written - in Perl) and were completely separated from the IKE daemon code itself. I think that fact is the key of all problems I had (and still have :) during the development.

When I looked at the Python's support for automated testing, I was shocked! It supports exactly what I was thinking off. The doctest module provides a very elegant way to test the code. What you need is to simply embed correct interpreter answers to the docstrings. E.g., I wrote a function which returns a list of prime factors of a number:


import math
import doctest
import fact

def factorize(n):
    """
        Returns list of number's prime factors.
        Examples:
        >>> factorize(32)
        [2, 2, 2, 2, 2]
        >>> factorize(1234)
        [2, 617]
    """
    factors = []
    while True:
        found = False
        for i in xrange(2, int(math.sqrt(n)) + 1):
            if not n % i:
                found = True
                factors.append(i)
                n = n / i
                break
        if not found:
            factors.append(n)
            return factors
             
                                                                                  if __name__ == '__main__":
    doctest.testmod(fact)

Note the three important sections in the code: you need to import module that you are testing (the file is named fact.py), then insert the correct answers from the interpreter into the docstrings, and finally call doctest.testmod() method. There are two ways to run this. First, a verbose mode: just type


$ python fact.py -v

(Not python -v fact.py!). It will print details during the testing process.


Running fact.__doc__
0 of 0 examples failed in fact.__doc__
Running fact.factorize.__doc__
Trying: factorize(1)
Expecting: [1]
ok
... [cut] ...

The second way to run it is without -v switch. That will suppress any output unless some errors are found during the testing. For example, let's change line


for i in xrange(2, int(math.sqrt(n)) + 1):


for i in xrange(2, int(math.sqrt(n))):

which is obviously wrong. Now let's run the program again:


$ python fact.py
****************************************
Failure in example: factorize(32)
from line #7 of fact.factorize
Expected: [2, 2, 2, 2, 2]
Got: [2, 2, 8]
****************************************
Failure in example: factorize(100)
from line #9 of fact.factorize
Expected: [2, 2, 5, 5]
Got: [2, 2, 25]
****************************************
1 items had failures:
  2 of   5 in fact.factorize
***Test Failed*** 2 failures.

Next time I'll write about even better thing than this - the unittest module.

Python's OOP

2006-02-08T23:13:00.000+01:00

One of the things I found Python's OOP model lacks are proper constructors. For example this C++ code works just as I would expect - it first calls constructor of the base class and then that of the derived class:


#include <iostream>

using namespace std;

class First {
    public:
        First() {
            cout << "First" <<>
But when you try the same thing in Python you get this:

class First:
    def __init__(self):
        print "First"

class Second(First):
    def __init__(self):
        print "Second"

if __name__ == '__main__':
    Second()

%  python foo.py
Second

That is not what I want. I want that class hierarchy works as in other object oriented languages. Besides that, Python doesn't know anything about private or public visibility of variables - everything in object is accessible. Is that all what Python's OOP model offers to me?

map and lambda strike again
2006-01-19T14:52:00.001+01:00
Few weeks ago I wrote a post in which I said that lambda, filter, map and reduce are not needed in Python and that all can be done with simple list operations. Well, today I found a problem where that kind of approach just gets ugly and inapplicable.

Here is the problem. I have a class Graph and I need a constructor for it which will take data from the file and fill the appropriate adjacency matrix. The input format is straightforward, i.e. for a simple weighted graph:

0 2 0 1
2 0 2 3
0 2 0 0
1 3 0 0

Here is the code I wrote:

class Graph:
  def __init__(self, filename):
    try:
      f = file(filename)
      self.adj_m = [e.split() for e in [e.rstrip() for e in f]]
      f.close()
    except IOError:
      print "IOError: " + filename
      sys.exit(1)

After this, the adjacency matrix looks like this:

[['0', '2', '0', '1'], ['2', '0', '2', '3'], ['0', '2', '0', '0'], ['1', '3', '0', '0']]

Naturally, graph theory algorithms are designed to deal with numbers, not strings. What I needed was this:

[[0, 2, 0, 1], [2, 0, 2, 3], [0, 2, 0, 0], [1, 3, 0, 0]]

I spent 20 minutes on examining how to convert a multidimensional list elements from strings to numbers using only list comprehension. I saw no simple and elegant way to do it. Thus, I asked a friend (btw, do visit his blog ;) for help and his solution was this:

self.adj_m = map(lambda x: map(int, x), self.adj_m)

This is very nice. It seems that we need constructs from functional programming languages after all :).

No "sucks" this time
2006-01-14T22:03:00.000+01:00
Today I talked with a friend and he told me that he had noticed most of my posts contains sentences like "X sucks" of "I hate X". Well, that's true, what other can you expect when you look at the title of the blog ;). But, this time I'll have a different approach. The lucky object of my praises is - Mono.

Lately I play a lot with Mono and I can talk about it only in superlatives. It is a fast and robust CLI (Common Language Infrastructure) implementation. Novell have done a great job. C# is world for itself. Two years ago I was writing most of my programs in plain C. Then I discovered Python and C#. As Python became my primary language and I worked mostly on Linux, I neglected C# a lot. Few days ago when I upgraded my Debian GNU/Linux from Sarge to Etch I noticed that Mono deb packages were added to the distribution (maybe they were available before? I don't know). Ofcourse, I installed it instantly and I wasn't disappointed.

As I worked in C# before I knew most of it's capabilities, so I wanted to test how fast Mono's VES (Virtual Execution System) executes CIL (Common Intermediate Language) produced by mcs (Mono C# compiler). Particularly, I was interested in Python vs Mono speed. I wrote several small programs (Sieve of Eratosthenes's, recursive quick sort, use of built in API to get a web page source...). No words are needed, I'll just say the number: C# beats Python in every segment of the test for more than 40%. I'm just interested whether Python is so slow, or Mono is so fast...

I think Mono has a bright future on Linux. Moves of several big players on the market confirm that. Even Fedora 5 will contain Mono :). And I will definitely use it more often from now on. The next thing I'll examine is Gtk#...

Prolog is stupid
2006-01-10T21:00:00.000+01:00
I have a new story... It is called "Prolog - The King of the Useless Programming Languages". I have learned a dozen of programming languages but none of them hasn't irritated and disappointed me as much as Prolog. Honestly, I hate it. I hate every of his predicates, rules, facts, lists and stupid examples of more stupid logic games (bonus question: have you ever seen a Prolog program which actually does something useful). I hate logical programming paradigm in general.

I have to learn Prolog as part of "Intelligent systems" class that I'm listening to this semester. Why, oh why don't we learn a normal programming language like Scheme or Haskel (err, what a heck, even CLISP would be better)?

BTW, check out this Prolog "pearl" (one of the best I've seen). You can write something like this:

foo(bar, baz)

but not (note the position of the white space):

foo (bar, baz)

Prolog just sucks!

lambda, map, filter and reduce no more
2006-01-03T17:25:00.000+01:00
There is simply no need to use bad words on decision to remove these functions from Python. As one of the main concepts of Python is "there should be only one (and it should be an obvious one!) way to do it", it was only matter of time for language architects to make this decision. In my opinion, these functions are ugly and completely unnecessary in Python, and are probably stuffed into the language definition to rip off some programmers from the functional languages which use these functions a lot.

Of course, each of those functions taken from functional languages can be easily replaced with clearer and more Python-like syntax. For example:

>>> l = [1, 2, 3, 4, 5]
>>> map (lambda x: x+1, l)
[2, 3, 4, 5, 6]
>>>
>>> [element+1 for element in l]
[2, 3, 4, 5, 6]

There is simply no need for using lambda, and second example is much more cleaner nd easy to understand. For filter, similar syntax could be used:

>>> l = [1, 2, 3, 4, 5]
>>> filter (lambda x:x%2, l)
[1, 3, 5]
>>>
>>> [element for element in l if element % 2]
[1, 3, 5]

Again, no need for lambda, and all can be done with simple list operations.

Then we come to reduce. I must admit that there is no elegant syntax to replace it like in the previous two examples (although it is not problem to implement it with iterations and slices), but on the other hand I can hardly remember a real world example where reduce was used.

If you ask me, this step should have been done a long time ago. Python is developing in the right direction (at least on this matter, but that is topic for one of the incoming posts :).
l can be done with simple list operations.

Config file parser
2006-01-01T19:00:00.000+01:00
One of the things I like in Python at most is that Python comes with batteries included. Yesterday when I finished my first web service written in Python I decided to make it fully configurable by using a simple configuration file. What I had in mind was a config file with syntax like this:

# example section
section {
   option
   option = value

   subsection {
       ...
   }
}

that is, a config file like most Unix utilities have. After few minutes spent on googling I found a standard Python module called ConfigParser. ConfigParser contains three classes (RawConfigParser, ConfigParser and SafeConfigParser) for config file parsing, but unfortunately none of them supported format which I needed. Format which ConfigParser could parse was something like this:

# example section
[section1]
option: value

[section2]
option: value

Similar format is commonly used by .ini files on Windows and some applications (e.g. Opera). Main disadvantage I found in this format was lack of sub-sectioning ability. I knew I had an alternative - XML config file, but I decided it would be an overkill for this kind of application :). So I selected ConfigParser after all.

It is really a great thing that Python provides such a large number of various modules, but in this case I think lack of parser for Unix style config files is one of the notable drawbacks. I would like to see that feature in some of the incoming releases of Python.

Blog engines sux!
2005-12-30T12:37:00.000+01:00
At the time I decided to write this blog I wanted to make my own blog engine and RSS feed generator using XHTML and Python. I decided to use XHTML because I thought using it would make it easy to convert the posts from RSS to HTML (or vice verse). For the engine itself I selected Python for two reasons. First, I have never made a single Python web script so I wanted to try something new (and I don't like PHP, it is horrible if you ask me). Second, I knew that Python provides a large number modules for manipulating XML data.

After short time I realized that making an own blog engine is not so easy as it seems on the first look, and possibly a complete waste of time. I find XML parsing and generating with Python rather unintuitive and complicated and I really didn't have time to look at it in the depth. So I turned to Perl and hard core text processing (good old noob approach, who cares for XML parsing after all :)... The result was a horrible code that I deleted as soon as I completed it. After that, I consulted google and found two blog engines (I won't advertise them here, they really don't deserve that :). Both of them were horrible, bloated and unsuitable for my needs. I had a simple request - all I needed was a simple system that I could incorporate into my homepage, a simple CMS for making new blog entries and comments, and which could generate RSS feed (any version, 1.0, 2.0 or Atom, I don't care) on the fly. After all the trouble, I realized that I was asking too much, so I decided to use blogger. And the result is here.