Tracking your DHL package in conky

(or converting the DHL tracking page from HTML to plain text, using XSLT)

So, let me explain the problem: You have that package shipped by DHL, and its tracking number. And you're so eager to receive it, that you end up checking the package tracking page every 5 minutes. Your productivity falls to zero.

But, worry no more! Let's put the package status inside your conky, so that you can just have a quick look on the side of your screen, and continue working.

Just in case you don't know (but really, you should), conky is the information bar on the right of the screenshot below:

conky is the information bar on the right on the display. CPU/RAM usage, disk free space, network status, temperature, a calendar, my DHL tracking thing, then weather in a few places.
In case you wonder, and the background is somewhere in Hue, Vietnam.
And don't worry about the gimp error, really.

The idea is to write the code that can generate this:

Yes, they spelled my name wrong...

From something as ugly looking as this:

I don't care how it works, I just want to get it running

Ok! After all the point of this was to increase your productivity, right? You can fetch the script from my github.

Then call it with:

./dhl <AWB>

Where <AWB> is the Waybill number (tracking number). It produces a text-only tracking information for your package.

You can integrate it in your conky with something like:

${font Monospace:size=6}${execi 60 ~/.conky/dhl <AWB> | head -n 3 | fold -w 16}$font

Replace ~/.conky/dhl with the path to where you copied the script. Change head parameter if you want more lines, and fold inserts new lines every 16 characters (change that depending on your conky width).

Now, if you want to know how it works, so you can fix it if it breaks, or update the code for other shipping companies, continue reading.

Inspecting the HTML source

The tracking URL looks like this (where <AWB> is your tracking number):

http://www.dhl-usa.com/content/us/en/express/tracking.shtml?brand=DHL&AWB=<AWB>

Looking at the HTML source, we notice that the interesting stuff is enclosed in a table:

<table border="0" summary="Summary of table content">

Then, you have a succession of thead/tbody tags. The first thead contains general information about the package, that we are not interested in. It starts like this (notice it has class "tophead"):

<thead class="tophead">

The next thead shows the date valid for the following entries. We are only interested in the first column here (the one that contains the date).

<thead>
    <tr>
        <td colspan="5" class="emptyRow"></td>
    </tr>
    <tr>
        <th scope="col" colspan="2" axis="length"
            style="width: 40% ;text-align:left">Thursday, September 19, 2013         </th>
        <th scope="col" axis="length"
            style="width: 30% ;text-align:left ">Location</th>
        <th scope="col" axis="length"
            style="width: 9%;text-align:left">Time</th>
        <th scope="col" axis="length" class="lastChild"
            style="width: 25% ;text-align:left">&nbsp;</th>
    </tr>
</thead>

Finally, the bulk of the events are enclosed in tbody. The first column is a incremented number, the second one is a description of what happened (passed customs, arrived at destination, etc.), the third one tell you the location (but this is often repeated in the description), and the fourth one is the time.

<tbody>
    <tr>
        <td class="" style="width: 5% ;text-align:left">18</td>
        <td class="" style="text-align:left">With delivery courier</td>
        <td class="" style="text-align:left">SINGAPORE - SINGAPORE</td>
        <td class="">7:27 PM</td>
        <td class="lastChild "><!--start contentteaser -->
        <div class="dhl">
        <div><div class="clearAll">&nbsp;</div></div>
        </div><!--end contentteaser --></td>
    </tr>
</tbody>

Ok, now we have an idea of the structure, let's parse that!

Parse HTML with XSLT

Ok, so let's say you have the DHL tracking page downloaded to /tmp/dhl.tmp, and an XSLT file in dhl.xslt, you can parse the page with:

xsltproc --html dhl.xslt /tmp/dhl.tmp

The XSLT file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="text" encoding="utf-8" />
    <xsl:template match="/">
        <xsl:for-each select="//table[@summary='Summary of table content']/*[self::thead|self::tbody][not(@class)]">
            <xsl:choose>
                <xsl:when test="name(.) = 'thead'">
                    <xsl:value-of select="tr/th[1]"/>
                    <xsl:text>
</xsl:text>
                </xsl:when>
                <xsl:otherwise>
                    <xsl:if test="floor(tr/td[1]) = tr/td[1]">
                        <xsl:value-of select="normalize-space(tr/td[4])"/>
                        <xsl:text>: </xsl:text>
                        <xsl:value-of select="normalize-space(tr/td[2])"/>
                        <xsl:text>
</xsl:text>
                    </xsl:if>
                </xsl:otherwise>
            </xsl:choose>
        </xsl:for-each>
    </xsl:template>
</xsl:stylesheet>

Let's take it step by step. It starts like this:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="text" encoding="utf-8" />
    <xsl:template match="/">

Nothing special here, apart from the text output mode, so that xsltproc outputs a text file (and not another XML file...).

Now begins the fun. We look for a table with summary attribute 'Summary of table content'. Inside that table, we look for thead and tbody elements, that do not have a class attribute set, so we can exclude the 'tophead' row, that we are not interested in.

<xsl:for-each select="//table[@summary='Summary of table content']/*[self::thead|self::tbody][not(@class)]">

Now, thead (containing only the date of the following events) and tbody (containing events) need to be parsed differently. This is done with xsl:choose:

<xsl:choose>
    <xsl:when test="name(.) = 'thead'">
...
    </xsl:when>
    <xsl:otherwise>
...
    </xsl:otherwise>
</xsl:choose>

For thead, we just want to show the date, that is the first th inside a tr (tr/th[1]). Then we print a new line with xsl:text.

<xsl:when test="name(.) = 'thead'">
    <xsl:value-of select="tr/th[1]">
    <xsl:text>
</xsl:text>
</xsl:value-of></xsl:when>

For tbody, it is slightly more complicated. First, we check that the first column is indeed a number (this removes the last row in the table, which is another type of summary): this is done with a "trick" (floor(tr/td[1]) = tr/td[1]). Then we print the time (4th column), followed by a colon, and the event description (2nd column).

<xsl:otherwise>
    <xsl:if test="floor(tr/td[1]) = tr/td[1]">
        <xsl:value-of select="normalize-space(tr/td[4])"/>
        <xsl:text>: </xsl:text>
        <xsl:value-of select="normalize-space(tr/td[2])"/>
        <xsl:text>
</xsl:text>
    </xsl:if>
</xsl:otherwise>

That's it! Then you can put everything in a shell script, see the complete code on github for details.

It's so cool, I want more!

I get it. I, too, have become of fan of parsing XML/HTML from scripts. See this post for another example.

Bokeh-fixing: Opening and cleaning an Olympus OM 50mm f/1.8

In the previous post, I talked about the Olympus OM 50mm f/1.8, and how I find it interesting to use on my Micro Four Thirds camera (Panasonic DMC-GX1), along with some sample shots.

When taking some night shots, with an object close in focus, I could see obvious defects in the bokeh, that is, the round circle of light coming from a distant, out-of-focus, light.

The next image is taken by pointing at a spot light about 200m away, but setting the focus at its closest position (0.45m). This gives a large bokeh:

Spot light is about 200m away, focus set at 0.45m, aperture f/1.8. ISO 1600, 1/40s. The bokeh dimension is about 1000x1000 pixels, that is a little more than a fifth of the width of the image.

Clearly, something is wrong here: there are some black dots and strange reflections on the left side of the bokeh circle.

By looking inside the lens, I can see something that looks like oil drops, apparently not far from the back, maybe behind the outermost lens. I'm wondering if it comes from the aperture mechanism, since it's slow and has obvious oil marks on it, but I can't tell for sure.

I looked up online, and some people on dpreview forums advise that it may not be worth the fuss trying to open it up, and it would be easier to buy a new one, considered the price. On the other hand, it is such a cheap lens (~25USD with shipping) that it would not be a disaster if I broke it. Looking further, I found some diagrams on Olympus Dementia, but even if you can figure out which exact model of lens you have (Olympus made multiple fairly different versions over the years), it still does not tell you how to open it.

Anyway, since my problem looked like to be at the back, and since there are 3 obvious screw there, I decided to start on that side:

It comes out easy. The lever to unlock the lens from the mount falls down (left on the picture below), but it isn't very tricky to find out how to put it back:

Then, a big part of the aperture mechanism comes out easily. This mechanism contains a spring that opens the aperture to the maximum. When the aperture lever is pressed (right of the picture), the spring is extended, and the lens stops down to the desired setting on the aperture ring. I took out the whole thing, taking care of keeping all the elements together. The lever falls out, but it's easy to figure out how to put it in again:

Then I'm left with this, and nothing obvious to remove. I want to remove the metal ring at the top, as it looks like there is oil right behind the glass that it is holding. It is screwed to the bottom part, but hard to remove. I notice some glue near the joint, so I scratch it off with a box cutter:

And after this, I managed to open it up, using a soft cloth to give me more grip and avoid damaging the lens (you can see some scratch on the screw thread, that's where the glue was):

The top glass is now free, and the easiest is to remove it by gravity: invert the lens, hold it in a soft cloth so that the glass does not fall down too hard, and shake it a bit.

No oil on that lens, but, luckily, I could spot it on the lens just below. I did not want to introduce any liquid in the lens, so I removed it the best I could, possibly smudging around instead of properly removing it, actually. A more proper way would have been to find the way to take out that glass, but, well, that would have required significantly more work.

After getting convinced that most of it was removed (or at least evened out...), I reassembled it, and took the same picture. Notice the improvement!

Left: before, Right: after. There is still a slight smudge on the right, but it is noticeably better.

And my 13.5 USD 50mm f/1.8 recovers it's original beautiful bokeh!

Olympus OM 50mm f/1.8 on Micro Four Thirds

One of the strong points of the Micro Four Thirds (MFT) system is that, thanks to its short flange focal distance, you can mount lenses designed for almost any other camera system.

I believe you can get the best deals by buying Olympus OM lenses: There is no current camera supporting those lenses anymore, but they were produced in mass in the 80's and 90's. These are ingredients for a high supply, low demand, therefore low prices on auction websites.

This is especially true of the Olympus OM 50mm, f/1.8, that used to be a kit lens with many film Olympus cameras. Almost a year ago, I bought one on eBay, for 13.50 USD (+ 11 USD shipping). I mounted it on my Panasonic DMC-GX1, using a OM to MFT adapter (less than 10 USD).

I originally bought this lens to use it as part of a custom tilt-shift adapter, but realised that the 50mm focal length is usually too narrow, and purchased a Promaster 28mm f/2.8 for that purpose (OM mount as well).

This lens is really amazing (especially considered its price): It becomes a short telephoto lens on the MFT system (100mm full-frame equivalent), which gives you interesting constraints: you have to focus on details, or put some distance between you and your subject. The large aperture makes it particularly interesting in low-light conditions (museums, night markets, etc.). On the other hand, it does require ND filters in bright daylight, as you are hitting the maximum shutter speed of the camera (1/4000s for the GX1): a 3-stop ND filter, that is ND8 or 0.9 optical density, works perfectly for these situations. I actually never stop the aperture down: I would rather switch to another lens if I want more depth of field.

Focusing is not easy, especially without a viewfinder. MFT cameras provide a magnified view to help you focus, but, with a bit of practice, I'm able to get a reasonably good focus without using that mode, by moving the ring back and forth until I have a good idea of the best position.

The lens I got was in good condition, except for the aperture, that is a bit sluggish: you need to jiggle the aperture ring to get it back to f/1.8 if you stop it down. I could also see some oil on the aperture blades: probably the reason why the mechanism is not working as well as expected. But again, since I only use it at maximum aperture, this is not really a concern for me.

I used that lens for a number of night shots, and realised that the bokeh is not exactly as round and nice as it should be: there is some "dirt" on the left side of the disk (when held in landscape orientation). This does not show up clearly in most shots, but it looks quite silly when the same pattern repeats in different locations on the frame:

Each of the bokeh rings shows some black spots at the bottom: looking through the lens, I can see some oil marks.

The next post will show you how I managed to fix the problem, by opening up the lens.

In the mean time, I uploaded on Flickr a collection of photos taken with that lens:

Good morning haze!

Yesterday morning Singapore woke up under thick haze due to forest fires in nearby Sumatra. Not healthy: You can feel it in your throat, and some corridors smell like Scamorza (some delicious Italian smoked cheese).

It smells just like that... (Image from Necrophorus@Wikipedia, GFDL)

Anyway... it gives some "interesting" light when the sun is low.

Good morning purée... (slightly underexposed)

Red sun, still high above the horizon (~1h before sunset).

Parse XML from shell scripts

As part of the crouton/chroagh project, I wanted to be able parse D-bus configuration file, to figure out what user account the system D-bus runs under.

The configuration file looks like this:

# /etc/dbus-1/system.conf
<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>

  <!-- Our well-known bus type, do not change this -->
  <type>system</type>

  <!-- Run as special user -->
  <user>dbus</user>

  <!-- Fork into daemon mode -->
  <fork/>

.....

</busconfig>

Yes, that's XML. And normally, XML and shell scripts are not exactly good friends...

What we want to be able to do here is to fetch the content of the user tag.

Solution 1 - sed

Well, that's without doubt the easiest:

sed -n 's|.*<user>\(.*\)</user>|\1|p' /etc/dbus-1/system.conf

Problem is that the is no guarantee that there would be no other user tag^[1], in other parts of the file, and, who knows, the tag might be commented out. We also do not handle "misplaced" newlines in the file...

^[1] Actually, there is no other user tag, according to the dtd file, but not necessarily in a general case... And well, this post is boring if I stop here, right?

Solution 2a - xmllint

A more proper and generic solution, making use of the xmllint parser:

echo "cat /busconfig/user/text()" | xmllint --shell /etc/dbus-1/system.conf

Problem is, xmllint is not really script-friendly, and outputs some garbage along with the desired output.

$ echo "cat /busconfig/user/text()" | xmllint --shell /etc/dbus-1/system.conf
/ > cat /busconfig/user/text()
 -------
dbus
/ > $

In my case, I know how a valid username looks like, so I just pipe the output through grep '^[a-z][-a-z0-9_]*$', and that's the solution that is used in my code. Also, xmllint is installed by default on Chrome OS, so that's good enough for me.

Solution 2b - xmllint and write

An improved version would be:

echo "cd //busconfig/user/text()
     write tmp" | xmllint --shell /etc/dbus-1/system.conf
cat tmp

This is cleaner, as the full text of the tag is written to a file, but this requires an intermediate file, so, maybe not that nice...

Solution 2c - xmllint, write, and fd/3

Maybe an even nicer one, assuming /dev/fd exists (as it does on recent Linux distributions):

exec 3>/dev/null
echo "cd //busconfig/user/text()
     write /dev/fd/3" | xmllint --shell /etc/dbus-1/system.conf 3>&1 1>/dev/null 2>/dev/null
exec 3<&-

No temporary file!

Solution 3 - xsltproc

Finally, the most powerful version, using XSLT:

xsltproc - /etc/dbus-1/system.conf <<END
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text" encoding="utf-8" />
<xsl:template match="/"><xsl:value-of select="/busconfig/user"/></xsl:template>
</xsl:stylesheet>
END

That's as good as it gets, and you could easily do much more complicated things with XSLT. But, that's a bit overkill for our purpose (and xsltproc is not installed in Chrome OS).

Chromebook

Since the adventure during my last trip, where my (work/home/main) laptop suddenly died, I realized that carrying a heavy and expensive laptop is not the best idea while travelling (in terms of risk of theft, damage, but also weight to carry around).

A tablet would be an option, but they do not offer the flexibility I want (keyboard, photo editing, etc.), and they are generally not easy to hack. Netbooks seems to be a thing of the past, so I went for a Chromebook.

I went for the second cheapest (Samsung ARM Chromebook, USD 249), the cheapest being a bit too heavy and bulky (Acer C7, USD 199), and maybe less exciting with a standard x86 processor.

It is unfortunately not available in Singapore (at least not through usual channels), and some websites, such as Amazon, do not want to ship it to Singapore. Fortunately for me, B&H Photo Video does not care, and shipped it to me for USD 52. Three days later, I received my new toy, with no GST to pay (total amount below SGD 400, see here).

Samsung ARM Chromebook (running Archlinux/XFCE using chroagh, but more on that later).

Well, I won't repeat what other reviews say online (there are plenty), but, basically:

• First, you pay what you get for. Don't expect it to behave like a laptop with a price tag 10 times higher (a friend said: "Fake Macbook!! Oh no, Korean!!", and, well, that's not completely inaccurate): The body is plastic, and this is not a powerhouse. But still, for the price, I find it awesome.
• Very light (1.1kg) and thin (1.8cm).
• No fan, no moving parts: No noise at all, little heat, and long battery life (~6h).
• Dual-core ARM processor (1.7Ghz): This is a bit better than your last generation cell phone (well, it does out-of-order execution so it's probably a step faster), but it seems to do the job fairly well.
• Chrome OS:
• Very good browsing experience, very fast. Youtube works well in general, but tends to lag when you do other things at the same time: not ideal if you have your music playlist on Youtube.
• You need connectivity for most stuff. There are a few offline apps (I haven't tried many), but most things will require a Wifi connection.
• Movie playback (from files, non-Youtube) is a problem: It could not read some x264-encoded files, even in SD quality. I mean, it could, but only displayed one frame per second or so... Maybe this could be improved in later versions of Chrome OS by optimizing the video decoder (e.g. use whatever hardware acceleration facilities the processor has).
• Finally, and this is critical for me: hackability. This is not an Apple device or a locked Android phone. Google lets you do whatever you want with your computer, including wiping Chrome OS to install something else, or installing another Linux in parallel (crouton/chroagh).

In short, mildly recommended for the normal user, Chrome OS is great, but really needs a working connection to be used to its full potential. And this is not always the case while travelling.

Strongly recommended for the more advanced user, as you can install a regular Linux in parallel: either as Dual-boot, or as a chroot inside Chrome OS, where you can switch between Chrome OS and your Linux with a simple key combination. I used the wonderful crouton to install Ubuntu, and ported it to support Archlinux (chroagh). More on that for a later post.

Archlinux - Swapping hard drives between computers

My good old Dell Lattitude E6400 didn't survive my last week-end trip to Vietnam...

For no apparent reason, the power button stays on for a few seconds, then switches off. The num lock light blinks, indicating a processor error... Luckily, I bought a 4th year of warranty, which would expire in 2 months time. Called up Dell on Tuesday, they came on Thursday (next business day), swapped the logic board, and it works again!

In the mean time, I found a Lenovo Thinkpad T410 at the office, which is roughly as old as my Dell (and the keyboard is covered by something that looks like mould, but that's another story). I could use the Windows installation on that computer, but I suffer from a strong allergy to Windows, so I decided to do something slightly smarter instead of risking mental breakdown.

Both the E6400 and the T410 are good professional laptops, and the hard drives can be taken out in less than 5 minutes - read: those are not Macs where you need to dismantle the whole computer to swap a component, only 5 screws need to be removed on each computer (1 to take out the hard drive assembly, and 4 to remove the drive itself).

Anyway, took out the T410 hard drive, replaced it with my drive from the E6400, booted the T410, and immediately got my Archlinux running, with X server, network, and everything. The fact that both computers have similar hardware, and both use a nVidia graphics card, made the task slightly easier (no extra driver to install or xorg.conf to reconfigure), and the new screen resolution was automatically detected (the T410 features a pathetic 1280x800, vs a slightly better 1440x900 for the E6400).

Now, just for a minute, imagine how complicated it would have been to do the same with a Windows system (re-installation? problems with key activation?).

Anyway, that got me thinking that I should carry such a big laptop on holidays, so I ordered a Samsung Chromebook, with the idea of install Archlinux on it. And I will soon need a new laptop to replace my E6400...

Just one picture of my trip to Ho Chi Minh City:

Somewhere in Saigon: Doll waiting to cross the road. Or celebrating Reunification day, not sure...