Monthly Archives: January 2016

My favourite debugging story

This was some years ago, when I was living in Vientiane, the capital of the Lao Peoples’ Democratic Republic. It was 1994 or thereabouts – just prior to the release of Windows 95. I had written a piece of software called “Keyman” which was being increasingly used to type in Lao in Windows 3.1, overloading characters in the 128-255 range of the standard US English character set at the time (I don’t want to be too technical here). Before Unicode.

The owner of a local computer store had had reports of an issue with Keyman from one of their clients in the provincial capital of Savannakhet, about a one hour flight from Vientiane in a small plane. The technical minutiae of the problem escape me now, but it was something to do with a certain set of keystrokes which gave the wrong output in some applications – I think Excel. The report had been communicated over the telephone to the computer shop technical staff, and then translated into English for my benefit – as my Lao was probably not good enough to really get the detail. So as you can imagine, Chinese Whispers is a good way to describe the final report I received.

I tried to diagnose the problem from the description, and tried to reproduce it on my computer, but could not figure it out.

Now it is important to remember that Laos in 1994 was still pretty much unknown to the outside world. There were few tourists; it was (and is) a communist country, at least in principle. Things there didn’t work quite the same as in Australia. There was no Internet access in Laos at that time, telephones were unreliable and the use of modems was technically illegal. This meant that remote diagnosis was only possible by means of telephoned conversations over noisy phone lines, by fax, or with posted letters. The post often took weeks, even in-country.

So after a few days of fruitless telephoning back and forth, the owner of the computer shop suggested I accompany him on a trip down to Savannakhet. (From memory, he was already planning to visit). I was but a callow youth, of 17, and so this was a fantastic opportunity!

When we met up at the airport, the first thing I remember was standing in the security line behind a Lao businessman who caused a bit of a ruckus at the hand luggage screening, because his briefcase had two pistols in it. This seemed a little unusual, even in Laos. After some discussion, his pistols were removed from his hand luggage and given to a guard, who told him he could not take them on the plane because there was not a separate luggage hold. I don’t know what happened to them after that as we were ushered through the security.

When our plane started boarding, a second problem arose. It appeared that the flight had been overbooked, or perhaps they’d substituted a smaller aircraft. The plane we could see was a Xian Y-7, a Chinese clone of a Russian Antonov An-24. The Wikipedia page linked above shows a picture, coincidentally enough of a Lao Aviation plane, perhaps the very plane we were to fly on (they only had 4).

But, as I said, the plane was overbooked, and we ended up in the group of about 10 that didn’t get onto the plane. Pistol-man was in the group that boarded the Y-7, and we didn’t see him again.

Fortunately for us, Lao Aviation had a solution to the problem. They simply rolled a second plane out of the hanger, a Harbin Y-12 this time, and fired it up.

Well, they tried to fire it up. It coughed and spluttered, and lots of black smoke poured out of the engines, but it didn’t start. Boh pen nyang. They pushed it back into the hanger, and rolled out yet another Y-12.

At this point I was feeling a little nervous.

The thired plane coughed and spluttered, poured out lots of black smoke, but it started! After a minute, they shut off the engines and asked us to board.

You can see in the picture below how part of the engine cowling is painted black. You can also see, if you look closely, how there are black smudge marks around that black painted area. Yeah. Smoke. I guess that the smoke mustn’t be a big problem, but it wasn’t inspiring at the time.

Lao_Aviation_Harbin_Y-12_Sibille-1

https://upload.wikimedia.org/wikipedia/commons/4/4b/Lao_Aviation_Harbin_Y-12_Sibille-1.jpg

Image © 2000 Regis Sibille, used under CCSA. (Another picture: http://www.airliners.net/photo/Iran—Revolutionary/Harbin-Y12-II/1503896/L/)

We rolled out and took off moments after the larger first plane. For a while, we could see the larger plane ahead and slightly above us – I don’t know why they didn’t go straight up to cruising altitude as the Y-7 is a lot faster than the Y-12. But eventually the Y-7 was out of sight. The scenery was in places spectacular. As I recall, the plane stayed in Lao airspace for the whole flight, despite this making the flight significantly longer.

Arriving in Savannakhet, we first travelled to the house of a friend of the computer store owner. This man happened to be one of the richest men in southern Laos. He had a beautiful house, filled with beautifully carved tables, paintings and collected antiques. After a brief meeting there, we were escorted by this man to a café in the city for a coffee. Well, some of us drank coffee; I didn’t. I was but 17 and at that age drank far too much Pepsi.

At this sidewalk café, an interesting encounter occurred, which has stuck with me. A street sweeper stopped and ordered a drink, and sat at the same table as this very wealthy man, and they struck up conversation. For some time, all at the table talked. The friendly interactions between two very different social classes was remarkable to me at the time – especially coming from Thailand where the social strata were clearly delineated.

Finally, social requirements met, we made our way to the computer with the problem. It was about 3 or 4 flights up dusty stairs in one of the tallest buildings in the city. There was a lift, but use of it was definitely not recommended. The problem was demonstrated to me, and I was able to observe there what had flummoxed me from afar. After just a few minutes, I realised what the problem was, and had enough information to fix it. I didn’t have my laptop with me so I had to write down some notes – and then we left them, a little sad that we couldn’t fix the problem immediately.

What was the problem? I actually don’t remember the detail. I just remember how we got there and back again!

We took a ferry across the river to Thailand, and took a bus to Nakhon Phanom. There were two reasons: first, my friend the computer shop owner wanted to visit some relatives there, and second, the roads in Laos were at the time in such poor condition that travel on them was best avoided if an alternative was available. Thai roads were busy but generally in excellent condition.

Once in Nakhon Phanom, it was a short motorcycle taxi ride to the relatives’ house by the river. We stopped there for a couple of hours and drank tea with them (yes, even I, Mr Pepsi Boy drank tea).

But when we tootled back to the bus stop, we found our bus had left a few minutes earlier than we had planned!

This was not a big problem. A bystander offered to chase down the bus in his pickup truck. This pickup was sparkling clean, had bright alloy wheels with a thin slick of rubber spray painted on, and a lowered chassis, so much so that the wheels would have been superfluous if they hadn’t been required for their locomotive capability.

pickup

It wasn’t this car, but it could well have been its older brother. Now in Thailand, the buses moved fast. Especially once they got out onto the open road. I’ve been on Thai buses doing 150 km/h or more.

So our intrepid young driver started chasing down the bus, flying down the city streets at over double the speed limit, braking hard for corners and easily avoiding the (fortunately) light traffic, and once he got out onto the open road he was eager to show us what his car would do. So here we were, flying down a Thai highway in a stranger’s car doing a ridiculous speed, chasing down a bus driver that didn’t know we existed. I’m pretty sure my parents would not have been pleased.

At the speed we were doing, we caught up to the bus pretty easily. After some vigorous flashing of the headlights, the bus’s left indicator came on, and it slowed and finally stopped. My friend paid the pickup race driver a token of our appreciation, and with slightly wobbly legs we climbed onto the bus, and off we went to Nong Khai.

We arrived in Nong Khai well after midnight, and ended up at a noisy Thai transit hotel, in a room without windows a couple of floors above the nightclub. One bed: a queen bed.

Now, remember, I was a callow 17 year old youth. The idea of sharing a bed with any man was pretty terrifying. But my friend, who I guess was in his 40s at that stage, kindly noted my abject and unwarranted fear, and gave me the whole bed – I think he sat in the chair and dozed. Not fair, I know.

At about 4am the nightclub finally quietened down, and at about 6am we were awoken by the daily noise that accompanies the start of every day – car doors slamming, shouts, trucks reversing. So we gave up on sleep, got up and made our way back across the border to Vientiane.

Upon returning to Vientiane, I fixed the problem in the Keyman code in a few minutes, prepared a new version on a floppy disk, and rode my bicycle over to the computer shop to deliver it. The computer shop sent the disk on to the user in Savannakhet, and as far as I know, that was that.

That’s how debugging used to work in the nearly olden days. None of these fancy remote desktop VPN SSH thingies. It was a lot more fun.

More windbg tricks with Delphi – how to ignore specific exceptions

This is a really quick post, just noting more flexible ways of handling exceptions in WinDBG, for example ignoring EIdConnClosedGracefully or EAbort by default.

There are two parts to this:

  1. Use script files to build complex breakpoint or exception statements. While you could technically embed it all in one line, it becomes increasingly unwieldy. It’s still painful in a script file but slightly less so.
  2. Use as to create aliases for specific memory addresses in the script, making string comparisons much simpler.

To tell WinDbg to use a script file when a Delphi exception occurs:

sxe -c "$$><C:\\scripts\\windbg_delphi_exception.txt" 0EEDFADE

Then, the script file itself:

$$
$$ Report on Delphi exceptions: setup aliases
$$

.block {
    as ${/v:exception_cleanup} ad /q ${/v:exception_message}; ad /q ${/v:exception_name}; ad /q ${/v:exception_address}
}
.block {
    exception_cleanup
}
.block {
    aS /mu ${/v:exception_message} poi(poi(ebp+1c)+4)
    aS /ma ${/v:exception_name} poi(poi(poi(ebp+1c))-38)+1
    aS /x ${/v:exception_address} poi(ebp+4)
}

$$
$$ Do everything in this block, no matter what the exception is. This way we 
$$ get reports on exceptions without interrupting program execution. You may want to
$$ add extra reporting, e.g. stack traces, other variables.
$$

.block {
    .echo Delphi exception exception_name at ${exception_address}: exception_message
}

$$
$$ In this block, we add conditionals for exception types we want to ignore, or even
$$ specific addresses or other conditions as you need.
$$
$$ Don't include anything below this block, not even comments, because that breaks the 
$$ "gc" command
$$

.block {
    .if ($scmp( "${exception_name}", "EIdConnClosedGracefully") == 0) { exception_cleanup; gc }
    .elsif ($scmp( "${exception_name}", "EAbort") == 0) { exception_cleanup; gc }
        .else {
      .echo
      exception_cleanup
    }
}

Yes, this file has a bit of complexity in it. The WinDbg script interpreter is excessively pernickity. Important things to note are:

  • aliases are not expanded until the block is started (hence usage of the aliases is in a separate block to the definition)
  • you can’t have any additional statements after a block containing a gc or t or similar command, otherwise the script interpreter has a little hissy fit
  • the exception_cleanup alias is pretty essential to avoid leaving aliases about that then get expanded at the wrong time (leading to premature expansion of aliases from a previous exception).
  • the exception_name alias is not 100% reliable. This is because it is referencing a Delphi short string, which is not null-terminated, leading to garbage characters displayed at the end in some cases. Sadface. There is probably a way to work around this but I haven’t found it yet.
  • You’ll see sometimes I use the ${alias_name} expansion and other times I use just alias_name. Use ${alias_name} where non-whitespace characters may be immediately after the alias, as they will be interpreted as part of the alias.
  • The ${/v:alias_name} version prevents expansion of the alias, which is useful for referring to the name of the alias if it is already defined, for example to delete it.

Useful knowledge on WinDbg scripts from other sources:

Update 12 Oct 2020: Fixed up final block to use .elsif and .else to avoid script execution past gc