All posts by Marc Durdin

Creating and using an Internationalized Domain Name in July 2024

We needed to setup a web host for the domain convert.ភាសាខ្មែរ.com for an upcoming project. It should have been simple, but this turned into quite the journey.

A very brief background on IDNs

Some background. Notice the use of Khmer characters: ភាសាខ្មែរ /pʰiesaa kmae/ or “Khmer Language”. ភាសាខ្មែរ.com is a domain I’ve owned for some time, having purchased it when I was learning Khmer. Because this domain names uses letters not found in the English alphabet, it is considered to be an Internationalized Domain Name (or IDN). Now, behind the scenes, ភាសាខ្មែរ is not stored using Khmer characters in a Domain Name System (DNS) record. For reasons of history, domain names are restricted to a handful of Latin script letters, digits, and hyphen, and so the domain must be re-encoded using a system known as ‘punycode‘ (truly!)

The punycode representation of ភាសាខ្មែរ is the rather bamboozling xn--j2e7beiw1lb2hqg.

Now, most of the time, this gory detail will be hidden from you in the user interface of your browser — but if you copy the URL from the address bar and paste it into a text editor, you’ll be presented with the gory details! (The domain punycoder.com allows you to easily play with other scripts and non-English-alphabet names and see how they are punycoded.)

We needed a backend host with Python for the project, which ruled out a lot of free options — particularly since we wanted to bind to this custom domain name. As I had access to an Azure subscription already, I went that way.

So first, I attempted to create the domain quickly in the Azure Portal web front end. I had no trouble creating https://khconvert.azurewebsites.net/ — took just a few seconds. Then I had to bind our domain convert.ភាសាខ្មែរ.com to this azure hostname. But computer said no:

The name convert.xn--j2e7beiw1lb2hqg.com is not valid.

Fine, perhaps the UI needed the domain name as a Unicode string.

Hmm. Blocked again.

Now, often the Azure Portal does not allow you to do things which can be done via their APIs or with their az CLI tool. So I tried again with az.

PS> az webapp config hostname add --hostname convert.xn--j2e7beiw1lb2hqg.com --webapp-name khconvert --resource-group <REDACTED> --verbose
The name convert.xn--j2e7beiw1lb2hqg.com is not valid.
Command ran in 4.228 seconds (init: 0.493, invoke: 3.736)

Okay then … perhaps I have to use the non-punycode version of the hostname? After all, that error in the Azure Portal was purely in the UI — it didn’t even call into the API.

And it worked!

Side track: the old Powershell console did not do well with rendering Khmer in its default settings…

So it looks like those were not question marks? Interestingly, I can paste Khmer characters into the console, but I can’t copy them back out from the command — those letters copy out as question marks. Although as you can see from a screenshot and corresponding text later, I can copy the response and show it correctly. (Admittedly, I was using the old Powershell console, not Terminal, which probably would have none of these problems.)

But it worked, hurrah! We are done!

Well, not quite. We need a TLS certificate for the site. Fortunately, Azure supports creating certificates automatically for web apps.

But not for IDNs.

And not even the CLI saved me this time.

PS> az webapp config ssl create --hostname 'convert.?????????.com' --name khconvert -g <REDACTED> --verbose
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Operation returned an invalid status 'Bad Request'
Content: {"Code":"BadRequest","Message":"Properties.CanonicalName is invalid.  Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed.","Target":null,"Details":[{"Message":"Properties.CanonicalName is invalid.  Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed."},{"Code":"BadRequest"},{"ErrorEntity":{"ExtendedCode":"51021","MessageTemplate":"{0} is invalid.  {1}","Parameters":["Properties.CanonicalName","Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed."],"Code":"BadRequest","Message":"Properties.CanonicalName is invalid.  Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed."}}],"Innererror":null}

At this point, I threw my hands in the air, put the domain behind a Cloudflare proxy and used their SSL certificate, and it just worked.

But it is no wonder there are so few IDNs. The experience is still frightful. The errors are weird. We have a long, long way to go.

(Oh, and do take a look at ភាសាខ្មែរ.com and convert.ភាសាខ្មែរ.com: the problems they are playing with have quite a lot to do with IDNs too!)

dynamic import on node.js with circular dependencies leads to interesting failure modes

So, given these three ES modules below, what output do you expect to get when you run node one.js?

one.js:

import { two } from './two.js';

await two();

export function one() {
  console.log('one');
}

two.js:


export async function two() {
  console.log('two');
  const { three } = await import('./three.js');
  three();
}

three.js:

import { one } from './one.js';
export function three() {
  console.log('three');
  one();
}

Following through the calls, you’d expect to get:

two
three
one

But instead you get just the following output:

two

And node exits with exit code 13, which means “Unfinished Top-Level Awaitawait was used outside of a function in the top-level code, but the passed Promise never resolved.” There are no messages reported to the console, which is very disconcerting!

It turns out that the await import() causes this, because of the circular dependency introduced in module three.js, back to one.js. Now, if there is no async involved, then this circular dependency just works; that is, instead of using import('./three.js') you use import { three } from './three.js':

two
three
one

Not sure yet how to move forward with this. Tested in Node 20.9.0 and 18.14.1. Should I be raising it as a bug? It’s certainly not a documented behaviour that I’ve been able to find (node documentationmdn documentation).

What is marriage?

Written the day after my daughter’s wedding.

 

What is marriage?

Is it a record of property duly
Signed and sealed by stroke of dreary pen
So lawyers and judges can argue
Over who owns what

Or a walled garden within which
A family can grow and thrive
In the sun protected from
The fierce winds of society

Is it companionship walking
The Path of Life, two as one
Helping each other over
The stiles and through
The thickets and alongside
The peaceful streams

Could it be a sanctuary for love
Of mutual fascination and exploration
And closeness and raw nakedness
And risk
Of bodies freely given and
Trust held in fragile vessels
Of humanity

What if it was a reflection?
A reflection of Christ’s love
For his people and
His people’s love of Him
Sacrificial
Joyful
All consuming
Life giving

All this and more
Is found in the
Binding of two souls
What God has brought together
Let not man break asunder

Debugging a Windows Service

This is a set of notes on how to debug a Windows service starting up, mostly for my reference. Building on https://www.sysadmins.lv/retired-msft-blogs/alejacma/how-to-debug-windows-services-with-windbg.aspx with command line steps where possible.

In this example, we’ll be debugging mycool.exe, which has the service name mycoolservice.

Enabling debugging

  1. Find the path to cdb.exe, windbg.exe, gflags.exe. (e.g. C:\Program Files (x86)\Windows Kits\10\Debuggers\x86).

  2. Start an elevated command prompt. Set the service to manual start (and stop it if it is currently running, … duh):

    sc config mycoolservice start=demand
    sc stop mycoolservice
    
  3. Find the short path for cdb.exe (pasting the path from point 1 as appropriate):

    for %A in ("C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe") do @echo %~sA
    
  4. Enable the debug hook for the service, using gflags, replacing the path as necessary:

    C:\PROGRA~2\WI3CF2~1\10\DEBUGG~1\x86\gflags /p /enable mycool.exe /debug "C:\PROGRA~2\WI3CF2~1\10\DEBUGG~1\x86\cdb.exe -server tcp:port=9999"
    
  5. Change the service startup timeout to 1 hour to avoid Windows killing the service on startup:

    reg add HKLM\System\CurrentControlSet\Control /v ServicesPipeTimeout /t REG_DWORD /d 3600000
    
  6. Reboot, start an elevated command prompt again.

  7. Start the service, which will appear to hang:

    sc start mycoolservice
    
  8. Open Windbg, Ctrl+R tcp:server=localhost,port=9999

  9. Go forth and debug.

Disable debugging

  1. Start an elevated command prompt, and enter the following commands:

    C:\PROGRA~2\WI3CF2~1\10\DEBUGG~1\x86\gflags /p /disable mycool.exe
    reg delete HKLM\System\CurrentControlSet\Control /v ServicesPipeTimeout
    
  2. Reset the service startup parameters to your preferred startup type.

  3. Reboot to reset the service control timeout.

location.hash can be re-entrant on Safari Mobile

So Josh and I were observing some really strange behaviour in the Keyman for iOS beta. When typing rapidly on the touch keyboard, we would sometimes get the wrong character emitted. We could not see anything immediately wrong in the code. So Josh added some logging. Then things got really weird: we would get the start of a touch event, then before the touch event handler finished, we’d get logging that indicated another touch event was received.

Whoa. In JavaScript, that shouldn’t be possible, right?

Each message is processed completely before any other message is processed. This offers some nice properties when reasoning about your program, including the fact that whenever a function runs, it cannot be pre-empted and will run entirely before any other code runs (and can modify data the function manipulates). This differs from C, for instance, where if a function runs in a thread, it can be stopped at any point to run some other code in another thread.

— https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop

After we dug in further, we found that it was indeed possible for a new touch event (and perhaps others) to be received when location.hash was set from the JavaScript code, in Safari for iOS.

Here’s a fairly minimal repro. Load this up on your iPhone, and start rapidly touching the Whack div. You’ll probably need to use two fingers repeatedly to trigger the event (I can usually get it to happen with about 50-100 rapid touches). When it happens, you’ll get a log message with a call stack showing how the whackIt() function has apparently called itself!

<!doctype html>
<html>
  <head>
    <meta charset='utf8'>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" /> 
    <title>location.hash re-entrancy</title>
    <style>
      #whack { font-size: 64pt }
      #log, #stack { font-family: courier }
    </style>
  </head>
  <body>
    <div id=whack>Whack</div>
    
    <div id=log></div>
    
    <div id=stack></div>
    
    <script>
    
      var tick = 0, inWhack = false;
      
      function whackIt(e) {
        e.preventDefault();

        tick++;
        
        if(inWhack) {
          stack.innerHTML = 'Re-entrant event: '+(new Error()).stack;
          return false;
        }
        
        var localTick = tick;
        log.innerHTML = tick;

        inWhack = true;

        // Setting location.hash seemingly can cause the 
        // event queue to be polled, resulting in a 
        // re-entrant touch event
        location.hash = '#'+localTick;

        inWhack = false;
        
        if(localTick != tick) {
          // alert('We had a re-entrant event');
        }
        
        return false;
      }
      
      whack.addEventListener('touchstart', whackIt, false);
      whack.addEventListener('touchend', whackIt, false);
      
    </script>
  </body>
</html>

We’ll be reporting this to Apple… Just a heads up.

The Case of the Overly Busy Process

The other day, I was running a routine Process Monitor (Procmon) trace to debug an issue in Keyman, when I noticed something strange: over 50% of the events displayed with the default filter (which excludes a lot of system-level noise and procmon-related feedback) were coming from a single process: services.exe.

You can see in the image below I’ve added services.exe to the filter (Process Name is services.exe), and then the status bar shows 52% of events belonging to it.

Puzzled, I set aside some time to dig a little further (which means I went to bed late one evening). Watching Process Explorer, I could see that services.exe and wmiprvse.exe were between them consuming about 10% of my CPU. This did not seem normal. Nor did it seem to be a good thing for my battery life.

Deciding to examine the trace a little, I filtered out common registry keys and events, such as RegCloseKey, which made it easier to spot a pattern. It became obvious that every 5 seconds, services.exe, with the help of wmiprvse.exe, would enumerate the list of services from the registry, sending about 120,000 events to the Procmon trace in the process. Nearly 80% of the events captured each minute by Procmon were generated by either services.exe or wmiprvse.exe!

Nearly 80% of the events captured each minute by Procmon were generated by either services.exe or wmiprvse.exe!

Given that wmiprvse.exe, the Windows Management Instrumentation (WMI) provider host, was involved, it seemed likely that there was a process issuing WMI queries against the Services provider, such as you can do with PowerShell:

Get-WmiObject Win32_Service | Format-Table Name, DisplayName, State, StartMode, StartName

It was just a matter of figuring out which one.

I started off by trying to dig into WMI logging. I don’t know if you’ve ever dug into that, but it’s huge, complex and somewhat impenetrable. It is likely that with the right knowledge I could have issued a command that gave me a list of queries being issued and who was issuing them. But I have not yet acquired that knowledge, sadly, and late at night my brain did not feel up to the attempt.

It seemed easier to instead to use a process of elimination of processes (yeah, I did that on purpose). I started the CPU monitor in Process Explorer for the services.exe process, which showed lovely 5 second spikes.

Then I started to stop various services, watching to see if the spiking stopped. It didn’t. Once I was down to a handful of critical services (do I really need to run the Firewall service?) I started looking at background user-level processes, such as the icons sitting in the System Notification Area.

And here I hit gold. After shutting down a few, including my own programs, with no noticeable change, I shutdown MySQL Notifier 1.1.7.

All of a sudden, CPU activity dropped to zero on the services.exe process, and the next Procmon trace showed a mere 85 events in a minute for the services.exe and wmiprvse.exe pair.

Success!

I checked the MySQL Notifier forums and saw no discussion of this issue, but I found a closed bug report in the bug database. I’ll have to add my comment to the bug report.

Once again, Procmon comes to the rescue 🙂 I’m looking forward to the increased battery life already!

I know it’s not the most elegant way to debug a problem, but sometimes it is quicker and easier than the alternatives. It’s especially easy to use process of elimination like this late at night, without having to think hard about it. 😉

The case for Keyman

Recent podcast

I was recently privileged to be a guest on Scott Hanselman’s Hanselminutes podcast. It was great to talk about the history of Keyman and how it solved real problems for us 20+ years ago. But afterwards I realised I hadn’t really described what sorts of problems Keyman solves today and why it is still relevant.

So this post is that story.

Note: I’ve put this on my personal blog rather than the Keyman blog because it’s full of my own personal opinions. There’s lots of useful content on the Keyman blog as well!

The elevator pitch – and why I always struggled with it

A common story around venture capitalists and tech startups is that you have to have an amazing elevator pitch, the idea that within about 20-30 seconds, you can sell a problem and its solution to a person just begging to give you great wads of cash. And if you can’t describe the problem in 20 seconds then you probably don’t understand it yourself.

With all due respect, that might be fine for selling package drones or IoT nerf guns, but there are plenty of hard problems out there that cannot be understood in 20 seconds.

I think keyboard input for the world’s writing systems is one of those problems. For many people growing up with limited contact with Asian writing systems, they will look pretty and exotic. But that’s about the extent of it.

So, let’s imagine you and I are in an elevator, and just as you ask me to explain the benefit of Keyman to you, the power goes out and the elevator lurches to a halt. Now you are stuck with me and I can give you the extended elevator pitch!

An illustration from Khmer

Khmer is the national language of Cambodia, a small nation in South East Asia, home of one of the world’s ancient wonders, Angkor Wat.

Photo by Marc Durdin

Here’s a couple of samples of Khmer text. The first is a photo of an inscription from Angkor Wat.

This second image shows two common styles for writing Khmer, as computer fonts:

In a little bit, I’ll dig into how the writing system is used on computer. The script is beautiful, and it is complex, and it has a long and interesting history.

But first, I would like to tell you a story. I recently learned about an effort by a group of Cambodian linguists to revise and prepare a new edition of a Khmer language dictionary. The current “gold standard” Khmer dictionary, compiled by Samdech Porthinhean Chuon Nath, has not been updated since 1967. So a new edition is very much needed.

A number of different people have been involved in typing up entries for the dictionary. In the process, they discovered a problem: some entries which looked identical on screen were actually encoded differently in the computer. This meant that they were unable to consistently search for words, or even sort their dictionary correctly.

How had this happened? This happened because it is possible to type Khmer words in a number of different ways and get identical output on the screen. And I don’t mean by using different input methods, either. I mean with the standard Khmer keyboard layouts supplied with all major operating systems, desktop and mobile.

The Khmer Script

In this post, I won’t get too technical with lots of references to Unicode encodings. Instead, I’m trying to illustrate the issues from the point of view of a user, who shouldn’t need to worry about those details.

How would we go about typing a Khmer word? I’ve picked a famous Khmer word for us to start with:

This is the word ខ្មែរ /kmae/, Khmer, which is the name of the language and the people.

So, with the standard Khmer keyboard, how do you type this word?

Let’s imagine you had this word on a piece of paper and a standard Khmer keyboard in front of you. We can tell that it’s made up of a group of characters and some sort of diacritic marks.

Start with the first shape. It looks like a   symbol and a  diacritic mark. But the two marks belong together and make up a vowel sound, which we can find on the [SHIFT]+[E] key:

Let’s go with that.

Next we have the  character and the diacritic mark. The first bit is easy enough to find, on the [X] key.

But that second bit? Okay, you won’t find that printed on a key cap. Fortunately, those using the Khmer keyboard tend to know their writing system. The character is a form of the  /m/ consonant used when it is the second letter in a consonant cluster. It is called a ជើង /cəəŋ/ letter in Khmer. This means leg or foot, base or support. (Diversion: The Unicode charts call them COENG consonants. I bet you read that as CO-ENG, didn’t you. I know I used to!)

Okay, that’s all very interesting but how does one type these cəəŋ consonants? Well, the Unicode encoding standard added a special control character that’s not part of the writing system, called a KHMER SIGN COENG (U+17D2), just to support these letters. This is typed with a [J] key on the standard keyboard. Now we’re getting somewhere. So we’ll need to type [J] [M] to get that  appearing under the .

Now before we go any further, this magic sign is a problem. It’s a problem because now the user has to know something about the way their text is encoded in order to type it. It’s not a particularly hard thing to learn, but it isn’t obvious. And this is one of the easier things that a native Khmer speaker needs to learn in order to type their own language.

The last letter is easy enough. Here it is on the [R] key.

The full sequence, first try

So the full sequence is: [SHIFT]+[E] [X] [J] [M] [R].

Let’s go ahead and type it.

Ugh. Where did that dotted circle come from?

The dotted circle is a visual hint. It tells us that the vowel  is written before the consonant it is attached to, but that the encoding places it after.

Second try

Okay, so let’s change it to [X] [SHIFT]+[E] [J] [M] [R].

Okay, that looks correct now.

Well, okay, it looks fine … but it’s not. In order to meet the Unicode ordering rules, the vowel has to be placed after the entire consonant cluster, just as if it was spoken.

Third time lucky

Let’s go back and fix that again. Here’s our final sequence. [X] [J] [M] [SHIFT]+[E] [R].

This is the correct sequence.

Now from a linguistic point of view most of this seems fairly logical, and it generally makes sense to users, once they learn it. But I hope this illustrates how some of these things are not obvious. If you haven’t been taught these rules, or if you do accidentally mis-type something, you won’t ever know about it.

It’s not hard to find examples that have a heap of different ways you could type them. For example:

At first glance that doesn’t look a lot more complex, right? However, we’ve found 14 different ways to encode this in Unicode, all of which look correct on Android devices! And if you repeat diacritic marks, then you can expand that to over 35 different combinations, all displayed identically! My colleagues and I wrote a paper on the problem. Can you imagine searching an online dictionary with these kinds of problems?

Other parts of the solution

Now I know some of you are going to be saying that this isn’t a problem that is unique to Khmer. That it can be solved with post-processing and normalisation. And you are correct, this problem extends to many scripts. And some problems can be solved with normalisation. But I think Unicode normalisation is an appropriate solution only with correctly encoded data. Our goal is for data to be encoded correctly at the time it enters the system.

You might argue that Autocorrect-type technologies could help with this, and they certainly can. But again, Autocorrect is really targeted at spelling and word completion, not avoiding encoding issues. (The inconsistently available implementations of Autocorrect don’t help either).

Improved rendering engines could also give helpful visual feedback to users, alerting them to mistyped sequences.

But none of these negate the importance of what Keyman brings to the party.

Keyman: A Comprehensive Input Method Solution

So what does Keyman bring to this party? None of the system supplied Khmer keyboards make any attempt to quality control the input – generally they just output a single character for a single key.

Given we have well defined, structured linguistic and encoding rules, we can leverage those in the input method and ensure that sequences that are invalid according to the specification just don’t make it into the text store.

The power of Keyman

Keyman can automatically reorder and replace input as you type. For our example, if you type a vowel before typing a KHMER COENG SIGN and a consonant, Keyman will transparently and instantly fix the order in your document, and you won’t even know.

You can try a Khmer keyboard which automatically solves these encoding issues online at https://keymanweb.com/?tier=alpha#km,Keyboard_khmer_angkor

That, in a nutshell, is the power of Keyman.

Keyman makes possible amazingly intuitive input for complex writing systems.

Keyman is open source and completely free.

Keyman keyboard layouts are defined with a clear and easy to understand keyboard grammar, so that anyone can write a keyboard layout for their language. The development tools include visual editors, interactive debuggers and automated testing to help you develop sophisticated keyboard layouts.

Keyman runs everywhere. A keyboard layout can deployed to Windows, macOS, iOS, Android, Linux and even as a Javascript web keyboard. The Keyman project has a repository of over 700 existing keyboard layouts, and more are added every week.

Version 10 of Keyman is about to hit Beta. Would you like to be involved?

Oh look, the power is back on. Elevator pitch over. Oh by the way, aren’t writing systems cool?

Working around Delphi’s default grid scrolling behaviour

Delphi’s T*Grid components have an annoying little feature whereby they will scroll the cell into view if you click on a partially visible cell at the right or the bottom of the window. Then, this couples with a timer that causes the scroll to continue as long as the mouse button is held down and the cell it is over is partially visible. This typically means that if a user clicks on a partially visible cell, they end up selecting a cell several rows or columns away from where they intended to click.

In my view, this is a bug that should be fixed in Delphi. I’m not the only person who thinks this. I’ve reported it to Embarcadero at RSP-18542.

In the meantime, here’s a little unit that works around the issue.

{
  Stop scroll on mousedown on bottom row of grid when bottom row
  is a partial cell: have to block both initial scroll and timer-
  based scroll.

  This code is pretty dependent on the implementation in Vcl.Grids.pas,
  so it should be checked if we upgrade to new version of Delphi.
}

{$IFNDEF VER320}
{$MESSAGE ERROR 'Check that this fix is still applicable for a new version of Delphi. Checked against Delphi 10.2' }
{$ENDIF}

unit ScrollFixedStringGrid;

interface

uses
  System.Classes,
  Vcl.Controls,
  Vcl.Grids,
  Winapi.Windows;

type
  TScrollFixedStringGrid = class(TStringGrid)
  private
    TimerStarted: Boolean;
    HackedMousedown: Boolean;
  protected
    procedure MouseDown(Button: TMouseButton; Shift: TShiftState; X: Integer;
      Y: Integer); override;
    procedure MouseMove(Shift: TShiftState; X: Integer; Y: Integer); override;
    function SelectCell(ACol, ARow: Longint): Boolean; override;
  end;

implementation

{ TScrollFixedStringGrid }

procedure TScrollFixedStringGrid.MouseDown(Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  // When we first mouse-down, we know the grid has
  // no active scroll timer
  TimerStarted := False;

  // Call the inherited event, blocking the default MoveCurrent
  // behaviour that scrolls the cell into view
  HackedMouseDown := True;
  try
    inherited;
  finally
    HackedMouseDown := False;
  end;

  // Cancel scrolling timer started by the mousedown event for selecting
  if FGridState = gsSelecting then
    KillTimer(Handle, 1);
end;

procedure TScrollFixedStringGrid.MouseMove(Shift: TShiftState; X, Y: Integer);
begin
  // Start the scroll timer if we are selecting and mouse
  // button is down, on our first movement with mouse down
  if not TimerStarted and (FGridState = gsSelecting) then
  begin
    SetTimer(Handle, 1, 60, nil);
    TimerStarted := True;
  end;
  inherited;
end;


function TScrollFixedStringGrid.SelectCell(ACol, ARow: Longint): Boolean;
begin
  Result := inherited;
  if Result and HackedMousedown then
  begin
    // MoveColRow calls MoveCurrent, which
    // calls SelectCell. If SelectCell returns False, then
    // movement is blocked. But we fake it by re-calling with Show=False
    // to get the behaviour we want
    HackedMouseDown := False;
    try
      MoveColRow(ACol, ARow, True, False);
    finally
      HackedMouseDown := True;
    end;
    Result := False;
  end;
end;

end.

 

Substituted layouts in Text Services Framework

There is a tantalising parameter in the ITfInputProcessorProfileMgr::RegisterProfile function called hklSubstitute.  The description (used to) read “The substitute keyboard layout of this profile.   When this profile is active, the keyboard will use this keyboard layout instead of the default keyboard layout.  This can be NULL if your text service doesn’t change the keyboard layout.”

This functionality also surfaces in ITfInputProcessorProfileSubstituteLayout::GetSubstituteKeyboardLayout and ITfInputProcessorProfiles::SubstituteKeyboardLayout.

From the minimal description in MSDN, this sounds like a feature that we want to use in Keyman Desktop (allowing us to, for instance, use kbdus.dll as a base layout for some Keyman keyboard layouts).  However, we have been unable to get it to work.

  1. It seems the parameter type in the documentation is not quite right.  This parameter is not a HKL that you can obtain from LoadKeyboardLayout or some such.  Instead it needs to be a KLID, a keyboard layout ID, such as 0x00010409 for US Dvorak.
  2. The base layout is limited to one linked to the language the TIP is being installed for.  This means you cannot, for instance, substitute German QWERTZ for a French TIP, even if you wanted to. (And we want to!)
  3. Despite these restrictions, the RegisterProfile function does not check the validity of the parameter and happily tells you that the profile registered OK.
  4. While GetSubstituteKeyboardLayout returns the exact value that you registered, ITfInputProcessorProfileMgr::GetProfile returns 0 for the hklSubstitute member unless you meet the preconditions described above.  Furthermore, hklSubstitute, when available, is not a HKL but a KLID.
  5. Finally, even after all this, it seems that the substitute keyboard layout does not apply.

I received some clarification from Microsoft (edited slightly, thank you Vladimir Smirnov):

In fact, the feature is not that generic as you apparently expect. Its whole purpose was to enable smooth deprecation of IMM32 IMEs and their replacement (substitution) with TSF IMEs (aka TIPs). It doesn’t support substituting an arbitrary plain keyboard layout with another layout or IME (let alone of a different language).

The hklSubstitute is supposed to be a real HKL, but for IMM32 IMEs it actually matches the IME’s KLID. There’s old verification code there which looks for a matching KLID in HKLM for a given hklSubstitute and ignores the latter if there’s no match. That’s why you get 0 from GetProfile for your invalid or unsupported hklSubstitute.

Generic substitution was just never meant to be supported in TSF.

It’s a shame, but we work with what we’ve got. So for Keyman Desktop 9, we wrote a mnemonic layout recompiler that recompiles a Keyman mnemonic layout keyboard, merging it with a given Windows system base layout, at keyboard installation time in Keyman Desktop.

The Day Henry Broke the Internet – A Short Story – Part 2

previous episode

Henry was pale.

We heard an Arts major swearing behind us.  They have dreadful language, Arts majors.  “What the **** is wrong with this iPad?”

Henry bolted over to the girl, and stared at the screen, horrified, numbers pouring down its Retina display and mixed in the middle of it all, the dreaded words “acquiring node”.

“Switch it off, switch it off, switch it off!”  Henry tried to grab the iPad out of her hands and turn it off, which was probably not wise.

“Hey, get off!” she yelled, and grabbed back.

“Switch it off!”

“Okay, okay!”  She pressed the power button and the tablet screen flicked off.

“No!  That’s not enough, it’s gotta be powered down, now!”  Henry cried.

I left him to it, and looked around the café.  There were only a handful of other students and staff there, but they were all using computers or phones or tablets, and they were all, every one of them, staring at their screens in consternation.  I raced around and asked them – ordered them – to switch them off.

Under a minute later, Henry and I met in the middle of the café.  All the computers and phones and tablets were off.  We were safe.  Weren’t we?

No.  Over at the counter, we heard the barista talking to himself.  “What is up with this thing?”

Henry and I looked at each other.  I beat him to the counter.  Scrolling down the cash register screen were, of course, the digits from hell, and as I watched, not one, but four “acquiring node” messages scrolled past, seeming to bump into each other in their hurry to make their way up and off the screen.  I started to realise it was too late.  Mechanically, I reached over and switched off the cash register, as everyone in the café watched.

Henry stumbled out of the coffee bar and into the wide world.  It was early, and not many students were around, but those that were, were fighting their phones and tablets.  “Oh no,” Henry breathed, “It’s made the jump to the campus WiFi.”

I took charge.  “Henry, we’ve got to get to Campus IT and get this switched off before it gets off campus.”

“Oh man, I am in so much trouble…” Henry, at a loss for once in his life, obediently followed me as I ran, leaping up the stairs and diving through the shortcut behind the Engineering department (even in this crisis, my mind briefly flicked over to Jenny – she smiled at me last night!), and arrived, out of breath at the IT building just as an IT tech was making his early way into the building.  I realised I knew him, slightly, through interactions in the PC labs when the lab computers inevitably went down.  But I couldn’t remember his name.

“You’ve got to switch off the campus WiFi, like, right now,” I panted.

“What?  Don’t be silly!  Why would we need to do that?” he prevaricated.

“Just pull out your phone,” I said.  Sure enough, the phone’s screen was filled with numbers.  “It’s a virus,” I explained, “and it’s spreading rapidly!”  I thought that more detailed and accurate explanations could wait, though Henry stared at me, aghast, when I said the word “virus”.

I caught the words “It’s not a virus, it’s a worm…” muttered under his breath.

 

But that was enough to convince the tech.  Gerald, that was his name.  Together, we ran to the network management centre, burst in, and stopped, horrified.  Every screen in the room was filled with numbers, scrolling past faster than we could read.  And of course, “acquiring node”.  As I looked, I saw one computer that seemed to be spending most of its time acquiring nodes, and its screen was filled with the dreaded message, and only the occasional number, scrolling past faster than we could read.  At this rate, the entire University network would be down in minutes.

I just hoped the firewalls would stop Bang from escaping the University.  But I was none too confident.  Knowing Henry, he’d probably designed in a firewall bridging routine.  He always did create perfect code and account for every possibility.  Okay, almost every possibility.  He missed a big one this time, I thought ironically.

Gerald looked, accusingly, at me and Henry.  Henry just slumped against the wall shaking his head and looking as green as a VT420 terminal.  It was up to me then.  “There’s no time to explain,” I explained.  “We’re just going to have to shut down all the computers and disconnect the Uni from the Internet to stop the virus spreading.”

“Whoa, whoa, whoa!  I can’t do that!  That’s way above my pay grade,” Gerald flustered, “I’m calling in my boss.”

He pulled out his Samsung phone, stared at the numbers marching across its screen, and tossed it aside.  “Here’s hoping the PABX is still going,” he muttered.  Picking up a nearby handset, his shoulders slumped with relief when he heard a dial tone.  Immediately, Gerald speed-dialed the head of Campus IT.

“Better try and bring in the professors, I reckon.  This is going to take some big brains to solve,” I suggested.

Gerald glared at me.  “Way ahead of you, kiddo.  Why don’t you go look after your mate?”  He dialled number after number, explained briefly to each the crisis, hung the phone up, and turned to look at us.  “Boss has given me the go-ahead to disconnect the campus network.  Sit here, and don’t touch anything.  I’m going to unplug the fibre now.”

He stalked out of the room.  Henry looked up at me.  He was just as green as before.  “I think I’ve broken the Internet,” he whispered.

“Nah, I don’t reckon it’ll be outside the Uni network,” I reassured him, “we’ll just be expelled, not arrested.”

Moments later, Gerald strode back into the room, followed by his boss, who’d turned up with his tie unfastened and his shirt half buttoned.  Clearly Gerald had been able to impress a sense of urgency upon him.  He too stopped dead in the entrance, staring at the Network Operation Centre’s various computer screens, all filled with rapidly scrolling numbers, except that one that was filled with rapidly scrolling “acquiring node”.  I was really starting to hate that phrase.

“What on earth?  Who is responsible for this?”  He swung around and bore down on us.  “Second year students?  Have you been messing with my network?”

I cringed, but Henry mustered his courage, stood up and said, “Actually, sir, that’s not precisely correct.  Through an unfortunate series of events, and in fact my completely accidental overlooking of a key …”

“I don’t have time for this,” cried Head of IT, “What I want to know is how are we going to fix this?  Have you quarantined the network?”  This last to Gerald.  Yeah, okay, Henry did sound a bit weaselly that time.

“Yes, fibre links are disconnected, WiFi has been shut down, it should be contained, sir,” responded Gerald immediately.

“Okay, so how do we get this damned virus off my network?”

Henry felt obliged to pipe up again.  “Technically, sir, it’s not a virus, because it isn’t attaching to other executable files, but rather is more like a worm, a bit like the Morris worm from 1988 …”  I remembered the Morris worm, unleashed on a young and unsuspecting Internet by a less-than-cautious graduate student.  Thinking back to the Computer History class from the previous year, I could see that Bang bore some distinct similarities.  I just hoped Bang wouldn’t infect 10% of the world’s computers!  More worrying to me was the fact that the fact that the creator of the worm, Robert Morris, was prosecuted for releasing the worm.  But there’d be time to stress over that later.  Henry was continuing, “It continues on by piggybacking on the TCP handshake when connection is established.”  Which made no sense to me.  I should probably remind you that I am going on my own memory here, and I’ve probably got the technical details all wrong.  It was complicated, anyway.

As Henry spoke, the NOC telephone rang.  Gerald grabbed it.  It was the University Vice Chancellor, calling because not only could he not use his mobile phone, or his desktop computer, but his car had stalled, on the street, on the way into the University, with the same horrid numbers scrolling up the multifunction display in the car.  In fact, he was calling on his wife’s ancient Nokia phone which she refused to give up and which to him was looking a more attractive device by the second.  Henry and I looked at each other.  If Bang had made it onto his car computer, it was definitely out in the wild.

From there, a tidal wave of events took us, and flung us headlong across the pages of history, all in one fateful morning.  A television was wheeled into the NOC, and we watched a harried presenter in a makeshift studio (their normal studio computers had crashed, of course) report one network after another succumbing to Bang.  Henry watched, silently.  It was clear that he’d recovered a bit, because he was no longer green, more of a 1984 Mac white.  He was also clearly thinking.  I watched, also silently, as events unfolded, like a train wreck, too horrible to look away from.  Airports were shut down.  The stock market was offline.  While some telephones were still working, and most power networks were still up, the companies responsible for those systems were unable to perform even the most basic tasks.  All around the world, computers were busy calculating Henry’s fantastic factorial.

Eventually the Vice Chancellor made his way onto campus, on a hastily borrowed bicycle.  He clearly had no idea who I was, and just as clearly knew very well who Henry was.  Funny that.  He convened a crisis meeting, and invited both Henry, as author of the crisis, and I, as honorary co-author apparently (an honour I was anxious to forgo), to sit in and listen.  Professor Eisenfaktor was there, as the guru in the department, and frowning at Henry he started with a simple question: “Just how does this worm” (he accepted the term without blinking) “replicate?”

For the third time, Henry started to explain.  “It piggybacks off the SYN packet during the handshake, and … “ and my eyes glazed over, again.  To be honest, I wasn’t sure I wanted to know how it replicated.  The less I knew, the better.  “… and the program uses a distributed parallel multiplication algorithm to calculate the next factor, based on the work by Bunimov in 2003.”  Henry paused and looked at Professor Eisenfaktor.

“Yes, yes, I know who Bunimov is,” growled Eisenfaktor.  “Continue.”

“And anyway, the end of the story is because the program is self-healing, I managed to overlook the fact that it has no way to stop it short of turning off the computer and disconnecting from the network so it cannot be reinstated,” finished Henry.

Professor Eisenfaktor surprised me then by praising Henry.  “That is remarkable,” he enthused, “a self-replicating, self-healing, resilient, cross-platform, distributed computing worm, all written in one evening?  How many hard problems did you solve without realising it?  You’ve got a bright future ahead of you!  Assuming you survive this, of course,” he continued, somewhat less encouragingly.  “But now, you have created this crisis.  To think that no one had spotted that flaw in TCP.  So obvious now, is it not?  Ach!”  (He tended to lapse into a more Germanic turn of phrase as he got more excited – this kept us amused during lectures.)  “As I say, you have created this crisis, and now this crisis you shall solve.”

“Ah, I am not sure that is all that wise, Professor,” the VC cut in, “this young man, while meaning well, no doubt lacks experience and I would feel more comfortable if he took merely, say, an advisory role, in any action or steps which we as a University corporate may take in addressing this, which I may say without exaggeration, existential crisis.  I have no doubt that even at this minute, the source of this virus, ah, worm, is being hunted down, and I am sure that the authorities will be anxious to speak to this young man in very short order.”

“Precisely,” retorted Professor Eisenfaktor, “and I think you underestimate young Henry here.  How long has that flaw in the protocols been there?  Decades!  And in one evening, Henry, purely in the interests of furthering his, if I may say so, extraordinary algorithm project, has not only identified this flaw, but perfectly and precisely utilised it in the creation of this wondrously beautiful replicating worm, what was it called?  Bang?  What a clever pun!”  The Professor was obviously getting a bit excited, and even I was starting to get the idea that what Henry had created was, indeed, extraordinary, both in its technical beauty, and in the way it so effortlessly overwhelmed the entire planet’s computing infrastructure.

“No doubt Henry has a suggestion?” the Professor finished, looking at Henry quizzically.

“Well, actually,” started Henry, uncharacteristically but perhaps unsurprisingly diffidently, “I do.  I believe I can create a program that follows Bang, in its footsteps as it were, and both kills Bang on each system it encounters, and closes the hole in the protocol at the same time.”  He paused.

“Go on,” the Vice Chancellor prompted.

“Um, there’s not much more to say, really,” said Henry, “except that I thought we could call the program Crash.  Crash Bang.”

“Right.”  The VC did not seem amused.  “Well, why don’t you and um, your friend, go back to the library and start doing whatever it is you need to do.  But don’t you put that computer online until after you have cleared it with the Professor.”

Henry and I stood.  I glanced at the television.  The President of the United States was now on screen, talking about the crisis, although we could not hear him as the sound was muted.  I gulped.  Well, if Henry hadn’t been noticed when was growing up, he’d certainly be noticed now.  I’d probably end up notorious as well.  I wondered if we’d both go to jail, or worse.  The entire world was grinding to a halt.  Banks had closed their doors.  Supermarkets were not admitting customers.  Leaders were telling their populace to stay at home and prepare for disaster.  And yet, despite this, some phones were still working, and we still had electricity.  I didn’t know what would happen if the power went out.  I wasn’t sure Henry had enough juice to finish his Crash program.

I’m sure you know the rest.  Henry went on to write Crash, even faster than he’d written Bang, and the Vice Chancellor himself held off the police until he’d finished.  Solemnly, Henry turned on his WiFi, Gerald switched the Campus WiFi back on, and we watched, entranced, as one after another, the numbers flicked off screens, replaced briefly by a smiling face – Henry thought it appropriate – before restoring the computer, tablet or phone to its usual duties.  One by one, networks around the world restored themselves.  And in the process, the security hole in the protocol was patched, automatically.  And then the Federal Police asked Henry, and I (poor innocent I) to please come with them for a little chat.

All’s well that ends well.  Henry is now pretty famous, and some of it has even rubbed off on me, though I don’t think I deserve it.  Although many clamoured for a prison sentence, Henry escaped conviction, because, as his lawyer said, it was a simple bug and anyone could see he didn’t have a malicious bone in his body.  But Henry and I were both banned from the Uni’s labs for the rest of the semester, which I thought was quite unfair, given I hadn’t done anything.

Henry was mostly disappointed that in the rush he didn’t find out what factorial result Bang had reached before being shut down.  But then as he said, even if he knew the answer, it’s not like we could have recorded it anywhere for posterity.  Not without using all the computers in all the world.