All posts by Marc Durdin

dynamic import on node.js with circular dependencies leads to interesting failure modes

So, given these three ES modules below, what output do you expect to get when you run node one.js?

one.js:

import { two } from './two.js';

await two();

export function one() {
  console.log('one');
}

two.js:


export async function two() {
  console.log('two');
  const { three } = await import('./three.js');
  three();
}

three.js:

import { one } from './one.js';
export function three() {
  console.log('three');
  one();
}

Following through the calls, you’d expect to get:

two
three
one

But instead you get just the following output:

two

And node exits with exit code 13, which means “Unfinished Top-Level Awaitawait was used outside of a function in the top-level code, but the passed Promise never resolved.” There are no messages reported to the console, which is very disconcerting!

It turns out that the await import() causes this, because of the circular dependency introduced in module three.js, back to one.js. Now, if there is no async involved, then this circular dependency just works; that is, instead of using import('./three.js') you use import { three } from './three.js':

two
three
one

Not sure yet how to move forward with this. Tested in Node 20.9.0 and 18.14.1. Should I be raising it as a bug? It’s certainly not a documented behaviour that I’ve been able to find (node documentationmdn documentation).

What is marriage?

Written the day after my daughter’s wedding.

 

What is marriage?

Is it a record of property duly
Signed and sealed by stroke of dreary pen
So lawyers and judges can argue
Over who owns what

Or a walled garden within which
A family can grow and thrive
In the sun protected from
The fierce winds of society

Is it companionship walking
The Path of Life, two as one
Helping each other over
The stiles and through
The thickets and alongside
The peaceful streams

Could it be a sanctuary for love
Of mutual fascination and exploration
And closeness and raw nakedness
And risk
Of bodies freely given and
Trust held in fragile vessels
Of humanity

What if it was a reflection?
A reflection of Christ’s love
For his people and
His people’s love of Him
Sacrificial
Joyful
All consuming
Life giving

All this and more
Is found in the
Binding of two souls
What God has brought together
Let not man break asunder

Debugging a Windows Service

This is a set of notes on how to debug a Windows service starting up, mostly for my reference. Building on https://www.sysadmins.lv/retired-msft-blogs/alejacma/how-to-debug-windows-services-with-windbg.aspx with command line steps where possible.

In this example, we’ll be debugging mycool.exe, which has the service name mycoolservice.

Enabling debugging

  1. Find the path to cdb.exe, windbg.exe, gflags.exe. (e.g. C:\Program Files (x86)\Windows Kits\10\Debuggers\x86).

  2. Start an elevated command prompt. Set the service to manual start (and stop it if it is currently running, … duh):

    sc config mycoolservice start=demand
    sc stop mycoolservice
    
  3. Find the short path for cdb.exe (pasting the path from point 1 as appropriate):

    for %A in ("C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe") do @echo %~sA
    
  4. Enable the debug hook for the service, using gflags, replacing the path as necessary:

    C:\PROGRA~2\WI3CF2~1\10\DEBUGG~1\x86\gflags /p /enable mycool.exe /debug "C:\PROGRA~2\WI3CF2~1\10\DEBUGG~1\x86\cdb.exe -server tcp:port=9999"
    
  5. Change the service startup timeout to 1 hour to avoid Windows killing the service on startup:

    reg add HKLM\System\CurrentControlSet\Control /v ServicesPipeTimeout /t REG_DWORD /d 3600000
    
  6. Reboot, start an elevated command prompt again.

  7. Start the service, which will appear to hang:

    sc start mycoolservice
    
  8. Open Windbg, Ctrl+R tcp:server=localhost,port=9999

  9. Go forth and debug.

Disable debugging

  1. Start an elevated command prompt, and enter the following commands:

    C:\PROGRA~2\WI3CF2~1\10\DEBUGG~1\x86\gflags /p /disable mycool.exe
    reg delete HKLM\System\CurrentControlSet\Control /v ServicesPipeTimeout
    
  2. Reset the service startup parameters to your preferred startup type.

  3. Reboot to reset the service control timeout.

location.hash can be re-entrant on Safari Mobile

So Josh and I were observing some really strange behaviour in the Keyman for iOS beta. When typing rapidly on the touch keyboard, we would sometimes get the wrong character emitted. We could not see anything immediately wrong in the code. So Josh added some logging. Then things got really weird: we would get the start of a touch event, then before the touch event handler finished, we’d get logging that indicated another touch event was received.

Whoa. In JavaScript, that shouldn’t be possible, right?

Each message is processed completely before any other message is processed. This offers some nice properties when reasoning about your program, including the fact that whenever a function runs, it cannot be pre-empted and will run entirely before any other code runs (and can modify data the function manipulates). This differs from C, for instance, where if a function runs in a thread, it can be stopped at any point to run some other code in another thread.

— https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop

After we dug in further, we found that it was indeed possible for a new touch event (and perhaps others) to be received when location.hash was set from the JavaScript code, in Safari for iOS.

Here’s a fairly minimal repro. Load this up on your iPhone, and start rapidly touching the Whack div. You’ll probably need to use two fingers repeatedly to trigger the event (I can usually get it to happen with about 50-100 rapid touches). When it happens, you’ll get a log message with a call stack showing how the whackIt() function has apparently called itself!

<!doctype html>
<html>
  <head>
    <meta charset='utf8'>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" /> 
    <title>location.hash re-entrancy</title>
    <style>
      #whack { font-size: 64pt }
      #log, #stack { font-family: courier }
    </style>
  </head>
  <body>
    <div id=whack>Whack</div>
    
    <div id=log></div>
    
    <div id=stack></div>
    
    <script>
    
      var tick = 0, inWhack = false;
      
      function whackIt(e) {
        e.preventDefault();

        tick++;
        
        if(inWhack) {
          stack.innerHTML = 'Re-entrant event: '+(new Error()).stack;
          return false;
        }
        
        var localTick = tick;
        log.innerHTML = tick;

        inWhack = true;

        // Setting location.hash seemingly can cause the 
        // event queue to be polled, resulting in a 
        // re-entrant touch event
        location.hash = '#'+localTick;

        inWhack = false;
        
        if(localTick != tick) {
          // alert('We had a re-entrant event');
        }
        
        return false;
      }
      
      whack.addEventListener('touchstart', whackIt, false);
      whack.addEventListener('touchend', whackIt, false);
      
    </script>
  </body>
</html>

We’ll be reporting this to Apple… Just a heads up.

The Case of the Overly Busy Process

The other day, I was running a routine Process Monitor (Procmon) trace to debug an issue in Keyman, when I noticed something strange: over 50% of the events displayed with the default filter (which excludes a lot of system-level noise and procmon-related feedback) were coming from a single process: services.exe.

You can see in the image below I’ve added services.exe to the filter (Process Name is services.exe), and then the status bar shows 52% of events belonging to it.

Puzzled, I set aside some time to dig a little further (which means I went to bed late one evening). Watching Process Explorer, I could see that services.exe and wmiprvse.exe were between them consuming about 10% of my CPU. This did not seem normal. Nor did it seem to be a good thing for my battery life.

Deciding to examine the trace a little, I filtered out common registry keys and events, such as RegCloseKey, which made it easier to spot a pattern. It became obvious that every 5 seconds, services.exe, with the help of wmiprvse.exe, would enumerate the list of services from the registry, sending about 120,000 events to the Procmon trace in the process. Nearly 80% of the events captured each minute by Procmon were generated by either services.exe or wmiprvse.exe!

Nearly 80% of the events captured each minute by Procmon were generated by either services.exe or wmiprvse.exe!

Given that wmiprvse.exe, the Windows Management Instrumentation (WMI) provider host, was involved, it seemed likely that there was a process issuing WMI queries against the Services provider, such as you can do with PowerShell:

Get-WmiObject Win32_Service | Format-Table Name, DisplayName, State, StartMode, StartName

It was just a matter of figuring out which one.

I started off by trying to dig into WMI logging. I don’t know if you’ve ever dug into that, but it’s huge, complex and somewhat impenetrable. It is likely that with the right knowledge I could have issued a command that gave me a list of queries being issued and who was issuing them. But I have not yet acquired that knowledge, sadly, and late at night my brain did not feel up to the attempt.

It seemed easier to instead to use a process of elimination of processes (yeah, I did that on purpose). I started the CPU monitor in Process Explorer for the services.exe process, which showed lovely 5 second spikes.

Then I started to stop various services, watching to see if the spiking stopped. It didn’t. Once I was down to a handful of critical services (do I really need to run the Firewall service?) I started looking at background user-level processes, such as the icons sitting in the System Notification Area.

And here I hit gold. After shutting down a few, including my own programs, with no noticeable change, I shutdown MySQL Notifier 1.1.7.

All of a sudden, CPU activity dropped to zero on the services.exe process, and the next Procmon trace showed a mere 85 events in a minute for the services.exe and wmiprvse.exe pair.

Success!

I checked the MySQL Notifier forums and saw no discussion of this issue, but I found a closed bug report in the bug database. I’ll have to add my comment to the bug report.

Once again, Procmon comes to the rescue 🙂 I’m looking forward to the increased battery life already!

I know it’s not the most elegant way to debug a problem, but sometimes it is quicker and easier than the alternatives. It’s especially easy to use process of elimination like this late at night, without having to think hard about it. 😉

The case for Keyman

Recent podcast

I was recently privileged to be a guest on Scott Hanselman’s Hanselminutes podcast. It was great to talk about the history of Keyman and how it solved real problems for us 20+ years ago. But afterwards I realised I hadn’t really described what sorts of problems Keyman solves today and why it is still relevant.

So this post is that story.

Note: I’ve put this on my personal blog rather than the Keyman blog because it’s full of my own personal opinions. There’s lots of useful content on the Keyman blog as well!

The elevator pitch – and why I always struggled with it

A common story around venture capitalists and tech startups is that you have to have an amazing elevator pitch, the idea that within about 20-30 seconds, you can sell a problem and its solution to a person just begging to give you great wads of cash. And if you can’t describe the problem in 20 seconds then you probably don’t understand it yourself.

With all due respect, that might be fine for selling package drones or IoT nerf guns, but there are plenty of hard problems out there that cannot be understood in 20 seconds.

I think keyboard input for the world’s writing systems is one of those problems. For many people growing up with limited contact with Asian writing systems, they will look pretty and exotic. But that’s about the extent of it.

So, let’s imagine you and I are in an elevator, and just as you ask me to explain the benefit of Keyman to you, the power goes out and the elevator lurches to a halt. Now you are stuck with me and I can give you the extended elevator pitch!

An illustration from Khmer

Khmer is the national language of Cambodia, a small nation in South East Asia, home of one of the world’s ancient wonders, Angkor Wat.

Photo by Marc Durdin

Here’s a couple of samples of Khmer text. The first is a photo of an inscription from Angkor Wat.

This second image shows two common styles for writing Khmer, as computer fonts:

In a little bit, I’ll dig into how the writing system is used on computer. The script is beautiful, and it is complex, and it has a long and interesting history.

But first, I would like to tell you a story. I recently learned about an effort by a group of Cambodian linguists to revise and prepare a new edition of a Khmer language dictionary. The current “gold standard” Khmer dictionary, compiled by Samdech Porthinhean Chuon Nath, has not been updated since 1967. So a new edition is very much needed.

A number of different people have been involved in typing up entries for the dictionary. In the process, they discovered a problem: some entries which looked identical on screen were actually encoded differently in the computer. This meant that they were unable to consistently search for words, or even sort their dictionary correctly.

How had this happened? This happened because it is possible to type Khmer words in a number of different ways and get identical output on the screen. And I don’t mean by using different input methods, either. I mean with the standard Khmer keyboard layouts supplied with all major operating systems, desktop and mobile.

The Khmer Script

In this post, I won’t get too technical with lots of references to Unicode encodings. Instead, I’m trying to illustrate the issues from the point of view of a user, who shouldn’t need to worry about those details.

How would we go about typing a Khmer word? I’ve picked a famous Khmer word for us to start with:

This is the word ខ្មែរ /kmae/, Khmer, which is the name of the language and the people.

So, with the standard Khmer keyboard, how do you type this word?

Let’s imagine you had this word on a piece of paper and a standard Khmer keyboard in front of you. We can tell that it’s made up of a group of characters and some sort of diacritic marks.

Start with the first shape. It looks like a   symbol and a  diacritic mark. But the two marks belong together and make up a vowel sound, which we can find on the [SHIFT]+[E] key:

Let’s go with that.

Next we have the  character and the diacritic mark. The first bit is easy enough to find, on the [X] key.

But that second bit? Okay, you won’t find that printed on a key cap. Fortunately, those using the Khmer keyboard tend to know their writing system. The character is a form of the  /m/ consonant used when it is the second letter in a consonant cluster. It is called a ជើង /cəəŋ/ letter in Khmer. This means leg or foot, base or support. (Diversion: The Unicode charts call them COENG consonants. I bet you read that as CO-ENG, didn’t you. I know I used to!)

Okay, that’s all very interesting but how does one type these cəəŋ consonants? Well, the Unicode encoding standard added a special control character that’s not part of the writing system, called a KHMER SIGN COENG (U+17D2), just to support these letters. This is typed with a [J] key on the standard keyboard. Now we’re getting somewhere. So we’ll need to type [J] [M] to get that  appearing under the .

Now before we go any further, this magic sign is a problem. It’s a problem because now the user has to know something about the way their text is encoded in order to type it. It’s not a particularly hard thing to learn, but it isn’t obvious. And this is one of the easier things that a native Khmer speaker needs to learn in order to type their own language.

The last letter is easy enough. Here it is on the [R] key.

The full sequence, first try

So the full sequence is: [SHIFT]+[E] [X] [J] [M] [R].

Let’s go ahead and type it.

Ugh. Where did that dotted circle come from?

The dotted circle is a visual hint. It tells us that the vowel  is written before the consonant it is attached to, but that the encoding places it after.

Second try

Okay, so let’s change it to [X] [SHIFT]+[E] [J] [M] [R].

Okay, that looks correct now.

Well, okay, it looks fine … but it’s not. In order to meet the Unicode ordering rules, the vowel has to be placed after the entire consonant cluster, just as if it was spoken.

Third time lucky

Let’s go back and fix that again. Here’s our final sequence. [X] [J] [M] [SHIFT]+[E] [R].

This is the correct sequence.

Now from a linguistic point of view most of this seems fairly logical, and it generally makes sense to users, once they learn it. But I hope this illustrates how some of these things are not obvious. If you haven’t been taught these rules, or if you do accidentally mis-type something, you won’t ever know about it.

It’s not hard to find examples that have a heap of different ways you could type them. For example:

At first glance that doesn’t look a lot more complex, right? However, we’ve found 14 different ways to encode this in Unicode, all of which look correct on Android devices! And if you repeat diacritic marks, then you can expand that to over 35 different combinations, all displayed identically! My colleagues and I wrote a paper on the problem. Can you imagine searching an online dictionary with these kinds of problems?

Other parts of the solution

Now I know some of you are going to be saying that this isn’t a problem that is unique to Khmer. That it can be solved with post-processing and normalisation. And you are correct, this problem extends to many scripts. And some problems can be solved with normalisation. But I think Unicode normalisation is an appropriate solution only with correctly encoded data. Our goal is for data to be encoded correctly at the time it enters the system.

You might argue that Autocorrect-type technologies could help with this, and they certainly can. But again, Autocorrect is really targeted at spelling and word completion, not avoiding encoding issues. (The inconsistently available implementations of Autocorrect don’t help either).

Improved rendering engines could also give helpful visual feedback to users, alerting them to mistyped sequences.

But none of these negate the importance of what Keyman brings to the party.

Keyman: A Comprehensive Input Method Solution

So what does Keyman bring to this party? None of the system supplied Khmer keyboards make any attempt to quality control the input – generally they just output a single character for a single key.

Given we have well defined, structured linguistic and encoding rules, we can leverage those in the input method and ensure that sequences that are invalid according to the specification just don’t make it into the text store.

The power of Keyman

Keyman can automatically reorder and replace input as you type. For our example, if you type a vowel before typing a KHMER COENG SIGN and a consonant, Keyman will transparently and instantly fix the order in your document, and you won’t even know.

You can try a Khmer keyboard which automatically solves these encoding issues online at https://keymanweb.com/?tier=alpha#km,Keyboard_khmer_angkor

That, in a nutshell, is the power of Keyman.

Keyman makes possible amazingly intuitive input for complex writing systems.

Keyman is open source and completely free.

Keyman keyboard layouts are defined with a clear and easy to understand keyboard grammar, so that anyone can write a keyboard layout for their language. The development tools include visual editors, interactive debuggers and automated testing to help you develop sophisticated keyboard layouts.

Keyman runs everywhere. A keyboard layout can deployed to Windows, macOS, iOS, Android, Linux and even as a Javascript web keyboard. The Keyman project has a repository of over 700 existing keyboard layouts, and more are added every week.

Version 10 of Keyman is about to hit Beta. Would you like to be involved?

Oh look, the power is back on. Elevator pitch over. Oh by the way, aren’t writing systems cool?

Working around Delphi’s default grid scrolling behaviour

Delphi’s T*Grid components have an annoying little feature whereby they will scroll the cell into view if you click on a partially visible cell at the right or the bottom of the window. Then, this couples with a timer that causes the scroll to continue as long as the mouse button is held down and the cell it is over is partially visible. This typically means that if a user clicks on a partially visible cell, they end up selecting a cell several rows or columns away from where they intended to click.

In my view, this is a bug that should be fixed in Delphi. I’m not the only person who thinks this. I’ve reported it to Embarcadero at RSP-18542.

In the meantime, here’s a little unit that works around the issue.

{
  Stop scroll on mousedown on bottom row of grid when bottom row
  is a partial cell: have to block both initial scroll and timer-
  based scroll.

  This code is pretty dependent on the implementation in Vcl.Grids.pas,
  so it should be checked if we upgrade to new version of Delphi.
}

{$IFNDEF VER320}
{$MESSAGE ERROR 'Check that this fix is still applicable for a new version of Delphi. Checked against Delphi 10.2' }
{$ENDIF}

unit ScrollFixedStringGrid;

interface

uses
  System.Classes,
  Vcl.Controls,
  Vcl.Grids,
  Winapi.Windows;

type
  TScrollFixedStringGrid = class(TStringGrid)
  private
    TimerStarted: Boolean;
    HackedMousedown: Boolean;
  protected
    procedure MouseDown(Button: TMouseButton; Shift: TShiftState; X: Integer;
      Y: Integer); override;
    procedure MouseMove(Shift: TShiftState; X: Integer; Y: Integer); override;
    function SelectCell(ACol, ARow: Longint): Boolean; override;
  end;

implementation

{ TScrollFixedStringGrid }

procedure TScrollFixedStringGrid.MouseDown(Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  // When we first mouse-down, we know the grid has
  // no active scroll timer
  TimerStarted := False;

  // Call the inherited event, blocking the default MoveCurrent
  // behaviour that scrolls the cell into view
  HackedMouseDown := True;
  try
    inherited;
  finally
    HackedMouseDown := False;
  end;

  // Cancel scrolling timer started by the mousedown event for selecting
  if FGridState = gsSelecting then
    KillTimer(Handle, 1);
end;

procedure TScrollFixedStringGrid.MouseMove(Shift: TShiftState; X, Y: Integer);
begin
  // Start the scroll timer if we are selecting and mouse
  // button is down, on our first movement with mouse down
  if not TimerStarted and (FGridState = gsSelecting) then
  begin
    SetTimer(Handle, 1, 60, nil);
    TimerStarted := True;
  end;
  inherited;
end;


function TScrollFixedStringGrid.SelectCell(ACol, ARow: Longint): Boolean;
begin
  Result := inherited;
  if Result and HackedMousedown then
  begin
    // MoveColRow calls MoveCurrent, which
    // calls SelectCell. If SelectCell returns False, then
    // movement is blocked. But we fake it by re-calling with Show=False
    // to get the behaviour we want
    HackedMouseDown := False;
    try
      MoveColRow(ACol, ARow, True, False);
    finally
      HackedMouseDown := True;
    end;
    Result := False;
  end;
end;

end.

 

Substituted layouts in Text Services Framework

There is a tantalising parameter in the ITfInputProcessorProfileMgr::RegisterProfile function called hklSubstitute.  The description (used to) read “The substitute keyboard layout of this profile.   When this profile is active, the keyboard will use this keyboard layout instead of the default keyboard layout.  This can be NULL if your text service doesn’t change the keyboard layout.”

This functionality also surfaces in ITfInputProcessorProfileSubstituteLayout::GetSubstituteKeyboardLayout and ITfInputProcessorProfiles::SubstituteKeyboardLayout.

From the minimal description in MSDN, this sounds like a feature that we want to use in Keyman Desktop (allowing us to, for instance, use kbdus.dll as a base layout for some Keyman keyboard layouts).  However, we have been unable to get it to work.

  1. It seems the parameter type in the documentation is not quite right.  This parameter is not a HKL that you can obtain from LoadKeyboardLayout or some such.  Instead it needs to be a KLID, a keyboard layout ID, such as 0x00010409 for US Dvorak.
  2. The base layout is limited to one linked to the language the TIP is being installed for.  This means you cannot, for instance, substitute German QWERTZ for a French TIP, even if you wanted to. (And we want to!)
  3. Despite these restrictions, the RegisterProfile function does not check the validity of the parameter and happily tells you that the profile registered OK.
  4. While GetSubstituteKeyboardLayout returns the exact value that you registered, ITfInputProcessorProfileMgr::GetProfile returns 0 for the hklSubstitute member unless you meet the preconditions described above.  Furthermore, hklSubstitute, when available, is not a HKL but a KLID.
  5. Finally, even after all this, it seems that the substitute keyboard layout does not apply.

I received some clarification from Microsoft (edited slightly, thank you Vladimir Smirnov):

In fact, the feature is not that generic as you apparently expect. Its whole purpose was to enable smooth deprecation of IMM32 IMEs and their replacement (substitution) with TSF IMEs (aka TIPs). It doesn’t support substituting an arbitrary plain keyboard layout with another layout or IME (let alone of a different language).

The hklSubstitute is supposed to be a real HKL, but for IMM32 IMEs it actually matches the IME’s KLID. There’s old verification code there which looks for a matching KLID in HKLM for a given hklSubstitute and ignores the latter if there’s no match. That’s why you get 0 from GetProfile for your invalid or unsupported hklSubstitute.

Generic substitution was just never meant to be supported in TSF.

It’s a shame, but we work with what we’ve got. So for Keyman Desktop 9, we wrote a mnemonic layout recompiler that recompiles a Keyman mnemonic layout keyboard, merging it with a given Windows system base layout, at keyboard installation time in Keyman Desktop.

The Day Henry Broke the Internet – A Short Story – Part 2

previous episode

Henry was pale.

We heard an Arts major swearing behind us.  They have dreadful language, Arts majors.  “What the **** is wrong with this iPad?”

Henry bolted over to the girl, and stared at the screen, horrified, numbers pouring down its Retina display and mixed in the middle of it all, the dreaded words “acquiring node”.

“Switch it off, switch it off, switch it off!”  Henry tried to grab the iPad out of her hands and turn it off, which was probably not wise.

“Hey, get off!” she yelled, and grabbed back.

“Switch it off!”

“Okay, okay!”  She pressed the power button and the tablet screen flicked off.

“No!  That’s not enough, it’s gotta be powered down, now!”  Henry cried.

I left him to it, and looked around the café.  There were only a handful of other students and staff there, but they were all using computers or phones or tablets, and they were all, every one of them, staring at their screens in consternation.  I raced around and asked them – ordered them – to switch them off.

Under a minute later, Henry and I met in the middle of the café.  All the computers and phones and tablets were off.  We were safe.  Weren’t we?

No.  Over at the counter, we heard the barista talking to himself.  “What is up with this thing?”

Henry and I looked at each other.  I beat him to the counter.  Scrolling down the cash register screen were, of course, the digits from hell, and as I watched, not one, but four “acquiring node” messages scrolled past, seeming to bump into each other in their hurry to make their way up and off the screen.  I started to realise it was too late.  Mechanically, I reached over and switched off the cash register, as everyone in the café watched.

Henry stumbled out of the coffee bar and into the wide world.  It was early, and not many students were around, but those that were, were fighting their phones and tablets.  “Oh no,” Henry breathed, “It’s made the jump to the campus WiFi.”

I took charge.  “Henry, we’ve got to get to Campus IT and get this switched off before it gets off campus.”

“Oh man, I am in so much trouble…” Henry, at a loss for once in his life, obediently followed me as I ran, leaping up the stairs and diving through the shortcut behind the Engineering department (even in this crisis, my mind briefly flicked over to Jenny – she smiled at me last night!), and arrived, out of breath at the IT building just as an IT tech was making his early way into the building.  I realised I knew him, slightly, through interactions in the PC labs when the lab computers inevitably went down.  But I couldn’t remember his name.

“You’ve got to switch off the campus WiFi, like, right now,” I panted.

“What?  Don’t be silly!  Why would we need to do that?” he prevaricated.

“Just pull out your phone,” I said.  Sure enough, the phone’s screen was filled with numbers.  “It’s a virus,” I explained, “and it’s spreading rapidly!”  I thought that more detailed and accurate explanations could wait, though Henry stared at me, aghast, when I said the word “virus”.

I caught the words “It’s not a virus, it’s a worm…” muttered under his breath.

 

But that was enough to convince the tech.  Gerald, that was his name.  Together, we ran to the network management centre, burst in, and stopped, horrified.  Every screen in the room was filled with numbers, scrolling past faster than we could read.  And of course, “acquiring node”.  As I looked, I saw one computer that seemed to be spending most of its time acquiring nodes, and its screen was filled with the dreaded message, and only the occasional number, scrolling past faster than we could read.  At this rate, the entire University network would be down in minutes.

I just hoped the firewalls would stop Bang from escaping the University.  But I was none too confident.  Knowing Henry, he’d probably designed in a firewall bridging routine.  He always did create perfect code and account for every possibility.  Okay, almost every possibility.  He missed a big one this time, I thought ironically.

Gerald looked, accusingly, at me and Henry.  Henry just slumped against the wall shaking his head and looking as green as a VT420 terminal.  It was up to me then.  “There’s no time to explain,” I explained.  “We’re just going to have to shut down all the computers and disconnect the Uni from the Internet to stop the virus spreading.”

“Whoa, whoa, whoa!  I can’t do that!  That’s way above my pay grade,” Gerald flustered, “I’m calling in my boss.”

He pulled out his Samsung phone, stared at the numbers marching across its screen, and tossed it aside.  “Here’s hoping the PABX is still going,” he muttered.  Picking up a nearby handset, his shoulders slumped with relief when he heard a dial tone.  Immediately, Gerald speed-dialed the head of Campus IT.

“Better try and bring in the professors, I reckon.  This is going to take some big brains to solve,” I suggested.

Gerald glared at me.  “Way ahead of you, kiddo.  Why don’t you go look after your mate?”  He dialled number after number, explained briefly to each the crisis, hung the phone up, and turned to look at us.  “Boss has given me the go-ahead to disconnect the campus network.  Sit here, and don’t touch anything.  I’m going to unplug the fibre now.”

He stalked out of the room.  Henry looked up at me.  He was just as green as before.  “I think I’ve broken the Internet,” he whispered.

“Nah, I don’t reckon it’ll be outside the Uni network,” I reassured him, “we’ll just be expelled, not arrested.”

Moments later, Gerald strode back into the room, followed by his boss, who’d turned up with his tie unfastened and his shirt half buttoned.  Clearly Gerald had been able to impress a sense of urgency upon him.  He too stopped dead in the entrance, staring at the Network Operation Centre’s various computer screens, all filled with rapidly scrolling numbers, except that one that was filled with rapidly scrolling “acquiring node”.  I was really starting to hate that phrase.

“What on earth?  Who is responsible for this?”  He swung around and bore down on us.  “Second year students?  Have you been messing with my network?”

I cringed, but Henry mustered his courage, stood up and said, “Actually, sir, that’s not precisely correct.  Through an unfortunate series of events, and in fact my completely accidental overlooking of a key …”

“I don’t have time for this,” cried Head of IT, “What I want to know is how are we going to fix this?  Have you quarantined the network?”  This last to Gerald.  Yeah, okay, Henry did sound a bit weaselly that time.

“Yes, fibre links are disconnected, WiFi has been shut down, it should be contained, sir,” responded Gerald immediately.

“Okay, so how do we get this damned virus off my network?”

Henry felt obliged to pipe up again.  “Technically, sir, it’s not a virus, because it isn’t attaching to other executable files, but rather is more like a worm, a bit like the Morris worm from 1988 …”  I remembered the Morris worm, unleashed on a young and unsuspecting Internet by a less-than-cautious graduate student.  Thinking back to the Computer History class from the previous year, I could see that Bang bore some distinct similarities.  I just hoped Bang wouldn’t infect 10% of the world’s computers!  More worrying to me was the fact that the fact that the creator of the worm, Robert Morris, was prosecuted for releasing the worm.  But there’d be time to stress over that later.  Henry was continuing, “It continues on by piggybacking on the TCP handshake when connection is established.”  Which made no sense to me.  I should probably remind you that I am going on my own memory here, and I’ve probably got the technical details all wrong.  It was complicated, anyway.

As Henry spoke, the NOC telephone rang.  Gerald grabbed it.  It was the University Vice Chancellor, calling because not only could he not use his mobile phone, or his desktop computer, but his car had stalled, on the street, on the way into the University, with the same horrid numbers scrolling up the multifunction display in the car.  In fact, he was calling on his wife’s ancient Nokia phone which she refused to give up and which to him was looking a more attractive device by the second.  Henry and I looked at each other.  If Bang had made it onto his car computer, it was definitely out in the wild.

From there, a tidal wave of events took us, and flung us headlong across the pages of history, all in one fateful morning.  A television was wheeled into the NOC, and we watched a harried presenter in a makeshift studio (their normal studio computers had crashed, of course) report one network after another succumbing to Bang.  Henry watched, silently.  It was clear that he’d recovered a bit, because he was no longer green, more of a 1984 Mac white.  He was also clearly thinking.  I watched, also silently, as events unfolded, like a train wreck, too horrible to look away from.  Airports were shut down.  The stock market was offline.  While some telephones were still working, and most power networks were still up, the companies responsible for those systems were unable to perform even the most basic tasks.  All around the world, computers were busy calculating Henry’s fantastic factorial.

Eventually the Vice Chancellor made his way onto campus, on a hastily borrowed bicycle.  He clearly had no idea who I was, and just as clearly knew very well who Henry was.  Funny that.  He convened a crisis meeting, and invited both Henry, as author of the crisis, and I, as honorary co-author apparently (an honour I was anxious to forgo), to sit in and listen.  Professor Eisenfaktor was there, as the guru in the department, and frowning at Henry he started with a simple question: “Just how does this worm” (he accepted the term without blinking) “replicate?”

For the third time, Henry started to explain.  “It piggybacks off the SYN packet during the handshake, and … “ and my eyes glazed over, again.  To be honest, I wasn’t sure I wanted to know how it replicated.  The less I knew, the better.  “… and the program uses a distributed parallel multiplication algorithm to calculate the next factor, based on the work by Bunimov in 2003.”  Henry paused and looked at Professor Eisenfaktor.

“Yes, yes, I know who Bunimov is,” growled Eisenfaktor.  “Continue.”

“And anyway, the end of the story is because the program is self-healing, I managed to overlook the fact that it has no way to stop it short of turning off the computer and disconnecting from the network so it cannot be reinstated,” finished Henry.

Professor Eisenfaktor surprised me then by praising Henry.  “That is remarkable,” he enthused, “a self-replicating, self-healing, resilient, cross-platform, distributed computing worm, all written in one evening?  How many hard problems did you solve without realising it?  You’ve got a bright future ahead of you!  Assuming you survive this, of course,” he continued, somewhat less encouragingly.  “But now, you have created this crisis.  To think that no one had spotted that flaw in TCP.  So obvious now, is it not?  Ach!”  (He tended to lapse into a more Germanic turn of phrase as he got more excited – this kept us amused during lectures.)  “As I say, you have created this crisis, and now this crisis you shall solve.”

“Ah, I am not sure that is all that wise, Professor,” the VC cut in, “this young man, while meaning well, no doubt lacks experience and I would feel more comfortable if he took merely, say, an advisory role, in any action or steps which we as a University corporate may take in addressing this, which I may say without exaggeration, existential crisis.  I have no doubt that even at this minute, the source of this virus, ah, worm, is being hunted down, and I am sure that the authorities will be anxious to speak to this young man in very short order.”

“Precisely,” retorted Professor Eisenfaktor, “and I think you underestimate young Henry here.  How long has that flaw in the protocols been there?  Decades!  And in one evening, Henry, purely in the interests of furthering his, if I may say so, extraordinary algorithm project, has not only identified this flaw, but perfectly and precisely utilised it in the creation of this wondrously beautiful replicating worm, what was it called?  Bang?  What a clever pun!”  The Professor was obviously getting a bit excited, and even I was starting to get the idea that what Henry had created was, indeed, extraordinary, both in its technical beauty, and in the way it so effortlessly overwhelmed the entire planet’s computing infrastructure.

“No doubt Henry has a suggestion?” the Professor finished, looking at Henry quizzically.

“Well, actually,” started Henry, uncharacteristically but perhaps unsurprisingly diffidently, “I do.  I believe I can create a program that follows Bang, in its footsteps as it were, and both kills Bang on each system it encounters, and closes the hole in the protocol at the same time.”  He paused.

“Go on,” the Vice Chancellor prompted.

“Um, there’s not much more to say, really,” said Henry, “except that I thought we could call the program Crash.  Crash Bang.”

“Right.”  The VC did not seem amused.  “Well, why don’t you and um, your friend, go back to the library and start doing whatever it is you need to do.  But don’t you put that computer online until after you have cleared it with the Professor.”

Henry and I stood.  I glanced at the television.  The President of the United States was now on screen, talking about the crisis, although we could not hear him as the sound was muted.  I gulped.  Well, if Henry hadn’t been noticed when was growing up, he’d certainly be noticed now.  I’d probably end up notorious as well.  I wondered if we’d both go to jail, or worse.  The entire world was grinding to a halt.  Banks had closed their doors.  Supermarkets were not admitting customers.  Leaders were telling their populace to stay at home and prepare for disaster.  And yet, despite this, some phones were still working, and we still had electricity.  I didn’t know what would happen if the power went out.  I wasn’t sure Henry had enough juice to finish his Crash program.

I’m sure you know the rest.  Henry went on to write Crash, even faster than he’d written Bang, and the Vice Chancellor himself held off the police until he’d finished.  Solemnly, Henry turned on his WiFi, Gerald switched the Campus WiFi back on, and we watched, entranced, as one after another, the numbers flicked off screens, replaced briefly by a smiling face – Henry thought it appropriate – before restoring the computer, tablet or phone to its usual duties.  One by one, networks around the world restored themselves.  And in the process, the security hole in the protocol was patched, automatically.  And then the Federal Police asked Henry, and I (poor innocent I) to please come with them for a little chat.

All’s well that ends well.  Henry is now pretty famous, and some of it has even rubbed off on me, though I don’t think I deserve it.  Although many clamoured for a prison sentence, Henry escaped conviction, because, as his lawyer said, it was a simple bug and anyone could see he didn’t have a malicious bone in his body.  But Henry and I were both banned from the Uni’s labs for the rest of the semester, which I thought was quite unfair, given I hadn’t done anything.

Henry was mostly disappointed that in the rush he didn’t find out what factorial result Bang had reached before being shut down.  But then as he said, even if he knew the answer, it’s not like we could have recorded it anywhere for posterity.  Not without using all the computers in all the world.

The Day Henry Broke the Internet – A Short Story – Part 1

“He’s wrong, you know,” stated Henry, decidedly.

“Um what?” I said, intelligently, as I dazedly looked up from my iPhone.

“Professor Eisenfaktor is wrong.  He said BigInts were essentially infinite.  But they’re not at all.  BigInts are limited.  They’re limited by design by the memory on the computer.”

“Ah ha ha,” I laughed weakly. We were making our way out of the lecture theatre, Henry leading the way as usual, weaving mysteriously through the ragged masses of students, somehow unconsciously finding the shortest path through the crowd, and simultaneously looking over his shoulder and firing commentary at me about the obvious shortcomings of the lecture.

Me? I was happy just to wander out in his wake, checking Facebook as I went. I didn’t really feel a big need to debate the lecture but knew that Henry did have that need. So I humoured him.

“That’d be a, um, pretty big number, wouldn’t it?” I stated cautiously. “Like, why’d you need to work with numbers that big?  An integer with four billion digits?” (I wasn’t just a pretty face.)

“That’s not the point, and you know it. The entire premise of his lecture is flawed. He introduced these BigInt algorithms to us as an alternative for classical integer overflow problems, didn’t he? Well he just moved the boundary conditions. He didn’t fix the problem at all!”

I didn’t quite know what to say in reply to this, so in an unusual fit of intelligence, I said nothing at all.

Undeterred, Henry plowed on. “Who knows what uses there are for huge integers?  I mean, look at cryptography.  Large numbers are a cryptographer’s bread and butter.  And you know how often those crypto systems fall to brute force attacks?  Moore’s Law man!  It’s inevitable and at some point the Prof’s system will also stop meeting our needs.  But … what if …”

Henry abruptly stopped, creating a substantial eddy in the flow of students, as I carefully moored myself next to him. It wasn’t like Henry to ever stop so I was pretty curious to see what would come next.

“Here!” came next, followed by a textbook and a pile of notebooks, tossed into my arms, which I expertly caught. “See you later.”

“Hey? Where’re you going? We’ve got Expert Systems now,” I blurted.

“I’ve got something to figure out. Say hi to Prof Dexter for me,” Henry flung back as he disappeared into the crowd.

I shrugged, used to Henry’s ideas by now, and made my solitary way to Professor Dextor’s Expert Systems, resigning myself to figuring out what on earth the Prof was on about without Henry’s impatient yet insightful interpretation to guide me.

After I’d struggled through Professor Dexter’s Expert Systems, my brain filled with forward chains, backward chains and half understood inference engines, I headed for Henry’s favourite hideout.  The library, of course.  I desperately needed his help to make sense of the lecture.

Now, I’m sure you’ve seen The Social Network movie, right?  Okay, well, Henry is so much like Mark in that movie.  Totally brilliant, totally focused.  Always buried in an idea, solving, creating.  But that’s where the similarities stop.  To be fair, the Mark of that movie is focused and brilliant but he’s also a total jerk, yeah?  And Henry’s about as nice as they come.  Absent minded as hell, oblivious to his own intelligence, but drag him away from his oversized brain and he’s incredibly generous and my best mate.

He and I started Uni together, and by that I mean, at the same time, because he was of course light years ahead of me.  We’d grown up together in a small rural town, not too far from the city, but just far enough that Henry’d never been noticed.   His mother was always talking about his brilliance, but who listens to mothers?  In high school, we’d been the two geeks hiding in the computer lab – that is, the two old Macs sitting in the corner of the classroom – coding away at teenage game projects, Henry leading, myself following, contributing an idea here or there, and happily doing the hack coding.

He’d been noticed at Uni though.  After acing his first year without seeming to ever pay attention, and at the same time getting a reputation for being the student who could be relied upon to correct the lecturer if they made a mistake – I must hasten to add, gracefully, and with tact – Henry’s professors none-the-less were somewhat afraid of him, even the intimidating brainbox that was Professor Eisenfaktor.  It would start with a raised hand, and the lecturer would say, “Yes, Henry?” (after their first run-in, they never forgot his name), and Henry would reply, “I’m really sorry, but I’m not sure that’s entirely correct.  You see, …” and he’d descend into a complex and yet one hundred percent correct explanation of just where they’d gone wrong.

And credit where it’s due, too, to the lecturers, because Henry had a gift for elocution and explanation, which meant that the whole class would listen, and a light would come on in their collective brains as Henry effortlessly reconstructed the lecture.  Instead of pandering to their egos, and shutting him down as they so easily could have, the faculty got together in a meeting (I call it the “What do we do with Henry meeting”) and ended up with a concerted plan to encourage him.  That takes guts, to accept that a student may be able to take that lecture you took hours preparing, and actually make it hang together, and make it look so easy.  So I took my hat off to them.

No doubt, if they can nab him, they’ll have him lecturing First Year by the time he’s in Honours.  If they can keep him on topic, anyway.  And assuming he doesn’t get snarfed up by someone like Defense Signals Directorate.

Anyway, where was I?  That’s right, getting to the library.  Henry was in the library, in his usual corner, with a pile of books next to him as he hammered away at his MacBook with its Tux stuck over the Apple.  Beautiful hardware, but who’d run an operating system designed for Arts majors?

“Hey,” I said, “can I ask you a question?”

Henry glanced up at me.  “Oh, great to see you!  Did you know how amazing the early hardware guys were?  They did ridiculously cool things.  Like, look at this guy, Howard Aiken, he’s my new hero.” (Henry had lots of heroes.) “So he built this computer which could do calculations automatically, and the processors were driven by a 15 metre shaft off an electric motor.  How cool is that?  Way cooler than this thing!” he said, as he patted his notebook affectionately.  “What was your question?”

I dived into the morass of Expert Systems with Henry, and within a couple of minutes he had me straightened out.  You see?  Great guy.

Henry was instantly back into his computer.  I still didn’t know what he was up to, and was loathe to interrupt him again, so I started looking through the textbooks piled up haphazardly next to him on the desk.  “Computational Complexity”.  Nope, that didn’t help.  “Self-Stabilization”.  What?  “Combinatorial Optimization: Algorithms and Complexity”   Okay, I admit it, I was way out of my depth.  Just then Henry grabbed another book off the pile, flicked through a few pages, emitted a satisfied grunt, and with his pen, started circling paragraphs and drawing diagrams.  In the library textbook.  Whoa.  That was really not the done thing.

“What’re you doing?”  I stuttered.

“What?  Nothing, just this book makes so much sense!  She’s brilliant.  She’s like my new hero, see, look at this.”  And he flipped the book around and I stared blankly at the paragraphs of text, now annotated with Henry’s scribblings.  Curiously, I looked at the cover.  “Distributed Algorithms” by Nancy Lynch.  I had never heard of her.  I was no further on in trying to figure out what Henry was up to.  But I was used to that feeling.

In any case, it didn’t matter.  Henry and I had promised to meet some other students for board games after class (yes, we are that geeky) at the English-style pub around the corner from the campus.  I didn’t tell Henry, but I had a double-purpose for going to that board game meet-up, because there was this girl there, Jenny, you see, studying engineering, and I was pretty sure that she liked me.  Not quite sure enough to ask her out, but I was working towards it.  Henry would be oblivious to that, of course, but he did seem to enjoy board games.  I dragged Henry away from his computer, told him off again about the book, and Henry acquiesced and shoved his MacBook into his backpack, shrugging it on and following me (for once) out of the library and down towards the pub.

As we crossed the road, Henry started muttering to himself, pulled his phone out of his pocket, and started tapping away on it.  Of course.  Not on Facebook like any normal human being would be, but writing code.  How on earth do you write code with your thumbs?  I have no idea; I have enough trouble writing coherent English.  He’s a freak, I tell ya.  But we made it into the pub, found the players, and got our drinks and counter meals sorted.  Henry looked up from his phone for a moment, and unexpectedly said, “Do you know, I think I’ll sit this one out.”

“Your funeral,” said Chris.  Chris and I and Jenny and a few others settled down for some good old-fashioned Risk – we always start the night with an easy, light game so that we don’t intimidate newcomers.  Henry, back on his notebook, hammered away in the corner.

Several hours later, hopelessly defeated in Kamchatka, out-negotiated in Diplomacy, I threw in the towel.  I looked up from my dismal position and caught Jenny smiling at me.  That made it all right.  But I still didn’t have quite the courage to actually, really, talk to her.  And I had to find Henry.  Henry had found a power point – another of his magical talents – and was sitting on the floor beside it, still coding.  He’ll probably be coding all night, I figured, so I dragged him out of the pub and back to our student digs, and took myself off to bed.  He’s a big boy now, he can look after himself, I told myself sleepily and somewhat guiltily.  I knew his mother was hoping I’d look after him – she knew all too well what he was like when he got a bee in his bonnet.

I woke up in the morning, and found Henry fast asleep on the floor in the lounge room, his computer next to him.  I tried to make my way stealthily past him and into the kitchen, but I wasn’t quite quiet enough, and he sat up and looked at me a little blearily.

“What time did you get to sleep, you ninny?” I asked politely.

“Um, not too late, about 5ish, I think, but you’ve gotta see this,” Henry fired back.  He reached over to his computer, typed in one word – bang – and pressed Enter.  The screen instantly filled with numbers, scrolling past much faster than I could read them.  “Isn’t it awesome?”

“Yeah, but what is it?”  Even I could have whipped up a program to fill my screen with random numbers in an hour.  There had to be more to it.

“So I solved the Good Professor’s problem, the finite space problem.”  With a flourish, he announced pompously,  “I hereby introduce you to Distributed Infinite-Precision Integer Arithmetic.  DIPIA for short.”

“Great!  What is it?”  I asked, no further ahead.

Henry was used to my witty repartee, and didn’t hesitate to elucidate.

“BigInt in the Cloud.  Networked Bigint.  Unlimited Bigint.”  I was starting to see an inkling.  “But man, I’m hungry,” he abruptly cried – he had of course not eaten his counter meal last night – and leapt up to head out the door, again tossing his computer into his bag as he went.  I knew he wanted to head somewhere with coffee.  He liked good coffee.  It was another of those little foibles: I mean what sort of Uni student is fussy about the coffee he drinks?  I followed him to the hip coffee joint which opened early under the quad, and served coffee and bagels and orange juice, and WiFi.

The barista poured our coffees almost as soon as we walked in.  Yeah, regulars.  Henry pulled out his computer, and launched into a lecture.  Behind him, I was fascinated by the numbers scrolling up the screen.

“It’s a factorial generator!” Henry proclaimed, “and I figured out how to do arbitrary mathematical functions on BigInts in a distributed algorithm. Factorials are a great way to test BigInts because they get so big so fast, you see, and plus that meant I could call the app Bang, like the factorial sign, exclamation mark, you know.”

I nodded weakly.  Henry liked puns.  I watched his screen as the numbers continued to scroll past. “Wait for it, … wait for it … there!” exclaimed Henry, jabbing the screen with a handful of bagel.  I just made out the words “acquiring node” buried amongst the chaos of digits as they scrolled up and off the screen.

Just then my phone buzzed.  I pulled it out of my pocket, and was shocked to see numbers scrolling down its screen.

“What have you done!?” I cried.

“Isn’t it cool?” Henry asked.  “My computer started to run out of space, so it acquired a proximate node and offloaded onto that.  It finds the closest nodes on the network for minimal latency, to optimise the synchronisation and hand-off.  Of course, your phone is a bit slower and has a bit less memory so it won’t be able to contribute to the algorithm at the same rate, so I had to work out synchronisation and load balancing and … “

I was with him, almost.  “Do you mean that you wrote an iPhone app as well as a Linux app?  All last night?”  As I asked him, I tried to unlock my phone but the numbers kept scrolling past.  I held the power button to switch it off.  I was relieved that the phone switched off, as normal, but nearly as soon as I switched it on again, the numbers started scrolling past.

“Yeah, an iPhone app, and an Android app, and Windows and Mac clients as well.  It’s not too hard.  I got stuck with the automated deployment of the client until I figured out how to piggy-back onto the TCP handshake…” At this point, I must apologise because he went really technical and it was way over my head.  I’m not even sure I got that much of it right.

“So what’s it doing?” I asked, worriedly.  As I asked, another “node acquired” message scrolled past on his screen.

“Well, it’s calculating factorials, of course!  The best part is, the algorithm is resilient and self-healing so even if I turn off my computer, it won’t lose data and will continue on your phone … and probably my phone too I guess…” He paused and pulled his phone out of his pocket, and sure enough, numbers were scrolling past.

“Wait, so it jumped onto my phone because my phone is on the same WiFi as your laptop, right?”

“Right…” said Henry, a worried look on his face now, because he was way ahead of me, of course.  “Turn it off.  Quick.”  As he so ordered, he powered down TuxBook and his phone, and again I switched my phone off.  “Do you think we got to it fast enough?” he asked.

I caught up.  “Has it spread, do you mean?”  I was really worried now.  “Man, Henry, you’ve really done it this time.  Oh man.”

continued