Category Archives: Computing

Working around Delphi’s default grid scrolling behaviour

Delphi’s T*Grid components have an annoying little feature whereby they will scroll the cell into view if you click on a partially visible cell at the right or the bottom of the window. Then, this couples with a timer that causes the scroll to continue as long as the mouse button is held down and the cell it is over is partially visible. This typically means that if a user clicks on a partially visible cell, they end up selecting a cell several rows or columns away from where they intended to click.

In my view, this is a bug that should be fixed in Delphi. I’m not the only person who thinks this. I’ve reported it to Embarcadero at RSP-18542.

In the meantime, here’s a little unit that works around the issue.

{
  Stop scroll on mousedown on bottom row of grid when bottom row
  is a partial cell: have to block both initial scroll and timer-
  based scroll.

  This code is pretty dependent on the implementation in Vcl.Grids.pas,
  so it should be checked if we upgrade to new version of Delphi.
}

{$IFNDEF VER320}
{$MESSAGE ERROR 'Check that this fix is still applicable for a new version of Delphi. Checked against Delphi 10.2' }
{$ENDIF}

unit ScrollFixedStringGrid;

interface

uses
  System.Classes,
  Vcl.Controls,
  Vcl.Grids,
  Winapi.Windows;

type
  TScrollFixedStringGrid = class(TStringGrid)
  private
    TimerStarted: Boolean;
    HackedMousedown: Boolean;
  protected
    procedure MouseDown(Button: TMouseButton; Shift: TShiftState; X: Integer;
      Y: Integer); override;
    procedure MouseMove(Shift: TShiftState; X: Integer; Y: Integer); override;
    function SelectCell(ACol, ARow: Longint): Boolean; override;
  end;

implementation

{ TScrollFixedStringGrid }

procedure TScrollFixedStringGrid.MouseDown(Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  // When we first mouse-down, we know the grid has
  // no active scroll timer
  TimerStarted := False;

  // Call the inherited event, blocking the default MoveCurrent
  // behaviour that scrolls the cell into view
  HackedMouseDown := True;
  try
    inherited;
  finally
    HackedMouseDown := False;
  end;

  // Cancel scrolling timer started by the mousedown event for selecting
  if FGridState = gsSelecting then
    KillTimer(Handle, 1);
end;

procedure TScrollFixedStringGrid.MouseMove(Shift: TShiftState; X, Y: Integer);
begin
  // Start the scroll timer if we are selecting and mouse
  // button is down, on our first movement with mouse down
  if not TimerStarted and (FGridState = gsSelecting) then
  begin
    SetTimer(Handle, 1, 60, nil);
    TimerStarted := True;
  end;
  inherited;
end;


function TScrollFixedStringGrid.SelectCell(ACol, ARow: Longint): Boolean;
begin
  Result := inherited;
  if Result and HackedMousedown then
  begin
    // MoveColRow calls MoveCurrent, which
    // calls SelectCell. If SelectCell returns False, then
    // movement is blocked. But we fake it by re-calling with Show=False
    // to get the behaviour we want
    HackedMouseDown := False;
    try
      MoveColRow(ACol, ARow, True, False);
    finally
      HackedMouseDown := True;
    end;
    Result := False;
  end;
end;

end.

 

Substituted layouts in Text Services Framework

There is a tantalising parameter in the ITfInputProcessorProfileMgr::RegisterProfile function called hklSubstitute.  The description (used to) read “The substitute keyboard layout of this profile.   When this profile is active, the keyboard will use this keyboard layout instead of the default keyboard layout.  This can be NULL if your text service doesn’t change the keyboard layout.”

This functionality also surfaces in ITfInputProcessorProfileSubstituteLayout::GetSubstituteKeyboardLayout and ITfInputProcessorProfiles::SubstituteKeyboardLayout.

From the minimal description in MSDN, this sounds like a feature that we want to use in Keyman Desktop (allowing us to, for instance, use kbdus.dll as a base layout for some Keyman keyboard layouts).  However, we have been unable to get it to work.

  1. It seems the parameter type in the documentation is not quite right.  This parameter is not a HKL that you can obtain from LoadKeyboardLayout or some such.  Instead it needs to be a KLID, a keyboard layout ID, such as 0x00010409 for US Dvorak.
  2. The base layout is limited to one linked to the language the TIP is being installed for.  This means you cannot, for instance, substitute German QWERTZ for a French TIP, even if you wanted to. (And we want to!)
  3. Despite these restrictions, the RegisterProfile function does not check the validity of the parameter and happily tells you that the profile registered OK.
  4. While GetSubstituteKeyboardLayout returns the exact value that you registered, ITfInputProcessorProfileMgr::GetProfile returns 0 for the hklSubstitute member unless you meet the preconditions described above.  Furthermore, hklSubstitute, when available, is not a HKL but a KLID.
  5. Finally, even after all this, it seems that the substitute keyboard layout does not apply.

I received some clarification from Microsoft (edited slightly, thank you Vladimir Smirnov):

In fact, the feature is not that generic as you apparently expect. Its whole purpose was to enable smooth deprecation of IMM32 IMEs and their replacement (substitution) with TSF IMEs (aka TIPs). It doesn’t support substituting an arbitrary plain keyboard layout with another layout or IME (let alone of a different language).

The hklSubstitute is supposed to be a real HKL, but for IMM32 IMEs it actually matches the IME’s KLID. There’s old verification code there which looks for a matching KLID in HKLM for a given hklSubstitute and ignores the latter if there’s no match. That’s why you get 0 from GetProfile for your invalid or unsupported hklSubstitute.

Generic substitution was just never meant to be supported in TSF.

It’s a shame, but we work with what we’ve got. So for Keyman Desktop 9, we wrote a mnemonic layout recompiler that recompiles a Keyman mnemonic layout keyboard, merging it with a given Windows system base layout, at keyboard installation time in Keyman Desktop.

The Day Henry Broke the Internet – A Short Story – Part 2

previous episode

Henry was pale.

We heard an Arts major swearing behind us.  They have dreadful language, Arts majors.  “What the **** is wrong with this iPad?”

Henry bolted over to the girl, and stared at the screen, horrified, numbers pouring down its Retina display and mixed in the middle of it all, the dreaded words “acquiring node”.

“Switch it off, switch it off, switch it off!”  Henry tried to grab the iPad out of her hands and turn it off, which was probably not wise.

“Hey, get off!” she yelled, and grabbed back.

“Switch it off!”

“Okay, okay!”  She pressed the power button and the tablet screen flicked off.

“No!  That’s not enough, it’s gotta be powered down, now!”  Henry cried.

I left him to it, and looked around the café.  There were only a handful of other students and staff there, but they were all using computers or phones or tablets, and they were all, every one of them, staring at their screens in consternation.  I raced around and asked them – ordered them – to switch them off.

Under a minute later, Henry and I met in the middle of the café.  All the computers and phones and tablets were off.  We were safe.  Weren’t we?

No.  Over at the counter, we heard the barista talking to himself.  “What is up with this thing?”

Henry and I looked at each other.  I beat him to the counter.  Scrolling down the cash register screen were, of course, the digits from hell, and as I watched, not one, but four “acquiring node” messages scrolled past, seeming to bump into each other in their hurry to make their way up and off the screen.  I started to realise it was too late.  Mechanically, I reached over and switched off the cash register, as everyone in the café watched.

Henry stumbled out of the coffee bar and into the wide world.  It was early, and not many students were around, but those that were, were fighting their phones and tablets.  “Oh no,” Henry breathed, “It’s made the jump to the campus WiFi.”

I took charge.  “Henry, we’ve got to get to Campus IT and get this switched off before it gets off campus.”

“Oh man, I am in so much trouble…” Henry, at a loss for once in his life, obediently followed me as I ran, leaping up the stairs and diving through the shortcut behind the Engineering department (even in this crisis, my mind briefly flicked over to Jenny – she smiled at me last night!), and arrived, out of breath at the IT building just as an IT tech was making his early way into the building.  I realised I knew him, slightly, through interactions in the PC labs when the lab computers inevitably went down.  But I couldn’t remember his name.

“You’ve got to switch off the campus WiFi, like, right now,” I panted.

“What?  Don’t be silly!  Why would we need to do that?” he prevaricated.

“Just pull out your phone,” I said.  Sure enough, the phone’s screen was filled with numbers.  “It’s a virus,” I explained, “and it’s spreading rapidly!”  I thought that more detailed and accurate explanations could wait, though Henry stared at me, aghast, when I said the word “virus”.

I caught the words “It’s not a virus, it’s a worm…” muttered under his breath.

 

But that was enough to convince the tech.  Gerald, that was his name.  Together, we ran to the network management centre, burst in, and stopped, horrified.  Every screen in the room was filled with numbers, scrolling past faster than we could read.  And of course, “acquiring node”.  As I looked, I saw one computer that seemed to be spending most of its time acquiring nodes, and its screen was filled with the dreaded message, and only the occasional number, scrolling past faster than we could read.  At this rate, the entire University network would be down in minutes.

I just hoped the firewalls would stop Bang from escaping the University.  But I was none too confident.  Knowing Henry, he’d probably designed in a firewall bridging routine.  He always did create perfect code and account for every possibility.  Okay, almost every possibility.  He missed a big one this time, I thought ironically.

Gerald looked, accusingly, at me and Henry.  Henry just slumped against the wall shaking his head and looking as green as a VT420 terminal.  It was up to me then.  “There’s no time to explain,” I explained.  “We’re just going to have to shut down all the computers and disconnect the Uni from the Internet to stop the virus spreading.”

“Whoa, whoa, whoa!  I can’t do that!  That’s way above my pay grade,” Gerald flustered, “I’m calling in my boss.”

He pulled out his Samsung phone, stared at the numbers marching across its screen, and tossed it aside.  “Here’s hoping the PABX is still going,” he muttered.  Picking up a nearby handset, his shoulders slumped with relief when he heard a dial tone.  Immediately, Gerald speed-dialed the head of Campus IT.

“Better try and bring in the professors, I reckon.  This is going to take some big brains to solve,” I suggested.

Gerald glared at me.  “Way ahead of you, kiddo.  Why don’t you go look after your mate?”  He dialled number after number, explained briefly to each the crisis, hung the phone up, and turned to look at us.  “Boss has given me the go-ahead to disconnect the campus network.  Sit here, and don’t touch anything.  I’m going to unplug the fibre now.”

He stalked out of the room.  Henry looked up at me.  He was just as green as before.  “I think I’ve broken the Internet,” he whispered.

“Nah, I don’t reckon it’ll be outside the Uni network,” I reassured him, “we’ll just be expelled, not arrested.”

Moments later, Gerald strode back into the room, followed by his boss, who’d turned up with his tie unfastened and his shirt half buttoned.  Clearly Gerald had been able to impress a sense of urgency upon him.  He too stopped dead in the entrance, staring at the Network Operation Centre’s various computer screens, all filled with rapidly scrolling numbers, except that one that was filled with rapidly scrolling “acquiring node”.  I was really starting to hate that phrase.

“What on earth?  Who is responsible for this?”  He swung around and bore down on us.  “Second year students?  Have you been messing with my network?”

I cringed, but Henry mustered his courage, stood up and said, “Actually, sir, that’s not precisely correct.  Through an unfortunate series of events, and in fact my completely accidental overlooking of a key …”

“I don’t have time for this,” cried Head of IT, “What I want to know is how are we going to fix this?  Have you quarantined the network?”  This last to Gerald.  Yeah, okay, Henry did sound a bit weaselly that time.

“Yes, fibre links are disconnected, WiFi has been shut down, it should be contained, sir,” responded Gerald immediately.

“Okay, so how do we get this damned virus off my network?”

Henry felt obliged to pipe up again.  “Technically, sir, it’s not a virus, because it isn’t attaching to other executable files, but rather is more like a worm, a bit like the Morris worm from 1988 …”  I remembered the Morris worm, unleashed on a young and unsuspecting Internet by a less-than-cautious graduate student.  Thinking back to the Computer History class from the previous year, I could see that Bang bore some distinct similarities.  I just hoped Bang wouldn’t infect 10% of the world’s computers!  More worrying to me was the fact that the fact that the creator of the worm, Robert Morris, was prosecuted for releasing the worm.  But there’d be time to stress over that later.  Henry was continuing, “It continues on by piggybacking on the TCP handshake when connection is established.”  Which made no sense to me.  I should probably remind you that I am going on my own memory here, and I’ve probably got the technical details all wrong.  It was complicated, anyway.

As Henry spoke, the NOC telephone rang.  Gerald grabbed it.  It was the University Vice Chancellor, calling because not only could he not use his mobile phone, or his desktop computer, but his car had stalled, on the street, on the way into the University, with the same horrid numbers scrolling up the multifunction display in the car.  In fact, he was calling on his wife’s ancient Nokia phone which she refused to give up and which to him was looking a more attractive device by the second.  Henry and I looked at each other.  If Bang had made it onto his car computer, it was definitely out in the wild.

From there, a tidal wave of events took us, and flung us headlong across the pages of history, all in one fateful morning.  A television was wheeled into the NOC, and we watched a harried presenter in a makeshift studio (their normal studio computers had crashed, of course) report one network after another succumbing to Bang.  Henry watched, silently.  It was clear that he’d recovered a bit, because he was no longer green, more of a 1984 Mac white.  He was also clearly thinking.  I watched, also silently, as events unfolded, like a train wreck, too horrible to look away from.  Airports were shut down.  The stock market was offline.  While some telephones were still working, and most power networks were still up, the companies responsible for those systems were unable to perform even the most basic tasks.  All around the world, computers were busy calculating Henry’s fantastic factorial.

Eventually the Vice Chancellor made his way onto campus, on a hastily borrowed bicycle.  He clearly had no idea who I was, and just as clearly knew very well who Henry was.  Funny that.  He convened a crisis meeting, and invited both Henry, as author of the crisis, and I, as honorary co-author apparently (an honour I was anxious to forgo), to sit in and listen.  Professor Eisenfaktor was there, as the guru in the department, and frowning at Henry he started with a simple question: “Just how does this worm” (he accepted the term without blinking) “replicate?”

For the third time, Henry started to explain.  “It piggybacks off the SYN packet during the handshake, and … “ and my eyes glazed over, again.  To be honest, I wasn’t sure I wanted to know how it replicated.  The less I knew, the better.  “… and the program uses a distributed parallel multiplication algorithm to calculate the next factor, based on the work by Bunimov in 2003.”  Henry paused and looked at Professor Eisenfaktor.

“Yes, yes, I know who Bunimov is,” growled Eisenfaktor.  “Continue.”

“And anyway, the end of the story is because the program is self-healing, I managed to overlook the fact that it has no way to stop it short of turning off the computer and disconnecting from the network so it cannot be reinstated,” finished Henry.

Professor Eisenfaktor surprised me then by praising Henry.  “That is remarkable,” he enthused, “a self-replicating, self-healing, resilient, cross-platform, distributed computing worm, all written in one evening?  How many hard problems did you solve without realising it?  You’ve got a bright future ahead of you!  Assuming you survive this, of course,” he continued, somewhat less encouragingly.  “But now, you have created this crisis.  To think that no one had spotted that flaw in TCP.  So obvious now, is it not?  Ach!”  (He tended to lapse into a more Germanic turn of phrase as he got more excited – this kept us amused during lectures.)  “As I say, you have created this crisis, and now this crisis you shall solve.”

“Ah, I am not sure that is all that wise, Professor,” the VC cut in, “this young man, while meaning well, no doubt lacks experience and I would feel more comfortable if he took merely, say, an advisory role, in any action or steps which we as a University corporate may take in addressing this, which I may say without exaggeration, existential crisis.  I have no doubt that even at this minute, the source of this virus, ah, worm, is being hunted down, and I am sure that the authorities will be anxious to speak to this young man in very short order.”

“Precisely,” retorted Professor Eisenfaktor, “and I think you underestimate young Henry here.  How long has that flaw in the protocols been there?  Decades!  And in one evening, Henry, purely in the interests of furthering his, if I may say so, extraordinary algorithm project, has not only identified this flaw, but perfectly and precisely utilised it in the creation of this wondrously beautiful replicating worm, what was it called?  Bang?  What a clever pun!”  The Professor was obviously getting a bit excited, and even I was starting to get the idea that what Henry had created was, indeed, extraordinary, both in its technical beauty, and in the way it so effortlessly overwhelmed the entire planet’s computing infrastructure.

“No doubt Henry has a suggestion?” the Professor finished, looking at Henry quizzically.

“Well, actually,” started Henry, uncharacteristically but perhaps unsurprisingly diffidently, “I do.  I believe I can create a program that follows Bang, in its footsteps as it were, and both kills Bang on each system it encounters, and closes the hole in the protocol at the same time.”  He paused.

“Go on,” the Vice Chancellor prompted.

“Um, there’s not much more to say, really,” said Henry, “except that I thought we could call the program Crash.  Crash Bang.”

“Right.”  The VC did not seem amused.  “Well, why don’t you and um, your friend, go back to the library and start doing whatever it is you need to do.  But don’t you put that computer online until after you have cleared it with the Professor.”

Henry and I stood.  I glanced at the television.  The President of the United States was now on screen, talking about the crisis, although we could not hear him as the sound was muted.  I gulped.  Well, if Henry hadn’t been noticed when was growing up, he’d certainly be noticed now.  I’d probably end up notorious as well.  I wondered if we’d both go to jail, or worse.  The entire world was grinding to a halt.  Banks had closed their doors.  Supermarkets were not admitting customers.  Leaders were telling their populace to stay at home and prepare for disaster.  And yet, despite this, some phones were still working, and we still had electricity.  I didn’t know what would happen if the power went out.  I wasn’t sure Henry had enough juice to finish his Crash program.

I’m sure you know the rest.  Henry went on to write Crash, even faster than he’d written Bang, and the Vice Chancellor himself held off the police until he’d finished.  Solemnly, Henry turned on his WiFi, Gerald switched the Campus WiFi back on, and we watched, entranced, as one after another, the numbers flicked off screens, replaced briefly by a smiling face – Henry thought it appropriate – before restoring the computer, tablet or phone to its usual duties.  One by one, networks around the world restored themselves.  And in the process, the security hole in the protocol was patched, automatically.  And then the Federal Police asked Henry, and I (poor innocent I) to please come with them for a little chat.

All’s well that ends well.  Henry is now pretty famous, and some of it has even rubbed off on me, though I don’t think I deserve it.  Although many clamoured for a prison sentence, Henry escaped conviction, because, as his lawyer said, it was a simple bug and anyone could see he didn’t have a malicious bone in his body.  But Henry and I were both banned from the Uni’s labs for the rest of the semester, which I thought was quite unfair, given I hadn’t done anything.

Henry was mostly disappointed that in the rush he didn’t find out what factorial result Bang had reached before being shut down.  But then as he said, even if he knew the answer, it’s not like we could have recorded it anywhere for posterity.  Not without using all the computers in all the world.

The Day Henry Broke the Internet – A Short Story – Part 1

“He’s wrong, you know,” stated Henry, decidedly.

“Um what?” I said, intelligently, as I dazedly looked up from my iPhone.

“Professor Eisenfaktor is wrong.  He said BigInts were essentially infinite.  But they’re not at all.  BigInts are limited.  They’re limited by design by the memory on the computer.”

“Ah ha ha,” I laughed weakly. We were making our way out of the lecture theatre, Henry leading the way as usual, weaving mysteriously through the ragged masses of students, somehow unconsciously finding the shortest path through the crowd, and simultaneously looking over his shoulder and firing commentary at me about the obvious shortcomings of the lecture.

Me? I was happy just to wander out in his wake, checking Facebook as I went. I didn’t really feel a big need to debate the lecture but knew that Henry did have that need. So I humoured him.

“That’d be a, um, pretty big number, wouldn’t it?” I stated cautiously. “Like, why’d you need to work with numbers that big?  An integer with four billion digits?” (I wasn’t just a pretty face.)

“That’s not the point, and you know it. The entire premise of his lecture is flawed. He introduced these BigInt algorithms to us as an alternative for classical integer overflow problems, didn’t he? Well he just moved the boundary conditions. He didn’t fix the problem at all!”

I didn’t quite know what to say in reply to this, so in an unusual fit of intelligence, I said nothing at all.

Undeterred, Henry plowed on. “Who knows what uses there are for huge integers?  I mean, look at cryptography.  Large numbers are a cryptographer’s bread and butter.  And you know how often those crypto systems fall to brute force attacks?  Moore’s Law man!  It’s inevitable and at some point the Prof’s system will also stop meeting our needs.  But … what if …”

Henry abruptly stopped, creating a substantial eddy in the flow of students, as I carefully moored myself next to him. It wasn’t like Henry to ever stop so I was pretty curious to see what would come next.

“Here!” came next, followed by a textbook and a pile of notebooks, tossed into my arms, which I expertly caught. “See you later.”

“Hey? Where’re you going? We’ve got Expert Systems now,” I blurted.

“I’ve got something to figure out. Say hi to Prof Dexter for me,” Henry flung back as he disappeared into the crowd.

I shrugged, used to Henry’s ideas by now, and made my solitary way to Professor Dextor’s Expert Systems, resigning myself to figuring out what on earth the Prof was on about without Henry’s impatient yet insightful interpretation to guide me.

After I’d struggled through Professor Dexter’s Expert Systems, my brain filled with forward chains, backward chains and half understood inference engines, I headed for Henry’s favourite hideout.  The library, of course.  I desperately needed his help to make sense of the lecture.

Now, I’m sure you’ve seen The Social Network movie, right?  Okay, well, Henry is so much like Mark in that movie.  Totally brilliant, totally focused.  Always buried in an idea, solving, creating.  But that’s where the similarities stop.  To be fair, the Mark of that movie is focused and brilliant but he’s also a total jerk, yeah?  And Henry’s about as nice as they come.  Absent minded as hell, oblivious to his own intelligence, but drag him away from his oversized brain and he’s incredibly generous and my best mate.

He and I started Uni together, and by that I mean, at the same time, because he was of course light years ahead of me.  We’d grown up together in a small rural town, not too far from the city, but just far enough that Henry’d never been noticed.   His mother was always talking about his brilliance, but who listens to mothers?  In high school, we’d been the two geeks hiding in the computer lab – that is, the two old Macs sitting in the corner of the classroom – coding away at teenage game projects, Henry leading, myself following, contributing an idea here or there, and happily doing the hack coding.

He’d been noticed at Uni though.  After acing his first year without seeming to ever pay attention, and at the same time getting a reputation for being the student who could be relied upon to correct the lecturer if they made a mistake – I must hasten to add, gracefully, and with tact – Henry’s professors none-the-less were somewhat afraid of him, even the intimidating brainbox that was Professor Eisenfaktor.  It would start with a raised hand, and the lecturer would say, “Yes, Henry?” (after their first run-in, they never forgot his name), and Henry would reply, “I’m really sorry, but I’m not sure that’s entirely correct.  You see, …” and he’d descend into a complex and yet one hundred percent correct explanation of just where they’d gone wrong.

And credit where it’s due, too, to the lecturers, because Henry had a gift for elocution and explanation, which meant that the whole class would listen, and a light would come on in their collective brains as Henry effortlessly reconstructed the lecture.  Instead of pandering to their egos, and shutting him down as they so easily could have, the faculty got together in a meeting (I call it the “What do we do with Henry meeting”) and ended up with a concerted plan to encourage him.  That takes guts, to accept that a student may be able to take that lecture you took hours preparing, and actually make it hang together, and make it look so easy.  So I took my hat off to them.

No doubt, if they can nab him, they’ll have him lecturing First Year by the time he’s in Honours.  If they can keep him on topic, anyway.  And assuming he doesn’t get snarfed up by someone like Defense Signals Directorate.

Anyway, where was I?  That’s right, getting to the library.  Henry was in the library, in his usual corner, with a pile of books next to him as he hammered away at his MacBook with its Tux stuck over the Apple.  Beautiful hardware, but who’d run an operating system designed for Arts majors?

“Hey,” I said, “can I ask you a question?”

Henry glanced up at me.  “Oh, great to see you!  Did you know how amazing the early hardware guys were?  They did ridiculously cool things.  Like, look at this guy, Howard Aiken, he’s my new hero.” (Henry had lots of heroes.) “So he built this computer which could do calculations automatically, and the processors were driven by a 15 metre shaft off an electric motor.  How cool is that?  Way cooler than this thing!” he said, as he patted his notebook affectionately.  “What was your question?”

I dived into the morass of Expert Systems with Henry, and within a couple of minutes he had me straightened out.  You see?  Great guy.

Henry was instantly back into his computer.  I still didn’t know what he was up to, and was loathe to interrupt him again, so I started looking through the textbooks piled up haphazardly next to him on the desk.  “Computational Complexity”.  Nope, that didn’t help.  “Self-Stabilization”.  What?  “Combinatorial Optimization: Algorithms and Complexity”   Okay, I admit it, I was way out of my depth.  Just then Henry grabbed another book off the pile, flicked through a few pages, emitted a satisfied grunt, and with his pen, started circling paragraphs and drawing diagrams.  In the library textbook.  Whoa.  That was really not the done thing.

“What’re you doing?”  I stuttered.

“What?  Nothing, just this book makes so much sense!  She’s brilliant.  She’s like my new hero, see, look at this.”  And he flipped the book around and I stared blankly at the paragraphs of text, now annotated with Henry’s scribblings.  Curiously, I looked at the cover.  “Distributed Algorithms” by Nancy Lynch.  I had never heard of her.  I was no further on in trying to figure out what Henry was up to.  But I was used to that feeling.

In any case, it didn’t matter.  Henry and I had promised to meet some other students for board games after class (yes, we are that geeky) at the English-style pub around the corner from the campus.  I didn’t tell Henry, but I had a double-purpose for going to that board game meet-up, because there was this girl there, Jenny, you see, studying engineering, and I was pretty sure that she liked me.  Not quite sure enough to ask her out, but I was working towards it.  Henry would be oblivious to that, of course, but he did seem to enjoy board games.  I dragged Henry away from his computer, told him off again about the book, and Henry acquiesced and shoved his MacBook into his backpack, shrugging it on and following me (for once) out of the library and down towards the pub.

As we crossed the road, Henry started muttering to himself, pulled his phone out of his pocket, and started tapping away on it.  Of course.  Not on Facebook like any normal human being would be, but writing code.  How on earth do you write code with your thumbs?  I have no idea; I have enough trouble writing coherent English.  He’s a freak, I tell ya.  But we made it into the pub, found the players, and got our drinks and counter meals sorted.  Henry looked up from his phone for a moment, and unexpectedly said, “Do you know, I think I’ll sit this one out.”

“Your funeral,” said Chris.  Chris and I and Jenny and a few others settled down for some good old-fashioned Risk – we always start the night with an easy, light game so that we don’t intimidate newcomers.  Henry, back on his notebook, hammered away in the corner.

Several hours later, hopelessly defeated in Kamchatka, out-negotiated in Diplomacy, I threw in the towel.  I looked up from my dismal position and caught Jenny smiling at me.  That made it all right.  But I still didn’t have quite the courage to actually, really, talk to her.  And I had to find Henry.  Henry had found a power point – another of his magical talents – and was sitting on the floor beside it, still coding.  He’ll probably be coding all night, I figured, so I dragged him out of the pub and back to our student digs, and took myself off to bed.  He’s a big boy now, he can look after himself, I told myself sleepily and somewhat guiltily.  I knew his mother was hoping I’d look after him – she knew all too well what he was like when he got a bee in his bonnet.

I woke up in the morning, and found Henry fast asleep on the floor in the lounge room, his computer next to him.  I tried to make my way stealthily past him and into the kitchen, but I wasn’t quite quiet enough, and he sat up and looked at me a little blearily.

“What time did you get to sleep, you ninny?” I asked politely.

“Um, not too late, about 5ish, I think, but you’ve gotta see this,” Henry fired back.  He reached over to his computer, typed in one word – bang – and pressed Enter.  The screen instantly filled with numbers, scrolling past much faster than I could read them.  “Isn’t it awesome?”

“Yeah, but what is it?”  Even I could have whipped up a program to fill my screen with random numbers in an hour.  There had to be more to it.

“So I solved the Good Professor’s problem, the finite space problem.”  With a flourish, he announced pompously,  “I hereby introduce you to Distributed Infinite-Precision Integer Arithmetic.  DIPIA for short.”

“Great!  What is it?”  I asked, no further ahead.

Henry was used to my witty repartee, and didn’t hesitate to elucidate.

“BigInt in the Cloud.  Networked Bigint.  Unlimited Bigint.”  I was starting to see an inkling.  “But man, I’m hungry,” he abruptly cried – he had of course not eaten his counter meal last night – and leapt up to head out the door, again tossing his computer into his bag as he went.  I knew he wanted to head somewhere with coffee.  He liked good coffee.  It was another of those little foibles: I mean what sort of Uni student is fussy about the coffee he drinks?  I followed him to the hip coffee joint which opened early under the quad, and served coffee and bagels and orange juice, and WiFi.

The barista poured our coffees almost as soon as we walked in.  Yeah, regulars.  Henry pulled out his computer, and launched into a lecture.  Behind him, I was fascinated by the numbers scrolling up the screen.

“It’s a factorial generator!” Henry proclaimed, “and I figured out how to do arbitrary mathematical functions on BigInts in a distributed algorithm. Factorials are a great way to test BigInts because they get so big so fast, you see, and plus that meant I could call the app Bang, like the factorial sign, exclamation mark, you know.”

I nodded weakly.  Henry liked puns.  I watched his screen as the numbers continued to scroll past. “Wait for it, … wait for it … there!” exclaimed Henry, jabbing the screen with a handful of bagel.  I just made out the words “acquiring node” buried amongst the chaos of digits as they scrolled up and off the screen.

Just then my phone buzzed.  I pulled it out of my pocket, and was shocked to see numbers scrolling down its screen.

“What have you done!?” I cried.

“Isn’t it cool?” Henry asked.  “My computer started to run out of space, so it acquired a proximate node and offloaded onto that.  It finds the closest nodes on the network for minimal latency, to optimise the synchronisation and hand-off.  Of course, your phone is a bit slower and has a bit less memory so it won’t be able to contribute to the algorithm at the same rate, so I had to work out synchronisation and load balancing and … “

I was with him, almost.  “Do you mean that you wrote an iPhone app as well as a Linux app?  All last night?”  As I asked him, I tried to unlock my phone but the numbers kept scrolling past.  I held the power button to switch it off.  I was relieved that the phone switched off, as normal, but nearly as soon as I switched it on again, the numbers started scrolling past.

“Yeah, an iPhone app, and an Android app, and Windows and Mac clients as well.  It’s not too hard.  I got stuck with the automated deployment of the client until I figured out how to piggy-back onto the TCP handshake…” At this point, I must apologise because he went really technical and it was way over my head.  I’m not even sure I got that much of it right.

“So what’s it doing?” I asked, worriedly.  As I asked, another “node acquired” message scrolled past on his screen.

“Well, it’s calculating factorials, of course!  The best part is, the algorithm is resilient and self-healing so even if I turn off my computer, it won’t lose data and will continue on your phone … and probably my phone too I guess…” He paused and pulled his phone out of his pocket, and sure enough, numbers were scrolling past.

“Wait, so it jumped onto my phone because my phone is on the same WiFi as your laptop, right?”

“Right…” said Henry, a worried look on his face now, because he was way ahead of me, of course.  “Turn it off.  Quick.”  As he so ordered, he powered down TuxBook and his phone, and again I switched my phone off.  “Do you think we got to it fast enough?” he asked.

I caught up.  “Has it spread, do you mean?”  I was really worried now.  “Man, Henry, you’ve really done it this time.  Oh man.”

continued

The KeymanWeb Hello World app

A couple of days ago I helped some developers put together a very basic web page that uses KeymanWeb to showcase their keyboard layout. The idea is that a visitor to the page will immediately be able to try out their keyboard layout without any fiddling around.

Another requirement was that the page work well on desktop and mobile devices. This requires a few little tricks, mostly because KeymanWeb has some special requirements to work smoothly on mobiles and tablets.

I’ve presented snippets of the code, in the order they appear in the document.

And here are screenshots just for posterity.

The usual HTML 5 boilerplate is thrown in at the top of the page. Our first trick is to put the page into an appropriate zoom and scroll mode for mobile and tablet devices with a viewport meta tag. This is done with the following code:

<!-- at present, KMW requires this for mobile -->
<meta name="viewport" content="width=device-width, user-scalable=no">

At the time of writing, KeymanWeb (build 408) requires the viewport meta tag in order to render correctly on a variety of devices. Both clauses in the content attribute are required, and the keyboard will not scale correctly without them.

Next, I add a link to the stylesheet for the on screen keyboard. This is optional, because KeymanWeb will inject the link itself anyway, but it does prevent a Flash Of Unstyled Content.

<!-- this is optional but eliminates a flash of unstyled content -->
<link href='https://s.keyman.com/kmw/engine/408/src/osk/kmwosk.css' rel='stylesheet' type='text/css'>

I throw in some basic styling for the desktop and touch devices. You’ll see a selector of body.is-desktop on a couple of the style selectors. I use this instead of a @media query because this is differentiating between viewport-controlled devices and plain old boring desktop browsers. There may be a better way of doing this now, but I haven’t found it yet. The code for setting the is-desktop class is found further on in this post.

/* Styles for the page */

* {
  font-family: Tahoma, sans-serif;
}

body.is-desktop {
  padding: 22px 2px 0 2px;
  width: 900px;
  margin: auto;
}

h1 {
  font-size: 24pt;
  font-weight: bold;
  margin: 0 0 24px 0;
  padding: 0;
}

body.is-desktop h1 {  
  font-size: 48pt;
}

/* Style the text area */

textarea {
  width: 100%;
  height: 250px;
  font-family: "Khmer OS";
  font-size: 24px;
}

I need to talk a bit more about the On Screen Keyboard (OSK) styling. While I could use the default styling (as shown below), this looks a little dated and I wanted to show how the keyboard could be styled as you like. There is some complexity to the OSK styling as it has many moving parts, around sizing, scaling and positioning of each individual key. I don’t want to throw away that baby, so instead I keep all the bathwater and just add some bubbles to jazz up the display the way I want it. That’s a mixed metaphor, but it was late when I wrote this.

The default KeymanWeb keyboard style

First, this CSS rule-set removes the thick red border and dark background, and adjusts the padding around the sides to compensate.

/* Style the on screen keyboard */

body .desktop .kmw-osk-inner-frame {
  background: white;  
  border: solid 1px #404040;
  padding-bottom: 2px;
  padding-top: 8px;
}

Next, I hide the header and footer for the OSK. I don’t need them for this demo.


body .desktop .kmw-title-bar,
body .desktop .kmw-footer {
  display: none;
}

I tweak the spacing between keys and rows and make the keys slightly less rounded.

body .desktop .kmw-key-square {
  border-radius: 2px;
  padding-bottom: 8px;
}
body .desktop .kmw-key {
  border-radius: 2px;
  border: solid 1px #404040;
  background: #f2f4f6;
}
body .desktop .kmw-key:hover {
  background: #c0c8cf;
}

And that’s it for the CSS changes. It’s really pretty easy to restyle the keyboard without losing the benefits of a scalable, cross-platform keyboard. You’ll note that sneaky .desktop selector creeping in there. That’s because I’ve opted to style only the desktop keyboard; the touch layouts are already pretty nice and I’ll keep them as is.

I load the KeymanWeb code from the KeymanWeb CDN. You can load from the KeymanWeb CDN (running on Azure) or keep a locally hosted copy, of course. I’ve commented out the source version and because we have only a single keyboard, I have elected not to include a menu for switching languages.

<!-- Uncomment these lines if you want the source version of KeymanWeb (and remove the keymanweb.js reference)
<script src="https://s.keyman.com/kmw/engine/408/src/kmwstring.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwbase.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/keymanweb.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwosk.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwnative.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwcallback.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwkeymaps.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwlayout.js"></script>
<script src="https://s.keyman.com/kmw/engine/408/src/kmwinit.js"></script>
-->

<script src="https://s.keyman.com/kmw/engine/408/keymanweb.js"></script>

<!-- Uncomment if you want the user to be able to switch keyboards with a toggle popup ...
or you can create a custom switch if you prefer
<script src='https://s.keyman.com/kmw/engine/408/kmwuitoggle.js'></script>
-->

Now, no self-respecting web developer is going to include inline script, except in a Hello World demo. So let’s call this the KeymanWeb Hello World demo, so I can keep my self respect. Or something.

I use a simple page load listener. You could use an earlier listener, such as DOMContentLoaded, but window.onload has a rich and nearly honourable history.

// This should really be in a separate file. 
// But for now in one file is easier to understand
       
// After everything has loaded
window.addEventListener('load', function() {

Next some voodoo. Allows me to style the touch and desktop versions differently, as I touched on earlier. Why? Because KeymanWeb works very differently on touch devices. If you think about this, it must be so. A touch device has its own soft keyboard, which KeymanWeb must, by slightly convoluted means, hide, and replace with its own soft keyboard. Whereas, on a desktop device, KeymanWeb can show a utility soft keyboard, but does most of its interaction by getting in there between the hardware keyboard and the input fields.

Don’t ask me about Windows touch devices. Does anyone actually use those? (Okay, I’m sure they do, but it’s a rocky path for us poor keyboard devs to tread!)

if(!tavultesoft.keymanweb.util.isTouchDevice()) {
  document.body.className += 'is-desktop';
}

Next, make sure KeymanWeb is initialised. If I don’t do this myself, KeymanWeb will do so when things are ready, but I need KeymanWeb to be ready in order to load keyboards and attach events, and so on.

tavultesoft.keymanweb.init();

I could add another keyboard, or a stock keyboard from the repository. I haven’t, but this shows you how.

//tavultesoft.keymanweb.addKeyboards('@eng'); // Loads default English keyboard from Keyman Cloud (CDN) if you want it

In this case, I have a custom keyboard, developed by Lyheng Phuoy. This is an early beta version of the keyboard but already it’s very impressive.

// Add our custom keyboard        
tavultesoft.keymanweb.addKeyboards({
  name: 'Khmerism',       // Display name of the keyboard
  id: 'lyhengkeyboard',   // ID of the keyboard for reference in code
  filename: './lyhengkeyboard-1.0.js',  // source of the keyboard for dynamic load
  version: '1.0',         // version of the keyboard, optional
  language: [{
    name: 'Khmer',        // language name for UI elements
    id: 'khm',            // language ID for tagging text
    region: 'as'          // region of the language, for UI elements, optional
  }]
});

In this section, I want to control the size, position and flexibility of the keyboard. I don’t want it to be resizable or movable. So I set nomove and nosize accordingly.

var ta = document.getElementsByTagName('textarea')[0];

// Watch for the keyboard being shown and set its position
        
tavultesoft.keymanweb.osk.addEventListener('show', function() {
  var rectParams = {
    left: ta.offsetLeft, 
    top: ta.offsetTop + ta.offsetHeight + 8, // a small gap below the text area
    width: ta.offsetWidth, 
    height: ta.offsetWidth * 0.4, // a pleasing aspect ratio
    nomove: true,
    nosize: true
  };
  tavultesoft.keymanweb.osk.setRect(rectParams);
});
      
// Focus text area after everything loads
        
ta.focus(); 

Finally, in the body element, I set a special magic class, osk-always-visible, so that KeymanWeb doesn’t hide its on screen keyboard after displaying it the first time. And then we have basically the world’s simplest page with a textarea.

<!-- The osk-always-visible class tells KeymanWeb not to hide the osk on blur, after it is
     shown the first time -->
<body class="osk-always-visible">
  <h1>Typing Khmer</h1>
  <textarea rows="6" cols="60"></textarea>	
</body>

And that’s it! I’d love to see what you do with KeymanWeb on your own sites.

Here’s the full keyboard source, and that link to the demo page again, including Lyheng’s keyboard (with his permission).

When ញ៉ាំ meets ញ៉ំា

The Khmer script was added to the Unicode standard in September 1999. Today, nearly 18 years later, operating system renderers still get it wrong.

This is a quick post to document the difference in how several Khmer words are wrongly rendered on different current operating systems. I ran these tests on Windows 10 (10.0.14393), Android 7.1.1 Nougat, iOS 10.2.1, Mac OS X Sierra <> and Ubuntu 16.04 LTS with Firefox 47. The good news is that Windows 10 and Ubuntu passed all the tests (bar a font style issue with Leelawadee UI). Android passed nearly everything, except the bad encoding test.

Now, admittedly, the rules around triisap (U+17CA) and muusikatoan (U+17C9) are very complex. The Unicode standard description covers most of the difficulties, but not all of them.

Muusiaktoan is also sometimes called ធ្មេញកណ្ដុរ /tmɨɲ kɑndao/ – rat’s teeth, which is a fun name.

On to the words. In every case, the DauhPenh rendering is correct.

ញ៉ាំ /ɲam/ To eat
U+1789 U+17C9 U+17B6 U+17C6

ស៊ី /sii/ To eat (for young)
U+179F U+17CA U+17B8

As of Mac OS X Sierra, /sii/ now displays correctly. But contrast with /ʔum/, /ʔom/ below.

អ៊ំ /ʔum/, /ʔom/ Uncle, aunt
U+17A2 U+17CA U+17C6

Note how Leelawadee UI renders this wrongly; but that is a font rather than a renderer bug.

ប៊ី /bii/ A type of egg roll
U+1794 U+17CA U+17B8

ប៉ី /pəy/ A type of wind instrument
U+1794 U+17C9 U+17B8

As of Mac OS X Sierra, /pəy/ now displays correctly. But contrast with /bii/ above!

Yum yum yum

ញ៉ាំ /ɲam/ To eat

I’d like to pull out the word ញ៉ាំ for further analysis. Every operating system has some trouble with this word, because it could be encoded in several different ways. The correct way works on everything except iOS and Mac OS X. The incorrect encodings should really display wrongly, but none of the renderers complain about both invalid forms!

Correct order (ញ៉ាំ)

U+1789 U+17C9 U+17B6 U+17C6

Incorrect order (ញ៉ំា)

U+1789 U+17C9 U+17C6 U+17B6

Incorrect vowel (ញុំា)

U+1789 U+17BB U+17C6 U+17B6

In this instance, The DauhPenh rendering is appropriate for the first and second lines; the Apple rendering is ironically most appropriate for the third line!

Many thanks to Makara for his suggestion on the second incorrect rendering; I updated this post shortly after initial posting to include the extra example. There are other possible letter orders which may or may not display “correctly”; I will leave finding these as an exercise for the reader.

ZWNJ FTW

Here’s one I’ll examine in detail another time. Some words can be written in two different ways, neither really incorrect. The Unicode standard caters for these by allowing for insertion of a Zero Width Non Joiner (U+200C) to force the superscripted form of triisap (៊) or muusikatoan (៉). Windows 10’s Leelawadee UI font gets this one wrong (but its DauhPenh font doesn’t).

អ៊‌ី or អ៊ី /ʔii/ An exclamation of surprise
U+17A2 U+17CA U+17B8
ZWNJ
U+17A2 U+17CA U+200C U+17B8

Let’s Encrypt on Windows, redux

A couple of months ago I wrote a script to automatically renew Let’s Encrypt certificates in PowerShell on Windows. The renewal process works really well, however, there is one wrinkle that I did not cover. In this blog post I smooth out that wrinkle!

The wrinkle

When it came time to renew my certificate, a few days ago (well within the 90 day limit for the certificate, mind you), I discovered that the identifier for the certificate had expired. Why was this a problem? Well, I had originally used a manual DNS challenge to validate the identifier with Let’s Encrypt. This worked fine, but of course, manually creating a new identifier and challenge every 90 days completely undoes the benefit of automatic certificate renewal.

Smoothing out the wrinkle

In order to resolve this, I needed to automatically generate and validate an identifier at the same time as I generated a new certificate. The only automatic challenge provider at this time with ACMESharp is the http-01 provider with the IIS handler.

So I updated the script to generate the identifier automatically and validate with the http-01 challenge provider. However, the http-01 challenge provider requires a HTTP request, not a HTTPS request, to the hostname in question, because, and I quote the IETF draft here:

… Because many webservers allocate a default HTTPS virtual host to a particular low-privilege tenant user in a subtle and non-intuitive manner, the challenge must be completed over HTTP, not HTTPS.

I love subtle and non-intuitive computing!

But this was a problem for me, because the secure site I was working on did not have a http endpoint, only a https endpoint, which is the whole reason I used a DNS challenge in the first place.

I finally threw in the towel and I decided to setup a http endpoint, and configure automatic redirection to https (yes, I could go further here). You may find you need to setup an exception for automatic redirection for the .well-known/ root folder (where http challenges are kept as static files). I’ll leave that tweak for you to figure out (it’s just another line or two in your site root web.config).

A deeper wrinkle

The wrinkles get a little deeper, when you look at what happens if you already have a valid challenge response for an identifier. This could happen if you had manually validated another alias for the same identifier using e.g. dns, as I had in the past, or if you are renewing before your existing identifier expires. Because the challenge responses are matched to the hostname, and not the aliases, the previously valid responses continue to be acceptable. In this situation, the existing challenge response was used by the Let’s Encrypt server, and it never checked my shiny new web server challenge endpoint. This meant we needed to see if any existing challenge responses were considered valid, rather than relying on checking the challenge we’d just setup. The script changes handle this scenario.

Just read me the script

The full PowerShell script is shown below; the changes begin at the start of the Try block, and finish after the New-ACMECertificate call. More detail on how the script works is provided in the previous blog post.

import-module ACMESharp

#
# Script parameters
#

$domain = "my.example.com"
$iissitename = "my.example.com"
$certname = "my.example.com-$(get-date -format yyyy-MM-dd--HH-mm)"

#
# Environmental variables
#

$PSEmailServer = "localhost"
$LocalEmailAddress = "[email protected]"
$OwnerEmailAddress = "[email protected]"
$pfxfile = "c:\Admin\Certs\$certname.pfx"
$CertificatePassword = "PASSWORD!"

#
# Script setup - should be no need to change things below this point
#

$ErrorActionPreference = "Stop"
$EmailLog = @()

#
# Utility functions
#

function Write-Log {
  Write-Host $args[0]
  $script:EmailLog  += $args[0]
}

Try {
  Write-Log "Generating a new identifier for $domain"
  New-ACMEIdentifier -Dns $domain -Alias $certname
  
  Write-Log "Completing a challenge via http"
  Complete-ACMEChallenge $certname -ChallengeType http-01 -Handler iis -HandlerParameters @{ WebSiteRef = $iissitename }
  
  Write-Log "Submitting the challenge"
  Submit-ACMEChallenge $certname -ChallengeType http-01
  
  # Check the status of the identifier every 6 seconds until we have an answer; fail after a minute
  $i = 0
  do {
    $identinfo = (Update-ACMEIdentifier $certname -ChallengeType http-01).Challenges | Where-Object {$_.Status -eq "valid"}
    if($identinfo -eq $null) {
      Start-Sleep 6
      $i++
    }
  } until($identinfo -ne $null -or $i -gt 10)

  if($identinfo -eq $null) {
    Write-Log "We did not receive a completed identifier after 60 seconds"
    $Body = $EmailLog | out-string
    Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Attempting to renew Let's Encrypt certificate for $domain" -Body $Body
    Exit
  }
  
  # We now have a new identifier... so, let's create a certificate
  Write-Log "Attempting to renew Let's Encrypt certificate for $domain"

  # Generate a certificate
  Write-Log "Generating certificate for $domain"
  New-ACMECertificate $certname -Generate -Alias $certname

  # Submit the certificate
  Submit-ACMECertificate $certname

  # Check the status of the certificate every 6 seconds until we have an answer; fail after a minute
  $i = 0
  do {
    $certinfo = Update-AcmeCertificate $certname
    if($certinfo.SerialNumber -eq "") {
      Start-Sleep 6
      $i++
    }
  } until($certinfo.SerialNumber -ne "" -or $i -gt 10)

  if($i -gt 10) {
    Write-Log "We did not receive a completed certificate after 60 seconds"
    $Body = $EmailLog | out-string
    Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Attempting to renew Let's Encrypt certificate for $domain" -Body $Body
    Exit
  }

  # Export Certificate to PFX file
  Get-ACMECertificate $certname -ExportPkcs12 $pfxfile -CertificatePassword $CertificatePassword

  # Import the certificate to the local machine certificate store 
  Write-Log "Import pfx certificate $pfxfile"
  $certRootStore = "LocalMachine"
  $certStore = "My"
  $pfx = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
  $pfx.Import($pfxfile,$CertificatePassword,"Exportable,PersistKeySet,MachineKeySet") 
  $store = New-Object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore) 
  $store.Open('ReadWrite')
  $store.Add($pfx) 
  $store.Close() 
  $certThumbprint = $pfx.Thumbprint

  # Bind the certificate to the requested IIS site (all https bindings)
  Write-Log "Bind certificate with Thumbprint $certThumbprint"
  $obj = get-webconfiguration "//sites/site[@name='$iissitename']"
  for($i = 0; $i -lt $obj.bindings.Collection.Length; $i++) {
    $binding = $obj.bindings.Collection[$i]
    if($binding.protocol -eq "https") {
      $method = $binding.Methods["AddSslCertificate"]
      $methodInstance = $method.CreateInstance()
      $methodInstance.Input.SetAttributeValue("certificateHash", $certThumbprint)
      $methodInstance.Input.SetAttributeValue("certificateStoreName", $certStore)
      $methodInstance.Execute()
    }
  }

  # Remove expired LetsEncrypt certificates for this domain
  Write-Log "Remove old certificates"
  $certRootStore = "LocalMachine"
  $certStore = "My"
  $date = Get-Date
  $store = New-Object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore) 
  $store.Open('ReadWrite')
  foreach($cert in $store.Certificates) {
    if($cert.Subject -eq "CN=$domain" -And $cert.Issuer.Contains("Let's Encrypt") -And $cert.Thumbprint -ne $certThumbprint) {
      Write-Log "Removing certificate $($cert.Thumbprint)"
      $store.Remove($cert)
    }
  }
  $store.Close() 

  # Finished
  Write-Log "Finished"
  $Body = $EmailLog | out-string
  Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Let's Encrypt certificate renewed for $domain" -Body $Body
} Catch {
  Write-Host $_.Exception
  $ErrorMessage = $_.Exception | format-list -force | out-string
  $EmailLog += "Let's Encrypt certificate renewal for $domain failed with exception`n$ErrorMessage`r`n`r`n"
  $Body = $EmailLog | Out-String
  Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Let's Encrypt certificate renewal for $domain failed with exception" -Body $Body
  Exit
}

Side note: I discovered it’s important to let ACMESharp do its thing in the script and not try and do anything with it in another process because it tends to fall over with an exception if some other process is accessing its vault when it wants to.

Another minor update (18 Feb 2017): my $alias in my live script happened to be the same as my $domain; if they differed, then the script would fail. The $alias variable is no longer needed and has been removed from the script above. Thank you to BdN3504 for reporting this in the comments!

Automating certificate renewal with Let’s Encrypt and ACMESharp on Windows

 

UPDATE: 10 February 2017! I’ve updated the script in a new blog post to handle identifer expiry. 

 

Let’s Encrypt is a free, automated, and open Certificate Authority. And it is awesome. It is being used by over 15 million domains already to date.

le-logo-standard

Let’s Encrypt is a certificate authority. So that means that they issue certificates, specifically for secure https (TLS) websites. Lots of other organisations do this as well. But two things stand out about Let’s Encrypt. First, it’s free! Given that I was paying over $100/year for a certificate for one of my sites until recently, that’s a big win already. The second is, it’s automated!

The automated bit cannot be understated. It means that the first time I use Let’s Encrypt, I have to do a bunch of setup. But from then on, I no longer have to remember the arcane and complicated process of generating a certificate request, uploading it to a CA, waiting for the CA to process the request, and finally importing the certificate along with all the incidentals such as intermediate certificates.

However, the one thing about Let’s Encrypt that has stopped me using it so far is that I run some of my sites on IIS on Windows, but Let’s Encrypt is very *nix-focused. While there are clients for Windows, none of them are very complete and so it’s been a bit of hit and miss using them.

The most up-to-date client/library that I have found appears to be ACMESharp. This library is a PowerShell module, and while there is a GUI front end available, I haven’t used the GUI. I’ve worked entirely with the PowerShell module. ACMESharp is pretty flexible and covers everything I need, except one thing: renewals. It has no built-in automated renewal support. So I rolled my own.

I did most of my work in the Let’s Encrypt staging environment, after foolishly starting in the live environment and rapidly hitting the duplicate certificate rate limit. I recommend you do your testing in the staging environment also!

The easiest way to work in the staging environment is to setup a separate vault profile for ACMESharp. I ended up using my :user profile for staging and my :sys profile for the live host. To specify which vault profile you want to use, it’s best to use an environment variable, as otherwise you’ll inevitably forget to append the -VaultProfile parameter to one of your setup commands and leave yourself in a bit of a mess:

$env:ACMESHARP_VAULT_PROFILE=":user"

What follows is a script setup for my servers, but which should work for most Windows PowerShell scenarios. I have set it up to email me on success or failure; I’ll be watching it over the next little while to ensure that it gets things right. I have set it up as a scheduled task to run every 60 days, per Let’s Encrypt’s recommendation.

The script assumes you have already followed the ACMESharp Quick Start to configure your environment. The variables at the top of the script could be configured as script parameters, but for simplicity I’ve just put them at the top of the script. The script will request a new certificate, import the newly issued certificate into the localmachine certificate store, then assign it to all https bindings on the specified web site instance. Finally, it will delete all expired certificates associated with the domain in question.

UPDATE: 10 February 2017! Please see the revised script!

import-module ACMESharp

#
# Script parameters
#

$domain = "my.example.com"
$alias = "my.example.com-01"
$iissitename = "my.example.com"
$certname = "my.example.com-$(get-date -format yyyy-MM-dd--HH-mm)"

#
# Environmental variables
#

$PSEmailServer = "localhost"
$LocalEmailAddress = "[email protected]"
$OwnerEmailAddress = "[email protected]"
$pfxfile = "c:\Admin\Certs\$certname.pfx"
$CertificatePassword = "PASSWORD!"

#
# Script setup - should be no need to change things below this point
#

$ErrorActionPreference = "Stop"
$EmailLog = @()

#
# Utility functions
#

function Write-Log {
  Write-Host $args[0]
  $script:EmailLog  += $args[0]
}

Try {
  Write-Log "Attempting to renew Let's Encrypt certificate for $domain"

  # Generate a certificate
  Write-Log "Generating certificate for $alias"
  New-ACMECertificate ${alias} -Generate -Alias $certname

  # Submit the certificate
  Submit-ACMECertificate $certname

  # Check the status of the certificate every 6 seconds until we have an answer; fail after a minute
  $i = 0
  do {
    $certinfo = Update-AcmeCertificate $certname
    if($certinfo.SerialNumber -ne "") {
      Start-Sleep 6
      $i++
    }
  } until($certinfo.SerialNumber -ne "" -or $i -gt 10)

  if($i -gt 10) {
    Write-Log "We did not receive a completed certificate after 60 seconds"
    $Body = $EmailLog | out-string
    Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Attempting to renew Let's Encrypt certificate for $domain" -Body $Body
    Exit
  }

  # Export Certificate to PFX file
  Get-ACMECertificate $certname -ExportPkcs12 $pfxfile -CertificatePassword $CertificatePassword

  # Import the certificate to the local machine certificate store 
  Write-Log "Import pfx certificate $pfxfile"
  $certRootStore = "LocalMachine"
  $certStore = "My"
  $pfx = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
  $pfx.Import($pfxfile,$CertificatePassword,"Exportable,PersistKeySet,MachineKeySet") 
  $store = New-Object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore) 
  $store.Open('ReadWrite')
  $store.Add($pfx) 
  $store.Close() 
  $certThumbprint = $pfx.Thumbprint

  # Bind the certificate to the requested IIS site (all https bindings)
  Write-Log "Bind certificate with Thumbprint $certThumbprint"
  $obj = get-webconfiguration "//sites/site[@name='$iissitename']"
  for($i = 0; $i -lt $obj.bindings.Collection.Length; $i++) {
    $binding = $obj.bindings.Collection[$i]
    if($binding.protocol -eq "https") {
      $method = $binding.Methods["AddSslCertificate"]
      $methodInstance = $method.CreateInstance()
      $methodInstance.Input.SetAttributeValue("certificateHash", $certThumbprint)
      $methodInstance.Input.SetAttributeValue("certificateStoreName", $certStore)
      $methodInstance.Execute()
    }
  }

  # Remove expired LetsEncrypt certificates for this domain
  Write-Log "Remove old certificates"
  $certRootStore = "LocalMachine"
  $certStore = "My"
  $date = Get-Date
  $store = New-Object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore) 
  $store.Open('ReadWrite')
  foreach($cert in $store.Certificates) {
    if($cert.Subject -eq "CN=$domain" -And $cert.Issuer.Contains("Let's Encrypt") -And $cert.Thumbprint -ne $certThumbprint) {
      Write-Log "Removing certificate $($cert.Thumbprint)"
      $store.Remove($cert)
    }
  }
  $store.Close() 

  # Finished
  Write-Log "Finished"
  $Body = $EmailLog | out-string
  Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Let's Encrypt certificate renewed for $domain" -Body $Body
} Catch {
  Write-Host $_.Exception
  $ErrorMessage = $_.Exception | format-list -force | out-string
  $EmailLog += "Let's Encrypt certificate renewal for $domain failed with exception`n$ErrorMessage`r`n`r`n"
  $Body = $EmailLog | Out-String
  Send-MailMessage -From $LocalEmailAddress -To $OwnerEmailAddress -Subject "Let's Encrypt certificate renewal for $domain failed with exception" -Body $Body
  Exit
}

I guess I’ll find out in 60 days if the script still works! With many thanks to the users of StackOverflow, and other bloggers, for working code samples which saved me a lot of time reading reference documentation for so many of the different bits of glue here, from exception management, through to sending emails with PowerShell, through to assigning certificates to IIS website bindings, and more…

Update 2 December 2016: The original script had a bug that didn’t affect use with IIS. However, when I tried to use the Let’s Encrypt certificate with another program, in this case MailEnable, I found that the program did not have access to the certificate, even though it seemed it should have had.

When I started the relevant MailEnable service, it would display an error such as:

12/02/16 20:13:00 **** Error 0x8009030d returned by AcquireCredentialsHandle
12/02/16 20:13:00 **** Error creating credentials object for SSL session
12/02/16 20:13:00 Unable to locate or bind to certificate with name "my.example.com"

I checked the permissions and various other factors but only when I did a deep comparison of the details of a working certificate against the one that was failing, using certutil, as suggested in that blog linked above, did I spot the problem. The problem lay in the following CRYPT_KEY_PROV_INFO structure:

CERT_KEY_PROV_INFO_PROP_ID(2):
    Key Container = {XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}
  Unique container name: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
    Provider = Microsoft Enhanced Cryptographic Provider v1.0
    ProviderType = 1
    Flags = 0
    KeySpec = 1 -- AT_KEYEXCHANGE

When I compared this against a working certificate, I saw that the Flags member had a value of 20 (hex), not 0. 0x20 turns out to be CRYPT_MACHINE_KEYSET. Because I was missing the MachineKeySet flag in the certificate import call, this meant that the key was stored under the Administrator user’s keyset instead of the machine keyset. IIS coped with this, but not MailEnable, which runs under a more restricted user’s credentials.

I have also updated the script to delete all Let’s Encrypt certificates for the domain that have been obsoleted by the new certificate, rather than just certificates that have expired (-And $cert.NotAfter -lt $date), mostly to avoid the risk of accidentally selecting an old certificate manually when doing configuration via UI.

On my servers, I’ve also added a section to the end of the script that restarts various services that depend on the certificate and will not use a new certificate until after restarting.

Don’t forget to navigate to about:blank when embedding IWebBrowser2

Today I spent several hours trying to figure out why an embedded web browser component (in this case TEmbeddedWB) in a Delphi test app never received the appropriate IHttpSecurity and IWindowForBindingUI QueryService requests.

I was doing this in order to provide more nuanced handling of self-signed certificates in an intranet context. We all do this, right? Here the term “nuanced” means “Of course I trust self signed certificates on my intranet, don’t you?” Feel free to rant and rave on this. 😉

But no matter what I did, what incantations I tried, or what StackOverflow posts I perused, I was unable to find an answer. Until finally I stumbled on a side comment in a thread from 2010. Igor Tandetnik notes that:

Right after creating the control, navigate it to about:blank. Right after that, navigate it to the page you wanted to go to. It’s a known problem that IServiceProvider doesn’t work for the very first navigation.

And this was something that I kinda knew in the back of my head, but of course had forgotten. Thank you Igor.

This post would not be complete without some splendiferous code. Just for reference, it’s so simple if you don’t blank out and forget about:blank.

unit InsecureBrowser;

interface

uses
  Winapi.Windows,
  Winapi.Messages,
  Winapi.Urlmon,
  Winapi.WinInet,
  System.SysUtils,
  System.Variants,
  System.Classes,
  Vcl.Graphics,
  Vcl.Controls,
  Vcl.Forms,
  Vcl.Dialogs,
  Vcl.OleCtrls,
  Vcl.StdCtrls,
  SHDocVw_EWB,
  EwbCore,
  EmbeddedWB;

type
  TInsecureBrowserForm = class(TForm, IHttpSecurity, IWindowForBindingUI)
    web: TEmbeddedWB;
    cmdGoInsecure: TButton;
    procedure webQueryService(Sender: TObject; const [Ref] rsid,
      iid: TGUID; var Obj: IInterface);
    procedure FormCreate(Sender: TObject);
    procedure cmdGoInsecureClick(Sender: TObject);
  private
    { IWindowForBindingUI }
    function GetWindow(const guidReason: TGUID; out hwnd): HRESULT; stdcall;

    { IHttpSecurity }
    function OnSecurityProblem(dwProblem: Cardinal): HRESULT; stdcall;
  end;

var
  InsecureBrowserForm: TInsecureBrowserForm;

implementation

{$R *.dfm}

function TInsecureBrowserForm.GetWindow(const guidReason: TGUID;
  out hwnd): HRESULT;
begin
  Result := S_FALSE;
end;

function TInsecureBrowserForm.OnSecurityProblem(dwProblem: Cardinal): HRESULT;
begin
  if (dwProblem = ERROR_INTERNET_INVALID_CA) or
     (dwProblem = ERROR_INTERNET_SEC_CERT_CN_INVALID)
    then Result := S_OK
    else Result := E_ABORT;
end;

procedure TInsecureBrowserForm.webQueryService(Sender: TObject;
  const [Ref] rsid, iid: TGUID; var Obj: IInterface);
begin
  if IsEqualGUID(IID_IWindowForBindingUI, iid) then
    Obj := Self as IWindowForBindingUI
  else if IsEqualGUID(IID_IHttpSecurity, iid) then
    Obj := Self as IHttpSecurity;
end;

procedure TInsecureBrowserForm.cmdGoInsecureClick(Sender: TObject);
begin
  web.Navigate('https://evil.intranet.site/');
end;

procedure TInsecureBrowserForm.FormCreate(Sender: TObject);
begin
  web.Navigate('about:blank');
end;

end.