Windbg and Delphi – a quick reference

This is a list of my blog posts on using WinDBG with Delphi apps, mostly for my reference. Internally, I use a version of tds2dbg with some private modifications, but you should have reasonable results with the public version.

WinDBG and Delphi exceptions

WinDBG and Delphi exceptions in x64

Locating Delphi exceptions in a live session or dump using WinDbg

Debugging a stalled Delphi process with Windbg and memory searches

Finding class instances in a Delphi process using WinDbg

More windbg tricks with Delphi – how to ignore specific exceptions (Jan 2016)

I also have some other posts that talk about WinDBG and/or Delphi which can be helpful for illustrative purposes:

Another partial malware diagnosis

Detecting the Citadel Trojan with an Application Failure

When characters go astray: diagnosing missing characters when printing with IE9

WaitForSingleObject. Why you should never use it.

IE11, Windbg, JavaScript = joy

 

 

 

 

Notes on a Khmer mobile keyboard for Keyman

While other Khmer keyboards exist for iOS and Android, I wanted to try playing with one myself, given I am currently learning Khmer. Creating a keyboard layout is a great way to rapidly become very familiar with a script!

Keyman Developer makes it easy to play around with the layout of a keyboard and rapidly iterate the design. I was able to turn the original desktop layout into a mobile-optimised layout in under two hours, using only the visual editor.

keyman-developer

The base keyboard comes from the khmer10 Keyman keyboard created by Andrew Cunningham, which is based on the NiDA Khmer keyboard layout. This is a desktop keyboard, which follows a phonetic-style input, with letters placed as far as possible on keys with a similar sound in the English alphabet. For example, ក is on the [k] key.

Design principles

I had a number of goals I wanted to achieve with this keyboard.

A design good for a language learner

I wanted the design to help me remember the script, the sounds and related letters. This may not be optimal for a person fluent in the language, but for me, the NiDA layout’s phonetic-style layout was a good starting point.

Reduce the number of keys …

As the original design is for a desktop keyboard, there are too many keys to fit on a normal mobile layout. As a mobile keyboard should ideally have no more than ten keys in a row, I started with reducing that as one goal.

… But don’t lose all relationship with the desktop keyboard

I tried to avoid moving keys around on the keyboard, or removing keys other than the ones described below. This way, once I do become familiar with the mobile layout, it is not a difficult transition to using the NiDA layout on a desktop computer.

Move symbols and numbers off base layers

A number of non-alphabetic symbols are placed in a seemingly haphazard fashion on both the unshifted and Shift layers. The position of these symbols is probably pragmatic – the keys were available and not used for any other purpose. On a touch layout, we don’t need to maintain this because we can have as many or as few keys as we wish.

I moved all non-alphabetic symbols and numbers to a numeric/symbol layer.

symbol-numeric-layer

Relate sub consonants to base consonants

ក្ក  ក្ខ  ក្គ  ក្ឃ  ក្ង  ក្ច  ក្ឆ  ក្ជ  ក្ឈ  ក្ញ  ក្ដ  ក្ឋ  ក្ឌ  ក្ឍ  ក្ណ  ក្ត  ក្ថ  ក្ទ  ក្ធ  ក្ន  ក្ប  ក្ផ  ក្ព  ក្ភ  ក្ម  ក្យ  ក្រ  ក្ល  ក្វ  ក្ឝ  ក្ឞ  ក្ស  ក្ហ  ក្ឡ  ក្អ

The original keyboard uses the key [j] as a prefix to create a sub consonant, by emitting the Unicode U+17D2 sub consonant marker character. This is a little obscure, and meant that the shapes of sub consonants were never visible on the keyboard.

I wanted to hide this encoding-based knowledge. I have added the sub consonants to the keyboard under a longpress menu for each base consonant, and added a rule to delete both the consonant and the prefix U+17D2 marker together when backspace is pressed. Thus, the average keyboard user need never know about the existence of U+17D2.

longpress-sub-consonant

Independent vowels

As these are less frequently typed than the dependent vowels, they really didn’t need their own key on the keyboard. Again, from a language learner point of view, placing the independent vowels under the related dependent vowel symbol, as longpress menus, helps me to learn the shapes more rapidly. It also means that I don’t confuse the independent vowel shapes with similarly shaped consonants on the layout.

Adding missing characters

The Khmer digits were already on the keyboard, but the Hindu-Arabic numerals were not. I added these as long-press under the Khmer digit keys on the numeric/symbol layer.

Things that are not yet right

This keyboard is nowhere near finished, but it’s now at a good point to start using it, before making further optimisations. After using it for a while, I will have a clearer understanding of what is uncomfortable or awkward to use, and will also probably have better ideas of how to resolve the issues.

Some of the issues I already know about are:

  • Still too many keys per row: some rows have 12 keys. I should try to reduce this to 10 keys.
  • There are vowel combinations that may or may not be necessary. These do not render correctly on most operating systems on the keyboard, but do when used in writing.
  • The backspace key should be either on the top row, or on the third row. It’s only on the second row at present because that was where there was space!
  • The keyboard is missing a number of symbols that I probably won’t be using for now, but should be available for a complete solution, such as the Lek Attak numeric divination symbols. Some archaic letters are also not yet present.
  • The Khmer OS fonts do not use the dotted circle for isolated diacritics, which makes them hard to read on the keyboard.
  • The layers are not precisely the same width, which leads to slightly disconcerting movement on layer switches.
  • The shift key on the shift layer is missing its icon.
  • I just realised the independent vowels are currently missing – I need to re-add those soon!

Screenshot on Android:

Issues with using the keyboard include:

  • iOS has rendering bugs with Khmer, for example ខ្ញុំ on iOS overlaps the two subscript marks.

khmer-on-ios-khnyom-render-bug

 

  • On Android, there are different rendering issues, for example the font is too large for the keys by default (showing the original keyboard as I don’t have an Android device handy for an up-to-date screenshot).

khmer-on-android-render-bug

Get the keyboard + source

Despite these limitations, the keyboard should be usable for typing most Khmer text today, as least on iOS. It certainly covers most of my needs as a language learner today. As such, I’ve made it available for install into Keyman for iPhone and iPad or Keyman for Android.

First, install the app from the appropriate link above. Then click the link below to add the keyboard to your device. I haven’t included a font in the keyboard, so you’ll need a device which already supports rendering the script.

Install the keyboard (version 1.1)

Install the keyboard (version 1.2)

New! Install the Khmer Angkor beta keyboard (version 1.0) – by Makara Sok

I welcome any feedback, of course!

The source of the keyboard is available on GitHub at https://github.com/mcdurdin/experimental-keyboards.

Windows 10 shows a blank screen after login

Quick post for future reference and in case it helps other users.

Got to work this morning, and couldn’t login to my Windows 10 machine. Windows Updates had been installed overnight (oh dear).

The symptoms were slightly different to most of the other reports I’ve seen online (Google Windows 10 blank screen after login). I could login on one account all fine, but my primary account would just go to a black screen (with cursor). Waited quite some time (10 minutes or more) with no change. Rebooted. No change.

Pressing Ctrl+Alt+Del would bring up Windows Security, and from there I could switch user or go back to the lock screen, but attempting to log out would fail: the “wait for logout” spin appeared but never logged out. Other options from that screen had no effect.

I vaguely tried multi monitor options (Windows+P key combination), typing password into the blank screen, and other magic mummery without effect (and without expectation of success).

Eventually I logged in on the working account, turned off Windows firewall, enabled Remote Registry, and got to work with SysInternals tools. I logged back out of that account and used the command pslist \\machine -t from a working computer to see what processes were running.


pslist v1.3 - Sysinternals PsList
Copyright (C) 2000-2012 Mark Russinovich
Sysinternals - www.sysinternals.com

Process information for <machine>:

Name                             Pid Pri Thd  Hnd      VM      WS    Priv
Idle                               0   0   4    0      64       4       0
  System                           4   8 141 1217  393012  271644    1560
    smss                         316  11   3   49    4864     592     376
csrss                            428  13  11  415  163348    3240    1620
wininit                          500  13   4   91   43984    2792    1232
  services                       572   9   9  294   35020    7900    4648
    svchost                      304   8  35  956  184400   23016   13240
    svchost                      356   8  17  321   82168    6432    3248
    svchost                      496   8  26 1043  204540   24268   20616
    svchost                      748   8  30  773   87432   16040   10172
      ShellExperienceHost       7032   8  44  876  369236   35880   58392
      WmiPrvSE                  8876   8   6  144   33264    8172    1880
      SearchUI                  9820   8  27  900  362948   37788   48896
    svchost                      812   8  18  528   59364   11544    9764
    svchost                      936   8  52 1346 1417096   26592   22524
    svchost                      968   8  78 4722 1321944   71912  797808
      taskhostw                 3952   8   8  291  275572   10268    6492
    svchost                     1064   8  37 1130  719392   25880   12960
      WUDFHost                  1816   8   8  476   38964    3852    2212
      dasHost                   2084   8  13  302   37880    7968    3764
    mvbtrcsvcx64                1108   8   3  107   58624    1608    1124
    svchost                     1344   8  14  527  217200   29352   56716
    svchost                     1348   8  29  605  167772   23016   18484
    spoolsv                     1792   8  24  576   91808   13972   11384
    svchost                     2156   8  48  485 1200004   19240    9308
    MsMpEng                     2268   8  45 1122  523848  127360  144964
    svchost                     2684   8  15 3980  303696   53068   95100
    NisSrv                      3084   8  10  287   71416   10276   23812
    SearchIndexer               5800   8  50 1045  434516   77688   95752
      SearchProtocolHost        9568   4  10  353   74552   12456    2240
      SearchFilterHost          9856   4   4  114   35096    6560    1208
    officeclicktorun            8500   8  18 3570  213028   27932   48116
    svchost                    11928   8   5  128   23372    6500    1276
  lsass                          580   9  14 1272   58540   17008    9876
csrss                           3356  13  15  634  160220   16276    2520
winlogon                        3396  13   6  229   65460   12380    2472
  dwm                           3520  13  12  332  282440   34416   62088

Nothing obviously wrong that I could see. I then ran pskill \\machine 3396 to kill winlogon, in the hope that at least that would cause the account to log out.

And of course it did.

The surprise was that after that I was able to login…

Unfortunately, I have not yet traced what caused the lock but at least this was an answer for me, which hopefully may help someone else one day.

Possibly related: I did find this in the Event Log:

Application pop-up: ShellExperienceHost.exe – Application Error : The instruction at 0x00007FFE569D50CB referenced memory at 0x0000000000000000. The memory could not be read.

And, an unhelpful error from win32k:

The description for Event ID 267 from source Win32k cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

The specified resource type cannot be found in the image file

Three things I did last Friday

I took the day off last Friday. It was the first warm day of spring. The weather in the morning was absolutely perfect: still, bright warm sun, without being too hot as the Tasmanian sun often is, and crisp, clear, clean air to breathe.

A Walk to Sphinx Rock

I spent most of the day with my little boy Peter, and we explored the mountain. After his sisters left to go to school, and mum to go to work, he and I grabbed some snacks and drove to The Springs. We were going to walk to Sphinx Rock, have some morning tea there, and then maybe walk a bit further. The day was glorious and we didn’t feel the need to be too specific.

Bushwalking at a two-year-old’s pace can be frustrating. After a minute, Peter found a great stick, and so we had to stop at each rock and each tree to give these trees and rocks a poke. Patience is good for the soul. Peter had many questions about the drains on the side of the path, and the leaves, and the forest. The track from The Springs to Sphinx Rock (Lenah Valley Track) is well formed and very accessible to little legs.

Eventually we made it to the rock, which is gated and seriously signposted for danger. I think we counted 5 signs warning us that THERE BE CLIFFS. We sat down in the early spring sun and basked and ate our morning tea.

The view from Sphinx Rock

We took our time there. I can’t remember when I have last enjoyed sitting in the sun so much. Most of the time, I just get burnt! I was of course pretty careful with Peter, but he was very sensible and stayed well away from the edge.

The cliff Peter and The Mountain Peter and The ViewPeter watches the mountain Peter drinks water with the view

After that, we weren’t ready to head back to the car, so we decided to climb up the Sawmill Track to the Organ Pipes Track. This was a much more challenging track, so Peter traversed it on my shoulders.

Sawmill Track

We crossed the road and eventually made it up to the 1000m mark, about where the Organ Pipes Track runs.

The Organ Pipes

On the Organ Pipes Track we were excited to find snow! Peter was very pleased and ate a nice handful.

Snow! Eat some snow!

After that, he spent some time on my shoulders, and some walking, and climbing over rocks, as we made our way back to the car.

Near The Springs, we saw a triangular shaped building. I expected it to be a utility building of some kind but it turned out to be a Cosmic Ray Observatory. That’s pretty cool.

This building turned out to be interesting

Finally, we treated ourselves to a coffee and a babycino.

Babycino time

A Bike Ride to The Chalet

Later that day, we decided to go on a ride, from our house in Ridgeway. The weather was still good, so we rode back up the mountain, past The Springs, all the way to The Chalet, at 1000m, at the far end of the Organ Pipes Track. It was a bit funny to be riding past the same areas we’d walked that morning, but the mountain was nearby and was beckoning.

Riding up the mountain is hard work with a baby seat on the front! The seat stops you from getting out of the saddle, and you have to ride with your knees out like you are a cowboy or something, which gets a bit uncomfortable. I was glad to pause at The Springs to get some food out of the pack for Peter, and even more glad after the last 100m to The Chalet, where the wind picked up and ever so nearly brought us to a complete stop!

At The Chalet Our steed

After a few minutes at The Chalet, we decided that it was getting too cold to go further up the mountain, and we rolled slowly back down. We stopped at The Springs to give the brakes a chance to cool, and explored the hut. Peter enjoyed the crackling log fire.

Warming hands at The Springs

We had a slow and safe descent, and arrived home just in time to go and collect the girls from school.

An Indoor Rock Climbing Session

My legs were very tired after this effort, but I had promised my eldest daughter that I’d take her rock climbing that evening. I had not been climbing for a long time. Hobart has a great rock climbing place, Rock It Climbing Centre, and so the two of us headed down that evening to get in some climbs.

No pictures of the evening. We had a good time but my body was ready to pack it in by the end! Four or five climbs each, some bouldering, and for my daughter, some tunneling. We mutually agreed that our muscles could take no more after an hour and a half, and we finished off the evening with a hot chocolate at the State Cinema cafe.

A great day. I was very sore on Saturday though!

Twenty Dismembered Men

Yesterday, I received in the mail twenty dismembered men. This may sound horrific to you, but I was pleased. They’d only cost me $4 on eBay. What a bargain!

I opened the package and here is what I found.

The packageWhat could all these little men be for?

My first mission was clearly to re-member these men.  I mean, to assemble them. After just five minutes, things were starting to come together. Literally.

The men assemble

But then I ran into a problem. The Wizard, Gandalf, had clearly been involved in some seriously wonky magic, because he had two right legs. And Vincent’s hand kept falling off. Vincent was an artist.

Gandalf and Vincent

As I assembled each man, the others, already able-bodied, assisted me.

The men assist in assembly

Sven was very clever, and assembled himself.

The self assembling man

Finally, I had a team. I waited while they talked amongst themselves.

The men gather around

The men are in discussion

Then I asked them to gather. The men had a mission.

The men are ready for their mission

The test lab at my office has many devices. But it also has a mess of cables, held in place with Blu-Tack. That gets a bit yucky and sticky. The mission of these men was to resolve this cable situation.

The mission

This is not an unusual mission for men of this calibre. But the stalwart men readily accepted their mission in all its orthodoxy.

The stalwart men listen

Thus I led them to the situation. They put their heads together and figured out a game plan.

The men examine the situation grimly

Shortly thereafter, the cable tidy was complete. Well done men!

Jake and Albert with cable number 1 Jimmy tells Alphonse how to hold the cable while Cam watches for danger Gandalf and Vincent are just hangers-on, along with Clint and Bob The Nokia cable required both Sven and Michael, but Ivan was able to handle the Samsung cable on his own A fight nearly broke out between Robert and his mate George Tim and Tom worked together well, as Peter leaned on the cable Matt just listened as Dave gave instructions on the right way to hold the cable Fred didn't care. He was happy to have a job at last

Gandalf was pretty unhappy about his leg.

Gandalf is not very happy about his leg situation

The men complete their mission

 

The case of the unexplained: When no Windows apps (aka Windows Store apps) will start

Yes, I’m shamelessly stealing @MarkRussinovich‘s blog series title for this post!

One of my machines running Windows 10 here would not run any Windows apps (formerly known as Universal apps, Metro, Modern UI and I’m not sure if I’ve missed any names). Classic desktop apps would work fine.

Finding Microsoft Edge in the Start Menu

I’d click the link to the app in the Start Menu (those missing names may be another case to chase!), and the app would flash onto the screen and then almost immediately disappear.

Microsoft Edge starts and immediately exits

Of course this was more than a little bit frustrating, with no hints as to how to resolve the problem. Checking event logs and reliability provided no pointers towards solutions.

Reliability Monitor is aware of the problem but doesn't know how to fix it

After a couple of pointless web searches (“Edge won’t start”, what was I thinking?), and a bizarre side trip into deep conspiracy theories on Microsoft forums, I realised it was time to break out Procmon to try and trace the problem.

Procmon to the rescue

Procmon lets you watch and log events happening on your file system, registry and network in real time. Running Procmon for even just a minute will often generate hundreds of thousands of events, so it’s fantastic that it includes a powerful set of filtering tools to help you locate specific events.

I started Procmon, and then started, or tried to start Microsoft Edge. After it fell over again, I went back into Procmon, stopped the trace (Ctrl+E), and started to filter the 452,626 events that had been captured in those few seconds.

Initial results in procmon

Procmon’s initial setup includes some filters that exclude events that are of less interest to mere mortals, such as reading and writing to the pagefile, or events caused by Procmon itself. Those default filters cut the results down by 55% to begin with!

While you can use the Filter dialog (Ctrl+L) to manually enter filters, and I often do this for complex filtering, it’s often faster to simply right-click on a cell that you don’t want to see again, and select Exclude <value> from the popup menu. Conversely, if you want to focus on that particular value, select Include <value>.

First, I excluded some processes I wasn’t interested in, such as Explorer.exe, and then excluded a number of different values from the Result column. I was really looking for the ACCESS DENIED result, because that’s probably the most common result that causes apps to crash. I ended up with the following filters on the Result column:

Filtering results to find the errors I am looking for in Procmon

Now, there were few enough events (only 1,625 of them) that I could scan through quickly and hopefully spot something going wrong. And, again Procmon found the answer!

Finding the point where Microsoft Edge encounters an error in Procmon

There you can see MicrosoftEdge.exe receiving an ACCESS DENIED result when trying to read the folder C:\Users\mcdurdin\AppData\Local\Packages\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC\Microsoft\Windows\1605653898. Shortly thereafter, we see the WerFault.exe process which is the Windows Error Reporting process that started after Edge decided to crash.

Note: it’s merely serendipitous that WerFault.exe is visible in the filtered results; remember that there are probably thousands of additional events between the highlighted ACCESS DENIED event and the start of the WerFault.exe process, and the only reason it is visible at all is that WerFault itself had received ACCESS DENIED and other results from its own events!

I could alternatively have looked at the process tree (Ctrl+T) to find when the WerFault.exe process started (or the MicrosoftEdge.exe process had stopped) and traced back from there. But usually I find filtering to be a faster way of finding the specific issue.

What’s wrong with this folder?

Now I wanted to figure out what was wrong with this folder. Here’s what I saw on this machine:

C:\Users\mcdurdin\AppData\Local\Packages>icacls Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC
Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
                                         BUILTIN\Administrators:(I)(OI)(CI)(F)
                                         TAVULTESOFT\mcdurdin:(I)(OI)(CI)(F)

Successfully processed 1 files; Failed processing 0 files

And this is what I saw on a machine where Edge was working:

C:\Users\mcdurdin\AppData\Local\Packages>icacls Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC
Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC S-1-15-2-3624051433-2125758914-1423191267-1740899205-1073925389-3782572162-737981194:(OI)(CI)(F)
                                         NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
                                         BUILTIN\Administrators:(I)(OI)(CI)(F)
                                         TAVULTESOFT\mcdurdin:(I)(OI)(CI)(F)
                                         Mandatory Label\Low Mandatory Level:(OI)(CI)(NW)

Successfully processed 1 files; Failed processing 0 files

I saw two differences: a missing S-1-15-2-… entry and a missing Low Mandatory Level entry. Now, that S-1-15-2-… entry is an App Package SID. I checked a few other installed app packages, and they were all missing the relevant security settings on this machine. So it wasn’t specific to Edge, but was a general issue on my computer.

At this point, I did find a relevant discussion on Microsoft’s forums that had some answers, but did not solve the general case that I was experiencing.

I  have not been able to find the root cause of this. Lost in the deep dark mists of time it is.

Fixing the problem

To fix the Low integrity level was a pretty straightforward command, run from a command prompt in the %LOCALAPPDATA%\Packages folder:

for /d %d in (*) do icacls %d\AC /setintegritylevel (OI)(CI)L

However, determining the correct SID to add to each folder was a little more work. It turns out that in the registry, there is a mapping between the app’s moniker (so, in this case, the folder names) and the relevant SID at HKEY_CLASSES_ROOT\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppContainer\Mappings. Learn more.
The AppContainer Mappings registry, showing the relationship between SID and Moniker
I sucked a list of those SIDs into a text file with the following command:

reg query "HKCR\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppContainer\Mappings" > m.txt

That gave me a file that looked like this, shown as an image just because:

The contents of m.txt

From there, I wanted to extract just the last part of each key:

for /f "tokens=9 delims=\" %i in (m.txt) do echo %i >> n.txt

Now n.txt looked like:

The contents of n.txt

Now to take each of those and map it to its moniker, and from there update the security on the folder accordingly. That command turned out to be a bit more hoopy.

for /f %a in (n.txt) do for /f "tokens=2*" %b in ('reg query "HKCR\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppContainer\Mappings\%a" /V "Moniker" 2^>NUL ^| FIND "REG_SZ"') DO icacls "%c\AC" /grant *%a:(OI)(CI)(F)

Putting that all together, in a batch file (I’ve combined the integrity level setting and SID grant in this script):

@echo off
del n.txt
reg query "HKCR\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppContainer\Mappings" > m.txt
for /f "tokens=9 delims=\" %%i in (m.txt) do echo %%i >> n.txt
for /f %%a in (n.txt) do for /f "tokens=2*" %%b in ('reg query "HKCR\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppContainer\Mappings\%%a" /V "Moniker" 2^>NUL ^| FIND "REG_SZ"') DO icacls "%%c\AC" /grant *%%a:(OI)(CI)(F) /setintegritylevel (OI)(CI)L

And yay, now Edge starts!

Yay, Microsoft Edge starts now!

One of these days I’ll have to get more into PowerShell, which would probably make some of these scripts a lot easier!

With thanks to examples from http://www.robvanderwoude.com/ntregquery.php which saved a lot of fuss (but warning if you copy and paste: the examples use the wrong caret character, ˆ instead of ^).

GUI Info Utility for Windows

A very quick one today. GUIInfo is a tiny little utility that has a narrow purpose: it shows, in realtime, the current active, focus, foreground and capture window information for Windows.

Why this utility? Debugging focus issues is always frustrating. Attempting to observe current window focus information in a debugger results in the focus changing (to the debugger, of course). You can work around this – use remote debugging, or add logging in the debugger – but it’s a hassle.

Debugging focus issues would make Werner Heisenberg feel right at home.

Screenshot

GUIInfo Screenshot

Features

  • Updates 10 times a second so changes are reflected promptly.
  • Changes are highlighted in green, and slowly fade back to black.
  • Hovering over bold window labels will highlight the relevant window on the screen.
  • Topmost, so you can see the information without needing to try and keep the window visible.
  • Open Source – written in Delphi, MIT license.

Download and Source

An update for EncodeURIComponent

Way way back in the dark ages of Delphi XE2, I wrote a function to encode components of a URI. Now, this function has been updated for use on mobile platforms, by Nicolas Dusart, and I quote Nicolas:

I had to make some modifications on it to compile for the mobile platforms, as the strings are 0-based on these platforms.

I also modified it to escape non-ASCII characters using their UTF-8 encoding as the standards advices. For multi-bytes characters, each byte is percent-encoded as usual.

Here’s the code, maybe it could interests you and the future readers of that article 🙂

And here’s Nicolas’s updated function in all its glory:

function EncodeURIComponent(const ASrc: string): string;
const
  HexMap: string = '0123456789ABCDEF';

  function IsSafeChar(ch: Byte): Boolean;
  begin
    if (ch >= 48) and (ch <= 57) then Result := True    // 0-9
    else if (ch >= 65) and (ch <= 90) then Result := True  // A-Z
    else if (ch >= 97) and (ch <= 122) then Result := True  // a-z
    else if (ch = 33) then Result := True // !
    else if (ch >= 39) and (ch <= 42) then Result := True // '()*
    else if (ch >= 45) and (ch <= 46) then Result := True // -.
    else if (ch = 95) then Result := True // _
    else if (ch = 126) then Result := True // ~
    else Result := False;
  end;

var
  I, J: Integer;
  Bytes: TBytes;
begin
  Result := '';
    
  Bytes := TEncoding.UTF8.GetBytes(ASrc);
    
  I := 0;
  J := Low(Result);

  SetLength(Result, Length(Bytes) * 3); // space to %xx encode every byte

  while I < Length(Bytes) do
  begin
    if IsSafeChar(Bytes[I]) then
    begin
      Result[J] := Char(Bytes[I]);
      Inc(J);
    end
    else
    begin
      Result[J] := '%';
      Result[J+1] := HexMap[(Bytes[I] shr 4) + Low(ASrc)];
      Result[J+2] := HexMap[(Bytes[I] and 15) + Low(ASrc)];
      Inc(J,3);
    end;
    Inc(I);
  end;
    
  SetLength(Result, J-Low(ASrc));
end;

Many thanks, Nicolas 🙂

Fixing the incorrect client size for Delphi VCL Forms that use styles

Delphi XE2 and later versions have a robust theming system that has a frustrating flaw: the client width and height are not reliably preserved when the theme changes the border widths for dialog boxes.

For forms that are sizeable this is not typically a problem, but for dialogs laid out statically this can look really ugly, as shown in this Stack Overflow question.

The problem in pictures

Here’s a little form, shown in the Delphi form designer. I’ve placed 4 buttons right in the corners of the form. I’m going to populate the Memo with notes on the form size at runtime.

Design time form with four buttons at corners

When I have no custom style set to the project (i.e. “Windows” style), I can run on a variety of platforms and see the buttons are where they should be. Shown here on Windows 10, Windows 7 and Windows XP (just because):
Windows theme form on Windows 10Windows theme form on Windows 7

Windows theme form on Windows XP

But when I apply a custom style to the project — I chose “Glossy” — then my dialog appears like so, instead:

Glow theme form on Windows 7

You’ll note that the vertical is adjusted but the horizontal is not: Button2 and Button4 are now chopped off on the right. Because we are using themes, the form looks identical on all platforms.

This problem has not been addressed as of Delphi XE8.

The workaround

For my needs, I found a workaround using a class helper, which can be applied to the forms which need to maintain their design-time ClientWidth and ClientHeight. This is typically the case for dialog boxes.

This workaround should be used with care as it has been designed to address a single issue and may have side effects.

  • It will trigger resize events at load time
  • Setting AutoScroll = true means that ClientWidth and ClientHeight are not stored in the form .dfm, and so this does not work.
  • It may not address other layout issues such as scaled elements scaling wrongly (I haven’t tested this).
type
  TFormHelper = class helper for Vcl.Forms.TCustomForm
  private
    procedure RestoreDesignClientSize;
  end;

procedure TfrmTestSize.FormCreate(Sender: TObject);
begin
  RestoreDesignClientSize;
end;

{ TFormHelper }

procedure TFormHelper.RestoreDesignClientSize;
begin
  if BorderStyle in [bsSingle, bsDialog] then
  begin
    if Self.FClientWidth > 0 then ClientWidth := Self.FClientWidth;
    if Self.FClientHeight > 0 then ClientHeight := Self.FClientHeight;
  end;
end;

After adding in this little snippet, the form is now restored to its design-time size, like thus:

Fixed glow theme form on Windows 7

Success 🙂

Concatenating strings in SQL Server, or undefined behaviour by design

We just ran into a funny problem here, using a “tried and true” technique in SQL Server to concatenate strings. I use the quotes advisedly. This technique is often suggested on blogs and sites such as Stack Overflow, but we found out (by painful experience) that it is not to be relied on.

Update, 9 Mar 2016: Bruce Gordon from Webucator has turned this into a great little 5 minute video. Thanks Bruce! I don’t know anything much about Webucator, but they are doing some good stuff with creating well-attributed videos about blog posts such as this one and apparently they do SQL Server training.

The problem

So, given the following setup:

CREATE TABLE BadConcat (
  BadConcatID INT NOT NULL,
  Description NVARCHAR(100) NOT NULL,
  SortIndex INT NOT NULL
  CONSTRAINT PK_BadConcat PRIMARY KEY CLUSTERED (BadConcatID)
)
GO

INSERT BadConcat 
  SELECT 1, 'First Item', 1 union all
  SELECT 2, 'Second Item', 2 union all
  SELECT 3, 'Third Item', 3
GO

We need to concatenate those Descriptions. I have avoided fine tuning such as dropping the final comma or handling NULLs for the purpose of this example. This example shows one of the most commonly given answers to the problem:

DECLARE @Summary NVARCHAR(100) = ''

SELECT @Summary = @Summary + ec.Description + ', '
FROM BadConcat ec
ORDER BY ec.SortIndex 

PRINT @Summary

And we get the following:

First Item, Second Item, Third Item, 

And that works fine. However, if we want to include a WHERE clause, even if that clause still selects everything, then we suddenly get something weird:

SET @Summary = ''

SELECT @Summary = @Summary + ec.Description + ', '
FROM BadConcat ec
WHERE ec.BadConcatID in (1,2,3)
ORDER BY ec.SortIndex 

PRINT @Summary

Now we get the following:

Third Item, 

What? What has SQL Server done? What’s happened to the first two items?

You’ll probably do what we did, which is to go through and make sure that you are selecting everything properly, which we are, and eventually come to the conclusion that “there must be a bug in SQL Server”.

The answer

It turns out that this iterative concatenation is unsupported functionality. Microsoft Knowledge Base article 287515 states:

You may encounter unexpected results when you apply any operators or expressions to the ORDER BY clause of aggregate concatenation queries.

Now, at first glance that does not directly apply. But we can extrapolate from that, as Microsoft developer support have done, in response to a bug report on SQL Server, to learn that:

The variable assignment with SELECT statement is a proprietary syntax (T-SQL only) where the behavior is undefined or plan dependent if multiple rows are produced

And again, in response to another bug report:

Using assignment operations (concatenation in this example) in queries with ORDER BY clause has undefined behavior. This can change from release to release or even within a particular server version due to changes in the query plan. You cannot rely on this behavior even if there are workarounds.

Some alternative solutions are given, also, in that second report:

The ONLY guaranteed mechanism are the following:

1. Use cursor to loop through the rows in specific order and concatenate the values
2. Use for xml query with ORDER BY to generate the concatenated values
3. Use CLR aggregate (this will not work with ORDER BY clause)

And the article “Concatenating Row Values in Transact-SQL” by Anith Sen goes through some of those solutions in detail. Sadly, none of them are as clean or as easy to understand as that original example.

Another example is given on Stack Overflow, which details how to safely use XML PATH to concatenate, without breaking on the XML special characters &, < and >. Applying that example into my problem code given above, we should use the following:

SELECT @Summary = (
  SELECT ec.Description + ', ' 
  FROM BadConcat ec 
  WHERE ec.BadConcatID in (1,2,3)
  ORDER BY ec.SortIndex 
  FOR XML PATH(''), TYPE
).value('.','varchar(max)')

PRINT @Summary

Voilà.

First Item, Second Item, Third Item,