Indy, TIdURI.PathEncode, URLEncode and ParamsEncode and more

Frequently in Delphi we come across the need to encode a string to stuff into a URL query string parameter (as per web forms).  One would expect that Indy contains well-tested functions to handle this.  Well, Indy contains some functions to help with this, but they may not work quite as you expect.  In fact, they may not be much use at all.

Indy contains a component called TIdURI.  It contains, among other things, the member functions URLEncode, PathEncode, and ParamsEncode. At first glance, these seem to do what you would need.  But in fact, they don’t.

URLEncode will take a full URL, split it into path, document and query components, encode each of those, and return the full string.  PathEncode is intended to handle the nuances of the path and document components of the URL, and ParamsEncode handles query strings.

Sounds great, right?  Well, it works until you have a query parameter that has an ampersand (&) in it.  Say my beloved end user want to search for big&little.  It seems that you could pass the following in:

s := TIdURI.URLEncode('http://www.google.com/search?q='+SearchText);

But then we get no change in our result:

s = 'http://www.google.com/search?q=big&little';

And you can already see the problem: little is now a separate parameter in the query string.  How can we work around this?  Can we pre-encode ampersand to %26 before you pass in the parameters?

s := TIdURI.URLEncode('http://www.google.com/search?q='+ReplaceStr(SearchText, '&', '%26'));

No:

s = 'http://www.google.com/search?q=big%25%26little';

And obviously we can’t do it ourselves afterwards, because we too won’t know which ampersands are which.  You could do correction of ampersand by encoding each parameter component separately and then post-processing the component for ampersand and other characters before final assembly using ParamsEncode. But you’ll soon find that it’s not enough anyway.  =, / and ? are also not encoded, although they should be.  Finally, URLEncode does not support internationalized domain names (IDN).

Given that these functions are not a complete solution, it’s probably best to avoid them altogether.

The problem is analogous to the Javascript encodeURI vs encodeURIComponent issue.

So to write your own…  I haven’t found a good Delphi solution online (and I searched a bit), so here’s a function I’ve cobbled together (use at your own risk!) to encode parameter names and values. You do need to encode each component of the parameter string separately, of course.

function EncodeURIComponent(const ASrc: string): UTF8String;
const
  HexMap: UTF8String = '0123456789ABCDEF';

  function IsSafeChar(ch: Integer): Boolean;
  begin
    if (ch >= 48) and (ch <= 57) then Result := True // 0-9
    else if (ch >= 65) and (ch <= 90) then Result := True // A-Z
    else if (ch >= 97) and (ch <= 122) then Result := True // a-z
    else if (ch = 33) then Result := True // !
    else if (ch >= 39) and (ch <= 42) then Result := True // '()* 
    else if (ch >= 45) and (ch <= 46) then Result := True // -.
    else if (ch = 95) then Result := True // _
    else if (ch = 126) then Result := True // ~
    else Result := False;
  end;
var
  I, J: Integer;
  ASrcUTF8: UTF8String;
begin
  Result := '';    {Do not Localize}

  ASrcUTF8 := UTF8Encode(ASrc);
  // UTF8Encode call not strictly necessary but
  // prevents implicit conversion warning

  I := 1; J := 1;
  SetLength(Result, Length(ASrcUTF8) * 3); // space to %xx encode every byte
  while I <= Length(ASrcUTF8) do
  begin
    if IsSafeChar(Ord(ASrcUTF8[I])) then
    begin
      Result[J] := ASrcUTF8[I];
      Inc(J);
    end
    else
    begin
      Result[J] := '%';
      Result[J+1] := HexMap[(Ord(ASrcUTF8[I]) shr 4) + 1];
      Result[J+2] := HexMap[(Ord(ASrcUTF8[I]) and 15) + 1];
      Inc(J,3);
    end;
    Inc(I);
  end;

  SetLength(Result, J-1);
end;

To use this, do something like the following:

function GetAURL(const param, value: string): UTF8String;
begin
  Result := 'http://www.example.com/search?'+
    EncodeURIComponent(param)+
    '='+
    EncodeURIComponent(value);
end;

Hope this helps. Sorry, I haven’t got an IDN solution in this post!

Updated 15 Nov 2018: Fixed bug with handling of space (should output %20, not +).

Understanding and correcting interface reference leaks in Delphi’s Vcl.OleCtrls.pas

Update 21 Sep 2015: This bug has been fixed in Delphi 10 Seattle.

I have spent quite some time recently tracing a memory leak in a Delphi application.  It is quite a large application and makes a lot of use of embedded MSHTML (TWebBrowser and TEmbeddedWB) controls for presentation.  Somehow, somewhere, we were leaking memory: users were reporting that after a few hours of use, the application slowed down, and checking Task Manager certainly reflected excessive memory usage.

The normal procedure to reproduce the memory leak was followed, including using tools such as AQtime and other debug logging tools.  However, no leaks were detected using these tools, although we could see the memory usage increasing in Task Manager on our test machines.  This suggested the memory we were leaking was not allocated by Delphi code: i.e. it was Windows or a 3rd party DLL.  This doesn’t mean, of course, that it wasn’t our fault — just that it wasn’t allocated directly from Delphi source!

At this point, I was asked to trace this issue further.  I ran Performance Monitor with a couple of key counters: Handle Count and Working Set.  Running the test in question (involving opening and closing a window with an embedded web browser control) showed a gradual increase in both handle count and working set size.  However, Performance Monitor unfortunately does not include a user handle (i.e. window handles) counter.  It was when I noticed in Process Explorer that user handles were also increasing that I got my first break in tracing the cause.

It turned out that the embedded web browser window was not always being destroyed when its parent window was.  This window had the class name “Internet Explorer_Server”.

With a bit more tracing, I found that the trigger was the Document or Application properties.  If the Document property of the TWebBrowser control was ever referenced, the window was never destroyed (note, there are also some other properties that trigger the same behaviour — look for properties returning type IDispatch).

This set me to researching the Document property.  It looks like this:

property Document: IDispatch index 203 read GetIDispatchProp;

Looking at GetIDispatchProp in Vcl.OleCtrls.pas, we see the following code:

function TOleControl.GetIDispatchProp(Index: Integer): IDispatch;
var
  Temp: TVarData;
begin
  GetProperty(Index, Temp);
  Result := IDispatch(Temp.VDispatch);
end;

And here some alarm bells go off.  Delphi, rather nicely, manages all the reference counting on interfaces.  This works pretty smoothly, until you trick the compiler by casting other types to interfaces.  Here the code in question is triggering an invisible call to IntfCopy in the line:

Result := IDispatch(Temp.VDispatch);

The IntfCopy function internally calls  _AddRef on the object, but because of the cast to IDispatch from a Pointer, this new reference is never released.  The fix is to change the function (and the similar GetIUnknownProp function) to:

function TOleControl.GetIDispatchProp(Index: Integer): IDispatch;
var
  Temp: TVarData;
begin
  GetProperty(Index, Temp);
  Pointer(Result) := Temp.VDispatch;
end;

function TOleControl.GetIUnknownProp(Index: Integer): IUnknown;
var
  Temp: TVarData;
begin
  GetProperty(Index, Temp);
  Pointer(Result) := Temp.VUnknown;
end;

By casting this way, we avoid the call to IntfCopy and hence the call to _AddRef.  Alternatively, you could have called _Release on Result, but this would require another test to ensure that Result wasn’t nil (and also, of course, redundant _AddRef and _Release calls).

It turns out that this problem was identified way back in 1999 and the workaround has been commonly referenced since then.  So I am not taking credit for the fix here!  And yet it is still unresolved in Delphi XE2 — and so still causing trouble for Delphi programmers today!  There are no clear references to the problem in QualityCentral that I could find (that’s about to change!)

But don’t stop reading yet!

“Now my app crashes”

There are frequent complaints online that this fix results in crashes.  This is because other developers have engineered fixes to this reference leak in places where these GetIDispatchProp and/or GetIUnknownProp calls are made, rather than where the problem actually occurs.  I have found this in TEmbeddedWB.  TEmbeddedWB is a web browser hosting component that takes up where TWebBrowser leaves off, and it does fix a lot of the limitations of TWebBrowser.

But here are the places in the TEmbeddedWB source that you’ll need to “unfix” once you fix the root problem in Vcl.OleCtrls.pas:

EmbeddedWB.pas [2603]: (procedure TEmbeddedWB.SetUserAgentInt): Delete _Release call
EwbCore.pas [1367]: (procedure TCustomEmbeddedWB.SetDesignMode): Delete _Release call
EwbCore.pas [1404]: (procedure TCustomEmbeddedWB.SetDownloadOptions): Delete _Release call
EwbCore.pas [1468]: (procedure TCustomEmbeddedWB.SetUserInterfaceOptions): Delete _Release call
EwbTools.pas [1015]: (function GetBmpFromBrowser): Delete _Release call
EwbTools.pas [3164]: (function InvokeCMD): Delete _Release call

Note, you’ll also see an experimental fix buried — and disabled — in the TEmbeddedWB code (but without fixing the lines above), and without a whole lot of documentation as to why!

I have created a Quality Central report, along with a test case and example fix.  I also checked the Delphi VCL source, and about 30 other components that we use, and found no more calls to _Release relating to this issue.

Hobart Dirty Dozen ride

This is just a small note that my post about the Hobart Dirty Dozen — a kinda epic ride up a bunch of Hobart’s steepest hills with my mates Dan and Iain — has been kindly published on The Climbing Cyclist’s blog.

Climbing Brent St

If you want to see just the relevant climbs for the Dirty Dozen on the Strava ride, drag the link “Show only the Hobart Dirty Dozen Climbs” to your bookmarks toolbar, then click the bookmark when you are on the Dirty Dozen Strava ride page:

Show only the Hobart Dirty Dozen Climbs

Rainforest Walk

Once upon a time, there lived a child and a daddy. The child’s name was Hannah.

Daddy said to her, “let’s go to Myrtle Forest”.

“Yes,” said Hannah.  Hannah’s sister Bethany came too.

Daddy saw a ruin.  Hannah wanted to go in the trail to the ruin, but it had a gate in front of it, which meant that you couldn’t go in it.

They saw heaps of ferns and waterfalls.

 They started exploring the waterfalls and ferns.

They saw an old fern stump, and it had little green things growing on top of it, and it was mossy all over!

There was a pretty waterfall on one side of a bridge, and on the other side there was a creek which went under the bridge.

They saw some snow trees, which are trees that are kind of aqua, and if you are far away, they look like they are covered with snow.  Hannah had a small sprig off a snow tree.  She made a wish.  She wished that she could fly.  The wish hasn’t come true yet.

Hannah and Bethany hid in some hiding places.

There were beautiful views and lots of trees and creeks.

Here is a photo of the creek hidden behind the fronds of the ferns.

— The End —

Corrie: why is she a hero?

“Father sat down on the edge of the narrow bed. “Corrie,” he began gently, “when you and I go to Amsterdam-when do I give you your ticket?”
I sniffed a few times, considering this.
“Why, just before we get on the train.”
“Exactly. And our wise Father in heaven knows when we’re going to need things, too. Don’t run out ahead of Him, Corrie. When the time comes that some of us will have to die, you will look into your heart and find the strength you need-just in time.”
― Corrie Ten Boom, The Hiding Place

Corrie Ten Boom suffered more than most of us ever will. And yet she was full of love. She had learned to live in the grace of God. Through the most devestating of circumstances, she depended on Him and His grace was sufficient for her needs.
If that bleak, black time comes for us, I guess all we have to do is ask our Lord for that grace. But perhaps the hardest thing of all is to do just that: it means giving up the hate and giving the hurt to Him, just to make that simplest of requests. No revenge, just forgiveness is all that can remain. He will help us, if we just once ask Him.
Corrie Ten Boom is one of my heroes, for she did not survive the most awful of times on her own willpower, or through determination, or even by being a super holy person. She is my hero because she was able to ask God for that grace.
I pray that I may be spared such sufferings. And yet, that if such sufferings come my way that He will protect my soul and not allow it to become hardened with resentment or hate. And that I can continually ask Him for that grace freely provided by the sacrifice He already made through His son, which purges our sins, and by so doing allows us to love those who have hurt us.

Race Shape for Strava and a Bookmarklet to link them

Race Shape is a nice polished mashup for Strava that allows you to compare efforts on a segment in much greater detail than the Strava interface allows.  It can be very interesting to see how and where your opponents stole time on you in their KOM efforts.

To make it a little easier to use, I whipped up a little bookmarklet (not very well written) that you can add to your Bookmarks toolbar.  When looking at the list of segment efforts in a Strava activity, click the bookmarklet and a little Race Shape icon will appear next to each segment in the list.  You can then just click on that icon to open the segment in Race Shape!

Drag this link to your Bookmarks toolbar to install the bookmarklet: Race Shape

Here’s what it looks like — the icon is pretty subtle…:

The raw code, for those who like such things (or want to fix or improve the bookmarklet):

(function(){
this.i=function(id,e)
{
  while(e.firstChild.className == ‘raceShape’) e.removeChild(e.firstChild);
  var segment=id.match(/-([0-9]+)$/)[1];
  var activity=location.pathname.match(/\/([0-9]+)/)[1];
  var a = window.document.createElement(‘a’);
  a.className=’raceShape’;
  a.href=’http://raceshape.com/strava.redirect.php?url=http%253A%252F%252Fapp.strava.com%252Frides%252F’+activity+’%2523’+segment;
  a.target=’_blank’;
  var img = window.document.createElement(‘img’);
  img.src=’http://d1touf4erjk31x.cloudfront.net/images/brand.png’;
  img.style=’border:none;margin-right:4px’;
  img.width=’20’;
  a.appendChild(img);
  e.insertBefore(a,e.firstChild);
};
var e = document.getElementById(‘segment-efforts-table’).getElementsByTagName(‘tr’);
for(var m = 0; m < e.length; m++)
{
  if(e[m].id) { this.i(e[m].id,e[m].cells[1]); }
}
})();

My blog hit 100,000 visits a couple of days ago.  Wow…  Thank you for reading!  I don’t know how you find anything interesting in all the twaddle… 😉

Delphi XE2’s hidden hints and warnings options

There are a number of warnings available in Delphi XE2 that are not very well documented.  While you can control them in the Project options dialog, and you can turn them on using {$WARN} directives or in command line compiler options, the documentation for the warning identifiers is currently pretty piecemeal, and there is no clear link between the project options, warning directive identifiers, and numeric identifiers.

The {$WARN} compiler directive takes effect only in the unit it is in.  It overrides project settings and command line compiler flags.  To turn a warning on or off for a block of code (although you’ll have to work hard to convince me that this is an acceptable solution 99% of the time):

{$WARN SYMBOL_DEPRECATED ON}
{$WARN SYMBOL_DEPRECATED OFF}

To restore the warning to its previous setting:

{$WARN SYMBOL_DEPRECATED DEFAULT}

And finally, you can convert a warning into an error:

{$WARN SYMBOL_DEPRECATED ERROR}

To change the setting in your project, go to Project|Options, Delphi Compiler, Hints and Warnings, and expand the Output warnings setting:


Delphi’s project options dialog – hints and warnings

To change the setting in the command line compiler, use the -W flag, for example:

dcc32 -W+SYMBOL_DEPRECATED myproject.dpr
dcc32 -W-SYMBOL_DEPRECATED myproject.dpr

Or to turn the warning into an error:

dcc32 -W^SYMBOL_DEPRECATED myproject.dpr

Note, if you type this from the Windows command prompt, you will have to repeat the ^ symbol, as ^ is an escape symbol:

dcc32 -W^^SYMBOL_DEPRECATED myproject.dpr

These are the hints and warning identifiers I am aware, along with what I understand are the default settings and numeric identifiers, and links to the relevant help topics.

ID Directive ID Default Description Notes
W1000 SYMBOL_DEPRECATED OFF Symbol ‘%s’ is deprecated (Delphi)
W1001 SYMBOL_LIBRARY OFF Symbol ‘%s’ is specific to a library (Delphi)
W1002 SYMBOL_PLATFORM OFF Symbol ‘%s’ is specific to a platform (Delphi)
W1003 SYMBOL_EXPERIMENTAL ON Symbol ‘%s’ is experimental (Delphi)
W1004 UNIT_LIBRARY OFF Unit ‘%s’ is specific to a library (Delphi)
W1005 UNIT_PLATFORM OFF Unit ‘%s’ is specific to a platform (Delphi)
W1006 UNIT_DEPRECATED OFF Unit ‘%s’ is deprecated (Delphi)
W1007 UNIT_EXPERIMENTAL ON Unit ‘%s’ is experimental (Delphi)
W1008 HRESULT_COMPAT ON Integer and HRESULT interchanged
W1009 HIDING_MEMBER ON Redeclaration of ‘%s’ hides a member in the base class (Delphi)
W1010 HIDDEN_VIRTUAL ON Method ‘%s’ hides virtual method of base type ‘%s’ (Delphi)
W1011 GARBAGE ON Text after final ‘END.’ – ignored by compiler (Delphi)
W1012 BOUNDS_ERROR ON Constant expression violates subrange bounds
W1013 ZERO_NIL_COMPAT ON Constant 0 converted to NIL (Delphi)
W1014 STRING_CONST_TRUNCED ON String constant truncated to fit STRING%ld (Delphi)
W1015 FOR_LOOP_VAR_VARPAR ON FOR-Loop variable ‘%s’ cannot be passed as var parameter (Delphi)
W1016 TYPED_CONST_VARPAR ON Typed constant ‘%s’ passed as var parameter (Delphi)
W1017 ASG_TO_TYPED_CONST ON Assignment to typed constant ‘%s’ (Delphi)
W1018 CASE_LABEL_RANGE ON Case label outside of range of case expression (Delphi)
x1019 FOR_VARIABLE ON For loop control variable must be simple local variable (Delphi)
x1020 CONSTRUCTING_ABSTRACT ON Constructing instance of ‘%s’ containing abstract method ‘%s.%s’ (Delphi)
W1021 COMPARISON_FALSE ON Comparison always evaluates to False (Delphi)
W1022 COMPARISON_TRUE ON Comparison always evaluates to True (Delphi)
W1023 COMPARING_SIGNED_UNSIGNED ON Comparing signed and unsigned types – widened both operands (Delphi)
W1024 COMBINING_SIGNED_UNSIGNED ON Combining signed and unsigned types – widened both operands (Delphi)
x1025 UNSUPPORTED_CONSTRUCT ON Unsupported language feature: ‘%s’ (Delphi)
x1026 FILE_OPEN ON File not found ‘%s’ (Delphi) Despite being listed as a warning, turning this off appears to have no effect.
F1027 FILE_OPEN_UNITSRC ON Unit not found ‘%s’ or binary equivalents (%s) (Delphi)
x1028 BAD_GLOBAL_SYMBOL ON Bad global symbol definition ‘%s’ in object file ‘%s’ (Delphi)
W1029 DUPLICATE_CTOR_DTOR ON Duplicate %s ‘%s’ with identical parameters will be inacessible from C++ (Delphi)
x1030 INVALID_DIRECTIVE ON Invalid compiler directive – ‘%s’ (Delphi)
W1031 PACKAGE_NO_LINK ON Package ‘%s’ will not be written to disk because -J option is enabled (Delphi)
W1032 PACKAGED_THREADVAR ON Exported package threadvar ‘%s.%s’ cannot be used outside of this package (Delphi)
W1033 IMPLICIT_IMPORT ON Unit ‘%s’ implicitly imported into package ‘%s’ (Delphi)
W1034 HPPEMIT_IGNORED ON $HPPEMIT ‘%s’ ignored (Delphi)
W1035 NO_RETVAL ON Return value of function ‘%s’ might be undefined (Delphi)
W1036 USE_BEFORE_DEF ON Variable ‘%s’ might not have been initialized (Delphi)
W1037 FOR_LOOP_VAR_UNDEF ON FOR-Loop variable ‘%s’ may be undefined after loop (Delphi)
E1038 UNIT_NAME_MISMATCH ON Unit identifier ‘%s’ does not match file name (Delphi)
W1039 NO_CFG_FILE_FOUND ON No configuration files found (Delphi)
W1040 IMPLICIT_VARIANTS ON Implicit use of Variants unit (Delphi)
W1041 UNICODE_TO_LOCALE ON Error converting Unicode char to locale charset. String truncated. Is your LANG environment variable set correctly (Delphi)
W1042 LOCALE_TO_UNICODE ON Error converting locale string ‘%s’ to Unicode. String truncated. Is your LANG environment variable set correctly (Delphi)
W1043 IMAGEBASE_MULTIPLE ON Imagebase $%X is not a multiple of 64k. Rounding down to $%X (Delphi)
W1044 SUSPICIOUS_TYPECAST ON Suspicious typecast of %s to %s (Delphi)
W1045 PRIVATE_PROPACCESSOR ON Property declaration references ancestor private ‘%s.%s’ (Delphi)
W1046 UNSAFE_TYPE OFF Unsafe type ‘%s%s%s’ (Delphi)
W1047 UNSAFE_CODE OFF Unsafe code ‘%s’ (Delphi)
W1048 UNSAFE_CAST OFF Unsafe typecast of ‘%s’ to ‘%s’ (Delphi)
W1049 OPTION_TRUNCATED ON value ‘%s’ for option %s was truncated (Delphi)
W1050 WIDECHAR_REDUCED ON WideChar reduced to byte char in set expressions (Delphi)
W1051 DUPLICATES_IGNORED ON Duplicate symbol names in namespace. Using ‘%s.%s’ found in %s. Ignoring duplicate in %s (Delphi)
W1052 UNIT_INIT_SEQ ON Can’t find System.Runtime.CompilerServices.RunClassConstructor. Unit initialization order will not follow uses clause order Does not seem to be documented in XE2
W1053 LOCAL_PINVOKE ON Local PInvoke code has not been made because external routine ‘%s’ in package ‘%s’ is defined using package local types in its custom attributes Does not seem to be documented in XE2
x1054 MESSAGE_DIRECTIVE ON %s (Delphi) User-defined warning messages (see below). Turns off message hints as well but not message errors.
W1055 TYPEINFO_IMPLICITLY_ADDED ON Published caused RTTI ($M+) to be added to type ‘%s’ (Delphi)
x1056 RLINK_WARNING ON Duplicate resource Type %s, ID %s; File %s resource kept; file %s resource discarded (Delphi)
W1057 IMPLICIT_STRING_CAST ON Implicit string cast from ‘%s’ to ‘%s’ (Delphi)
W1058 IMPLICIT_STRING_CAST_LOSS ON Implicit string cast with potential data loss from ‘%s’ to ‘%s’ (Delphi)
W1059 EXPLICIT_STRING_CAST OFF Explicit string cast from ‘%s’ to ‘%s’ (Delphi)
W1060 EXPLICIT_STRING_CAST_LOSS OFF Explicit string cast with potential data loss from ‘%s’ to ‘%s’ (Delphi)
W1061 CVT_WCHAR_TO_ACHAR ON W1061 Narrowing given WideChar constant (‘%s’) to AnsiChar lost information (Delphi)
W1062 CVT_NARROWING_STRING_LOST ON Narrowing given wide string constant lost information (Delphi)
W1063 CVT_ACHAR_TO_WCHAR ON Widening given AnsiChar constant (‘%s’) to WideChar lost information (Delphi)
W1064 CVT_WIDENING_STRING_LOST ON Widening given AnsiString constant lost information (Delphi)
W1066 NON_PORTABLE_TYPECAST ON Lost Extended floating point precision. Reduced to Double (Delphi)
W1201 XML_WHITESPACE_NOT_ALLOWED ON XML comment on ‘%s’ has badly formed XML-‘Whitespace is not allowed at this location.’ (Delphi)
W1202 XML_UNKNOWN_ENTITY ON XML comment on ‘%s’ has badly formed XML- ‘Reference to undefined entity ‘%s (Delphi)
W1203 XML_INVALID_NAME_START ON XML comment on ‘%s’ has badly formed XML-‘A name was started with an invalid character.’ (Delphi)
W1204 XML_INVALID_NAME ON XML comment on ‘%s’ has badly formed XML-‘A name contained an invalid character.’ (Delphi)
W1205 XML_EXPECTED_CHARACTER ON XML comment on ‘%s’ has badly formed XML-‘The character ‘%c’ was expected.’ (Delphi)
W1206 XML_CREF_NO_RESOLVE ON XML comment on ‘%s’ has cref attribute ‘%s’ that could not be resolved (Delphi)
W1207 XML_NO_PARM ON XML comment on ‘%s’ has a param tag for ‘%s’, but there is no parameter by that name (Delphi)
W1208 XML_NO_MATCHING_PARM ON Parameter ‘%s’ has no matching param tag in the XML comment for ‘%s’ (but other parameters do) (Delphi)

The $MESSAGE directive allows generation of H1054, W1054, E1054 and F1054 messages.  User-defined hint and warning messages can be turned off with {$WARN MESSAGE_DIRECTIVE OFF} but logically enough, user-defined error and fatal messages can not be disabled.  For example, add the following line to MyUnit.pas:

{$MESSAGE WARN 'This is a user warning'}

Which will gives a compiler warning similar to the following:

[DCC Warning] MyUnit.pas(25): W1054 This is a user warning

Finally, I’m not going to discuss the $WARNINGS directive, except to say that I don’t think it’s ever necessary or sensible to turn it off.  If you must turn a warning off for a specific section of code, turn just that warning off as discussed above.  Using {$WARNINGS OFF} is just as unhelpful as an empty except block.

Updates to kmwString functions

A couple of weeks ago I published on this blog a set of functions for handling supplementary characters in JavaScript.  We have since fixed a few bugs in the code and have now made the kmwString script available on github at https://github.com/tavultesoft/keyman-tools

This update:

  • Fixes an error which could occur when only 1 parameter was passed to kmwSubstring and that parameter was negative
  • Fixed an incorrect NaN return value for kmwCodePointToCodeUnit when passed an offset at the very end of a string
  • Adds kmwCodeUnitToCodePoint function
  • Adds kmwCharAt function — which is basically just kmwSubstr(index,1)

Again, suggestions, comments, fixes and flames all gratefully received.

WinDBG and Delphi exceptions in x64

I recently was asked whether the Delphi exception event filter for WinDBG that I wrote about would also work with x64 Delphi applications.  The answer was no, it wouldn’t work, but that made me curious to find out what was different with x64.  I knew x64 exception handling was completely different to x86, being table based instead of stack based, but I wasn’t sure how much of this would be reflected in the event filter.

The original post contains the details about how the exception record was accessible at a known location on the stack, and how we could dig in from there.

Before firing up WinDBG, I had a look at System.pas, and found the x64 virtual method table offsets.  I have highlighted the key field we want to pull out:

{ Virtual method table entries }
{$IF defined(CPUX64)}
vmtSelfPtr           = -176;
vmtIntfTable         = -168;
vmtAutoTable         = -160;
vmtInitTable         = -152;
vmtTypeInfo          = -144;
vmtFieldTable        = -136;
vmtMethodTable       = -128;
vmtDynamicTable      = -120;
vmtClassName         = -112;
vmtInstanceSize      = -104;
vmtParent            = -96;
vmtEquals            = -88 deprecated 'Use VMTOFFSET in asm code';
vmtGetHashCode       = -80 deprecated 'Use VMTOFFSET in asm code';
vmtToString          = -72 deprecated 'Use VMTOFFSET in asm code';
vmtSafeCallException = -64 deprecated 'Use VMTOFFSET in asm code';
vmtAfterConstruction = -56 deprecated 'Use VMTOFFSET in asm code';
vmtBeforeDestruction = -48 deprecated 'Use VMTOFFSET in asm code';
vmtDispatch          = -40 deprecated 'Use VMTOFFSET in asm code';
vmtDefaultHandler    = -32 deprecated 'Use VMTOFFSET in asm code';
vmtNewInstance       = -24 deprecated 'Use VMTOFFSET in asm code';
vmtFreeInstance      = -16 deprecated 'Use VMTOFFSET in asm code';
vmtDestroy           =  -8 deprecated 'Use VMTOFFSET in asm code';

vmtQueryInterface    =  0 deprecated 'Use VMTOFFSET in asm code';
vmtAddRef            =  8 deprecated 'Use VMTOFFSET in asm code';
vmtRelease           = 16 deprecated 'Use VMTOFFSET in asm code';
vmtCreateObject      = 24 deprecated 'Use VMTOFFSET in asm code';
{$ELSE !CPUX64}
vmtSelfPtr           = -88;
vmtIntfTable         = -84;
vmtAutoTable         = -80;
vmtInitTable         = -76;
vmtTypeInfo          = -72;
vmtFieldTable        = -68;
vmtMethodTable       = -64;
vmtDynamicTable      = -60;
vmtClassName         = -56;
vmtInstanceSize      = -52;
vmtParent            = -48;
vmtEquals            = -44 deprecated 'Use VMTOFFSET in asm code';
vmtGetHashCode       = -40 deprecated 'Use VMTOFFSET in asm code';
vmtToString          = -36 deprecated 'Use VMTOFFSET in asm code';
vmtSafeCallException = -32 deprecated 'Use VMTOFFSET in asm code';
vmtAfterConstruction = -28 deprecated 'Use VMTOFFSET in asm code';
vmtBeforeDestruction = -24 deprecated 'Use VMTOFFSET in asm code';
vmtDispatch          = -20 deprecated 'Use VMTOFFSET in asm code';
vmtDefaultHandler    = -16 deprecated 'Use VMTOFFSET in asm code';
vmtNewInstance       = -12 deprecated 'Use VMTOFFSET in asm code';
vmtFreeInstance      = -8 deprecated 'Use VMTOFFSET in asm code';
vmtDestroy           = -4 deprecated 'Use VMTOFFSET in asm code';

vmtQueryInterface    = 0 deprecated 'Use VMTOFFSET in asm code';
vmtAddRef            = 4 deprecated 'Use VMTOFFSET in asm code';
vmtRelease           = 8 deprecated 'Use VMTOFFSET in asm code';
vmtCreateObject      = 12 deprecated 'Use VMTOFFSET in asm code';
{$IFEND !CPUX64}

I also noted that the exception code for Delphi x64 was the same as x86:

cDelphiException   = $0EEDFADE;

Given this, I put together a test x64 application in Delphi that would throw an exception, and loaded it into WinDBG.  I enabled the event filter for unknown exceptions, and triggered an exception in the test application.  This broke into WinDBG, where I was able to take a look at the raw stack:

(2ad4.2948): Unknown exception - code 0eedfade (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
KERNELBASE!RaiseException+0x39:
000007fe`fd6ccacd 4881c4c8000000  add     rsp,0C8h
0:000> dd rbp
00000000`0012eab0  00000008 00000000 00000021 00000000
00000000`0012eac0  0059e1f0 00000000 0059e1f0 00000000
00000000`0012ead0  0eedfade 00000001 00000000 00000000
00000000`0012eae0  0059e1dd 00000000 00000007 00000000
00000000`0012eaf0  0059e1dd 00000000 0256cff0 00000000
00000000`0012eb00  00000000 00000000 00000000 00000000
00000000`0012eb10  00000000 00000000 00000000 00000000
00000000`0012eb20  00000000 00000000 0256cff8 00000000

We can see at rbp+20 is the familiar looking 0EEDFADE value.  This is the start of the EXCEPTION_RECORD structure, which I’ve reproduced below from Delphi’s System.pas with a little annotation of my own:

  TExceptionRecord = record
    ExceptionCode: Cardinal;                 // +0
    ExceptionFlags: Cardinal;                // +4
    ExceptionRecord: PExceptionRecord;       // +8
    ExceptionAddress: Pointer;               // +10
    NumberParameters: Cardinal;              // +18
    case {IsOsException:} Boolean of
      True:  (ExceptionInformation : array [0..14] of NativeUInt);
      False: (ExceptAddr: Pointer;           // +20
              ExceptObject: Pointer);        // +28
  end;

We do have to watch out for member alignment with this structure — because it contains both 4 byte DWORDs and 8 byte pointers, there are 4 bytes of hidden padding after the NumberParameters member, as shown below (this is from MSDN, sorry to switch languages on you!):

typedef struct _EXCEPTION_RECORD64 {
  DWORD ExceptionCode;
  DWORD ExceptionFlags;
  DWORD64 ExceptionRecord;
  DWORD64 ExceptionAddress;
  DWORD NumberParameters;
  DWORD __unusedAlignment;
  DWORD64 ExceptionInformation[EXCEPTION_MAXIMUM_PARAMETERS];
} EXCEPTION_RECORD64, *PEXCEPTION_RECORD64;

But what we can see from TExceptionRecord is that at offset 0x28 in the record is a pointer to our ExceptObject.  Great!  That’s everything we need.  We can now put together our x64-modified event filter.

You may remember the x86 event filter:

sxe -c "da poi(poi(poi(ebp+1c))-38)+1 L16;du /c 100 poi(poi(ebp+1c)+4)" 0EEDFADE

So here is the x64 version:

sxe -c "da poi(poi(poi(rbp+48))-70)+1 L16;du /c 100 poi(poi(rbp+48)+8)" 0EEDFADE

And with this filter installed, here is how a Delphi exception is now displayed in WinDBG:

(2ad4.2948): Unknown exception - code 0eedfade (first chance)
00000000`0059e0cf  "MyException"
00000000`02573910  "My very own kind of error message"
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
KERNELBASE!RaiseException+0x39:
000007fe`fd6ccacd 4881c4c8000000  add     rsp,0C8h

I’ll dissect the pointer offsets a little more than I did in the previous blog, because they can be a bit confusing:

  • rbp+48 is a pointer to the exception object (usually a type that inherits from Exception).
  • poi(rbp+48) dereferences that, and at offset 0 right here, we have a pointer to the class type.
  • Before we look at the class type, poi(rbp+48)+8 is the first member of the object (don’t forget ancestor classes), which happens to be FMessage from the Exception class. That gives us our message.
  • Diving deeper, poi(poi(rbp+48)) is now looking at the class type.
  • And we know that the offset of vmtClassName is -112 (-0x70).  So poi(poi(poi(rbp+48))-70) gives us the the ShortString class name, of which the first byte is the length.
  • So we finish with poi(poi(poi(rbp+48))-70)+1, which lets us look at the string itself.

You will see that to access the exception message, I have opted to look directly at the Exception object rather than use the more direct pointer which is on the stack.  I did this to make it easier to see how it might be possible to pull out other members of descendent exception classes, such as ErrorCode from EOSError.

And one final note: looking back on that previous blog, I see that one thing I wrote was a little misleading: the string length of FMessage is indeed available at poi(poi(rbp+48)+8)-4, but the string is null-terminated, so we don’t need to use it — WinDBG understands null-terminated strings.  Where this is more of a problem is with the ShortString type, which is not null-terminated.  This is why sometimes exception class names displayed using this method will show a few garbage characters afterwards, because we don’t bother about accounting for that; the L16 parameter prevents us dumping memory until we reach a null byte.

What value my life?

I sometimes ponder on what I value in my life. Or to put it another way, what it would mean to lose certain things. I go through the things, the activities, the people in my life and imagine what it would be to no longer have them there. My house? It’s a lovely house… We put a lot of effort into building it and I enjoy living in it. All the same, I can imagine not having it, and it doesn’t hurt that much. It’s a place to live and I thank God for His provision and yet I don’t feel bound to it. My car? Pfft. It gets us from A to B and the thing I value most about it is that it rarely goes wrong.

So how about my bike? I certainly enjoy riding it. But it just costs money to replace, and God has blessed us with enough money to buy a bike. I don’t feel so attached to the bike itself. Ok then, how about cycling itself?

I do love riding my bike. The challenges, racing and riding with mates, pushing myself physically, the exercise and the well-being it brings. And yet… I could let it all go, and my only real regrets would be losing touch with riding buddies, and the loss of fitness. I’d miss riding, but it wouldn’t destroy me.

My job, my career? The code I create — Keyman; the business I’ve poured myself into — Tavultesoft; and the relationships built, respect earned? I derive a lot of satisfaction from creating computer software: turning a concept into a tool that can really help people. So what if that all disappeared? My position, my abilities, my reputation, my creations? Would that devastate?

It would hurt. I’m sure it would hurt more than I can really imagine. And yet, I can imagine a world without this job. I can’t imagine what I would do, but I can imagine it.

And so as we cut closer to the quick, I start to draw back. All these various blessings of God — those that I take for granted through to those that I spend much of my waking life focused on — stripped away. Like Job. What if it happened to me? Can I really comprehend the pain of the loss of all that?

That pain. It would be as nothing to the loss of my family. That’s the one place I cannot imagine arriving at. The loneliness, my tether to reality severed, my life, my soul, spinning through the void. My wife and daughters: those who are so close and can infuriate, hurt, and exasperate. Those three people who bring such joy to me, a joy that others can only glimpse a pale reflection of when I post a comment on Facebook, or drop an email to them. Even to contemplate losing them stops my heart and starts a cold, cold hurt deep within.

I can’t imagine it. I shy away from the very idea. Job lost his whole family, everything, and yet he would not curse God, nor turn from Him. Do I have such a love for Him that I could continue to worship Him in the face of such devastation? To look for comfort, probably. To be angry with Him, almost certainly! But to worship Him? I don’t know, and my prayer is that He would not allow me to suffer to such a degree that I could come to doubt His love for me.

It is a selfish prayer. So many people in this world suffer such loss that I couldn’t bear. I sit here safe and contemplate this from the safety of my warm home, with my family around me, in a safe and comfortable country. Lord God, shatter me, make me dependent on You and You alone. I am too scared to ask this of You. But this is my tiny step of faith. I am like the father who asked Jesus “help me believe!”