All posts by Marc Durdin

Understanding and correcting interface reference leaks in Delphi’s Vcl.OleCtrls.pas

Bugs, Computing, Delphi, Development

Update 21 Sep 2015: This bug has been fixed in Delphi 10 Seattle.

I have spent quite some time recently tracing a memory leak in a Delphi application.  It is quite a large application and makes a lot of use of embedded MSHTML (TWebBrowser and TEmbeddedWB) controls for presentation.  Somehow, somewhere, we were leaking memory: users were reporting that after a few hours of use, the application slowed down, and checking Task Manager certainly reflected excessive memory usage.

The normal procedure to reproduce the memory leak was followed, including using tools such as AQtime and other debug logging tools.  However, no leaks were detected using these tools, although we could see the memory usage increasing in Task Manager on our test machines.  This suggested the memory we were leaking was not allocated by Delphi code: i.e. it was Windows or a 3rd party DLL.  This doesn’t mean, of course, that it wasn’t our fault — just that it wasn’t allocated directly from Delphi source!

At this point, I was asked to trace this issue further.  I ran Performance Monitor with a couple of key counters: Handle Count and Working Set.  Running the test in question (involving opening and closing a window with an embedded web browser control) showed a gradual increase in both handle count and working set size.  However, Performance Monitor unfortunately does not include a user handle (i.e. window handles) counter.  It was when I noticed in Process Explorer that user handles were also increasing that I got my first break in tracing the cause.

It turned out that the embedded web browser window was not always being destroyed when its parent window was.  This window had the class name “Internet Explorer_Server”.

With a bit more tracing, I found that the trigger was the Document or Application properties.  If the Document property of the TWebBrowser control was ever referenced, the window was never destroyed (note, there are also some other properties that trigger the same behaviour — look for properties returning type IDispatch).

This set me to researching the Document property.  It looks like this:

property Document: IDispatch index 203 read GetIDispatchProp;

Looking at GetIDispatchProp in Vcl.OleCtrls.pas, we see the following code:

function TOleControl.GetIDispatchProp(Index: Integer): IDispatch;
var
  Temp: TVarData;
begin
  GetProperty(Index, Temp);
  Result := IDispatch(Temp.VDispatch);
end;

And here some alarm bells go off.  Delphi, rather nicely, manages all the reference counting on interfaces.  This works pretty smoothly, until you trick the compiler by casting other types to interfaces.  Here the code in question is triggering an invisible call to IntfCopy in the line:

Result := IDispatch(Temp.VDispatch);

The IntfCopy function internally calls  _AddRef on the object, but because of the cast to IDispatch from a Pointer, this new reference is never released.  The fix is to change the function (and the similar GetIUnknownProp function) to:

function TOleControl.GetIDispatchProp(Index: Integer): IDispatch;
var
  Temp: TVarData;
begin
  GetProperty(Index, Temp);
  Pointer(Result) := Temp.VDispatch;
end;

function TOleControl.GetIUnknownProp(Index: Integer): IUnknown;
var
  Temp: TVarData;
begin
  GetProperty(Index, Temp);
  Pointer(Result) := Temp.VUnknown;
end;

By casting this way, we avoid the call to IntfCopy and hence the call to _AddRef.  Alternatively, you could have called _Release on Result, but this would require another test to ensure that Result wasn’t nil (and also, of course, redundant _AddRef and _Release calls).

It turns out that this problem was identified way back in 1999 and the workaround has been commonly referenced since then.  So I am not taking credit for the fix here!  And yet it is still unresolved in Delphi XE2 — and so still causing trouble for Delphi programmers today!  There are no clear references to the problem in QualityCentral that I could find (that’s about to change!)

But don’t stop reading yet!

“Now my app crashes”

There are frequent complaints online that this fix results in crashes.  This is because other developers have engineered fixes to this reference leak in places where these GetIDispatchProp and/or GetIUnknownProp calls are made, rather than where the problem actually occurs.  I have found this in TEmbeddedWB.  TEmbeddedWB is a web browser hosting component that takes up where TWebBrowser leaves off, and it does fix a lot of the limitations of TWebBrowser.

But here are the places in the TEmbeddedWB source that you’ll need to “unfix” once you fix the root problem in Vcl.OleCtrls.pas:

EmbeddedWB.pas [2603]: (procedure TEmbeddedWB.SetUserAgentInt): Delete _Release call
EwbCore.pas [1367]: (procedure TCustomEmbeddedWB.SetDesignMode): Delete _Release call
EwbCore.pas [1404]: (procedure TCustomEmbeddedWB.SetDownloadOptions): Delete _Release call
EwbCore.pas [1468]: (procedure TCustomEmbeddedWB.SetUserInterfaceOptions): Delete _Release call
EwbTools.pas [1015]: (function GetBmpFromBrowser): Delete _Release call
EwbTools.pas [3164]: (function InvokeCMD): Delete _Release call

Note, you’ll also see an experimental fix buried — and disabled — in the TEmbeddedWB code (but without fixing the lines above), and without a whole lot of documentation as to why!

I have created a Quality Central report, along with a test case and example fix.  I also checked the Delphi VCL source, and about 30 other components that we use, and found no more calls to _Release relating to this issue.

Hobart Dirty Dozen ride

Cycling, silly ideas

This is just a small note that my post about the Hobart Dirty Dozen — a kinda epic ride up a bunch of Hobart’s steepest hills with my mates Dan and Iain — has been kindly published on The Climbing Cyclist’s blog.

Climbing Brent St

If you want to see just the relevant climbs for the Dirty Dozen on the Strava ride, drag the link “Show only the Hobart Dirty Dozen Climbs” to your bookmarks toolbar, then click the bookmark when you are on the Dirty Dozen Strava ride page:

Show only the Hobart Dirty Dozen Climbs

Rainforest Walk

Family, Family|Bible|More, Photos

Once upon a time, there lived a child and a daddy. The child’s name was Hannah.

Daddy said to her, “let’s go to Myrtle Forest”.

“Yes,” said Hannah.  Hannah’s sister Bethany came too.

Daddy saw a ruin.  Hannah wanted to go in the trail to the ruin, but it had a gate in front of it, which meant that you couldn’t go in it.

They saw heaps of ferns and waterfalls.

 They started exploring the waterfalls and ferns.

They saw an old fern stump, and it had little green things growing on top of it, and it was mossy all over!

There was a pretty waterfall on one side of a bridge, and on the other side there was a creek which went under the bridge.

They saw some snow trees, which are trees that are kind of aqua, and if you are far away, they look like they are covered with snow.  Hannah had a small sprig off a snow tree.  She made a wish.  She wished that she could fly.  The wish hasn’t come true yet.

Hannah and Bethany hid in some hiding places.

There were beautiful views and lots of trees and creeks.

Here is a photo of the creek hidden behind the fronds of the ferns.

— The End —

Corrie: why is she a hero?

Bible, Family|Bible|More, Pontificating

“Father sat down on the edge of the narrow bed. “Corrie,” he began gently, “when you and I go to Amsterdam-when do I give you your ticket?”
I sniffed a few times, considering this.
“Why, just before we get on the train.”
“Exactly. And our wise Father in heaven knows when we’re going to need things, too. Don’t run out ahead of Him, Corrie. When the time comes that some of us will have to die, you will look into your heart and find the strength you need-just in time.”
― Corrie Ten Boom, The Hiding Place

Corrie Ten Boom suffered more than most of us ever will. And yet she was full of love. She had learned to live in the grace of God. Through the most devestating of circumstances, she depended on Him and His grace was sufficient for her needs.
If that bleak, black time comes for us, I guess all we have to do is ask our Lord for that grace. But perhaps the hardest thing of all is to do just that: it means giving up the hate and giving the hurt to Him, just to make that simplest of requests. No revenge, just forgiveness is all that can remain. He will help us, if we just once ask Him.
Corrie Ten Boom is one of my heroes, for she did not survive the most awful of times on her own willpower, or through determination, or even by being a super holy person. She is my hero because she was able to ask God for that grace.
I pray that I may be spared such sufferings. And yet, that if such sufferings come my way that He will protect my soul and not allow it to become hardened with resentment or hate. And that I can continually ask Him for that grace freely provided by the sacrifice He already made through His son, which purges our sins, and by so doing allows us to love those who have hurt us.

Race Shape for Strava and a Bookmarklet to link them

Cycling, mashups, Strava

Race Shape is a nice polished mashup for Strava that allows you to compare efforts on a segment in much greater detail than the Strava interface allows.  It can be very interesting to see how and where your opponents stole time on you in their KOM efforts.

To make it a little easier to use, I whipped up a little bookmarklet (not very well written) that you can add to your Bookmarks toolbar.  When looking at the list of segment efforts in a Strava activity, click the bookmarklet and a little Race Shape icon will appear next to each segment in the list.  You can then just click on that icon to open the segment in Race Shape!

Drag this link to your Bookmarks toolbar to install the bookmarklet: Race Shape

Here’s what it looks like — the icon is pretty subtle…:

The raw code, for those who like such things (or want to fix or improve the bookmarklet):

(function(){
this.i=function(id,e)
{
  while(e.firstChild.className == ‘raceShape’) e.removeChild(e.firstChild);
  var segment=id.match(/-([0-9]+)$/)[1];
  var activity=location.pathname.match(/\/([0-9]+)/)[1];
  var a = window.document.createElement(‘a’);
  a.className=’raceShape’;
  a.href=’http://raceshape.com/strava.redirect.php?url=http%253A%252F%252Fapp.strava.com%252Frides%252F’+activity+’%2523’+segment;
  a.target=’_blank’;
  var img = window.document.createElement(‘img’);
  img.src=’http://d1touf4erjk31x.cloudfront.net/images/brand.png’;
  img.style=’border:none;margin-right:4px’;
  img.width=’20’;
  a.appendChild(img);
  e.insertBefore(a,e.firstChild);
};
var e = document.getElementById(‘segment-efforts-table’).getElementsByTagName(‘tr’);
for(var m = 0; m < e.length; m++)
{
  if(e[m].id) { this.i(e[m].id,e[m].cells[1]); }
}
})();

My blog hit 100,000 visits a couple of days ago.  Wow…  Thank you for reading!  I don’t know how you find anything interesting in all the twaddle… 😉

Delphi XE2’s hidden hints and warnings options

Computing, Delphi, Development

There are a number of warnings available in Delphi XE2 that are not very well documented.  While you can control them in the Project options dialog, and you can turn them on using {$WARN} directives or in command line compiler options, the documentation for the warning identifiers is currently pretty piecemeal, and there is no clear link between the project options, warning directive identifiers, and numeric identifiers.

The {$WARN} compiler directive takes effect only in the unit it is in.  It overrides project settings and command line compiler flags.  To turn a warning on or off for a block of code (although you’ll have to work hard to convince me that this is an acceptable solution 99% of the time):

{$WARN SYMBOL_DEPRECATED ON}
{$WARN SYMBOL_DEPRECATED OFF}

To restore the warning to its previous setting:

{$WARN SYMBOL_DEPRECATED DEFAULT}

And finally, you can convert a warning into an error:

{$WARN SYMBOL_DEPRECATED ERROR}

To change the setting in your project, go to Project|Options, Delphi Compiler, Hints and Warnings, and expand the Output warnings setting:


Delphi’s project options dialog – hints and warnings

To change the setting in the command line compiler, use the -W flag, for example:

dcc32 -W+SYMBOL_DEPRECATED myproject.dpr
dcc32 -W-SYMBOL_DEPRECATED myproject.dpr

Or to turn the warning into an error:

dcc32 -W^SYMBOL_DEPRECATED myproject.dpr

Note, if you type this from the Windows command prompt, you will have to repeat the ^ symbol, as ^ is an escape symbol:

dcc32 -W^^SYMBOL_DEPRECATED myproject.dpr

These are the hints and warning identifiers I am aware, along with what I understand are the default settings and numeric identifiers, and links to the relevant help topics.

ID Directive ID Default Description Notes
W1000 SYMBOL_DEPRECATED OFF Symbol ‘%s’ is deprecated (Delphi)
W1001 SYMBOL_LIBRARY OFF Symbol ‘%s’ is specific to a library (Delphi)
W1002 SYMBOL_PLATFORM OFF Symbol ‘%s’ is specific to a platform (Delphi)
W1003 SYMBOL_EXPERIMENTAL ON Symbol ‘%s’ is experimental (Delphi)
W1004 UNIT_LIBRARY OFF Unit ‘%s’ is specific to a library (Delphi)
W1005 UNIT_PLATFORM OFF Unit ‘%s’ is specific to a platform (Delphi)
W1006 UNIT_DEPRECATED OFF Unit ‘%s’ is deprecated (Delphi)
W1007 UNIT_EXPERIMENTAL ON Unit ‘%s’ is experimental (Delphi)
W1008 HRESULT_COMPAT ON Integer and HRESULT interchanged
W1009 HIDING_MEMBER ON Redeclaration of ‘%s’ hides a member in the base class (Delphi)
W1010 HIDDEN_VIRTUAL ON Method ‘%s’ hides virtual method of base type ‘%s’ (Delphi)
W1011 GARBAGE ON Text after final ‘END.’ – ignored by compiler (Delphi)
W1012 BOUNDS_ERROR ON Constant expression violates subrange bounds
W1013 ZERO_NIL_COMPAT ON Constant 0 converted to NIL (Delphi)
W1014 STRING_CONST_TRUNCED ON String constant truncated to fit STRING%ld (Delphi)
W1015 FOR_LOOP_VAR_VARPAR ON FOR-Loop variable ‘%s’ cannot be passed as var parameter (Delphi)
W1016 TYPED_CONST_VARPAR ON Typed constant ‘%s’ passed as var parameter (Delphi)
W1017 ASG_TO_TYPED_CONST ON Assignment to typed constant ‘%s’ (Delphi)
W1018 CASE_LABEL_RANGE ON Case label outside of range of case expression (Delphi)
x1019 FOR_VARIABLE ON For loop control variable must be simple local variable (Delphi)
x1020 CONSTRUCTING_ABSTRACT ON Constructing instance of ‘%s’ containing abstract method ‘%s.%s’ (Delphi)
W1021 COMPARISON_FALSE ON Comparison always evaluates to False (Delphi)
W1022 COMPARISON_TRUE ON Comparison always evaluates to True (Delphi)
W1023 COMPARING_SIGNED_UNSIGNED ON Comparing signed and unsigned types – widened both operands (Delphi)
W1024 COMBINING_SIGNED_UNSIGNED ON Combining signed and unsigned types – widened both operands (Delphi)
x1025 UNSUPPORTED_CONSTRUCT ON Unsupported language feature: ‘%s’ (Delphi)
x1026 FILE_OPEN ON File not found ‘%s’ (Delphi) Despite being listed as a warning, turning this off appears to have no effect.
F1027 FILE_OPEN_UNITSRC ON Unit not found ‘%s’ or binary equivalents (%s) (Delphi)
x1028 BAD_GLOBAL_SYMBOL ON Bad global symbol definition ‘%s’ in object file ‘%s’ (Delphi)
W1029 DUPLICATE_CTOR_DTOR ON Duplicate %s ‘%s’ with identical parameters will be inacessible from C++ (Delphi)
x1030 INVALID_DIRECTIVE ON Invalid compiler directive – ‘%s’ (Delphi)
W1031 PACKAGE_NO_LINK ON Package ‘%s’ will not be written to disk because -J option is enabled (Delphi)
W1032 PACKAGED_THREADVAR ON Exported package threadvar ‘%s.%s’ cannot be used outside of this package (Delphi)
W1033 IMPLICIT_IMPORT ON Unit ‘%s’ implicitly imported into package ‘%s’ (Delphi)
W1034 HPPEMIT_IGNORED ON $HPPEMIT ‘%s’ ignored (Delphi)
W1035 NO_RETVAL ON Return value of function ‘%s’ might be undefined (Delphi)
W1036 USE_BEFORE_DEF ON Variable ‘%s’ might not have been initialized (Delphi)
W1037 FOR_LOOP_VAR_UNDEF ON FOR-Loop variable ‘%s’ may be undefined after loop (Delphi)
E1038 UNIT_NAME_MISMATCH ON Unit identifier ‘%s’ does not match file name (Delphi)
W1039 NO_CFG_FILE_FOUND ON No configuration files found (Delphi)
W1040 IMPLICIT_VARIANTS ON Implicit use of Variants unit (Delphi)
W1041 UNICODE_TO_LOCALE ON Error converting Unicode char to locale charset. String truncated. Is your LANG environment variable set correctly (Delphi)
W1042 LOCALE_TO_UNICODE ON Error converting locale string ‘%s’ to Unicode. String truncated. Is your LANG environment variable set correctly (Delphi)
W1043 IMAGEBASE_MULTIPLE ON Imagebase $%X is not a multiple of 64k. Rounding down to $%X (Delphi)
W1044 SUSPICIOUS_TYPECAST ON Suspicious typecast of %s to %s (Delphi)
W1045 PRIVATE_PROPACCESSOR ON Property declaration references ancestor private ‘%s.%s’ (Delphi)
W1046 UNSAFE_TYPE OFF Unsafe type ‘%s%s%s’ (Delphi)
W1047 UNSAFE_CODE OFF Unsafe code ‘%s’ (Delphi)
W1048 UNSAFE_CAST OFF Unsafe typecast of ‘%s’ to ‘%s’ (Delphi)
W1049 OPTION_TRUNCATED ON value ‘%s’ for option %s was truncated (Delphi)
W1050 WIDECHAR_REDUCED ON WideChar reduced to byte char in set expressions (Delphi)
W1051 DUPLICATES_IGNORED ON Duplicate symbol names in namespace. Using ‘%s.%s’ found in %s. Ignoring duplicate in %s (Delphi)
W1052 UNIT_INIT_SEQ ON Can’t find System.Runtime.CompilerServices.RunClassConstructor. Unit initialization order will not follow uses clause order Does not seem to be documented in XE2
W1053 LOCAL_PINVOKE ON Local PInvoke code has not been made because external routine ‘%s’ in package ‘%s’ is defined using package local types in its custom attributes Does not seem to be documented in XE2
x1054 MESSAGE_DIRECTIVE ON %s (Delphi) User-defined warning messages (see below). Turns off message hints as well but not message errors.
W1055 TYPEINFO_IMPLICITLY_ADDED ON Published caused RTTI ($M+) to be added to type ‘%s’ (Delphi)
x1056 RLINK_WARNING ON Duplicate resource Type %s, ID %s; File %s resource kept; file %s resource discarded (Delphi)
W1057 IMPLICIT_STRING_CAST ON Implicit string cast from ‘%s’ to ‘%s’ (Delphi)
W1058 IMPLICIT_STRING_CAST_LOSS ON Implicit string cast with potential data loss from ‘%s’ to ‘%s’ (Delphi)
W1059 EXPLICIT_STRING_CAST OFF Explicit string cast from ‘%s’ to ‘%s’ (Delphi)
W1060 EXPLICIT_STRING_CAST_LOSS OFF Explicit string cast with potential data loss from ‘%s’ to ‘%s’ (Delphi)
W1061 CVT_WCHAR_TO_ACHAR ON W1061 Narrowing given WideChar constant (‘%s’) to AnsiChar lost information (Delphi)
W1062 CVT_NARROWING_STRING_LOST ON Narrowing given wide string constant lost information (Delphi)
W1063 CVT_ACHAR_TO_WCHAR ON Widening given AnsiChar constant (‘%s’) to WideChar lost information (Delphi)
W1064 CVT_WIDENING_STRING_LOST ON Widening given AnsiString constant lost information (Delphi)
W1066 NON_PORTABLE_TYPECAST ON Lost Extended floating point precision. Reduced to Double (Delphi)
W1201 XML_WHITESPACE_NOT_ALLOWED ON XML comment on ‘%s’ has badly formed XML-‘Whitespace is not allowed at this location.’ (Delphi)
W1202 XML_UNKNOWN_ENTITY ON XML comment on ‘%s’ has badly formed XML- ‘Reference to undefined entity ‘%s (Delphi)
W1203 XML_INVALID_NAME_START ON XML comment on ‘%s’ has badly formed XML-‘A name was started with an invalid character.’ (Delphi)
W1204 XML_INVALID_NAME ON XML comment on ‘%s’ has badly formed XML-‘A name contained an invalid character.’ (Delphi)
W1205 XML_EXPECTED_CHARACTER ON XML comment on ‘%s’ has badly formed XML-‘The character ‘%c’ was expected.’ (Delphi)
W1206 XML_CREF_NO_RESOLVE ON XML comment on ‘%s’ has cref attribute ‘%s’ that could not be resolved (Delphi)
W1207 XML_NO_PARM ON XML comment on ‘%s’ has a param tag for ‘%s’, but there is no parameter by that name (Delphi)
W1208 XML_NO_MATCHING_PARM ON Parameter ‘%s’ has no matching param tag in the XML comment for ‘%s’ (but other parameters do) (Delphi)

The $MESSAGE directive allows generation of H1054, W1054, E1054 and F1054 messages.  User-defined hint and warning messages can be turned off with {$WARN MESSAGE_DIRECTIVE OFF} but logically enough, user-defined error and fatal messages can not be disabled.  For example, add the following line to MyUnit.pas:

{$MESSAGE WARN 'This is a user warning'}

Which will gives a compiler warning similar to the following:

[DCC Warning] MyUnit.pas(25): W1054 This is a user warning

Finally, I’m not going to discuss the $WARNINGS directive, except to say that I don’t think it’s ever necessary or sensible to turn it off.  If you must turn a warning off for a specific section of code, turn just that warning off as discussed above.  Using {$WARNINGS OFF} is just as unhelpful as an empty except block.

Updates to kmwString functions

Bugs, Computing, Development, JavaScript, Unicode

A couple of weeks ago I published on this blog a set of functions for handling supplementary characters in JavaScript.  We have since fixed a few bugs in the code and have now made the kmwString script available on github at https://github.com/tavultesoft/keyman-tools

This update:

  • Fixes an error which could occur when only 1 parameter was passed to kmwSubstring and that parameter was negative
  • Fixed an incorrect NaN return value for kmwCodePointToCodeUnit when passed an offset at the very end of a string
  • Adds kmwCodeUnitToCodePoint function
  • Adds kmwCharAt function — which is basically just kmwSubstr(index,1)

Again, suggestions, comments, fixes and flames all gratefully received.

WinDBG and Delphi exceptions in x64

Computing, Delphi, Development, Tools, Windows, Windows 7

I recently was asked whether the Delphi exception event filter for WinDBG that I wrote about would also work with x64 Delphi applications.  The answer was no, it wouldn’t work, but that made me curious to find out what was different with x64.  I knew x64 exception handling was completely different to x86, being table based instead of stack based, but I wasn’t sure how much of this would be reflected in the event filter.

The original post contains the details about how the exception record was accessible at a known location on the stack, and how we could dig in from there.

Before firing up WinDBG, I had a look at System.pas, and found the x64 virtual method table offsets.  I have highlighted the key field we want to pull out:

{ Virtual method table entries }
{$IF defined(CPUX64)}
vmtSelfPtr           = -176;
vmtIntfTable         = -168;
vmtAutoTable         = -160;
vmtInitTable         = -152;
vmtTypeInfo          = -144;
vmtFieldTable        = -136;
vmtMethodTable       = -128;
vmtDynamicTable      = -120;
vmtClassName         = -112;
vmtInstanceSize      = -104;
vmtParent            = -96;
vmtEquals            = -88 deprecated 'Use VMTOFFSET in asm code';
vmtGetHashCode       = -80 deprecated 'Use VMTOFFSET in asm code';
vmtToString          = -72 deprecated 'Use VMTOFFSET in asm code';
vmtSafeCallException = -64 deprecated 'Use VMTOFFSET in asm code';
vmtAfterConstruction = -56 deprecated 'Use VMTOFFSET in asm code';
vmtBeforeDestruction = -48 deprecated 'Use VMTOFFSET in asm code';
vmtDispatch          = -40 deprecated 'Use VMTOFFSET in asm code';
vmtDefaultHandler    = -32 deprecated 'Use VMTOFFSET in asm code';
vmtNewInstance       = -24 deprecated 'Use VMTOFFSET in asm code';
vmtFreeInstance      = -16 deprecated 'Use VMTOFFSET in asm code';
vmtDestroy           =  -8 deprecated 'Use VMTOFFSET in asm code';

vmtQueryInterface    =  0 deprecated 'Use VMTOFFSET in asm code';
vmtAddRef            =  8 deprecated 'Use VMTOFFSET in asm code';
vmtRelease           = 16 deprecated 'Use VMTOFFSET in asm code';
vmtCreateObject      = 24 deprecated 'Use VMTOFFSET in asm code';
{$ELSE !CPUX64}
vmtSelfPtr           = -88;
vmtIntfTable         = -84;
vmtAutoTable         = -80;
vmtInitTable         = -76;
vmtTypeInfo          = -72;
vmtFieldTable        = -68;
vmtMethodTable       = -64;
vmtDynamicTable      = -60;
vmtClassName         = -56;
vmtInstanceSize      = -52;
vmtParent            = -48;
vmtEquals            = -44 deprecated 'Use VMTOFFSET in asm code';
vmtGetHashCode       = -40 deprecated 'Use VMTOFFSET in asm code';
vmtToString          = -36 deprecated 'Use VMTOFFSET in asm code';
vmtSafeCallException = -32 deprecated 'Use VMTOFFSET in asm code';
vmtAfterConstruction = -28 deprecated 'Use VMTOFFSET in asm code';
vmtBeforeDestruction = -24 deprecated 'Use VMTOFFSET in asm code';
vmtDispatch          = -20 deprecated 'Use VMTOFFSET in asm code';
vmtDefaultHandler    = -16 deprecated 'Use VMTOFFSET in asm code';
vmtNewInstance       = -12 deprecated 'Use VMTOFFSET in asm code';
vmtFreeInstance      = -8 deprecated 'Use VMTOFFSET in asm code';
vmtDestroy           = -4 deprecated 'Use VMTOFFSET in asm code';

vmtQueryInterface    = 0 deprecated 'Use VMTOFFSET in asm code';
vmtAddRef            = 4 deprecated 'Use VMTOFFSET in asm code';
vmtRelease           = 8 deprecated 'Use VMTOFFSET in asm code';
vmtCreateObject      = 12 deprecated 'Use VMTOFFSET in asm code';
{$IFEND !CPUX64}

I also noted that the exception code for Delphi x64 was the same as x86:

cDelphiException   = $0EEDFADE;

Given this, I put together a test x64 application in Delphi that would throw an exception, and loaded it into WinDBG.  I enabled the event filter for unknown exceptions, and triggered an exception in the test application.  This broke into WinDBG, where I was able to take a look at the raw stack:

(2ad4.2948): Unknown exception - code 0eedfade (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
KERNELBASE!RaiseException+0x39:
000007fe`fd6ccacd 4881c4c8000000  add     rsp,0C8h
0:000> dd rbp
00000000`0012eab0  00000008 00000000 00000021 00000000
00000000`0012eac0  0059e1f0 00000000 0059e1f0 00000000
00000000`0012ead0  0eedfade 00000001 00000000 00000000
00000000`0012eae0  0059e1dd 00000000 00000007 00000000
00000000`0012eaf0  0059e1dd 00000000 0256cff0 00000000
00000000`0012eb00  00000000 00000000 00000000 00000000
00000000`0012eb10  00000000 00000000 00000000 00000000
00000000`0012eb20  00000000 00000000 0256cff8 00000000

We can see at rbp+20 is the familiar looking 0EEDFADE value.  This is the start of the EXCEPTION_RECORD structure, which I’ve reproduced below from Delphi’s System.pas with a little annotation of my own:

  TExceptionRecord = record
    ExceptionCode: Cardinal;                 // +0
    ExceptionFlags: Cardinal;                // +4
    ExceptionRecord: PExceptionRecord;       // +8
    ExceptionAddress: Pointer;               // +10
    NumberParameters: Cardinal;              // +18
    case {IsOsException:} Boolean of
      True:  (ExceptionInformation : array [0..14] of NativeUInt);
      False: (ExceptAddr: Pointer;           // +20
              ExceptObject: Pointer);        // +28
  end;

We do have to watch out for member alignment with this structure — because it contains both 4 byte DWORDs and 8 byte pointers, there are 4 bytes of hidden padding after the NumberParameters member, as shown below (this is from MSDN, sorry to switch languages on you!):

typedef struct _EXCEPTION_RECORD64 {
  DWORD ExceptionCode;
  DWORD ExceptionFlags;
  DWORD64 ExceptionRecord;
  DWORD64 ExceptionAddress;
  DWORD NumberParameters;
  DWORD __unusedAlignment;
  DWORD64 ExceptionInformation[EXCEPTION_MAXIMUM_PARAMETERS];
} EXCEPTION_RECORD64, *PEXCEPTION_RECORD64;

But what we can see from TExceptionRecord is that at offset 0x28 in the record is a pointer to our ExceptObject.  Great!  That’s everything we need.  We can now put together our x64-modified event filter.

You may remember the x86 event filter:

sxe -c "da poi(poi(poi(ebp+1c))-38)+1 L16;du /c 100 poi(poi(ebp+1c)+4)" 0EEDFADE

So here is the x64 version:

sxe -c "da poi(poi(poi(rbp+48))-70)+1 L16;du /c 100 poi(poi(rbp+48)+8)" 0EEDFADE

And with this filter installed, here is how a Delphi exception is now displayed in WinDBG:

(2ad4.2948): Unknown exception - code 0eedfade (first chance)
00000000`0059e0cf  "MyException"
00000000`02573910  "My very own kind of error message"
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
KERNELBASE!RaiseException+0x39:
000007fe`fd6ccacd 4881c4c8000000  add     rsp,0C8h

I’ll dissect the pointer offsets a little more than I did in the previous blog, because they can be a bit confusing:

  • rbp+48 is a pointer to the exception object (usually a type that inherits from Exception).
  • poi(rbp+48) dereferences that, and at offset 0 right here, we have a pointer to the class type.
  • Before we look at the class type, poi(rbp+48)+8 is the first member of the object (don’t forget ancestor classes), which happens to be FMessage from the Exception class. That gives us our message.
  • Diving deeper, poi(poi(rbp+48)) is now looking at the class type.
  • And we know that the offset of vmtClassName is -112 (-0x70).  So poi(poi(poi(rbp+48))-70) gives us the the ShortString class name, of which the first byte is the length.
  • So we finish with poi(poi(poi(rbp+48))-70)+1, which lets us look at the string itself.

You will see that to access the exception message, I have opted to look directly at the Exception object rather than use the more direct pointer which is on the stack.  I did this to make it easier to see how it might be possible to pull out other members of descendent exception classes, such as ErrorCode from EOSError.

And one final note: looking back on that previous blog, I see that one thing I wrote was a little misleading: the string length of FMessage is indeed available at poi(poi(rbp+48)+8)-4, but the string is null-terminated, so we don’t need to use it — WinDBG understands null-terminated strings.  Where this is more of a problem is with the ShortString type, which is not null-terminated.  This is why sometimes exception class names displayed using this method will show a few garbage characters afterwards, because we don’t bother about accounting for that; the L16 parameter prevents us dumping memory until we reach a null byte.

What value my life?

Bible, Family|Bible|More, Pontificating

I sometimes ponder on what I value in my life. Or to put it another way, what it would mean to lose certain things. I go through the things, the activities, the people in my life and imagine what it would be to no longer have them there. My house? It’s a lovely house… We put a lot of effort into building it and I enjoy living in it. All the same, I can imagine not having it, and it doesn’t hurt that much. It’s a place to live and I thank God for His provision and yet I don’t feel bound to it. My car? Pfft. It gets us from A to B and the thing I value most about it is that it rarely goes wrong.

So how about my bike? I certainly enjoy riding it. But it just costs money to replace, and God has blessed us with enough money to buy a bike. I don’t feel so attached to the bike itself. Ok then, how about cycling itself?

I do love riding my bike. The challenges, racing and riding with mates, pushing myself physically, the exercise and the well-being it brings. And yet… I could let it all go, and my only real regrets would be losing touch with riding buddies, and the loss of fitness. I’d miss riding, but it wouldn’t destroy me.

My job, my career? The code I create — Keyman; the business I’ve poured myself into — Tavultesoft; and the relationships built, respect earned? I derive a lot of satisfaction from creating computer software: turning a concept into a tool that can really help people. So what if that all disappeared? My position, my abilities, my reputation, my creations? Would that devastate?

It would hurt. I’m sure it would hurt more than I can really imagine. And yet, I can imagine a world without this job. I can’t imagine what I would do, but I can imagine it.

And so as we cut closer to the quick, I start to draw back. All these various blessings of God — those that I take for granted through to those that I spend much of my waking life focused on — stripped away. Like Job. What if it happened to me? Can I really comprehend the pain of the loss of all that?

That pain. It would be as nothing to the loss of my family. That’s the one place I cannot imagine arriving at. The loneliness, my tether to reality severed, my life, my soul, spinning through the void. My wife and daughters: those who are so close and can infuriate, hurt, and exasperate. Those three people who bring such joy to me, a joy that others can only glimpse a pale reflection of when I post a comment on Facebook, or drop an email to them. Even to contemplate losing them stops my heart and starts a cold, cold hurt deep within.

I can’t imagine it. I shy away from the very idea. Job lost his whole family, everything, and yet he would not curse God, nor turn from Him. Do I have such a love for Him that I could continue to worship Him in the face of such devastation? To look for comfort, probably. To be angry with Him, almost certainly! But to worship Him? I don’t know, and my prayer is that He would not allow me to suffer to such a degree that I could come to doubt His love for me.

It is a selfish prayer. So many people in this world suffer such loss that I couldn’t bear. I sit here safe and contemplate this from the safety of my warm home, with my family around me, in a safe and comfortable country. Lord God, shatter me, make me dependent on You and You alone. I am too scared to ask this of You. But this is my tiny step of faith. I am like the father who asked Jesus “help me believe!”

Freedom Ride

Uncategorized

Snow to 800 metres. 55km/h winds. Possible hail. That was the forecast that greeted us at 4:30am as we met at Tim’s house, all psyched up for our 200km Freedom Ride over the central plateau back to Hobart. As we drove up north in the bus, 11 testosterone fuelled blokes discussed whether or not we would be so soft that we would fall back on the “easy” bad weather backup route. Or whether we would be hard men and tough out the weather.

As each rider spoke up and agreed that they wanted to do the central plateau route, and then also said that it would probably be a bad idea, I recalled the Launceston – New Norfolk race a few years back, that took the same route, when only 7 pro riders finished in similarly atrocious weather.

Amazingly, we had almost unanimous agreement to go the soft option. Still 200km, still over 2000m climbing, but via the east coast, with its generally milder weather. I was relieved: the plateau route was sounding more and more like a really dangerous choice, with freezing cold, wet and gusty descents and general misery!

We were riding for the charity Live Free Tassie, which works to help young people break free from addictions.  It’s not too late to donate — please do!

Preparing to leave Campbell Town.  Weather looks great so far.

At just after 7am, we set out from Campbell Town towards the east coast, immediately hitting the largest climb of the whole ride, which we took gently while we warmed up. The strong winds were behind us at this point, and we were somewhat sheltered on most of the climb. Over the top of the range it was cold, but it was over surprisingly quickly and with little pain.

A tight well-organised bunch, rolling smoothly

The descent into Swansea was great fun and we rolled into the small seaside town well ahead of schedule. With no sense of urgency, and warm and pleasant weather, the brief toilet stop took somewhat longer than originally intended!

Swansea Stop

We rode the whole route in a tight, well organised bunch, with smooth roll-throughs and good communication through the group. No one got left behind and I really felt like we were all working well together. I’ve never been on a ride before where the bunch stayed organised for 50km, let alone 200km!  Well done Tim (our organiser) for his success in motivating the fast boys to support the bunch!

Once we got to Triabunna, the traffic got a little heavier, and we had a couple of impatient drivers attempt to overtake on blind corners and similarly silly places. As we went through Orford, we had our one and only puncture of the day, and the local police stopped by to ask us to ride single file through the narrow gorge out of Orford to reduce the likelihood of more silly moves by drivers. The puncture meant that this became our lunch stop, and again, the beneficial winds meant we were still ahead of schedule.  So again, we were in no rush…

Lunch stop at Orford, 125km in, everyone looks happy!

Then we turned our noses towards Hobart. And hit the cross-headwinds. 50km/h with lots of gusts thrown in for good measure. Our average speed plummeted, and we were riding at less than 20km/h most of the time now. A few of the riders in the group were starting to feel their legs, and we encouraged them to spend as little time on the front as they possible.  As we rode single file through the gorge, a phalanx of probably 200-odd motorbikes roared past the other way, followed by a long line of classic cars.  There was to be no overtaking anyway…

Mechanical — rear wheel change meant a bit of waiting for the group.  Just before the rain!

We struggled against the wind, as the rain and a little bit of hail started to assault us. But we survived. The worst part, for me, was the steep descents in driving wind and rain. They were not much fun.  At this point, Simon experienced a serious cramp that put him in the bus. But only for 5 minutes! He got out again, like a madman, in time for the sketchy descent. Not sure I could have done that…

The rain let up again after the hills and we just worked our way slowly back into Hobart against the winds.  We finally and triumphantly arrived, the last group to finish for the day, all our early time gains destroyed by the ferocious winds and unexpected stops through to the finish.

Personally, I enjoyed the whole day, bar the sketchy descents and the handful of crazy drivers.  I’ve never been as well prepared for a long ride as I was for this one, despite already having done 280km in the 5 days preceding the ride.  Kudos must go out to Trev, Mark D and Mark P as the oldest in the bunch (by some margin, too), and to Tim who did a brilliant job of organising the ride.  The whole bunch was great to ride with, and I’d do the ride again with the same guys without hesitation.  Thanks for an awesome epic ride, Brendon, Dan, Simon, Ben, Tim, Michael, Ant, Trev, Mark, Mark and Mark!  Updated: And massive thank you to Henry and Henry for driving and supporting on the ride.  Sorry I forgot to mention you initially!