Category Archives: Suggestions

Concatenating strings in SQL Server, or undefined behaviour by design

We just ran into a funny problem here, using a “tried and true” technique in SQL Server to concatenate strings. I use the quotes advisedly. This technique is often suggested on blogs and sites such as Stack Overflow, but we found out (by painful experience) that it is not to be relied on.

Update, 9 Mar 2016: Bruce Gordon from Webucator has turned this into a great little 5 minute video. Thanks Bruce! I don’t know anything much about Webucator, but they are doing some good stuff with creating well-attributed videos about blog posts such as this one and apparently they do SQL Server training.

The problem

So, given the following setup:

CREATE TABLE BadConcat (
  BadConcatID INT NOT NULL,
  Description NVARCHAR(100) NOT NULL,
  SortIndex INT NOT NULL
  CONSTRAINT PK_BadConcat PRIMARY KEY CLUSTERED (BadConcatID)
)
GO

INSERT BadConcat 
  SELECT 1, 'First Item', 1 union all
  SELECT 2, 'Second Item', 2 union all
  SELECT 3, 'Third Item', 3
GO

We need to concatenate those Descriptions. I have avoided fine tuning such as dropping the final comma or handling NULLs for the purpose of this example. This example shows one of the most commonly given answers to the problem:

DECLARE @Summary NVARCHAR(100) = ''

SELECT @Summary = @Summary + ec.Description + ', '
FROM BadConcat ec
ORDER BY ec.SortIndex 

PRINT @Summary

And we get the following:

First Item, Second Item, Third Item, 

And that works fine. However, if we want to include a WHERE clause, even if that clause still selects everything, then we suddenly get something weird:

SET @Summary = ''

SELECT @Summary = @Summary + ec.Description + ', '
FROM BadConcat ec
WHERE ec.BadConcatID in (1,2,3)
ORDER BY ec.SortIndex 

PRINT @Summary

Now we get the following:

Third Item, 

What? What has SQL Server done? What’s happened to the first two items?

You’ll probably do what we did, which is to go through and make sure that you are selecting everything properly, which we are, and eventually come to the conclusion that “there must be a bug in SQL Server”.

The answer

It turns out that this iterative concatenation is unsupported functionality. Microsoft Knowledge Base article 287515 states:

You may encounter unexpected results when you apply any operators or expressions to the ORDER BY clause of aggregate concatenation queries.

Now, at first glance that does not directly apply. But we can extrapolate from that, as Microsoft developer support have done, in response to a bug report on SQL Server, to learn that:

The variable assignment with SELECT statement is a proprietary syntax (T-SQL only) where the behavior is undefined or plan dependent if multiple rows are produced

And again, in response to another bug report:

Using assignment operations (concatenation in this example) in queries with ORDER BY clause has undefined behavior. This can change from release to release or even within a particular server version due to changes in the query plan. You cannot rely on this behavior even if there are workarounds.

Some alternative solutions are given, also, in that second report:

The ONLY guaranteed mechanism are the following:

1. Use cursor to loop through the rows in specific order and concatenate the values
2. Use for xml query with ORDER BY to generate the concatenated values
3. Use CLR aggregate (this will not work with ORDER BY clause)

And the article “Concatenating Row Values in Transact-SQL” by Anith Sen goes through some of those solutions in detail. Sadly, none of them are as clean or as easy to understand as that original example.

Another example is given on Stack Overflow, which details how to safely use XML PATH to concatenate, without breaking on the XML special characters &, < and >. Applying that example into my problem code given above, we should use the following:

SELECT @Summary = (
  SELECT ec.Description + ', ' 
  FROM BadConcat ec 
  WHERE ec.BadConcatID in (1,2,3)
  ORDER BY ec.SortIndex 
  FOR XML PATH(''), TYPE
).value('.','varchar(max)')

PRINT @Summary

Voilà.

First Item, Second Item, Third Item, 

Everything you (thought you) knew about Internet password security is wrong

Time and time again, we see calls from security experts for complex passwords.   Make sure your password includes lower case letters, upper case letters (how xenoglossophobic!), numerals, punctuation, star signs, and emoji.  It must be no less than 23 characters long and not use the same character twice.  Change your password every 60 days, every 30 days, every half hour.  Don’t use the same password again this year, or next year, or for the next 6 galactic years.  Never write your password down.  Especially not on a post-it on your monitor.  Or under your keyboard.
d00fus

The Golden Password Rules

And it’s all wrong.  There are just two rules you need to remember, as an Internet password user:

  1. Never use the same password in two places.  Like, if you have a Yahoo account and a Google account, don’t let them share the same password.  They’d be offended if they knew you did anyway.
  2. Make sure your password isn’t “guessable”, like your pet’s name, or your middle name.  Or anyone’s middle name.  Or “password”.  Or anything like that.  But “correct horse battery staple” is probably ok, or it was until xkcd published that cartoon.

It’s all wrong because all that complexity jazz is just part of an arms race in the brute force or dictionary attack password cracking game.

Say what? So a brute force attack is, in its simplest form, a computer — or hundreds of computers — trying everything single password combination they can think of, against your puny little password. And trust me when I say a computer can think of more passwords than you can. Have you ever played Scrabble against a computer?

Brute force attacks on Internet passwords are only effective on well designed sites when that site has already been compromised.  At which point who cares if they know your password to that site (because rule 1, remember): they also know everything else that site has recorded about you.  And anyway you can just change that password.  No problem (for you; the site owners have a big problem).

Now, if you are unlucky enough to be targeted, then complex passwords are not going to save you, because the attackers will find another way to get your data without needing to brute force your Google Apps password.  We’ve seen this demonstrated time and time again.  And if you are not being targeted, then you can only be a victim of random, drive-by style password attacks.  And if you followed rule 2, then random, drive-by style password attacks still won’t get you, because the site owner has followed basic security principles to prevent this.  And if the site owner has not followed basic security principles, then you are stuffed anyway, and password complexity still doesn’t matter.

In fact, complex passwords and cycling regimes actually hurt password security.  The first thing that happens is that users, forced to change passwords regularly, can no longer remember any passwords.  So they start to use easier to guess passwords.  And eventually, their passwords all end up being some variation of “password123!”.

The Bearers of Responsibility

The people who really have to do the hard yards are the security experts, software developers, and the site owners.  They are the keepers of the password databases and bear a heavy burden for the rest of us.  Some suggestions, by no means comprehensive, for this to-be-pitied crew:

  1. Thwart dictionary attacks on Internet-facing password entry.  That is, throttle connection attempts, delay for 15 seconds after 10 attempts, require 2nd level authentication after failed attempts, that kind of thing.  These solutions are well documented.
  2. Control access to your password database (duh).  Remember, in the good ol’ days of Unix, password were stored in /etc/passwd, which was world readable and so the enterprising young hacker could just copy the file and try and crack it in their own good time elsewhere.  So keep other people’s dirty paws off your (hashed) password database.
  3. Don’t ever display passwords in plain text.  No “here is your password” emails.  Not even for registration.  That has to be a one time token.  Your password database is hashed, right?  Not ROT13?
  4. Notify a user if someone tries to access their account multiple times.  Give them the power to fret and stress.
  5. If your site gets hacked, tell your users as soon as you possibly can, and reset your password database.  Mind you, they’ll just have to change their password for your site because they’ve been following rule 1 above, right?  Oh, and don’t be too ashamed to tell anyone.  It happens to all the best site owners and there’s nothing worse than covering it up.

The Flaw in My Rant

Still, my rant has a problem.  It’s to do with Rule 1:  A separate password for every site.  But just how many sites do I have accounts for?  Right now?  402 sites.

Yikes.

How do I manage that?  How can I remember passwords for 402 sites?  I can’t!  So I put them into a database, of course.  Managed by a password app such as KeePass or 1Password or Password Safe. Ironically, as soon as I use a password manager, I no longer have to make up passwords, and they actually end up being random strings of letters, numbers and symbols. Which keeps those old-fashioned password nazis happy too.

Personally, I keep a few high-value passwords out of the password database (think Internet Banking) and memorise them instead.

Of course, my password safe itself has just a single password to be cracked.  And because I (the naive end user) store this database on my computer, it’s probably easy enough to steal once I click on that dodgy link on that site…  At which point, our black hat password cracker can roll out their full armada of brute force password cracking flying robots to get into my password database.  So perhaps you still need that complex password to secure your password database?

What do you think?

Note: Roger Grimes stole my thunder.

Nearly crushed by a cement truck on my ride today

Update 2 July 2014: Added two diagrams to mitigation of Boral Concrete Forecourt

An unfolding story

So, I was nearly crushed by a cement truck today.  It came around a corner, at about 40km/h, without indicating.  I was doing just 25km/h on my bike, which was fortunate, as otherwise I probably would be writing this from a hospital bed, or from a comfy freezer down at the neighbourhood morgue (do they have wifi?).

Perhaps he’d stepped in some cement during his delivery run, and found it hard to ease off the accelerator pedal.  Whatever the case, I don’t want to lay all the blame for this near miss at the feet of the driver.

That’s because the real problem lies with the Hobart City Council. This incident occurred on the primary, and best cycling route North out of the city.  The HCC maps describe this route as a “Off-road – Shared Footway/Bicycle Path.”  I think I will now describe it as an “Off-road – Shared Footway/Bicycle/Cement Truck Path.”

The site in question is the Boral Concrete Depot, through which the cycle route happily wends its way, and is probably the most dangerous of the obstacles which the intrepide commuter cyclist must negotiate on his or her way out of Hobart City.  But it is by no means the only obstacle.

An interview

Before I go into more detail on the obstacles, with pictures and lots of fun, I have taken down an Official Report from myself, viz.:

I was proceeding on my pedal cycle in a Northerly direction, at approximately twenty-five (25) kilometres per hour, through the Forecourt of the Boral Concrete Depot, upon the principal cycle route as shown on Council Maps, and paying due attention to traffic on the adjacent Highway, when my attention was caught by an approaching Cement Mixer Truck (henceforth, CMT).  Said CMT was proceeding in a Southerly direction at a speed which I estimate at no less than forty (40) kilometres per hour, and as CMT had not indicated that it would be leaving the aforementioned Highway, I presumed that it would continue past the entrance into the forecourt.

To my surprise, when the CMT reached the entrance of the Forecourt, it abruptly swung off the Highway and into the Forecourt, at speed, at which point I executed Evasive Manœuvres, to wit, braking sharply and turning my vehicle (2 wheeled pedal cycle) towards the West.  Additionally, I immediately alerted the driver of CMT to the impending danger with a carefully worded, shouted, phrase.

CMT then braked heavily; however this action was no longer necessary as I had already averted the danger with my Evasive Manœuvres.  I then proceeded, unharmed, on my journey, albeit with an elevated heart rate (see Figure 1 – ed).

Heart Rate and Speed
Figure 1 – Heart Rate and Speed
Incident Diagram
Figure 2 – Incident Diagram

Your daily obstacle course commute

The Intercity Cycleway is by far the most established and concrete (there’s that word again) piece of bicycle infrastructure in Hobart. Following the railway line North from the Docks, through Moonah, Glenorchy and Montrose, it is used by hundreds (in Summer, thousands) of cyclists a day for commuting and exercise. And until you reach the Cenotaph, it is, by and large, a decent piece of cycle infrastructure.

The bliss of the Intercity Cycleway
The bliss of the Intercity Cycleway

I think a good question to ask when looking at bicycle infrastructure design is: is it safe for an 8 year old to ride? Not necessarily unaccompanied, but looking more at bicycle control and potential danger points. And at the Cenotaph, things start to go downhill. First, we encounter a confusing road crossing, up-hill, with traffic approaching from 4 different directions. The confusion is mitigated by typically low speeds, but it’s not a good start.

Traffic comes from four different directions as you exit the Intercity Cycleway
Traffic comes from four different directions as you exit the Intercity Cycleway

After crossing the road, a cyclist is presented with two possible routes. The official route heads slightly up hill, and a secondary route heads past the Cenotaph. All well and good, almost.

Approaching the Cenotaph - Two Routes
Approaching the Cenotaph – Two Routes

The “almost” comes into play shortly. The official route turns abruptly at the edge of the highway, where traffic is passing at 70km/h. There is no safety barrier.

Approach the Highway, and Turn Left
Approach the Highway, and Turn Left

Here the path goes downhill, literally. The typical cyclist picks up a bit of speed here, coasting down the hill. We reach the other end of the Cenotaph route.

This point is just plain dangerous, which is no doubt why the newer, ‘official’ route was introduced. However, without signage or recommendation, there is nothing to encourage riders to use the slightly less dangerous, slightly longer route. So what’s the problem?

Mind you don't miss the corner!
Mind you don’t miss the corner!
  1. There is a conflict point with cyclists merging, at speed, coming down hill both on the official route, and the Cenotaph route. This can be a conflict with pedestrians as well.
  2. Worse, cyclists coming down the Cenotaph route run a significant risk of overshooting, if not careful, into the highway. I have seen a cyclist do this. They were lucky: no cars were in the near lane.

Now we approach the bottom of the hill, with a blind corner. Pedestrians regularly round this corner on the “wrong” side of the shared path. Cyclists should ride their brakes down here to avoid picking up too much speed.

Approaching Boral Concrete
Approaching Boral Concrete

Confusion ensues: there are three marked routes here. Which is the proper route? The only frequently used route is the closest exit onto the forecourt roadway. But this exit is also the most dangerous, as I found today. The two more distant exits are just awkward to access. This forecourt is dangerous: with traffic entering from the highway, potentially at speed, and trucks turning and reversing, it’s just not a great place for bikes. Yet it is smack bang on the primary bike route into Hobart.

The iPhone does a Telephoto Spy Shot into Boral Concrete's Depot
The iPhone does a Telephoto Spy Shot into Boral Concrete’s Depot
The Forecourt, Heading North
The Forecourt, Heading North
Yes, Ride Past the No Entry Sign
Yes, Ride Past the No Entry Sign to exit South
The Route North to the Intercity Cycleway in all its glory
The Route North to the Intercity Cycleway in all its glory
One of the three off ramps into the forecourt
One of the three off ramps into the forecourt

Things improve a little on the far side: we have a reasonably well marked pathway, albeit with another sharp corner right on the edge of the highway.

Turn Hard Left.  This does not qualify as high quality infrastructure, sorry!
Turn Hard Left. This does not qualify as high quality infrastructure, sorry!

Now we are faced with a traffic light pole in the middle of the path, narrowing the path in one direction to less than a metre right beside a very busy roadway. That’s nasty.

The Pole
The Pole

The next section, however, is quite pleasant, offset from the road and through an avenue of trees. Apart from some minor maintenance on the ‘cobblestones’ to level them out, I have no complaints.

Pleasant Times
Pleasant Times

Now we come to the Hobart Docks precinct. First we have a road crossing, with a separate light for and control system for bicycles. I’m not sure why. The button is on the wrong side of the path, causing conflict for oncoming bicycles.

Road crossing
Road crossing

Enough has been said about the placement of this Cafe. But perhaps the signs which frequently encroach into the bike lane (not too badly in the photo today, but worse on other days) should be relocated.

The Cafe
The Cafe
A Sign Encroaches
A Sign Encroaches

Crossing the docks themselves is not ideal, with a path shared with pedestrians and parking cars. But it is a low speed area and most of the conflicts are overcome without too much trouble. However, the Mures carpark entrance is still dangerous, even with the upgraded crossing treatment. Sight lines are poor and I have observed drivers crossing this intersection at speed, attempting to make it into a break in the traffic on Davey St.

Crossing the Mures Entrance
Crossing the Mures Entrance

Finally, we have another shared path, with a somewhat ambiguously marked bike lane on the street side of it. Perhaps it would be better to treat the whole path as shared, and not ‘reserve’ a section for bikes if it isn’t going to be clearly marked, but it’s not a big issue.

Shared Path Past Docks
Shared Path Past Docks

Mitigations

The sections of the track that need attention most urgently are those along the edge of the highway, and where the route crosses the Boral Concrete forecourt area.

Engineers Australia Building

Travelling from the city this time, the first danger point, where the path traverses the edge of the highway and narrows around the traffic light pole, could be improved by shifting the bike path away from the edge of the road, and across the otherwise empty lawns outside the Engineers Australia building. No doubt there are some property boundary issues there. But perhaps it wouldn’t hurt to ask them? Even a one or two metre setback would improve the situation considerably.

Adjusting the shared path past the Engineers Australia building
Adjusting the shared path past the Engineers Australia building

Boral Concrete Forecourt

The safest solution to this area would be to close the car and truck access to and from the highway entirely, and reroute traffic to Boral Concrete and the Engineers Australia building through the dockyards. This would also address the problematic entrance of vehicles onto the highway in the middle of a major intersection.

Alternative access to Boral Concrete
Alternative access to Boral Concrete
Close highway access to forecourt
Close highway access to forecourt

This may be a hard sell, however if the Hobart City Council wants to increase the bike share into the city, it will need to take serious steps to improve the safety of this primary route through this area.

Realignment of path past Cenotaph

The bike path along the side of the highway could be rerouted behind the Cenotaph, or with some work, shifted away from the edge of the highway. Alternatively, a safety barrier could be put into place along the path beside the highway.

Alternate Cenotaph Routes: both would take some work
Alternate Cenotaph Routes: both would take some work

I’ve been wanting to write this post for quite a while. The Incident of the Cement Truck was sufficient to rekindle my blogging ‘abilities’. Other posts in the Hobart Bike Infrastructure series:

Fifty-nine vulnerabilities, or do you feel safe using Windows XP?

In today’s Microsoft Security Bulletin release was a very long list of vulnerabilities fixed in Internet Explorer. A very long list. 59 separate vulnerabilities to be exact. I do believe that is a record.

But I’m not here to talk about the record — I am more interested in the steps Windows XP users will take to mitigate the flaws, because Microsoft are not patching any of these vulnerabilities for Windows XP! Some people I’ve talked to, from individuals up to enterprises, seem to have the idea that they’ll practice “Safe Computing” and be able to continue using Windows XP and avoid paying for an upgrade.

What do I mean by Safe Computing? Y’know, don’t open strange attachments, use an alternate web browser, view emails with images switched off, keep antivirus and malware protection software up to date, remove unused applications, disable unwanted features, firewalls, mail and web proxies, so on and so forth.

So let’s look at what the repercussions are of practicing Safe Computing in light of these disclosures.

The first mitigation you are going to take is, obviously, to stop using Internet Explorer. With this many holes, you are clearly not going to be able to use Internet Explorer at all. This means a loss of functionality, though: those Internet Explorer-optimised sites (I’m looking at you, just about every corporate intranet) often don’t even work with non-IE browsers. So if you have to use IE to view these ‘trusted’ sites, you must ensure you never click on an external link, or you will be exposed again. Doable, but certainly a hassle.

Okay, so you don’t use IE now. You use Firefox, or Chrome. But you’re still in trouble, because it turns out that the very next security bulletin announces that GDI+ and Uniscribe are both vulnerable as well, today. GDI+ is used to display images and render graphics in Windows, and Uniscribe is used by just about every application to draw text onto the screen, including all the major web browsers. The Uniscribe flaw relates to how it processes fonts. The GDI+ flaw relates to a specific metafile image format.

So, disable support for downloadable fonts in your browser, and disable those specific metafile image types in the Windows Registry. Yes, it can be done. Now you’ll be all good, right? You don’t need those fonts, or those rare image types, do you? You can still surf the web okay?

But you’ve lost functionality, which we might not value all that highly, but it’s still a trade-off you’ve had to make.

From today, every security flaw that is announced will force you to trade more functionality for security.

And this is my point. From today, and on into the future, every security flaw that is announced will force you to trade yet more functionality for security. Eventually, you will only be able to use Windows XP offline — it simply will not be possible to safely access Internet resources without your computer and your data being compromised. It’s going to get worse from here, folks. It is well and truly past time to upgrade.

Only 21? Do you feel safe yet?

Using Delphi attributes to unify source, test and documentation

Updated 28 May 2014: Removed extraneous unit reference from AttributeValidation.pas. Sorry…

What problem was I trying to solve?

Recently, while using the Indy Internet components in Delphi XE2, I was struggling to track the thread contexts in which certain code paths ran, to ensure that resource contention and deadlocks were correctly catered for.

Indy components are reasonably robust, but use a multithreaded model which it turns out is difficult to get 100% correct.  Component callbacks can occur on many different threads:

  • The thread that constructed the component
  • The VCL thread
  • The server listener thread
  • The connection’s thread
  • Some, e.g. exceptions, can occur on any thread

Disentangling this, especially when in conjunction with third party solutions that are based on Indy and may add several layers of indirection, quickly becomes an unenjoyable task.

I started adding thread validation assertions to each function to ensure that I was (a) understanding which thread context the function was actually running in, and (b) to ensure that I didn’t call the function in the wrong context myself.  However, when browsing the code, it was still very difficult to get a big picture view of thread usage.

Introducing attributes

Enter attributes.  Delphi 2010 introduced support for attributes in Win32, and a nice API to query them with extended Run Time Type Information (RTTI).  This is nice, except for one thing: it’s difficult at runtime to find the RTTI associated with the current method.

In this unit, I have tried to tick a number of boxes:

  • Create a simple framework for extending runtime testing of classes with attributes
  • Use attributes to annotate methods, in this case about thread safety, to optimise self-documentation
  • Keep a single, consistent function call in each member method, to test any attributes associated with that method.
  • Sensible preprocessor use to enable and disable both the testing and full RTTI in one place.

One gotcha is that by default, RTTI for Delphi methods is only available for public and published member methods.  This can be changed with the $RTTI compiler directive but you have to remember to do it in each unit!  I have used a unit-based $I include in order to push the correct RTTI settings consistently.

I’ve made use of Delphi’s class helper model to give direct access to any object at compile time.  This is a clean way of injecting this support into all classes which are touched by the RTTI, but does create larger executables.  I believe this to be a worthwhile tradeoff.

Example code

The code sample below demonstrates how to use the attribute tests in a multi-threaded context. In this example, an assertion will be raised soon after cmdDoSomeHardWorkClick is called. Why is this? It happens because the HardWorkCallback function on the main thread is annotated with [MainThread] attribute, but it will be called from TSomeThread‘s thread context, not the main thread.

In order for the program run without an assertion, you could change the annotation of HardWorkCallback to [NotMainThread]. Making this serves as an immediate prompt that you should not be accessing VCL properties, because you are no longer running on the main thread. In fact, unless you can prove that the lifetime of the form will exceed that of TSomeThread, you shouldn’t even be referring to the form. The HardWorkCallback function here violates these principles by referring to the Handle property of TForm. However, because we can show that the form is destroyed after the thread exits, it’s safe to make the callback to the TAttrValForm object itself.

You can download the full source for this project from the link at the bottom of this post in order to compile it and run it yourself.

Exercise: How could you restructure this to make HardWorkCallback thread-safe? There’s more than one way to skin this cat.

unit AttrValSample;

interface

uses
  System.Classes,
  System.SyncObjs,
  System.SysUtils,
  System.Variants,
  Vcl.Controls,
  Vcl.Dialogs,
  Vcl.Forms,
  Vcl.Graphics,
  Vcl.StdCtrls,
  Winapi.Messages,
  Winapi.Windows,

  {$I AttributeValidation.inc};

type
  TSomeThread = class;

  TAttrValForm = class(TForm)
    cmdStartThread: TButton;
    cmdDoSomeHardWork: TButton;
    cmdStopThread: TButton;
    procedure cmdStartThreadClick(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
    procedure cmdStopThreadClick(Sender: TObject);
    procedure cmdDoSomeHardWorkClick(Sender: TObject);
  private
    FThread: TSomeThread;
  public
    [MainThread] procedure HardWorkCallback;
  end;

  TSomeThread = class(TThread)
  private
    FOwner: TAttrValForm;
    FEvent: TEvent;
    [NotMainThread] procedure HardWork;
  protected
    [NotMainThread] procedure Execute; override;
  public
    [MainThread] constructor Create(AOwner: TAttrValForm);
    [MainThread] destructor Destroy; override;
    [MainThread] procedure DoSomeHardWork;
  end;

var
  AttrValForm: TAttrValForm;

implementation

{$R *.dfm}

procedure TAttrValForm.cmdStartThreadClick(Sender: TObject);
begin
  FThread := TSomeThread.Create(Self);

  cmdDoSomeHardWork.Enabled := True;
  cmdStopThread.Enabled := True;
  cmdStartThread.Enabled := False;
end;

procedure TAttrValForm.cmdDoSomeHardWorkClick(Sender: TObject);
begin
  FThread.DoSomeHardWork;
end;

procedure TAttrValForm.cmdStopThreadClick(Sender: TObject);
begin
  FreeAndNil(FThread);
  cmdDoSomeHardWork.Enabled := False;
  cmdStopThread.Enabled := False;
  cmdStartThread.Enabled := True;
end;

procedure TAttrValForm.FormDestroy(Sender: TObject);
begin
  FreeAndNil(FThread);
end;

procedure TAttrValForm.HardWorkCallback;
begin
  ValidateAttributes;
  SetWindowText(Handle, 'Hard work done');
end;

{ TSomeThread }

constructor TSomeThread.Create(AOwner: TAttrValForm);
begin
  ValidateAttributes;
  FEvent := TEvent.Create(nil, False, False, '');
  FOwner := AOwner;
  inherited Create(False);
end;

destructor TSomeThread.Destroy;
begin
  ValidateAttributes;
  if not Terminated then
  begin
    Terminate;
    FEvent.SetEvent;
    WaitFor;
  end;
  FreeAndNil(FEvent);
  inherited Destroy;
end;

procedure TSomeThread.DoSomeHardWork;
begin
  ValidateAttributes;
  FEvent.SetEvent;
end;

procedure TSomeThread.Execute;
begin
  ValidateAttributes;
  while not Terminated do
  begin
    if FEvent.WaitFor = wrSignaled then
      if not Terminated then
        HardWork;
  end;
end;

procedure TSomeThread.HardWork;
begin
  ValidateAttributes;
  FOwner.HardWorkCallback;
end;

end.

The AttributeValidation.inc file referenced in the uses clause above controls RTTI and debug settings, in one line. This pattern makes it easy to use the unit without forgetting to set the appropriate RTTI flags in one unit.

// Disable the following $DEFINE to remove all validation from the project
// You may want to do this with {$IFDEF DEBUG} ... {$ENDIF}
{$DEFINE ATTRIBUTE_DEBUG}

// Shouldn't need to touch anything below here
{$IFDEF ATTRIBUTE_DEBUG}
{$RTTI EXPLICIT METHODS([vcPrivate,vcProtected,vcPublic,vcPublished])}
{$ENDIF}

// This .inc file is also included from AttributeValidation.pas, so
// don't use it again in that context.
{$IFNDEF ATTRIBUTE_DEBUG_UNIT}
AttributeValidation
{$ENDIF}

Finally, the AttributeValidation.pas file itself contains the assembly stub to capture the return address for the caller, and the search through the RTTI for the appropriate method to test in each case. This will have a performance cost so should really only be present in Debug builds.

unit AttributeValidation;

interface

{$DEFINE ATTRIBUTE_DEBUG_UNIT}
{$I AttributeValidation.inc}

uses
  System.Rtti;

type
  // Base class for all validation attributes
  ValidationAttribute = class(TCustomAttribute)
    function Execute(Method: TRTTIMethod): Boolean; virtual;
  end;

  // Will log to the debug console whenever a deprecated
  // function is called
  DeprecatedAttribute = class(ValidationAttribute)
    function Execute(Method: TRTTIMethod): Boolean; override;
  end;

  // Base class for all thread-related attributes
  ThreadAttribute = class(ValidationAttribute);

  // This indicates that the procedure can be called from
  // any thread.  No test to pass, just a bare attribute
  ThreadSafeAttribute = class(ThreadAttribute);

  // This indicates that the procedure must only be called
  // in the context of the main thread
  MainThreadAttribute = class(ThreadAttribute)
    function Execute(Method: TRTTIMethod): Boolean; override;
  end;

  // This indicates that the procedure must only be called
  // in another thread context.
  NotMainThreadAttribute = class(ThreadAttribute)
    function Execute(Method: TRTTIMethod): Boolean; override;
  end;

  TAttributeValidation = class helper for TObject
{$IFDEF ATTRIBUTE_DEBUG}
  private
    procedure IntValidateAttributes(FReturnAddress: UIntPtr);
{$ENDIF}
  protected
    procedure ValidateAttributes;
  end;

implementation

uses
  Winapi.Windows,

  classes;

{ TAttributeValidation }

{
 Function:    TAttributeValidation.ValidateAttributes

 Description: Save the return address to an accessible variable
              on the stack.  We could do this with pure Delphi and
              some pointer jiggery-pokery, but this is cleaner.
}
{$IFNDEF ATTRIBUTE_DEBUG}
procedure TAttributeValidation.ValidateAttributes;
begin
end;
{$ELSE}
{$IFDEF CPUX64}
procedure TAttributeValidation.ValidateAttributes;
asm
  push rbp
  sub  rsp, $20
  mov  rbp, rsp
                          // rcx = param 1; will already be pointing to Self.
  mov  rdx, [rbp+$28]     // rdx = param 2; rbp+$28 is return address on stack
  call TAttributeValidation.IntValidateAttributes;

  lea  rsp, [rbp+$20]
  pop  rbp
end;
{$ELSE}
procedure TAttributeValidation.ValidateAttributes;
asm
                            // eax = Self
  mov edx, dword ptr [esp]  // edx = parameter 1
  call TAttributeValidation.IntValidateAttributes
end;
{$ENDIF}

{
 Function:    TAttributeValidation.IntValidateAttributes

 Description: Find the closest function to the return address,
              and test the attributes in that function.  Assumes
              that the closest function is the correct one, so
              if RTTI is missing then you'll be in a spot of
              bother.
}
procedure TAttributeValidation.IntValidateAttributes(FReturnAddress: UIntPtr);
var
  FRttiType: TRttiType;
  FClosestRttiMethod, FRttiMethod: TRTTIMethod;
  FAttribute: TCustomAttribute;
begin
  with TRttiContext.Create do
  try
    FRttiType := GetType(ClassType);
    if not Assigned(FRttiType) then Exit;

    FClosestRttiMethod := nil;

    // Find nearest function for the return address
    for FRttiMethod in FRttiType.GetMethods do
    begin
      if (UIntPtr(FRttiMethod.CodeAddress) <= FReturnAddress) then
      begin
        if not Assigned(FClosestRttiMethod) or
            (UIntPtr(FRttiMethod.CodeAddress) > UIntPtr(FClosestRttiMethod.CodeAddress)) then
          FClosestRttiMethod := FRttiMethod;
      end;
    end;

    // Check attributes for the function
    if Assigned(FClosestRttiMethod) then
    begin
      for FAttribute in FClosestRttiMethod.GetAttributes do
      begin
        if FAttribute is ValidationAttribute then
        begin
          if not (FAttribute as ValidationAttribute).Execute(FClosestRttiMethod) then
          begin
            Assert(False, 'Attribute '+FAttribute.ClassName+' did not validate on '+FClosestRttiMethod.Name);
          end;
        end;
      end;
    end;
  finally
    Free;
  end;
end;
{$ENDIF}

{ ValidationAttribute }

function ValidationAttribute.Execute(Method: TRTTIMethod): Boolean;
begin
  Result := True;
end;

{ MainThreadAttribute }

function MainThreadAttribute.Execute(Method: TRTTIMethod): Boolean;
begin
  Result := GetCurrentThreadID = MainThreadID;
end;

{ NotMainThreadAttribute }

function NotMainThreadAttribute.Execute(Method: TRTTIMethod): Boolean;
begin
  Result := GetCurrentThreadID <> MainThreadID;
end;

{ DeprecatedAttribute }

function DeprecatedAttribute.Execute(Method: TRTTIMethod): Boolean;
begin
  OutputDebugString(PChar(Method.Name + ' was called.'#13#10));
  Result := True;
end;

end.

There you have it — a “real” use case for attributes in Delphi. The key advantages I see to this approach, as opposed to, say function-level assertions, is that a birds-eye view of your class will help you to understand the preconditions for each member function, and these preconditions can be consistently and simply tested.

Using a class helper makes it easy to inject the additional functionality into every class that is touched by attribute validation, without polluting the class hierarchy. This means that attribute tests can be seamlessly added to existing infrastructure and Delphi child classes such as TForm.

Full source: AttrVal.zip. License: MPL 2.0. YMMV and use at your own risk.

Generics and Delphi enumerated types without RTTI

Some Delphi types do not have RTTI. This is no fun. This happens when, and I quote:

whereas enumerated constants with a specific value, such as the following, do not have RTTI:
type SomeEnum = (e1 = 1, e2 = 2, e3 = 3);

In normal use, this will go unnoticed, and not cause you any grief, until you throw these enumerated types into a generic construct (or have any other need to use RTTI). As soon as you do that, you’ll start getting the unhelpful and misleading “Invalid Class Typecast” exception. (No it’s not a Class!)

To avoid this problem, you must wander into the dark world of pointer casting, because once you are pointing at some data, Delphi no longer cares what its actual type is.

Here’s an example of how to convert a Variant value into a generic type, including support for RTTI-free enums, in a reasonably type-safe way. This is part of a TNullable record type, which mimics, in some ways, the .NET Nullable type. The workings of this type are not all that important for the example, however. This example works with RTTI types, and with one byte non-RTTI enumerated types &mdash you’d need to extend it to support larger enumerated types. While I could reduce the number of steps in the edge case by spelunking directly into the Variant TVarData, that would not serve to clarify the murk.

constructor TNullable<T>.Create(AValue: Variant);
type
  PT = ^T;
var
  v: Byte;
begin
  if VarIsEmpty(AValue) or VarIsNull(AValue) then
    Clear
  else if (TypeInfo(T) = nil) and
    (SizeOf(T) = 1) and
    (VarType(AValue) = varByte) then
  begin
    { Assuming an enum type without typeinfo, have to
      do some cruel pointer magics here to avoid type
      cast errors, so am very careful to validate
      first! }
    v := AValue;
    FValue := PT(@v)^;
  end
  else
    Create(TValue.FromVariant(AValue).AsType<T>);
end;

So what is going on here? Well, first if we are passed Null or “Empty” variant values, then we just clear our TNullable value.

Otherwise we test if (a) we have no RTTI for our generic, and (b) it’s one byte in size, and (c) our variant is also a Byte value. If all these prerequisites are met, we perform the casting, in which we hark back to the ancient incantations with a pointer typecast, taking the address of the value and dereferencing it, fooling the compiler along the way. (Ha ha!)

Finally, we find a modern TValue incantation suffices to wreak the type change for civilised types such as Integer or String.

Testing for design time in Delphi, in an initialization section

Sometimes it can be handy to test for design-time in a component unit when the component package is first loaded, e.g. within an initialization section, rather than when a component is created or registered. We use this to validate that runtime units that interoperate with a component are linked into a project, and raise an error as early as possible if they are not.

With Delphi’s RTTI, this is fairly straightforward, I believe:

function IsDesignTime: Boolean;
begin
  Result := TRttiContext.Create.FindType('ToolsAPI.IBorlandIDEServices') <> nil;
end;

Is there anything wrong with this?

Rant: Why can’t Microsoft provide actually useful titles on their updates?

Windows Updates have improved dramatically over the last few years.  With Windows 7, the integrated updates install smoothly and without much fuss (apart from the occasional EULA or Internet Explorer Upgrade to throw a spanner in the works).

There’s just one thing.  In general, the update titles are useless.  Completely useless. “Security Update for Windows 7”? Why else would I be running Windows Update?

update-2

Furthermore, the detailed description is also useless — it doesn’t actually provide any details!  It’s even more ambiguous than the title! “A security issue has been identified in a Microsoft software product that could affect your system.”

update-1

Let’s look at what’s wrong with “Update for Windows 7 for x64-based Systems (KB2830477)”:

  • It doesn’t tell us what the update actually provides
  • We already know it’s for Windows 7 — that’s in the group title.
  • We don’t need to know it’s for x64-based Systems — Windows Update won’t serve us updates for the wrong system type

We couldn’t we see “Update for RemoteApp and Desktop Connections features is available for Windows (KB2830477)”, instead? So which sleeve did I pull that descriptive and useful title from?

Well, the thing is, Microsoft already do know exactly what the update is providing.  They have even taken the time to write a succinct title for the update: it’s the title of the Knowledge Base article associated with the update, and it’s even linked to from the update. For example, instead of “Update for Windows 7 (KB2852386)”, we could have “Update: Disk Cleanup Wizard addon lets users delete outdated Windows updates on Windows 7 SP1 (KB2852386)”

Now it’s even worse when using WSUS — you now have to trawl through hundreds of nearly identically titled updates, with only a KB article number to differentiate.  So easy to accidentally approve the wrong update.  Why, Microsoft, why?  Is it so you don’t scare consumers who don’t understand what the update provides?  They just press the big “Automatic Updates” button anyway!

update-4

Admittedly, Microsoft have taken a big step in the right direction with Visual Studio updates: the description for Visual Studio updates generally gives you some information about what is being updated:

update-3

But even that could be improved. We’ve got a lot of repeated information: “Visual Studio 2010” is referenced 4 times: in the group title, in the update title, in the update title in the preview pane, and in the description of the update, again in the preview pane! Surely we don’t need to know that 4 times! And why don’t we go with a title of “Update fixes coded UI test issues for Visual Studio 2010 SP1 in IE9 or IE10 when KB 2870699 is installed (KB2890573)”. Sure it’s a little bit long, but it’s better than “Update for Microsoft Visual Studio 2010 Service Pack 1 (KB2890573)”.

So in conclusion, may I ask you, Microsoft, please, fix these update titles? Just start giving us titles that mean something? And if you are feeling particularly generous, you could even update the description of the update to add more meaning, not less!

Delphi’s TJSONString.ToString is broken, and how to fix it

As per several QC reports, Data.DBXJSON.TJSONString.ToString is still very broken. Which means, for all intents and purposes, TJSONAnything.ToString is also broken. Fortunately, you can just use TJSONAnything.ToBytes for a happy JSON outcome.

The following function will take any Delphi JSON object and convert it to a string:

function JSONToString(obj: TJSONAncestor): string;
var
  bytes: TBytes;
  len: Integer;
begin
  SetLength(bytes, obj.EstimatedByteSize);
  len := obj.ToBytes(bytes, 0);
  Result := TEncoding.ANSI.GetString(bytes, 0, len);
end;

Because TJSONString.ToBytes escapes all characters outside U+0020-U+007F, we can assume that the end result is 7-bit clean, so we can use TEncoding.ANSI.  You could instead stream the TBytes to a file or do other groovy things with it.

Second Rant: Why oh why does Adobe Reader need to restart?

Updated: 3:50pm — It gets worse — see below!
 
Have you ever restarted Windows (e.g. after installing Windows updates) and then been presented with the following dialog:

 And of course, once you click that Install button, you eventually end up with the following:

(For the pedantic reader, yes, I captured different versions of Adobe Reader Updater on different machines.  I’ve had to deal with this on 3 machines so far today. Get over it!)

Here’s why Adobe Reader needs to restart, even if you have nothing apparently running: the installer does not shut down the background processes AdobeARM.exe and reader_sl.exe (and possibly others).

For example on my Windows 7 x64 test machine, in Event Viewer, you can see the following events.

Product: Adobe Reader 9.5.2. The file C:\Program Files (x86)\Adobe\Reader 9.0\Reader\reader_sl.exe is being used by the following process: Name: reader_sl , Id 3084.

Product: Adobe Reader 9.5.2. The file C:\Program Files (x86)\Common Files\Adobe\ARM\1.0\AdobeARM.exe is being used by the following process: Name: AdobeARM , Id 3096.

Both these processes are part of Adobe Reader, and they should be shut down by the installer.  But the updater does not do this.  So, you are forced to restart.  This happens even if you don’t open your web browser or any other applications!

So before clicking that fateful Install button, go and kill those Adobe processes hanging around so you can avoid that restart.  Also, you will need to close your web browser(s), Office applications, and anything else that you can think of that may also be involved.  It’s a pain, but it’s still better than restarting.

Update: Killing AdobeARM.exe at the wrong time is a bit of an issue.  Because AdobeARM.exe presents the front end to the updater!  This means that if you kill AdobeARM.exe, the update dialog also disappears.  The tricky fix then, is to kill AdobeARM.exe just after clicking the Install button.  You get no feedback on the install, but you can see msiexec.exe (up to 3 instances during the install) running in Task Manager or Process Explorer.  When it finishes, two of those will close down (the last instance hangs around for a while).  It will usually leave a happy message in the Event Log.

A little extra rant?  How poor is that design? Pushing updates for the updater in such a way that it forces a restart…  A little bit of forethought could have avoided that one, Adobe.