A common problem that you, as a developer, may run into on Windows is the need to replace a file that is in use. This commonly happens with installations and upgrades, but can of course also happen in general use.

In earlier versions of Windows, when most users worked in full administrator mode, the MoveFileEx function with the MOVEFILE_DELAY_UNTIL_REBOOT flag was suggested by Microsoft as a great approach for updating files that were in use.  This flag would, as it sounds, allow you to schedule the move or deletion of a file at a time when it was (pretty much) guaranteed to succeed.

For example:

// This will delete c:\temp\coolcorelibrary.dll on the next reboot
MoveFileEx(“c:\\temp\\coolcorelibrary.dll”, NULL, MOVEFILE_DELAY_UNTIL_REBOOT);

Nowadays, of course, this flag does not work unless you are running in the context of an administrative user.  That’s great, you think, this will still work for my install or upgrade program.

But don’t trust that feeling of security.  Things are never as easy as they seem.  I first realised that this was a problem when researching issues occasionally reported by some of our Keyman Desktop users.

Take this scenario:

1. A user, Joe, decides to uninstall your awesome app CoolCoreProgram.
2. The uninstaller finds that a critical file (let’s call it coolcorelibrary.dll) is in use and can’t delete it
3. Installer calls a MoveFileEx with MOVEFILE_DELAY_UNTIL_REBOOT to schedule deletion of coolcorelibrary.dll.

4. Would you click Restart Now?  Why not leave it till later?

   The uninstall completes and presents Joe with the dreaded “Hey, you need to restart Windows now” dialog.

5. Poor unhappy Joe swears and cancels the restart, and continues his work.  He can’t see any good reason to restart Windows…
6. A short while later, Joe realises that he actually loves CoolCoreProgram and so he downloads and reinstalls the latest, greatest version (now with extra shiny!)
7. Shortly thereafter, Joe finishes up for the day and turns off his computer.
8. The next morning, after Joe starts up his computer, Windows notes its instructions from the previous day, and obediently deletes coolcorelibrary.dll.
9. And now Joe is now really, really unhappy when he tries to start CoolCoreProgram and he gets a bizarre coolcorelibrary.dll missing error.

Who is Joe going to blame?  Is it his fault for not restarting?  Is it yours for using cool APIs such as MoveFileEx?  Or is it a woeful confluence of unintended consequences?

This is probably one of the simplest scenarios in which this problem can crop up.  Things get much worse when you talk about shared libraries, fonts, or other resources which may be in use by the system, or multi-user systems.

Some reasons I have encountered for files in use:

1. Program is still running (duh!)
2. System has locked the file (common with fonts, hook libraries)
3. Antivirus or security software is scanning the file
4. Another application is trying to update the file (i.e. don’t run multiple installers at once)

One possible fix would be to check and update the registry key
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations for listed file before creating or updating them.  But that’s a big performance and reliability hit.  I can’t say that solution sits well with me.

Another idea is to block further install and upgrade tasks until Joe actually does restart his computer, for example with a registry setting or a RunOnce entry.  But that’s going to make Joe hate me even more than he already does!

Some scenarios can be fixed with the Windows Installer automatic repair functionality.  But that assumes access to the original source files is always possible.  So that’s not a general solution.

I don’t really have a good general solution to this problem.  Any ideas?

I was recently tasked with addressing a problem with an application that would not shut down when given a WM_QUERYENDSESSION message. On the surface, this seemed easy enough to fix — just make sure that the WM_QUERYENDSESSION message was appropriately handled by all the top level windows in the application.

Step 1 was to add a handler into the main top level window that avoided the normal shutdown dialog boxes that warned of backup and similar tasks. Where possible, Microsoft advises, you should not block the shutdown of Windows with dialog boxes, etc.

That done, I tested and found that the application was still not shutting down when it was supposed to. So I fired up Spy++ and got started on watching the WM_QUERYENDSESSION chatter in the application. Here I ran into my first little catch-22. I decided that I wanted to do a real shutdown, not a simulated one, to make sure that I was testing the correct scenario.

Problem was, Spy++ was always being shut down before I could capture the messages needed in my target application. Eventually I realised that I needed to start Spy++ after my application, because Windows would ask each application in turn to WM_QUERYENDSESSION, and as we know, my application would cancel the process because it wasn’t accepting a shutdown! Then I realised I could also make sure the shutdown was cancelled even when my application behaved properly, simply by loading Notepad after my application and typing a character or two into it. Then during the shutdown sequence, Iwould be asked if I want to save changes to the text document, and I could just click Cancel. Crude but effective.

With that out of the way, I ran through a Spy++ session and was able to spot an offending window that returned 0 to WM_QUERYENDSESSION. A little tracing and I found a utility window procedure that failed to call DefWindowProc. Naughty. Easily solved.

Well so I thought. Started testing again. The application was still failing to shutdown. I ran through Spy++, traced all the WM_QUERYENDSESSION messages for the process, and every single window procedure responded to the message, virtually instantly, with a nice clean 1, indicating that the window was happy to end the session. After 5 seconds, Windows would show its “Application is not responding” message, indicating that it had not received a response from one of the top level windows.

This was rather puzzling. I threw in a bit of logging into a dll using SetWindowsHookEx for WH_CALLWNDPROC and WH_CALLWNDPROCRET, just in case I was missing something with Spy++. No difference, except that it was somewhat easier to read.

So then I got thinking. This application runs quite a few background threads for loading data from a database. What if, I wondered, one of those threads had a window created but was not running a message loop? It sounded logical but I wasn’t very confident that that was the cause. Problem was, how to tell? I looked for ways to check the queue status for a thread in Windbg but could not find an easy solution. Eventually I decided to add some debugging into the common ThreadProc wrapper for the application, called GetQueueStatus(QS_SENDMESSAGE), and logged the response. Re-ran the application, and bingo, there was a thread closing down with a message in its queue. No prizes if you guess what that message was!

After that, it was simple. Reviewing the thread’s main function I quickly discovered a call to WaitForSingleObject. The thread did not appear to create any windows in any of our code, but it did use COM in its database connectivity, which meant that a helper window was created behind the scenes. I changed the WaitForSingleObject to a MsgWaitForMultipleObjects, and processed any outstanding messages when signalled with a PeekMessage loop.

And then it finally worked (and I forgot to start Notepad, so Windows restarted…) But it really stumped me for a bit today.