Category Archives: Tools

A story, or will pay anybody five pounds to remove database from one computer to another

Today I needed to download a recent backup of a database from a server in the data centre for testing major changes to the database locally…  Things don’t really go as well as I expect:

  1. Backup is 12GB.  Yike, that’s going to take ages to download!
  2. OK, so let’s zip it.  20 minutes later… we have a 2GB zip file.  Well, that was worth doing.
  3. Now copy (encrypted) zip to a server with HTTP access (much faster than downloading over SSH).  About 1 minute (yep, nice fast network in the data centre)
  4. Download zip to database server on local network.  Takes about 45 minutes.
  5. So try to unzip on server with Win Server 2003 Zip tool.  Huh.  It doesn’t support files > 4GB (plus it’s running low on disk space, so a bit of a juggling act to have enough space to unzip anyway).
  6. OK, move the zip file to a client machine.  5 minutes on LAN
  7. Unzip locally then move 12GB backup back to server.  Best part of an hour.  Yes, it’s a slow client machine and the LAN is only 100mbit.
  8. Try to restore to SQL Server.  Oops, old version of SQL Server on that server.
  9. So install SQL Server 2008 on another server.  That takes 45 minutes (including adding .NET framework).
  10. Can’t do anything with it until we have patched SQL Server.  So download and run SQL Server 2008 SP3.  20 minutes to download.
  11. Installer starts while I’m not at computer, automatically cancels when screensaver starts (consent.exe)
  12. Drat.  Download SQL Server 2008 SP3 again as IE has deleted installer from cache.  This time I save the service pack installer.  20 minutes.
  13. Finally! Install SQL Server 2008 SP3.  10 minutes.
  14. Restore database to new server.  5 minutes.

And finally it works.

Why do the little things take so much energy?

Indy Sockets: an example of how to not distribute software libraries

Indy Sockets is a library of network components for Embarcadero Delphi, which has been included in the Delphi distribution for many years now.  Like all libraries, bug fixes and patches are regularly made to the source and thus it is typically a good idea to update your the library periodically.
Unfortunately the method of distribution that the Indy Sockets developers have chosen could be a classic example of how to not distribute software libraries.  Here’s why.
When you visit the Indy download page, you are presented with the following blurb:

Installation

Development Snapshot – Instructions to obtain live source and compile manually.

Alternatively you can download the Source Code for Version 10.0.52. This is however a rather old version and is no longer recommended.

So naturally, one follows the Development Snapshot link as the rather old version is no longer recommended, right?  Once you get to the Development Snapshot page, you are then presented with the following scary message:

“You are being warned. This will provide you with a direct link into our current development files. At various times the files may not compile, or in some cases may cause strange errors. Use at your own risk! However please see the version specific notes below. If you are unlucky enough to download a bad version please try again a little later. We apologise for any inconvenience caused.”

This means that when you download Indy, you have no guarantee of getting a stable or even a compiling version.  There is no version information, and you have to rely on their SVN commit logs to figure out what the status of the libraries are.  Oh yes, and the “version specific notes” are missing.

Poor show.

Netbook Surgery

Joey’s computer had a little accident recently and is due to go to hospital shortly for some serious transplant work once a donor can be located.

However before sending Joey’s netbook to the doctor we wanted to take a backup of the drive, as Acer will wipe and reinstall the drive in the repair process. Joey had a recent backup but not everything from the last few days. Oh yes, her netbook doesn’t start up.

So I figured I’d swap the hard drive out of her netbook into mine, and back it up onto a USB hard drive. It was 2 minutes work to pull the drive out of her netbook:

2 exterior screws, 1 interior screw and 4 screws on the hard drive enclosure. Quick and easy.

Mine proved a little more complicated. To get to the hard drive, I had to remove 21 screws, unclip the casing, remove the keyboard, the mainboard, the wi-fi card, a secondary circuit board, undo the screen hinge mount, and unplug 6 different cables:

It worked though: I was able to take a backup of the drive and wipe it, before reassembling both computers. Yay!

Now to see how repairable Joey’s netbook is… This bit is Acer’s job!

Fixing Windows font scaling without restarting

Windows 7 and Windows Server 2008 include the ability for each user to set their font scale. This is fantastic, except for a legacy complication: the old bitmap fonts MS Sans Serif, MS Serif and Courier have specific versions for each font scale, but these are never changed after Windows is installed. In previous versions of Windows, the fonts were replaced with the correct versions for the selected font scale, which is why a system restart was required

This means that these bitmap fonts can be out of sync with the currently selected font scale. This is typically only a problem for legacy applications, but it is ugly in those cases!

More background is available at the MSDN blog http://blogs.msdn.com/b/developingfordynamicsgp/archive/2009/11/25/windows-7-bitmap-fonts-and-microsoft-dynamics-gp.aspx and the follow-up post http://blogs.msdn.com/b/developingfordynamicsgp/archive/2009/12/02/more-on-windows-7-bitmap-fonts-and-dpi-settings.aspx

In our situation, it was even worse: the client was running a Remote Desktop Services environment, where restarting the server was really out of the question.

So I wrote a little fix-it app that dynamically adjusts all the font scaling registry settings and installs the correct fonts for the selected font scale.  You may need to log off and log on again, but in most cases, no restart is required.  It is setup for 100% and 125% only, and I provide this app here only as a useful tool.  No support or warranties, etc, etc.  Use at your own risk!

Download Fontsizefix.zip.

Update 1 Jul: As I discussed this blog with Peter Constable, I realised that I didn’t really describe what the tool did.  So: fontsizefix updates the various metrics in HKCU\Control Panel\Desktop, and a couple of LogPixels registry settings in HKLM\SYSTEM\CurrentControlSet\Hardware Profiles\CurrentSoftware\Fonts and HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\FontDPI\LogPixels, updates the fonts key in the registry to point to the correct versions of MS Sans Serif, MS Serif and Courier, and then RemoveFontResource and AddFontResource in order to get the correct version of the font loaded.  I’m sure it’s not 100% but it got us over a hurdle with the terminal services environment.  For purposes of support, it was easiest to make a tool that did the whole lot rather than document a bunch of registry tweaks which are easy to trip over on, and then we figured we might as well make it available to other users as well…

Notes on a virus

 This morning a coworker logged in and was greeted with a dialog box he did not expect:

There is no disk in the drive. Please insert a disk into drive \Device\Harddisk2\DR6.

This was immediately suspicious.  We quickly fired up Process Explorer to identify what could be causing the problem, and found a cmd.exe process that had been spawned by winlogon.exe, with the following command line:

“C:\Windows\System32\cmd.exe” /c echo [autorun]>>F:\autorun.ini
Alarm bells ringing!  There’d be no reason a legitimate app would be trying to do this…  F: is a card reader on my coworker’s machine, which he has never used, and hence is empty, which is why this message arose in the first place.  He is running the most recent version of F-Prot, with up to date signatures, but that did not pick up this malware.
Now we knew there was something malicious afoot on his computer, we decided to try and capture as much information about the issue as possible.  Unfortunately, we missed the first few steps that the virus was taking, and have no information on how it got onto to the computer in the first place, but here’s the rest of what we found.
The first thing we spotted was that winlogon.exe had a version resource indicating it was created by Microsoft, but it also had no valid certificate, and the details were not consistent:
Curiously enough, we also spotted that it was a .NET application.  I haven’t seen a .NET virus before…  Looking at the process details, we could see it was running from %AppData%, which is certainly not kosher:
We took a look at the folder, and saw a couple of other files, spread.exe and MSNMessengerAPI.dll.  Looks like it may spread through Windows Messenger?
In Process Explorer, sure enough, spread.exe was running.  Then something which was concerning: it appeared to be writing to what I believe is Firefox’s certificate database, accessing cert8.db and key3.db:
We also spotted a third process was spotted, svchosts.exe, which was stored in another folder: %LOCALAPPDATA%\Microsoft.  It may not be related but seems to be similar in construction.  This process was notable for a handle leak; at the time we spotted it, it had over 4700 handles open, mostly thread and duplicate file handles.  Now it was time to kill the processes.  I was expecting, given 3 processes, some complex watchdogging to prevent the processes being killed by the end user, but no, we were able to kill one after the other without any interference.
Now that there seemed to be no suspicious processes running, we took a look at the registry.  It fairly naively installed itself into HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run.

The HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\RunOnce key also contained a simple reference to winlogon.exe:
And curiously enough, HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows key had a reference to winlogon.exe as well:

At this stage, we zipped up the various virus files and deleted them, restarted, and so far, no residue has been detected.  It appears that the virus did not manage to elevate itself and attain administrator access, which was reassuring.
Finally we ran the following SysInternals command to give us a rough idea of what winlogon.exe references:
strings -n 16 winlogon.exe
This command picked up function import references to sqllite3 and cryptography functions, which strengthened my fears about Firefox’s certificate store.  Very few interesting strings were spotted, which suggests that they may have been encrypted.  We opted to uninstall and reinstall Firefox, creating a new profile and all, but I remain concerned about whether or not Internet Explorer’s certificate store has been compromised.
We have not diagnosed how this virus spreads, but have submitted it to F-Prot for evaluation.  I’ll be interested to see the outcome.

VMWare to Hyper-V: it’s never as easy as promised

Today I took on the task of converting one of our last VMWare Server-based VMs to Hyper-V.  This was an XP VM that is used for accounting, and with lots of software and configuration the idea of reinstalling Windows did not appeal.

So I used vmdk2vhd to convert the disk to Hyper-V format, making sure that I followed the initial preparation steps as noted in various blogs.

No joy, ended up with the dreaded “A disk read error occurred. Press ctrl+alt+del to restart” error.  with some research found sites that suggested using FIXBOOT, FIXMBR and CHKDSK in the XP Recovery Console, but I didn’t expect much success with this route and I didn’t get any.

As a side note, I could not remember the local Administrator password for the box, and ended up using Petter Nordahl-Hagen’s excellent offline password editor.

Finally, found a blog that explains how to resolve the problem using TestDisk by CGSecurity.  Ran that, went through the various steps, and finally ended up with a VM that would at least boot to Safe Mode.  After a few minutes of churn, Windows installed various drivers and I could boot to normal Windows and complete Hyper-V Integration Services install.

Or could I?  Nope.  Windows wanted to be reactivated before I could login — but I no longer had network drivers.  As I was not willing to spend the time on the phone with a Microsoft operator when at home late at night, I booted back into Safe Mode, and ran the Integration Services install from there.

No!  Another error: “Setup cannot upgrade the HAL in this virtual machine”.  Yet another blog post (and comment thereunder) to the rescue.  I ended up having to copy HALACPI.DLL from %windows%\servicepackfiles\i386 into %system%.  Rinse, reboot.  And now follow the primary steps in that same blog post – deleting the “Standard PC” entry, and now finally it’s installing.

Wait, no, it failed.  “An Error has occurred: One of the update processes returned error code 1601.”  Google again.  Wow, a Technet discussion thread returns a possible solution.  So booting into Recovery Console again to delete some VMWare drivers (VMWare Tools would not uninstall because it had a Windows Installer reference to a network drive on a machine that had been decomissioned…  Yuck.)

No way, Jose.  A bit more research and eventlog spelunking and I realised that Integration Services setup was trying to start Windows Installer — which doesn’t run in Safe Mode.  Yay!  Now what: can’t install network drivers in Safe Mode, can’t login in normal mode because I have no network drivers for boot.  Ah!  Turns out adding a legacy network adapter may give me a way forward here.  Gosh, that actually worked.  Reactivated Windows successfully.  Now I am booted in normal Windows mode, should be able to install Integration Services, right?

Well, I could now use Remote Desktop to connect to the machine — great, now I had a mouse again – and am now installing Integration Services.  We’ll see how it goes — but I’m not waiting around for it any longer tonight.

Was it worth all the pain?  Probably not.  At least I collected a few useful Hyper-V resources.  Perhaps a rebuild would have been smarter but I really like to understand the hows and whys of these problems.  And perhaps this may help someone else.

The case of the hidden exception

I was building a new installer for an application today using WiX v3, and ran into a problem when trying to extract the COM SelfReg from a DLL using heat.

The error that heat returned was not very helpful.

heat.exe : warning HEAT5150 : Could not harvest data from a file that was expected to be a SelfReg DLL: PrinterBvr.dll. If this file does not support SelfReg you can ignore this warning. Otherwise, this error detail may be helpful to diagnose the failure: Exception has been thrown by the target of an invocation..

No tracing options were available in heat to figure out what actually caused the exception or even what the exception was! Time to pull out one of my favourite debugging tools: Procmon. Procmon traces activities on your system and records registry, file, network and similar activities to a log file.

So, I ran Procmon and added a filter to restrict the log to processes named “heat.exe”:

Then I started the trace and re-ran my command. It failed (as expected) with the same error. I came back to Procmon, stopped the trace, and scanned through looking for a failure point. Near the bottom of the trace, I found the issue. Part of the work heat does is to redirect registry actions to a special location in the registry so it can capture the work the DLL’s DllRegisterServer call does. Unfortunately, printerbvr.dll was depending on a specific key in HKLM already existing, which of course it did not in the redirected registry keys.

At this point I was able to understand the cause of the problem. But how could I work around this to get heat to do its work? I only needed to do this once to capture the registry settings. I could have used Procmon to watch all the activity and manually built the .wxs source file from that. But I was loath to do so much work.

Looking at the trace a bit closer I noted that the key that heat used as a temporary root for its registry redirection was based on its process ID. So I couldn’t create the keys beforehand because I wouldn’t know what the process ID was until after heat had already started — and the process only took a fraction of a second to run.

Windbg to the rescue! I fired up windbg, and started a new debug session with heat.exe and the appropriate command line parameters for heat. I didn’t want to create the registry key based on the PID before heat did that itself because I figured heat might complain if the registry key was already there. So I quickly peeked at the handy stack trace that Procmon had captured for the event in question:

From that I discerned that I could add a breakpoint when PrinterBvr.dll + 0x18BB (subtracting 6 bytes for the size of the ‘call’ opcode), and continued execution. I could probably have added the breakpoint at printerbvr.dll!DllRegisterServer, but this worked for me:

bu printerbvr.dll+0x18bb
g

At the right time, we hit the breakpoint and I then jumped into the registry (after quickly looking up the PID for this instance of heat.exe) and created my new registry key:

This time, everything completed happily, I had my .wxs output file, and I was able to get on with some real work (i.e. writing this blog post).

Spy++ in the case of the missing message

Yesterday, I wrote about WM_QUERYENDSESSION and the app that would not shut down. In that blog, I found a thread that never checked its message queue, and this meant that Windows thought the app was not responding.

So why didn’t Spy++ show the sent message in its log when it was sent? Spy++ uses a SetWindowsHookEx(WH_CALLWNDPROC) hook to log messages sent to windows. It turns out that in Windows NT 4.0 and later versions, the WH_CALLWNDPROC hook is called just before the message is sent to the window procedure. So our message never made it to the hook.

However, in Win9x and NT 3.51, the hook was called when the SendMessage function was called. Spy++ has its heritage in that era. Perhaps now the message delivery title in Spy++ should be changed from “sent” to “received” in order to clarify this difference.

A debug tool wish list item: MsgMon

After running through a painful message debugging session today, I realised again that at the top of my debug tool wishlist tool is MsgMon, a “Procmon for messages”. Spy++ and Winspector Spy are just not up to the task. Once you start using Procmon, with its caching of call stacks, filtering, phenomenal performance and more, life seems very hard each time you start Spy++.

I would love to write this tool, if only I had the time… It would be nice to be able to use the existing Procmon framework as it is so solid…

PHP security updates are like malaria treatments

Applying PHP security updates is somewhat like taking a malaria treatment: they are, temporarily, worse than the disease itself. Let me explain.

Malaria is not a nice disease. I have had malaria a couple of times. We treated it with chloroquine (this was a few years ago). The treatment dose of chloroquine makes you feel worse than the malaria itself. But then you get better.

A PHP security hole is obviously a big issue for your average PHP site. I have had to apply patches to address these holes numerous times in the last few years. Unfortunately, it seems that each patch version for PHP introduces either new bugs or changes the published API. This causes all sorts of chaos and panic when the upgrade goes through, and lots of scrambling to fix a site that no longer works correctly. Sometimes you may not find the problem for several weeks in an infrequently used area of the site, so running a test server does not address this (besides, who wants to leave a known security hole online for several weeks?)

For example, PHP 5.2.7 was released to address a number of bugs and security holes but then was removed from distribution 3 days later because of an introduced bug changing the behaviour of magic quotes. That didn’t affect me because I did not use magic quotes… (Magic quotes were a majorly broken silly idea in the first place, but even worse is making it a configurable option so any code that I write has to test the setting… But let’s not get distracted.)

Or, to take an even more serious example, strtotime function return values changed in 5.1.0. As of 5.1.0, when strtotime is passed an invalid date, it returns FALSE instead of -1. This change was made without notice, and as far as I can tell, without any reference whatsoever in the huge changelog or even in bugs referenced in the changelog. That would have been better in the first place but this type of breaking change should never be made otherwise. I shouldn’t have to review all the changes to the PHP documentation, and then audit all 150,000+ lines of PHP code each time we update PHP!

That’s just two of the more obvious examples of the horrible PHP upgrade situation. Every time I have to upgrade, I just hold my breath and hope that no one has made any more silly breaking changes.