Monthly Archives: September 2013

The case of the stack trace that wouldn’t (trace)

Computing, Development, Tools, Windows

One of my favourite Windows tools is Procmon.  I pull it out regularly, often as a first port of call when diagnosing complicated and opaque problems in the software I develop.  Or in anyone’s software, really.

Procmon captures a trace of key I/O activities on your computer, including file, registry and network activity, and makes it really easy to spot operations that have failed or that may be causing problems.  It’s great for spotting authentication problems, sharing violations, missing files and more (… malware).  Procmon goes as far as recording a stack trace for nearly every operation it captures!

Today, we were trying to diagnose a problem with a process that was taking 15 seconds or longer to start on a Windows XP computer.  The normal start time for that process should have been 1-2 seconds.  None of the usual culprits came forward and admitted fault, so it was time to pull out Procmon again.

We quickly spotted a big fat delay in the trace.  Note the time stamps in the two selected rows  in the screen capture below.

A big time gap between 4:10:04 and 4:10:17

Now it was time to try and find out what was causing this.  So we examined the stack trace for each entry, except … there were no symbols.  Easy enough to fix — copy dbghelp.dll from a version of Microsoft’s Debugging Tools for Windows onto the system temporarily, fixup the symbol path in Procmon’s options, and … nope, still no symbols.  Now this is one area where Procmon falls down a little bit.  If symbol loading fails, it just silently fails.  No warnings, errors or hints as to what might be going on.

This issue was occurring on a client’s computer, so it was time to take the investigation elsewhere for examination.  Before we could really examine the captured trace, we needed to get symbols going.  But how?

Procmon to the rescue!

That’s right, we realised we can use Procmon to diagnose itself!  I booted up a clean new Windows XP virtual machine, loaded Procmon onto it, ran a basic capture of some random events.

A trace on XP
A trace on XP
No symbols showing, only exports
No symbols showing, only exports
Configuring symbols for Procmon
Configuring symbols for Procmon

Even after configuring symbols, they still silently failed to load.  So I stopped the capture, saved it and immediately opened the saved capture, to stop this instance of Procmon from capturing events on the local computer.  I then started a second instance of Procmon, removed the Procmon exclusion from the filtering, and instead, added a filter to include Procmon (I also filtered specifically for the PID of the original Procmon, later):

Configuring Procmon to watch itself
Configuring Procmon to watch itself

Then I started the trace, switched back to the first Procmon, and tried to examine the stack.  Of course, still no symbols, but now it was time to switch back to the second, active Procmon process and see what we found.

And what we found was that dbghelp.dll was looking for symsrv.dll in order to download its symbols.  So we copied that also into the folder with procmon.exe  and suddenly everything worked!

dbghelp wants symsrv to help it as well
dbghelp wants symsrv to help it as well
The undecorated stack for the symsrv load request
The undecorated stack for the symsrv load request

Update 19 Sep 2013: Oops.  Forgot to attach the decorated stack (sorry!):

The call stack with symbols
The call stack with symbols loaded.  Note how the function names differ from the original stack.

So that’s the first takeaway from this story: when you want symbols, copy both dbghelp.dll and symsrv.dll from your copy of Debugging Tools for Windows.  We found no other dependencies, even with the latest version of these files.

A diversion

One curious anomaly we spotted: Procmon (or possibly Dbghelp) is looking in some strange places for debug symbols, including appending a SRV*path*url style symbol path to procmon’s parent path, and looking there, without much success:

Some weirdness in symbol loading
Some weirdness in symbol loading

I leave that one for you to solve.

Backtrack to the stack (trace)

Back to the original trace.  We loaded up the saved trace, and found that we now got kernel mode symbols just fine, but no user mode symbols would load.  In fact, Procmon doesn’t even appear to be looking for symbols for these user mode frames — either on the local drive or on the network.  And this time Procmon isn’t able to give us any more detail.  However, when we debugged the call that Procmon made to SymFindFileInPath when viewing a call stack in this log vs another new log, we found that Procmon wasn’t even providing the necessary identifying information.

What information is this?  The identifying information that the symbol servers use is the TimeDateStamp and the SizeOfImage fields from the PE header of the executable file (slightly different for .pdb files).

I surmise that this identifying information is missing from our original trace because this trace was captured before we copied version 6.0 or later of dbghelp.dll onto the client’s computer — meaning that the version that Procmon used when capturing the trace did not record this identifying information.

Therefore, the second takeaway of the story is: always copy a recent version of dbghelp.dll and symsrv.dll into the folder with procmon.exe, before starting a trace.  Even if you intend to analyze the trace later, you’ll find that without these, you won’t get full stack traces.

(Dear Microsoft, please can you consider including these in the Procmon and Procexp downloads, given that you now own Sysinternals?  Saves a lot of hassle!)

Giro-Your-Strava now does Strava Routes as well

Computing, Cycling, mashups, Strava

Update May 2014: As Strava’s ride pages have changed format significantly, Giro-Your-Strava no longer works. The good news is that Mesmeride does rides — but unfortunately not routes as yet.

After a comment from Mike on my previous Giro-Your-Strava post asking if the bookmarklet could support Strava’s new Routes feature, I took a few minutes over breakfast to spelunk and found it wouldn’t be too hard.

Firebug’s DOM explorer actually made the task much, much simpler. My approach was simply to look for Great Big Arrays Of Numbers. I soon found, amongst all the Google, jQuery, Modernizr, Optimizely and other objects, a pageView object, which contained a number of juicy functions, including pageView.chartContext().dataContext().elevationStream() and pageView.pageContext().routeSegments(), which gave me all the information I needed. The data structures for routes are somewhat different to those for rides, so I opted to massage the Great Big Arrays those functions returned into the same basic structure as the ride data already used by the bookmarklet, rather than touch the rendering code at all.

So… after that overly detailed introduction, here ’tis. I’ve updated the bookmarklet to draw Giro-style elevation profiles for Strava Routes as well as for Rides. And of course, Le Tour-style elevation profiles for segments still work, within individual ride pages.

Note: this still won’t work in IE10, as Github returns the wrong Content-Type for the Javascript and IE gets a little panicky about it, and, well, just use a different browser.

1. Install the bookmarklet.

Here’s the bookmarklet. Just drag it to your Bookmarks toolbar to install it:

Giro-Your-Strava

2. Load a favourite Strava Route and click the bookmark.

(It’s best to wait for the page to load completely before clicking the bookmark.)

Here’s one of my favourite Strava Routes — Hobart 10,000 Day 1:

H10K 2013 Day 1 - Strava Route 2013-09-05 08-26-54

After a few seconds the gradient graphic may refresh with the correct font — this takes a second or two to download.

That’s it! This new bookmarklet still works with rides (so delete the old one!)

The source is still all on GitHub.  Again, if you improve the code, or accidentally hurt yourself while reading it, please do share with a comment on this post.

And as DC Rainmaker says, thanks for reading!

Our collective wrongs

Bible, Family|Bible|More, Pontificating

It has been said that every generation and every culture harbour a collective wrong. When we read through history, it is easy to find cultural attitudes and actions which violate the rights of a group of people. These attitudes are often couched in pseudo-scientific terms, and dehumanising phrases are commonly used by the proponents of these attitudes. Societies’ leaders would use moral imperatives in support of their arguments. In some cases, these wrongs were even perpetuated by well-meaning people with a sadly misguided world view.

Those who spoke out against what they saw as grievous wrongs were marginalised, or even hunted down.

First they came for the communists,
and I didn’t speak out because I wasn’t a communist.

Then they came for the socialists,
and I didn’t speak out because I wasn’t a socialist.

Then they came for the trade unionists,
and I didn’t speak out because I wasn’t a trade unionist.

Then they came for me,
and there was no one left to speak for me.

– Martin Niemöller

These injustices have ranged from the tragic (stolen children) through to the terrible (slavery), right through to the monstrous (Nazi genocides).

It is easy to look back at our ancestors and condemn their actions. But are we as a society, today, ready to examine our own motives, understandings, and prejudices?

What are our collective wrongs? What injustices do we not see, or see, and not speak out against?