Introducing Mesmeride

Computing, Cycling, Development, mashups, Strava

So I recently had some holidays. Weird, I know. I took two whole weeks off and only had to go into the office twice during that time. My first week had unseasonably nice weather, so I spent some time on my bike making the most of it.

In the second week, the weather soured, so I took the opportunity to learn something of Ruby on Rails with the great Rails tutorial. I am not generally a big fan of tutorials but this particular one covered a lot of bases, and was well organised. Equally excellent were Railscasts.

After working through the first few chapters of the tutorial, I was comfortable enough to start on my own project to test my newly acquired knowledge.

Enter Mesmeride. With this project, I had two objectives:

  • Get a functional and “useful” Ruby on Rails site live in a week.
  • Get my Strava gradient rendering code running again with the new v3 Strava API.

mesmeride-1

Mesmeride allows you to take any Strava activity or segment, and graph it out in a number of different styles. You can add waypoints and control the length, height and size of the presentation, making it suitable for print or web. After tweaking the style of the graph to perfection, you can share the result on Twitter or Facebook, embed the image on your blog, or save it for printing or offline sharing.

Waypoints

Any ride of a reasonable length will have points of interest. The Giro renderer will draw these onto the profile. You can add and delete waypoints, move them along the ride, and change their names in the left hand box in the controls section.

Mountains or Molehills?

The most popular or remarked-upon feature is the ability to make any of your rides, even the most flat and featureless, look like a day attacking the biggest climbs of the Alps. You can control the mountainosity of your ride with the Netherlands-Switzerlands slider (also called the Molehills-Mountain slider).

Size and Length

To help you adjust the dimensions of the graphic, for print or for web, you can rescale the entire ride graphic with the “Teensy – Ginormous slider”, or make the ride appear longer or shorter with the “Shopping Trip – Grand Tour” slider.

Sharing

What good is a graphic without eyes to look at it? Mesmeride has tools to share any of the graphics you create on Twitter, Facebook or even by embedding them in your blog. Or of course you can save the image and download it. The images are stored on Amazon S3, and you can save up to 3 for any given route.

Sharing your ride
Sharing your ride

I even drew the logo myself. Can you tell?

mesmeride-2

Mesmeride will save the design you create as well, and you can come back later and change it round into many other styles.

mesmeride-3

In the future I may add mapping, additional gradient styles, and more controls and waypoint types to existing styles.

Here are a few examples from my race last weekend, via Strava. No, I didn’t do well, but never mind 😉 The screenshots above show the editor in action; what you see below are the resulting files.  I even fixed a bug in Mesmeride when preparing this…

Hell of the South
Hell of the South, full route profile, with the Mesmeride “Giro” Renderer. The waypoints are fully customisable!
Hell of the South Climb 1
The Gardiner’s Bay Climb at the start of Hell of the South. Presented with the Mesmeride “Le Tour” segment renderer
Hell of the South Kettering Climb
The climb out of Kettering, presented in the “Le Tour” rendering style. This is the climb I came unstuck on…
Nicholl's Rivulet Climb
The Nicholl’s Rivulet Climb, a lovely, smooth winding climb which I suffered greatly on. Off the back… 🙂

To finish with, the whole ride again, in another style.

HotS "Hobart 10,000 Banner" Style
HotS “Hobart 10,000 Banner” Style

Almost Caving, and Tourist Caving

Family, Family|Bible|More, Uncategorized

The forecast was bleak — 10°C and rain showers.  So we abandoned our original plans and decided to go bush instead.  Last night, Hannah prepared an itinerary, including Mystery Creek Cave, Thermal Springs, Hastings Cave and Surprise Destination (at my suggestion).  This morning, we executed that carefully shaped itinerary, nearly perfectly.  Apart from wet clothes all day.  That was a bit of a dampener.

On the way down, we stopped in Geeveston to buy some food.  And happened to park right outside a huge lolly shop.  What a huge lolly shop is doing in a tiny little place like Geeveston, I don’t know.  Philosophical considerations were not top of my daughters’ minds.  They happily accepted the existence of this incongruous edifice, and came away with — I’m sorry, Joey! — lollies.

Bellies charged, we made our way to our first objective: Almost Caving.  I knew we were regrettably not adequately prepared for Real Caving, so instead we were aiming for Almost Caving.  That is, looking into the cave, shining torches around the entrance and wondering what was around the next corner.

Almost was still an adventure.  The track was muddy.  Very muddy.  Wind had brought many trees down.  Climbing over wet branches and landing in big mud puddles was a great introduction to the day.  Instantly soggy pants.  The rain kept coming down, soaking from above, while the puddles soaked from below.

Cleaning boots at the start of the track
Cleaning boots at the start of the track
Trees down
Trees down
Trees down
Trees down

We found a midden of abandoned boots and bottles.  Presumably collected and deposited there over time.  Creepy, according to the girls.

Midden
Midden

Then we arrived at an obstacle.

Initially, I did not think we would be able to cross: the river was flowing fast and I was not keen on getting even more soaked than we already were.  But then I found some exposed rocks and a fallen tree.  So crossed we did!  Apologies for the quality of the pictures and video.  They were taken on SoggyiPhoneCam.

First Raging Torrent Crossing
First Raging Torrent Crossing

Shortly thereafter, we arrived triumphantly at a quarry.  The rain decided to come down just a bit more heavily.  That didn’t stop the girls from checking out what lay beyond the rock piles.

Success! Crossed the raging torrent!
Success! Crossed the raging torrent!
Quarry Exploration
Quarry Exploration

But on to the cave.  A couple minutes more trekking through mud and puddles, and we spotted it.  The river was not (yet) high.  So we were able to approach the entrance of the cave, and look into what lay beyond.  It was also dry, which was a nice change.

Almost Caving
Almost Caving
Looking at the cliff face above the cave
Looking at the cliff face above the cave
Looking into the cave
Looking into the cave
Looking into the cave
Looking into the cave
The mysterious beyond
The mysterious beyond
And out of the cave entrance
And out of the cave entrance

Lolly snakes all round, and back to the car.  With the crossing of the raging torrent, now a little higher.  But not more than any adventurous (nearly) 7 or 10 year old could handle.

Back to the car, super soggy
Back to the car, super soggy

Back at the dry and cosy car, there was unanimous agreement that we would head to the Thermal Springs for some additional warmth and recovery.  With car heater on full bore, your merry correspondent and his soggy herd proceeded thence forthwith.  And a fun time was had by all, steaming in front of open log fires, swimming, in the rain, in the thermal pools (much warmer than the rain), but cut short all too soon by the need for sustenance, and tourist cave booking.

Perhaps Newdegate Cave was just too tame, after the morning expedition.  Or maybe we were too cold and tired.  But the review of the cave, as expressed by Hannah, and not contested by the rest of us, was “it wasn’t as good as I expected”.  Still, we “did” the tour, saw a solitary glowworm, many stalactites and -gmites, shawls and curtains and straws, holes and flowstone and more, and, then more than a little cold, made it back to the car.

Amazingly, just enough time for the final Surprise Destination.  This Surprise Destination was augmented by a multi-dollar budget for each of the girls.  A Gemstone Shop!  The initial joy rapidly gave way to serious perturbation.  This was difficult.  So many pretty rocks.  So much choice.  Yet, finally, both girls came away satisfied, thanks to a friendly shopowner, who was willing to mix and match presorted bags to her appreciative and choosy customers’ satisfaction.  Amethysts, moonstone, mookaite and more.  And, from there we made our tired, soggy, smelly way home, stopping only once to buy drinks.

I expected the girls to collapse on arrival home, but they bounced in the door, greeted baby Peter, and exuberantly leapt into the bath.  And while I was off preparing Peter for bed, they set the table for a fancy meal, including candles, wine glasses (no wine, more’s the pity) and neatly arranged place settings.  Maybe lollies are good for them, after all?

The case of the stack trace that wouldn’t (trace)

Computing, Development, Tools, Windows

One of my favourite Windows tools is Procmon.  I pull it out regularly, often as a first port of call when diagnosing complicated and opaque problems in the software I develop.  Or in anyone’s software, really.

Procmon captures a trace of key I/O activities on your computer, including file, registry and network activity, and makes it really easy to spot operations that have failed or that may be causing problems.  It’s great for spotting authentication problems, sharing violations, missing files and more (… malware).  Procmon goes as far as recording a stack trace for nearly every operation it captures!

Today, we were trying to diagnose a problem with a process that was taking 15 seconds or longer to start on a Windows XP computer.  The normal start time for that process should have been 1-2 seconds.  None of the usual culprits came forward and admitted fault, so it was time to pull out Procmon again.

We quickly spotted a big fat delay in the trace.  Note the time stamps in the two selected rows  in the screen capture below.

A big time gap between 4:10:04 and 4:10:17

Now it was time to try and find out what was causing this.  So we examined the stack trace for each entry, except … there were no symbols.  Easy enough to fix — copy dbghelp.dll from a version of Microsoft’s Debugging Tools for Windows onto the system temporarily, fixup the symbol path in Procmon’s options, and … nope, still no symbols.  Now this is one area where Procmon falls down a little bit.  If symbol loading fails, it just silently fails.  No warnings, errors or hints as to what might be going on.

This issue was occurring on a client’s computer, so it was time to take the investigation elsewhere for examination.  Before we could really examine the captured trace, we needed to get symbols going.  But how?

Procmon to the rescue!

That’s right, we realised we can use Procmon to diagnose itself!  I booted up a clean new Windows XP virtual machine, loaded Procmon onto it, ran a basic capture of some random events.

A trace on XP
A trace on XP
No symbols showing, only exports
No symbols showing, only exports
Configuring symbols for Procmon
Configuring symbols for Procmon

Even after configuring symbols, they still silently failed to load.  So I stopped the capture, saved it and immediately opened the saved capture, to stop this instance of Procmon from capturing events on the local computer.  I then started a second instance of Procmon, removed the Procmon exclusion from the filtering, and instead, added a filter to include Procmon (I also filtered specifically for the PID of the original Procmon, later):

Configuring Procmon to watch itself
Configuring Procmon to watch itself

Then I started the trace, switched back to the first Procmon, and tried to examine the stack.  Of course, still no symbols, but now it was time to switch back to the second, active Procmon process and see what we found.

And what we found was that dbghelp.dll was looking for symsrv.dll in order to download its symbols.  So we copied that also into the folder with procmon.exe  and suddenly everything worked!

dbghelp wants symsrv to help it as well
dbghelp wants symsrv to help it as well
The undecorated stack for the symsrv load request
The undecorated stack for the symsrv load request

Update 19 Sep 2013: Oops.  Forgot to attach the decorated stack (sorry!):

The call stack with symbols
The call stack with symbols loaded.  Note how the function names differ from the original stack.

So that’s the first takeaway from this story: when you want symbols, copy both dbghelp.dll and symsrv.dll from your copy of Debugging Tools for Windows.  We found no other dependencies, even with the latest version of these files.

A diversion

One curious anomaly we spotted: Procmon (or possibly Dbghelp) is looking in some strange places for debug symbols, including appending a SRV*path*url style symbol path to procmon’s parent path, and looking there, without much success:

Some weirdness in symbol loading
Some weirdness in symbol loading

I leave that one for you to solve.

Backtrack to the stack (trace)

Back to the original trace.  We loaded up the saved trace, and found that we now got kernel mode symbols just fine, but no user mode symbols would load.  In fact, Procmon doesn’t even appear to be looking for symbols for these user mode frames — either on the local drive or on the network.  And this time Procmon isn’t able to give us any more detail.  However, when we debugged the call that Procmon made to SymFindFileInPath when viewing a call stack in this log vs another new log, we found that Procmon wasn’t even providing the necessary identifying information.

What information is this?  The identifying information that the symbol servers use is the TimeDateStamp and the SizeOfImage fields from the PE header of the executable file (slightly different for .pdb files).

I surmise that this identifying information is missing from our original trace because this trace was captured before we copied version 6.0 or later of dbghelp.dll onto the client’s computer — meaning that the version that Procmon used when capturing the trace did not record this identifying information.

Therefore, the second takeaway of the story is: always copy a recent version of dbghelp.dll and symsrv.dll into the folder with procmon.exe, before starting a trace.  Even if you intend to analyze the trace later, you’ll find that without these, you won’t get full stack traces.

(Dear Microsoft, please can you consider including these in the Procmon and Procexp downloads, given that you now own Sysinternals?  Saves a lot of hassle!)

Giro-Your-Strava now does Strava Routes as well

Computing, Cycling, mashups, Strava

Update May 2014: As Strava’s ride pages have changed format significantly, Giro-Your-Strava no longer works. The good news is that Mesmeride does rides — but unfortunately not routes as yet.

After a comment from Mike on my previous Giro-Your-Strava post asking if the bookmarklet could support Strava’s new Routes feature, I took a few minutes over breakfast to spelunk and found it wouldn’t be too hard.

Firebug’s DOM explorer actually made the task much, much simpler. My approach was simply to look for Great Big Arrays Of Numbers. I soon found, amongst all the Google, jQuery, Modernizr, Optimizely and other objects, a pageView object, which contained a number of juicy functions, including pageView.chartContext().dataContext().elevationStream() and pageView.pageContext().routeSegments(), which gave me all the information I needed. The data structures for routes are somewhat different to those for rides, so I opted to massage the Great Big Arrays those functions returned into the same basic structure as the ride data already used by the bookmarklet, rather than touch the rendering code at all.

So… after that overly detailed introduction, here ’tis. I’ve updated the bookmarklet to draw Giro-style elevation profiles for Strava Routes as well as for Rides. And of course, Le Tour-style elevation profiles for segments still work, within individual ride pages.

Note: this still won’t work in IE10, as Github returns the wrong Content-Type for the Javascript and IE gets a little panicky about it, and, well, just use a different browser.

1. Install the bookmarklet.

Here’s the bookmarklet. Just drag it to your Bookmarks toolbar to install it:

Giro-Your-Strava

2. Load a favourite Strava Route and click the bookmark.

(It’s best to wait for the page to load completely before clicking the bookmark.)

Here’s one of my favourite Strava Routes — Hobart 10,000 Day 1:

H10K 2013 Day 1 - Strava Route 2013-09-05 08-26-54

After a few seconds the gradient graphic may refresh with the correct font — this takes a second or two to download.

That’s it! This new bookmarklet still works with rides (so delete the old one!)

The source is still all on GitHub.  Again, if you improve the code, or accidentally hurt yourself while reading it, please do share with a comment on this post.

And as DC Rainmaker says, thanks for reading!

Our collective wrongs

Bible, Family|Bible|More, Pontificating

It has been said that every generation and every culture harbour a collective wrong. When we read through history, it is easy to find cultural attitudes and actions which violate the rights of a group of people. These attitudes are often couched in pseudo-scientific terms, and dehumanising phrases are commonly used by the proponents of these attitudes. Societies’ leaders would use moral imperatives in support of their arguments. In some cases, these wrongs were even perpetuated by well-meaning people with a sadly misguided world view.

Those who spoke out against what they saw as grievous wrongs were marginalised, or even hunted down.

First they came for the communists,
and I didn’t speak out because I wasn’t a communist.

Then they came for the socialists,
and I didn’t speak out because I wasn’t a socialist.

Then they came for the trade unionists,
and I didn’t speak out because I wasn’t a trade unionist.

Then they came for me,
and there was no one left to speak for me.

РMartin Niem̦ller

These injustices have ranged from the tragic (stolen children) through to the terrible (slavery), right through to the monstrous (Nazi genocides).

It is easy to look back at our ancestors and condemn their actions. But are we as a society, today, ready to examine our own motives, understandings, and prejudices?

What are our collective wrongs? What injustices do we not see, or see, and not speak out against?

I am happy to announce that the missing characters have been found!

Bugs, Computing, Delphi, Development, Windows

Last year I published a blog post about characters missing from print jobs with Internet Explorer 9 and my adventures in tracing and reporting the issue to Microsoft.

We saw situations where a letter would be printed with random letters missing, as per the following example:

Here’s how it should have been printed (oh yes, that’s a completely fictional name we used for testing non-English Latin script letters in printing):

Today, I was notified that Microsoft have finally publicly published a hotfix!  We received the hotfix from them a few weeks ago, before it was publicly available, and it certainly solved the problem on our test machines and end user computers.

Download the hotfix from:

http://support.microsoft.com/kb/2853777 (Windows 7 SP1, Windows Server 2008 SP1) 

http://support.microsoft.com/kb/2855336 (Windows 8, Windows Server 2012)

(The first link notes that the hotfix is included in the rollup in the second link, though the second link doesn’t mention the hotfix!)

Sadly, they’ve only published a solution for Windows 7 Service Pack 1 onwards, and not for Vista, which could be an issue for some users. 

It was interesting to see that the issue was indeed a race condition (two threads ending up with the same random seed, because, and I quote:

This issue occurs because a conflict causes the text that uses the font to become corrupted when the two threads try to install the font at the same time. The name of the font is generated by the RAND function together with a SEEDvalue whose time value is set to srand(time(NULL)). When the two documents are printed at the same time, the SEEDvalue for the font is the same in both documents. Therefore, the conflict occurs.

So not related to the threading model, which was a little bit of misdirection on my part 😉  That’s par for the course though when debugging complex issues without source…

Fantastic to finally have the fix 🙂

Giro-Your-Strava updated to give Le Tour treatment to the climbs!

Computing, Cycling, Development, JavaScript, mashups, Strava

Update May 2014: As Strava’s ride pages have changed format significantly, Giro-Your-Strava no longer works. The good news is that Mesmeride does the same and more!

After the Strava API debacle, my little Tour Segment Gradient tool no longer works, which is sad.  I’d put together a number of other Strava API-based widgets, but this was the only one which was really at all popular.  Yesterday, DC Rainmaker himself mentioned (thank you Ray!) the Giro-Your-Strava elevation graph tool (which does still work) on his blog, so what better time to update the Tour Segment Gradient tool?

In short, what I have done is to dump both the Giro and Le Tour gradient mashups into the same bookmarklet.  One click and you get beautiful isometric graphs for your ride (in Giro style) and your efforts (in Le Tour style).  Yes, I get the inconsistency, but what would life be without idiosyncracies?

1. Install the bookmarklet.

Here’s the bookmarklet.  Just drag it to your Bookmarks toolbar to install it:

Giro-Your-Strava

2. Load a favourite Strava ride and click the bookmark.

(It’s best to wait for the page to load completely before clicking the bookmark.)

Presto, you’ll get two spiffy new buttons, one for your ride:

And one for the segment view:

So go ahead and click the Giro button, and you’ll see:

Click the Le Tour button for the new elevation profile for a segment:

Have fun!

One danger with bookmarklets that fiddle with an existing site in this way is that they will tend to break when the site updates.  There are no stability guarantees that APIs (ususally!) provide, so YMMV.  However, if the Strava site layout changes, it’s probably only a simple tweak to the code to get it working again.

There’s nothing beautiful about the code on the backend.  It really needs rewriting and modularisation etc etc etc but hey, it works 😉  Do what you want with the code, just share it with us all if you improve things!

Strava have decided to abandon their developer community

Cycling, Griping, mashups, Strava

I received an email from Strava today which was very disappointing.

Hello Marc,

As of July 1, 2013, V1/V2 API endpoints have been retired. Libraries, sites, and applications using these endpoints will cease to work.

In previous blog updates, we’ve discussed status and access to V3 of our API.  As mentioned then, we had to make difficult decisions this year about where to invest time and resources – feature development or a full-fledged API program.  We have chosen to focus on feature development at this time and so access to V3 of our API is extremely limited.

Any developer who has been granted access to V3 of the API has been contacted. We will revisit our API program and applications from time to time, but for the time being, we have no plans to grant further access in 2013.

If you have questions or comments, please send an email to [email protected]. Given our limited resources, you should not expect an immediate response.

Thanks for your understanding,

Your Friends at Strava

The highlights (or lowlights) from this email:

  • Strava have no plans to allow anyone else to access their new API for at least six months
  • It seems Strava aren’t interested in dialogue with their community about this decision.

I’ve been using the Strava V1 and V2 APIs now for several years, and have written quite a few blog posts about how to use it.  The V1 and V2 APIs were always fairly experimental, but that was okay.  We all understood that and it was part of the fun of working with Strava.

Discontinuing the V1 and V2 APIs was in the pipeline, and again we Strava API users could understand limiting the V3 API beta program to a small number of developers.  However, the abrupt announcement today that not only are the V1 and V2 APIs no longer available, but now Strava won’t be making the V3 API available to anyone else for an Internet Eternity (that is, at least 6 months) either is completely unexpected.

The only Windows Phone app that integrates with Strava no longer works.  My apps no longer work.  And there appears to be no future for any of these apps.

Now, I’ve been a Strava Ambassador since before they started their official Ambassador program, and have had a lot of fun promoting Strava.  I’ve developed popular tools that integrate with the Strava API.  It is a very big disappointment to me that they’ve decided to throw away all that good-will with their developer community.  This is not the way to treat the most dedicated and enthusiastic portion of your user base, Strava!

Giro-style elevation graphs for Strava

Computing, Cycling, Development, Hobart's Top Ten Climbs, JavaScript, mashups, Strava

Update May 2014: As Strava’s ride pages have changed format significantly, Giro-Your-Strava no longer works. The good news is that Mesmeride does the same and more!
Update 13 July 2013: An updated version of Giro-Your-Strava is now available here.

Just in time for the last few days of the Giro, I’ve finished a little after-hours Strava mashup project that builds on the segment graphs that I originally created for the Hobart 10,000.

I’m sure you’ve seen some of the Giro elevation graphs. Here’s one, from Stage 11:

Now, here’s a bookmarklet.  Drag it to your bookmarks toolbar, load up a favourite hilly ride on Strava, and click the bookmarklet.

Giro-Your-Strava

A mysterious new button will appear in the graph menu.  Go, on click it!

And up pops an elevation graph that makes it look like you’ve been riding the Giro!

The algorithm picks the categorised climbs from your ride (and tries to figure out the most appropriate segment where multiple segments finish at the top of a hill). Non-categorised segments are currently ignored.  The whole project is published on GitHub, so you can tweak it and improve it to your heart’s content.  Your first step should be to tidy up the mess I’ve left you 🙂

RSA Key Exchange between CryptoAPI and CNG (BCrypt)

Computing, Development, Windows

Microsoft, a few years ago, wrote a new cryptography API for Windows Vista called Cryptography Next Generation (CNG).  This replaces the existing cryptography API in Windows XP and earlier versions which was simply called CryptoAPI.

Now, compatibility between the two APIs is touted as a feature of the new CNG API, but a somewhat more sombre reality tends to set in pretty quickly once you start working with them.  A motley collection of loosely documented data structures, endianness differences, and “Not Supported” footnotes pile up as quickly as you can open MSDN documentation tabs in your browser.  In my research, I also found a dearth of concrete examples of how to interoperate between the two APIs online.  In fact, there was only one page which really helped, buried in Microsoft’s Output Protection Manager documentation.

There is something magical about all your bits lining up in a row when mixing crypto libraries of any ilk, and I wanted to share!  Hence this story.

My requirement was pretty simple: I wanted to generate an RSA key pair using CryptoAPI in my client application, hand off the public key to another application, which happens to use CNG, wave my hands a little, and magically transact a key exchange.  Here’s how to do it (with some error handling elided for your own reading sanity).  You should be able to load this into any recent version of Visual Studio, add bcrypt.lib to your library includes and build.  The code should be reasonably self-explanatory, so I’ll leave it there.

The usual disclaimers apply. Source on GitHub.