We needed to setup a web host for the domain convert.ភាសាខ្មែរ.com for an upcoming project. It should have been simple, but this turned into quite the journey.
A very brief background on IDNs
Some background. Notice the use of Khmer characters: ភាសាខ្មែរ /pʰiesaa kmae/ or “Khmer Language”. ភាសាខ្មែរ.com is a domain I’ve owned for some time, having purchased it when I was learning Khmer. Because this domain names uses letters not found in the English alphabet, it is considered to be an Internationalized Domain Name (or IDN). Now, behind the scenes, ភាសាខ្មែរ is not stored using Khmer characters in a Domain Name System (DNS) record. For reasons of history, domain names are restricted to a handful of Latin script letters, digits, and hyphen, and so the domain must be re-encoded using a system known as ‘punycode‘ (truly!)
The punycode representation of ភាសាខ្មែរ is the rather bamboozling xn--j2e7beiw1lb2hqg.
Now, most of the time, this gory detail will be hidden from you in the user interface of your browser — but if you copy the URL from the address bar and paste it into a text editor, you’ll be presented with the gory details! (The domain punycoder.com allows you to easily play with other scripts and non-English-alphabet names and see how they are punycoded.)
We needed a backend host with Python for the project, which ruled out a lot of free options — particularly since we wanted to bind to this custom domain name. As I had access to an Azure subscription already, I went that way.
So first, I attempted to create the domain quickly in the Azure Portal web front end. I had no trouble creating https://khconvert.azurewebsites.net/ — took just a few seconds. Then I had to bind our domain convert.ភាសាខ្មែរ.com to this azure hostname. But computer said no:
The name convert.xn--j2e7beiw1lb2hqg.com is not valid.
Fine, perhaps the UI needed the domain name as a Unicode string.
Hmm. Blocked again.
Now, often the Azure Portal does not allow you to do things which can be done via their APIs or with their az CLI tool. So I tried again with az.
PS> az webapp config hostname add --hostname convert.xn--j2e7beiw1lb2hqg.com --webapp-name khconvert --resource-group <REDACTED> --verbose
The name convert.xn--j2e7beiw1lb2hqg.com is not valid.
Command ran in 4.228 seconds (init: 0.493, invoke: 3.736)
Okay then … perhaps I have to use the non-punycode version of the hostname? After all, that error in the Azure Portal was purely in the UI — it didn’t even call into the API.
And it worked!
Side track: the old Powershell console did not do well with rendering Khmer in its default settings…
So it looks like those were not question marks? Interestingly, I can paste Khmer characters into the console, but I can’t copy them back out from the command — those letters copy out as question marks. Although as you can see from a screenshot and corresponding text later, I can copy the response and show it correctly. (Admittedly, I was using the old Powershell console, not Terminal, which probably would have none of these problems.)
But it worked, hurrah! We are done!
Well, not quite. We need a TLS certificate for the site. Fortunately, Azure supports creating certificates automatically for web apps.
But not for IDNs.
And not even the CLI saved me this time.
PS> az webapp config ssl create --hostname 'convert.?????????.com' --name khconvert -g <REDACTED> --verbose
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Operation returned an invalid status 'Bad Request'
Content: {"Code":"BadRequest","Message":"Properties.CanonicalName is invalid. Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed.","Target":null,"Details":[{"Message":"Properties.CanonicalName is invalid. Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed."},{"Code":"BadRequest"},{"ErrorEntity":{"ExtendedCode":"51021","MessageTemplate":"{0} is invalid. {1}","Parameters":["Properties.CanonicalName","Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed."],"Code":"BadRequest","Message":"Properties.CanonicalName is invalid. Canonical name convert.ភាសាខ្មែរ.com includes at least one special character. Only letters and numbers are allowed."}}],"Innererror":null}
At this point, I threw my hands in the air, put the domain behind a Cloudflare proxy and used their SSL certificate, and it just worked.
But it is no wonder there are so few IDNs. The experience is still frightful. The errors are weird. We have a long, long way to go.
(Oh, and do take a look at ភាសាខ្មែរ.com and convert.ភាសាខ្មែរ.com: the problems they are playing with have quite a lot to do with IDNs too!)
This morning, Firefox stalled while loading the login page for the ANZ Internet Banking website. Looking at the status bar, I could see that Firefox was attempting to connect to a website, australianewzealandb.tt.omtrdc.net. This raised immediate alarm bells, because I didn’t recognise the website address, and it certainly wasn’t an official anz.com sub-domain.
ANZ login delay – note the status message at the bottom of the window
The connection eventually started, and the page finished loading — just one of those little glitches loading web pages that everyone encounters all the time, right? But before I entered my ID and password, I decided I wasn’t comfortable to continue without knowing what that website was, and what resources it was providing to the ANZ site.
And here’s where things got a little scary.
It turns out that australianewzealandb.tt.omtrdc.net is a user tracking web site run by marketing firm Omniture, now a part of Adobe. The ANZ Internet Banking login page is requesting a Javascript script from the server, which currently returns the following tiny piece of code:
if (typeof(mboxFactories) !== 'undefined') {mboxFactories.get('default').get('SiteCatalyst: event', 0).setOffer(new mboxOfferDefault()).loaded();}
The Scare Factor
This script is run within the context of the Internet Banking login page. What can scripts that run within that context do? At worst, a script can be used to watch your login and password and send them (pretty silently) to a malicious host. This interaction may even be undetectable by the bank, and it would be up to you and your computer to be aware of this and block it — a big ask!
At worst, a script can be used to watch your login and password and send them to a malicious host.
The Relief
Now, this particular script is fortunately not malicious! In fact, as the mboxFactories variable actually is undefined on this page, this script does nothing at all. In other words, it’s useless and doesn’t even need to be there! (It’s defintely possible that the request for the script is being used on the server side to log client statistics, given the comprehensive parameters that are passed in the HTTPS request for the script.)
What are the risks?
So what’s the big deal with running third party script on a website?
The core issue is that scripts from third party sites can be changed at any time, without the knowledge of the ANZ Internet Banking team. In fact, different scripts can be served for different clients — a smart hacker would serve the original script for IP addresses owned by ANZ Bank, and serve a malicious script only to specific targeted clients. There would be no reliable way for the ANZ Internet Banking security team to detect this.
Scripts from third party sites can be changed at any time, without the knowledge of the Internet Banking team.
Another way of looking at this: it’s standard practice in software development to include code developed by other organisations in applications or websites. This is normal, sensible, and in fact unavoidable. The key here is that any code must be vetted and validated by a security team before deployment. If the bank hosts this code on their own servers, this is a straightforward part of the deployment process. When the bank instead references a third party site, this crucial security step is impossible.
Banking websites are among the most targeted sites online, for obvious reasons. I would expect their security policies to be conservative and robust. My research today surprised me.
Trust
How could third party scripts go wrong?
First, australianewzealandb.tt.omtrdc.net is not controlled by ANZ Bank. It’s controlled by a marketing organisation in another country. We don’t know how much emphasis they place on security. We are required to trust a third party from another country in order to login to our Internet Banking.
This means we need to trust that none of their employees are malicious, that they have strong procedures in place for managing updates to the site, the servers and infastructure, and that their organisation’s aims are coincident with the tight security requirements of Internet Banking. They need to have the same commitment to security that you would expect your bank to have. That’s a big ask for a marketing firm.
Security
The ANZ Internet Banking website is of course encrypted, served via HTTPS, the industry standard method of serving encrypted web pages.
We can tell, just by looking at the address bar, that anz.com uses an Extended Validation certificate.
With a little simple detective work, we can also see that anz.com serves those pages using the TLS_RSA_WITH_AES_256_CBC_SHA encryption suite, using 256-bit keys. This is a good strong level of encryption, today.
However, australianewzealandb.tt.omtrdc.net does not measure up. In fact, this site uses 128-bit RC4+SHA encryption and integrity and does not have an Extended Validation certificate. RC4 is not a good choice today, and neither is SHA. This suggests immediately that security is not their top concern, which should then be an immediate concern to us!
ANZ vs Omtrdc Security
I should qualify this a little: Extended Validation certificates are not available for wildcard domains, which is the type of certificate that tt.omtrdc.net is using. This is for a good reason: “in order to ensure that EV SSL Certificates are not issued fraudulently or misused after issuance.” It’s worth thinking through that reason and seeing how it applies to this context.
Malicious Actors
So how could a nasty person steal your money?
In theory, if a nasty person managed to hack into the Adobe server, they could simply supply a script to the Internet Banking login page that captures your login details and sends them to a server, somewhere, anywhere, on the Internet. This means that we have to trust (there’s that word again) that the marketing firm will be proactive in updating and patching not only their Internet-facing servers, but their infrastructure behind those servers as well.
If a bad actor has compromised a certificate authority, as has happened severaltimesrecently, they can target these third party servers . Together with a DNS cache poisoning or Man-In-The-Middle (MITM) attack, even security-savvy users will be unlikely to notice fraudulent certificates on the script servers.
Security flaws like Heartbleed are exacerbated by this setup. Not only do the bank security team have to patch their own servers, they also have to push the third party vendors to patch theirs as a priority.
Protect Yourself
As a user, run security software. That’s an important first step. Security software is regularly refreshed with blacklists of known malicious sites, and this will hopefully minimize any window of opportunity that an untargeted attack may have. I’m not going to recommend any particular brand, because I pretty much hate them all.
If you want to unleash your inner geek and be aware of how sites are using third party script servers, you can use Developer Tools included in your browser — press F12 in Internet Explorer, Chrome or Firefox, and look for the Network tab to see a list of all resources referenced by the site. You may need to press Ctrl+F5 to trigger a ‘hard’ refresh before the list is fully populated.
I’ve shown below the list of resources, filtered for Javascript, for the National Australia Bank Internet Banking site. You can see two scripts are loaded from one site — again, a market research firm.
Simplistic Advice for Banks
Specifically to mitigate this risk, banks should consider the following:
Serve all scripts from your own domain and vet any third party scripts that you serve before deployment.
In particular, check third party scripts for back end communication, via AJAX or other channels.
Minimize the number of third party scripts anywhere that secure content must be presented.
Use the Content-Security-Policy HTTP header to prevent third party scripts on supported browsers (most browsers today support this).
There are of course other mitigations, such as Two Factor Authentication (2FA), which do reduce risk. However, even 2FA should not be considered a silver bullet: it is certainly possible to modify the login page to take over your current login in real time — all the user would see is a message that they’d mistyped their password, and as they login again, the malicious hacker is actively draining money from their account.
A final thought on 2FA: do you really want a hacker to have your banking password, even if they don’t have access to your phone? Why do we have these passwords in the first place?
Browser Developers
I believe that browser vendors could mitigate the situation somewhat by warning users if secure sites reference third party sites for resources, in particular where these secure sites have lower quality protection than the first party site. This protection is already in place where content is requested over HTTP from a HTTPS site, known as mixed content warnings.
There is no value in an Extended Validation certificate if any of the resources requested by the site are served from a site with lower quality encryption! Similarly, if a bank believes that 256-bit AES encryption is needed for their banking website, a browser could easily warn the user that resources are being served with lower quality 128-bit RC4 encryption.
Australian Banks
After this little investigation, I took a quick look at the big four Australian banking sites — ANZ, Commonwealth Bank, National Australia Bank, and Westpac. Here’s what I found; this is a very high-level overview and contains only information provided by any web browser!
Bank
Bank site security
# 3rd party scripts
Third party sites
Third party security
ANZ Bank
256-bit AES (EV certificate)
1
australianewzealandb.tt.omtrdc.net
128-bit RC4
NAB
256-bit AES (EV certificate)
2
survey.112.2o7.net
256-bit AES
Westpac
128-bit AES (EV certificate)
None!
Commonwealth Bank
128-bit AES (EV certificate)
9!
ssl.google-analytics.com
128-bit AES
commonwealthbankofau.tt.omtrdc.net
128-bit RC4
google-analytics.com
128-bit AES
d1wscoizcbxzhp.cloudfront.net
128-bit AES
cba.demdex.net
128-bit AES
Do you see how the *.tt.omtrdc.net subdomains are used by two different banking sites? In fact, this domain is used by a large number of banking websites. That would make it an attractive target, wouldn’t you think?
I reached out to all 4 banks via Twitter (yeah, I know, I know, “reached out”, “Twitter”, I apologise already), and NAB was the first, and so far only, bank to respond:
@MarcDurdin Hi Marc, I'm keen to have our tech team look into this for you – could you pls DM me more details? ^TC
Today I needed to download a recent backup of a database from a server in the data centre for testing major changes to the database locally… Things don’t really go as well as I expect:
Backup is 12GB. Yike, that’s going to take ages to download!
OK, so let’s zip it. 20 minutes later… we have a 2GB zip file. Well, that was worth doing.
Now copy (encrypted) zip to a server with HTTP access (much faster than downloading over SSH). About 1 minute (yep, nice fast network in the data centre)
Download zip to database server on local network. Takes about 45 minutes.
So try to unzip on server with Win Server 2003 Zip tool. Huh. It doesn’t support files > 4GB (plus it’s running low on disk space, so a bit of a juggling act to have enough space to unzip anyway).
OK, move the zip file to a client machine. 5 minutes on LAN
Unzip locally then move 12GB backup back to server. Best part of an hour. Yes, it’s a slow client machine and the LAN is only 100mbit.
Try to restore to SQL Server. Oops, old version of SQL Server on that server.
So install SQL Server 2008 on another server. That takes 45 minutes (including adding .NET framework).
Can’t do anything with it until we have patched SQL Server. So download and run SQL Server 2008 SP3. 20 minutes to download.
Installer starts while I’m not at computer, automatically cancels when screensaver starts (consent.exe)
Drat. Download SQL Server 2008 SP3 again as IE has deleted installer from cache. This time I save the service pack installer. 20 minutes.
Every now and then I receive a payment via PayPal. That’s not unusual, right? PayPal would send me an email notifying me of the payment, and I’d open up Outlook to take a look at it. All well and good. In the last week, however, something changed. When I clicked on any PayPal email, Outlook would take the best part of a minute to open the email, and what’s more, would freeze its user interface entirely while doing this. But this was only happening with emails from PayPal — everything else was fine.
Not good. At first I suspected an addin was mucking things up, so I disabled all the Outlook addins and restarted Outlook for good measure. No difference. Now I was getting worried — what if this was some unintended side-effect of some malware that had somehow got onto my machine, and it was targeting PayPal emails?
So I decided to do some research. I fired up SysInternals’ Process Monitor, set it up to show only Outlook in its filtering, and turned on the Process Monitor trace.
Process Monitor filter window – filtering for OUTLOOK.EXE
Then I went and clicked on a PayPal email. Waited the requisite time for the email to display, then went back to Process Monitor and turned off the trace. I added the Duration column to make it easier to spot an anomalous entry. This doesn’t always help but given the long delay, I was expecting some file or network activity to be taking a long time to run.
Adding the Duration column
Then scrolling up the log I quickly spotted the following entry. It had a duration of nearly 3 seconds which stood out like a sore thumb.
The first offending entry
This entry was a Windows Networking connection to connect to a share on the remote host \\102.112.2o7.net. This came back, nearly 3 seconds later, with ACCESS DENIED. Then there were a bunch of follow up entries that related to this, all in all taking over 30 seconds to complete. A quick web search revealed that this domain with its dubious looking name 102.112.2o7.net belongs to a well known web statistics company called Omniture. That took some of the load off my mind, but now I was wondering how on earth opening a PayPal email could result in Internet access when I didn’t have automatic downloads of pictures switched on.
One of the emails that caused the problem, redacted of course 🙂
I opened the source of the PayPal HTML email and searched for “102.112“. And there it was.
The HTML email in notepad
That’s a classic web bug. Retrieving that image of a 1×1 pixel size, no doubt transparent, with some details encoded in the URL to record my visit to the web page (or in this case, opening of the email):
What was curious about this web bug was the use of the “//” shorthand to imply the same protocol (e.g. HTTP or HTTPS) as the base page. That’s all well and good in a web browser, but in Outlook, the email is not being retrieved over HTTP. So Outlook interprets this as a Windows Networking address, and attempts a Windows Networking connection to the host instead, \\102.112.2o7.net\b….
At this point I realised this could be viewed as a security flaw in Outlook. So I wrote to Microsoft instead of publishing this post immediately. Microsoft responded that they did not view this as a vulnerability (as it does not result in system compromise), but that they’d pass it on to the Outlook team as a bug.
Nevertheless, this definitely has the potential of being exploited for tracking purposes. One important reason that external images are not loaded by default is to prevent third parties from being notified that you are reading a particular email. While this little issue does not actually cause the image to load (it is still classified as an external image), it does cause a network connection to the third party server which could easily be logged for tracking purposes. This network connection should not be happening.
So what was my fix? Well, I don’t really care about Omniture or whether PayPal get their statistics, so I added an entry in my hosts file to block this domain. Using an invalid IP address made it fail faster than the traditional use of 127.0.0.1:
Trying to connect my Windows 7 beta AspireOne to our wireless network (with a Netgear DG834G router), was receiving an unspecified error – “Windows failed to connect”. After playing with a multitude of settings, including security, access control, manual configuration and more, I discovered that changing wireless mode on the router from “g only” to “g & b” solved all my problems…
After that I went back and turned on all the security settings on the router again!
Security flaws like 
