You may have read my last 2 blog posts about the TrustedInstaller.exe leak that seems to be present in Windows 7 (Does Microsoft’s TrustedInstaller.exe have a leak? and Reproducing the TrustedInstaller.exe leak). It was not yet clear to me whether this leak that I observed was a bug or if it is by design, so I decided that the next time I had to build a Windows VM and install updates, I would capture some more log information about the update. I thought about using Process Monitor to capture everything, and this would certainly have captured reams of very interesting data, but this would have slowed the update install process down unacceptably on the day.
So instead, I ran the Handle program, also a part of the SysInternals suite, once a minute, and dumped its output to a text file. The update process ran as normal, and the handles logging generated a 50mb text file, which was fairly simple to import into a database. Now I had some data I could play with!
I picked a filename at random to look at in the logs. I closed my eyes and pointed, and the lucky filename chosen was winlogon! A simple search in the database came up with 4 different objects matching winlogon in the handle list:
C:\Windows\winsxs\amd64_microsoft-windows-winlogon_31bf3856ad364e35_6.1.7600.16447_none_cbe534e7ee8042ad C:\Windows\winsxs\amd64_microsoft-windows-winlogon_31bf3856ad364e35_6.1.7600.16447_none_cbe534e7ee8042ad\winlogon.exe C:\Windows\winsxs\Temp\PendingRenames\0859c097e7b9cc0153330000d0088004.amd64_microsoft-windows-winlogon_31bf3856ad364e35_6.1.7600.16447_none_cbe534e7ee8042ad_winlogon.exe_ac37d0c5 C:\Windows\winsxs\Temp\PendingRenames\aef6bd97e7b9cc0152330000d0088004.amd64_microsoft-windows-winlogon_31bf3856ad364e35_6.1.7600.16447_none_cbe534e7ee8042ad.manifest
The first entry in this list was a directory, in the Windows component store (aka Side-by-Side Store). The second was a reference to winlogon.exe within this same folder. The third and fourth entries gave a bit of a clue as to what may be going on, with the PendingRenames temporary folder hinting at a possible reason why the files were being kept open.
OK, so that was a very rough analysis. But things got a bit more complicated when I started looking a bit deeper. I picked the second entry, our winlogon.exe in the Windows component store, and had a look at the handles which were opened to that. Here’s what I found. Remember that this is based on minute-by-minute snapshots, so this is incomplete, but still paints a curious picture:
- 41 different handles were logged for this one file.
- 14 of these handles were still open at the end of the session, when TrustedInstaller.exe had no further CPU or disk activity. (TrustedInstaller.exe remains in memory until the restart, it seems)
Why does TrustedInstaller.exe need to maintain 14 open handles to a single file until Windows is restarted? And why does it need to open at least 41 different handles over time to the same file? This seems horribly inefficient, as the same pattern is manifest for the 2000 or more other objects that were opened by the TrustedInstaller.exe process.
I could understand a requirement to maintain a single open handle for concurrency and replacement (PendingRenames) operations. I’m sure there’s a reason… There may be another update to this story yet!
Hi Marc- I’ve been following your blog about the TrustedInstaller.exe memory leak. Have you found anything else? and has Microsoft ever responded to your initial findings?
Thanks
Paul
Hi Marc-
Thanks for your fantastic work on this thus far. We are experiencing the same issues here and was wondering if you have found anything new or if Microsoft has ever responded to your initial findings?
Thanks a TON!
Paul
Hi Paul, thanks for reading. No further news from Microsoft and I have not pursued this any further yet; hard to know where to take this from here. The obvious workaround is to multiple batches of updates, but given the default is to install all updates in one batch, it’s not great…
I have been experiencing the same problem on Windows 2008 R2, so looks like it isn’t just a Windows 7 issue.
Same here, when using microsoft supplied update vbs script. Seems to cause significant performance slowdown.
Can confirm this bug. I resolved it by using a microsoft VBS script to install just 30 – 40 patches at a time and reboot until everything is installed.
Experiencing same issue. Domain Controller ran out of resources whenever we were doing patching. It stopped all DNS functions.
Hi, I’m having this problem on 2 different Windows 2008R2 servers. Any new info available?
Has anyone tried reporting this as as a bug to Microsoft?
I haven’t seen any further answers at this point. Recommendation is staged install: install 30-40 updates at a time, reboot, and install further updates. I haven’t gone through a formal bug reporting process with MS.
I’m seeing this on a Win7 x64 system. TrustedInstaller.exe has 1,307,698 handles open! Working set ~3.3 GB, Virtual Size ~12.2GB.
I reinstalled Windows 7 x64 on my EliteBook 8540w. I lost patience with 117 of 134 important updates installed. Resource Monitor reported 13.65GB committed for TrustedInstaller.exe—on a machine with 4GB of physical RAM. I will retry with the staged approach recommended above.
Hi Marc — thanks for the fantastic posts!
I noticed that this problem was most prevalent in my lab environment where I run W2K8R2 guests with little memory (between 512MB and a maximum of 4GB).
I found that an interim solution was to change the Windows Update settings to “Never check for updates …” (even though it’s not recommended). Once set to never check, the “TrustedInstaller.exe” process doesn’t appear to spawn when performing Windows Update. I wouldn’t dare suggest this is a viable production solution, but at least my lab is a little more stable now. 🙂
Perhaps you found a more elegant solution?
Thanks!
Interesting! I have certainly never tried this but will investigate next time I build a VM from scratch and update the post accordingly.
I don’t know if this would be of any help but my system that has close to 1GB of RAM held by TrustedInstaller has in fact been disabled for Windows updates since it used in a production environment. We run Win7 64bit Ent. CPU use isn’t an issue but RAM is.
Recent reboot dropped the memory utilization back to 30,000K
Hi Everyone,
I’m just wondering if you can check your current UAC settings in Control Panel?
We’re seeing this on a LOT of servers that have has UAC practically disabled.
I think the Trusted Installer is a part of UAC which prompts the user within the “secure desktop” to trust an installer and allow it to install files.
I could be wrong, but this is my 5 cents.
Some admin don’t like UAC and rather than setting a GPO to allow UAC to be suppressed for Admins, they simply disable UAC.
Start > Control Panel (Category View) > User Accounts > User Accounts > Change User Account Control Settings
**Default Settings **
Default Option: Notify me only when programs try to make changes to my computer
o Don’t notify me when I make changes to Windows settings
** Modified Setting that seems to cause the Trusted Installer issue **
Never notify me when:
o Programs try to install software or make changes to my computer
o I make changes to Windows settings
I’m only guess here, but I’m thinking that this allows software to bypass UAC and gives open slather to the Trustedinstaller.exe process. Rather than prompting the admin.
For example, third party apps that constantly and silently update themselves Oracle Java or Adobe Flash Player for example may be triggering updates which rely on the Trusted Installer process?
Thanks for the comments.
On my test systems, this isn’t related to UAC — we are doing this with a clean install of Windows with default UAC settings, and have observed the same behaviour on many different systems, none of which have changed UAC defaults, either by policy or directly. We don’t have Java or Flash installed on the test machines, nor any other third party software which is doing automatic updates.
Finally, I seriously doubt that third-party apps are causing this issue anyway — I’m not aware of any automatic updaters that try to install Windows Updates silently, and suspect if there were any, they’d be red flagged pretty quickly.
Seeing this behavior before with Windows XP… http://beta.slashdot.org/story/195683
Yeah, it’s probably related 🙂
thanks for the analysis, was wondering why two 2008R2 servers with the same patch sets pending after 80+ days have different TrustedInstaller memory usage – one is 540MB, the other 143 MB. Has anyone heard if Microsoft is aware of the behavior?
I’ve come to the conclusion that the patch infrastructure must have data structures with O(n*n) patterns. Yes, there are a lot of different patches and updates for Windows, but nothing to justify the performance characteristics we’ve seen.
I was looking into this recently on a 2008 r2 server and below is what I found.
To see if trustedinstaller is doing anything have a look at c:\windows\logs\cbs\cbs.log
If everything’s good trustedinstaller should shut itself down after 10 mins of inactivity.
In my situation I found people leaving Server Manager open was keeping trustedinstaller running due to the “features summary” section that it displays and the default 2-minute refresh interval.
To check if this is your problem see if there are any mmc.exe processes running the server manager snap-in (add the command line column in task manager and look for servermanager.msc).
If there are then end them and watch for new entries being written to cbs.log.
If no new entries are being written to cbs.log then the trustedinstaller.exe process should shut down after 10 minutes.
If that works you could look at changing or disabling the server manager refresh (I haven’t tried this yet but looks like it can be disabled for all users: https://www.windows-security.org/b4f6440b461433025a005e47d1583550/configure-the-refresh-interval-for-server-manager).
If you’re still seeing new entries being written to the cbs.log after ending all server manager instances then suggest trying process monitor to track down the source (you may capture a script or something kicking off just before the cbs.log gets updated).
Hope this helps someone!
Nice analysis 🙂
Brilliant, problems disolved just as you predicted! Thanks!
Referring to my previous message.
Handler stats on my workstation: post: https://postimg.org/image/6uuttcj0b/