Microsoft's middle finger to devs: IFileDialog edition

The set

It is the 1st of September 2025. I wake up to a GitHub comment notification posted on a Pull Request I had submitted to FreeCAD two weeks prior about hiding the excruciatingly long file extension lists that exist on wildcard filters you can see when opening a file, e.g. "Supported formats".

Screenshot of the "Supported formats" file filter entry, spanning the entire screen's width yet still overflowing it.

The comment comes from Chris Hennes, one of the biggest FreeCAD maintainers, and the content is pretty simple: my PR does not work on Windows, the long lists are still displayed. Being busy at the time, I leave that to the side as the PR is not a priority, and FreeCAD had other priorities in its merge queue anyway.

Fast forward to the 21st of March 2026, and Chris comments that Windows visibly does not support the QFileDialog::HideNameFilterDetails flag, which surprised me. At my day job of embedded Linux systems integration (that I still had at the time) digging into Qt's source was a common occurence, so I immediately dig into qtbase's repository to find the Win32 implementation of QFileDialog. Like with other OS platform integrations, this is code that rarely ever changes, so it's no surprise to find the HideNameFilterDetails implementation landed 14 years prior with no major modifications since.

Curious to see the bug for myself, I install a fresh Windows 11 VM with libvirt, not having used Windows for development in more than 10 years now, and download Visual Studio 2022 to compile FreeCAD; the latter of which proves especially slow across the VM layer. The pixi-based workflow described in the FreeCAD build docs prove especially useful for avoiding tedious dependency downloads.1

I manage to reproduce the bug. My first instinct is to suspect a bug in Qt's code, the HideNameFilterDetails implementation thereof meddles with the file format filter lists in some fashion I did not bother to deeply understand yet, but which splits the strings to process further. My Pull Request, on the other hand, abuses the HideNameFilterDetails to still show extension lists when they are short enough; the first logical action is to test if they actually work as intended, stepping aside from the "Supported formats" issue for a moment.

Edit, recompile, observe. The extension lists I willingly display do show up modified as expected. This means neither Qt or Win32 check for consistency between the displayed list and the actual filter list.

FreeCAD's Open file dialog filter combo box, with entries' extensions replaced with (*.EXTSWOULDGOHERE)

The setting

Testing some more quickly reveals the absence of a visible extension list makes the dialog force the display of the underlying list. Luck would have it that I installed a fresh Windows mere hours prior, and doing exactly that reminded me of a setting one has to change with every install since it was introduced in Windows XP:

The "Hide extensions for known file types" checkbox in Explorer's Folder Options

The "Hide extensions for known file types" option, which is stored in the registry at HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\HideFileExt, changes pretty much all first party Microsoft software to hide extensions known to the registry. It used to be easily toggleable from Explorer's ribbon menu in Windows 8 and 10, but that got removed in 11 when they trashedredesigned the Explorer UI. Turning it back on reveals a behaviour change in the file dialogs:

FreeCAD's Open file dialog filter combo box, showing what was expected from the start

The onset

Not knowing if this is a Qt specific thing or general Win32 API, I grep for instances of SHGetSettings() or SHGetSetSettings() in qtcore, to no avail. So the difference is in Win32's shell.

Qt relies on the IFileDialog COM interface that provides the extended open/save windows introduced with Vista's release. That's nice and all, but the documentation for IFileDialog::SetFileTypes says nothing about how the filter lists actually work.

In my hay day of messing about with C# and .NET 3.5 I had a small obsession with the new Vista/7 look and the newfangled UI controls that dropped. Action selectors, enhanced dialog boxes, extending Aero glass borders into the client area, all that jazz. Microsoft might be exceptionally bad at documentation but at least they provide code samples buried in the 11+ GiB Windows SDKs, nowadays easily reachable on GitHub, among them being CommonFileDialog.

Fetch, build, test. The sample conveniently has the filter list at the top:

const COMDLG_FILTERSPEC c_rgSaveTypes[] =
{
    {L"Word Document (*.doc)",       L"*.doc"},
    {L"Web Page (*.htm; *.html)",    L"*.htm;*.html"},
    {L"Text Document (*.txt)",       L"*.txt"},
    {L"All Documents (*.*)",         L"*.*"}
};

If one was to change the extension list on the left, you would expect whatever you replaced it with to show up in the filter box right? Not quite. My tests reveal that Windows... expects something that looks like an extension list and will append the one on the right no questions asked if it didn't find one and the HideFileExt shell setting is off. Even as simple as (*.) is enough to satisfy it, but no shorter; it has to have one or more filename stem wildcards. I went to the lengths of installing a Vista VM to check if this was present on launch, and the answer is yes.

The come up

No, this can't be right. I do distinctly remember software whose filter dialogs includes entries without visible extensions, and easy enough, turns out mspaint.exe is one of them. Armed with the old and venerable API Monitor v2 alpha-r13 (fitting for Vista's era), I hook IFileDialog methods, run Paint, hit Ctrl+O and... nothing. Disappointed, I turn to GHIDRA to quickly find the strings and what Win32 API function gets called.

I should've seen it coming, it's obvious. It's the older GetOpenFileName() function. Usage examples for it are available all over the internet, and no further setup is required outside of the big OPENFILENAME struct it takes as an argument, unlike IFileDialog and its ugly COM instantiation.

Pass a filter like "Shown (*.abc)\0*.abc\0Hidden\0*.def\0\0" and... it works. Acts just fine. In fact it does not care about HideFileExt and will always show exactly what you asked it to.

GetOpenFileName() visual result: Shown (*.abc), and Hidden with no visible extensions.

Where is IFileDialog implemented anyway? And can I tear it to shreds?

I've been referring to IFileDialog this whole time but what's really at play here is an implementation of that interface, pointed to by the CLSID_FileOpenDialog constant passed to CoCreateInstance(). I expect this to be in some shell DLL, but Microsoft, in their ever great wisdom, made it so any COM class' code location is essentially opaque, hidden in the implementation details of HKEY_CURRENT_CLASSES\CLSID. OleView can help shed some light on this, but it is buried in the SDK and severely outdated, so OleView.NET will do. Just need to plug in the class' GUID visible in Wine's version of IDL files, since those are much more readable than the official ones:

[
    helpstring("File Open Dialog"),
    threading(apartment),
    uuid(dc1c5a9c-e88a-4dde-a5a1-60f82a20aef7)
]
coclass FileOpenDialog { interface IFileOpenDialog; }
OleView.NET displaying FileOpenDialog's server dll is comdlg32.dll.

The rest of this article will use comdlg32.dll version 6.0.6002.18005 loaded at 0x7FEFE240000. If one jumps to the IFileDialog VTable location as given to us by OleView.NET, GHIDRA will show the address as referenced by 2 functions, the former of which calls RegisterWindowMessageW() and the latter CoTaskMemFree(), clearly being a constructor and destructor (as those, in basically all languages, reinstate VTables as the ctor/dtor chains run). How convenient!

The constructor is only called by a single function at 0x7fefe24ea88, and above the call site is what suspiciously look like an allocation of what's further passed as the this pointer:

puVar3 = (undefined8 *)FUN_7fefe241ed0(0x6f0);
if (puVar3 == (undefined8 *)0x0) {
  uVar2 = FUN_7fefe24eaff();
  return uVar2;
}
FileOpenDialog_ctor(puVar3,param_1,param_2,param_4);
uVar2 = FUN_7fefe24eaff();

Turns out, this yet unnamed function just calls HeapAlloc(). So here we have it, FileOpenDialog is 0x6f0 bytes big; and knowing this will tremendously help GHIDRA's decompiler infer data locations.

COM's IDL files describe the VTable contents and keep the order in the resulting binary interface, but inheritance exists and appends its own function pointers in front of the rest. I can't be bothered to figure out the actual offset of SetFileTypes, so let's just get it by hacking up the dialog example above:

#include <type_traits>
#define CINTERFACE
#include <Shobjidl.h>
// ...
printf("IFileDialog::SetFileTypes @ vtable+0x%llX\n",
  offsetof(std::remove_reference_t<decltype(*pFileOpen->lpVtbl)>, SetFileTypes));
IFileDialog::SetFileTypes @ vtable+0x20

Pointed by that address is our culprit. Or is it? The function body seems to only copy file filters, not modify them:

i = 0;
this->filterSpecCount = cFileTypes;
if (cFileTypes != 0) {
  do {
    ok = SHStrDupW(rgFilterSpec->pszName,&this->filterSpecs[i].pszName);
    if (((int)ok < 0) ||
        (ok = SHStrDupW(rgFilterSpec->pszSpec,&this->filterSpecs[i].pszSpec), (int)ok < 0))
    goto cleanup;
    i = i + 1;
    rgFilterSpec = rgFilterSpec + 1;
  } while (i < cFileTypes);
}
ok = FUN_7fefe24b9f4(this->filterSpecCount,this->filterSpecs,
                      (undefined8 *)&this->field_0x530);

The peak (/s)

Throwing things at the wall

If the list doesn't get modified when we set it, perhaps it gets changed when the file picker dialog is built and shown? ::Show is just above in the VTable, and the function body sets the dialog window up with DialogBoxIndirectParamW(), to which a "dialog box procedure" (0x7fefe242210) pointer is passed—I expect the filter dropdown to be populated there.

You know what, let's search for constants. You set the entries of any given dropdown ("Combo Box") by sending it a CB_ADDSTRING message, the ordinal of which is 0x143... 10 results, 8 of which are movs into RDX2, most definitely what we want.

The first of those (0x7fefe25ccb4) is down the reference chain for the sister GetOpenFileName() function seen previously, and happens to be in a loop sending CB_ADDSTRING followed by 0x151=CB_SETITEMDATA to control ID 0x470 of the dialog...

szFilterCursor = (LPCWSTR)pOFN->lpstrFilter;
if ((szFilterCursor == (LPCWSTR)0x0) || (*szFilterCursor == L'\0')) {
  pOFN->nFilterIndex = 0;
}
else {
  do {
    _ = SendDlgItemMessageW(param_1,0x470,0x143,0,(LPARAM)szFilterCursor);
    uNameLen = lstrlenW(szFilterCursor);
    szFilter = szName + uNameLen + 1U;
    SendDlgItemMessageW(param_1,0x470,0x151,_ & 0xffffffff,(ulonglong)szFilter);
    uSpecLen = lstrlenW(szFilterCursor + (uNameLen + 1U));
    szName = szFilter + uSpecLen + 1U;
    szFilterCursor = szFilterCursor + (uNameLen + 1U) + (uSpecLen + 1U);
  } while (*szFilterCursor != L'\0');
}

So now we know why it works as exactly specified with the legacy API: the processing is as basic as it gets! No filtering, no splitting.

🤔 The file dialog is the same between the legacy and the new API, meaning the 0x470 ID is probably consistent between both. Actually, inspect.exe on the modern window confirms that: AutomationId: "1136", and 1136 is the decimal of 0x470. However of the remaining search results for CB_ADDSTRING, all call sites refer to other IDs, the closest one being 0x471. This probably means other codepaths use SendMessageW() directly with a HWND of the combo box instead. No bueno.

Seeing it does stick

A sidetrack about SHGetSetSettings()

New approach: we know the filters change depending on the results of SHGet(Set)Settings(), so let's hook on those with x64dbg3

Address          To               From             Size Comment                              Party 
000000000012E218 000007FEFE24ED1C 000007FEFE323EB0 60   shell32.000007FEFE323EB0             System
000000000012E278 000007FEFE24EAFA 000007FEFE24ED1C 40   comdlg32.000007FEFE24ED1C            System
000000000012E2B8 000007FEFE276058 000007FEFE24EAFA 60   comdlg32.000007FEFE24EAFA            System
000000000012E318 000007FEFE2761FB 000007FEFE276058 50   comdlg32.000007FEFE276058            System
000000000012E368 000007FEFE25BEF3 000007FEFE2761FB 60   comdlg32.000007FEFE2761FB            System
000000000012E3C8 000007FEFE264A76 000007FEFE25BEF3 1150 comdlg32.000007FEFE25BEF3            System
000000000012F518 000007FEFE25BCCF 000007FEFE264A76 30   comdlg32.000007FEFE264A76            System
000000000012F548 000000013F6B1A74 000007FEFE25BCCF 4E0  comdlg32.000007FEFE25BCCF            User
000000000012FA28 000000013F6B2A49 000000013F6B1A74 50   consoleapplication1.000000013F6B1A74 User

The 2nd line of that call stack is... in the FileOpenDialog_ctor function ._.
Don't you love how Microsoft maintains an ABI that uses function ordinals for some but not all functions? They aren't even stable, so tools that don't ship with matching tables for every possible DLL version aren't aware of them, and so the line just looks like Ordinal_68(local_38,2,0); in GHIDRA. The literal 2 is the value of SSF_SHOWEXTENSIONS as expected. Resolving some names, we get:

SHELLSTATEW shellSettings;
// ...
SHGetSetSettings(&shellSettings,2,0);
// ...
// field_0x1b0 is apparently a 32-bit bitfield, in which the 2nd bit is set to shellSettings.fShowExtensions
*(uint *)&this->field_0x1b0 =
     *(uint *)&this->field_0x1b0 |
     (int)(shellSettings._0_4_ << 0x1e) >> 0x1e & 2U | (param_3 & 1) << 7;

But finding uses of a bitfield in an optimised x86_64 binary is difficult; there's many ways of doing bit testing. This isn't working.

Earlier I observed a visible filter as simple as *. is enough to calm the beast. Searching for this in the decompiled output won't work, surely. It can't be this simple, right? Right??

// FUN_7fefe2454b8, simplified version
for (specIdx = 0; specIdx < (this->dialog).filterSpecCount; ++specIdx) {
  const auto &spec = (this->dialog).filterSpecs[specIdx];
  if (((~((this->dialog).field_0x2a0 >> 4) & 1) == 0) ||
      (((this->dialog).field_0x1a8 & 2) == 0) ||
      (StrStrIW(spec->pszName, L"*.") != NULL)) {
    add_to_list(spec->pszName, spec->pszFilter);
  } else {
    uVisualBufferLen = lstrlenW(spec->pszFilter) + 4 + lstrlenW(spec->pszName);
    pszVisualBuffer = CoTaskMemAlloc(uVisualBufferLen * 2);
    if (pszVisualBuffer != NULL) {
      printf_thing(pszVisualBuffer, uVisualBufferLen, L"%s (%s)", spec->pszName, spec->pszFilter);
      add_to_list(pszVisualBuffer, spec->pszFilter);
      CoTaskMemFree(pszVisualBuffer);
    }
  }
}

If I patch the if jump out it definitely won't wor--

x64dbg showing a patched jump (jnz → jmp), Shell options showing "Hide extensions for known file types" is off, and the Open File dialog with no appended extensions

How uniquely Microsoftesque of a solution.

The offset

haha get it? the offset? into the struct?

The after effects

ms api shite, had to use old af func in FC
1

A significant development hindrance that Microsoft's solution vcpkg did not meaningfully solve by still forcing you to build everything from scratch, which is not reasonable for very large libraries like Qt.

2

Or into EDX, which is the sign-extended 32-bit view into RDX.

3

If you're old school, you've used OllyDbg before; x64dbg is the open source spiritual sucessor. It stopped working on Vista long ago though, so the last version you can use there is the 2020-12-14_15-31 snapshot.