The offset (into the struct)

What we previously observed when I patched the if jump out is that only 3 cases ever prevent the extension list from being appended to a filter dropdown entry:

1. (~(this->field_0x2a8 >> 4) & 1) == 0, aka this->field_0x2a8 & 0x10 is set
2. (this->field_0x1b0 & 2) == 0, aka this->field_0x1b0 & 0x2 is unset
3. StrStrIW(spec->pszName, L"*.") != NULL, aka L"*." is found somewhere in the filter name

The third option is obviously not acceptable: the point is to display just the name, without any whacky constraints. The other two cases though might show some promise.

After spending many hours decompiling other parts of comdlg32's file dialog implementation, I managed to figure out every single spot where these two fields are used.

• field_0x1b0 is a bitfield used for tracking various internal behaviour bits, which I renamed dwInternalFlags instead, and the 0x2 flag seen above is only ever set once, to wit, we have already seen it prior in the sidetrack about SHGetSetSettings():

  // In FileDialog::FileDialog(), starting at 0x7fefe24eb40
  SHGetSetSettings(&shellSettings, 2, 0);
  this->dwInternalFlags =
    this->dwInternalFlags & ~(FDIF_HIDE_SCROLLBAR | FDIF_SHELL_SHOW_EXTENSIONS) |
    (shellSettings.fShowExtensions ? FDIF_SHELL_SHOW_EXTENSIONS : 0) |
    (bHideScrollbar ? FDIF_HIDE_SCROLLBAR : 0);

  As the names I've introduced show, this is just relaying the "Hide extensions for known file types" user setting. All other code locations that write to this variable do it in a masked fashion, and none of those masks include FDIF_SHELL_SHOW_EXTENSIONS, so there is no possibility for a hidden function changing it.
• field_0x2a8 on the other hand is referenced in a fair number of places and seem to complement dwInternalFlags's abilities, and two of those places are functions that very suspiciously look like a setter and getter function, their addresses being present in some kind of VTable... we'll come back to this shortly.

COM and its implementation

This article is no crash course on Microsoft's Component Object Model, but pointing out a few basics is appropriate here:

• Classes and Interfaces are the two major aspects of using COM, and while classes need to be declared in the registry at HKEY_CURRENT_CLASSES\CLSID for obvious code-loading reasons, interfaces (HKEY_CURRENT_CLASSES\IID) have no such mandatory declaration
• Any class can implement as many interfaces as it wants (and this is not declared in the registry or anything)
• Internally, obtaining a COM object only gives you access to that object through a pointer of the specific interface you requested, and you cannot natively re-request or translate that to another interface from that same underlying object
• The lifetime model of COM-obtained objects is undefined by the system and left to implementors

Solving the latter two points led Microsoft to implement an über-interface called IUnknown, providing a QueryInterface() method that allows for interface pointer conversion for a given backing object, and AddRef()/Release() for reference-counted lifetime management. This interface is central to COM in that (almost) all interfaces in existence must inherit it, as it's the root that base functions like CoCreateInstance() expect.

In most COM provider code, an object implementing IWhatever will have an actual struct IWhatever as a member, where that struct only contains a, "lpVtbl" pointer to its implementation VTable. All functions referenced therein simply refer to the actual object they're part of with some offset on the This pointer.

For Microsoft's system DLLs specifically, the implementation of QueryInterface is commonly done through a function present in shlwapi.dll that was undocumented for a while. And by "for a while", I mean Microsoft had to be forced by U.S. antitrust hearings to publicise its existence along with many other undocumented functions, as prior to that the function was only known as shlwapi's Ordinal 219. It is now publicly known as QISearch() and is documented on MSDN.

Simply put, it takes a this pointer, and a table of (Interface ID, Interface's offset into this) tuples, and gives you a pointer to any requested VTable, which is used as the pointer for that IWhatever you requested. This creates very convenient tables that say 1. which VTable pointers assigned in constructors are and 2. what offset the pointed-to functions will apply (negated) to get the real this pointer:

The third convenient fact with these QISearch() tables ("QITABs") is they're—bar potential manual handling code in a QueryInterface()—a full enumeration of supported interfaces, even if they are private, i.e. undocumented and possibly undeclared in HKEY_CURRENT_CLASSES\IID thereby invisible to tools like OleView.

Private until it isn't

As it so happens, the third table entry, right below those in the previous screenshot, is one for Interface ID 2539E31C-857F-43C4-8872-45BD6A024892 at offset 0x10, which is actually listed by tools like OleView.NET on Vista, known as IFileDialogPrivate there, and IFileDialogPrivate_Vista by some Wine testcases.

Sadly no IDL or VTable for this private interface exists around. No matter, it can be figured out by reverse-engineering it later. And later means now! Remember field_0x2a8 from earlier? Its getter and setter functions are pointed to by the 4th and 5th entry of the VTable being set up at the very offset of the private interface. As for the preceding 3 function pointers, a good assumption to make with COM is whatever you're dealing with inherits from IUnknown, and sure enough the pointed-to functions call their respective parent FileOpenDialog functions. Give field_0x2a8 a name of dwPrivateOptions, a new FileDialogPrivateOptions enum type, and we get the following partial VTable:

struct IFileDialogPrivate_Vista_PartialVtbl {
  HRESULT (*QueryInterface)(IFileDialogPrivate_Vista *This, REFIID riid, void **ppvObject);
  ULONG (*AddRef)(IFileDialogPrivate_Vista *This);
  ULONG (*Release)(IFileDialogPrivate_Vista *This);
  HRESULT (*GetPrivateOptions)(IFileDialogPrivate_Vista *This, FileDialogPrivateOptions *pPrivateOptions);
  HRESULT (*SetPrivateOptions)(IFileDialogPrivate_Vista *This, FileDialogPrivateOptions dwPrivateOptions);
};

As recalled at the start of this 2nd episode, the 0x10 flag is used to disable the extension display, so let's name it FDPO_DONT_FORCE_SHOW_EXTS, slap together some code to get the private interface and call it, and Bob's your uncle:

FileDialogPrivateOptions opts;
FileDialogPrivate_Vista *privVista;
pFileOpen->QueryInterface(IID_IFileDialogPrivate_Vista, &privVista);
privVista->lpVtbl->GetPrivateOptions(privVista, &opts);
privVista->lpVtbl->SetPrivateOptions(privVista, opts | FDPO_DONT_FORCE_SHOW_EXTS);
privVista->lpVtbl->Release(privVista);
// ...
pFileOpen->Show();

The after effects

There isn't only one

The code working on Vista is great, but the OS might be a tad outdated by this point. Windows 10 isn't keen to the kind of meddling I'm doing here, the private interface seems absent altogether. Maybe not so surprising considering ReactOS calls it IFileDialogPrivate_Vista, but does the interface still exist in more modern versions of the OS? I can check in my Windows 11 VM—it doesn't—but can't be bothered to install 7, 8 and 10 just for that. Maybe WinSxS can help us here, since Microsoft's strategy to backwards compatibility is keeping a copy of everything.

Sadly my Windows 11's SxS store only has a few versions, and none feature the Vista interface. As for prior versions, I discovered Winbindex, a site listing DLLs that can exist in SxS alongside their download links, upon reading Auscitte's blog post on SxS exploration; this covered my bases for Windows 10, but not 7 or 8. For those, good ol' "comdlg32.dll download" on [insert search engine here] will point you to various version archives around, allowing me to collect DLLs version 6.1.x and 6.2.x, probably matching Windows NT 6.1 and 6.2, aka "7" and "8".

Some small amount of revering later, the results are in: it's still possible to set the very same private option bits in modern versions of the DLL, albeit with different interfaces and VTables to match:

• IFileDialogPrivate_Vista (2539E31C-857F-43C4-8872-45BD6A024892): 6.0 = Vista
• IFileDialogPrivate_W7 (AC92FFC5-F0E9-455A-906B-4A83E74A803B): 6.1 - 6.2 = W7 & W8
• IFileDialogPrivate_W10 (9EA5491C-89C8-4BEF-93D3-7F665FB82A33): 10.0 = W10 & W11

The Vista one is sorted already. The Windows 10+ one shows up in searches on GitHub as it is actually used and fully named in Files UWP, an explorer.exe alternative featuring tabs and other quality of life enhancements.

As for Windows 7 and 8, the VTable has the same layout as the Vista one as far as calling SetPrivateOptions() is concerned.

What now?

This all started from my contribution to FreeCAD, and now I have figured out a way to bend the file dialog to my initial will.

Too bad I ended up committing code using the legacy GetOpenFileName() anyway because this Just Works™ 🤡

In all seriousness, I might come back and improve that code with trying out the private interfaces first, and if all fail then use the crusty old function. In the meantime, you can download the messy C++ code I used to test things out in these 2 posts.