MinGW - Minimalist GNU for Windows

Fork

• mingw-dist
• build-aux
• wtklite
• pexports
• website
• mingw-get
• mingw-org-wsl
• mingw-pkgspecs

Donation

Reply To eliz

First, we need a value for _WIN32_WINNT that specifies Windows 10, in sdkddkver.h:
#define _WIN32_WINNT_WIN10 0x0A00

Yes, we definitely need this; it's even almost sufficient, in isolation, but I think that, for completeness, we should also add the NTDDI_WIN10 sub-version variants, so I committed this.

Additionally, I note that the _WIN32_IE features selectors have not been updated, since IE8, so I also committed this; (curiously, docs.microsoft.com indicates that _WIN32_IE_IE100 and _WIN32_IE_IE110 are characterized by the same value).

To keep this ball rolling, perhaps you (or someone else) would care to propose patches, to address the remaining omissions?

Reply To eliz

Second, we need functions, data structures, and macros to create and update lists of attributes for process and thread creation.

Okay. In the absence of any third party patch submissions, I decided to have a look into this, myself.

These are:
* InitializeProcThreadAttributeList function
* UpdateProcThreadAttribute function
* DeleteProcThreadAttributeList function

I can find prototypes for these, on docs.microsoft.com, so no problem to add them to <winbase.h>. AFAICT, they should be available from Vista onwards — in fact, they must be if STARTUPINFOEX is, because the additional data which this incorporates, over STARTUPINFO itself, cannot be manipulated without them.

* PROC_THREAD_ATTRIBUTE_LIST structure

AFAICT, this is undocumented (by Microsoft), but maybe we can get away with a opaque declaration, since this should only ever be accessed via an LPPROC_THREAD_ATTRIBUTE_LIST pointer, and manipulated using the preceding three functions.

* STARTUPINFOEX structure

Once again, I can implement this, in terms of the documentation on docs.microsoft.com.

* PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE macro
* Other PROC_THREAD_ATTRIBUTE_* macros

What do you mean by other PROC_THREAD_ATTRIBUTE_* macros? I'm guessing you are referring to those enumerated as valid for the Attribute parameter of UpdateThreadAttribute(), but that documentation is utterly useless, because the enumeration is no more than a meaningless list of macro names; it does not specify appropriate values for any of those macros.

These seem to be supported since Windows 7, with the exception of PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, which is only supported since Windows 10, and STARTUPINFOEX, which is supported since Vista. I think the proper place for them is in the winbase.h header file.

Actually, some seem to be supported in Vista, some in Win7, and some not until Win8; PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE appears not to be supported even for all Win10, but only from Win10-v1809 (October 2018 update). I agree that <winbase.h> is the appropriate header for deployment, but that hardly matters; the lack of appropriate publicly accessible documentation is a blocker! Unless someone can provide appropriate third-party documentation, (not copies of Microsoft SDK headers), this request cannot be fulfilled.

We also need the EXTENDED_STARTUPINFO_PRESENT flag for the CreateProcess function (this flag is supported since Vista and should be in winbase.h).

This one is easy enough; it is adequately documented here.

Reply To keith

Reply To eliz

Second, we need functions, data structures, and macros to create and update lists of attributes for process and thread creation.

Okay. In the absence of any third party patch submissions, I decided to have a look into this, myself.

I'm sorry I dropped the ball on this. I actually collected the data, but didn't have time to generate patches so that I could submit them to you.

These are:
* InitializeProcThreadAttributeList function
* UpdateProcThreadAttribute function
* DeleteProcThreadAttributeList function

I can find prototypes for these, on docs.microsoft.com, so no problem to add them to <winbase.h>. AFAICT, they should be available from Vista onwards — in fact, they must be if STARTUPINFOEX is, because the additional data which this incorporates, over STARTUPINFO itself, cannot be manipulated without them.

Thanks.

* PROC_THREAD_ATTRIBUTE_LIST structure

AFAICT, this is undocumented (by Microsoft), but maybe we can get away with a opaque declaration, since this should only ever be accessed via an LPPROC_THREAD_ATTRIBUTE_LIST pointer, and manipulated using the preceding three functions.

Don't you need it to declare STARTUPINFOEX?

* PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE macro
* Other PROC_THREAD_ATTRIBUTE_* macros

What do you mean by other PROC_THREAD_ATTRIBUTE_* macros? I'm guessing you are referring to those enumerated as valid for the Attribute parameter of UpdateThreadAttribute(), but that documentation is utterly useless, because the enumeration is no more than a meaningless list of macro names; it does not specify appropriate values for any of those macros.

These seem to be supported since Windows 7, with the exception of PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, which is only supported since Windows 10, and STARTUPINFOEX, which is supported since Vista. I think the proper place for them is in the winbase.h header file.

Actually, some seem to be supported in Vista, some in Win7, and some not until Win8; PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE appears not to be supported even for all Win10, but only from Win10-v1809 (October 2018 update). I agree that <winbase.h> is the appropriate header for deployment, but that hardly matters; the lack of appropriate publicly accessible documentation is a blocker! Unless someone can provide appropriate third-party documentation, (not copies of Microsoft SDK headers), this request cannot be fulfilled.

Are we allowed to look in MinGW64 headers and take the info from there?

We also need the EXTENDED_STARTUPINFO_PRESENT flag for the CreateProcess function (this flag is supported since Vista and should be in winbase.h).

This one is easy enough; it is adequately documented here.

Thanks. And sorry again for not seeing this through.

Reply To eliz

Reply To keith

Reply To eliz

* PROC_THREAD_ATTRIBUTE_LIST structure

AFAICT, this is undocumented (by Microsoft), but maybe we can get away with a opaque declaration, since this should only ever be accessed via an LPPROC_THREAD_ATTRIBUTE_LIST pointer, and manipulated using the preceding three functions.

Don't you need it to declare STARTUPINFOEX?

No, I don't think so. All that STARTUPINFOEX needs is a pointer to a PROC_THREAD_ATTRIBUTE_LIST, which must be allocated separately. Its size is not fixed, but rather, must be determined at initialization time, by calling InitializeProcThreadAttributeList() twice, (once to get the size of memory block to be allocate, then again to initialize the allocated block), and its content must then be specified using UpdateProcThreadAttribute() (as many times as there are attributes). We never access the content directly, (and, since Microsoft don't provide an accessor for content retrieval, that content must be viewed as write-only, from the user perspective), simply declaring PROC_THREAD_ATTRIBUTE_LIST as an opaque struct

 typedef struct _PROC_THREAD_ATTRIBUTE_LIST *LPPROC_THREAD_ATTRIBUTE_LIST;

Reply To eliz

Reply To keith

Reply To eliz

* PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE macro
* Other PROC_THREAD_ATTRIBUTE_* macros

What do you mean by other PROC_THREAD_ATTRIBUTE_* macros? I'm guessing you are referring to those enumerated as valid for the Attribute parameter of UpdateThreadAttribute(), but that documentation is utterly useless, because the enumeration is no more than a meaningless list of macro names; it does not specify appropriate values for any of those macros.

These seem to be supported since Windows 7, with the exception of PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, which is only supported since Windows 10, and STARTUPINFOEX, which is supported since Vista. I think the proper place for them is in the winbase.h header file.

Actually, some seem to be supported in Vista, some in Win7, and some not until Win8; PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE appears not to be supported even for all Win10, but only from Win10-v1809 (October 2018 update). I agree that <winbase.h> is the appropriate header for deployment, but that hardly matters; the lack of appropriate publicly accessible documentation is a blocker! Unless someone can provide appropriate third-party documentation, (not copies of Microsoft SDK headers), this request cannot be fulfilled.

Are we allowed to look in MinGW64 headers and take the info from there?

Sorry, but I have to say "no". Although I don't have conclusive proof, our history with that project leads me to a strong suspicion that their headers will have been plagiarized from Microsoft's, in violation of Microsoft's terms of use; I will not make myself an accessory to such suspected plagiarism.

Reply To keith

Are we allowed to look in MinGW64 headers and take the info from there?

Sorry, but I have to say "no". Although I don't have conclusive proof, our history with that project leads me to a strong suspicion that their headers will have been plagiarized from Microsoft's, in violation of Microsoft's terms of use; I will not make myself an accessory to such suspected plagiarism.

What about packages under Mozilla Public License, then? Or Wine?

I guess my point is that the Internet is full of places which cite the values of these constants, so maybe some of these places is okay for us to glean the info? Especially since these are just numerical values, not really code.

Failing that, I guess each one who needs to use these will have to discover the values him/herself.

Reply To eliz

Reply To keith

Are we allowed to look in MinGW64 headers and take the info from there?

Sorry, but I have to say "no". Although I don't have conclusive proof, our history with that project leads me to a strong suspicion that their headers will have been plagiarized from Microsoft's, in violation of Microsoft's terms of use; I will not make myself an accessory to such suspected plagiarism.

What about packages under Mozilla Public License, then? Or Wine?

I would also rule Wine out, for the same reason as I reject mingw-w64; in the past, some Wine developers have admitted, (to me), that they are willing to download a Windows SDK, ignore its stipulated terms of use, and read the header files to extract values which Microsoft have, otherwise, declined to disclose.

I guess my point is that the Internet is full of places which cite the values of these constants, so maybe some of these places is okay for us to glean the info?

Indeed. If you can find sources, which aren't encumbered by an exclusive licence, that's fine; for example, from this GitHub page, I deduce that

 internal const uint PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE = 0x00020016;

Especially since these are just numerical values, not really code.

The Wine guys state this argument as justification for violating the Microsoft terms of use, but IANAL, and I have no desire to be required to defend this standpoint in a court of law.

Failing that, I guess each one who needs to use these will have to discover the values him/herself.

Yes. Failing a permissively licensed source, that's the best option. In the past, I have used MinGW-compiled code, in a brute force fashion, to determine undisclosed values, based solely on the observed effects of trying different candidate values. Last resort, maybe, but occasionally the only option.

Reply To keith

Indeed. If you can find sources, which aren't encumbered by an exclusive licence, that's fine; for example, from this GitHub page, I deduce that

 internal const uint PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE = 0x00020016;

A further example: on pinvoke.net I see

 PROC_THREAD_ATTRIBUTE_PARENT_PROCESS = 0x00020000,
 PROC_THREAD_ATTRIBUTE_HANDLE_LIST    = 0x00020002

Reply To keith

Reply To keith

Indeed. If you can find sources, which aren't encumbered by an exclusive licence, that's fine; for example, from this GitHub page, I deduce that

 internal const uint PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE = 0x00020016;

A further example: on pinvoke.net I see

 PROC_THREAD_ATTRIBUTE_PARENT_PROCESS = 0x00020000,
 PROC_THREAD_ATTRIBUTE_HANDLE_LIST    = 0x00020002

  with terms of use which grant us complete freedom to use this information, subject to us accepting any liabilities for any consequence of such use.

I can suggest this: https://github.com/mozilla/positron/blob/master/security/sandbox/chromium-shim/base/win/sdkdecls.h It is under the Mozilla license.

Reply To eliz

I can suggest this: https://github.com/mozilla/positron/blob/master/security/sandbox/chromium-shim/base/win/sdkdecls.h It is under the Mozilla license.

Yes, I stumbled upon that myself. It all looks kind of back-to-front, to me; e.g., predicated on #if _WIN32_WINNT < 0x0602, (i.e. < _WIN32_WINNT_VISTA), it sets about defining features which are unsupported prior to Vista. Furthermore, apart from the mitigation policy properties suitable for use from Win8 onwards, it doesn't appear to define anything for which I haven't found a more useful reference, and it has what appears to be an unnecessarily convoluted mechanism for defining properties such as PROC_THREAD_ATTRIBUTE_HANDLE_LIST ... possibly inspired by a study of Microsoft's headers, although, since I have not examined (and will not examine) those myself, I cannot be certain.

FWIW, within this golang code review, I see

  const (
 +       // attributes for ProcThreadAttributeList
 +       PROC_THREAD_ATTRIBUTE_PARENT_PROCESS    = 0x00020000
 +       PROC_THREAD_ATTRIBUTE_HANDLE_LIST       = 0x00020002
 +       PROC_THREAD_ATTRIBUTE_GROUP_AFFINITY    = 0x00030003
 +       PROC_THREAD_ATTRIBUTE_PREFERRED_NODE    = 0x00020004
 +       PROC_THREAD_ATTRIBUTE_IDEAL_PROCESSOR   = 0x00030005
 +       PROC_THREAD_ATTRIBUTE_MITIGATION_POLICY = 0x00020007
 +       PROC_THREAD_ATTRIBUTE_UMS_THREAD        = 0x00030006
 +       PROC_THREAD_ATTRIBUTE_PROTECTION_LEVEL  = 0x0002000b
 +)

Reply To keith

Reply To eliz

I can suggest this: https://github.com/mozilla/positron/blob/master/security/sandbox/chromium-shim/base/win/sdkdecls.h It is under the Mozilla license.

Yes, I stumbled upon that myself. It all looks kind of back-to-front, to me; e.g., predicated on #if _WIN32_WINNT < 0x0602, (i.e. < _WIN32_WINNT_VISTA), it sets about defining features which are unsupported prior to Vista.

I think this is because they want to be able to compile the sources on versions of Windows older than Vista, and then detect the actual support at run time.

We don't need to copy the conditions, just the values, of course.

FWIW, within this golang code review, I see {{{ code diffu const ( + // attributes for ProcThreadAttributeList + PROC_THREAD_ATTRIBUTE_PARENT_PROCESS = 0x00020000 + PROC_THREAD_ATTRIBUTE_HANDLE_LIST = 0x00020002 + PROC_THREAD_ATTRIBUTE_GROUP_AFFINITY = 0x00030003 + PROC_THREAD_ATTRIBUTE_PREFERRED_NODE = 0x00020004 + PROC_THREAD_ATTRIBUTE_IDEAL_PROCESSOR = 0x00030005 + PROC_THREAD_ATTRIBUTE_MITIGATION_POLICY = 0x00020007 + PROC_THREAD_ATTRIBUTE_UMS_THREAD = 0x00030006 + PROC_THREAD_ATTRIBUTE_PROTECTION_LEVEL = 0x0002000b +) }}} and can see no documented restrictions, which would prohibit us from using this as a reference source.

This doesn't include PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, though, which was the trigger for this ticket. I guess you will take its value from other sources?

Reply To eliz

Reply To keith

Reply To eliz

I can suggest this: https://github.com/mozilla/positron/blob/master/security/sandbox/chromium-shim/base/win/sdkdecls.h It is under the Mozilla license.

Yes, I stumbled upon that myself. It all looks kind of back-to-front, to me; e.g., predicated on #if _WIN32_WINNT < 0x0602, (i.e. < _WIN32_WINNT_VISTA), it sets about defining features which are unsupported prior to Vista.

I think this is because they want to be able to compile the sources on versions of Windows older than Vista, and then detect the actual support at run time.

That may be so, but it is a degenerate may of going about it. The normal approach would be to #define _WIN32_WINNT, or #define NTDDI_VERSION, to expose the appropriate APIs, but still to perform checks, at runtime, for actual availability of those APIs. (It is entirely at the developer's discretion, to define _WIN32_WINNT or NTDDI_VERSION appropriately; why would you choose a value which is not appropriate to the APIs you wish to employ?)

We don't need to copy the conditions, just the values, of course.

Sure.

FWIW, within this golang code review, I see

  const (
 +       // attributes for ProcThreadAttributeList
 +       PROC_THREAD_ATTRIBUTE_PARENT_PROCESS    = 0x00020000
 +       PROC_THREAD_ATTRIBUTE_HANDLE_LIST       = 0x00020002
 +       PROC_THREAD_ATTRIBUTE_GROUP_AFFINITY    = 0x00030003
 +       PROC_THREAD_ATTRIBUTE_PREFERRED_NODE    = 0x00020004
 +       PROC_THREAD_ATTRIBUTE_IDEAL_PROCESSOR   = 0x00030005
 +       PROC_THREAD_ATTRIBUTE_MITIGATION_POLICY = 0x00020007
 +       PROC_THREAD_ATTRIBUTE_UMS_THREAD        = 0x00030006
 +       PROC_THREAD_ATTRIBUTE_PROTECTION_LEVEL  = 0x0002000b
 +)

  and can see no documented restrictions, which would prohibit us from using this as a reference source.

This doesn't include PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, though,

To be fair, neither does the reference you suggested.

which was the trigger for this ticket. I guess you will take its value from other sources?

Yes. I already have an MIT licensed reference for it, as noted in a prior comment.

Reply To eliz

To compile programs that use the Pseudo Console feature introduced recently with Windows 10, there's a need in several additions to the MinGW w32api headers and import libraries.

I think I've now added sufficient content to my local working copies of <sdkddkver.h>, <winbase.h>, and <wincon.h>, to achieve the pseudo-console support objective, but there is a potential problem with simply adding the import library thunks, for the missing functions. It's not insurmountable — in fact, it's easy enough to do — and perhaps is more of a caveat for users, but it does kind of make a mockery of the shenanigans exhibited in the MPL code, to which you previously referred me.

Fundamentally, the caveat is that, once the header files have been updated, and the import library thunks have been added, any code which is compiled with an appropriate NTDDI_VERSION definition may call the new functions directly, and, in the absence of coding errors, the program will compile, and link successfully. However, it will not load, much less run, on any legacy version of Windows, (which, in the case of the pseudo-console support, is anything pre-dating Win10-RS5); it will crash, at load time, with a "kernel32.dll entry point not found" exception, before the main() function even starts to run, and thus before it has any chance to offer a hint that it requires a newer version of Windows!

Now, I'm sure that you, Eli, are already aware of this limitation, and there is already precedent for such pitfalls, in MinGW import libraries, but I thought it worthy of mention, for the benefit of (possibly) less enlightened users. The defensive way, to manage this caveat, is to always call potentially unsupported functions via a pointer retrieved by GetProcAddress(), and never call them directly; in this way, the application can ensure that GetProcAddress() returns a valid pointer, or can provide a graceful fallback action, if NULL is returned. I've written quite a few such "legacy capable" functions, in the past year or so; they all look very similar, and I'm seriously considering factoring out the commonality, into generic helper functions, (perhaps in libkernel32.a), to make the task less repetitive, but that's probably a subject for another feature request ticket.

Reply To keith

Reply To eliz
Fundamentally, the caveat is that, once the header files have been updated, and the import library thunks have been added, any code which is compiled with an appropriate NTDDI_VERSION definition may call the new functions directly, and, in the absence of coding errors, the program will compile, and link successfully. However, it will not load, much less run, on any legacy version of Windows, (which, in the case of the pseudo-console support, is anything pre-dating Win10-RS5); it will crash, at load time, with a "kernel32.dll entry point not found" exception, before the main() function even starts to run, and thus before it has any chance to offer a hint that it requires a newer version of Windows!

Now, I'm sure that you, Eli, are already aware of this limitation, and there is already precedent for such pitfalls, in MinGW import libraries, but I thought it worthy of mention, for the benefit of (possibly) less enlightened users. The defensive way, to manage this caveat, is to always call potentially unsupported functions via a pointer retrieved by GetProcAddress(), and never call them directly; in this way, the application can ensure that GetProcAddress() returns a valid pointer, or can provide a graceful fallback action, if NULL is returned. I've written quite a few such "legacy capable" functions, in the past year or so; they all look very similar, and I'm seriously considering factoring out the commonality, into generic helper functions, (perhaps in libkernel32.a), to make the task less repetitive, but that's probably a subject for another feature request ticket.

Yes, indeed. Any program that wants to use APIs only available on some modern versions of Windows must use GetProcAddress to test for the availability, and then call the API through a pointer populated by it. For example, GNU Emacs, when built for MS-Windows, uses this technique all over the place, because (when compiled by MinGW -- NOT MinGW64, not even their 32-bit variant!) it still supports every Windows version from Windows 98 onwards.

I've attached my current W32API patch, to address this issue; please check if it meets your requirements.

Note that, to expose the pseudo-console API, you will need to #define NTDDI_VERSION NTDDI_WIN10_RS5, (which, with our <sdkddkver.h> should also implicitly #define _WIN32_WINNT _WIN32_WINNT_WIN10; if not, you will also need to define this explicitly).

Further note that, while I've also added a significant swathe of ancillary manifest constant definitions, gleaned from Microsoft's UpdateProcThreadAttribute() documentation, I've omitted some information which may be found there; I found that page to be confusing, w.r.t. which version of Windows was required to support each attribute, and I've omitted those about which I was uncertain.