Mailing List Archive

What version of visual studio are you using? What version of python? I’ve had success with using the cython package in python and cl from visual studio, but I haven’t tried visual studio alone.

Sent from my iPhone

> On Jun 13, 2023, at 11:59 AM, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>
> Fehler beim Buildvorgang

--
https://mail.python.org/mailman/listinfo/python-list

Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz:
> What version of visual studio are you using?

Visual Studio 2022, aka 17.6.2.

> What version of python?

python 3.10.11 or 3.11.4

> I’ve had success with using the cython package in python and cl from visual studio, but I haven’t tried visual studio alone.

Same problem at the same place: directory "../modules/..." not found,
Renaming it from "Modules" to "modules" it is found, but then fails to
find "Modules".

Looks like it awaits, compiling in Windows an filesystem only case
aware, not case sensitive -- I'm assuming this a bug now. Building
within cygwin (or MSYS, Ubuntu) this works as expected. But there it
does not search for "modules" once and "Modules" at an other place.

>> On Jun 13, 2023, at 11:59 AM, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>>
>> Fehler beim Buildvorgang
--
Thomas

On 6/13/23 12:12, Thomas Schweikle via Python-list wrote:
>
>
> Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz:
>> What version of visual studio are you using?
>
> Visual Studio 2022, aka 17.6.2.
>
>> What version of python?
>
> python 3.10.11 or 3.11.4
>
>> I’ve had success with using the cython package in python and cl from
>> visual studio, but I haven’t tried visual studio alone.
>
> Same problem at the same place: directory "../modules/..." not found,
> Renaming it from "Modules" to "modules" it is found, but then fails to
> find "Modules".
>
> Looks like it awaits, compiling in Windows an filesystem only case
> aware, not case sensitive -- I'm assuming this a bug now. Building
> within cygwin (or MSYS, Ubuntu) this works as expected. But there it
> does not search for "modules" once and "Modules" at an other place.

I just did this build the other day for the first time even from a git
checkout (so VS22, and not a versioned release but top of main branch),
and there was no such problem - did you follow the instructions at
https://devguide.python.org/getting-started/setup-building/index.html?


--
https://mail.python.org/mailman/listinfo/python-list

One expert told me to do the following when compiling via cython and cl:

cython -3 --embed -o c_file_name python_file_name

Then, assuming python is installed in your apps directory and not your program files directory:

set "PYTHON_DIR=%LocalAppData%\Programs\Python\Python311"

or whatever directory you put python in.

cl /O2 /I"%PYTHON_DIR%\Include" c_file_name /link /libpath:"%PYTHON_DIR%\libs"

If that doesn't work, that's all I have. Sorry.

-----Original Message-----
From: Python-list <python-list-bounces+jschwar=sbcglobal.net@python.org> On Behalf Of Thomas Schweikle via Python-list
Sent: Tuesday, June 13, 2023 1:12 PM
To: Python <python-list@python.org>
Cc: Thomas Schweikle <tschweikle@bfs.de>
Subject: Re: Compiling python on windows with vs



Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz:
> What version of visual studio are you using?

Visual Studio 2022, aka 17.6.2.

> What version of python?

python 3.10.11 or 3.11.4

> I’ve had success with using the cython package in python and cl from visual studio, but I haven’t tried visual studio alone.

Same problem at the same place: directory "../modules/..." not found, Renaming it from "Modules" to "modules" it is found, but then fails to find "Modules".

Looks like it awaits, compiling in Windows an filesystem only case aware, not case sensitive -- I'm assuming this a bug now. Building within cygwin (or MSYS, Ubuntu) this works as expected. But there it does not search for "modules" once and "Modules" at an other place.

>> On Jun 13, 2023, at 11:59 AM, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>>
>> Fehler beim Buildvorgang
--
Thomas


--
https://mail.python.org/mailman/listinfo/python-list

On 6/13/23, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>
> Since Git enables Windows NTFS case sensitivity while checking out
> sources ... is it a bug or a "feature"? And: is there a simple

AFAIK the Windows version of Git (you're not using the Linux version
of Git via WSL, right?) does not automatically enable NTFS case
sensitivity. But a newly created directory does inherit the case
sensitivity of its parent directory. Make sure to clone the CPython
repo in a directory that has case sensitivity disabled.

> _testconsole.c
> C:\Users\sct-muc\Documents\Projekte\cpython\PC\_testconsole.c(13,10):
> fatal error C1083: Datei (Include) kann nicht geöffnet werde
> n: "..\modules\_io\_iomodule.h": No such file or directory
> [C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\_testconsole.vcxpro
> j]

I just built the main branch in a case sensitive tree. I had no
problem building "_testconsole.c". However, building the _decimal
extension module raised a couple of serious warnings. In
"PCbuild/_decimal.vcxproj", there's an include for
"..\Modules\_decimal\libmpdec\vcdiv64.asm". However, MSBuild resolved
this relative path with all lower-case names, i.e. "modules" instead
of the correct name "Modules", and it incorrectly tried to output
"vcdiv64.obj" in a subdirectory of "pcbuild" instead of the correct
name "PCbuild". This appears to be a bug in MSBuild. A lot of Windows
programs don't handle case-sensitive directories well, including
Python's standard library. It's understandable when comparing paths,
but the behavior in this case is inexcusably bad.
--
https://mail.python.org/mailman/listinfo/python-list

Am Di., 13.Juni.2023 um 20:36:17 schrieb Mats Wichmann via Python-list:
> On 6/13/23 12:12, Thomas Schweikle via Python-list wrote:
>>
>>
>> Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz:
>>> What version of visual studio are you using?
>>
>> Visual Studio 2022, aka 17.6.2.
>>
>>> What version of python?
>>
>> python 3.10.11 or 3.11.4
>>
>>> I’ve had success with using the cython package in python and cl from
>>> visual studio, but I haven’t tried visual studio alone.
>>
>> Same problem at the same place: directory "../modules/..." not found,
>> Renaming it from "Modules" to "modules" it is found, but then fails to
>> find "Modules".
>>
>> Looks like it awaits, compiling in Windows an filesystem only case
>> aware, not case sensitive -- I'm assuming this a bug now. Building
>> within cygwin (or MSYS, Ubuntu) this works as expected. But there it
>> does not search for "modules" once and "Modules" at an other place.
>
> I just did this build the other day for the first time even from a git
> checkout (so VS22, and not a versioned release but top of main branch),
> and there was no such problem - did you follow the instructions at
> https://devguide.python.org/getting-started/setup-building/index.html?

Yes.
Had git installed (2.41.0.windows.1), did "git clone
https://github.com/python/cpython.git", then checked out the branch I
wanted: "git checkout 3.10.12" (or 3.11.4, 3.12.0b2).

Python was already installed and in path:
"python --version" gives back: "Python 3.11.4"

Changed into cpython directory, then called: ".\PCbuild\build.bat"
(".\pcbuild\build.bat" will, within cmd.exe, throw an error: File not
found -- remember directory cpython and all beyond are case sensitive,
not only case aware -- git sets case sensitiveness for checked out
directories).

It compiles until it does not find "modules", because the directory is
named "Modules" (all posix file systems behave this way and NTFS can be
forced to behave this way too. It is easy and any user can do it:

"fsutil.exe file SetCaseSensitiveInfo <path> enable"

You may query this with:

"fsutil.exe file queryCaseSensitiveInfo <path>"

In my case:
~/Documents/Projekte/cpython> fsutil.exe file queryCaseSensitiveInfo .
Das Attribut für Groß-/Kleinschreibung für das Verzeichnis ^
"C:\Users\user\Documents\Projekte\cpython" ist aktiviert.

Or: the directory we're in is case sensitive and distinguishes between
"Modules", "modules" and "MODULES" and you can have them all:

C:\Users\user\Documents\Projekte\cpython>dir
Datenträger in Laufwerk C: ist Windows
Volumeseriennummer: BC5E-F466

Verzeichnis von C:\Users\user\Documents\Projekte\cpython

Di, 13.Jun.2023 21:23 <DIR> .
Di, 13.Jun.2023 21:23 <DIR> ..
[...]
Di, 13.Jun.2023 21:23 <DIR> MODULES
Mo, 12.Jun.2023 22:13 <DIR> Modules
Di, 13.Jun.2023 21:23 <DIR> modules
[...]
16 Datei(en), 1.320.340 Bytes
21 Verzeichnis(se), 134.699.622.400 Bytes frei

Since case sensitivity is necessary for a variety of projects normally
only used with *nix-OS I've configured git to create directories with
case sensitivity set enabled (its default meanwhile if git is installed
on windows).
--
Thomas

> Since Git enables Windows NTFS case sensitivity while checking out sources

I didn't know that. Would you give us a link to this feature?
As far as I know, `git config core.ignorecase` doesn't mean NTFS case
sensitive.

On Wed, Jun 14, 2023 at 1:57?AM Thomas Schweikle via Python-list <
python-list@python.org> wrote:

> Hi!
>
> Trying to compile python on windows leads to following error:
>
> _testimportmultiple.vcxproj ->
>
> C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\amd64\_testimportmultiple.pyd
> _testmultiphase.c
> Bibliothek
> "C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.lib"
>
> und Objekt "C:\Users\sct-muc\Docume
> nts\Projekte\cpython\PCbuild\amd64\_testmultiphase.exp" werden erstellt.
> Code wird generiert.
> Codegenerierung ist abgeschlossen.
> _testmultiphase.vcxproj ->
>
> C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.pyd
> _testconsole.c
> C:\Users\sct-muc\Documents\Projekte\cpython\PC\_testconsole.c(13,10):
> fatal error C1083: Datei (Include) kann nicht geöffnet werde
> n: "..\modules\_io\_iomodule.h": No such file or directory
> [C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\_testconsole.vcxpro
> j]
>
> Fehler beim Buildvorgang.
>
> if I rename "Modules" to "modules" it will find
> "..\modules\_io\_iomodule.h" but wont find "..\Modules\..."
>
> Since Git enables Windows NTFS case sensitivity while checking out
> sources ... is it a bug or a "feature"? And: is there a simple
> workaround available besides disabling case sensitivity (which will
> break others)?
> --
> Thomas
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


--
Inada Naoki <songofacandy@gmail.com>
--
https://mail.python.org/mailman/listinfo/python-list

On 6/14/23, Inada Naoki via Python-list <python-list@python.org> wrote:
>> Since Git enables Windows NTFS case sensitivity while checking out sources
>
> I didn't know that. Would you give us a link to this feature?
> As far as I know, `git config core.ignorecase` doesn't mean NTFS case
> sensitive.

If a repo is cloned into a case-insensitive directory, then
core.ignorecase should be enabled automatically. If a repo is cloned
into a case-sensitive directory, then core.ignorecase should not be
enabled automatically.

I searched through relevant issues on the Git for Windows repo on
GitHub, and I found nothing to indicate that a capability to
automatically enable NTFS case sensitivity has been added. I searched
through the source of Git and Git for Windows, and I didn't find any
references to WinAPI SetFileInformationByHandle: FileCaseSensitiveInfo
or NTAPI NtSetInformationFile: FileCaseSensitiveInformation, nor the
use of fsutil file setCaseSensitiveInfo.
--
https://mail.python.org/mailman/listinfo/python-list

Am Mi., 14.Juni.2023 um 15:10:50 schrieb Eryk Sun:
> On 6/14/23, Inada Naoki via Python-list <python-list@python.org> wrote:
>>> Since Git enables Windows NTFS case sensitivity while checking out sources
>>
>> I didn't know that. Would you give us a link to this feature?
>> As far as I know, `git config core.ignorecase` doesn't mean NTFS case
>> sensitive.
>
> If a repo is cloned into a case-insensitive directory, then
> core.ignorecase should be enabled automatically. If a repo is cloned
> into a case-sensitive directory, then core.ignorecase should not be
> enabled automatically.
>
> I searched through relevant issues on the Git for Windows repo on
> GitHub, and I found nothing to indicate that a capability to
> automatically enable NTFS case sensitivity has been added. I searched
> through the source of Git and Git for Windows, and I didn't find any
> references to WinAPI SetFileInformationByHandle: FileCaseSensitiveInfo
> or NTAPI NtSetInformationFile: FileCaseSensitiveInformation, nor the
> use of fsutil file setCaseSensitiveInfo.

In this case: not sure what is going on. In a git-shell on Windows:
If caseSensitiveInfo is disabled:
user@host MINGW64 ~/K
$ fsutil file queryCaseSensitiveInfo .
Das Attribut für Groß-/Kleinschreibung für das Verzeichnis
"C:\Users\user\K" ist deaktiviert.

$ git config --global -l
core.ignorecase=true

$ git clone https://github.com/python/cpython.git
Cloning into 'cpython'...
remote: Enumerating objects: 956870, done.
remote: Counting objects: 100% (1304/1304), done.
remote: Compressing objects: 100% (801/801), done.
Receiving objects: 100% (956870/956870), 557.02 MiB | 9.75 MiB/s, done.55566

Resolving deltas: 100% (760802/760802), done.
Updating files: 100% (4488/4488), done.

$ find . -type d -exec fsutil.exe file queryCaseSensitiveInfo {} \;
does not show any directory having caseSensitiveInfo enabled.

If caseSesitiveInfo is enabled:
user@host MINGW64 ~/K
$ fsutil file queryCaseSensitiveInfo .
Das Attribut für Groß-/Kleinschreibung für das Verzeichnis
"C:\Users\user\K" ist aktiviert.

$ git config --global -l
core.ignorecase=true

$ git clone https://github.com/python/cpython.git
Cloning into 'cpython'...
remote: Enumerating objects: 956870, done.
remote: Counting objects: 100% (1304/1304), done.
remote: Compressing objects: 100% (801/801), done.
Receiving objects: 100% (956870/956870), 557.02 MiB | 9.75 MiB/s, done.55566

Resolving deltas: 100% (760802/760802), done.
Updating files: 100% (4488/4488), done.

$ find . -type d -exec fsutil.exe file queryCaseSensitiveInfo {} \;
All directories created by git have caseSensitiveInfo enabled.

core.ignorecase is not regarded in any way. It does not mater if it is
set or not.

--
Thomas

Then, git doesn't enable Windows NTFS case sensitivity.

You enabled NTFS case sensitivity on "C:\Users\user\K".
And Windows enabled case sensitivity for all new directories under the
directory.

Since it is not default and minor setting, it is not a bug that
current Python doesn't support building on case sensitive directory.
But I think it is a nice improvement if next Python supports it.

Regards,

--
Inada Naoki <songofacandy@gmail.com>
--
https://mail.python.org/mailman/listinfo/python-list

On 6/15/23, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>
> In this case: not sure what is going on.

Possibly you have a setting configured that affects the behavior of
Git via the MinGW-w64 runtime, such that calling mkdir() ends up
calling NtSetInformationFile() to set the FileCaseSensitiveInformation
for the directory.

Does the mkdir command in Git bash create a case-sensitive directory?
It doesn't for me. I have to manually enable case sensitivity via
`chattr +C`.

What do you get for `which git` and `git --version`?

$ which git
/mingw64/bin/git

$ git --version
git version 2.41.0.windows.1

> $ fsutil file queryCaseSensitiveInfo .

The MSYS2 environment includes lsattr and chattr commands, with the
case-sensitive flag mapped to "C". It's probably more convenient than
typing `fsutil file queryCaseSensitiveInfo` or `fsutil file
setCaseSensitiveInfo`.

$ lsattr -d test
------------ test
$ chattr +C test
$ lsattr -d test
-----------C test

> core.ignorecase is not regarded in any way. It does not mater if it is
> set or not.

Git tests the case-sensitivity of the target directory to configure
core.ignorecase when cloning a repo. If it's case insensitive, then
core.ignorecase is enabled. This overrides the global value. AFAIK,
the ignorecase setting is unrelated to actually setting the case
sensitivity of created directories; it just affects how Git behaves on
a case-insensitive filesystem.
--
https://mail.python.org/mailman/listinfo/python-list

Am Do., 15.Juni.2023 um 15:44:42 schrieb Inada Naoki:
> Then, git doesn't enable Windows NTFS case sensitivity.
>
> You enabled NTFS case sensitivity on "C:\Users\user\K".
> And Windows enabled case sensitivity for all new directories under the
> directory.

No. This flag is not inherited. Someone has to set it for created
directories. It is easy to confirm: take a directory not under MSYS or
cygwin control (because it is mounted by MSYS or cygwin), set the flag,
then create directories. They all will have caseSensitivInfo disabled.

> Since it is not default and minor setting, it is not a bug that
> current Python doesn't support building on case sensitive directory.
> But I think it is a nice improvement if next Python supports it.

Python itself isn't the problem here. It is MSBuild.exe. For some reason
this tool lowercases sometimes whole paths to files included. This does
not matter if case sensitivity is disabled. It matters if case
sensitivity is enabled! There is no reason MSBUild.exe does it. But it
is done for some paths (as someone else pointed out).

>
> Regards,
>
> --
> Inada Naoki <songofacandy@gmail.com>

--
Thomas

Am Do., 15.Juni.2023 um 16:28:21 schrieb Eryk Sun:
> On 6/15/23, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>>
>> In this case: not sure what is going on.
>
> Possibly you have a setting configured that affects the behavior of
> Git via the MinGW-w64 runtime, such that calling mkdir() ends up
> calling NtSetInformationFile() to set the FileCaseSensitiveInformation
> for the directory.

It is a mount option for MSYS or cygwin controlled parts of the file
system. posix=1 switches case sensitivity on, posix=0 switches it off.

MSYS or cygwin are then make caseSensitiveInfo "inheritated" by setting
this flag for all newly created directories. This is a feature of the
latest cygwin-dll. It might not be seen on older variants.

> Does the mkdir command in Git bash create a case-sensitive directory?
> It doesn't for me. I have to manually enable case sensitivity via
> `chattr +C`.
>
> What do you get for `which git` and `git --version`?
>
> $ which git
> /mingw64/bin/git
>
> $ git --version
> git version 2.41.0.windows.1

the same:
$ which git
/mingw64/bin/git

$ git --version
git version 2.41.0.windows.1

And in cmd.exe:
+>where git
C:\Program Files\Git\bin\git.exe

+>git --version
git version 2.41.0.windows.1

>> $ fsutil file queryCaseSensitiveInfo .
>
> The MSYS2 environment includes lsattr and chattr commands, with the
> case-sensitive flag mapped to "C". It's probably more convenient than
> typing `fsutil file queryCaseSensitiveInfo` or `fsutil file
> setCaseSensitiveInfo`.
>
> $ lsattr -d test
> ------------ test
> $ chattr +C test
> $ lsattr -d test
> -----------C test

True. But if you frequently change between environments fsutil is the
command working in cmd.exe as in git-shell.

>> core.ignorecase is not regarded in any way. It does not mater if it is
>> set or not.
>
> Git tests the case-sensitivity of the target directory to configure
> core.ignorecase when cloning a repo. If it's case insensitive, then
> core.ignorecase is enabled. This overrides the global value. AFAIK,
> the ignorecase setting is unrelated to actually setting the case
> sensitivity of created directories; it just affects how Git behaves on
> a case-insensitive filesystem.

--
Thomas

On 6/15/23, Thomas Schweikle via Python-list <python-list@python.org> wrote:
>
> No. This flag is not inherited. Someone has to set it for created
> directories. It is easy to confirm: take a directory not under MSYS or
> cygwin control (because it is mounted by MSYS or cygwin), set the flag,
> then create directories. They all will have caseSensitivInfo disabled.

That was how the attribute was implemented initially in Windows 10,
but subsequently it was made inheritable. For example:

C:\Temp\test>mkdir spam
C:\Temp\test>fsutil file setCaseSensitiveInfo spam enable
Error: Access is denied.

Setting the case-sensitive attribute requires the right to add files
and directories (i.e. "WD" = "write data" / "add file"; "AD" = "append
data" / "add subdirectory") and the right to remove files and
directories (i.e. "DC" = "delete child"). The owner of a directory
doesn't necessarily inherit these rights from the parent directory
(which is my case here), but the owner of any object usually has the
implicit right to modify discretionary security. Let's simply grant
the owner (i.e. "OW" = "owner rights") full control of the directory
(i.e. "F"), inheritable to child directories (i.e. "CI" = "container
inherit").

C:\Temp\test>icacls spam /grant *OW:(CI)(F)
processed file: spam
Successfully processed 1 files; Failed processing 0 files

C:\Temp\test>fsutil file setCaseSensitiveInfo spam enable
Case sensitive attribute on directory C:\Temp\test\spam is enabled.

Now, create a child directory and confirm that it inherits the
case-sensitive flag.

C:\Temp\test>mkdir spam\eggs
C:\Temp\test>fsutil file queryCaseSensitiveInfo spam\eggs
Case sensitive attribute on directory C:\Temp\test\spam\eggs is enabled.

> Python itself isn't the problem here. It is MSBuild.exe. For some reason
> this tool lowercases sometimes whole paths to files included. This does
> not matter if case sensitivity is disabled. It matters if case
> sensitivity is enabled! There is no reason MSBUild.exe does it. But it
> is done for some paths (as someone else pointed out).

For the specific problem you had when building 3.10 and 3.11, it's
actually a bug in Python's source code, which is no longer present in
3.12+. It can be fixed in 3.11, but 3.10 no longer gets bug fixes.
Here's the link to the issue on GitHub:

https://github.com/python/cpython/issues/105737

I encountered a different bug when building the main branch. Building
the _decimal extension module includes an .asm file. The relative path
in the project file has the correct case, but the build system
resolved the fully-qualified path as all lower case. This problem only
occurred for this single file, out of hundreds of relative paths in
the project files, so it's not like the build system is completely
broken when working in case-sensitive directories.

There are probably a few such bugs that need to be fixed in msbuild,
the compiler, and linker. After all, these tools have been developed
and tested for decades on only case-insensitive filesystems. But you
don't have to be on the bleeding edge. There's no reason to make
directories case-sensitive for repositories that are intended for use
on Windows, such as CPython.
--
https://mail.python.org/mailman/listinfo/python-list