Error `git: command not found` on GitHub-hosted arm64 runner that got GfW installed at runtime

[dennisameling]

While working on dennisameling-org/git-for-windows-automation#1, I ran into a weird issue where the setup-git-for-windows-sdk action didn't work, even though I manually installed it in an earlier step (the arm64 image doesn't have GfW installed yet).

I've created a consistent reproducer here, which shows some interesting behavior:

• The first attempt to use setup-git-for-windows-sdk fails with Error: spawn C:/Program Files/Git/cmd/git.exe ENOENT
• I then install Git for Windows and add it to the path
• In the second attempt, it actually uses the Git executable to clone git-sdk-arm64 and build-extra. However, Creating build-installers artifact still fails with .tmp/build-extra/please.sh: line 106: git: command not found.

What I did so far to debug this issue

• In Debug Git not found issue dennisameling/setup-git-for-windows-sdk#3, I added some logging to better understand the environment variables that get set by the NodeJS process when it invokes bash.exe from the C:/Program Files/Git/usr/bin folder (gitForWindowsUsrBinPath). Here you can see the PATH value that NodeJS passes to the bash.exe process. Bash itself, however, reports this, where it seems like C:/Program Files/Git/usr/bin got replaced with simply /usr/bin (I assume this is expected behavior). This folder doesn't seem to contain the git.exe executable.
• I have a strong feeling that this issue hasn't popped up before because both GitHub-hosted runners and our self-hosted runners have Git for Windows pre-installed, before (!!!) the GitHub runner agent starts and loads the environment variables from the OS. I added a which git line, which resulted in the following:
  • GitHub-hosted arm64 runner (without Git preinstalled, but installed it in the pipeline itself): which: no git in (/usr/bin:/bin:/.....)
  • GitHub-hosted x64 runner (with GfW preinstalled): /cmd/git (so it somehow found Git in the /cmd folder, even though it doesn't show in the path??)

What I did to work around the issue

Here, I just explicitly added PATH="/c/Program Files/Git/bin:$PATH" to please.sh, so that it could find something to work with.

Noe that this issue will probably disappear as soon as GfW gets preinstalled on GitHub-hosted ARM64 runners, but I think we could make the setup-git-for-windows-sdk action more resilient to this type of issue moving forward.

I'm happy to implement a fix if you could point me in the right direction. Thanks!

[rimrul]

Bash itself, however, reports this, where it seems like C:/Program Files/Git/usr/bin got replaced with simply /usr/bin (I assume this is expected behavior).

It is expected.

This folder doesn't seem to contain the git.exe executable.

That is expected, too. /usr/bin/git.exe would be MSYS2 Git. We're looking for /cmd/git.exe or /mingw64/git.exe.

GitHub-hosted x64 runner (with GfW preinstalled): /cmd/git (so it somehow found Git in the /cmd folder, even though it doesn't show in the path??)

/cmd is on the PATH there.

[…]Program Files/nodejs:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/GitHub CLI[…]

I have a strong feeling that this issue hasn't popped up before because both GitHub-hosted runners and our self-hosted runners have Git for Windows pre-installed, before (!!!) the GitHub runner agent starts and loads the environment variables from the OS.

I think this feeling is correct.

In the second attempt, it actually uses the Git executable to clone git-sdk-arm64 and build-extra. However, Creating build-installers artifact still fails with .tmp/build-extra/please.sh: line 106: git: command not found.

The cloning steps call the git.exe they want with an absolute path, but the please.sh step just calls git.

[dscho]

Maybe this code is the culprit for the behavior where /cmd/git.exe is found but Git can then not find its commands?

But that does not really match the error message, which looks more as if /cmd/git.exe does not find /clangarm64/bin/git.exe (and then let that do the actual work), even though it should find it.

[rimrul]

Maybe this code is the culprit for the behavior where /cmd/git.exe is found

I think it's this

but Git can then not find its commands?

But that does not really match the error message, which looks more as if /cmd/git.exe does not find /clangarm64/bin/git.exe

I think it's just bash doesn't have /cmd or /clangarm64/bin or /mingw64/bin on the path when calling please.sh

[dscho]

looks more as if /cmd/git.exe does not find /clangarm64/bin/git.exe

I think it's just bash doesn't have /cmd or /clangarm64/bin or /mingw64/bin on the path when calling please.sh

Good point. It happens to work on hosted runners because git.exe is already in the PATH there.

So we'd need to change this code around from someting like:

export const gitForWindowsUsrBinPath = `${gitRoot}/usr/bin`
[...]
              PATH: `${gitForWindowsUsrBinPath}${delimiter}${process.env.PATH}`

to something like:

export const gitForWindowsBinPaths =
  ['clangarm64', 'mingw64', 'mingw32', 'usr'].map(p => `${gitRoot}/${p}/bin`)
[...]
              PATH: `${gitForWindowsBinPaths.join(delimiter)}${delimiter}${process.env.PATH}`

I pushed something along those lines into my fork as fix-on-arm64 branch.

@dennisameling could you please try running this (switching out uses: dscho/setup-git-for-windows-sdk@fix-on-arm64 for uses: git-for-windows/setup-git-for-windows-sdk@v1 should do the trick)? This should even run without installing Git for Windows, as it should simply use the runner's own MinGit in that case.

[dennisameling]

@dennisameling could you please try running this (switching out uses: dscho/setup-git-for-windows-sdk@fix-on-arm64 for uses: git-for-windows/setup-git-for-windows-sdk@v1 should do the trick)?

That worked, thank you!!

This should even run without installing Git for Windows, as it should simply use the runner's own MinGit in that case.

It doesn't, because the cloning git-sdk-arm64 step still expects C:/Program Files/Git/cmd/git.exe to be present. But I'd say that's outside the scope of this issue.

I was working on a similar fix, and ended up extracting the spawn logic into a wrapper and adding tests. Curious to hear if you think that'd still add value to this repo.

[dscho]

It doesn't, because the cloning git-sdk-arm64 step still expects C:/Program Files/Git/cmd/git.exe to be present.

Hmm. But that clone() call implicitly spawns git.exe at gitExePath, which has previously been set to ${gitRoot}/cmd/git.exe, and that gitRoot should find the agent runner's MinGit. Does that not exist?

[dennisameling]

Does that not exist?

It doesn't, indeed. By default, the GitHub Actions Runner doesn't seem to have any flavor of Git pre-installed. It only seems to have NodeJS installed. Having Git pre-installed only seems to be the case on the default GitHub-hosted runners, like windows-2019 and windows-2022.

Even the AGENT_HOMEDIRECTORY environment variable doesn't exist at runtime (docs). Isn't that a leftover from the Azure Pipelines days?

[dscho]

Does that not exist?

It doesn't, indeed. By default, the GitHub Actions Runner doesn't seem to have any flavor of Git pre-installed.

Huh. I thought it needed Git to run actions/checkout and the likes?

Even the AGENT_HOMEDIRECTORY environment variable doesn't exist at runtime (docs).
Isn't that a leftover from the Azure Pipelines days?

Hmm. Possibly. Maybe RUNNER_TOOL_CACHE has it?

[dennisameling]

Huh. I thought it needed Git to run actions/checkout and the likes?

Apparently not, as can be seen here:

The repository will be downloaded using the GitHub REST API
To create a local Git repository instead, add Git 2.18 or higher to the PATH

It falls back to the GitHub REST API if Git isn't installed. However, in my earlier testing, that did cause issues later down the line:

fatal: not a git repository (or any of the parent directories): .git

Hmm. Possibly. Maybe RUNNER_TOOL_CACHE has it?

Negative. Looks like that folder doesn't even exist on a fresh installation of the Runner. So it looks like GfW really doesn't come bundled with the runner itself, only NodeJS.