feat: enable multi-arch builds with ARM compatibility

[rhdedgar]

Setup multi-arch builds with a manifest that will automatically point to the image of supported architectures.

On Kubernetes, with these changes, images are still referred to in the same manner, but will resolve to the architecture that is relevant to the node where it is to be run.

                                                  --> quay.io/<user>/llama-stack-k8s-operator/<version>-linux-arm64
quay.io/<user>/llama-stack-k8s-operator/<version>-|
                                                  --> quay.io/<user>/llama-stack-k8s-operator/<version>-linux-amd64

This should allow users to seamlessly transition between clusters with different architectures, once the distribution changes are also in place.

Closes: RHAIENG-1941

[derekhiggins]

How long does a arm64 image build take? Building an arm64 image with podman locally has taken over an hour and is still running, are we going to hit timeouts in CI ? (or could it be a local setup problem)

[derekhiggins]

For podman, --platform automatically sets BUILDPLATFORM and TARGETPLATFORM

If podman is setting BUILDPLATFORM do we still need to hard code it here (same question above in podman-buildx-multiarch)?

[rhdedgar]

I tried a few more builds and was able to replicate some transient issues with the arm64 builds. I've narrowed it down to the inclusion of CGO_ENABLED=0 (FIPS related) on the build. The builds were either taking 5-10 minutes, or getting stuck entirely.

It doesn't happen with amd64 builds, and I have finally settled on an arm64 workaround that I still need to test further. I'll see if I can get access to some FIPS-enabled ARM clusters to test on.

[rhdedgar]

Cool, that did improve things quite a bit. I now see all the PR tests pass within a reasonable amount of time. It looks like the longest one was the e2e tests, at 9m. I'll see if the ARM-specific CGO nuances can reflected in downstream documentation.

[mfleader]

We only want to build the image here as a part of the validation process, but not push it. From my understanding of the image-buildx rule that it also pushes the image. Would you create and use a Makefile rule here that encapsulates only the image building logic?

[VaishnaviHire]

@mergify rebase

[mergify]

rebase

✅ Branch has been successfully rebased

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @rhdedgar please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork