option --copy-host-keys not working

[dearfl]

Prerequisites

• I have updated to the latest version using nix run --refresh github:nix-community/nixos-anywhere
• I have reproduced the issue with the --debug flag
• I have searched existing issues to make sure this isn't a duplicate

Bug Description

Sorry I don't have the time to take all these steps to reproduce, anyway, it seems when --copy-host-keys not working, at least in my case.

I'm triyng to install a minimal NixOS on a 512MiB memory vultr vps, naturally I use the minimal NixOS ISO as the installation medium. I need copy the host keys for bootstrap vaultix(secrets) setup, however when the installation finished, the host keys on VPS is not what on my local machine, thus vaultix failed to decrypt my secrets.

I belive this should be the relevant code, however I can't really understand it.

nixos-anywhere/src/nixos-anywhere.sh

Lines 894 to 905 in 25d23ef

	if [ ${copyHostKeys-n} = "y" ]; then
	# NB we copy host keys that are in turn copied by kexec installer.
	mkdir -m 755 -p /mnt/etc/ssh
	for p in /etc/ssh/ssh_host_*; do
	# Skip if the source file does not exist (i.e. glob did not match any files)
	# or the destination already exists (e.g. copied with --extra-files).
	if [ ! -e "\$p" ] || [ -e "/mnt/\$p" ]; then
	continue
	fi
	cp -a "\$p" "/mnt/\$p"
	done
	fi

Could it be only occur when using the minimal NixOS ISO? Since it not kexec?

Steps to Reproduce

I have my nixos config on https://github.com/dearfl/nyx, however I don't think it necessary to reproduce.

1. deploy a vultr vps with a minimal NixOS ISO attached.
2. install any nixosconfig with --copy-host-keys
3. after the installation finish, you should see ssh-keyscan return a public key different from your local machine.

Debug Logs

https://fars.ee/cMYe

Command Used

nix run nixpkgs#nixos-anywhere -- -f .#vultr --build-on local --copy-host-keys --no-disko-deps --target-host root@207.148.71.76

Target System

Other

NixOS Version

No response

Environment Information

• Host OS:
• Nix version:
• Target architecture:

Configuration Files

Additional Context

No response

[phozzy]

I can confirm that --copy-host-keys option is a little bit flaky. It may work one time, but won't work the next one.
It is a specially annoying since host ssh keys work nicely in pair with sops-nix! But now during each deployment, there is a 50/50 chance that I'll need to re-encrypt sops files.

[phozzy]

These logs in kexec stage:

+ kexec_extra_flags=
+ '[' 2 -gt 0 ']'
+ case "$1" in
+ kexec_extra_flags=
+ shift
+ shift
+ '[' 0 -gt 0 ']'
+ init=/nix/store/br4064v0fjpnp6kvg961wciban3ikh8a-nixos-system-nixos-installer-24.11pre-git/init
+ kernelParams='nouveau.modeset=0 console=tty0 console=ttyS0,115200 zswap.enabled=1 zswap.max_pool_percent=50 zswap.compressor=zstd zswap.zpool=zsmalloc root=fstab loglevel=4'
+++ readlink -f /root/kexec/kexec/run
++ dirname /root/kexec/kexec/run
+ SCRIPT_DIR=/root/kexec/kexec
++ TMPDIR=/root/kexec/kexec
++ mktemp -d
+ INITRD_TMP=/root/kexec/kexec/tmp.kvFzpsSg7N
+ cd /root/kexec/kexec/tmp.kvFzpsSg7N
+ trap cleanup EXIT
+ mkdir -p ssh
+ extractPubKeys /root
+ home=/root
+ for file in .ssh/authorized_keys .ssh/authorized_keys2
+ key=/root/.ssh/authorized_keys
+ test -e /root/.ssh/authorized_keys
+ grep -o '\(\(ssh\|ecdsa\|sk\)-[^ ]* .*\)' /root/.ssh/authorized_keys
+ for file in .ssh/authorized_keys .ssh/authorized_keys2
+ key=/root/.ssh/authorized_keys2
+ test -e /root/.ssh/authorized_keys2
+ test -n ''
+ test -n root
++ sh -c 'echo ~root'
+ sudo_home=/root
+ extractPubKeys /root
+ home=/root
+ for file in .ssh/authorized_keys .ssh/authorized_keys2
+ key=/root/.ssh/authorized_keys
+ test -e /root/.ssh/authorized_keys
+ grep -o '\(\(ssh\|ecdsa\|sk\)-[^ ]* .*\)' /root/.ssh/authorized_keys
+ for file in .ssh/authorized_keys .ssh/authorized_keys2
+ key=/root/.ssh/authorized_keys2
+ test -e /root/.ssh/authorized_keys2
+ test -e /etc/ssh/authorized_keys.d/root
+ cat /etc/ssh/authorized_keys.d/root
+ test -n root
+ test -e /etc/ssh/authorized_keys.d/root
+ cat /etc/ssh/authorized_keys.d/root
+ for p in /etc/ssh/ssh_host_*
+ test -e /etc/ssh/ssh_host_ed25519_key
+ cp -a /etc/ssh/ssh_host_ed25519_key ssh
+ for p in /etc/ssh/ssh_host_*
+ test -e /etc/ssh/ssh_host_ed25519_key.pub
+ cp -a /etc/ssh/ssh_host_ed25519_key.pub ssh
+ for p in /etc/ssh/ssh_host_*
+ test -e /etc/ssh/ssh_host_rsa_key
+ cp -a /etc/ssh/ssh_host_rsa_key ssh
+ for p in /etc/ssh/ssh_host_*
+ test -e /etc/ssh/ssh_host_rsa_key.pub
+ cp -a /etc/ssh/ssh_host_rsa_key.pub ssh
+ /root/kexec/kexec/ip --json addr
+ /root/kexec/kexec/ip -4 --json route
+ /root/kexec/kexec/ip -6 --json route
+ '[' -f /etc/machine-id ']'
+ cp /etc/machine-id machine-id
+ find .
+ cpio -o -H newc
+ gzip -9
27 blocks
+ kexecSyscallFlags=
+ sort -c -V
++ uname -r
+ printf '%s\n' 6.1 6.12.57
+ kexecSyscallFlags=--kexec-syscall-auto
+ sh -c ''\''/root/kexec/kexec/kexec'\'' --load '\''/root/kexec/kexec/bzImage'\''   --kexec-syscall-auto      --initrd='\''/root/kexec/kexec/initrd'\'' --no-checks   --command-line '\''init=/nix/store/br4064v0fjpnp6kvg961wciban3ikh8a-nixos-system-nixos-installer-24.11pre-git/init nouveau.modeset=0 console=tty0 console=ttyS0,115200 zswap.enabled=1 zswap.max_pool_percent=50 zswap.compressor=zstd zswap.zpool=zsmalloc root=fstab loglevel=4'\'''
machine will boot into nixos in 6s...

Then it results in the following error:

### Installing NixOS ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added 'XXX.XXX.XXX.XXX' (ED25519) to the list of known hosts.
installing the boot loader...
setting up /etc...
generating machine-specific age key...
Public key: age167su43cgfdl8hsrp4p9jyckxj52rzy0ruk03r9eqf857zuj6tuwqf6jdml
setting up secrets...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key as GPG key with fingerprint 60ccce7224e502267064c881c5040fb6dccf9ba8
sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age1t39pzt2dvrph5ttemh7kked0xefuk5p92pak4nkpjtt84r0pm9hsmgxyu0
/nix/store/fyr4h78mlrh7nw29y87lw092hdyik1v9-sops-install-secrets-0.0.1/bin/sops-install-secrets: failed to decrypt '/nix/store/y82y8h4l00m71aw9p6b0pz978zzxvjrd-secrets.yaml': Error getting data key: 0 successful groups required, got 0
Activation script snippet 'setupSecrets' failed (1)