ceph10 and ceph11 second interface: SAME MAC ADDRESS!

[rbo]

isar% oc debug -q node/ceph10 -- ip -br l show dev enp65s0f1
enp65s0f1        UP             a0:36:9f:07:ff:ff <BROADCAST,MULTICAST,UP,LOWER_UP>
isar% oc debug -q node/ceph11 -- ip -br l show dev enp65s0f1
enp65s0f1        UP             a0:36:9f:07:ff:ff <BROADCAST,MULTICAST,UP,LOWER_UP>
isar% oc debug -q node/ceph12 -- ip -br l show dev enp65s0f1
enp65s0f1        UP             a0:36:9f:06:8e:2e <BROADCAST,MULTICAST,UP,LOWER_UP>
isar%

[rbo]

ceph12 NIC Slot 4: Intel(R) 10G 2P X540-t Adapter
    Firmware version: 18.3.6
    Port 1 A0:36:9F:06:8E:2C
    Port 2 A0:36:9F:06:8E:2E

ceph11 NIC Slot 4: Intel(R) 10G 2P X540-t Adapter
    Firmware version: 18.3.6
    Port 1 A0:36:9F:07:03:2C
    Port 2 A0:36:9F:07:FF:FF

ceph10 NIC Slot 4: Intel(R) Ethernet 10G 2P X540-t Adapter - A0:36:9F:07:04:C0
    Firmware version: 14.5.9
    Port 1 A0:36:9F:07:04:C0
    Port 2 A0:36:9F:07:FF:FF

[rbo]

Firmware 19.5.12, available.

[rbo]

ceph10

drained

ssh -i ~/.ssh/coe-muc -l core 10.32.97.10

[root@ceph10 core]# ip -br l | grep enp
enp65s0f0        UP             a0:36:9f:07:04:c0 <BROADCAST,MULTICAST,UP,LOWER_UP>
enp65s0f1        UP             a0:36:9f:07:ff:ff <BROADCAST,MULTICAST,UP,LOWER_UP>
[root@ceph10 core]# ip link set dev enp65s0f1 address a0:36:9f:07:04:c1

[root@ceph10 core]#
[root@ceph10 core]# ip -br l | grep enp
enp65s0f0        UP             a0:36:9f:07:04:c0 <BROADCAST,MULTICAST,UP,LOWER_UP>
enp65s0f1        UP             a0:36:9f:07:04:c1 <BROADCAST,MULTICAST,UP,LOWER_UP>
[root@ceph10 core]#

iDrac still shows the old mac address.

Reboot and again the old one:

[core@ceph10 ~]$ ip -br l | grep enp
enp65s0f0        UP             a0:36:9f:07:04:c0 <BROADCAST,MULTICAST,UP,LOWER_UP>
enp65s0f1        UP             a0:36:9f:07:ff:ff <BROADCAST,MULTICAST,UP,LOWER_UP>
[core@ceph10 ~]$

We need an intel tool to change the mac..

[rbo]

PROADMIN don't work https://www.intel.de/content/www/de/de/download/2593/administrative-tools-for-intel-network-adapters.html

[root@ceph10 core]# ./epct64e -devices branding
Ethernet Port Configuration Tool
EPCT version: v1.43.24.3
Copyright 2019 - 2025 Intel Corporation.


No supported adapters found.
[root@ceph10 core]#

[rbo]

[root@ceph10 core]# ./Network_Firmware_40NTK_LN_19.5.12_A00.BIN
Collecting inventory...
...
Running validation...

Intel(R) Ethernet 10G 2P X540-t Adapter

The version of this Update Package is newer than the currently installed version.
Software application name: Intel(R) Ethernet 10G 2P X540-t Adapter
Package version: 19.5.12
Installed version: 14.5.9

Intel(R) Ethernet 10G 2P X540-t Adapter

The version of this Update Package is newer than the currently installed version.
Software application name: Intel(R) Ethernet 10G 2P X540-t Adapter
Package version: 19.5.12
Installed version: 14.5.9



Continue? Y/N:Y
Executing update...
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER PRODUCTS WHILE UPDATE IS IN PROGRESS.
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!
..................................
Device: Intel(R) Ethernet 10G 2P X540-t Adapter
  Application: Intel(R) Ethernet 10G 2P X540-t Adapter
  An error occurred when updating a firmware module

Device: Intel(R) Ethernet 10G 2P X540-t Adapter
  Application: Intel(R) Ethernet 10G 2P X540-t Adapter
  An error occurred when updating a firmware module

[rbo]

Applied following Updates:

Network_Firmware_40NTK_LN_19.5.12_A00.BIN
BIOS_V134X_LN_2.9.0.BIN
iDRAC-with-Lifecycle-Controller_Firmware_0GHF4_LN_2.65.65.65_A00.BIN

After BIOS and iDRAC update the new NIC Firmware version is visible.

Shutdown the server and power for 10/15minutes. Now the MAC is different:

ceph10 NIC Slot 4: Intel(R) Ethernet 10G 2P X540-t Adapter - A0:36:9F:07:04:C0
    Firmware version: 19.5.12
    Port 1 A0:36:9F:07:04:C0
    Port 2 A0:36:9F:07:FF:FF

Still the old MAC...

[dmoessne]

ceph10 corrected

[core@ceph10 ~]$ sudo ./eeupdate64e 
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode

Warning: No Adapter Selected

NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

[core@ceph10 ~]$ 
[core@ceph10 ~]$ sudo ./eeupdate64e /NIC=1 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 1: LAN MAC Address is A0369F0704C0.
[core@ceph10 ~]$ 
[core@ceph10 ~]$ sudo ./eeupdate64e /NIC=2 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: LAN MAC Address is A0369F07FFFF.
[core@ceph10 ~]$ 
[core@ceph10 ~]$ sudo ./eeupdate64e /NIC=2 /MAC=A0369F0704C1
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: Updating Mac Address to A0369F0704C1...Done.
 2: Updating Checksum and CRCs...Done.
[core@ceph10 ~]$ 
[core@ceph10 ~]$ sudo ./eeupdate64e /NIC=2 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: LAN MAC Address is A0369F07FFFF.
[core@ceph10 ~]$ sudo ./eeupdate64e /NIC=2 /MAC=A0369F0704C1
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: Updating Mac Address to A0369F0704C1...Done.
 2: Updating Checksum and CRCs...Done.
[core@ceph10 ~]$ 
[core@ceph10 ~]$ sudo systemctl reboot
[core@ceph10 ~]$ Connection to 10.32.97.10 closed by remote host.
[...]
[core@ceph10 ~]$ ip -br l |grep ^enp
enp65s0f0        UP             a0:36:9f:07:04:c0 <BROADCAST,MULTICAST,UP,LOWER_UP> 
enp65s0f1        UP             a0:36:9f:07:04:c1 <BROADCAST,MULTICAST,UP,LOWER_UP> 
[core@ceph10 ~]$ 

[dmoessne]

fixed ceph11 as well

enp65s0f1        UP             a0:36:9f:07:ff:ff <BROADCAST,MULTICAST,UP,LOWER_UP> 

core@ceph11
------------
[core@ceph11 ~]$ sudo ./eeupdate64e
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode

Warning: No Adapter Selected

NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

[core@ceph11 ~]$ 
[core@ceph11 ~]$ sudo ./eeupdate64e /NIC=1 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 1: LAN MAC Address is A0369F07032C.
[core@ceph11 ~]$ 
[core@ceph11 ~]$ sudo ./eeupdate64e /NIC=2 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: LAN MAC Address is A0369F07FFFF.
[core@ceph11 ~]$ 
[core@ceph11 ~]$ sudo ./eeupdate64e /NIC=2 /MAC=A0369F07032D
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: Updating Mac Address to A0369F07032D...Done.
 2: Updating Checksum and CRCs...Done.
[core@ceph11 ~]$ 
[core@ceph11 ~]$ sudo ./eeupdate64e /NIC=2 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: LAN MAC Address is A0369F07FFFF.
[core@ceph11 ~]$ sudo systemctl reboot
[core@ceph11 ~]$ Connection to 10.32.97.11 closed by remote host.

[core@ceph11 ~]$ 
[core@ceph11 ~]$ sudo ./eeupdate64e /NIC=2 /MAC_dump
Connection to QV driver failed - please reinstall it!

Using: Intel (R) PRO Network Connections SDK v2.30.25
EEUPDATE v5.30.25.06 
Copyright (C) 1995 - 2017 Intel Corporation
Intel (R) Confidential and not for general distribution.

Driverless Mode


NIC Bus Dev Fun Vendor-Device  Branding string
=== === === === ============= =================================================
  1  65  00  00   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter
  2  65  00  01   8086-1528    Intel(R) Ethernet 10G 2P X540-t Adapter

 2: LAN MAC Address is A0369F07032D.
[core@ceph11 ~]$ 
[core@ceph11 ~]$ ip link show enp65s0f0
4: enp65s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:07:03:2c brd ff:ff:ff:ff:ff:ff
[core@ceph11 ~]$ ip link show enp65s0f1
5: enp65s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master coe-bridge state DOWN mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:07:03:2d brd ff:ff:ff:ff:ff:ff
[core@ceph11 ~]$ 
logout
Connection to 10.32.97.11 closed.