setting custom timezone impacts VM HA

[TadiosAbebe]

I wasn’t entirely sure whether this should be classified as a bug or simply an edge case, but I’m documenting it here in case it helps someone in the future.

Earlier today, while testing VM HA on a lab cluster, I noticed that a VM with HA enabled took more than 3 hours to automatically restart after being shut down from inside the OS. When I investigated the delay, the management server logs repeatedly showed the message:

vm id: 258 - time since last state update (-10620285 ms) has not passed graceful period yet

The value -10620285 ms caught my attention, it corresponds to roughly 3 hours, which matched the delay I was observing.

looking into the code,

long milliSecondsSinceLastStateUpdate = currentTime.getTime() - vmStateUpdateTime.getTime();

Here, currentTime is obtained from:

Date currentTime = DateUtil.currentGMTTime();

Meanwhile, vmStateUpdateTime comes from the database (vm_instance.power_state_update_time), which is updated through:

        StringBuilder sql = new StringBuilder("UPDATE `cloud`.`vm_instance` SET " +
                "`power_host` = ?, `power_state_update_time` = now(), `power_state` = CASE ");

At that point, the root cause became clear: Our servers were configured with the local timezone (GMT+3) for logging convenience. This meant: currentGMTTime() returned UTC, NOW() in MySQL returned local time (UTC+3).

As a result, CloudStack ended up comparing timestamps from two different timezones, producing the large negative time difference and causing the HA mechanism to wait the full (incorrect) interval, approximately 3 hours.

To resolve this, I switched the MySQL server’s timezone to UTC (UTC+0) so that both values align. Before realizing this, I briefly changed the timezone on the management server, compute server, and database server to UTC as well. After adjusting the database timezone, I had to restart the MySQL service for the change to take effect.

I hope this helps someone encountering a similar issue. I would also appreciate any suggestions from the community on how to maintain local timezones on servers without impacting services that rely on strict time consistency like this.

[DaanHoogland]

@TadiosAbebe , thanks for you investigation and report. I think this has no solution in code that will suit everyone, but maybe a paragraph in the documentation somewhere? Do you think you can find a proper place to add this? (changing the category to “Show and tell" )