[doc][yba][DOC-521][2024.2] Auto master failover (yugabyte#24875)

ddhodge · aishwarya24 · web-flow · commit e8c9e7d352d4 · 2024-11-19T13:41:46.000-05:00
* auto master failover

* auto master failover

* review comments

* review comment

* Update docs/content/v2024.2/yugabyte-platform/manage-deployments/remove-nodes.md

Co-authored-by: Aishwarya Chakravarthy  &lt;achakravarthy@yugabyte.com&gt;

* review comments

---------

Co-authored-by: Aishwarya Chakravarthy &lt;achakravarthy@yugabyte.com&gt;
diff --git a/docs/content/preview/yugabyte-platform/manage-deployments/remove-nodes.md b/docs/content/preview/yugabyte-platform/manage-deployments/remove-nodes.md
@@ -16,6 +16,67 @@ menu:
 type: docs
 ---
 
+## Automatic YB-Master failover
+
+{{<tags/feature/ea>}} To avoid under-replication, YugabyteDB Anywhere can automatically detect a YB-Master server that is not responding to the master leader, or that is lagging WAL operations, and fail over to another available node in the same availability zone.
+
+Note that automatic failover only works for a single unhealthy master server.
+
+### Prerequisites
+
+- Automatic YB-Master failover is {{<tags/feature/ea>}}. To enable the feature for a universe, set the **Auto Master Failover** Universe Runtime Configuration option (config key `yb.auto_master_failover.enabled`) to true. Refer to [Manage runtime configuration settings](../../administer-yugabyte-platform/manage-runtime-config/).
+- The universe has the following characteristics:
+
+  - running v2.20.3.0, v2.21.0.0, or later
+  - is on-premises or on a cloud provider (Kubernetes is not supported)
+  - has a replication factor of 3 or more
+  - does not have dedicated masters
+
+- A replacement node (running a TServer) is available in the same availability zone. (Read replica nodes are not valid for failover.)
+
+### How it works
+
+Automatic master failover works as follows:
+
+1. When active, by default YugabyteDB Anywhere checks universe masters every minute to see if they are healthy.
+
+    You can customize this interval using the universe runtime configuration option `yb.auto_master_failover.detect_interval`.
+
+1. YugabyteDB Anywhere declares a master is failed or potentially failed when any of the following conditions are met:
+
+    - The Master heartbeat delay is greater than the threshold.
+    - Maximum tablet follower lag is greater than the threshold.
+
+1. When YugabyteDB Anywhere detects an unhealthy master in a universe, it displays a message on the universe **Overview** indicating a potential master failure, and indicating the estimated time remaining until auto failover.
+
+    The warning is displayed when a master lags more than the threshold defined by the universe runtime configuration option `yb.auto_master_failover.master_follower_lag_soft_threshold`.
+
+    You can configure the time to failover using the universe runtime configuration option `yb.auto_master_failover.master_follower_lag_hard_threshold`.
+
+1. During this time, you can investigate and potentially fix the problem. Navigate to the universe **Nodes** tab to check the status of the nodes. You may need to replace or eliminate unresponsive nodes, or fix a lagging master process. Refer to the following sections.
+
+    If you fix the problem, the warning is dismissed, and YugabyteDB Anywhere returns to monitoring the universe.
+
+    If you need more time to investigate or fix the problem manually, you can opt to snooze the failover.
+
+1. Failover is triggered if the time expires and the issue hasn't been fixed.
+
+    For a universe to successfully fail over masters, the following must be true:
+
+    - The universe is not paused.
+    - The universe is not locked (that is, another locking operation is running).
+    - All nodes are live; that is, there aren't any stopped, removed, or decommissioned nodes.
+
+    Note that master failover may not fix all the issues with the universe. Be sure to address other failed or unavailable nodes or other issues to bring your universe back to a healthy state.
+
+    For master failover, if the task fails, a retry is made automatically. The retry limit for failover tasks is set by the universe runtime configuration option `yb.auto_master_failover.max_task_retries`.
+
+1. After starting up a new master on a different node in the same availability zone as the failed master, YugabyteDB Anywhere waits for you to recover any failed VMs, including the failed master VM, so that it can update the master address configuration on those VMs. Follow the steps in [Replace a live or unreachable node](#replace-a-live-or-unreachable-node).
+
+    You can set the delay for post automatic master failover using the universe runtime configuration option `yb.auto_master_failover.sync_master_addrs_task_delay`. The reference start time is calculated from the time that YugabyteDB Anywhere finds that all processes are running fine on all the VMs.
+
+    Post failover, there is no retry limit as it is a critical operation.
+
 ## Replace a live or unreachable node
 
 To replace a live node for extended maintenance or replace an unhealthy node, do the following:
diff --git a/docs/content/v2024.2/yugabyte-platform/manage-deployments/remove-nodes.md b/docs/content/v2024.2/yugabyte-platform/manage-deployments/remove-nodes.md
@@ -12,6 +12,67 @@ menu:
 type: docs
 ---
 
+## Automatic YB-Master failover
+
+{{<tags/feature/ea>}} To avoid under-replication, YugabyteDB Anywhere can automatically detect a YB-Master server that is not responding to the master leader, or that is lagging WAL operations, and fail over to another available node in the same availability zone.
+
+Note that automatic failover only works for a single unhealthy master server.
+
+### Prerequisites
+
+- Automatic YB-Master failover is {{<tags/feature/ea>}}. To enable the feature for a universe, set the **Auto Master Failover** Universe Runtime Configuration option (config key `yb.auto_master_failover.enabled`) to true. Refer to [Manage runtime configuration settings](../../administer-yugabyte-platform/manage-runtime-config/).
+- The universe has the following characteristics:
+
+  - running v2.20.3.0, v2.21.0.0, or later
+  - is on-premises or on a cloud provider (Kubernetes is not supported)
+  - has a replication factor of 3 or more
+  - does not have dedicated masters
+
+- A replacement node (running a TServer) is available in the same availability zone. (Read replica nodes are not valid for failover.)
+
+### How it works
+
+Automatic master failover works as follows:
+
+1. When active, by default YugabyteDB Anywhere checks universe masters every minute to see if they are healthy.
+
+    You can customize this interval using the universe runtime configuration option `yb.auto_master_failover.detect_interval`.
+
+1. YugabyteDB Anywhere declares a master is failed or potentially failed when any of the following conditions are met:
+
+    - The Master heartbeat delay is greater than the threshold.
+    - Maximum tablet follower lag is greater than the threshold.
+
+1. When YugabyteDB Anywhere detects an unhealthy master in a universe, it displays a message on the universe **Overview** indicating a potential master failure, and indicating the estimated time remaining until auto failover.
+
+    The warning is displayed when a master lags more than the threshold defined by the universe runtime configuration option `yb.auto_master_failover.master_follower_lag_soft_threshold`.
+
+    You can configure the time to failover using the universe runtime configuration option `yb.auto_master_failover.master_follower_lag_hard_threshold`.
+
+1. During this time, you can investigate and potentially fix the problem. Navigate to the universe **Nodes** tab to check the status of the nodes. You may need to replace or eliminate unresponsive nodes, or fix a lagging master process. Refer to the following sections.
+
+    If you fix the problem, the warning is dismissed, and YugabyteDB Anywhere returns to monitoring the universe.
+
+    If you need more time to investigate or fix the problem manually, you can opt to snooze the failover.
+
+1. Failover is triggered if the time expires and the issue hasn't been fixed.
+
+    For a universe to successfully fail over masters, the following must be true:
+
+    - The universe is not paused.
+    - The universe is not locked (that is, another locking operation is running).
+    - All nodes are live; that is, there aren't any stopped, removed, or decommissioned nodes.
+
+    Note that master failover may not fix all the issues with the universe. Be sure to address other failed or unavailable nodes or other issues to bring your universe back to a healthy state.
+
+    For master failover, if the task fails, a retry is made automatically. The retry limit for failover tasks is set by the universe runtime configuration option `yb.auto_master_failover.max_task_retries`.
+
+1. After starting up a new master on a different node in the same availability zone as the failed master, YugabyteDB Anywhere waits for you to recover any failed VMs, including the failed master VM, so that it can update the master address configuration on those VMs. Follow the steps in [Replace a live or unreachable node](#replace-a-live-or-unreachable-node).
+
+    You can set the delay for post automatic master failover using the universe runtime configuration option `yb.auto_master_failover.sync_master_addrs_task_delay`. The reference start time is calculated from the time that YugabyteDB Anywhere finds that all processes are running fine on all the VMs.
+
+    Post failover, there is no retry limit as it is a critical operation.
+
 ## Replace a live or unreachable node
 
 To replace a live node for extended maintenance or replace an unhealthy node, do the following:
@@ -21,19 +82,20 @@ To replace a live node for extended maintenance or replace an unhealthy node, do
 
     ![Replace Node Actions](/images/ee/replace-node.png)
 
-1. Click OK to confirm.
+1. Click **OK** to confirm.
 
-YugabyteDB Anywhere (YBA) starts the node replacement process, and you can view the progress on the **Tasks** tab. As part of the node replacement process, all data (tablets) on the existing node will be moved to other nodes to ensure that the desired replication factor is maintained throughout the operation.
+YugabyteDB Anywhere starts the node replacement process, and you can view the progress on the **Tasks** tab. As part of the node replacement process, all data (tablets) on the existing node will be moved to other nodes to ensure that the desired replication factor is maintained throughout the operation.
 
-For cloud providers (AWS, Azure, or GCP), YBA returns the existing node back to the provider and provisions a new replacement node from the cloud provider. For on-premises universes, the existing node is returned to the [on-premises provider node pool](../../configure-yugabyte-platform/on-premises-nodes/) and a new replacement node is selected from the free pool.
+For cloud providers (AWS, Azure, or GCP), YugabyteDB Anywhere returns the existing node back to the provider and provisions a new replacement node from the cloud provider. For on-premises universes, the existing node is returned to the [on-premises provider node pool](../../configure-yugabyte-platform/on-premises-nodes/) and a new replacement node is selected from the free pool.
 
-For on-premises universes, clean up of existing data directories and running processes may fail if the node is unhealthy. In such cases, YBA sets the state to Decommissioned. This prevents the node from being added to a new universe.
+For on-premises universes, clean up of existing data directories and running processes may fail if the node is unhealthy. In such cases, YugabyteDB Anywhere sets the state to Decommissioned. This prevents the node from being added to a new universe.
 
 ### Check on-premises node state
 
 On-premises nodes have three states: In use, Free, and Decommissioned as described in the following illustration.
 
 ![Decommissioned node workflow](/images/ee/on-prem-replace-workflow.png)
+
 To check the state of an on-premises node, navigate to **Integrations > Infrastructure > On-Premises Datacenters**, select the associated on-premises configuration, and click **Instances**.
 
 ### Recommission a decommissioned on-premises node
@@ -44,13 +106,13 @@ Perform the following steps to recommission a node:
 
 1. Navigate to **Integrations > Infrastructure > On-Premises Datacenters**, select the associated on-premises configuration, and click **Instances**.
 
-1. Under Instances, for the decommissioned node, click **Actions > Recommission Node**. YBA will now re-attempt to clean up existing data directories and processes on this node.
+1. Under Instances, for the decommissioned node, click **Actions > Recommission Node**. YugabyteDB Anywhere will re-attempt to clean up existing data directories and processes on this node.
 
     ![Recommission Node](/images/ee/recommission-node.png)
 
 1. Click OK to confirm.
 
-YugabyteDB Anywhere (YBA) starts the node recommissioning process, and you can view the progress on the **Tasks** tab.
+YugabyteDB Anywhere starts the node recommissioning process, and you can view the progress on the **Tasks** tab.
 
 ## Eliminate an unresponsive node
 
@@ -134,11 +196,11 @@ A typical universe has an RF of 3 or 5. At the end of the [node removal](#remove
 
 1. Click **Actions > Start Master** corresponding to the node, as per the following illustration.
 
-   This action is only available if there are additional nodes in the same availability zone and these nodes do not have a running Master process.
+    This action is only available if there are additional nodes in the same availability zone and these nodes do not have a running Master process.
 
     ![Start master](/images/yp/start-master.png)
 
-   When you execute the start Master action, YugabyteDB Anywhere performs the following:
+    When you execute the start Master action, YugabyteDB Anywhere performs the following:
 
 1. Configures the Master on the subject node.
 
@@ -202,7 +264,7 @@ In some cases, depending on the node's status, YugabyteDB Anywhere allows you to
 
 1. Find a node with a Decommissioned status and click its corresponding **Actions > Add Node**, as per the following illustration:
 
-  ![Add Node Actions](/images/ee/node-actions-add-node.png)
+    ![Add Node Actions](/images/ee/node-actions-add-node.png)
 
 For Infrastructure as a service (IaaS) such as AWS and GCP, YugabyteDB Anywhere will spawn with the existing node instance type in the existing region and zone of that node. When the process completes, the node will have the Master and TServer processes running, along with data that is load-balanced onto this node. The node's name will be reused and the status will be shown as Live.
 
