STAC-23600: Restore Clickhouse

ARCHITECTURE.md

@@ -20,8 +20,10 @@ stackstate-backup-cli/
 │   ├── root.go              # Root command and global flags
 │   ├── version/             # Version information command
 │   ├── elasticsearch/       # Elasticsearch backup/restore commands
+│   ├── clickhouse/          # ClickHouse backup/restore commands
 │   ├── stackgraph/          # Stackgraph backup/restore commands
-│   └── victoriametrics/     # VictoriaMetrics backup/restore commands
+│   ├── victoriametrics/     # VictoriaMetrics backup/restore commands
+│   └── settings/            # Settings backup/restore commands
 │
 ├── internal/                # Internal packages (Layers 0-3)
 │   ├── foundation/          # Layer 0: Core utilities
@@ -63,9 +65,11 @@ stackstate-backup-cli/
 - Formats output for end users
 
 **Key Packages**:
-- `cmd/elasticsearch/`: Elasticsearch snapshot/restore commands (configure, list-snapshots, list-indices, restore-snapshot)
+- `cmd/elasticsearch/`: Elasticsearch snapshot/restore commands (configure, list, list-indices, restore, check-and-finalize)
+- `cmd/clickhouse/`: ClickHouse backup/restore commands (list, restore, check-and-finalize)
 - `cmd/stackgraph/`: Stackgraph backup/restore commands (list, restore, check-and-finalize)
 - `cmd/victoriametrics/`: VictoriaMetrics backup/restore commands (list, restore, check-and-finalize)
+- `cmd/settings/`: Settings backup/restore commands (list, restore, check-and-finalize)
 - `cmd/version/`: Version information
 
 **Dependency Rules**:
@@ -140,6 +144,7 @@ appCtx.Formatter
 **Key Packages**:
 - `k8s/`: Kubernetes API operations (Jobs, Pods, Deployments, ConfigMaps, Secrets, Logs)
 - `elasticsearch/`: Elasticsearch HTTP API (snapshots, indices, datastreams)
+- `clickhouse/`: ClickHouse Backup API and SQL operations (backups, restore operations, status tracking)
 - `s3/`: S3/Minio operations (client creation, object filtering)
 
 **Dependency Rules**:
@@ -407,7 +412,7 @@ endpoint := "http://localhost:9200"
 ### ❌ Don't: Create Clients Directly in Commands
 
 ```go
-// BAD: cmd/elasticsearch/list-snapshots.go
+// BAD: cmd/elasticsearch/list.go
 func runListSnapshots(globalFlags *config.CLIGlobalFlags) error {
     k8sClient, _ := k8s.NewClient(globalFlags.Kubeconfig, globalFlags.Debug)
     esClient, _ := elasticsearch.NewClient("http://localhost:9200")

README.md

@@ -8,10 +8,10 @@ This CLI tool replaces the legacy Bash-based backup/restore scripts with a singl
 
 **Current Support:**
 - Elasticsearch snapshots and restores
+- ClickHouse backups and restores
 - Stackgraph backups and restores
 - VictoriaMetrics backups and restores
-
-**Planned:** ClickHouse, Configuration backups
+- Settings backups and restores
 
 ## Installation
 
@@ -69,26 +69,43 @@ List Elasticsearch indices.
 sts-backup elasticsearch list-indices --namespace <namespace>
 ```
 
-#### list-snapshots
+#### list
 
 List available Elasticsearch snapshots.
 
 ```bash
-sts-backup elasticsearch list-snapshots --namespace <namespace>
+sts-backup elasticsearch list --namespace <namespace>
 ```
 
-#### restore-snapshot
+#### restore
 
 Restore Elasticsearch snapshot. Automatically scales down affected deployments before restore and scales them back up afterward.
 
 ```bash
-sts-backup elasticsearch restore-snapshot --namespace <namespace> --snapshot-name <name> [flags]
+sts-backup elasticsearch restore --namespace <namespace> [--snapshot <name> | --latest] [flags]
+```
+
+**Flags:**
+- `--snapshot, -s` - Name of snapshot to restore (mutually exclusive with --latest)
+- `--latest` - Restore from the most recent snapshot (mutually exclusive with --snapshot)
+- `--background` - Run restore in background without waiting for completion
+- `--yes, -y` - Skip confirmation prompt
+
+**Note**: Either `--snapshot` or `--latest` must be specified (mutually exclusive).
+
+#### check-and-finalize
+
+Check the status of a restore operation and finalize if complete.
+
+```bash
+sts-backup elasticsearch check-and-finalize --namespace <namespace> --operation-id <snapshot> [--wait]
 ```
 
 **Flags:**
-- `--snapshot-name, -s` - Name of snapshot to restore (required)
-- `--drop-all-indices, -r` - Delete all existing STS indices before restore
-- `--yes` - Skip confirmation prompt
+- `--operation-id` - Operation ID of the restore operation (snapshot name) (required)
+- `--wait` - Wait for restore to complete if still running
+
+**Use Case**: This command is useful when a restore was started with `--background` flag or was interrupted (Ctrl+C).
 
 ### stackgraph
 
@@ -183,6 +200,90 @@ sts-backup victoriametrics check-and-finalize --namespace <namespace> --job <job
 **Use Case**: This command is useful when a restore job was started with `--background` flag or was interrupted (
 Ctrl+C).
 
+### settings
+
+Manage Settings backups and restores.
+
+#### list
+
+List available Settings backups from S3/Minio.
+
+```bash
+sts-backup settings list --namespace <namespace>
+```
+
+#### restore
+
+Restore Settings from a backup archive. Automatically scales down affected deployments before restore and scales them
+back up afterward.
+
+```bash
+sts-backup settings restore --namespace <namespace> [--archive <name> | --latest] [flags]
+```
+
+**Flags:**
+
+- `--archive` - Specific archive name to restore (e.g., sts-backup-20251117-1404.sty)
+- `--latest` - Restore from the most recent backup
+- `--background` - Run restore job in background without waiting for completion
+- `--yes, -y` - Skip confirmation prompt
+
+**Note**: Either `--archive` or `--latest` must be specified (mutually exclusive).
+
+#### check-and-finalize
+
+Check the status of a background Settings restore job and clean up resources.
+
+```bash
+sts-backup settings check-and-finalize --namespace <namespace> --job <job-name> [--wait]
+```
+
+**Flags:**
+
+- `--job, -j` - Settings restore job name (required)
+- `--wait, -w` - Wait for job to complete before cleanup
+
+**Use Case**: This command is useful when a restore job was started with `--background` flag or was interrupted (
+Ctrl+C).
+
+### clickhouse
+
+Manage ClickHouse backups and restores.
+
+#### list
+
+List available ClickHouse backups from the backup API.
+
+```bash
+sts-backup clickhouse list --namespace <namespace>
+```
+
+#### restore
+
+Restore ClickHouse from a backup. Automatically scales down affected StatefulSets before restore and scales them back up afterward.
+
+```bash
+sts-backup clickhouse restore --namespace <namespace> --backup-name <name> [flags]
+```
+
+**Flags:**
+- `--backup-name` - Name of the backup to restore (required)
+- `--wait` - Wait for restore to complete (default: true)
+
+#### check-and-finalize
+
+Check the status of a ClickHouse restore operation and finalize if complete.
+
+```bash
+sts-backup clickhouse check-and-finalize --namespace <namespace> --operation-id <id> [--wait]
+```
+
+**Flags:**
+- `--operation-id` - Operation ID of the restore operation (required)
+- `--wait` - Wait for restore to complete if still running
+
+**Use Case**: This command is useful when checking the status of a restore operation or finalizing after completion.
+
 ## Configuration
 
 The CLI uses configuration from Kubernetes ConfigMaps and Secrets with the following precedence:
@@ -258,13 +359,22 @@ See [internal/foundation/config/testdata/validConfigMapConfig.yaml](internal/fou
 │   ├── elasticsearch/            # Elasticsearch subcommands
 │   │   ├── configure.go          # Configure snapshot repository
 │   │   ├── list-indices.go       # List indices
-│   │   ├── list-snapshots.go     # List snapshots
-│   │   └── restore-snapshot.go   # Restore snapshot
+│   │   ├── list.go               # List snapshots
+│   │   ├── restore.go            # Restore snapshot
+│   │   └── check-and-finalize.go # Check and finalize restore
+│   ├── clickhouse/               # ClickHouse subcommands
+│   │   ├── list.go               # List backups
+│   │   ├── restore.go            # Restore backup
+│   │   └── check-and-finalize.go # Check and finalize restore
 │   ├── stackgraph/               # Stackgraph subcommands
 │   │   ├── list.go               # List backups
 │   │   ├── restore.go            # Restore backup
 │   │   └── check-and-finalize.go # Check and finalize restore job
-│   └── victoriametrics/          # VictoriaMetrics subcommands
+│   ├── victoriametrics/          # VictoriaMetrics subcommands
+│   │   ├── list.go               # List backups
+│   │   ├── restore.go            # Restore backup
+│   │   └── check-and-finalize.go # Check and finalize restore job
+│   └── settings/                 # Settings subcommands
 │       ├── list.go               # List backups
 │       ├── restore.go            # Restore backup
 │       └── check-and-finalize.go # Check and finalize restore job
@@ -276,6 +386,7 @@ See [internal/foundation/config/testdata/validConfigMapConfig.yaml](internal/fou
 │   ├── clients/                  # Layer 1: Service clients
 │   │   ├── k8s/                  # Kubernetes client
 │   │   ├── elasticsearch/        # Elasticsearch client
+│   │   ├── clickhouse/           # ClickHouse client
 │   │   └── s3/                   # S3/Minio client
 │   ├── orchestration/            # Layer 2: Workflows
 │   │   ├── portforward/          # Port-forwarding lifecycle