Skip to content

STAC-23600: Restore Clickhouse #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
viliakov merged 7 commits into from
Nov 25, 2025
Merged

STAC-23600: Restore Clickhouse #7

viliakov merged 7 commits into from
Nov 25, 2025

Conversation

@viliakov
Copy link
Contributor

@viliakov viliakov commented Nov 23, 2025 •
edited
Loading

Summary

This PR adds ClickHouse backup/restore support and makes Elasticsearch commands follow the same pattern.

Key Changes

Elasticsearch Improvements

  • restore-snapshot and list-snapshots commands renamed to restore and list respectivelly.
  • Introduced check-and-finalize command allowing to scale up the deployments when the restore operation is done
  • Changed snapshot list sort order to show newest snapshots first

Clickhouse

clickhouse list

Lists available backups from Minio S3 storage.

Example:

❯ go run main.go clickhouse list --namespace stac-23374-ha
Setting up port-forward to suse-observability-clickhouse-backup:7171 in namespace stac-23374-ha...
✓ Port-forward established successfully
Listing Clickhouse backups...

NAME                             CREATED              SIZE
incremental_2025-11-23T15-45-00  2025-11-23 15:45:04  64MiB
incremental_2025-11-23T14-45-00  2025-11-23 14:45:04  137MiB
incremental_2025-11-23T13-45-00  2025-11-23 13:45:04  178MiB
incremental_2025-11-23T12-45-00  2025-11-23 12:45:04  62MiB
incremental_2025-11-23T11-45-04  2025-11-23 11:45:08  206MiB
incremental_2025-11-23T10-45-00  2025-11-23 10:45:04  11MiB
incremental_2025-11-23T09-45-00  2025-11-23 09:45:04  1GiB
incremental_2025-11-23T08-45-00  2025-11-23 08:45:04  65MiB
incremental_2025-11-23T07-45-00  2025-11-23 07:45:04  54MiB
incremental_2025-11-23T06-45-00  2025-11-23 06:45:04  77MiB
incremental_2025-11-23T05-45-00  2025-11-23 05:45:04  131MiB
incremental_2025-11-23T04-45-00  2025-11-23 04:45:04  65MiB
incremental_2025-11-23T03-45-00  2025-11-23 03:45:04  182MiB
full_2025-11-23T00-45-01         2025-11-23 00:45:04  2GiB
incremental_2025-11-22T23-45-00  2025-11-22 23:45:04  73MiB
incremental_2025-11-22T22-45-00  2025-11-22 22:45:04  71MiB
incremental_2025-11-22T21-45-04  2025-11-22 21:45:07  111MiB
incremental_2025-11-22T20-45-00  2025-11-22 20:45:04  51MiB
incremental_2025-11-22T19-45-00  2025-11-22 19:45:04  67MiB
incremental_2025-11-22T18-45-00  2025-11-22 18:45:04  87MiB
incremental_2025-11-22T17-45-00  2025-11-22 17:45:04  58MiB
incremental_2025-11-22T16-45-00  2025-11-22 16:45:04  71MiB
incremental_2025-11-22T15-45-00  2025-11-22 15:45:04  153MiB
incremental_2025-11-22T14-45-00  2025-11-22 14:45:04  65MiB
incremental_2025-11-22T13-45-00  2025-11-22 13:45:04  81MiB
incremental_2025-11-22T12-45-00  2025-11-22 12:45:04  83MiB
incremental_2025-11-22T11-45-00  2025-11-22 11:45:04  95MiB
incremental_2025-11-22T10-45-00  2025-11-22 10:45:04  72MiB
incremental_2025-11-22T09-45-00  2025-11-22 09:45:04  27MiB
incremental_2025-11-22T08-45-00  2025-11-22 08:45:04  2GiB
incremental_2025-11-22T07-45-00  2025-11-22 07:45:04  2GiB
incremental_2025-11-22T06-45-04  2025-11-22 06:45:08  2GiB
incremental_2025-11-22T05-45-00  2025-11-22 05:45:04  2GiB
incremental_2025-11-22T04-45-00  2025-11-22 04:45:04  2GiB
incremental_2025-11-22T03-45-00  2025-11-22 03:45:04  2GiB
full_2025-11-22T00-45-00         2025-11-22 00:45:04  2GiB
incremental_2025-11-21T23-45-00  2025-11-21 23:45:04  2GiB
incremental_2025-11-21T22-45-00  2025-11-21 22:45:04  2GiB
incremental_2025-11-21T21-45-00  2025-11-21 21:45:04  2GiB
incremental_2025-11-21T20-45-04  2025-11-21 20:45:08  2GiB
incremental_2025-11-21T19-45-00  2025-11-21 19:45:04  2GiB
incremental_2025-11-21T18-45-04  2025-11-21 18:45:07  2GiB
incremental_2025-11-21T17-45-00  2025-11-21 17:45:04  420KiB
incremental_2025-11-21T16-45-00  2025-11-21 16:45:04  12MiB
incremental_2025-11-21T15-45-00  2025-11-21 15:45:04  1GiB
incremental_2025-11-21T14-45-00  2025-11-21 14:45:04  60MiB
incremental_2025-11-21T13-45-04  2025-11-21 13:45:07  54MiB
incremental_2025-11-21T12-45-04  2025-11-21 12:45:07  48MiB
incremental_2025-11-21T11-45-00  2025-11-21 11:45:04  188MiB
incremental_2025-11-21T10-45-00  2025-11-21 10:45:04  49MiB
incremental_2025-11-21T09-45-04  2025-11-21 09:45:07  11MiB
incremental_2025-11-21T08-45-00  2025-11-21 08:45:03  11MiB
incremental_2025-11-21T07-45-02  2025-11-21 07:45:05  16MiB
incremental_2025-11-21T06-45-00  2025-11-21 06:45:03  12MiB
incremental_2025-11-21T05-45-00  2025-11-21 05:45:03  36MiB
incremental_2025-11-21T04-45-00  2025-11-21 04:45:03  13MiB
incremental_2025-11-21T03-45-00  2025-11-21 03:45:03  17MiB
full_2025-11-21T00-45-00         2025-11-21 00:45:03  1GiB
incremental_2025-11-20T23-45-00  2025-11-20 23:45:03  59MiB
incremental_2025-11-20T22-45-00  2025-11-20 22:45:03  76MiB
incremental_2025-11-20T21-45-00  2025-11-20 21:45:03  168MiB
incremental_2025-11-20T20-45-04  2025-11-20 20:45:07  71MiB
incremental_2025-11-20T19-45-00  2025-11-20 19:45:03  57MiB
incremental_2025-11-20T18-45-00  2025-11-20 18:45:03  133MiB
incremental_2025-11-20T17-45-00  2025-11-20 17:45:03  77MiB
incremental_2025-11-20T16-45-04  2025-11-20 16:45:07  27MiB
incremental_2025-11-20T15-45-00  2025-11-20 15:45:03  458MiB
incremental_2025-11-20T14-45-00  2025-11-20 14:45:03  34MiB
incremental_2025-11-20T13-45-00  2025-11-20 13:45:04  105MiB
incremental_2025-11-20T12-45-03  2025-11-20 12:45:07  38MiB
incremental_2025-11-20T11-45-04  2025-11-20 11:45:07  48MiB
incremental_2025-11-20T10-45-00  2025-11-20 10:45:03  931MiB
incremental_2025-11-20T09-45-00  2025-11-20 09:45:03  115MiB
incremental_2025-11-20T08-45-00  2025-11-20 08:45:04  62MiB
incremental_2025-11-20T07-45-00  2025-11-20 07:45:03  66MiB
incremental_2025-11-20T06-45-00  2025-11-20 06:45:03  122MiB
incremental_2025-11-20T05-45-00  2025-11-20 05:45:03  66MiB
incremental_2025-11-20T04-45-00  2025-11-20 04:45:03  61MiB
incremental_2025-11-20T03-45-00  2025-11-20 03:45:03  184MiB
full_2025-11-20T00-45-00         2025-11-20 00:45:03  726MiB
incremental_2025-11-19T23-45-00  2025-11-19 23:45:03  66MiB
incremental_2025-11-19T22-45-00  2025-11-19 22:45:03  438MiB
incremental_2025-11-19T21-45-00  2025-11-19 21:45:03  55MiB
incremental_2025-11-19T20-45-00  2025-11-19 20:45:03  59MiB
incremental_2025-11-19T19-45-00  2025-11-19 19:45:03  394MiB
incremental_2025-11-19T18-45-00  2025-11-19 18:45:03  69MiB
incremental_2025-11-19T17-45-00  2025-11-19 17:45:03  63MiB
incremental_2025-11-19T16-45-00  2025-11-19 16:45:03  138MiB
incremental_2025-11-19T15-45-00  2025-11-19 15:45:03  53MiB
incremental_2025-11-19T14-45-04  2025-11-19 14:45:07  61MiB
incremental_2025-11-19T13-45-00  2025-11-19 13:45:03  146MiB
incremental_2025-11-19T12-45-01  2025-11-19 12:45:04  43MiB
incremental_2025-11-19T11-45-00  2025-11-19 11:45:03  58MiB
incremental_2025-11-19T10-45-00  2025-11-19 10:45:03  67MiB
incremental_2025-11-19T09-45-00  2025-11-19 09:45:03  73MiB
incremental_2025-11-19T08-45-00  2025-11-19 08:45:03  166MiB
incremental_2025-11-19T07-45-04  2025-11-19 07:45:07  65MiB
incremental_2025-11-19T06-45-00  2025-11-19 06:45:03  58MiB
incremental_2025-11-19T05-45-00  2025-11-19 05:45:03  58MiB
incremental_2025-11-19T04-45-00  2025-11-19 04:45:03  187MiB
incremental_2025-11-19T03-45-00  2025-11-19 03:45:03  148MiB
full_2025-11-19T00-45-04         2025-11-19 00:45:07  395MiB
incremental_2025-11-18T23-45-00  2025-11-18 23:45:03  172MiB
incremental_2025-11-18T22-45-00  2025-11-18 22:45:03  66MiB
incremental_2025-11-18T21-45-00  2025-11-18 21:45:03  89MiB
incremental_2025-11-18T20-45-00  2025-11-18 20:45:03  69MiB
incremental_2025-11-18T19-45-04  2025-11-18 19:45:06  121MiB
incremental_2025-11-18T18-45-04  2025-11-18 18:45:07  68MiB
incremental_2025-11-18T17-45-00  2025-11-18 17:45:03  55MiB
incremental_2025-11-18T16-45-00  2025-11-18 16:45:03  77MiB
incremental_2025-11-18T15-45-00  2025-11-18 15:45:03  79MiB
incremental_2025-11-18T14-45-00  2025-11-18 14:45:03  87MiB
incremental_2025-11-18T13-45-07  2025-11-18 13:45:09  55MiB
incremental_2025-11-18T12-45-00  2025-11-18 12:45:03  21MiB
full_2025-11-18T11-45-04         2025-11-18 11:45:07  48KiB

clickhouse restore

Restores a backup from Minio S3 storage with automatic deployment scaling
Restore Workflow

Usage

Usage:
sts-backup clickhouse restore [flags]

Flags:
--background Run restore in background without waiting for completion
-h, --help help for restore
--latest Restore from the most recent backup
--snapshot string Specific snapshot/archive name to restore (e.g., full_2025-11-18T11-45-04)
-y, --yes Skip confirmation prompt

Example 1: Restore with Background Execution (Interactive).

❯ go run main.go clickhouse restore -n stac-23374-ha --snapshot full_2025-11-18T11-45-04 --background

Warning: WARNING: Restoring from backup will overwrite existing ClickHouse data!
Warning: This operation cannot be undone.

Backup to restore: full_2025-11-18T11-45-04
Namespace: stac-23374-ha

Do you want to continue? (yes/no): yes

Scaling down deployments (selector: observability.suse.com/scalable-during-clickhouse-restore=true)...
✓ Scaled down 0 deployment(s):
✓ Scaled down 1 statefulsets(s):
  - suse-observability-otel-collector (replicas: 1 -> 0)
Waiting for pods to terminate...
✓ All pods have terminated
Setting up port-forward to suse-observability-clickhouse-backup:7171 in namespace stac-23374-ha...
✓ Port-forward established successfully

Triggering restore for backup: full_2025-11-18T11-45-04
✓ Restore triggered successfully (operation ID: 0cb4fb23-c88c-11f0-ac61-5a4aba4d9429)

Restore is running for clickhouse: 0cb4fb23-c88c-11f0-ac61-5a4aba4d9429

To check status and finalize, run:
  sts-backup clickhouse check-and-finalize --operation-id 0cb4fb23-c88c-11f0-ac61-5a4aba4d9429 --wait -n stac-23374-ha

clickhouse check-and-finalize

Check the status of a background restore job and clean up resources.

Usage
sts-backup clickhouse check-and-finalize --operation-id [--wait] -n

Flags:
--operation-id string Operation ID of the restore operation (required)
--wait Wait for restore to complete before finalizing

Note: This command automatically scales up deployments that were scaled down during restore.


**Example: Waiting for the job to finish**
```sh
❯ go run main.go clickhouse check-and-finalize --operation-id 0cb4fb23-c88c-11f0-ac61-5a4aba4d9429 --wait -n stac-23374-ha
Setting up port-forward to suse-observability-clickhouse-backup:7171 in namespace stac-23374-ha...
✓ Port-forward established successfully

Checking restore status for operation: 0cb4fb23-c88c-11f0-ac61-5a4aba4d9429
✓ Restore completed successfully
Executing post-restore SQL commands...
Setting up port-forward to suse-observability-clickhouse-backup:9000 in namespace stac-23374-ha...
✓ Port-forward established successfully
✓ Post-restore SQL executed successfully

Scaling up deployments from annotations (selector: observability.suse.com/scalable-during-clickhouse-restore=true)...
✓ Scaled up 0 deployment(s) successfully:
✓ Scaled up 1 statefulset(s) successfully:
  - suse-observability-otel-collector (replicas: 0 -> 1)

✓ Restore finalized successfully

rb3ckers
rb3ckers reviewed Nov 25, 2025
View reviewed changes
@viliakov viliakov merged commit d5de0d2 into main Nov 25, 2025
5 checks passed
@viliakov viliakov deleted the STAC-23600 branch November 25, 2025 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rb3ckers rb3ckers rb3ckers approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@viliakov @rb3ckers