Skip to content

STAC-24013: Settings commands support pvc as a backup storage #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
viliakov merged 4 commits into from
Dec 18, 2025
Merged

STAC-24013: Settings commands support pvc as a backup storage #9

viliakov merged 4 commits into from
Dec 18, 2025

Conversation

@viliakov
Copy link
Contributor

@viliakov viliakov commented Dec 17, 2025 •
edited
Loading

This mr adds support for settings commands to get backups from a K8s pvc.

The changes:

  • CLI Config:

    • New fields:
      • config.minio.enabled set with Helm value .global.backup.enabled.
      • config.settings.restore.pvc set with the same value as PVC name for settings Cronjob.

  • a new runner function cmd/cmdutlis.Run that is used for all commands and wraps up some common logic:

    • ensures that config.minio.enabled is configured wherever needed. Otherwise cli fails with the error.
    • common error handling

  • settings list/restore commands:

    • get the list of the backups from PVC
    • get the list of the backups from Minio (only if config.minio.enabled is true)
    • since PVC and Minio have the same content, it removes the duplicated from the concatenated list before presenting it to the user

  • renaming some functions to more generic names, like createRestoreJob -> createJob.

Usage:

  • .Values.global.backup.enabled=false
    • list backups from PVC
❯ go run main.go settings list --namespace stac-23374-ha

Creating job to list Settings backups stored on PVC...
✓ List job created: settings-list-20251218t091743

✓ List job completed successfully
Cleaning up resources...
✓ Job deleted: settings-list-20251218t091743
NAME                          LAST MODIFIED            SIZE
sts-backup-20251217-0400.sty  2025-12-17 04:00:34 UTC  2MiB
sts-backup-20251216-0400.sty  2025-12-16 04:00:35 UTC  2MiB
sts-backup-20251203-0400.sty  2025-12-03 04:00:47 UTC  2MiB
sts-backup-20251202-0400.sty  2025-12-02 04:00:43 UTC  2MiB
sts-backup-20251201-0400.sty  2025-12-01 04:00:43 UTC  2MiB
sts-backup-20251130-0400.sty  2025-11-30 04:00:46 UTC  2MiB
sts-backup-20251129-0400.sty  2025-11-29 04:00:48 UTC  2MiB
sts-backup-20251128-1328.sty  2025-11-28 13:29:12 UTC  2MiB
sts-backup-20251127-0400.sty  2025-11-27 04:00:42 UTC  2MiB
sts-backup-20251126-0400.sty  2025-11-26 04:00:46 UTC  2MiB
  • restore backup from PVC
❯ go run main.go settings restore --namespace stac-23374-ha --archive sts-backup-20251217-0400.sty

Warning: WARNING: Restoring from backup will PURGE all existing Stackgraph (Topology) data!
Warning: This operation cannot be undone.

Backup to restore: sts-backup-20251217-0400.sty
Namespace: stac-23374-ha

Do you want to continue? (yes/no): yes

Scaling down deployments (selector: stackstate.com/connects-to-stackgraph=true)...
✓ Scaled down 9 deployment(s):
  - suse-observability-api (replicas: 1 -> 0)
  - suse-observability-authorization-sync (replicas: 1 -> 0)
  - suse-observability-checks (replicas: 1 -> 0)
  - suse-observability-health-sync (replicas: 1 -> 0)
  - suse-observability-initializer (replicas: 1 -> 0)
  - suse-observability-notification (replicas: 1 -> 0)
  - suse-observability-slicing (replicas: 1 -> 0)
  - suse-observability-state (replicas: 1 -> 0)
  - suse-observability-sync (replicas: 1 -> 0)
✓ Scaled down 0 statefulsets(s):
Waiting for pods to terminate...
Waiting for 1 pod(s) to terminate...
✓ All pods have terminated

Ensuring backup scripts ConfigMap exists...
✓ Backup scripts ConfigMap ready
Ensuring Minio keys secret exists...
✓ Minio keys secret ready

Creating restore job for backup: sts-backup-20251217-0400.sty
✓ Restore job created: settings-restore-20251218t093820

Waiting for restore job to complete (this may take significant amount of time depending on the archive size)...

You can safely interrupt this command with Ctrl+C.
To check status, scale up the required deployments and cleanup later, run:
  sts-backup settings check-and-finalize --job settings-restore-20251218t093820 --wait -n stac-23374-ha

✓ Restore completed successfully

Cleaning up resources...
✓ Job deleted: settings-restore-20251218t093820
Warning: Failed to delete PVC: persistentvolumeclaims "settings-restore-20251218t093820" not found

Scaling up deployments from annotations (selector: stackstate.com/connects-to-stackgraph=true)...
✓ Scaled up 9 deployment(s) successfully:
  - suse-observability-api (replicas: 0 -> 1)
  - suse-observability-authorization-sync (replicas: 0 -> 1)
  - suse-observability-checks (replicas: 0 -> 1)
  - suse-observability-health-sync (replicas: 0 -> 1)
  - suse-observability-initializer (replicas: 0 -> 1)
  - suse-observability-notification (replicas: 0 -> 1)
  - suse-observability-slicing (replicas: 0 -> 1)
  - suse-observability-state (replicas: 0 -> 1)
  - suse-observability-sync (replicas: 0 -> 1)
✓ Scaled up 0 statefulset(s) successfully:
  • .Values.global.backup.enabled=true
    • list when Minio is enabled
❯ go run main.go settings list --namespace stac-23374-ha
Setting up port-forward to suse-observability-minio:9000 in namespace stac-23374-ha...
✓ Port-forward established successfully
Listing Settings backups in bucket 'sts-configuration-backup'...

Creating job to list Settings backups stored on PVC...
✓ List job created: settings-list-20251218t095507

✓ List job completed successfully
Cleaning up resources...
✓ Job deleted: settings-list-20251218t095507
NAME                          LAST MODIFIED            SIZE
sts-backup-20251217-0400.sty  2025-12-17 04:00:34 UTC  2MiB
sts-backup-20251216-0400.sty  2025-12-16 04:00:35 UTC  2MiB
sts-backup-20251203-0400.sty  2025-12-03 04:00:47 UTC  2MiB
sts-backup-20251202-0400.sty  2025-12-02 04:00:43 UTC  2MiB
sts-backup-20251201-0400.sty  2025-12-01 04:00:43 UTC  2MiB
sts-backup-20251130-0400.sty  2025-11-30 04:00:46 UTC  2MiB
sts-backup-20251129-0400.sty  2025-11-29 04:00:48 UTC  2MiB
sts-backup-20251128-1328.sty  2025-11-28 13:29:12 UTC  2MiB
sts-backup-20251127-0400.sty  2025-11-27 04:00:42 UTC  2MiB
sts-backup-20251126-0400.sty  2025-11-26 04:00:46 UTC  2MiB
  • restore when Minio is enabled
❯ go run main.go settings restore --namespace stac-23374-ha --archive sts-backup-20251217-0400.sty

Warning: WARNING: Restoring from backup will PURGE all existing Stackgraph (Topology) data!
Warning: This operation cannot be undone.

Backup to restore: sts-backup-20251217-0400.sty
Namespace: stac-23374-ha

Do you want to continue? (yes/no): yes

Scaling down deployments (selector: stackstate.com/connects-to-stackgraph=true)...
✓ Scaled down 9 deployment(s):
  - suse-observability-api (replicas: 1 -> 0)
  - suse-observability-authorization-sync (replicas: 1 -> 0)
  - suse-observability-checks (replicas: 1 -> 0)
  - suse-observability-health-sync (replicas: 1 -> 0)
  - suse-observability-initializer (replicas: 1 -> 0)
  - suse-observability-notification (replicas: 1 -> 0)
  - suse-observability-slicing (replicas: 1 -> 0)
  - suse-observability-state (replicas: 1 -> 0)
  - suse-observability-sync (replicas: 1 -> 0)
✓ Scaled down 0 statefulsets(s):
Waiting for pods to terminate...
Waiting for 9 pod(s) to terminate...
Waiting for 9 pod(s) to terminate...
Waiting for 9 pod(s) to terminate...
Waiting for 9 pod(s) to terminate...
Waiting for 9 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
Waiting for 8 pod(s) to terminate...
✓ All pods have terminated

Ensuring backup scripts ConfigMap exists...
✓ Backup scripts ConfigMap ready
Ensuring Minio keys secret exists...
✓ Minio keys secret ready

Creating restore job for backup: sts-backup-20251217-0400.sty
✓ Restore job created: settings-restore-20251218t102135

Waiting for restore job to complete (this may take significant amount of time depending on the archive size)...

You can safely interrupt this command with Ctrl+C.
To check status, scale up the required deployments and cleanup later, run:
  sts-backup settings check-and-finalize --job settings-restore-20251218t102135 --wait -n stac-23374-ha

✓ Restore completed successfully

Cleaning up resources...
✓ Job deleted: settings-restore-20251218t102135
Warning: Failed to delete PVC: persistentvolumeclaims "settings-restore-20251218t102135" not found

Scaling up deployments from annotations (selector: stackstate.com/connects-to-stackgraph=true)...
✓ Scaled up 9 deployment(s) successfully:
  - suse-observability-api (replicas: 0 -> 1)
  - suse-observability-authorization-sync (replicas: 0 -> 1)
  - suse-observability-checks (replicas: 0 -> 1)
  - suse-observability-health-sync (replicas: 0 -> 1)
  - suse-observability-initializer (replicas: 0 -> 1)
  - suse-observability-notification (replicas: 0 -> 1)
  - suse-observability-slicing (replicas: 0 -> 1)
  - suse-observability-state (replicas: 0 -> 1)
  - suse-observability-sync (replicas: 0 -> 1)
✓ Scaled up 0 statefulset(s) successfully:

@viliakov viliakov merged commit a3310e9 into main Dec 18, 2025
5 checks passed
@viliakov viliakov deleted the STAC-24013 branch December 18, 2025 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rb3ckers rb3ckers rb3ckers approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@viliakov @rb3ckers