Prune not working with S3 storage.

[iamriajul]

tested using v1 and master of docker hub image. (eg: databack/mysql-backup:1.0.0 and databack/mysql-backup:master)

Tried running the entrypoint directly.

Note: AWS Credentials are passed via Environment Variables.

DEBUG LOGS:

With command: ./entrypoint prune --target=s3://***-backup-mumbai/mysql/dumps-frequent --retention=2c --verbose=2 --debug
Output:

DEBU[0000] starting prune
INFO[0000] beginning prune                               run=fa663208-6015-44c6-9cea-8201b75a295a
DEBU[0000] pruning target s3://********-backup-mumbai/mysql/dumps-frequent  run=fa663208-6015-44c6-9cea-8201b75a295a
SDK 2024/12/16 13:22:57 DEBUG Request
GET /?list-type=2&prefix=. HTTP/1.1
Host: *********-backup-mumbai.s3.ap-south-1.amazonaws.com
User-Agent: m/E aws-sdk-go-v2/1.32.3 os/linux lang/go#1.22.9 md/GOOS#linux md/GOARCH#amd64 api/s3#1.66.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 8ed63a4b-55ec-4444-b56*****************
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ2N**************/20241216/ap-south-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=5b9631e03f7713c6871a6b**************************
X-Amz-Content-Sha256: e3b0c44298fc1c149af**********************
X-Amz-Date: 20241216T132257Z

SDK 2024/12/16 13:22:57 DEBUG Response
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 16 Dec 2024 13:22:58 GMT
Server: AmazonS3
X-Amz-Bucket-Region: ap-south-1
X-Amz-Id-2: EjcP0Vt5MLu+fWE8**************************
X-Amz-Request-Id: GCMGH*************

DEBU[0000] pruning 0 files from target &{{s3  %!s(*url.Userinfo=<nil>) *******-backup-mumbai /mysql/dumps-frequent  %!s(bool=false) %!s(bool=false)   } %!s(bool=false)    }  run=fa663208-6015-44c6-9cea-8201b75a295a

With command: ./entrypoint prune --target=s3://****-backup-mumbai/mysql/dumps-frequent --retention=1h --verbose=2 --debug
Output:

DEBU[0000] starting prune
INFO[0000] beginning prune                               run=bcad4177-cceb-41fd-8364-8cf9c6baae06
DEBU[0000] pruning target s3://********-backup-mumbai/mysql/dumps-frequent  run=bcad4177-cceb-41fd-8364-8cf9c6baae06
SDK 2024/12/16 13:27:23 DEBUG Request
GET /?list-type=2&prefix=. HTTP/1.1
Host: *********-backup-mumbai.s3.ap-south-1.amazonaws.com
User-Agent: m/E aws-sdk-go-v2/1.32.3 os/linux lang/go#1.22.9 md/GOOS#linux md/GOARCH#amd64 api/s3#1.66.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 7606e64a-a784-4f31-8a**************
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ2N2O***********/20241216/ap-south-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=b05dbd22d7426c592a3269c3497******************
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8****************
X-Amz-Date: 20241216T132723Z

SDK 2024/12/16 13:27:23 DEBUG Response
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 16 Dec 2024 13:27:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: ap-south-1
X-Amz-Id-2: vHmc4QmXfID2d1d*************************
X-Amz-Request-Id: TBD4CE*********************

DEBU[0000] pruning 0 files from target &{{s3  %!s(*url.Userinfo=<nil>) ********-backup-mumbai /mysql/dumps-frequent  %!s(bool=false) %!s(bool=false)   } %!s(bool=false)    }  run=bcad4177-cceb-41fd-8364-8cf9c6baae06

Screenshot of Backup Directory:

[deitch]

Are these debug logs from master or v1.0.0? Some of those weird log messages should have been fixed.

[iamriajul]

I ran those commands from master too just now to make sure, it's the same.

[iamriajul]

And also just tried v1.1.0 using databack/mysql-backup:1.1.0 image with no luck, the same logs are being displayed.
Logs:

DEBU[0000] starting prune
INFO[0000] beginning prune                               run=7f4831e8-7247-4674-8bd2-4ccaefa848a1
DEBU[0000] pruning target s3://********-backup-mumbai/mysql/dumps-frequent  run=7f4831e8-7247-4674-8bd2-4ccaefa848a1
SDK 2024/12/17 03:39:50 DEBUG Request
GET /?list-type=2&prefix=. HTTP/1.1
Host: ***********-backup-mumbai.s3.ap-south-1.amazonaws.com
User-Agent: m/E aws-sdk-go-v2/1.32.3 os/linux lang/go#1.22.9 md/GOOS#linux md/GOARCH#amd64 api/s3#1.66.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 2c0c5b51-d**************************
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ*********************/20241217/ap-south-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=124a252fa1cf41595****************************************
X-Amz-Content-Sha256: e3b0c442*************************************
X-Amz-Date: 20241*************

SDK 2024/12/17 03:39:50 DEBUG Response
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 17 Dec 2024 03:39:51 GMT
Server: AmazonS3
X-Amz-Bucket-Region: ap-south-1
X-Amz-Id-2: rzjuYT9HeQ4DK5Cuk***********+*********************************
X-Amz-Request-Id: VKPZD************

DEBU[0000] pruning 0 files from target &{{s3  %!s(*url.Userinfo=<nil>) *********-backup-mumbai /mysql/dumps-frequent  %!s(bool=false) %!s(bool=false)   } %!s(bool=false)    }  run=7f4831e8-7247-4674-8bd2-4ccaefa848a1

[deitch]

Successfully recreated the issue in a test (which will be part of the standard set of tests). Now to figure out why it fails.

[deitch]

With #398 merged in, this should be fixed.

If you are just running the binary, compile it from source and run it. If you are using the Docker image, you should be able to find it as databack/mysql-backup:6764cb615d5386e3060ffb8e02c7a4157df5a254

Try it and report here, please. Once you see the fix as well, we can cut a patch release.

[iamriajul]

@deitch Thanks, It's working now, It's awesome. Thanks again.

[iamriajul]

Reopening the issue, Just noticed it's not respecting the RETENTION COUNT, I've this configuration, but the s3 storage only contains last 1 day backup.

[iamriajul]

I've two configuration, one is running hourly and keeping 24 count (it is working perfectly), but I've another which runs daily and it should keep last 31 backup (it's not working, it's only keeping last 1 backup).

My Compose:

version: '3.8'

services:
  mysql-backup-hourly:
    image: databack/mysql-backup:6764cb615d5386e3060ffb8e02c7a4157df5a254
    environment:
      DB_SERVER: mysql
      DB_USER: root
      DB_PASS: ${MYSQL_ROOT_PASSWORD:?err}
      AWS_DEFAULT_REGION: ${MYSQL_BACKUP_AWS_DEFAULT_REGION:?err}
      AWS_ACCESS_KEY_ID: ${MYSQL_BACKUP_AWS_ACCESS_KEY_ID:?err}
      AWS_SECRET_ACCESS_KEY: ${MYSQL_BACKUP_AWS_SECRET_ACCESS_KEY:?err}
      DB_DUMP_TARGET: s3://${MYSQL_BACKUP_S3_BUCKET:?err}/mysql/dumps-hourly
      DB_DUMP_CRON: '0 * * * *' # Every hour
      DB_DUMP_RETENTION: 24c
      NICE: 'true' # Don't be use too much CPU or RAM, just be nice.
      COMPRESSION: gzip
      TZ: UTC
    networks:
      - mysql_encrypted_network
    command: dump
    depends_on:
      - services_mysql
    deploy:
      resources:
        limits:
          memory: 8G
          cpus: '0.5'
      mode: global
      placement:
        constraints:
          - node.labels.role == primary
      update_config:
        order: stop-first
      rollback_config:
        order: stop-first

  mysql-backup-daily:
    image: databack/mysql-backup:6764cb615d5386e3060ffb8e02c7a4157df5a254
    environment:
      DB_SERVER: mysql
      DB_USER: root
      DB_PASS: ${MYSQL_ROOT_PASSWORD:?err}
      AWS_DEFAULT_REGION: ${MYSQL_BACKUP_AWS_DEFAULT_REGION:?err}
      AWS_ACCESS_KEY_ID: ${MYSQL_BACKUP_AWS_ACCESS_KEY_ID:?err}
      AWS_SECRET_ACCESS_KEY: ${MYSQL_BACKUP_AWS_SECRET_ACCESS_KEY:?err}
      DB_DUMP_TARGET: s3://${MYSQL_BACKUP_S3_BUCKET:?err}/mysql/dumps-daily
      DB_DUMP_CRON: '30 22 * * *' # At 22:30
      DB_DUMP_RETENTION: 31c
      NICE: 'true' # Don't be use too much CPU or RAM, just be nice.
      COMPRESSION: gzip
      TZ: UTC
    networks:
      - mysql_encrypted_network
    command: dump
    depends_on:
      - services_mysql
    deploy:
      resources:
        limits:
          memory: 8G
          cpus: '0.5'
      mode: global
      placement:
        constraints:
          - node.labels.role == primary
      update_config:
        order: stop-first
      rollback_config:
        order: stop-first

  mysql-backup-monthly:
    image: databack/mysql-backup:6764cb615d5386e3060ffb8e02c7a4157df5a254
    environment:
      DB_SERVER: mysql
      DB_USER: root
      DB_PASS: ${MYSQL_ROOT_PASSWORD:?err}
      AWS_DEFAULT_REGION: ${MYSQL_BACKUP_AWS_DEFAULT_REGION:?err}
      AWS_ACCESS_KEY_ID: ${MYSQL_BACKUP_AWS_ACCESS_KEY_ID:?err}
      AWS_SECRET_ACCESS_KEY: ${MYSQL_BACKUP_AWS_SECRET_ACCESS_KEY:?err}
      DB_DUMP_TARGET: s3://${MYSQL_BACKUP_S3_BUCKET:?err}/mysql/dumps-monthly
      DB_DUMP_CRON: '0 0 1 * *' # At 00:00 on day-of-month 1
      NICE: 'true' # Don't be use too much CPU or RAM, just be nice.
      COMPRESSION: gzip
      TZ: UTC
    networks:
      - mysql_encrypted_network
    command: dump
    depends_on:
      - services_mysql
    deploy:
      resources:
        limits:
          memory: 8G
          cpus: '0.5'
      mode: global
      placement:
        constraints:
          - node.labels.role == primary
      update_config:
        order: stop-first
      rollback_config:
        order: stop-first

networks:
  mysql_encrypted_network:
    external: true

[deitch]

This is helpful. Will see if I can figure it out, but will take some days.

[deitch]

@iamriajul what is the debug log for the one that is only keeping the last one, rather than the last 31?

[iamriajul]

As a workaround I've implemented a multi-bucket approach, where each bucket is only attached with only one service, now I've 3 separate buckets (hourly, daily, monthly).

[deitch]

Can you share the debug log for the one that is only keeping the last one? We prefer things work, rather than workaround.

[iamriajul]

Another strange thing happened when I was using this version: databack/mysql-backup:6764cb615d5386e3060ffb8e02c7a4157df5a254 then pruning was working for one, but after upgrading to 1.1.0 , now pruning is not working.

I also want to help you debug, but sorry to say I'm barely getting time, as I've switched Job, and I'm giving only my spare time to the old company.