Infinite attempt retryer causes memory to grow unbounded

[injust]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

I'm running a script that calls EC2 RunInstances with an infinite-attempt retryer. When the retryer retries repeatedly, the memory usage grows unbounded.

When the retryer fails quickly (e.g. because MaxSpotInstanceCountExceeded or SpotMaxPriceTooLow are not retryable), there is no increase in memory usage.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I expect memory usage not to grow unbounded when retrying.

Current Behavior

fish: Job 1, './ec2-run-instances --launch-te…' terminated by signal SIGKILL (Forced quit)

Reproduction Steps

package main

import (
	"context"
	"errors"
	"log"
	"os"
	"time"

	flag "github.com/spf13/pflag"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/aws/ratelimit"
	"github.com/aws/aws-sdk-go-v2/aws/retry"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/ec2"
	"github.com/aws/aws-sdk-go-v2/service/ec2/types"
	"github.com/aws/smithy-go"
)

var (
	count          = flag.Int32P("count", "n", 1, "Number of instances to launch simultaneously")
	interval       = flag.Duration("interval", 1*time.Second, "Interval between instance launch attempts")
	launchTemplate = flag.String("launch-template", "", "EC2 launch template name")
)

func InfiniteRetryer() aws.Retryer {
	retryer := retry.NewStandard(func(o *retry.StandardOptions) {
		o.Backoff = retry.BackoffDelayerFunc(func(attempt int, err error) (time.Duration, error) {
			return *interval, nil
		})
		o.RateLimiter = ratelimit.None
		o.Retryables = append(o.Retryables, retry.IsErrorRetryableFunc(
			func(err error) aws.Ternary {
				var apiErr smithy.APIError
				if errors.As(err, &apiErr) {
					switch apiErr.ErrorCode() {
					case "MaxSpotInstanceCountExceeded", "SpotMaxPriceTooLow":
						return aws.TrueTernary
					}
				}
				return aws.UnknownTernary
			},
		))
	})
	// NOTE(https://github.com/aws/aws-sdk-go-v2/issues/3193): Using functional option (`o.MaxAttempts = 0`) sets MaxAttempts to the default
	return retry.AddWithMaxAttempts(retryer, 0)
}

func RunInstances(ctx context.Context, client *ec2.Client, launchTemplate string) (*ec2.RunInstancesOutput, error) {
	input := &ec2.RunInstancesInput{
		LaunchTemplate: &types.LaunchTemplateSpecification{
			LaunchTemplateName: aws.String(launchTemplate),
		},
		MinCount: aws.Int32(1),
		MaxCount: count,
	}
	return client.RunInstances(ctx, input)
}

func main() {
	flag.Parse()
	if *launchTemplate == "" {
		flag.Usage()
		os.Exit(1)
	}

	ctx := context.Background()
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRetryer(InfiniteRetryer),
	)
	if err != nil {
		log.Fatalf("Failed to load config: %v", err)
	}
	client := ec2.NewFromConfig(cfg)

	for {
		resp, err := RunInstances(ctx, client, *launchTemplate)
		if err != nil {
			log.Printf("Launch failed: %v", err)
		} else {
			for _, instance := range resp.Instances {
				log.Printf("Launched %s instance in %s: %s", instance.InstanceType, *instance.Placement.AvailabilityZone, *instance.InstanceId)
			}
		}

		time.Sleep(*interval)
	}
}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

require (
        github.com/aws/aws-sdk-go-v2 v1.40.0
        github.com/aws/aws-sdk-go-v2/config v1.32.2
        github.com/aws/aws-sdk-go-v2/service/ec2 v1.275.0
)

require (
        github.com/aws/aws-sdk-go-v2/credentials v1.19.2 // indirect
        github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14 // indirect
        github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14 // indirect
        github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.14 // indirect
        github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
        github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3 // indirect
        github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.14 // indirect
        github.com/aws/aws-sdk-go-v2/service/signin v1.0.2 // indirect
        github.com/aws/aws-sdk-go-v2/service/sso v1.30.5 // indirect
        github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.10 // indirect
        github.com/aws/aws-sdk-go-v2/service/sts v1.41.2 // indirect
        github.com/aws/smithy-go v1.23.2 // indirect
)

Compiler and Version used

go version go1.24.9 linux/arm64

Operating System and version

Amazon Linux 2023.9.20251117

[injust]

Upon further thought, this probably isn't a regression. I hadn't noticed any SIGKILLs before, but I think I'm just consistently getting retryable errors in certain regions.

EDIT: After adding more error codes as retryable, I'm getting the same elevated memory usage in all regions.

Some napkin math suggests something on the order of 40k requests uses close to 1 GB of memory.

[wty-Bryant]

Could you provide the reproduction step that shows the memory usage grows unbounded? I didn't see related code in your snippet

[injust]

I've already provided a description of the issue and a reproduction. What are you looking for, specifically?

You can build the script and run it yourself to see. Use a launch template that will always fail, e.g. max spot price = $0.01. The retryer will loop infinitely, and the process' memory usage will keep growing.

[lucix-aws]

What tool did you use to measure memory usage? I believe that was the intent of Tianyi's question.