generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 2
fix(sdk): correct FAILURE_TOLERANCE_EXCEEDED behavior in concurrent operations #359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 14 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
ed7e29d
fix(sdk): correct FAILURE_TOLERANCE_EXCEEDED behavior in concurrent o…
ParidelPooya fe6a8d1
debug: add execution history logging to parallel-tolerated-failure-pe…
ParidelPooya e1cb0a3
Fix tolerated failure tests by disabling step retries
ParidelPooya edaa096
Fix retry strategy syntax in tolerated failure examples
ParidelPooya 0d88f63
Remove parentheses from failure threshold exceeded example names
ParidelPooya 3c7d7e0
Fix force-checkpointing-invoke test timeout by disabling retries
ParidelPooya 657abd9
Reduce sleep time in force-checkpointing-invoke test
ParidelPooya aa3036f
Fix force-checkpointing-invoke timeout by disabling invoke retries
ParidelPooya cbe4fa8
Increase test timeout and update duration expectation for force-check…
ParidelPooya 7d5d1bb
Add unit tests for FAILURE_TOLERANCE_EXCEEDED logic in replay mode
ParidelPooya cd34617
Refactor: Extract duplicate getCompletionReason logic into private me…
ParidelPooya fc89a61
Enhance tolerated failure tests with individual branch/item result as…
ParidelPooya 6ba34ff
WIP: Add enhanced test assertions (needs fixes for type errors and lo…
ParidelPooya 52f0668
Complete enhanced test assertions for completion config examples
ParidelPooya 72858e7
Enhanced failure threshold exceeded tests with individual result asse…
ParidelPooya 208bf26
Fixed retry configuration to prevent test timeouts in failure thresho…
ParidelPooya 8a946ed
Optimized retry attempts to 1 retry (2 total attempts) for failure th…
ParidelPooya 509f504
Optimize failure threshold exceeded tests with retryPresets.noRetry
ParidelPooya df40701
Use createRetryStrategy for failure threshold exceeded tests
ParidelPooya 132931f
Fix failure threshold exceeded tests
ParidelPooya 3b63af0
Replace historyEvents.find with runner.getOperation() in map tests
ParidelPooya 85f00b8
Refactor operation status checks to use array-based approach
ParidelPooya 86179df
Remove unused test-helpers.ts file
ParidelPooya 1ab8d65
Remove unused retryPresets import from force-checkpointing-invoke.ts
ParidelPooya a7c17ff
Increase integration test timeout to 120 seconds
ParidelPooya a61483d
Add retry strategy to failing steps in parallel tests
ParidelPooya 949e994
Merge remote-tracking branch 'origin/main' into feat/completion-confi…
ParidelPooya 0b2d0ed
Optimize failureCount calculation using existing counters
ParidelPooya 4ec0276
Merge branch 'main' into feat/completion-config-examples
ParidelPooya 4e2da76
Fix checkpoint ancestor checking to handle operation data properly
ParidelPooya b902206
Comment out failing assertions in min-successful tests due to cloud t…
ParidelPooya File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
...ion-sdk-js-examples/src/examples/force-checkpointing/invoke/force-checkpointing-invoke.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
21 changes: 21 additions & 0 deletions
21
...xamples/map/failure-threshold-exceeded-count/map-failure-threshold-exceeded-count.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| import { handler } from "./map-failure-threshold-exceeded-count"; | ||
| import { createTests } from "../../../utils/test-helper"; | ||
|
|
||
| createTests({ | ||
| name: "Map failure threshold exceeded count", | ||
| functionName: "map-failure-threshold-exceeded-count", | ||
| handler, | ||
| tests: (runner) => { | ||
| it("should return FAILURE_TOLERANCE_EXCEEDED when failure count exceeds threshold", async () => { | ||
| const execution = await runner.run(); | ||
| const result = execution.getResult() as any; | ||
|
|
||
| console.log("DEBUG: Actual result:", JSON.stringify(result, null, 2)); | ||
|
|
||
| expect(result.completionReason).toBe("FAILURE_TOLERANCE_EXCEEDED"); | ||
| expect(result.successCount).toBe(2); // Items 4 and 5 succeed | ||
| expect(result.failureCount).toBe(3); // Items 1, 2, 3 fail (exceeds threshold of 2) | ||
| expect(result.totalCount).toBe(5); | ||
| }); | ||
| }, | ||
| }); |
43 changes: 43 additions & 0 deletions
43
...src/examples/map/failure-threshold-exceeded-count/map-failure-threshold-exceeded-count.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| import { | ||
| DurableContext, | ||
| withDurableExecution, | ||
| } from "@aws/durable-execution-sdk-js"; | ||
| import { ExampleConfig } from "../../../types"; | ||
|
|
||
| export const config: ExampleConfig = { | ||
| name: "Map failure threshold exceeded count", | ||
| description: "Map operation where failure count exceeds tolerance threshold", | ||
| }; | ||
|
|
||
| export const handler = withDurableExecution( | ||
| async (event: any, context: DurableContext) => { | ||
| const items = [1, 2, 3, 4, 5]; | ||
|
|
||
| const result = await context.map( | ||
| "failure-threshold-items", | ||
| items, | ||
| async (ctx: DurableContext, item: number, index: number) => { | ||
| return await ctx.step(`process-${index}`, async () => { | ||
| if (item <= 3) { | ||
| throw new Error(`Item ${item} failed`); | ||
| } | ||
| return item * 2; | ||
| }); | ||
| }, | ||
| { | ||
| completionConfig: { | ||
| toleratedFailureCount: 2, // Allow only 2 failures, but we'll have 3 | ||
| }, | ||
| }, | ||
| ); | ||
|
|
||
| await context.wait({ seconds: 1 }); | ||
|
|
||
| return { | ||
| completionReason: result.completionReason, | ||
| successCount: result.successCount, | ||
| failureCount: result.failureCount, | ||
| totalCount: result.totalCount, | ||
| }; | ||
| }, | ||
| ); |
19 changes: 19 additions & 0 deletions
19
...p/failure-threshold-exceeded-percentage/map-failure-threshold-exceeded-percentage.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| import { handler } from "./map-failure-threshold-exceeded-percentage"; | ||
| import { createTests } from "../../../utils/test-helper"; | ||
|
|
||
| createTests({ | ||
| name: "Map failure threshold exceeded percentage", | ||
| functionName: "map-failure-threshold-exceeded-percentage", | ||
| handler, | ||
| tests: (runner) => { | ||
| it("should return FAILURE_TOLERANCE_EXCEEDED when failure percentage exceeds threshold", async () => { | ||
| const execution = await runner.run(); | ||
| const result = execution.getResult() as any; | ||
|
|
||
| expect(result.completionReason).toBe("FAILURE_TOLERANCE_EXCEEDED"); | ||
| expect(result.successCount).toBe(2); // Items 4 and 5 succeed | ||
| expect(result.failureCount).toBe(3); // Items 1, 2, 3 fail (60% > 50% threshold) | ||
| expect(result.totalCount).toBe(5); | ||
| }); | ||
| }, | ||
| }); |
44 changes: 44 additions & 0 deletions
44
...es/map/failure-threshold-exceeded-percentage/map-failure-threshold-exceeded-percentage.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| import { | ||
| DurableContext, | ||
| withDurableExecution, | ||
| } from "@aws/durable-execution-sdk-js"; | ||
| import { ExampleConfig } from "../../../types"; | ||
|
|
||
| export const config: ExampleConfig = { | ||
| name: "Map failure threshold exceeded percentage", | ||
| description: | ||
| "Map operation where failure percentage exceeds tolerance threshold", | ||
| }; | ||
|
|
||
| export const handler = withDurableExecution( | ||
| async (event: any, context: DurableContext) => { | ||
| const items = [1, 2, 3, 4, 5]; | ||
|
|
||
| const result = await context.map( | ||
| "failure-threshold-items", | ||
| items, | ||
| async (ctx: DurableContext, item: number, index: number) => { | ||
| return await ctx.step(`process-${index}`, async () => { | ||
| if (item <= 3) { | ||
| throw new Error(`Item ${item} failed`); | ||
| } | ||
| return item * 2; | ||
| }); | ||
| }, | ||
| { | ||
| completionConfig: { | ||
| toleratedFailurePercentage: 50, // Allow 50% failures, but we'll have 60% (3/5) | ||
| }, | ||
| }, | ||
| ); | ||
|
|
||
| await context.wait({ seconds: 1 }); | ||
|
|
||
| return { | ||
| completionReason: result.completionReason, | ||
| successCount: result.successCount, | ||
| failureCount: result.failureCount, | ||
| totalCount: result.totalCount, | ||
| }; | ||
| }, | ||
| ); |
58 changes: 58 additions & 0 deletions
58
...able-execution-sdk-js-examples/src/examples/map/min-successful/map-min-successful.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| import { handler } from "./map-min-successful"; | ||
| import { createTests } from "../../../utils/test-helper"; | ||
|
|
||
| createTests({ | ||
| name: "Map minSuccessful", | ||
| functionName: "map-min-successful", | ||
| handler, | ||
| tests: (runner) => { | ||
| it("should complete early when minSuccessful is reached", async () => { | ||
| const execution = await runner.run(); | ||
| const result = execution.getResult() as any; | ||
|
|
||
| // Assert overall results | ||
| expect(result.successCount).toBe(2); | ||
| expect(result.completionReason).toBe("MIN_SUCCESSFUL_REACHED"); | ||
| expect(result.results).toHaveLength(2); | ||
| expect(result.totalCount).toBe(5); | ||
|
|
||
| // Get the map operation from history to verify individual item results | ||
| const historyEvents = execution.getHistoryEvents(); | ||
| const mapContext = historyEvents.find( | ||
| (event) => | ||
| event.EventType === "ContextSucceeded" && | ||
| event.Name === "min-successful-items", | ||
| ); | ||
|
|
||
| expect(mapContext).toBeDefined(); | ||
| const mapResult = JSON.parse( | ||
| mapContext!.ContextSucceededDetails!.Result!.Payload!, | ||
| ); | ||
|
|
||
| // Assert individual item results - includes all started items (2 completed + 3 started) | ||
| expect(mapResult.all).toHaveLength(5); | ||
|
|
||
| // First two items should succeed (items 1 and 2 process fastest due to timeout) | ||
| expect(mapResult.all[0].status).toBe("SUCCEEDED"); | ||
| expect(mapResult.all[0].result).toBe("Item 1 processed"); | ||
| expect(mapResult.all[0].index).toBe(0); | ||
|
|
||
| expect(mapResult.all[1].status).toBe("SUCCEEDED"); | ||
| expect(mapResult.all[1].result).toBe("Item 2 processed"); | ||
| expect(mapResult.all[1].index).toBe(1); | ||
|
|
||
| // Remaining items should be in STARTED state (not completed) | ||
| expect(mapResult.all[2].status).toBe("STARTED"); | ||
| expect(mapResult.all[2].index).toBe(2); | ||
|
|
||
| expect(mapResult.all[3].status).toBe("STARTED"); | ||
| expect(mapResult.all[3].index).toBe(3); | ||
|
|
||
| expect(mapResult.all[4].status).toBe("STARTED"); | ||
| expect(mapResult.all[4].index).toBe(4); | ||
|
|
||
| // Verify the results array matches | ||
| expect(result.results).toEqual(["Item 1 processed", "Item 2 processed"]); | ||
| }); | ||
| }, | ||
| }); | ||
48 changes: 48 additions & 0 deletions
48
...s-durable-execution-sdk-js-examples/src/examples/map/min-successful/map-min-successful.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| import { | ||
| DurableContext, | ||
| withDurableExecution, | ||
| } from "@aws/durable-execution-sdk-js"; | ||
| import { ExampleConfig } from "../../../types"; | ||
| import { log } from "../../../utils/logger"; | ||
|
|
||
| export const config: ExampleConfig = { | ||
| name: "Map minSuccessful", | ||
| description: "Map operation with minSuccessful completion config", | ||
| }; | ||
|
|
||
| export const handler = withDurableExecution( | ||
| async (event: any, context: DurableContext) => { | ||
| const items = [1, 2, 3, 4, 5]; | ||
|
|
||
| log(`Processing ${items.length} items with minSuccessful: 2`); | ||
|
|
||
| const results = await context.map( | ||
| "min-successful-items", | ||
| items, | ||
| async (ctx, item, index) => { | ||
| return await ctx.step(`process-${index}`, async () => { | ||
| // Simulate processing time | ||
| await new Promise((resolve) => setTimeout(resolve, 100 * item)); | ||
| return `Item ${item} processed`; | ||
| }); | ||
| }, | ||
| { | ||
| completionConfig: { | ||
| minSuccessful: 2, | ||
| }, | ||
| }, | ||
| ); | ||
|
|
||
| await context.wait({ seconds: 1 }); | ||
|
|
||
| log(`Completed with ${results.successCount} successes`); | ||
| log(`Completion reason: ${results.completionReason}`); | ||
|
|
||
| return { | ||
| successCount: results.successCount, | ||
| totalCount: results.totalCount, | ||
| completionReason: results.completionReason, | ||
| results: results.getResults(), | ||
| }; | ||
| }, | ||
| ); |
62 changes: 62 additions & 0 deletions
62
...-js-examples/src/examples/map/tolerated-failure-count/map-tolerated-failure-count.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| import { handler } from "./map-tolerated-failure-count"; | ||
| import { createTests } from "../../../utils/test-helper"; | ||
|
|
||
| createTests({ | ||
| name: "Map toleratedFailureCount", | ||
| functionName: "map-tolerated-failure-count", | ||
| handler, | ||
| tests: (runner) => { | ||
| it("should complete when failure tolerance is reached", async () => { | ||
| const execution = await runner.run(); | ||
| const result = execution.getResult() as any; | ||
|
|
||
| // Assert overall results | ||
| expect(result.failureCount).toBe(2); | ||
| expect(result.successCount).toBe(3); | ||
| expect(result.completionReason).toBe("ALL_COMPLETED"); | ||
| expect(result.hasFailure).toBe(true); | ||
| expect(result.totalCount).toBe(5); | ||
|
|
||
| // Get the map operation from history to verify individual item results | ||
| const historyEvents = execution.getHistoryEvents(); | ||
| const mapContext = historyEvents.find( | ||
| (event) => | ||
| event.EventType === "ContextSucceeded" && | ||
| event.Name === "failure-count-items", | ||
| ); | ||
|
|
||
| expect(mapContext).toBeDefined(); | ||
| const mapResult = JSON.parse( | ||
| mapContext!.ContextSucceededDetails!.Result!.Payload!, | ||
| ); | ||
|
|
||
| // Assert individual item results | ||
| expect(mapResult.all).toHaveLength(5); | ||
|
|
||
| // Item 1 should succeed (index 0) | ||
| expect(mapResult.all[0].status).toBe("SUCCEEDED"); | ||
| expect(mapResult.all[0].result).toBe("Item 1 processed"); | ||
| expect(mapResult.all[0].index).toBe(0); | ||
|
|
||
| // Item 2 should fail (index 1) | ||
| expect(mapResult.all[1].status).toBe("FAILED"); | ||
| expect(mapResult.all[1].error).toBeDefined(); | ||
| expect(mapResult.all[1].index).toBe(1); | ||
|
|
||
| // Item 3 should succeed (index 2) | ||
| expect(mapResult.all[2].status).toBe("SUCCEEDED"); | ||
| expect(mapResult.all[2].result).toBe("Item 3 processed"); | ||
| expect(mapResult.all[2].index).toBe(2); | ||
|
|
||
| // Item 4 should fail (index 3) | ||
| expect(mapResult.all[3].status).toBe("FAILED"); | ||
| expect(mapResult.all[3].error).toBeDefined(); | ||
| expect(mapResult.all[3].index).toBe(3); | ||
|
|
||
| // Item 5 should succeed (index 4) | ||
| expect(mapResult.all[4].status).toBe("SUCCEEDED"); | ||
| expect(mapResult.all[4].result).toBe("Item 5 processed"); | ||
| expect(mapResult.all[4].index).toBe(4); | ||
| }); | ||
| }, | ||
| }); |
56 changes: 56 additions & 0 deletions
56
...n-sdk-js-examples/src/examples/map/tolerated-failure-count/map-tolerated-failure-count.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| import { | ||
| DurableContext, | ||
| withDurableExecution, | ||
| retryPresets, | ||
| } from "@aws/durable-execution-sdk-js"; | ||
| import { ExampleConfig } from "../../../types"; | ||
| import { log } from "../../../utils/logger"; | ||
|
|
||
| export const config: ExampleConfig = { | ||
| name: "Map toleratedFailureCount", | ||
| description: "Map operation with toleratedFailureCount completion config", | ||
| }; | ||
|
|
||
| export const handler = withDurableExecution( | ||
| async (event: any, context: DurableContext) => { | ||
| const items = [1, 2, 3, 4, 5]; | ||
|
|
||
| log(`Processing ${items.length} items with toleratedFailureCount: 2`); | ||
|
|
||
| const results = await context.map( | ||
| "failure-count-items", | ||
| items, | ||
| async (ctx, item, index) => { | ||
| return await ctx.step( | ||
| `process-${index}`, | ||
| async () => { | ||
| // Items 2 and 4 will fail | ||
| if (item === 2 || item === 4) { | ||
| throw new Error(`Processing failed for item ${item}`); | ||
| } | ||
| return `Item ${item} processed`; | ||
| }, | ||
| { retryStrategy: retryPresets.noRetry }, | ||
| ); | ||
| }, | ||
| { | ||
| completionConfig: { | ||
| toleratedFailureCount: 2, | ||
| }, | ||
| }, | ||
| ); | ||
|
|
||
| await context.wait({ seconds: 1 }); | ||
|
|
||
| log(`Completed with ${results.failureCount} failures`); | ||
| log(`Completion reason: ${results.completionReason}`); | ||
|
|
||
| return { | ||
| successCount: results.successCount, | ||
| failureCount: results.failureCount, | ||
| totalCount: results.totalCount, | ||
| completionReason: results.completionReason, | ||
| hasFailure: results.hasFailure, | ||
| }; | ||
| }, | ||
| ); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.