Skip to content

Commit 3281f06

Browse files
jeromevdlphipag
jeromevdl
and
phipag
authored
feat(v2): batch validation with partial failure (#1621)
* validation of sqs / kinesis batches with partial failures * cleanup jdk8 stuff + slf4j simple logs for tests * fix sonar potential NPE * documentation * update mockito (and bytebuddy) for java21 compat * comment * fix mvn pom * Preparing for checkstyle removal. Keeping the checkstyle.xml for backwards compatibility. --------- Co-authored-by: Philipp Page <pagejep@amazon.com> Co-authored-by: Philipp Page <github@philipp.page>
1 parent 7ca1b7e commit 3281f06

File tree

29 files changed

+757
-359
lines changed

29 files changed

+757
-359
lines changed

‎docs/utilities/validation.md‎

Lines changed: 28 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ This utility provides JSON Schema validation for payloads held within events and
88
**Key features**
99

1010
* Validate incoming events and responses
11-
* Built-in validation for most common events (API Gateway, SNS, SQS, ...)
11+
* Built-in validation for most common events (API Gateway, SNS, SQS, ...) and support for partial batch failures (SQS, Kinesis)
1212
* JMESPath support validate only a sub part of the event
1313

1414
## Install
@@ -100,10 +100,15 @@ The validator is configured to enable format assertions by default even for 2019
100100
`@Validation` annotation is used to validate either inbound events or functions' response.
101101

102102
It will fail fast if an event or response doesn't conform with given JSON Schema. For most type of events a `ValidationException` will be thrown.
103+
103104
For API gateway events associated with REST APIs and HTTP APIs - `APIGatewayProxyRequestEvent` and `APIGatewayV2HTTPEvent` - the `@Validation`
104105
annotation will build and return a custom 400 / "Bad Request" response, with a body containing the validation errors. This saves you from having
105106
to catch the validation exception and map it back to a meaningful user error yourself.
106107

108+
For SQS and Kinesis events - `SQSEvent` and `KinesisEvent`- the `@Validation` annotation will add the invalid messages
109+
to the batch item failures list in the response, respectively `SQSBatchResponse` and `StreamsEventResponse`
110+
and removed from the event so that you do not process them within the handler.
111+
107112
While it is easier to specify a json schema file in the classpath (using the notation `"classpath:/path/to/schema.json"`), you can also provide a JSON String containing the schema.
108113

109114
=== "MyFunctionHandler.java"
@@ -160,31 +165,31 @@ For the following events and responses, the Validator will automatically perform
160165

161166
** Events **
162167

163-
Type of event | Class | Path to content |
164-
------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------
165-
API Gateway REST | APIGatewayProxyRequestEvent | `body`
166-
API Gateway HTTP | APIGatewayV2HTTPEvent | `body`
167-
Application Load Balancer | ApplicationLoadBalancerRequestEvent | `body`
168-
Cloudformation Custom Resource | CloudFormationCustomResourceEvent | `resourceProperties`
169-
CloudWatch Logs | CloudWatchLogsEvent | `awslogs.powertools_base64_gzip(data)`
170-
EventBridge / Cloudwatch | ScheduledEvent | `detail`
171-
Kafka | KafkaEvent | `records[*][*].value`
172-
Kinesis | KinesisEvent | `Records[*].kinesis.powertools_base64(data)`
173-
Kinesis Firehose | KinesisFirehoseEvent | `Records[*].powertools_base64(data)`
174-
Kinesis Analytics from Firehose | KinesisAnalyticsFirehoseInputPreprocessingEvent | `Records[*].powertools_base64(data)`
175-
Kinesis Analytics from Streams | KinesisAnalyticsStreamsInputPreprocessingEvent | `Records[*].powertools_base64(data)`
176-
SNS | SNSEvent | `Records[*].Sns.Message`
177-
SQS | SQSEvent | `Records[*].body`
168+
| Type of event | Class | Path to content |
169+
|---------------------------------|-------------------------------------------------|----------------------------------------------|
170+
| API Gateway REST | APIGatewayProxyRequestEvent | `body` |
171+
| API Gateway HTTP | APIGatewayV2HTTPEvent | `body` |
172+
| Application Load Balancer | ApplicationLoadBalancerRequestEvent | `body` |
173+
| Cloudformation Custom Resource | CloudFormationCustomResourceEvent | `resourceProperties` |
174+
| CloudWatch Logs | CloudWatchLogsEvent | `awslogs.powertools_base64_gzip(data)` |
175+
| EventBridge / Cloudwatch | ScheduledEvent | `detail` |
176+
| Kafka | KafkaEvent | `records[*][*].value` |
177+
| Kinesis | KinesisEvent | `Records[*].kinesis.powertools_base64(data)` |
178+
| Kinesis Firehose | KinesisFirehoseEvent | `Records[*].powertools_base64(data)` |
179+
| Kinesis Analytics from Firehose | KinesisAnalyticsFirehoseInputPreprocessingEvent | `Records[*].powertools_base64(data)` |
180+
| Kinesis Analytics from Streams | KinesisAnalyticsStreamsInputPreprocessingEvent | `Records[*].powertools_base64(data)` |
181+
| SNS | SNSEvent | `Records[*].Sns.Message` |
182+
| SQS | SQSEvent | `Records[*].body` |
178183

179184
** Responses **
180185

181-
Type of response | Class | Path to content (envelope)
182-
------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------
183-
API Gateway REST | APIGatewayProxyResponseEvent} | `body`
184-
API Gateway HTTP | APIGatewayV2HTTPResponse} | `body`
185-
API Gateway WebSocket | APIGatewayV2WebSocketResponse} | `body`
186-
Load Balancer | ApplicationLoadBalancerResponseEvent} | `body`
187-
Kinesis Analytics | KinesisAnalyticsInputPreprocessingResponse} | `Records[*].powertools_base64(data)``
186+
| Type of response | Class | Path to content (envelope) |
187+
|-----------------------|---------------------------------------------|---------------------------------------|
188+
| API Gateway REST | APIGatewayProxyResponseEvent} | `body` |
189+
| API Gateway HTTP | APIGatewayV2HTTPResponse} | `body` |
190+
| API Gateway WebSocket | APIGatewayV2WebSocketResponse} | `body` |
191+
| Load Balancer | ApplicationLoadBalancerResponseEvent} | `body` |
192+
| Kinesis Analytics | KinesisAnalyticsInputPreprocessingResponse} | `Records[*].powertools_base64(data)`` |
188193

189194
## Custom events and responses
190195

‎pom.xml‎

Lines changed: 13 additions & 94 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@
101101
<aws.sdk.v1.version>1.12.781</aws.sdk.v1.version>
102102
<versions-maven-plugin.version>2.18.0</versions-maven-plugin.version>
103103
<elastic.version>1.6.0</elastic.version>
104+
<mockito.version>5.12.0</mockito.version>
104105

105106
<!-- As we have a .mvn directory at the root of the project, this will evaluate to the root directory
106107
regardless of where maven is run - sub-module, or root. -->
@@ -317,6 +318,12 @@
317318
</exclusion>
318319
</exclusions>
319320
</dependency>
321+
<dependency>
322+
<groupId>org.slf4j</groupId>
323+
<artifactId>slf4j-simple</artifactId>
324+
<version>${slf4j.version}</version>
325+
<scope>test</scope>
326+
</dependency>
320327
<dependency>
321328
<groupId>org.skyscreamer</groupId>
322329
<artifactId>jsonassert</artifactId>
@@ -328,6 +335,12 @@
328335
<artifactId>aspectjtools</artifactId>
329336
<version>${aspectj.version}</version>
330337
</dependency>
338+
<dependency>
339+
<groupId>org.mockito</groupId>
340+
<artifactId>mockito-core</artifactId>
341+
<version>${mockito.version}</version>
342+
<scope>test</scope>
343+
</dependency>
331344
<dependency>
332345
<groupId>com.amazonaws</groupId>
333346
<artifactId>aws-lambda-java-tests</artifactId>
@@ -603,100 +616,6 @@
603616
</plugins>
604617
</build>
605618
</profile>
606-
<profile>
607-
<id>olderThanJdk11</id>
608-
<activation>
609-
<jdk>(,11)</jdk>
610-
</activation>
611-
<properties>
612-
<!-- mockito 5+ is not compatible anymore with java < 11 -->
613-
<mockito.version>4.11.0</mockito.version>
614-
</properties>
615-
<dependencies>
616-
<dependency>
617-
<groupId>org.mockito</groupId>
618-
<artifactId>mockito-core</artifactId>
619-
<version>${mockito.version}</version>
620-
<scope>test</scope>
621-
</dependency>
622-
<dependency>
623-
<groupId>org.mockito</groupId>
624-
<artifactId>mockito-inline</artifactId>
625-
<version>${mockito.version}</version>
626-
<scope>test</scope>
627-
</dependency>
628-
</dependencies>
629-
</profile>
630-
<profile>
631-
<id>newerThanJdk11</id>
632-
<activation>
633-
<jdk>[11,)</jdk>
634-
</activation>
635-
<properties>
636-
<mockito.version>5.6.0</mockito.version>
637-
</properties>
638-
<dependencies>
639-
<!-- since mockito 5.3, no need to have mockito-inline -->
640-
<dependency>
641-
<groupId>org.mockito</groupId>
642-
<artifactId>mockito-core</artifactId>
643-
<version>${mockito.version}</version>
644-
<scope>test</scope>
645-
</dependency>
646-
</dependencies>
647-
<build>
648-
<plugins>
649-
<plugin>
650-
<groupId>org.apache.maven.plugins</groupId>
651-
<artifactId>maven-surefire-plugin</artifactId>
652-
<configuration>
653-
<systemPropertyVariables>
654-
<!-- TODO: remove when updating mockito / bytebuddy with Java21 compat -->
655-
<net.bytebuddy.experimental>true</net.bytebuddy.experimental>
656-
</systemPropertyVariables>
657-
</configuration>
658-
</plugin>
659-
</plugins>
660-
</build>
661-
</profile>
662-
<profile>
663-
<id>newerThanJdk8</id>
664-
<activation>
665-
<jdk>[9,)</jdk>
666-
</activation>
667-
<build>
668-
<plugins>
669-
<plugin>
670-
<!-- we run checkstyle only on Java 11, no need to run it for all JDK, it's just code style -->
671-
<groupId>org.apache.maven.plugins</groupId>
672-
<artifactId>maven-checkstyle-plugin</artifactId>
673-
<version>3.6.0</version>
674-
<configuration>
675-
<propertyExpansion>basedir=${project.rootdir}</propertyExpansion>
676-
<configLocation>checkstyle.xml</configLocation>
677-
<consoleOutput>true</consoleOutput>
678-
<failsOnError>true</failsOnError>
679-
<linkXRef>false</linkXRef>
680-
</configuration>
681-
<!-- does not work without this dependency -->
682-
<dependencies>
683-
<dependency>
684-
<groupId>com.puppycrawl.tools</groupId>
685-
<artifactId>checkstyle</artifactId>
686-
<version>10.18.1</version>
687-
</dependency>
688-
</dependencies>
689-
<executions>
690-
<execution>
691-
<goals>
692-
<goal>check</goal>
693-
</goals>
694-
</execution>
695-
</executions>
696-
</plugin>
697-
</plugins>
698-
</build>
699-
</profile>
700619
</profiles>
701620

702621
</project>

‎powertools-batch/pom.xml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,16 @@
7373
<artifactId>aws-lambda-java-tests</artifactId>
7474
<scope>test</scope>
7575
</dependency>
76+
<dependency>
77+
<groupId>org.mockito</groupId>
78+
<artifactId>mockito-core</artifactId>
79+
<scope>test</scope>
80+
</dependency>
81+
<dependency>
82+
<groupId>org.slf4j</groupId>
83+
<artifactId>slf4j-simple</artifactId>
84+
<scope>test</scope>
85+
</dependency>
7686
</dependencies>
7787

7888
</project>

‎powertools-cloudformation/pom.xml‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,16 @@
7575
<artifactId>junit-jupiter-params</artifactId>
7676
<scope>test</scope>
7777
</dependency>
78+
<dependency>
79+
<groupId>org.mockito</groupId>
80+
<artifactId>mockito-core</artifactId>
81+
<scope>test</scope>
82+
</dependency>
83+
<dependency>
84+
<groupId>org.slf4j</groupId>
85+
<artifactId>slf4j-simple</artifactId>
86+
<scope>test</scope>
87+
</dependency>
7888
<dependency>
7989
<groupId>org.assertj</groupId>
8090
<artifactId>assertj-core</artifactId>
@@ -86,13 +96,4 @@
8696
<scope>test</scope>
8797
</dependency>
8898
</dependencies>
89-
90-
<build>
91-
<plugins>
92-
<plugin>
93-
<groupId>org.apache.maven.plugins</groupId>
94-
<artifactId>maven-checkstyle-plugin</artifactId>
95-
</plugin>
96-
</plugins>
97-
</build>
98-
</project>
99+
</project>

‎powertools-common/pom.xml‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,16 @@
7676
<artifactId>assertj-core</artifactId>
7777
<scope>test</scope>
7878
</dependency>
79+
<dependency>
80+
<groupId>org.mockito</groupId>
81+
<artifactId>mockito-core</artifactId>
82+
<scope>test</scope>
83+
</dependency>
84+
<dependency>
85+
<groupId>org.slf4j</groupId>
86+
<artifactId>slf4j-simple</artifactId>
87+
<scope>test</scope>
88+
</dependency>
7989
</dependencies>
8090
<profiles>
8191
<profile>
@@ -182,11 +192,5 @@
182192
<directory>src/main/resources</directory>
183193
</resource>
184194
</resources>
185-
<plugins>
186-
<plugin>
187-
<groupId>org.apache.maven.plugins</groupId>
188-
<artifactId>maven-checkstyle-plugin</artifactId>
189-
</plugin>
190-
</plugins>
191195
</build>
192-
</project>
196+
</project>

‎powertools-e2e-tests/pom.xml‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -176,10 +176,6 @@
176176

177177
<build>
178178
<plugins>
179-
<plugin>
180-
<groupId>org.apache.maven.plugins</groupId>
181-
<artifactId>maven-checkstyle-plugin</artifactId>
182-
</plugin>
183179
<!-- Don't deploy the e2e tests -->
184180
<plugin>
185181
<groupId>org.apache.maven.plugins</groupId>

‎powertools-idempotency/powertools-idempotency-core/pom.xml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,5 +38,15 @@
3838
<artifactId>powertools-serialization</artifactId>
3939
</dependency>
4040
<!-- Test dependencies -->
41+
<dependency>
42+
<groupId>org.mockito</groupId>
43+
<artifactId>mockito-core</artifactId>
44+
<scope>test</scope>
45+
</dependency>
46+
<dependency>
47+
<groupId>org.slf4j</groupId>
48+
<artifactId>slf4j-simple</artifactId>
49+
<scope>test</scope>
50+
</dependency>
4151
</dependencies>
4252
</project>

‎powertools-large-messages/pom.xml‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,16 @@
9797
<artifactId>junit-pioneer</artifactId>
9898
<scope>test</scope>
9999
</dependency>
100+
<dependency>
101+
<groupId>org.mockito</groupId>
102+
<artifactId>mockito-core</artifactId>
103+
<scope>test</scope>
104+
</dependency>
105+
<dependency>
106+
<groupId>org.slf4j</groupId>
107+
<artifactId>slf4j-simple</artifactId>
108+
<scope>test</scope>
109+
</dependency>
100110
<dependency>
101111
<groupId>org.apache.commons</groupId>
102112
<artifactId>commons-lang3</artifactId>
@@ -130,11 +140,7 @@
130140
</environmentVariables>
131141
</configuration>
132142
</plugin>
133-
<plugin>
134-
<groupId>org.apache.maven.plugins</groupId>
135-
<artifactId>maven-checkstyle-plugin</artifactId>
136-
</plugin>
137143
</plugins>
138144
</build>
139145

140-
</project>
146+
</project>

‎powertools-logging/pom.xml‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,16 @@
6767
<artifactId>junit-jupiter-engine</artifactId>
6868
<scope>test</scope>
6969
</dependency>
70+
<dependency>
71+
<groupId>org.mockito</groupId>
72+
<artifactId>mockito-core</artifactId>
73+
<scope>test</scope>
74+
</dependency>
75+
<dependency>
76+
<groupId>org.slf4j</groupId>
77+
<artifactId>slf4j-simple</artifactId>
78+
<scope>test</scope>
79+
</dependency>
7080
<dependency>
7181
<groupId>org.junit-pioneer</groupId>
7282
<artifactId>junit-pioneer</artifactId>
@@ -201,10 +211,6 @@
201211
</resource>
202212
</resources>
203213
<plugins>
204-
<plugin>
205-
<groupId>org.apache.maven.plugins</groupId>
206-
<artifactId>maven-checkstyle-plugin</artifactId>
207-
</plugin>
208214
<plugin>
209215
<groupId>org.apache.maven.plugins</groupId>
210216
<artifactId>maven-surefire-plugin</artifactId>
@@ -217,4 +223,4 @@
217223
</plugin>
218224
</plugins>
219225
</build>
220-
</project>
226+
</project>

0 commit comments

Comments
 (0)