Skip to content

Commit bba1a8b

Browse files
sasanchezramirezsasanchezramirez
committed
docs(examples): initial scaffolding for large-messages module
Signed-off-by: Santiago <sasanchezramirez@gmail.com>
1 parent 7ea89de commit bba1a8b

File tree

12 files changed

+2234
-0
lines changed

12 files changed

+2234
-0
lines changed

‎examples/powertools-examples-large-messages/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Powertools for AWS Lambda (Java) - Kafka Example
2+
3+
This project demonstrates how to use Powertools for AWS Lambda (Java) to deserialize Kafka Lambda events directly into strongly typed Kafka ConsumerRecords<K, V> using different serialization formats.
4+
5+
## Overview
6+
7+
The example showcases automatic deserialization of Kafka Lambda events into ConsumerRecords using three formats:
8+
- JSON - Using standard JSON serialization
9+
- Avro - Using Apache Avro schema-based serialization
10+
- Protobuf - Using Google Protocol Buffers serialization
11+
12+
Each format has its own Lambda function handler that demonstrates how to use the `@Deserialization` annotation with the appropriate `DeserializationType`, eliminating the need to handle complex deserialization logic manually.
13+
14+
## Build and Deploy
15+
16+
### Prerequisites
17+
- [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
18+
- Java 11+
19+
- Maven
20+
21+
### Build
22+
23+
```bash
24+
# Build the application
25+
sam build
26+
```
27+
28+
### Deploy
29+
30+
```bash
31+
# Deploy the application to AWS
32+
sam deploy --guided
33+
```
34+
35+
During the guided deployment, you'll be prompted to provide values for required parameters. After deployment, SAM will output the ARNs of the deployed Lambda functions.
36+
37+
### Build with Different Serialization Formats
38+
39+
The project includes Maven profiles to build with different serialization formats:
40+
41+
```bash
42+
# Build with JSON only (no Avro or Protobuf)
43+
mvn clean package -P base
44+
45+
# Build with Avro only
46+
mvn clean package -P avro-only
47+
48+
# Build with Protobuf only
49+
mvn clean package -P protobuf-only
50+
51+
# Build with all formats (default)
52+
mvn clean package -P full
53+
```
54+
55+
## Testing
56+
57+
The `events` directory contains sample events for each serialization format:
58+
- `kafka-json-event.json` - Sample event with JSON-serialized products
59+
- `kafka-avro-event.json` - Sample event with Avro-serialized products
60+
- `kafka-protobuf-event.json` - Sample event with Protobuf-serialized products
61+
62+
You can use these events to test the Lambda functions:
63+
64+
```bash
65+
# Test the JSON deserialization function
66+
sam local invoke JsonDeserializationFunction --event events/kafka-json-event.json
67+
68+
# Test the Avro deserialization function
69+
sam local invoke AvroDeserializationFunction --event events/kafka-avro-event.json
70+
71+
# Test the Protobuf deserialization function
72+
sam local invoke ProtobufDeserializationFunction --event events/kafka-protobuf-event.json
73+
```
74+
75+
## Sample Generator Tool
76+
77+
The project includes a tool to generate sample JSON, Avro, and Protobuf serialized data. See the [tools/README.md](tools/README.md) for more information.

‎examples/powertools-examples-large-messages/pom.xml‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<groupId>software.amazon.lambda.examples</groupId>
5+
<version>2.8.0</version>
6+
<artifactId>powertools-examples-kafka</artifactId>
7+
<packaging>jar</packaging>
8+
<name>Powertools for AWS Lambda (Java) - Examples - Kafka</name>
9+
10+
<properties>
11+
<maven.compiler.source>11</maven.compiler.source>
12+
<maven.compiler.target>11</maven.compiler.target>
13+
<aspectj.version>1.9.20.1</aspectj.version>
14+
<avro.version>1.12.1</avro.version>
15+
<protobuf.version>4.33.0</protobuf.version>
16+
</properties>
17+
18+
<dependencies>
19+
<dependency>
20+
<groupId>software.amazon.lambda</groupId>
21+
<artifactId>powertools-kafka</artifactId>
22+
<version>${project.version}</version>
23+
</dependency>
24+
<dependency>
25+
<groupId>org.apache.kafka</groupId>
26+
<artifactId>kafka-clients</artifactId>
27+
<version>4.1.1</version> <!-- Supports >= 3.0.0 -->
28+
</dependency>
29+
<dependency>
30+
<groupId>org.apache.avro</groupId>
31+
<artifactId>avro</artifactId>
32+
<version>${avro.version}</version>
33+
</dependency>
34+
<dependency>
35+
<groupId>com.google.protobuf</groupId>
36+
<artifactId>protobuf-java</artifactId>
37+
<version>${protobuf.version}</version>
38+
</dependency>
39+
40+
<!-- Basic logging setup -->
41+
<dependency>
42+
<groupId>software.amazon.lambda</groupId>
43+
<artifactId>powertools-logging-log4j</artifactId>
44+
<version>${project.version}</version>
45+
</dependency>
46+
<dependency>
47+
<groupId>org.aspectj</groupId>
48+
<artifactId>aspectjrt</artifactId>
49+
<version>${aspectj.version}</version>
50+
</dependency>
51+
</dependencies>
52+
53+
<build>
54+
<plugins>
55+
<!-- Don't deploy the example -->
56+
<plugin>
57+
<groupId>org.apache.maven.plugins</groupId>
58+
<artifactId>maven-deploy-plugin</artifactId>
59+
<version>3.1.4</version>
60+
<configuration>
61+
<skip>true</skip>
62+
</configuration>
63+
</plugin>
64+
<plugin>
65+
<groupId>org.apache.maven.plugins</groupId>
66+
<artifactId>maven-shade-plugin</artifactId>
67+
<version>3.6.1</version>
68+
<executions>
69+
<execution>
70+
<phase>package</phase>
71+
<goals>
72+
<goal>shade</goal>
73+
</goals>
74+
<configuration>
75+
<createDependencyReducedPom>false</createDependencyReducedPom>
76+
<transformers>
77+
<transformer
78+
implementation="org.apache.logging.log4j.maven.plugins.shade.transformer.Log4j2PluginCacheFileTransformer" />
79+
</transformers>
80+
</configuration>
81+
</execution>
82+
</executions>
83+
<dependencies>
84+
<dependency>
85+
<groupId>org.apache.logging.log4j</groupId>
86+
<artifactId>log4j-transform-maven-shade-plugin-extensions</artifactId>
87+
<version>0.2.0</version>
88+
</dependency>
89+
</dependencies>
90+
</plugin>
91+
<plugin>
92+
<groupId>dev.aspectj</groupId>
93+
<artifactId>aspectj-maven-plugin</artifactId>
94+
<version>1.14.1</version>
95+
<configuration>
96+
<source>${maven.compiler.source}</source>
97+
<target>${maven.compiler.target}</target>
98+
<complianceLevel>${maven.compiler.target}</complianceLevel>
99+
<aspectLibraries>
100+
<aspectLibrary>
101+
<groupId>software.amazon.lambda</groupId>
102+
<artifactId>powertools-logging</artifactId>
103+
</aspectLibrary>
104+
</aspectLibraries>
105+
</configuration>
106+
<executions>
107+
<execution>
108+
<goals>
109+
<goal>compile</goal>
110+
</goals>
111+
</execution>
112+
</executions>
113+
<dependencies>
114+
<dependency>
115+
<groupId>org.aspectj</groupId>
116+
<artifactId>aspectjtools</artifactId>
117+
<version>${aspectj.version}</version>
118+
</dependency>
119+
</dependencies>
120+
</plugin>
121+
<!-- Generate Avro classes from schema -->
122+
<plugin>
123+
<groupId>org.apache.avro</groupId>
124+
<artifactId>avro-maven-plugin</artifactId>
125+
<version>${avro.version}</version>
126+
<executions>
127+
<execution>
128+
<phase>generate-sources</phase>
129+
<goals>
130+
<goal>schema</goal>
131+
</goals>
132+
<configuration>
133+
<sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
134+
<outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
135+
<stringType>String</stringType>
136+
</configuration>
137+
</execution>
138+
</executions>
139+
</plugin>
140+
<!-- Generate Protobuf classes from schema -->
141+
<plugin>
142+
<groupId>io.github.ascopes</groupId>
143+
<artifactId>protobuf-maven-plugin</artifactId>
144+
<version>3.10.3</version>
145+
<executions>
146+
<execution>
147+
<goals>
148+
<goal>generate</goal>
149+
</goals>
150+
<phase>generate-sources</phase>
151+
<configuration>
152+
<protocVersion>${protobuf.version}</protocVersion>
153+
<sourceDirectories>
154+
<sourceDirectory>${project.basedir}/src/main/proto</sourceDirectory>
155+
</sourceDirectories>
156+
<outputDirectory>${project.basedir}/src/main/java</outputDirectory>
157+
<clearOutputDirectory>false</clearOutputDirectory>
158+
</configuration>
159+
</execution>
160+
</executions>
161+
</plugin>
162+
</plugins>
163+
</build>
164+
165+
<profiles>
166+
<!-- Base profile without Avro or Protobuf (compatible with JSON only) -->
167+
<profile>
168+
<id>base</id>
169+
<properties>
170+
<active.profile>base</active.profile>
171+
</properties>
172+
<dependencies>
173+
<!-- Exclude both Avro and Protobuf -->
174+
<dependency>
175+
<groupId>org.apache.avro</groupId>
176+
<artifactId>avro</artifactId>
177+
<version>${avro.version}</version>
178+
<scope>provided</scope>
179+
</dependency>
180+
<dependency>
181+
<groupId>com.google.protobuf</groupId>
182+
<artifactId>protobuf-java</artifactId>
183+
<version>${protobuf.version}</version>
184+
<scope>provided</scope>
185+
</dependency>
186+
</dependencies>
187+
</profile>
188+
189+
<!-- Profile with only Avro -->
190+
<profile>
191+
<id>avro-only</id>
192+
<properties>
193+
<active.profile>avro-only</active.profile>
194+
</properties>
195+
<dependencies>
196+
<dependency>
197+
<groupId>com.google.protobuf</groupId>
198+
<artifactId>protobuf-java</artifactId>
199+
<version>${protobuf.version}</version>
200+
<scope>provided</scope>
201+
</dependency>
202+
</dependencies>
203+
</profile>
204+
205+
<!-- Profile with only Protobuf -->
206+
<profile>
207+
<id>protobuf-only</id>
208+
<properties>
209+
<active.profile>protobuf-only</active.profile>
210+
</properties>
211+
<dependencies>
212+
<dependency>
213+
<groupId>org.apache.avro</groupId>
214+
<artifactId>avro</artifactId>
215+
<version>${avro.version}</version>
216+
<scope>provided</scope>
217+
</dependency>
218+
</dependencies>
219+
</profile>
220+
221+
<!-- Profile with both Avro and Protobuf (default) -->
222+
<profile>
223+
<id>full</id>
224+
<activation>
225+
<activeByDefault>true</activeByDefault>
226+
</activation>
227+
<properties>
228+
<active.profile>full</active.profile>
229+
</properties>
230+
</profile>
231+
</profiles>
232+
</project>

‎examples/powertools-examples-large-messages/template.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
AWSTemplateFormatVersion: "2010-09-09"
2+
Transform: AWS::Serverless-2016-10-31
3+
Description: >
4+
Kafka Deserialization example with Kafka Lambda ESM
5+
6+
Globals:
7+
Function:
8+
Timeout: 20
9+
Runtime: java11
10+
MemorySize: 512
11+
Tracing: Active
12+
13+
Resources:
14+
JsonDeserializationFunction:
15+
Type: AWS::Serverless::Function
16+
Properties:
17+
CodeUri: .
18+
Handler: org.demo.kafka.JsonDeserializationFunction::handleRequest
19+
Environment:
20+
Variables:
21+
JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
22+
POWERTOOLS_LOG_LEVEL: DEBUG
23+
POWERTOOLS_SERVICE_NAME: JsonDeserialization
24+
POWERTOOLS_METRICS_NAMESPACE: JsonDeserializationFunction
25+
26+
AvroDeserializationFunction:
27+
Type: AWS::Serverless::Function
28+
Properties:
29+
CodeUri: .
30+
Handler: org.demo.kafka.AvroDeserializationFunction::handleRequest
31+
Environment:
32+
Variables:
33+
JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
34+
POWERTOOLS_LOG_LEVEL: DEBUG
35+
POWERTOOLS_SERVICE_NAME: AvroDeserialization
36+
POWERTOOLS_METRICS_NAMESPACE: AvroDeserializationFunction
37+
38+
ProtobufDeserializationFunction:
39+
Type: AWS::Serverless::Function
40+
Properties:
41+
CodeUri: .
42+
Handler: org.demo.kafka.ProtobufDeserializationFunction::handleRequest
43+
Environment:
44+
Variables:
45+
JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
46+
POWERTOOLS_LOG_LEVEL: DEBUG
47+
POWERTOOLS_SERVICE_NAME: ProtobufDeserialization
48+
POWERTOOLS_METRICS_NAMESPACE: ProtobufDeserializationFunction
49+
50+
Outputs:
51+
JsonFunction:
52+
Description: "Kafka JSON Lambda Function ARN"
53+
Value: !GetAtt JsonDeserializationFunction.Arn
54+
AvroFunction:
55+
Description: "Kafka Avro Lambda Function ARN"
56+
Value: !GetAtt AvroDeserializationFunction.Arn
57+
ProtobufFunction:
58+
Description: "Kafka Protobuf Lambda Function ARN"
59+
Value: !GetAtt ProtobufDeserializationFunction.Arn

0 commit comments

Comments
 (0)