Skip to content
This repository was archived by the owner on Sep 11, 2024. It is now read-only.

Commit 1a4ea1e

Browse files
committed
docs: refactor initial sections of how it works
1 parent c9066da commit 1a4ea1e

File tree

1 file changed

+14
-4
lines changed

1 file changed

+14
-4
lines changed

README.md

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,6 @@
44

55
This is a sink Apache Kafka Connect connector that stores Apache Kafka messages in an AWS S3 bucket.
66

7-
The connector requires Java 11 or newer for development and production.
8-
97
**Table of Contents**
108

119
- [How it works](#how-it-works)
@@ -18,16 +16,23 @@ The connector requires Java 11 or newer for development and production.
1816
## How it works
1917

2018
The connector subscribes to the specified Kafka topics and collects messages coming in them and periodically dumps the collected data to the specified bucket in AWS S3.
19+
20+
### Requirements
21+
22+
The connector requires Java 11 or newer for development and production.
23+
24+
#### Authorization
25+
2126
The connector needs the following permissions to the specified bucket:
2227
* ``s3:GetObject``
2328
* ``s3:PutObject``
2429
* ``s3:AbortMultipartUpload``
2530
* ``s3:ListMultipartUploadParts``
2631
* ``s3:ListBucketMultipartUploads``
2732

28-
In case of ``Access Denied`` error see https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
33+
In case of ``Access Denied`` error, see https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
2934

30-
### Credentials
35+
#### Authentication
3136

3237
To make the connector work, a user has to specify AWS credentials that allow writing to S3.
3338
There are two ways to specify AWS credentials in this connector:
@@ -48,6 +53,8 @@ It is also important to specify `aws.sts.role.external.id` for the security reas
4853

4954
### File name format
5055

56+
> File name format is tightly related to [Record Grouping](#record-grouping)
57+
5158
The connector uses the following format for output files (blobs):
5259
`<prefix><filename>`.
5360

@@ -98,6 +105,9 @@ record grouping modes are:
98105
- `topic`, `partition`, `start_offset`, and `timestamp` - grouping by the topic,
99106
partition, and timestamp;
100107
- `key` - grouping by the key.
108+
- `key`, `topic`, `partition` - grouping by the topic, partition, and key.
109+
110+
See record grouping in the next section for more details.
101111

102112
If the file name template is not specified, the default value is
103113
`{{topic}}-{{partition}}-{{start_offset}}` (+ `.gz` when compression is

0 commit comments

Comments
 (0)