You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 11, 2024. It is now read-only.
Copy file name to clipboardExpand all lines: README.md
+14-4Lines changed: 14 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,8 +4,6 @@
4
4
5
5
This is a sink Apache Kafka Connect connector that stores Apache Kafka messages in an AWS S3 bucket.
6
6
7
-
The connector requires Java 11 or newer for development and production.
8
-
9
7
**Table of Contents**
10
8
11
9
-[How it works](#how-it-works)
@@ -18,16 +16,23 @@ The connector requires Java 11 or newer for development and production.
18
16
## How it works
19
17
20
18
The connector subscribes to the specified Kafka topics and collects messages coming in them and periodically dumps the collected data to the specified bucket in AWS S3.
19
+
20
+
### Requirements
21
+
22
+
The connector requires Java 11 or newer for development and production.
23
+
24
+
#### Authorization
25
+
21
26
The connector needs the following permissions to the specified bucket:
22
27
*``s3:GetObject``
23
28
*``s3:PutObject``
24
29
*``s3:AbortMultipartUpload``
25
30
*``s3:ListMultipartUploadParts``
26
31
*``s3:ListBucketMultipartUploads``
27
32
28
-
In case of ``Access Denied`` error see https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
33
+
In case of ``Access Denied`` error, see https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
29
34
30
-
###Credentials
35
+
#### Authentication
31
36
32
37
To make the connector work, a user has to specify AWS credentials that allow writing to S3.
33
38
There are two ways to specify AWS credentials in this connector:
@@ -48,6 +53,8 @@ It is also important to specify `aws.sts.role.external.id` for the security reas
48
53
49
54
### File name format
50
55
56
+
> File name format is tightly related to [Record Grouping](#record-grouping)
57
+
51
58
The connector uses the following format for output files (blobs):
52
59
`<prefix><filename>`.
53
60
@@ -98,6 +105,9 @@ record grouping modes are:
98
105
-`topic`, `partition`, `start_offset`, and `timestamp` - grouping by the topic,
99
106
partition, and timestamp;
100
107
-`key` - grouping by the key.
108
+
-`key`, `topic`, `partition` - grouping by the topic, partition, and key.
109
+
110
+
See record grouping in the next section for more details.
101
111
102
112
If the file name template is not specified, the default value is
103
113
`{{topic}}-{{partition}}-{{start_offset}}` (+ `.gz` when compression is
0 commit comments