Skip to content

Commit 12b2384

Browse files
arthurGrigoarthurGrigo
committed
Added project files
1 parent 411c122 commit 12b2384

File tree

14 files changed

+650
-4
lines changed

14 files changed

+650
-4
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.idea
2+
target
3+
!target/*jar-with-dependencies.jar

‎Dockerfile‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
FROM openjdk:8u181-jdk-slim
2+
3+
ENV SERVER_INSTALL_FOLDER=/app/kafka-mqtt-connector/
4+
ENV JAR_FILE_NAME=simple_kafka_mqtt_connector-0.0.1-SNAPSHOT-jar-with-dependencies.jar
5+
6+
# Application propertis
7+
ENV KAFKA_HOST=localhostss
8+
ENV KAFKA_PORT=9092
9+
ENV KAFKA_CLIENT_ID=testing-kafka-producer-1
10+
11+
ENV MQTT_HOST=localhost
12+
ENV MQTT_PORT=1883
13+
ENV MQTT_CLIENT_ID=mqtt-client-1
14+
ENV MQTT_QOS=2
15+
16+
ENV TOPIC_MAPPING=robotgroup001/robot001>>>test;robotgroup001/robot002>>>test02;robotgroup001/robot003>>>test03
17+
18+
RUN mkdir -p "${SERVER_INSTALL_FOLDER}log"
19+
20+
#SERVER:
21+
ADD src/main/resources/application.properties ${SERVER_INSTALL_FOLDER}
22+
ADD target/${JAR_FILE_NAME} ${SERVER_INSTALL_FOLDER}
23+
24+
ADD docker/setConfiguration.sh ${SERVER_INSTALL_FOLDER}
25+
26+
WORKDIR ${SERVER_INSTALL_FOLDER}
27+
28+
29+
CMD bash ${SERVER_INSTALL_FOLDER}setConfiguration.sh "${SERVER_INSTALL_FOLDER}application.properties" \
30+
"$KAFKA_HOST" "$KAFKA_PORT" "$KAFKA_CLIENT_ID" \
31+
"$MQTT_HOST" "$MQTT_PORT" "$MQTT_CLIENT_ID" "$MQTT_QOS" \
32+
"$TOPIC_MAPPING" \
33+
&& java -jar -Xmx1024m -Xms512m ${JAR_FILE_NAME}
34+

‎LICENSE‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@
186186
same "printed page" as the copyright notice for easier
187187
identification within third-party archives.
188188

189-
Copyright [yyyy] [name of copyright owner]
189+
Copyright 2019 Daniel Schel, Fraunhofer Institute for Manufacturing Engineering and Automation (IPA)
190190

191191
Licensed under the Apache License, Version 2.0 (the "License");
192192
you may not use this file except in compliance with the License.
@@ -198,4 +198,4 @@
198198
distributed under the License is distributed on an "AS IS" BASIS,
199199
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200200
See the License for the specific language governing permissions and
201-
limitations under the License.
201+
limitations under the License.

‎README.md‎

Lines changed: 76 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,76 @@
1-
# simple-kafka-mqtt-connector
2-
This application receives messages from a mqtt broker and sends the messages to a kafka cluster. Topic mapping is configurable.
1+
# A simple MQTT to Apache Kafka Connector
2+
3+
4+
5+
## Usage
6+
7+
#### Prerequisites
8+
* Java 8 or higher
9+
10+
11+
#### Configuration
12+
13+
* Edit the application.properties-file
14+
15+
* kafka.host = IP THROUGH WHICH THE KAFKA BROKER WILL BE REACHED [HOSTNAME|IP]
16+
* kafka.port = [INTEGER]
17+
* kafka.client.id = [STRING]
18+
19+
* mqtt.host = IP THROUGH WHICH THE MQTT BROKER WILL BE REACHED [HOSTNAME|IP]
20+
* mqtt.port = [INTEGER]
21+
* mqtt.client.id = [STRING]
22+
* mqtt.qos = Quality of service for MQTT - [INTEGER] Allowed[0,1,2]
23+
24+
* topic.mapping = How are topics routed from MQTT to Kafka - (Separators >>> and ;) Example mqttTopicA>>>kafkaTopicA;mqttTopicB>>>kafkaTopicB;mqttTopicC>>>kafkaTopicC
25+
26+
#### How to build
27+
28+
* Run: `mvn clean install`
29+
30+
31+
#### How to run
32+
33+
* Place the jar with dependencies and your edited application.properties-file in the same directory
34+
35+
* Open a bash or CMD in the same directory as the .jar
36+
37+
* Run: `java -jar simple_kafka_mqtt_connector-0.0.1-SNAPSHOT-jar-with-dependencies.jar`
38+
39+
40+
## Usage with Docker
41+
42+
* Docker Hub https://hub.docker.com/r/arthurgrigo/simple-kafka-mqtt-connector
43+
44+
* Run (edit enviroment variables to your needs!) : `docker run -d -t -i -e KAFKA_HOST='localhost' -e KAFKA_PORT=9092 -e KAFKA_CLIENT_ID='testing-kafka-producer-1' -e MQTT_HOST='localhost' -e MQTT_PORT=1883 -e MQTT_CLIENT_ID='mqtt-client-1' -e MQTT_QOS=2 -e TOPIC_MAPPING='robotgroup001/robot001>>>test;robotgroup001/robot002>>>test02;robotgroup001/robot003>>>test03' --name simple-kafka-mqtt-connector arthurgrigo/simple-kafka-mqtt-connector:latest`
45+
46+
47+
## Usage with Docker-Compose
48+
49+
* See [docker-compose-examples](docker-compose)
50+
51+
####Standalone
52+
53+
* Stand-alone container
54+
55+
* [docker-compose.yml](docker-compose/standalone/docker-compose.yml)
56+
57+
* Run: `docker-compose up -d`
58+
59+
60+
####Full Stack
61+
62+
* Full Stack (mqtt-broker, zookeeper, kafka-broker, simple-kafka-mqtt-connector)
63+
64+
* [docker-compose.yml](docker-compose/fullstack/docker-compose.yml)
65+
66+
* [env.list](docker-compose/fullstack/env.list)
67+
68+
* Place docker-compose.yml and env.list in the same directory
69+
70+
* Edit env.list to your needs!
71+
72+
* Run: `docker-compose up -d`
73+
74+
75+
## License
76+
See [LICENSE](LICENSE) file for License

‎docker-compose/fullstack/docker-compose.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
version: '2'
2+
3+
services:
4+
5+
simple-kafka-mqtt-connector:
6+
image: arthurgrigo/simple-kafka-mqtt-connector:0.0.1-SNAPSHOT
7+
restart: always
8+
hostname: simple-kafka-mqtt-connector
9+
container_name: simple-kafka-mqtt-connector
10+
env_file:
11+
- env.list
12+
networks:
13+
- kafka_mqtt_network
14+
15+
mosquitto:
16+
image: eclipse-mosquitto:1.5.5
17+
restart: always
18+
hostname: mosquitto
19+
container_name: mosquitto
20+
ports:
21+
- "1883:1883"
22+
networks:
23+
- kafka_mqtt_network
24+
25+
zookeeper:
26+
image: wurstmeister/zookeeper
27+
restart: always
28+
hostname: zookeeper
29+
container_name: zookeeper
30+
ports:
31+
- "2181:2181"
32+
env_file:
33+
- env.list
34+
volumes:
35+
- ./zookeeper/data:/data
36+
- ./zookeeper/datalog:/datalog
37+
networks:
38+
- kafka_mqtt_network
39+
40+
kafka:
41+
image: wurstmeister/kafka:2.11-2.0.0
42+
restart: always
43+
hostname: kafka
44+
container_name: kafka
45+
ports:
46+
- "9092:9092"
47+
env_file:
48+
- env.list
49+
volumes:
50+
- /var/run/docker.sock:/var/run/docker.sock
51+
networks:
52+
- kafka_mqtt_network
53+
54+
networks:
55+
kafka_mqtt_network:
56+
driver: bridge
57+
ipam:
58+
config:
59+
- subnet: 192.168.115.0/24
60+
gateway: 192.168.115.1

‎docker-compose/fullstack/env.list‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
########
2+
#
3+
# simple-kafka-mqtt-connector
4+
#
5+
########
6+
7+
# !EDITME! IP THROUGH WHICH THE KAFKA BROKER WILL BE REACHED
8+
KAFKA_HOST= kafka
9+
10+
KAFKA_PORT= 9092
11+
KAFKA_CLIENT_ID= simple-kafka-mqtt-connector-1
12+
13+
# !EDITME! IP THROUGH WHICH THE MQTT BROKER WILL BE REACHED
14+
MQTT_HOST= mosquitto
15+
MQTT_PORT= 1883
16+
MQTT_CLIENT_ID= simple-kafka-mqtt-connector-1
17+
MQTT_QOS= 2
18+
19+
#!EDITME! mqttTopicA>>>kafkaTopicA;mqttTopicB>>>kafkaTopicB;mqttTopicC>>>kafkaTopicC
20+
TOPIC_MAPPING= robotgroup001/robot001>>>test;robotgroup001/robot002>>>test02;robotgroup001/robot003>>>test03
21+
22+
########
23+
#
24+
# zookeeper
25+
#
26+
########
27+
ZOO_MY_ID= 1
28+
ZOO_PORT= 2181
29+
ZOO_SERVERS= server.1=zookeeper:2888:3888
30+
31+
32+
########
33+
#
34+
# kafka
35+
#
36+
########
37+
38+
# !EDITME! How will kafka producers reach this broker?
39+
KAFKA_ADVERTISED_HOST_NAME= kafka
40+
41+
KAFKA_AUTO_CREATE_TOPICS_ENABLE= true
42+
# KAFKA_CREATE_TOPICS= "test:1:1"
43+
KAFKA_ZOOKEEPER_CONNECT= zookeeper:2181

‎docker-compose/standalone/docker-compose.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
version: '2'
2+
3+
services:
4+
5+
simple-kafka-mqtt-connector:
6+
image: arthurgrigo/simple-kafka-mqtt-connector:0.0.1-SNAPSHOT
7+
restart: always
8+
hostname: simple-kafka-mqtt-connector
9+
container_name: simple-kafka-mqtt-connector
10+
environment:
11+
KAFKA_HOST: 172.21.5.239 #EDITME IP THROUGH WHICH THE KAFKA BROKER WILL BE REACHED
12+
KAFKA_PORT: 9092
13+
KAFKA_CLIENT_ID: simple-kafka-mqtt-connector-1
14+
MQTT_HOST: 172.21.5.239 #EDITME IP THROUGH WHICH THE MQTT BROKER WILL BE REACHED
15+
MQTT_PORT: 1883
16+
MQTT_CLIENT_ID: simple-kafka-mqtt-connector-1
17+
MQTT_QOS: 2
18+
TOPIC_MAPPING: robotgroup001/robot001>>>test;robotgroup001/robot002>>>test02;robotgroup001/robot003>>>test03 #EDITME mqttTopicA>>>kafkaTopicA;mqttTopicB>>>kafkaTopicB;mqttTopicC>>>kafkaTopicC

‎docker/setConfiguration.sh‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/env bash
2+
3+
# Read arguments
4+
application_properties_file=${1}
5+
6+
KAFKA_HOST=${2}
7+
KAFKA_PORT=${3}
8+
KAFKA_CLIENT_ID=${4}
9+
10+
MQTT_HOST=${5}
11+
MQTT_PORT=${6}
12+
MQTT_CLIENT_ID=${7}
13+
MQTT_QOS=${8}
14+
15+
TOPIC_MAPPING=${9}
16+
17+
18+
echo "Configuration:"
19+
echo "Application properties file: $application_properties_file"
20+
echo "KAFKA_HOST: $KAFKA_HOST"
21+
echo "KAFKA_PORT: $KAFKA_PORT"
22+
echo "KAFKA_CLIENT_ID: $KAFKA_CLIENT_ID"
23+
echo "MQTT_HOST: $MQTT_HOST"
24+
echo "MQTT_PORT: $MQTT_PORT"
25+
echo "MQTT_CLIENT_ID: $MQTT_CLIENT_ID"
26+
echo "MQTT_QOS: $MQTT_QOS"
27+
echo "TOPIC_MAPPING: $TOPIC_MAPPING"
28+
29+
# overwrite properties
30+
sed -i "/kafka.host/c\kafka.host = $KAFKA_HOST" ${application_properties_file}
31+
sed -i "/kafka.port/c\kafka.port = $KAFKA_PORT" ${application_properties_file}
32+
sed -i "/kafka.client.id/c\kafka.client.id = $KAFKA_CLIENT_ID" ${application_properties_file}
33+
34+
sed -i "/mqtt.host/c\mqtt.host = $MQTT_HOST" ${application_properties_file}
35+
sed -i "/mqtt.port/c\mqtt.port = $MQTT_PORT" ${application_properties_file}
36+
sed -i "/mqtt.client.id/c\mqtt.client.id = $MQTT_CLIENT_ID" ${application_properties_file}
37+
sed -i "/mqtt.qos/c\mqtt.qos = $MQTT_QOS" ${application_properties_file}
38+
39+
sed -i "/topic.mapping/c\topic.mapping = $TOPIC_MAPPING" ${application_properties_file}

0 commit comments

Comments
 (0)