Skip to content

Home

Jump to bottom
atiqbitstream edited this page Mar 29, 2025 · 2 revisions

Welcome to the IntelliBank-Kafka wiki!

Event-Driven Architecture with IBM Event Streams

  • Use IBM Event Streams (Kafka on IBM Cloud) to enable real-time communication

  • Implement asynchronous messaging between Customer Service and Transaction Service and Notification Service

  • Decouple services for scalability and fault tolerance

1. First, Clean Up Existing Resources

ibmcloud ce application delete --name customer-service --force
ibmcloud ce application delete --name transaction-service --force
ibmcloud ce application delete --name notification-service --force

Run the following command to list all service instances in your IBM Cloud account:

ibmcloud resource service-instances

This command will return a list of all service instances, including the Event Streams instances.

ibmcloud resource service-instance my-event-streams --output json

This command will list things inside 'my-event-stream' instance.

Create an Event Streams instance

ibmcloud resource service-instance-create my-event-streams messagehub standard au-syd

Verify and Initialize Event Streams

After successful creation:

ibmcloud es init -i my-event-streams

2. Create Applications with Dummy Images (Temporary)

Customer Service

ibmcloud ce application create --name customer-service \
  --image nginx:alpine \
  --port 80 \
  --registry-secret icr-secret

Transaction Service

ibmcloud ce application create --name transaction-service \
  --image nginx:alpine \
  --port 80 \
  --registry-secret icr-secret

Notification Service

ibmcloud ce application create --name notification-service \
  --image nginx:alpine \
  --port 80 \
  --registry-secret icr-secret

3. Bind Event Streams Service

For Customer Service

ibmcloud ce application bind --name customer-service \
  --service-instance my-event-streams \
  --prefix EVENT_STREAMS

For Transaction Service

ibmcloud ce application bind --name transaction-service \
  --service-instance my-event-streams \
  --prefix EVENT_STREAMS

For Notification Service

ibmcloud ce application bind --name notification-service \
  --service-instance my-event-streams \
  --prefix EVENT_STREAMS

List the bindings of your application

You can use the ibmcloud ce application get command to check if a service instance is bound to your application. This will show you the details of the application, including any service bindings.

ibmcloud ce application get --name notification-service

5. Rebuild and Push Docker Images

Rebuild with updated code

docker build -t au.icr.io/mycodeengine/customer-service:latest -f Dockerfile.customer-service .
docker build -t au.icr.io/mycodeengine/transaction-service:latest -f Dockerfile.transaction-service .
docker build -t au.icr.io/mycodeengine/notification-service:latest -f Dockerfile.notification-service .

Push to registry

docker push au.icr.io/mycodeengine/customer-service:latest
docker push au.icr.io/mycodeengine/transaction-service:latest
docker push au.icr.io/mycodeengine/notification-service:latest

Update Applications with Real Images

ibmcloud ce application update --name customer-service \
  --image au.icr.io/mycodeengine/customer-service:latest

ibmcloud ce application update --name transaction-service \
  --image au.icr.io/mycodeengine/transaction-service:latest

ibmcloud ce application update --name notification-service \
  --image au.icr.io/mycodeengine/notification-service:latest

1. Initialize Event Streams with Instance Name

Use the -i flag instead of --service-instance-name:

ibmcloud es init -i my-event-streams

3. Initialize with Admin URL 

ibmcloud es init -a <ADMIN_REST_URL>

Example:

ibmcloud es init -a https://<your-admin-rest-url>

4. Create the Topic 

ibmcloud es topic-create --name user.created --partitions 1

5. Verify

ibmcloud es topics --filter "user.created"

Verify Topic and Consumer Groups

Use the available commands from your ES plugin to verify the topic and consumer groups:

  1. List all topics to verify user.created exists:
   ibmcloud es topics
  1. Get details about the user.created topic:
   ibmcloud es topic --name user.created
  1. List all consumer groups:
    ibmcloud es groups
  1. Get details about your consumer groups:
    ibmcloud es group  order-consumer
    ibmcloud es group  user-consumer
  1. View brokers Information: To get a broader view of your Event Streams cluster, including brokers, you can use:
ibmcloud es cluster

This will display the details of your Event Streams cluster, but it may not show individual broker-specific metrics directly.

Delete All Topics at Once

You can use a loop to delete all topics. Since ibmcloud es topic-delete deletes a single topic at a time, you will need to automate this using a for loop.

 topics=("user.created"  "notification-events"  "transaction-events"  "customer-events") # Loop through and delete each topic  


for topic in  "${topics[@]}"; do
 ibmcloud es topic-delete --name "$topic" -f done

This script will delete all topics in the list. The -f flag forces the deletion without prompting for confirmation.

Extra commands:

List your Event Streams instances: If you're using IBM Event Streams, list all the available Event Streams (Kafka) instances in your account:

ibmcloud resource service-instances

Get Event Streams Service Credentials: Once you know the name of your Event Streams service instance, use the following command to get the service credentials (including the broker URL and port):

`ibmcloud resource service-key-create <key-name> Writer --instance-name <your-service-instance-name>
ibmcloud resource service-instances
Name Location State Type Resource Group ID
my-devops-platform au-syd active service_instance Default 39d5d1ee4bd74274b72db57e654fff8e
my-event-streams au-syd active service_instance Default 39d5d1ee4bd74274b72db57e654fff8e
my-kafka-project au-syd active service_instance Default 39d5d1ee4bd74274b72db57e654fff8e

Great! You have found your Event Streams instance, which is named my-event-streams. Now, you can retrieve the Kafka broker URL and other credentials for this instance.

Steps to Retrieve Kafka Broker URL and Credentials:

  1. Get the Service Credentials for the Event Streams Instance: Since you have the name of your Event Streams instance (my-event-streams), you need to retrieve the service credentials. Use the following command to create and view the service key:

    `ibmcloud resource service-key-create kafka-credentials Writer --instance-name my-event-streams

Check the Created Service Key: After running the command, you should see a confirmation message that the service key has been created. Now, retrieve the credentials by running:

ibmcloud resource service-keys --instance-name my-event-streams

1. What is a Kafka Topic?

A topic in Kafka is like a bucket where messages are stored. Services send their messages to this bucket and other services can read them from there.

  • Example: If a new user registers, the User Service sends that info to a topic called user-registrations.

Where do you publish events?

  • You (the producer) send your message (like "User Created") to a topic.

Steps:

  1. The Producer (like the User Service) sends a message (e.g., "User Created") to a Topic (like user-events).
  2. Other services (like the Order Service) can then read this message from the Topic.

What is the role of the Broker?

A broker is like a server that stores the messages in the topic. It makes sure the messages are saved and can be accessed by the services that need them.

  • Brokers handle all the reading and writing of messages.

Why are there multiple brokers?

  • Scalability: More brokers help handle more data.
  • Reliability: If one broker crashes, others can still store the data.

In summary:

  1. You send your message (event) to a topic (like a bucket).

  2. Brokers are the servers that store and manage these messages.

  3. Multiple brokers make Kafka faster and safer by spreading the load and keeping data safe if something breaks.

  4. The Producer sends events to the Topic, and the Consumer reads them from the Topic.

2. What is a Partition in Kafka?

1

Scenario: Banking System

Imagine a banking system that handles customer transactions. This system needs to process lots of transactions and keep track of each customer's account activities in real-time. Let's say we have the following services:

  1. Customer Service: Manages customer accounts (e.g., creating new customers).
  2. Transaction Service: Processes money transfers between accounts.
  3. Notification Service: Sends notifications to customers about their transaction status.

In this system, we use Kafka to send messages between these services, and we will break down topics, partitions, and their role in this system.

What is a Kafka Topic in this Case?

2

Let's say we create a Kafka Topic called transactions.

  • This topic will hold all the messages (events) related to customer transactions, like:
    • "Customer A transferred money to Customer B."
    • "Customer A's transaction failed due to insufficient funds."

The topic is the general category that collects these events, just like a mailbox that holds letters.

What is a Kafka Partition in this Case?

3

Now, let's divide the transactions topic into multiple partitions. Imagine we have 3 partitions, like having 3 mail slots in the mailbox.

Why do we use partitions?

  1. Faster Processing: Each partition can be processed separately, so the system can handle more messages at once.

    • Example: If you have 3 partitions, you can have 3 workers (or consumers) working in parallel. One worker handles Partition 1, another handles Partition 2, and another handles Partition 3. This speeds things up.

  2. Scalability: As your banking system grows, you can easily add more partitions. More partitions mean more capacity to handle more messages.

    • Example: If you start getting millions of transactions, you can increase the number of partitions to manage the load.

Role of Partitions in This Scenario

  1. Message Distribution: When a transaction occurs, the message gets sent to the transactions topic, but Kafka distributes the message to one of the partitions.

    • Example: If Customer A makes a transaction, Kafka decides which partition to store this message in. It might go into Partition 1, and the next transaction might go into Partition 2.

  2. Ordering Within Partitions: Kafka ensures that messages in the same partition are read in the same order they were sent.

    • Example: If Customer A sends multiple transactions, Kafka will make sure they are processed in the order they occurred within the same partition (e.g., "Customer A transferred money to Customer B" before "Customer A requested balance check").

  3. Load Balancing and Parallelism: Each consumer (worker) can read messages from a different partition. If you have 3 partitions, you can have 3 consumers, each reading from one partition, processing transactions at the same time.

    • Example: One consumer processes Partition 1, another consumer processes Partition 2, and a third processes Partition 3.

  4. Fault Tolerance: Kafka ensures that if one partition's data gets lost (due to a failure), there is a copy (replica) stored on another broker to keep the data safe.

    • Example: If Partition 1 is on Broker A and Broker A goes down, Partition 1's replica is stored on Broker B, so the system can keep processing without losing any data.

Putting It All Together

In our banking system:

  1. Topic: The transactions topic collects all transaction messages.

  2. Partitions: The topic is split into multiple partitions (e.g., Partition 1, Partition 2, Partition 3) for better performance and scalability.

    • Each partition holds a part of the messages, making it faster to process them.
    • Kafka ensures that the messages in a partition are processed in order (e.g., "Transaction 1" before "Transaction 2").

  3. Producers: The Transaction Service is the producer, which sends transaction messages (like "Customer A transferred money").

  4. Consumers: The Transaction Service, Notification Service, and other services act as consumers that read messages from the topic and process them.

  5. Brokers: Kafka brokers are the servers that store these partitions and handle the read and write requests for the messages.

Example of How It Works:

  1. Transaction Happens: Customer A sends a transaction to Customer B.

    • This message goes to the transactions topic.
    • Kafka decides which partition to store this message in (e.g., Partition 1).

  2. Consumer Reads Message: The Transaction Service or Notification Service (a consumer) reads the transaction message from Partition 1 and processes it.

    • Transaction Service processes the transaction.
    • Notification Service sends a notification to Customer A about the transaction.

  3. Scaling: If the system starts receiving millions of transactions, we can add more partitions to handle the increased load and use more consumers to process them in parallel.

Summary

  • Partitions break down the topic into smaller pieces, allowing messages to be distributed and processed faster.

  • Each partition holds a subset of messages, and Kafka guarantees message ordering within each partition.

  • Multiple consumers can read from different partitions at the same time, which makes the system scalable and fast.

  • Kafka ensures fault tolerance by replicating partitions to other brokers in case of failure.

3. what is replica leader and followers?

4

In this banking system, we have multiple services like:

  • Customer Service: Manages customer accounts (e.g., creating new customers).
  • Transaction Service: Processes money transfers between accounts.
  • Notification Service: Sends notifications to customers about their transaction status.

Kafka is used to send messages between these services, specifically through the transactions topic. Now, let’s introduce replica leader and replica follower into the system.

Kafka Topic: transactions

  • The transactions topic holds all messages related to customer transactions (e.g., "Customer A transferred money to Customer B" or "Customer A's transaction failed due to insufficient funds").

Kafka Partitioning

  • The transactions topic is split into multiple partitions for better performance and scalability (e.g., Partition 1, Partition 2, Partition 3).
  • Each partition is stored on different brokers (Kafka servers) for fault tolerance and to ensure high availability.

Replica Leader and Replica Follower in the Banking System

Kafka replicates each partition for high availability. Here’s how replica leader and replica follower work within this setup:

1. Replica Leader

  • Role: The leader replica for each partition is the primary replica that handles all reads and writes for that partition.

  • In the Banking System:

    • Let’s say Partition 1 is where all transactions related to Customer A are stored.

    • Broker A holds the leader replica for Partition 1. When Customer A sends a transaction, the Transaction Service (producer) writes this event (e.g., "Customer A transferred $100 to Customer B") to Partition 1’s leader replica on Broker A.

    • The leader replica ensures that the transaction message is recorded in the correct order (important for consistency in banking transactions).

2. Replica Follower

  • Role: The follower replicas maintain backup copies of the data from the leader replica. These followers replicate the leader’s data and ensure that they are always up-to-date.

  • In the Banking System:

    • Broker B and Broker C each hold a follower replica of Partition 1. They regularly fetch the latest transactions from the leader replica on Broker A to ensure they have an up-to-date copy of the data.
    • These follower replicas do not handle writes or direct reads from the producer or consumer. They simply replicate data to ensure fault tolerance.

How Kafka Handles Failures with Replica Leader and Replica Follower

5

Scenario 1: Normal Operation

  1. Transaction Happens:

    • Customer A initiates a transaction, transferring money to Customer B. The Transaction Service sends this message to the transactions topic in Kafka.
    • Kafka writes this message to the leader replica of Partition 1 (stored on Broker A).

  2. Replication Process:

    • The follower replicas on Broker B and Broker C regularly replicate this transaction from the leader replica on Broker A.

  3. Message Processing:

    • Notification Service reads the transaction from Partition 1 (from the leader replica on Broker A) and sends a notification to Customer A about the successful transaction.

Scenario 2: Broker A (Leader) Fails

  1. Leader Failure:

    • Suppose Broker A (which holds the leader replica for Partition 1) fails or goes down. Kafka needs to maintain high availability and ensure no transaction data is lost.

  2. Leader Election:

    • Kafka automatically promotes one of the follower replicas to become the new leader. Let’s say Broker B (which holds a follower replica of Partition 1) is promoted to be the new leader for Partition 1.
    • The new leader replica (Broker B) now starts handling all new writes and reads for Partition 1.

  3. Continued Operations:

    • Even though Broker A (the original leader) is down, Broker B has an up-to-date copy of the data because of replication. Kafka guarantees that the transaction data will not be lost.
    • The Transaction Service continues to send transactions to the new leader (Broker B), and the Notification Service continues to process the transactions.
    • If Broker A comes back online, it can rejoin the cluster, and Kafka will ensure data consistency across all replicas.

Scenario 3: Broker B (Follower) Fails

  1. Follower Failure:
    • If Broker B (one of the follower replicas) fails, Broker C continues to act as a backup for the partition. Kafka automatically updates the replication to ensure the remaining replicas are synchronized with the leader.
  2. No Impact on System:
    • The leader replica on Broker A (or Broker B if promoted) continues to process transactions without any interruption. There’s no downtime for the banking system, and data loss is prevented.
    • Once Broker B comes back online, it will re-sync with the leader replica and catch up on any missed transactions.

Key Roles of Replica Leader and Replica Follower in the Banking System

  1. Fault Tolerance:
    • Kafka ensures that no data is lost by replicating partition data across multiple brokers. If the leader replica fails, one of the follower replicas is promoted to leader, and the system continues to function without downtime.
  2. High Availability:
    • With multiple follower replicas across different brokers, the system ensures that transaction data is always available, even if one broker goes down.
  3. Message Consistency:
    • The leader replica guarantees that messages within a partition are processed in the correct order. If Customer A makes multiple transactions, Kafka ensures they are processed in the correct sequence within the same partition, and replicas (followers) keep the data synchronized.

Putting It All Together: The Flow in Your Banking System with Replicas

  • Transaction Happens: Customer A sends a transaction.

    • The Transaction Service sends this event to Kafka, which writes it to the leader replica of Partition 1 on Broker A.

    • The followers on Broker B and Broker C replicate this message to ensure no data loss.

  • Failure Handling: If Broker A goes down, Broker B is promoted to the leader, and the system continues processing without interruption.

    • Transaction Service writes to the new leader (now on Broker B).
    • Notification Service processes the transaction and sends a notification to Customer A.

  • Scaling and Fault Tolerance: As the banking system grows, Kafka allows for more partitions and replica followers to be added, ensuring the system can handle higher transaction volumes and remain fault-tolerant.

Summary of Replica Leader and Replica Follower in Your Banking System

  • Replica Leader:

    • Handles all reads and writes for a partition.
    • Guarantees message ordering and consistency.
    • Coordinates with followers to replicate data.

  • Replica Follower:

    • Replicates data from the leader to ensure fault tolerance.
    • Ensures there are backup copies of the data for reliability.
    • Can be promoted to leader in case of a failure.

4. Simplified Explanation of Consumer Group, Group ID, and Client ID

Services in the Banking System:

  1. Customer Service: Manages customer accounts (e.g., creating new customers).
  2. Transaction Service: Processes money transfers between accounts.
  3. Notification Service: Sends notifications to customers about their transaction status.

Kafka is used to send messages between these services, specifically through the transactions topic. Now, let’s explore the terms Consumer Group, Group ID, and Client ID in this scenario.

Kafka Consumer Group

Consumer Group refers to a group of consumers that work together to read messages from a Kafka topic. The consumers in a group share the responsibility of processing messages in parallel.

  • In this banking system, there is a transactions topic with several partitions.
  • Transaction Service and Notification Service might both be consumers of the transactions topic. Instead of each service processing all the messages, they work together in a consumer group to share the workload.

How Consumer Group Works:

  • Kafka distributes messages from partitions to consumers in a consumer group.

  • Each partition is processed by only one consumer within a group. If you have 3 partitions and 2 consumers, Kafka will assign 2 partitions to 2 consumers, leaving 1 partition idle or handled by one consumer.

In the banking system, this means:

  • The Transaction Service might consume messages from Partition 1, processing transactions.
  • The Notification Service might consume messages from Partition 2, sending notifications to customers.

This parallel processing improves performance because multiple consumers share the task of consuming messages.

Group ID

The Group ID uniquely identifies a consumer group in Kafka. Every consumer in the same consumer group shares the same Group ID, which allows Kafka to track and manage them as a single unit.

  • For example, the Transaction Service and Notification Service may belong to the same consumer group, called transaction-processing-group.
  • Kafka ensures that within this group, each message is processed by only one consumer.

How Group ID Works:

  • If the Group ID is transaction-processing-group, both the Transaction Service and Notification Service will belong to this group.
  • If there are 3 partitions, Kafka will assign 3 partitions to the consumers within the group to process them in parallel.

Example:

  • Group ID = transaction-processing-group

    • Transaction Service (Consumer 1) processes messages from Partition 1.

    • Notification Service (Consumer 2) processes messages from Partition 2.

Client ID

The Client ID is a unique identifier for each consumer or producer that interacts with Kafka. It helps to track and log activities related to that particular client.

  • Each service, like Transaction Service or Notification Service, can be given its own Client ID to help Kafka identify and monitor their activities.

How Client ID Works:

  • Transaction Service could have a Client ID like transaction-service-client.

  • Notification Service could have a Client ID like notification-service-client.

This way, Kafka can track how Transaction Service and Notification Service are interacting with Kafka, and also help monitor their performance and identify issues.

Putting It All Together in the Banking System

Let’s use an example to see how these concepts work together in your banking system:

  1. Transaction Happens:

    • Customer A initiates a transaction, transferring money to Customer B.

    • The Transaction Service sends this transaction event to Kafka, which stores it in the transactions topic.

  2. Message Distribution:

    • The transactions topic has multiple partitions (e.g., Partition 1, Partition 2, and Partition 3).

    • Kafka divides the workload, so each consumer group can handle part of the processing.

  3. Consumer Group:

    • Transaction Service and Notification Service are part of the same consumer group called transaction-processing-group.

    • This means that both services will share the task of processing messages from the transactions topic.

  4. Partition Assignment:

    • Kafka assigns Partition 1 to Transaction Service and Partition 2 to Notification Service for parallel processing.

    • This way, the Transaction Service processes transaction messages, while the Notification Service sends out notifications at the same time.

  5. Handling Failures:

    • If one consumer (e.g., Transaction Service) fails, another consumer in the group can take over. This is possible because they share the Group ID and Kafka will reassign the partitions to other available consumers in the group.

  6. Client IDs:

    • Transaction Service might have the Client ID transaction-service-client.

    • Notification Service might have the Client ID notification-service-client.

    • Kafka uses Client IDs to track which service is interacting with Kafka and to log the activities of each client.

Summary of Kafka Concepts in Your Banking System

  1. Consumer Group:

    • A consumer group allows multiple consumers (like Transaction Service and Notification Service) to share the workload of processing messages from Kafka.

    • Each partition is assigned to one consumer within the group.

  2. Group ID:

    • The Group ID (transaction-processing-group) uniquely identifies the consumer group. All consumers within this group share the same Group ID, allowing Kafka to track and manage them.

  3. Client ID:

    • The Client ID (transaction-service-client or notification-service-client) uniquely identifies each consumer and helps Kafka track interactions for logging and monitoring purposes.

Clone this wiki locally