❔ Question: Server is not responding to chargers message after reconnection

[YashDhondugade]

Server is not responding to charger's message after reconnection

Current Behavior:

When the charger is connected to the server for the first time, it sends a message, and the server responds as expected. At some point, the server disconnects unexpectedly. The charger then initiates a new connection, and the logs confirm that this reconnection is successful. Despite the successful connection, the server does not send any acknowledgment or response to the subsequent message sent by the charger. Since no message is received from the server within one minute, the charger assumes the connection is lost and tries to initiate a new connection again. However, the server still holds the previous connection for that client and logs a message like "client **** already exists, closing duplicate client." This results in a loop of repeated reconnection attempts.

Captured logs:
2025-04-03 09:43:44 - handling new connection for XXXX from XXXXXXX
2025-04-03 09:43:44 - upgraded websocket connection for XXXX from XXXXXXX
2025-04-03 09:44:00 - handling read error for XXXX: websocket: unexpected reserved bits 0x40
2025-04-03 09:44:00 - handling forced close signal for XXXX
2025-04-03 09:44:00 - closed connection to XXXX
2025-04-03 09:46:18 - handling new connection for XXXX from XXXXXXX
2025-04-03 09:46:18 - upgraded websocket connection for XXXX from XXXXXXX
2025-04-03 09:46:19 - received JSON message from XXXX: [2,"1","StatusNotification",{"connectorId":1,"errorCode":"NoError","status":"Available"}]
2025-04-03 09:47:28 - handling new connection for XXXX from XXXXXXX
2025-04-03 09:47:28 - upgraded websocket connection for XXXX from XXXXXXX
2025-04-03 09:47:28 - client XXXX already exists, closing duplicate client

Expected Behavior:
After a successful reconnection, the server should resume communication and respond to messages from the charger.
If the connection is invalid or fails, the server should clean up the old connection and allow a new one to be established without entering a loop.

Please help to solve the problem
@lorenzodonini @xBlaz3kx

Which OCPP version referring to?

• OCPP 1.6
• OCPP 2.0.1

Are you using any OCPP extensions?

• OCPP 1.6 Security
• OCPP 1.6 Plug and Charge

👀 Have you spent some time to check if this question has been asked before?

• I checked and didn't find a similar issue

[xBlaz3kx]

Probably relates to #336 and #314. Please check the answers there; what you probably need to do is sync the websocket connections on both the server and charger (client).

Possible scenarios:

• Charger does not support server-sent ping-pong message or does not respond to the ping pong
• Charger does not send ping-pong message to the server (at all)
• Charger does not send ping-pong message to the server in time (ping-pong configuration not in sync)

What I would recommend is after the BootNotification, a configuration change (ChangeConfiguration message) is sent to the charger with the ping-pong interval that is in sync with the server.

[Mayuri-Mense]

@xBlaz3kx we do set heartbeatInterval to 60s after getting BootNotification, and server's ping wait is configured to 75s.
Problem is server doesnt close the connection once it goes unresponsive, It keeps rejecting all the connection saying duplicate clientId, client already exist. Shouldn't the server close those connections ? server doesnt become functionally unless we restart the server @lorenzodonini Please suggest workaround

[tanordheim]

I can confirm seeing the same; some times when restarting the Kubernetes pod(s) hosting a service built using ocpp-go, we see that the new instance after some minutes starts logging a lot of the errors mentioned above:

client $chargePointID already exists, closing duplicate client

A instance restart usually resolves it and it remains just fine after that. I dug into one specific charge point exhibiting this behavior last time it happened and I see this (full timeline)

1. [Time +0m0s] Chargepoint connects and authenticates. It sends a BootNotification.req
2. [Time +0m10s] We send TriggerMessage.req and GetConfiguration.req to the chargepoint to confirm some data we need in our end. I can see these get sent via "dispatched request X for $chargePointID" messages being logged.
3. [Time +0m11s] Chargepoint sends StatusNotification.req and GetConfiguration.conf based on the TriggerMessage.req and GetConfiguration.req sent previously
4. [Time +0m11s] Based on the previous two messages received, we issue 3x ChangeConfiguration.req to the charge point to ensure the correct configuration. I can see these get sent via "dispatched request X for $chargePointID" as well.
5. [Time +0m11s] Chargepoint sends MeterValues.req, which we respond to with MeterValues.conf, confirmed by 2x "dispatched request X for $chargePointID" logged.
6. [Time +7m11s] Radio silence for 7 minutes, when we see a new authentication attempt from the same charge point. This logs "client $chargePointID already exists, closing duplicate client"
7. [Time +8m29s] New authentication attempt from the same charge point, another "client $chargePointID already exists, closing duplicate client"
8. [Time +8m49s] The message "failed to send ping message for $chargePointID: write tcp $source->$target: write: broken pipe" gets logged, as well as "closed connection to $chargePointID"

This now repeats for this chargepoint until the service is restarted. Interestingly, we emit metrics for the total amount of connected charge points by tracking a counter via a sync.Map in SetNewChargePointHandler(..) and SetChargePointDisconnectedHandler(..) and the reported connected number of chargepoints is relatively stable through problem times like this.

We respond to BootNotification.req with a BootNotification.conf that sets the heartbeat interval to 45s. In addition, on the ws.Server we set the following:

• PingPeriod: 90s (heartbeat interval *2)
• PongPeriod: 45s (heartbeat interval)

Just looking at the raw logs here without any insights into what the code does, it feels like the chargepoints themselves feel like they have disconnected, and reconnect; yet the ocpp-go central system think the charger is still connected and refuses the duplicate connection. Only to close the connection later when realizing its dead after sending a ping.

It feels like a fix here, maybe specific to our use case, is to close the old connection on a duplicate connection detected, keeping the new one.

[xBlaz3kx]

@tanordheim Very nice and detailed description of the behavor you observed!

Regarding the fix though: Maybe the behavior on the duplicate connection could be configurable; either to reject until the system determines when to drop the connection or to replace the connection with the new one.

However, this might have a significant impact on security; what if someone would purposely spam with new websocket connections, trying to overload the central system or potentially wants to hijack the connection?

[tanordheim]

Regarding the fix though: Maybe the behavior on the duplicate connection could be configurable; either to reject until the system determines when to drop the connection or to replace the connection with the new one.

It would be a patchwork fix at least, the right fix is probably to not have this happen in the first place 🙂

However, this might have a significant impact on security; what if someone would purposely spam with new websocket connections, trying to overload the central system or potentially wants to hijack the connection?

True. We run auth on our chargers which limits the blast radius that somewhat, but that's very use case dependent.

Sidenote, I tried lowering the ping/pong period to 25s and observed a lot of chargepoint disconnects/reconnects so I assume not all chargepoints support this (we're an aggregator across multiple brands) and it isn't the way to go.

[tanordheim]

@xBlaz3kx we've been bitten by this a few more times over this week and can see no other solution to prevent it than to just alert on it and restart the service when it happens.

Do you think the "close original connection instead of the new one" approach is a valid fix for this (given its configurable and opt in)? Or do you have some suggestions for other solutions to the issue? I'm more than happy to contribute a fix.

[xBlaz3kx]

@tanordheim I understand your frustration, but Id suggest reading this article.

I also saw a potential fix for this behavior in another issue, but I think it would be a nice to have an option on how to handle duplicate connections, but with clear remarks on security implications.

[tanordheim]

@tanordheim I understand your frustration

I hope it was clear this was not at all intended as frustration against ocpp-go or the maintainers - more as a "this is a problem for us and I need to carve out some time and propose a decent fix" 🙂

Id suggest reading this article.

I also saw a potential fix for this behavior in another issue, but I think it would be a nice to have an option on how to handle duplicate connections, but with clear remarks on security implications.

I'll give that one a look before I start looking into this after the weekend! I'll create a PR next week on how to handle duplicate connections as well. My gut feeling - of course based on no research or really deep insights - is that using auth mitigates a good bunch of these attack vectors. Let's see what I'm able to cook up!

[YashDhondugade]

I'm currently debugging an issue where the OCPP server becomes unresponsive. I'm using version v0.18.0, and the issue typically occurs when a large number of chargers are connected. All chargers stop responding simultaneously, and restarting the server resolves the issue.

I've added debug logs and observed that the problem seems to originate from the ocppMessageHandler() function. All goroutines appear to get stuck while executing the GetClientState() function.
In GetClientState(), we acquire a write lock, but we only perform a write operation for the first message after a charger connects. For all subsequent messages, we are simply reading the state.

Is there a possibility of a deadlock here due to the write lock being used for read operations?
Would it be more appropriate to use a read lock for getting client state and retain a write lock only for the initial create operation?
Could this be the reason why the server becomes unresponsive for all chargers simultaneously?

@xBlaz3kx

[muaaz9329]

@YashDhondugade Hey , were you able to solve this issue, facing similar issue right now

[smulloni]

We're facing this issue as well. A restart fixed it but it's a worrying vulnerability (and potentially expensive, as chargers hammering us with connection attempts use a lot of SIM bandwidth).

[tanordheim]

I've added debug logs and observed that the problem seems to originate from the ocppMessageHandler() function. All goroutines appear to get stuck while executing the GetClientState() function. In GetClientState(), we acquire a write lock, but we only perform a write operation for the first message after a charger connects. For all subsequent messages, we are simply reading the state.

Is there a possibility of a deadlock here due to the write lock being used for read operations? Would it be more appropriate to use a read lock for getting client state and retain a write lock only for the initial create operation? Could this be the reason why the server becomes unresponsive for all chargers simultaneously?

I'm seeing something similar here too, I don't have any data on whether or not this is in user code or in the ocpp-go handling but during a small incident over the weekend we observed the number of goroutines increase over time while seemingly no data was being received from any of the chargers:

[xBlaz3kx]

@tanordheim I already have an open PR (#343) for OCPP server and some websocket metrics that would be bundled with the library. You can suggest additional metrics there, I think we need to add the callback and request queue metrics to get to the bottom of this issue.

Furthermore, we could add profiling with Grafana Pyroscope and maybe write some performance tests with Grafana K6 that would simulate charger connects and disconnects.

[tanordheim]

@xBlaz3kx Its not really clear to me where this is happening and what metrics are needed, but I rolled out an update of our server now that has pprof endpoints enabled so I can capture where the goroutines are stuck next time it happens. I'll let you know!