How can I better detect and handle server errors

[stormyweathers]

Hey Jan,

First off, thanks for your work in maintaining this library. The code and extensive discussions around it have been very helpful to me over the last year.

[Pymodbus 3.8.6]

Lately, I've been developing a modbus server device, and encountered issues when trying to use it on a multi-drop bus. I encountered Issue #2556, where you mentioned this is not supported behavior. Ok, I think I understand why this would be hard to support, and I have since moved my server to a dedicated network to avoid this issue.

However, this error did not cause my program to crash, instead I only noticed by the stacktrace printed in the logs, and then my server refusing communication afterwards.

handle: <Handle SerialTransport.intern_read_ready() created at /usr/lib/python3.9/asyncio/selector_events.py:259>
source_traceback: Object created at (most recent call last):
  File "/home/pi/PiLogger/controller-development/water_mfc.py", line 486, in <module>
    asyncio.run(mfc.run_async_server(), debug=True)
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1882, in _run_once
    handle._run()
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/serialtransport.py", line 44, in setup
    self.async_loop.add_reader(self.sync_serial.fileno(), self.intern_read_ready)
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 336, in add_reader
    self._add_reader(fd, callback, *args)
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 259, in _add_reader
    handle = events.Handle(callback, args, self, None)
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/serialtransport.py", line 134, in intern_read_ready
    self.intern_protocol.data_received(data)  # type: ignore[attr-defined]
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/transport.py", line 304, in data_received
    self.datagram_received(data, None)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/transport.py", line 338, in datagram_received
    cut = self.callback_data(self.recv_buffer, addr=addr)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/server/requesthandler.py", line 79, in callback_data
    used_len = super().callback_data(data, addr)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transaction/transaction.py", line 187, in callback_data
    used_len, pdu = self.framer.processIncomingFrame(self.trace_packet(False, data))
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/framer/base.py", line 76, in processIncomingFrame
    data_len, pdu = self._processIncomingFrame(data[used_len:])
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/framer/base.py", line 97, in _processIncomingFrame
    if (result := self.decoder.decode(frame_data)) is None:
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/pdu/decoders.py", line 112, in decode
    pdu.decode(frame[1:])
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/pdu/register_message.py", line 26, in decode
    self.address, self.count = struct.unpack(">HH", data)
struct.error: unpack requires a buffer of 4 bytes

I am concerned that some other future error could silently break my program.
But since the exception is handled somewhere deep (perhaps the asyncio library, perhaps pymodbus?), I am unable to handle it myself.

I am starting the server using an adaptation of one of the examples:

asyncio.run(mfc.run_async_server(), debug=True)

where mfc has a method:

    async def run_async_server(self) -> None:
        txt = f"### start ASYNC server, listening on serial: {self.port}"
        _logger.info(txt)
        # socat -d -d PTY,link=/tmp/ptyp0,raw,echo=0,ispeed=9600
        #             PTY,link=/tmp/ttyp0,raw,echo=0,ospeed=9600
        await StartAsyncSerialServer(
            context=self.server_context,  # Data storage
            identity=self.identity,  # server identify
            # timeout=1,  # waiting time for request to complete
            port=self.port,  # serial port
            # custom_functions=[],  # allow custom handling
            framer=self.framer,  # The framer strategy to use
            # stopbits=1,  # The number of stop bits to use
            # bytesize=8,  # The bytesize of the serial messages
            # parity="N",  # Which kind of parity to use
            baudrate=self.baudrate,  # The baud rate to use for the serial device
            # handle_local_echo=False,  # Handle local echo of the USB-to-RS485 adaptor
            ignore_missing_slaves=True,  # ignore request to a missing slave
            # broadcast_enable=False,  # treat slave 0 as broadcast address,
        )
        return

Could you offer some advice on this?
One option would be to add a special logging handler to detect this issue, i.e.:

    def emit(self, record):
        global critical_error_occurred
        log_entry = self.format(record)

        # Checking for specific error messages in the logs
        if "unpack requires a buffer of 4 bytes" in log_entry:
            critical_error_occurred = True

This feels wrong to me, but I don't know a better way.
I'd rather "un-handle" the exception, so the program crashes, and can then be restarted,

[janiversen]

First of all you should update to the newest version of pymodbus, we have made a number of bug fixes regarding stray exception.

The stack trace you show would normally be passed to the application, however data exchange runs in another task (due to asyncio) and thus are not always returned to the application (which is one the items we tried to do better in newer versions).

the only option I see is to write your own log handler, because some exception originate in asyncio and thus is outside control.

[janiversen]

There is another way, make a small script with a client in a loop...if the client fails restart the server.