Understanding SPI latency in recent Kernel versions

[matt-foundry]

Hello @msperl,

Thanks for all your work on improving SPI latency in Linux. As you note, the situation in recent Kernel versions is quite different from when you created this patch, and I notice you have made several recent contributions to https://github.com/torvalds/linux too. I have read through these patches as best as I can, but I am struggling to understand what kind of latency I can expect to achieve today?

For example I am using a Raspberry Pi Model 3A+ with an MCP2515, queuing bursts of 6 back-to-back 8-byte EFF frames from the Pi at 1 Mbps, using SocketCAN, and a recent build of https://github.com/raspberrypi/linux. I was hoping to see transmissions at "close" to the best-case 128 us period, but instead observe frames sent every 260-300 us.

I have tried tweaking several parameters, such as fixing the CPU and GPU frequency, assigning SCHED_FIFO priorities to user and kernel threads such as "spi0", and building a custom kernel setting cs_change_delay to 0 ns in mcp251x.c, and ->rt = 1 in mcp251x.c and spi-bcm2835.c, but nothing has made any substantial improvement.

Is this the best latency I can hope to achieve today, or am I missing some setting or patch which I could apply?

All the best,
Matt

[msperl]

For what it is worth I have to say that for lowest latencies (at highest Cpu cycle cost) you need an optimized spi driver.
The raspberry pi version is polling for some time while the majority rely on interrupts and typically that adds latencies - In the worsted case: from interrupt signal to interrupt handler, from interrupt handler to interrupt handler thread from there to the kernel thread that is accessing the spi bus.

So I do not know where your real latencies lie. Only spi bus captures can show/measure where the time is really spent.

From a performance perspective: my recommendation is to always read as much data as possible in one long transfer - reading more is typically faster than all those intermissions with Setup/tear down overheads (and possibly interrupts at longer transfers).

One thing that may also make it easier is setting aside a cpu core for the can interrupt handler thread and pin that one alone to this core. That may minimize scheduling latencies.

[matt-foundry]

Thank you @msperl for your helpful advice on this - I have a better idea of what I should be measuring in the Kernel now, and attempting to optimise.

[msperl]

I still think that applying a logic analyzer ist the thing you should do to identify the io pattern first and when you understand the underlying issue you can start tuning the kernel...

[matt-foundry]

Thanks, I have ordered an oscilloscope from picotech.com, hopefully when that arrives it'll shed some light on the problem.

[matt-foundry]

Hello @msperl,

Thanks again for your help with my issue. I got to the bottom of it eventually: the problem wasn't SPI latency in the end, but that the upstream mcp251x Linux kernel driver doesn't include the clever FIFO prioritisation scheme you use in this mcp2515a driver, it just uses one TX mailbox, and hence quite a bit of time is lost between transmissions. I've been running with the attached patch for about a month now without issues. Do you think there would be any interest from the kernel developers in merging this patch?

I'm aware that my patch doesn't deal properly with priority inversion (e.g. aborting lower priority messages in favour of higher priority messages) but the mainline kernel driver doesn't support aborting lower priority messages either, so perhaps I'm not not making the situation much worse, but it does substantially improves TX rates (probing the CAN bus I now see back-to-back frame transmission on my scope).

patch-mcp251x.txt