Hi,
I'm trying to evaluate the Pi 5 behavior for some low latency audio application, and use SPI as bus for some hypothetical external ADC/DAC.
Kernel is Linux pi5 6.1.70-rt21-v8-16k+ #1 SMP PREEMPT_RT Wed Jan 10 16:32:55 CET 2024 aarch64 GNU/Linux
I've created a simple test app similar to cyclictest that does two things:
- send off an SPI-transaction of 2KB that on average should take about 7-800us
- sleep until the next cycle period of 2ms
I then load the system with stress-ng --cpu 3, some I2C bus transfers, and iperf3, and gather both wakeup latency stats and SPI transaction duration stats.
The good news: wakeup latency is great, running for days under 100uS, I've seen 150 or so once, but that's good.
However the SPI transaction times can get significantly out of bounds: I've followed cyclictest tracing option and cut off after 1.5ms. Running a couple of hours (or sometimes just minutes, as these things go), I can see > 1.5ms (sometimes 2ms for example) SPI xfer times.
Looking at scheduler tracing with kernelshark I see that the irq/141-dw_axi_dmac_platform seems to be responsible for waking up my process. Worth mentioning here: I've elevated RT priorities for that IRQ as well as SPI5 IRQ and the SPI5 task, as well as my own worker thread of course, to 70. So above all other default 50 functions.
The latency between the dmac IRQ being triggered and then the spi test process running is within the expected low bounds in this case. But it appears the IRQ itself is just coming late. I see a lot of other IRQs especially from ETH0 coming through and interacting with the dmac IRQ.
Now I know that the RP1 contains all the peripheral functions, so my current hypothesis is that traffic over ETH coming though via PCIe is battling with the SPI transactions. And stalls them. Is this hypothesis valid and is there something that can be done? Is there some way to prioritize DMA channels or something like that? I have to admit that this kind of bus-based peripheral action is a gray area, so any insights are greatly appreciated.
For the record: in a prior life I was doing pretty much the same thing based off a CM4 with its direct SPI peripherals, the PCIe channel pushing WIFI data as much as possible, and the SPI transactions were rock steady. Thus my suspicion of some RP1-related root cause here.
I'm trying to evaluate the Pi 5 behavior for some low latency audio application, and use SPI as bus for some hypothetical external ADC/DAC.
Kernel is Linux pi5 6.1.70-rt21-v8-16k+ #1 SMP PREEMPT_RT Wed Jan 10 16:32:55 CET 2024 aarch64 GNU/Linux
I've created a simple test app similar to cyclictest that does two things:
- send off an SPI-transaction of 2KB that on average should take about 7-800us
- sleep until the next cycle period of 2ms
I then load the system with stress-ng --cpu 3, some I2C bus transfers, and iperf3, and gather both wakeup latency stats and SPI transaction duration stats.
The good news: wakeup latency is great, running for days under 100uS, I've seen 150 or so once, but that's good.
However the SPI transaction times can get significantly out of bounds: I've followed cyclictest tracing option and cut off after 1.5ms. Running a couple of hours (or sometimes just minutes, as these things go), I can see > 1.5ms (sometimes 2ms for example) SPI xfer times.
Looking at scheduler tracing with kernelshark I see that the irq/141-dw_axi_dmac_platform seems to be responsible for waking up my process. Worth mentioning here: I've elevated RT priorities for that IRQ as well as SPI5 IRQ and the SPI5 task, as well as my own worker thread of course, to 70. So above all other default 50 functions.
The latency between the dmac IRQ being triggered and then the spi test process running is within the expected low bounds in this case. But it appears the IRQ itself is just coming late. I see a lot of other IRQs especially from ETH0 coming through and interacting with the dmac IRQ.
Now I know that the RP1 contains all the peripheral functions, so my current hypothesis is that traffic over ETH coming though via PCIe is battling with the SPI transactions. And stalls them. Is this hypothesis valid and is there something that can be done? Is there some way to prioritize DMA channels or something like that? I have to admit that this kind of bus-based peripheral action is a gray area, so any insights are greatly appreciated.
For the record: in a prior life I was doing pretty much the same thing based off a CM4 with its direct SPI peripherals, the PCIe channel pushing WIFI data as much as possible, and the SPI transactions were rock steady. Thus my suspicion of some RP1-related root cause here.
Statistics: Posted by __deets__ — Fri Feb 02, 2024 10:17 am — Replies 0 — Views 40