I recently encountered a timing issue where multiple hundred reads from a PCIe device took about 70us longer than usual, at an interval of 20 seconds. It wasn’t the reads themselves (i.e. the time between Memory Read and Completion TLP packages) that took longer, as confirmed using a tracer, but the time between two consecutive reads.

The tracer also revealed a lot of SKP packages between two consecutive reads when the timing error appeared. The PCIe Spec says these are to combat clock drift - was this the issue? Or are SKP packages just sent when there is nothing else to do? The latter is the case. As you can read here, SKP packages are sent about every 6.152us, or accumulated during transmissions and sent when transmissions finish, explaining the large amount of them between delayed consecutive reads.

So… maybe it’s an OS (realtime-patched Linux) scheduling or interrupt issue?

After verifying that the isolation of the CPU worked by calculating the difference between two consecutive TSC reads for about a bazillion times, and also testing a different PCIe device to make sure it is not a configuration issue on the device, all to no success, I tried different BIOS settings.

One suspect was IOMMU (Intel VT-d), though it seemed unlikely, as the same address was read over and over again, and a TLB miss would only lead to a single delayed read, not hundreds. Anyway, since we don’t need any virtualization, we left it off.

The other suspect suggested by a colleague was the integrated GPU. Disabling it fixed the issue, the timing was now consistent. What was the GPU doing every 20 seconds? I don’t know. It should not communicate with the CPU via PCIe. But memory is shared, perhaps large amounts of data are moved in this interval, congesting some internal bus?