Mity CPU Platforms » MitySOM-A5

We’re using the MitySOM-A5E development kit with the PCIe root-port reference design to run a Broadcom MegaRAID 9660-16i (SAS4116, mpi3mr driver) on the M.2 slot (J10). The card enumerates and works, but it intermittently loses commands under normal I/O, and our testing points at the FPGA path the card uses to read HPS memory. We’d appreciate your help, since this path is part of your PCIe root-port integration.

Setup

SOM: MitySOM-A5E, model A5ED-B64-144-SRC-X (S/N 25021141), on the dev-kit carrier (80-001828RI-2)
FPGA: based on the mitysom-a5e-ref project, Quartus Prime Pro 25.3.1, PCIe root port subsystem (pcie_rp_subsys) with the socfpga_agilex5_mity_a5e_devkit_pcie0_rp device tree overlay
Linux 6.18.2 (Yocto, linux-cl-socfpga-lts), mpi3mr driver 8.15.0.5.50
HBA: MegaRAID 9660-16i, firmware 8.12.1.0-00000-00003, with 16 SATA SSDs; link trains at Gen3 x4
Our changes to the reference design:
The FPGA drives M.2 PERST# (BU109, open-drain) for about 100 ms after configuration. The card wouldn’t enumerate otherwise, because the TLV803 releases PERST# before U-Boot programs the Si5332.
pcie_aspm=off on the kernel command line, which stopped replay-timer AER errors.
Root-port max payload raised to 512 in the GTS PCIe IP.
Symptom

At random, a single command to one RAID volume never completes. It can be a 1 MiB write, or a tiny command such as READ CAPACITY or MODE SENSE. After 60–240 s the driver’s abort, LUN reset and target reset all fail, and only a full controller reset recovers it.
While the command is stuck:
The controller still answers management commands.
There are no AER errors, and TransPend is clear on both the root port and the endpoint.
The controller’s event log shows no drive, link or background-initialization events.
It happens on every volume type (JBOD, RAID 0, 1 and 5), on every I/O queue, and with very little load, roughly once per few minutes of I/O. Repeated forced resets eventually put the HBA into safe mode once.
What we ruled out

Drives, cabling, RAID level and background initialization
Temperature (controller at 42–45 °C)
ASPM, max payload and max read request size (128 to 512 bytes), relaxed ordering, no-snoop and extended tags
High-memory DMA (mem=2G didn’t help)
Changing the ACE5-Lite translator’s ARDOMAIN_OVERRIDE from 2 to 0
Separately, marking the PCIe node dma-coherent breaks initialization: the CPU reads stale data. That’s consistent with the translator config, where AWDOMAIN = 0 and ARDOMAIN = 2.

What points at the FPGA read path
The card’s reads from HPS memory go through:
pcie_gts_mcdma (BAM) → bam_axi_bridge → addr_filter → addr_filter_avmm_bridge → f2h_avmm_cdc (mm_ccb_9) → msi_ordering → cct_axi_bridge → ace5lite_cache_coherency_translator_0 → HPS

When we deepened f2h_avmm_cdc (RESPONSE_FIFO_DEPTH 32 → 512, COMMAND_FIFO_DEPTH 4 → 32) so more reads could be in flight, lost commands went from occasional to almost every run, often within seconds. Write throughput only rose about 10%, from 177 to 196 MB/s. So we believe that with several reads outstanding, a read from HPS memory sometimes never completes, or its response is lost or mismatched, somewhere around msi_ordering, cct_axi_bridge or the translator.

Separately, writes to the HBA cap at about 150–200 MB/s regardless of drive count, while reads reach about 1.75 GB/s. That also suggests the HPS read path through the FPGA is limited.

Questions

Have you validated the PCIe root-port reference design with a bus-mastering endpoint that does heavy DMA reads from HPS memory (NVMe, RAID/HBA)? Any known issues or a newer revision of the root-port subsystem?
Are there known limitations or required settings for msi_ordering, addr_filter and the f2h_avmm_cdc / cct_axi_bridge read path, such as outstanding reads, ID handling or burst sizes?
Is the mixed coherency setting on ace5lite_cache_coherency_translator_0 (AWDOMAIN_OVERRIDE = 0, ARDOMAIN_OVERRIDE = 2, cache override disabled) intentional? What do you recommend for PCIe endpoint DMA?
Any guidance on improving PCIe endpoint read bandwidth from HPS memory beyond about 200 MB/s?
We can provide the HBA snapdumps and event logs, kernel logs from the hangs, our test script, and the FPGA changes, and we’re happy to run SignalTap on the read path if you can suggest which signals to look at.