Intel Acceleration Stack for Intel® Xeon® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual

Download PDF
ID 683193
Date 11/04/2019
Public
Document Table of Contents
Document Table of Contents x
1. Acceleration Stack for Intel® Xeon® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual
1. Acceleration Stack for Intel® Xeon® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual x
1.1. About this Document 1.2. Introduction 1.3. CCI-P Interface 1.4. AFU Requirements 1.5. Intel® FPGA Basic Building Blocks 1.6. Device Feature List 1.7. Document Revision History for Intel® Acceleration Stack for Intel® Xeon® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual
1.1. About this Document x
1.1.1. Intended Audience 1.1.2. Conventions 1.1.3. Related Documentation 1.1.4. Acronym List for Acceleration Stack for Intel® Xeon® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual 1.1.5. Acceleration Glossary
1.2. Introduction x
1.2.1. FPGA Interface Manager (FIM) 1.2.2. Intel® FPGA Interface Unit (FIU) 1.2.3. Memory and Cache Hierarchy
1.2.2. Intel® FPGA Interface Unit (FIU) x
1.2.2.1. FIU for Intel® FPGA PAC 1.2.2.2. FIU for Intel Integrated FPGA Platform 1.2.2.3. Comparison of FIU Capabilities
1.3. CCI-P Interface x
1.3.1. Signaling Information 1.3.2. Read and Write to Main Memory 1.3.3. Interrupts 1.3.4. UMsg 1.3.5. MMIO Accesses to I/O Memory 1.3.6. CCI-P Tx Signals 1.3.7. Tx Header Format 1.3.8. CCI-P Rx Signals 1.3.9. Multi-Cache Line Memory Requests 1.3.10. Byte Enable Memory Request ( Intel® FPGA PAC D5005) 1.3.11. Additional Control Signals 1.3.12. Protocol Flow 1.3.13. Ordering Rules 1.3.14. Timing Diagram 1.3.15. CCI-P Guidance
1.3.2. Read and Write to Main Memory x
1.3.2.1. Reading from Main Memory 1.3.2.2. Writing to Main Memory
1.3.8. CCI-P Rx Signals x
1.3.8.1. Rx Header and Rx Data Format
1.3.10. Byte Enable Memory Request ( Intel® FPGA PAC D5005) x
1.3.10.1. Mixing Byte Enable and Full Cache Line Accesses
1.3.12. Protocol Flow x
1.3.12.1. Upstream Requests 1.3.12.2. Downstream Requests
1.3.13. Ordering Rules x
1.3.13.1. Memory Requests 1.3.13.2. MMIO Requests
1.3.13.1. Memory Requests x
1.3.13.1.1. Memory Write Fence 1.3.13.1.2. Memory Consistency Explained
1.4. AFU Requirements x
1.4.1. Mandatory AFU CSR Definitions 1.4.2. AFU Discovery Flow 1.4.3. AFU_ID
1. Acceleration Stack for Intel® Xeon® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual

1.3.7. Tx Header Format

Table 12.  Tx Header Field Definitions
Field Description
mode Memory Access Mode
  • eMOD_CL (1'b0)—cache aligned write. Enables 1, 2, or 4 cache line writes to host memory, specified by cl_len.
  • eMOD_BYTE(1'b1)—byte aligned write. Write a contiguous subset of a line to host memory, as specified by the byte_start and byte_len fields. Note: This memory request mode is not available for Intel® PAC with Intel® Arria® 10 GX FPGA.
    Note:

    When set to eMOD_BYTE, the cache length (cl_len) must be set to 0, indicating a single cache line write.

    When set to eMOD_CL, byte_len and byte_start must be set to 0.

Note: You cannot change modes in the middle of a multi cache line write.
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
byte_start Byte Start Index for Byte Access Mode
  • Indicates index of first byte in the 512-bit TX Data bus to write to host memory.
  • When mode = eMOD_CL, byte_start must be set to 0.
  • When mode = eMOD_BYTE, byte_start is set in byte_enable mode and the legal range is 0 - 63.
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
byte_len Byte Length for Byte Access Mode (mode = eMOD_BYTE)
  • Indicates how many bytes to write to host memory.
  • byte_len—specifies the number of bytes to the left (most significant) of the byte_start index to include a memory request in Byte Access Mode.
  • When mode = eMOD_CL, byte_len must be set to 0.
  • When mode = eMOD_BYTE, byte_len is set in byte enable mode and the legal range is 1 - 63.
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
mdata

Metadata: user defined request ID that is returned unmodified from request to response header.

For multi-CL writes on C1 Tx, mdata is only valid for the header when sop=1.

tid

Transaction ID: AFU must return the tid MMIO Read request to response header. It is used to match the response against the request.

vc_sel Virtual Channel selected
  • 2’h0 – VA
  • 2’h1 – VL0
  • 2’h2 – VH0
  • 2’h3 – VH1

All CLs that form a multi-CL write request are routed over the same virtual channel.

req_type Request types listed in Table 13.
sop Start of Packet for multi-CL memory write
  • 1’h1 – marks the first header. Must write in increasing address order.
  • 1’h0 – subsequent headers
cl_len Length for memory requests
  • 2’h0 – 64 bytes (1 CL)
  • 2’h1 – 128 bytes (2 CLs)
  • 2’h3 – 256 bytes (4 CLs)
Note: When mode = eMOD_BYTE, cl_len must be 2’h0.
address

64-byte aligned Physical Address, that is, byte_address>>6

The address must be naturally aligned with regards to the cl_len field. For example for cl_len=2’b01, address[0] must be 1'b0, similarity for cl_len=2'b11, address[1:0] must be 2'b00.

Table 13.   AFU Tx Request Encodings and Channels Mapping
Request Type Encoding Data Payload Description Header Format
t_if_ccip_c0_tx: enum t_ccip_c0_req
eREQ_RDLINE_I 4’h0 No

Memory read request with no intention to cache.

C0 Memory Request Header. Refer to Table 14.

eREQ_RDLINE_S 4’h1 No

Memory read request with caching hint set to Shared.

 
t_if_ccip_c1_tx: enum t_ccip_c1_req
eREQ_WRLINE_I 4’h0 Yes

Memory write request with no intention of keeping the data in FPGA cache.

Does not keep the cache line in the FPGA cache and does not provide guidance on the CPU-side caching.
Note: The CPU is responsible for the CPU-side caching.

C1 Memory Request Hdr. Refer to Table 15.

eREQ_WRLINE_M 4’h1 Yes

Memory write request with caching hint set to Modified.

eREQ_WRPUSH_I 4’h2 Yes

Memory Write Request, with caching hint set to Invalid. FIU writes the data into the processor’s last level cache (LLC) with no intention of keeping the data in FPGA cache. The LLC it writes to is always the LLC associated with the processor where the SDRAM address is homed.

Does not keep the cache line in the FPGA cache, but pushes the line into the CPU LLC.

eREQ_WRFENCE 4’h4 No

Memory write fence request.

Fence Hdr. Refer to Table 16.

eREQ_INTR 4'h6 No Interrupt Interrupt Hdr. Refer to Table 17
t_if_ccip_c2_tx – does not have a request type field
MMIO Rd NA Yes MMIO read response

MMIO Rd Response Hdr

Refer to Table 18.

All unused encodings are considered RSVD0.

Table 14.  C0 Read Memory Request Header Format Structure; t_ccip_c0_ReqMemHdr
Bit Number of Bits Field
[73:72] 2 vc_sel
[71:70] 2 RSVD
[69:68] 2 cl_len
[67:64] 4 req_type
[63:58] 6 RSVD
[57:16] 42 address
[15:0] 16 mdata
To determine if byte enable is available on your platform, you must use Verilog to verify CCIP_ENCODING_HAS_BYTE_WR is defined and the parameter ccip_cfg_pkg::BYTE_EN_SUPPORTED is non-zero. These two conditions must be true in order for byte enable to be available.
Note: When CCIP_ENCODING_HAS_BYTE_WR is defined, byte_start and byte_len are available. This does not mean that byte enable is available on your platform.
Table 15.  C1 Write Memory Request Header Format Structure: t_ccip_c1_ReqMemHdr
Bit Number of Bits Field SOP=1 Field SOP=0
[79:74] 6 byte_len (must be 0 when mode=eMOD_CL)
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
byte_len (must be 0 when sop=0)
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
[73:72] 2 vc_sel RSVD-DNC
[71] 1 sop=1 sop=0
[70] 1 mode
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
mode (must be eMOD_CL when sop=0)
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
[69:68] 2 cl_len RSVD-DNC
[67:64] 4 req_type req_type
[63:58] 6 byte_start (must be 0 when mode=eMOD_CL)
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
byte_start (must be 0 when sop=0)
Note: This field is RSVD0 for Intel® FPGA PAC N3000 and Intel® PAC with Intel® Arria® 10 GX FPGA
[57:18] 40 address[41:0] RSVD-DNC
[17:16] 2 address[1:0]
[15:0] 16 mdata RSVD-DNC
Table 16.  C1 Fence Header Format Structure: t_ccip_c1_ReqFenceHdr
Bit Number of Bits Field
[79:74] 6 RSVD
[73:72] 2 vc_sel
[71:68] 4 RSVD
[67:64] 4 req_type
[63:16] 48 RSVD
[15:0] 16 mdata
Table 17.  C1 Interrupt Header Format Structure: t_ccip_c1_ReqIntrHdr ( Intel® FPGA PAC only)
Bit Number of Bits Field
[79:74] 6 RSVD
[73:72] 2 vc_sel
[71:68] 4 RSVD
[67:64] 4 req_type
[63:12] 62 RSVD
[1:0] 2 id
Table 18.  C2 MMIO Response Header Format
Bit Number of Bits Field
[8:0] 9 tid
Level Two Title