Architecture & Design Verification: 3. PCIe for DV Engineers

Part 3 of the PCIe for DV Engineers series | Part 2: Physical Layer

In Part 3 of our PCIe series, we move up the stack to the Data Link Layer. This layer ensures reliable delivery of packets between two directly connected devices, handling acknowledgments, replay, and flow control.

Where the Data Link Layer Fits

The Data Link Layer sits between the Physical and Transaction layers:

Its primary job is to provide reliable delivery over the Physical Layer's bit pipe.

1. Data Link Layer Overview

The Data Link Layer sits between the Physical and Transaction layers, providing:

Reliable Delivery: ACK/NAK protocol ensures TLPs arrive correctly
Replay Buffer: Retransmits packets on NAK or timeout
Flow Control: Manages buffer credits to prevent overflow
CRC Protection: 32-bit LCRC for error detection

flowchart TB
    subgraph DL_LAYER["Data Link Layer"]
        subgraph TX_PATH["Transmit Path"]
            TLP_IN[TLP from Transaction Layer]
            SEQ[Add Sequence Number]
            LCRC[Calculate LCRC]
            REPLAY[Store in Replay Buffer]
            TX_OUT[To Physical Layer]
        end
        subgraph RX_PATH["Receive Path"]
            RX_IN[From Physical Layer]
            CRC_CHK[LCRC Check]
            SEQ_CHK[Sequence Check]
            ACK_GEN[Generate ACK/NAK]
            TLP_OUT[TLP to Transaction Layer]
        end
    end
    TLP_IN --> SEQ --> LCRC --> REPLAY --> TX_OUT
    RX_IN --> CRC_CHK --> SEQ_CHK --> TLP_OUT
    SEQ_CHK --> ACK_GEN

2. Data Link Layer Packets (DLLPs)

DLLPs are 8-byte packets used for link management. Unlike TLPs, they don't carry payload data.

DLLP Structure

Byte	Field	Description
0	Type	DLLP type identifier
1-2	Payload	Type-specific data
3-4	CRC	16-bit CRC

DLLP Types

Type	Code	Purpose
Ack	00h	Acknowledge TLPs received correctly
Nak	10h	Request retransmission
InitFC1-P	40h	Posted credits initialization
InitFC1-NP	50h	Non-Posted credits initialization
InitFC1-Cpl	60h	Completion credits initialization
InitFC2-P	C0h	Posted credits init (second phase)
InitFC2-NP	D0h	Non-Posted credits init
InitFC2-Cpl	E0h	Completion credits init
UpdateFC-P	80h	Update Posted credits
UpdateFC-NP	90h	Update Non-Posted credits
UpdateFC-Cpl	A0h	Update Completion credits
PM_Enter_L1	20h	Power management
PM_Enter_L23	21h	Power management
Vendor	30h	Vendor-specific

DV Insight: Your testbench should verify all DLLP types are correctly generated and processed. Pay special attention to flow control initialization - links cannot enter L0 without completing FC init.

3. ACK/NAK Protocol

The ACK/NAK mechanism ensures reliable TLP delivery.

Transmit Side

1. Assign 12-bit sequence number to each TLP

2. Calculate 32-bit LCRC over sequence number + TLP

3. Store TLP in replay buffer

4. Transmit TLP with sequence number and LCRC

5. Start REPLAY_TIMER

Receive Side

1. Verify LCRC

2. Check sequence number is expected (previous + 1)

3. If valid: forward TLP, send ACK DLLP

4. If invalid: send NAK DLLP, discard TLP

sequenceDiagram
    participant TX as Transmitter
    participant RX as Receiver
    TX->>RX: TLP (Seq=0)
    TX->>RX: TLP (Seq=1)
    TX->>RX: TLP (Seq=2) - Corrupted!
    RX-->>TX: ACK (AckSeq=1)
    RX-->>TX: NAK (Seq=2)
    TX->>RX: TLP (Seq=2) - Replay
    TX->>RX: TLP (Seq=3)
    RX-->>TX: ACK (AckSeq=3)

Sequence Number Rules

12-bit counter (0-4095), wraps around
ACK contains sequence number of last successfully received TLP
NAK contains sequence number of first bad/missing TLP
Transmitter retransmits all TLPs from NAK sequence onwards

DV Insight: Test sequence number wraparound carefully. Also verify behavior when ACKs are delayed or lost - the REPLAY_TIMER should trigger retransmission.

4. Replay Buffer & Timer

Interactive Retry Buffer Demo

Before diving into the details, try this interactive visualization of the ACK/NAK and Retry Buffer mechanism. Click Send TLP to transmit packets, Inject LCRC Error to simulate corruption, or Drop ACK to trigger a timeout replay:

Key observations from the demo:

Each TLP gets a sequence number and is stored in the Retry Buffer until ACKed
ACK removes all TLPs up to the acknowledged sequence number
NAK triggers replay of all TLPs from the NAK sequence onwards
Timer expiration triggers replay if no ACK received
After 4 consecutive replays (REPLAY_NUM), the link enters Recovery state

Replay Buffer

Stores transmitted TLPs until acknowledged:

Size varies by implementation (min 128 bytes)
Must hold enough TLPs to cover round-trip latency
TLPs removed when ACK received
All TLPs retransmitted from replay point on NAK

REPLAY_TIMER

Started when TLP transmitted and buffer non-empty
Restarted when ACK received
Timeout triggers full replay from oldest unacked TLP
Timeout value depends on link speed/width

REPLAY_NUM Counter

Counts consecutive replay events
Threshold typically 4
Exceeding threshold → enter Recovery LTSSM state


// Replay buffer verification coverage
covergroup replay_cg @(posedge clk);
  
  buffer_depth: coverpoint replay_buffer.depth {
    bins empty    = {0};
    bins low      = {[1:16]};
    bins medium   = {[17:64]};
    bins high     = {[65:$]};
  }
  
  replay_events: coverpoint replay_triggered {
    bins single   = (0 => 1);
    bins back2back = (1 => 1);
  }
  
  replay_reason: coverpoint replay_cause {
    bins nak_received  = {REPLAY_NAK};
    bins timer_expired = {REPLAY_TIMEOUT};
  }
  
  // Cross: buffer fullness when replay happens
  buffer_x_replay: cross buffer_depth, replay_events;
  
endgroup

5. Flow Control

Flow control prevents buffer overflow at the receiver. It uses a credit-based system.

Credit Types

Credit Type	Tracks	TLP Types
Posted Header (PH)	Headers	Memory Write, Message
Posted Data (PD)	Data payload	Memory Write data
Non-Posted Header (NPH)	Headers	Memory Read, Config, IO
Non-Posted Data (NPD)	Data payload	(rarely used)
Completion Header (CplH)	Headers	Completions
Completion Data (CplD)	Data payload	Completion data

Credit Units

Header credit = 1 TLP header (max 4 DW)
Data credit = 4 DW (16 bytes) of payload

Flow Control Mechanism

sequenceDiagram
    participant TX as Transmitter
    participant RX as Receiver
    Note over TX,RX: FC Initialization
    RX-->>TX: InitFC1 (PH=8, PD=32)
    RX-->>TX: InitFC2 (PH=8, PD=32)
    TX->>TX: Credits Available: PH=8, PD=32
    Note over TX,RX: Normal Operation
    TX->>RX: Posted Write (1 PH, 4 PD)
    TX->>TX: Credits: PH=7, PD=28
    TX->>RX: Posted Write (1 PH, 4 PD)
    TX->>TX: Credits: PH=6, PD=24
    Note over RX: Buffer freed
    RX-->>TX: UpdateFC (PH=+2, PD=+8)
    TX->>TX: Credits: PH=8, PD=32

Flow Control Rules

1. Cannot transmit if insufficient credits (must wait)

2. Receiver sends UpdateFC DLLPs when buffers freed

3. Infinite credits (FFFFh) = flow control disabled for that type

4. Credits never go negative

DV Insight: Flow control bugs are common and subtle. Key scenarios:

Verify transmitter stalls when credits exhausted
Test UpdateFC DLLP processing under heavy load
Check for credit leaks (credits consumed but never returned)
Verify infinite credits work correctly

6. Flow Control Initialization

Before L0, devices must exchange initial credits via two-phase handshake:

InitFC Sequence

1. InitFC1 Phase: Exchange initial credit values

Send InitFC1-P, InitFC1-NP, InitFC1-Cpl

Must receive all three types from partner

2. InitFC2 Phase: Confirm reception

Send InitFC2-P, InitFC2-NP, InitFC2-Cpl

Must receive all three types from partner

3. Link Ready: Can now transmit TLPs


// FC init state machine verification
typedef enum {
  FC_INIT_IDLE,
  FC_INIT1_TX,
  FC_INIT1_RX_WAIT,
  FC_INIT2_TX,
  FC_INIT2_RX_WAIT,
  FC_INIT_DONE
} fc_init_state_e;

// Verify all states are visited
covergroup fc_init_cg @(posedge clk);
  fc_state: coverpoint dut.fc_init_state {
    bins all_states[] = {FC_INIT_IDLE, FC_INIT1_TX, FC_INIT1_RX_WAIT,
                         FC_INIT2_TX, FC_INIT2_RX_WAIT, FC_INIT_DONE};
  }
  
  // Transitions
  fc_trans: coverpoint dut.fc_init_state {
    bins idle_to_init1   = (FC_INIT_IDLE => FC_INIT1_TX);
    bins init1_to_wait   = (FC_INIT1_TX => FC_INIT1_RX_WAIT);
    bins init1_to_init2  = (FC_INIT1_RX_WAIT => FC_INIT2_TX);
    bins init2_to_done   = (FC_INIT2_RX_WAIT => FC_INIT_DONE);
  }
endgroup

7. Data Integrity (LCRC)

LCRC Calculation

32-bit CRC using polynomial 0x04C11DB7
Calculated over: Sequence Number (2 bytes) + TLP (variable)
Appended to each TLP before transmission

LCRC Verification Points


// LCRC error injection and detection
class lcrc_error_test extends pcie_base_test;
  
  task run_phase(uvm_phase phase);
    pcie_tlp_seq seq;
    
    // Test 1: Corrupt LCRC on transmitted TLP
    seq = pcie_tlp_seq::type_id::create("seq");
    seq.inject_lcrc_error = 1;
    seq.start(env.agent.sequencer);
    
    // Verify NAK received
    wait_for_dllp(DLLP_NAK);
    
    // Verify TLP replayed
    ASSERT_EQ(env.agent.replay_count, 1, "TLP should be replayed")
    
    // Test 2: Corrupt random bit in TLP
    seq.inject_lcrc_error = 0;
    seq.corrupt_random_bit = 1;
    seq.start(env.agent.sequencer);
    
    wait_for_dllp(DLLP_NAK);
    
  endtask
  
endclass



8. DV Verification Scenarios

Must-Have Test Scenarios



Scenario Description Coverage Goal


ACK/NAK Basic Normal ACK flow, single NAK, replay Functional
Sequence Wrap Sequence number 4095 → 0 transition Corner case
Replay Timer Timeout without ACK triggers replay Error handling
Replay Exhaustion 4 consecutive replays → Recovery Error escalation
FC Init Timeout Partner doesn't complete FC init Error handling
Credit Exhaustion TX stall when credits = 0 Flow control
Credit Wrap Credit counters wrap correctly Corner case
Back-to-back NAK Multiple consecutive NAKs Stress


Example: Credit Exhaustion Test

Scenario	Description	Coverage Goal
ACK/NAK Basic	Normal ACK flow, single NAK, replay	Functional
Sequence Wrap	Sequence number 4095 → 0 transition	Corner case
Replay Timer	Timeout without ACK triggers replay	Error handling
Replay Exhaustion	4 consecutive replays → Recovery	Error escalation
FC Init Timeout	Partner doesn't complete FC init	Error handling
Credit Exhaustion	TX stall when credits = 0	Flow control
Credit Wrap	Credit counters wrap correctly	Corner case
Back-to-back NAK	Multiple consecutive NAKs	Stress


task test_credit_exhaustion();
  // Configure receiver with minimal credits
  partner.set_initial_credits(.ph(2), .pd(4), .nph(1), .npd(0), .cplh(4), .cpld(8));
  
  // Complete FC init
  wait_for_fc_init_done();
  
  // Send TLPs until credits exhausted
  fork
    begin
      // Transmitter: Send 10 posted writes (needs 10 PH, 40 PD)
      for (int i = 0; i < 10; i++) begin
        send_posted_write(.addr(32'h1000 + i*64), .length(4));
      end
    end
    begin
      // Monitor: Verify stall occurs
      wait(dut.tx_stalled && dut.credits_ph == 0);
      uvm_info("TEST", "TX correctly stalled on credit exhaustion", UVM_MEDIUM)
      
      // After some time, release credits
      #1000ns;
      partner.send_update_fc(.ph(8), .pd(32));
    end
  join
  
  // Verify all TLPs eventually sent
  `ASSERT_EQ(monitor.tlp_count, 10, "All TLPs should complete")
endtask

9. Common Data Link Layer Bugs

Watch for these issues during verification:

Credit Leak: Credits consumed but never returned on completion
ACK Ordering: Out-of-order ACK processing causing premature buffer release
Replay Storm: Continuous replay due to persistent errors
FC Deadlock: Circular dependency preventing credit return
Sequence Gap: Missing sequence numbers not triggering NAK
LCRC on Replay: Recalculating vs. using cached LCRC

Key Takeaways

Data Link Layer provides reliable delivery via ACK/NAK protocol
DLLPs handle acknowledgments, flow control, and power management
Flow control uses credit-based system with 6 credit types
12-bit sequence numbers track TLP ordering
Replay buffer + timer handle retransmission
LCRC (32-bit) protects data integrity

Next Up

Next: Part 4 — Transaction Layer

3. PCIe for DV Engineers - Data Link Layer