rtos_llm_opt/reference_file/05-irq-realtime.md

方向5: 中断管理与实时性保障

1. 问题陈述

RTOS上的LLM推理不是孤立的系统。设备中还有其他硬实时任务(传感器、通信、控制)，它们与LLM任务共存于同一个RTOS内核中。核心问题：

1. 抢占冲突: LLM的大计算量可能饿死其他实时任务
2. 中断风暴: NPU完成中断 + 通信中断 + 传感器中断的并发处理
3. Priority Inversion: 低优先级的LLM任务可能阻塞高优先级任务
4. 资源竞争: IRQ line、DMA channel、内存的共享竞争

2. 中断层级设计

2.1 中断优先级映射

┌─────────────────────────────────────────────────────────────────┐
│  IRQ Hierarchy (示例: 基于ARM GIC)                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  Priority 255 (最高):  Hard fault, Reset                          │
│  Priority 254:          Watchdog, Critical error                  │
│  Priority 253:          Real-time control (motor, actuator)      │
│  Priority 252:          Hard comm (ethernet MAC)                  │
│  Priority 251:          Sensor interrupt (IMU, camera)           │
│  Priority 250:          Network interrupt (TCP timeout)          │
│  ──────────────────────────────────────────────                   │
│  Priority 200:          NPU/GPU completion interrupt             │
│  Priority 199:          DMA completion                           │
│  Priority 198:          Timer interrupt (tick)                   │
│  ──────────────────────────────────────────────                   │
│  Priority 100:          Software interrupt (task notification)  │
│  Priority 50:           Low-priority I2C/SPI                    │
│  Priority 0 (最低):     Idle interrupt                           │
└─────────────────────────────────────────────────────────────────┘

Design Rule:
  Real-time control tasks > NPU completion > DMA > Timer > Software

2.2 中断类型与处理策略

┌─────────────────────────────────────────────────────────────┐
│  IRQ Type          | Handling Strategy                      │
├────────────────────┼────────────────────────────────────────┤
│  NPU Completion    | 立即唤醒Attention task, 不延迟         │
│  DMA Complete      | 标记完成, 通过event通知task            │
│  Timer Tick        | 标准tick, 用于调度时间片               │
│  Sensor Data       | 高优先级, 触发数据处理task             │
│  Comm Timeout      | 中优先级, 可能触发重连task             │
│  Control Command   | 最高优先级, 触发控制task               │
│  Error/Exception   | 最高优先级, 触发error handling task    │
└─────────────────────────────────────────────────────────────┘

3. NPU/GPU完成中断处理

3.1 中断处理流程

NPU完成指令 → Interrupt Controller → RTOS IRQ Handler

  1. NPU完成当前层 → 触发IRQ
  2. ARM GIC收到中断 → 保存寄存器, 跳转到ISR
  3. ISR: 读取NPU状态寄存器 → 确认完成 → 清中断
  4. ISR: xSemaphoreGiveFromISR() → 唤醒等待task
  5. ISR: 如果有更高优先级task ready → portYIELD_FROM_ISR()
  6. 返回 → 恢复寄存器 → 执行task

3.2 ISR设计原则

// NPU完成ISR (简化版)
volatile uint32_t npu_status_reg;  // 硬件寄存器
SemaphoreHandle_t npu_done_sem;    // RTOS信号量

void NPU_IRQ_Handler(void) {
    // 1. 读取并确认NPU状态 (纯硬件操作)
    uint32_t status = npu_status_reg;
    if (status & NPU_STATUS_COMPLETE) {
        // 2. 写寄存器清除中断标志
        npu_status_reg &= ~NPU_STATUS_COMPLETE;
        
        // 3. 释放信号量 (中断上下文)
        BaseType_t xHigherPriorityTaskWoken = pdFALSE;
        xSemaphoreGiveFromISR(npu_done_sem, &xHigherPriorityTaskWoken);
        
        // 4. 如果有更高优先级task ready, 立即切换
        if (xHigherPriorityTaskWoken) {
            portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
        }
    }
}

// ISR设计原则:
// 1. 最小化: ISR中只做最少的操作 (<10μs)
// 2. 不阻塞: ISR中不使用sleep/等待
// 3. 用FromISR版本: 所有RTOS API都用ISR版本
// 4. 状态寄存器直接访问: 硬件寄存器直接读写
// 5. 事件通知为主: 优先使用event/group, 避免queue拷贝

4. Priority Inversion处理

4.1 经典场景分析

Priority Inversion in LLM Inference:

Task A (HIGH): Real-time control (needs NPU results)
Task B (MEDIUM): LLM Attention (uses NPU, holds mutex)
Task C (LOW): Background logging (locks NPU mutex)

Timeline:
  t0: Task C acquires NPU mutex
  t1: Task B starts, wants NPU → BLOCKED (waiting for C)
  t2: Task A starts, wants NPU result → BLOCKED (waiting for B)
  t3: Preemptive? No - C is LOW, B is MEDIUM
  → A waits for B, B waits for C
  → A is blocked by lower-priority C = PRIORITY INVERSION

Solution: Priority Inheritance Protocol (PIP)
  t0: Task C acquires NPU mutex
  t2: Task A blocked by B, B blocked by C
  t3: C inherits A's priority → C becomes HIGH
  t4: C releases mutex → C returns to LOW, A can proceed

4.2 RTOS中的PIP实现

// PIP扩展: 适用于LLM推理的分级PIP
typedef struct {
    uint32_t mutex_id;
    uint32_t base_priority;    // 持有者的基础优先级
    uint32_t current_priority; // 继承后的优先级
    TaskHandle_t holder;       // 持有者
    StackType_t *saved_stack;  // 保存的栈指针
} pip_mutex_t;

void pip_take(pip_mutex_t *m) {
    // 保存原始优先级
    uint32_t orig_priority = task_get_priority(m->holder);
    m->base_priority = orig_priority;
    m->current_priority = orig_priority;
    
    // 提升持有者优先级到最高等待者
    uint32_t max_waiter = find_highest_waiting_priority(m->mutex_id);
    task_set_priority(m->holder, max_waiter);
    m->current_priority = max_waiter;
}

void pip_give(pip_mutex_t *m) {
    // 恢复基础优先级
    task_set_priority(m->holder, m->base_priority);
    m->current_priority = m->base_priority;
}

// LLM特例: NPU mutex的PIP
// NPU mutex持有者通常是DMA task或NPU驱动
// 提升为Attention task的优先级

4.3 Enhanced PIP (ePIP) for LLM

Standard PIP的问题: 只继承一次, 可能传递到更高级

Enhanced PIP: 继承所有等待者的最高优先级

Scenario:
  Task A (MAX): Control → waiting for NPU
  Task B (HIGH): Attention → waiting for NPU
  Task C (MED): KV Cache → waiting for NPU
  Task D (LOW): DMA → holding NPU mutex

Standard PIP:
  D inherits MAX (A's priority)
  Problem: B (HIGH) may starve

Enhanced PIP:
  D inherits MAX (A's priority)
  B inherits MAX too (via queue, not mutex)
  All high-priority tasks preempt D

RTOS Implementation:
  // 使用Priority Queue记录所有等待者
  typedef struct {
      pip_mutex_t base;
      PriorityQueueType_t wait_queue;  // RTOS wait queue
      uint32_t max_wait_priority;      // 最高等待优先级
  } epip_mutex_t;

  void epip_take(epip_mutex_t *m) {
      uint32_t max_prio = max_priority_in_queue(m->wait_queue);
      task_set_priority(m->holder, max_prio);
      m->max_wait_priority = max_prio;
  }

5. 中断屏蔽与上下文切换

5.1 中断屏蔽策略

Critical Section (中断屏蔽):
  ┌─────────────────────────────────────────────────────────┐
│  Level 0: Global IRQ Disable                             │
│  使用: 极短的critical section (<5μs)                      │
│  场景: 写硬件寄存器, 更新共享计数器                       │
│  开销: 所有IRQ被屏蔽, 可能错过中断                        │
│                                                         │
│  Level 1: Task-level IRQ Disable                         │
│  使用: task内critical section (≤50μs)                     │
│  场景: 原子更新task状态, 修改task control block           │
│  开销: 只屏蔽当前task的中断, 其他task不受影响              │
│                                                         │
│  Level 2: No IRQ Disable (锁+原子操作)                    │
│  使用: 大多数critical section                             │
│  场景: 修改共享数据结构, 使用atomic操作                    │
│  开销: 最小, 但需要确保原子性                             │
└─────────────────────────────────────────────────────────┘

LLM推理中的critical sections:
  1. Attention结果写入 (原子操作即可, 不需要全局disable)
  2. KV Cache更新 (lock-free queue, 不需要disable)
  3. Scheduler state update (task-level disable)
  4. Precision change flag (global disable for <10μs)

5.2 上下文切换开销

Context Switch in RTOS:
  1. Save current task registers (R0-R12, LR, PC, PSR)
  2. Save/restore stack pointer
  3. Update task control block
  4. Restore next task registers
  5. Restore stack pointer
  6. Execute 'BX LR' (return from exception)

Cost:
  Cortex-M7: ~10-50 cycles (with DCBP) ≈ 1-5μs @ 400MHz
  ARM Cortex-A: ~100-500 cycles ≈ 50-250ns @ 2GHz
  With MMU: ~2-10μs (TLB flush)

LLM Impact:
  - 频繁context switch增加E2E延迟
  - 每个switch增加 ~5μs (M7) ~2μs (A72)
  - 假设每token 10次context switch: +50μs
  
  Reduction strategies:
  - Pin critical tasks to core (reduce switch)
  - Use task notification instead of queue (reduce overhead)
  - Batch task switches (wait for multiple events)

6. 实时性分析

6.1 Response Time Analysis (RTA) with Interrupts

Modified RTA for LLM with interrupts:

R_i = C_i + I_i + Σ_{j∈hp(i)} ⌈R_j / T_j⌉ × C_j

Where:
  R_i: response time of task i
  C_i: execution time of task i
  I_i: interrupt-induced latency (中断导致的额外延迟)
  Σ: interference from higher priority tasks

Interrupt-induced latency:
  I_i = Σ_{k∈interrupts} (ISR_time_k + preemption_delay_k)
  
  ISR_time_k: 中断k的处理时间
  preemption_delay_k: 中断k导致的preemption开销

For LLM:
  I_attention = ISR_time(npu_complete) + ISR_time(dma_complete)
              + preemption(attention → higher_priority_control)
  
  Typical:
    ISR_time(npu) ≈ 2μs
    ISR_time(dma) ≈ 1μs
    Preemption ≈ 5μs
    Total interrupt overhead ≈ 8μs per inference step

6.2 中断导致的Jitter

Jitter来源分析:
  ┌─────────────────────────────────────────────────────────┐
│  Source                  | Latency (μs) | Probability    │
├──────────────────────────┼──────────────┼────────────────┤
│  NPU completion IRQ      | 2-5          | 100% (predictable) │
│  DMA completion IRQ      | 1-3          | 100% (predictable) │
│  Timer tick              | 1-2          | 100% (predictable) │
│  Preemption by control   | 5-10         | Low (<5%)        │
│  Preemption by comm      | 5-15         | Medium (<20%)    │
│  ISR overhead variance   | 1-5          | High             │
│  Cache miss (interrupt)  | 10-50        | Variable         │
└─────────────────────────────────────────────────────────┘

Worst-case Jitter:
  Jitter = Max(R_i) - Min(R_i) across all inference steps
  
  Predictable jitter: NPU/DMA IRQ (deterministic)
  Unpredictable jitter: Preemption, Cache miss (stochastic)
  
  Scheduling strategy:
  - Minimize unpredictable jitter sources
  - Bound predictable jitter through analysis
  - Design with worst-case jitter in mind

7. 各平台的中断差异

7.1 MCU级

中断控制器: NVIC (ARM) / SCB (RISC-V)
中断数量: 10-50
中断优先级: 3-8 bits
特点:
  - 中断延迟低 (<1μs)
  - 中断嵌套简单 (固定层级)
  - 无中断向量表动态重定位
  - RTOS成熟, 中断集成简单

关键中断:
  - SysTick (tick)
  - NPU/ISP (如有)
  - GPIO (传感器)
  - UART/SPI (通信)
  - DMA (数据传输)

7.2 SoC级

中断控制器: GICv3/v4 (ARM) / PLIC (RISC-V)
中断数量: 100-1000+
中断优先级: 8 bits
特点:
  - 中断虚拟化支持 (但NPU驱动可能不暴露)
  - 中断路由复杂 (多个master)
  - MSI/MSI-X支持
  - 中断聚合 (coalescing)

关键中断:
  - ARM CPU cores (each has IRQ)
  - NPU (interrupt via GIC)
  - DMA controllers (multiple)
  - Ethernet (MAC interrupt)
  - Modem/Radio (connectivity)
  - ISP (camera)

7.3 Edge盒子

中断控制器: GICv4 (Jetson Orin)
中断数量: 1000+
中断优先级: 8 bits
特点:
  - 丰富的中断源
  - PCIe MSIX支持
  - GPU中断丰富 (渲染/计算)
  - 中断亲和性可配置

关键中断:
  - GPU (CUDA completion)
  - NVLink (multi-GPU)
  - PCIe (SSD/NPU)
  - Ethernet
  - Timer

7.4 Server级

中断控制器: GICv4 / MSI-X
中断数量: 数千
特点:
  - 中断亲和性精细控制
  - 中断聚合
  - RSS (Receive Side Scattering)
  - 中断向量池

RTOS角色:
  - 主要处理网络/存储中断
  - GPU中断由用户态处理
  - RTOS保证关键路径上的中断响应

8. 关键设计决策

决策点	选项	推荐	理由
PIP/ePIP	PIP / ePIP / No	ePIP	LLM场景优先级复杂
ISR策略	Minimal / Full	Minimal	ISR尽量短
Critical Section	Global / Task-level / Lock-free	混合	按场景选择
Interrupt Mask	Disable / Priority / Vector	Priority	灵活+实时
Timer	Tick / Event	Event	降低开销
IRQ Affinity	Auto / Manual	Manual	保证确定性

9. 开放研究问题

1. Interrupt-aware Scheduling: 将中断处理时间纳入调度分析？
2. Interrupt Storm Handling: 多中断并发时的优先级管理？
3. Predictable Jitter: 如何分析和保证jitter的确定性？
4. Cross-core Interrupt: 多核间的中断传播与同步？
5. Interrupt-less Inference: 轮询模式下的LLM调度？

---

最后更新: 2026-09-17