Initial commit

04-quantization-scheduling.md

@@ -0,0 +1,514 @@
+# 方向4: 量化与精度感知调度
+
+## 1. 问题陈述
+
+LLM的量化(Int8/Int4)不仅影响计算精度和模型大小，还直接影响RTOS调度层面的行为：
+
+```
+量化精度 → 内存占用 → 数据传输量 → 计算周期 → 调度时间片 → 优先级调整
+```
+
+量化不仅是模型层面的优化，更是**调度层面的参数**。调度器需要感知量化精度，动态调整：
+1. 任务的执行时间估计
+2. 内存带宽需求
+3. 精度切换时的同步策略
+
+## 2. 量化层级分析
+
+### 2.1 LLM各组件的量化粒度
+
+```
+Component          | Typical Precision | Quantization Method
+-------------------|-------------------|---------------------
+Embedding          | FP16              | None (high sensitivity)
+Attention Q/K/V    | FP16 / Int8       | Per-tensor / Per-channel
+Attention Out Proj | FP16 / Int8       | Per-tensor
+FFN Gate           | FP16 / Int8       | Per-channel
+FFN Up             | FP16 / Int8       | Per-channel
+FFN Down           | FP16 / Int8       | Per-tensor
+LM Head            | FP16              | Per-tensor
+KV Cache           | FP16 / Int8       | Per-token
+
+Recommendation for Edge:
+  - MCU/Edge:  QAT (Quantization-Aware Training) Int8
+  - SoC:      AWQ (Activation-aware Weight Quantization) Int4
+  - Server:   FP8 / FP16 (abundant compute)
+```
+
+### 2.2 量化对调度参数的影响
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│  量化精度 → 调度参数映射                                     │
+├──────────────────────────────────────────────────────────────┤
+│                                                              │
+│  FP16 (基准):                                                │
+│    C_attention = 100μs  (base WCET)                         │
+│    BW_attention = 2.0 GB/s                                   │
+│    Memory = 2.0 GB (KV Cache)                               │
+│                                                              │
+│  Int8:                                                      │
+│    C_attention = 80μs  (-20% latency, 2x less bits)         │
+│    BW_attention = 1.5 GB/s  (less bandwidth)               │
+│    Memory = 1.0 GB (half KV Cache size)                    │
+│                                                              │
+│  Int4:                                                      │
+│    C_attention = 70μs  (-30% latency)                       │
+│    BW_attention = 1.2 GB/s                                  │
+│    Memory = 0.5 GB (quarter KV Cache)                      │
+│                                                              │
+│  Impact on Scheduling:                                      │
+│    - WCET decreases → can increase priority                 │
+│    - Bandwidth decreases → less contention                  │
+│    - Memory decreases → more concurrent requests            │
+│    - BUT: calibration needed → adds overhead                │
+│                                                              │
+└──────────────────────────────────────────────────────────────┘
+```
+
+## 3. 量化感知调度模型
+
+### 3.1 精度-调度联合建模
+
+```c
+// 扩展task模型, 加入精度感知
+typedef struct {
+    // 原有字段
+    uint32_t priority;
+    uint32_t wcet;
+    
+    // 精度感知字段
+    uint8_t precision;        // 0=FP16, 1=Int8, 2=Int4, 3=FP8
+    uint8_t quant_method;     // 0=PTQ, 1=QAT, 2=AWQ
+    uint32_t calibration_cost_us;  // 校准成本
+    float accuracy_loss;      // 精度损失百分比
+    
+    // 动态字段
+    uint32_t current_precision;  // 当前运行精度
+    bool precision_locked;       // 精度是否锁定
+} quant_aware_task_t;
+
+// 动态WCET计算: 基于精度
+uint32_t calc_wcet(quant_aware_task_t *task) {
+    // Base WCET at FP16
+    uint32_t base_wcet = task->wcet;
+    
+    // Precision scaling factor
+    float scale = 1.0f;
+    switch (task->current_precision) {
+        case 1: scale = 0.80f; break;  // Int8: 20% faster
+        case 2: scale = 0.70f; break;  // Int4: 30% faster
+        case 3: scale = 0.90f; break;  // FP8: 10% faster
+    }
+    
+    // Quantization method overhead
+    switch (task->quant_method) {
+        case 0: break;  // PTQ: no calibration overhead
+        case 1: return base_wcet * scale + task->calibration_cost_us;
+        case 2: return base_wcet * scale + task->calibration_cost_us;
+    }
+    
+    return (uint32_t)(base_wcet * scale);
+}
+```
+
+### 3.2 动态精度调度
+
+**核心思路**：根据系统负载和延迟需求，动态调整推理精度
+
+```
+High Load Scenario (CPU/NPU saturated):
+  → 降低精度(Int8→Int4)
+  → 减少计算量, 降低WCET
+  → 提高系统吞吐量
+
+Low Load Scenario (plenty of resources):
+  → 提高精度(Int4→Int8→FP16)
+  → 提高输出质量
+  → 满足SLA要求
+
+Low Latency Requirement:
+  → 降低精度(Int8→Int4)
+  → 降低WCET
+  → 优先满足延迟SLA
+
+High Quality Requirement:
+  → 提高精度(Int4→FP16)
+  → 牺牲部分性能
+  → 优先满足质量SLA
+```
+
+### 3.3 精度感知优先级调整
+
+```
+调度策略: 精度作为优先级的输入参数
+
+Priority = f(urgency, precision, deadline)
+
+Where:
+  urgency: 任务的紧急程度 (TTFT vs TPOT)
+  precision: 当前精度级别 (FP16 > Int8 > Int4)
+  deadline: 截止时间
+
+When precision drops:
+  WCET decreases → task finishes faster → priority can be increased
+  BUT accuracy_loss increases → may need to boost back later
+
+Dynamic Priority Adjustment:
+  1. Monitor system load (CPU/NPU utilization)
+  2. If load > threshold, reduce precision
+  3. Recalculate WCET with new precision
+  4. Update task priority based on new WCET
+  5. Adjust scheduling to maintain SLA
+
+RTOS Implementation:
+  void adjust_priority_for_precision(task_t *task) {
+      uint32_t wcet = calc_wcet_with_precision(task);
+      task->priority = wcet_to_priority(wcet);
+      
+      // High precision = higher priority (quality-critical)
+      if (task->precision == FP16) {
+          task->priority += PRECISION_BOOST;
+      }
+      
+      // Update scheduling parameters
+      task->deadline = task->period - wcet;
+      reschedule_with_edf(task);
+  }
+```
+
+## 4. 精度切换的RTOS同步
+
+### 4.1 同步原语
+
+```
+精度切换流程:
+┌─────────────────────────────────────────────────────────┐
+│  1. Control task decides to change precision             │
+│  2. Signal all inference tasks                           │
+│  3. Wait for barrier (all tasks at sync point)           │
+│  4. Apply new precision to weights/KV                    │
+│  5. Resume inference with new precision                  │
+└─────────────────────────────────────────────────────────┘
+
+RTOS Barrier实现:
+  // 精度切换barrier
+  SemaphoreHandle_t precision_barrier;  // 计数barrier
+  EventGroupHandle_t precision_sync;    // 事件同步
+
+  // Control task
+  void switch_precision(Precision new_prec) {
+      // 1. Signal all tasks
+      xEventGroupSetBits(precision_sync, SYNC_PRECISION_CHANGE);
+      
+      // 2. Wait for all tasks to acknowledge
+      EventBits_t bits = xEventGroupWaitBits(
+          precision_sync,
+          SYNC_ALL_ACK,
+          pdTRUE,  // 清除bits
+          EVENT_BITS_ALL_COMPLETED,
+          portMAX_DELAY
+      );
+      
+      // 3. Apply precision change
+      apply_precision(new_prec);
+      
+      // 4. Release barrier
+      for (int i = 0; i < N_TASKS; i++) {
+          xSemaphoreGive(precision_barrier);
+      }
+  }
+
+  // Inference task
+  void inference_task(void *params) {
+      for (;;) {
+          // Wait for precision change signal
+          EventBits_t bits = xEventGroupWaitBits(
+              precision_sync,
+              SYNC_PRECISION_CHANGE,
+              pdTRUE,
+              pdFALSE,
+              portMAX_DELAY
+          );
+          
+          if (bits & SYNC_PRECISION_CHANGE) {
+              // Acknowledge
+              xEventGroupSetBits(precision_sync, SYNC_ALL_ACK);
+              
+              // Wait at barrier
+              xSemaphoreTake(precision_barrier, portMAX_DELAY);
+              
+              // Apply new precision
+              apply_precision(current_precision);
+              
+              // Release barrier
+              xSemaphoreGive(precision_barrier);
+          }
+      }
+  }
+```
+
+### 4.2 精度切换开销建模
+
+```
+精度切换开销 = Preparing new precision + Swapping weights + Syncing
+
+Preparation:   50-200μs   (量化参数准备)
+Weight Swap:   100-500μs  (内存拷贝, 取决于模型大小)
+Sync:          50-100μs   (barrier同步)
+Total:         200-800μs
+
+Impact on Scheduling:
+  - 精度切换期间所有推理task暂停
+  - 切换时间需要计入RT调度分析
+  - 切换频率需控制 (避免频繁切换导致的调度抖动)
+
+Scheduling Adjustment:
+  - 将切换时间计入task的overhead
+  - 切换期间的task挂起不消耗调度时间片
+  - 切换完成后的task恢复需保持原有优先级
+```
+
+## 5. MoE (Mixture of Experts) 调度
+
+### 5.1 MoE架构分析
+
+```
+MoE Layer Structure:
+  Input (d_model) → Router → Expert Selection → Expert Computation → Combine
+
+  Router:        TopK routing (e.g., Top2 = 2 experts per token)
+  Expert:        FFN with shared weights
+  Load Balance:  Each expert has capacity limit
+
+  Example (Qwen2.5-72B-MoE):
+    - 64 experts, each FFN = 8B params
+    - Top2 routing → 16B active params per token
+    - Effective compute: 16B / 72B = 22% of full model
+  
+  MoE Scheduling Challenge:
+    - Router decision is dynamic → task graph changes at runtime
+    - Different experts have different execution times
+    - Expert capacity limits → queuing when overloaded
+```
+
+### 5.2 MoE的RTOS调度
+
+```
+MoE Layer Scheduling:
+┌──────────────────────────────────────────────────────┐
+│  Input Token → Router Task (LOW priority)            │
+│       ↓                                              │
+│  Expert Tasks (MEDIUM priority) × K_experts          │
+│  [Expert A] [Expert B] [Expert C] [Expert D]         │
+│       ↓       ↓          ↓          ↓                │
+│  Combine Task (HIGH priority)                        │
+└──────────────────────────────────────────────────────┘
+
+Scheduling Strategy:
+  1. Router: 最低优先级, 可延迟, 不影响关键路径
+  2. Expert: 中优先级, 并行执行, 独立task
+  3. Combine: 高优先级, 等待所有expert完成
+
+RTOS Implementation:
+  // Expert调度使用parallel task pool
+  void schedule_moe_layer(uint32_t *expert_ids, int k) {
+      for (int i = 0; i < k; i++) {
+          xTaskNotify(expert_tasks[expert_ids[i]], 
+                      token_data, eIncrement);
+      }
+      
+      // Wait for all experts
+      for (int i = 0; i < k; i++) {
+          xEventGroupWaitBits(complete_bits, 
+                              (1 << expert_ids[i]),
+                              pdTRUE, pdFALSE, portMAX_DELAY);
+      }
+  }
+```
+
+### 5.3 Expert负载均衡
+
+```
+问题: 某些expert可能被过多token激活, 造成排队
+
+负载不均衡:
+  Expert A: ████████████████████ (loaded)
+  Expert B: ████ (unloaded)
+  Expert C: ████████ (half-loaded)
+  Expert D: █ (nearly idle)
+
+RTOS负载均衡策略:
+  1. Capacity Tracking: 每个expert维护capacity counter
+  2. Priority Adjustment: 排队超限时, 临时提升priority
+  3. Queue Migration: 将排队中的task迁移到空闲expert
+  4. Load-aware Routing: Router感知负载, 调整TopK选择
+
+RTOS实现:
+  typedef struct {
+      uint32_t capacity;        // 总容量
+      uint32_t queued;          // 当前排队数
+      uint32_t executing;       // 当前执行数
+      uint32_t max_queue;       // 最大排队数
+      uint32_t priority_base;   // 基础优先级
+  } expert_load_t;
+
+  void adjust_export_priority(expert_load_t *load) {
+      // 排队越多, 优先级越高 (防止饿死)
+      uint32_t priority_boost = load->queued / load->max_queue;
+      task_priority = load->priority_base + priority_boost;
+  }
+```
+
+## 6. 精度与调度的联合优化
+
+### 6.1 优化目标
+
+```
+Maximize:  Throughput (tokens/sec)
+Subject to:
+  - WCL ≤ 200ms (硬实时约束)
+  - Accuracy ≥ 95% (质量约束)
+  - Power ≤ 5W (功耗约束)
+
+Decision Variables:
+  - Precision per layer (FP16/Int8/Int4)
+  - Scheduling strategy (FP/EDF/Hybrid)
+  - Batch size (concurrent requests)
+  - KV Cache size
+
+Trade-off:
+  Lower precision → Lower latency → Higher throughput
+  BUT → Lower accuracy → May violate quality constraint
+  
+  Higher precision → Higher accuracy → Lower throughput
+  BUT → May violate latency constraint
+
+Solution: Multi-objective optimization
+  - Find Pareto frontier of (latency, accuracy, throughput)
+  - Select operating point based on system mode
+```
+
+### 6.2 运行时模式切换
+
+```
+┌──────────────────────────────────────────────────────────┐
+│  System Modes                                             │
+├──────────────────────────────────────────────────────────┤
+│                                                           │
+│  MODE PERFORMANCE (性能模式):                             │
+│    - Precision: Int4 (lowest precision)                  │
+│    - Scheduling: Max throughput, aggressive batching     │
+│    - KV Cache: Large pool, aggressive eviction           │
+│    - Power: Max freq, no thermal limit                   │
+│    - Use: Real-time response, latency-critical           │
+│                                                           │
+│  MODE BALANCED (均衡模式):                                │
+│    - Precision: Int8 (balanced)                          │
+│    - Scheduling: Balanced throughput + latency           │
+│    - KV Cache: Moderate pool, moderate eviction          │
+│    - Power: Medium freq, thermal-aware                   │
+│    - Use: General purpose                                │
+│                                                           │
+│  MODE ACCURACY (精度模式):                                │
+│    - Precision: FP16 (highest precision)                 │
+│    - Scheduling: Lower throughput, prioritize quality    │
+│    - KV Cache: Large pool, conservative eviction         │
+│    - Power: May throttle for stability                   │
+│    - Use: High-quality output required                   │
+│                                                           │
+│  MODE POWER (省电模式):                                   │
+│    - Precision: Int4 + DVFS low freq                     │
+│    - Scheduling: Aggressive idle, power-saving           │
+│    - KV Cache: Minimal pool, aggressive eviction         │
+│    - Power: Min freq, aggressive sleep                   │
+│    - Use: Battery-powered, standby                       │
+│                                                           │
+│  Mode Transitions (RTOS-aware):                           │
+│    1. Mode change signaled by control task               │
+│    2. All tasks synchronize at barrier                   │
+│    3. Parameters updated (precision, priority, etc.)    │
+│    4. Resume with new parameters                         │
+└──────────────────────────────────────────────────────────┘
+```
+
+## 7. 各硬件平台的精度-调度差异
+
+### 7.1 MCU级
+
+```
+约束:
+- 仅支持INT8 (硬件限制, 无FPunit)
+- 无动态精度切换
+- 精度固定, 调度简单
+
+策略:
+- 静态精度(Int8), 固定优先级调度
+- 关注: 精度校准对WCET的影响
+- 量化开销: 固定, 可预先计算
+```
+
+### 7.2 SoC级
+
+```
+约束:
+- 支持INT8/INT4/FP16 (取决于NPU)
+- 动态精度切换可能有限
+- NPU驱动可能不支持运行时精度调整
+
+策略:
+- 静态精度选择, 启动时配置
+- 精度影响调度参数(WCET)
+- 可能支持精度切换但非无缝
+```
+
+### 7.3 Edge盒子
+
+```
+约束:
+- GPU支持FP16/FP32/INT8动态切换
+- 丰富的精度选项
+- 成熟软件栈支持
+
+策略:
+- 运行时动态精度调整
+- 基于负载的精度自适应
+- 精度感知调度优化
+```
+
+### 7.4 Server级
+
+```
+约束:
+- 全精度支持, 动态调整
+- 丰富的软件生态
+- 调度算法成熟
+
+策略:
+- 精细化精度调度
+- 各层不同精度
+- 量化感知 serving (vLLM FP8 support)
+```
+
+## 8. 关键设计决策
+
+| 决策点 | 选项 | 推荐 | 理由 |
+|-------|------|-----|------|
+| 精度选择 | 静态 / 动态 | **Hybrid** | 启动静态+运行时微调 |
+| 精度粒度 | Layer-level / Token-level | **Layer-level** | 实现简单+精度可接受 |
+| 切换开销 | 计入WCET / 不计入 | **计入WCET** | 实时性分析准确 |
+| 负载均衡 | 静态 / 动态 | **动态** | MoE负载不均衡常见 |
+| 模式切换 | 手动 / 自动 | **自动** | 自适应系统负载 |
+| 校准策略 | Offline / Online | **Offline** | 在线校准开销大 |
+
+## 9. 开放研究问题
+
+1. **Layer-specific Precision**: 每层不同精度对调度有何影响？
+2. **Precision Prediction**: 预测最佳精度, 避免频繁切换？
+3. **MoE Load Balancing in RT**: MoE的负载均衡如何满足实时约束？
+4. **Precision-aware Memory**: 精度变化时的内存自动伸缩？
+5. **Cross-model Precision**: 多模型共存时的精度分配？
+
+---
+
+*最后更新: 2026-09-17*