545 lines
20 KiB
Markdown
545 lines
20 KiB
Markdown
# 方向4: 量化与精度感知调度
|
||
|
||
## 1. 问题陈述
|
||
|
||
LLM的量化(Int8/Int4)不仅影响计算精度和模型大小，还直接影响RTOS调度层面的行为：
|
||
|
||
```
|
||
量化精度 → 内存占用 → 数据传输量 → 计算周期 → 调度时间片 → 优先级调整
|
||
```
|
||
|
||
量化不仅是模型层面的优化，更是**调度层面的参数**。调度器需要感知量化精度，动态调整：
|
||
1. 任务的执行时间估计
|
||
2. 内存带宽需求
|
||
3. 精度切换时的同步策略
|
||
|
||
### 1.1 本方向在总课题中的角色
|
||
|
||
本方向聚焦：
|
||
|
||
- 如何把量化精度纳入 RTOS 可分析的调度参数；
|
||
- 如何在人工智能目标负载时效性与输出有效性之间建立可控权衡；
|
||
- 如何避免精度切换、校准开销和 MoE 负载波动破坏关键保障负载的实时边界。
|
||
|
||
因此，这一方向服务的是总课题中的“质量-时效联合调度”主线。
|
||
|
||
### 1.2 与验证矩阵的对应关系
|
||
|
||
量化与精度感知调度在 `T5~T1` 中的作用方式不同，因此需要按部署形态看重点：
|
||
|
||
| 部署形态 | 精度调度侧重点 |
|
||
|---|---|
|
||
| `T5` 控制端 | 固定低精度、静态校准与可预测执行时间 |
|
||
| `T4` 终端设备 | Int8/Int4 下的质量-时延平衡 |
|
||
| `T3` 边缘节点 | 负载波动下的动态精度与服务模式切换 |
|
||
| `T2` 单机工作站 | 多精度混合与大上下文推理的联合优化 |
|
||
| `T1` 服务器/集群 | 多模型、多租户下的精度策略编排 |
|
||
|
||
## 2. 量化层级分析
|
||
|
||
### 2.1 LLM各组件的量化粒度
|
||
|
||
```
|
||
Component | Typical Precision | Quantization Method
|
||
-------------------|-------------------|---------------------
|
||
Embedding | FP16 | None (high sensitivity)
|
||
Attention Q/K/V | FP16 / Int8 | Per-tensor / Per-channel
|
||
Attention Out Proj | FP16 / Int8 | Per-tensor
|
||
FFN Gate | FP16 / Int8 | Per-channel
|
||
FFN Up | FP16 / Int8 | Per-channel
|
||
FFN Down | FP16 / Int8 | Per-tensor
|
||
LM Head | FP16 | Per-tensor
|
||
KV Cache | FP16 / Int8 | Per-token
|
||
|
||
Recommendation for Edge:
|
||
- MCU/Edge: QAT (Quantization-Aware Training) Int8
|
||
- SoC: AWQ (Activation-aware Weight Quantization) Int4
|
||
- Server: FP8 / FP16 (abundant compute)
|
||
```
|
||
|
||
### 2.2 量化对调度参数的影响
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ 量化精度 → 调度参数映射 │
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ FP16 (基准): │
|
||
│ C_attention = 100μs (base WCET) │
|
||
│ BW_attention = 2.0 GB/s │
|
||
│ Memory = 2.0 GB (KV Cache) │
|
||
│ │
|
||
│ Int8: │
|
||
│ C_attention = 80μs (-20% latency, 2x less bits) │
|
||
│ BW_attention = 1.5 GB/s (less bandwidth) │
|
||
│ Memory = 1.0 GB (half KV Cache size) │
|
||
│ │
|
||
│ Int4: │
|
||
│ C_attention = 70μs (-30% latency) │
|
||
│ BW_attention = 1.2 GB/s │
|
||
│ Memory = 0.5 GB (quarter KV Cache) │
|
||
│ │
|
||
│ Impact on Scheduling: │
|
||
│ - WCET decreases → can increase priority │
|
||
│ - Bandwidth decreases → less contention │
|
||
│ - Memory decreases → more concurrent requests │
|
||
│ - BUT: calibration needed → adds overhead │
|
||
│ │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## 3. 量化感知调度模型
|
||
|
||
### 3.1 精度-调度联合建模
|
||
|
||
```c
|
||
// 扩展task模型, 加入精度感知
|
||
typedef struct {
|
||
// 原有字段
|
||
uint32_t priority;
|
||
uint32_t wcet;
|
||
|
||
// 精度感知字段
|
||
uint8_t precision; // 0=FP16, 1=Int8, 2=Int4, 3=FP8
|
||
uint8_t quant_method; // 0=PTQ, 1=QAT, 2=AWQ
|
||
uint32_t calibration_cost_us; // 校准成本
|
||
float accuracy_loss; // 精度损失百分比
|
||
|
||
// 动态字段
|
||
uint32_t current_precision; // 当前运行精度
|
||
bool precision_locked; // 精度是否锁定
|
||
} quant_aware_task_t;
|
||
|
||
// 动态WCET计算: 基于精度
|
||
uint32_t calc_wcet(quant_aware_task_t *task) {
|
||
// Base WCET at FP16
|
||
uint32_t base_wcet = task->wcet;
|
||
|
||
// Precision scaling factor
|
||
float scale = 1.0f;
|
||
switch (task->current_precision) {
|
||
case 1: scale = 0.80f; break; // Int8: 20% faster
|
||
case 2: scale = 0.70f; break; // Int4: 30% faster
|
||
case 3: scale = 0.90f; break; // FP8: 10% faster
|
||
}
|
||
|
||
// Quantization method overhead
|
||
switch (task->quant_method) {
|
||
case 0: break; // PTQ: no calibration overhead
|
||
case 1: return base_wcet * scale + task->calibration_cost_us;
|
||
case 2: return base_wcet * scale + task->calibration_cost_us;
|
||
}
|
||
|
||
return (uint32_t)(base_wcet * scale);
|
||
}
|
||
```
|
||
|
||
### 3.2 动态精度调度
|
||
|
||
**核心思路**：根据系统负载和延迟需求，动态调整推理精度
|
||
|
||
```
|
||
High Load Scenario (CPU/NPU saturated):
|
||
→ 降低精度(Int8→Int4)
|
||
→ 减少计算量, 降低WCET
|
||
→ 提高系统吞吐量
|
||
|
||
Low Load Scenario (plenty of resources):
|
||
→ 提高精度(Int4→Int8→FP16)
|
||
→ 提高输出质量
|
||
→ 满足SLA要求
|
||
|
||
Low Latency Requirement:
|
||
→ 降低精度(Int8→Int4)
|
||
→ 降低WCET
|
||
→ 优先满足延迟SLA
|
||
|
||
High Quality Requirement:
|
||
→ 提高精度(Int4→FP16)
|
||
→ 牺牲部分性能
|
||
→ 优先满足质量SLA
|
||
```
|
||
|
||
### 3.3 精度感知优先级调整
|
||
|
||
```
|
||
调度策略: 精度作为优先级的输入参数
|
||
|
||
Priority = f(urgency, precision, deadline)
|
||
|
||
Where:
|
||
urgency: 任务的紧急程度 (TTFT vs TPOT)
|
||
precision: 当前精度级别 (FP16 > Int8 > Int4)
|
||
deadline: 截止时间
|
||
|
||
When precision drops:
|
||
WCET decreases → task finishes faster → priority can be increased
|
||
BUT accuracy_loss increases → may need to boost back later
|
||
|
||
Dynamic Priority Adjustment:
|
||
1. Monitor system load (CPU/NPU utilization)
|
||
2. If load > threshold, reduce precision
|
||
3. Recalculate WCET with new precision
|
||
4. Update task priority based on new WCET
|
||
5. Adjust scheduling to maintain SLA
|
||
|
||
RTOS Implementation:
|
||
void adjust_priority_for_precision(task_t *task) {
|
||
uint32_t wcet = calc_wcet_with_precision(task);
|
||
task->priority = wcet_to_priority(wcet);
|
||
|
||
// High precision = higher priority (quality-critical)
|
||
if (task->precision == FP16) {
|
||
task->priority += PRECISION_BOOST;
|
||
}
|
||
|
||
// Update scheduling parameters
|
||
task->deadline = task->period - wcet;
|
||
reschedule_with_edf(task);
|
||
}
|
||
```
|
||
|
||
## 4. 精度切换的RTOS同步
|
||
|
||
### 4.1 同步原语
|
||
|
||
```
|
||
精度切换流程:
|
||
┌─────────────────────────────────────────────────────────┐
|
||
│ 1. Control task decides to change precision │
|
||
│ 2. Signal all inference tasks │
|
||
│ 3. Wait for barrier (all tasks at sync point) │
|
||
│ 4. Apply new precision to weights/KV │
|
||
│ 5. Resume inference with new precision │
|
||
└─────────────────────────────────────────────────────────┘
|
||
|
||
RTOS Barrier实现:
|
||
// 精度切换barrier
|
||
SemaphoreHandle_t precision_barrier; // 计数barrier
|
||
EventGroupHandle_t precision_sync; // 事件同步
|
||
|
||
// Control task
|
||
void switch_precision(Precision new_prec) {
|
||
// 1. Signal all tasks
|
||
xEventGroupSetBits(precision_sync, SYNC_PRECISION_CHANGE);
|
||
|
||
// 2. Wait for all tasks to acknowledge
|
||
EventBits_t bits = xEventGroupWaitBits(
|
||
precision_sync,
|
||
SYNC_ALL_ACK,
|
||
pdTRUE, // 清除bits
|
||
EVENT_BITS_ALL_COMPLETED,
|
||
portMAX_DELAY
|
||
);
|
||
|
||
// 3. Apply precision change
|
||
apply_precision(new_prec);
|
||
|
||
// 4. Release barrier
|
||
for (int i = 0; i < N_TASKS; i++) {
|
||
xSemaphoreGive(precision_barrier);
|
||
}
|
||
}
|
||
|
||
// Inference task
|
||
void inference_task(void *params) {
|
||
for (;;) {
|
||
// Wait for precision change signal
|
||
EventBits_t bits = xEventGroupWaitBits(
|
||
precision_sync,
|
||
SYNC_PRECISION_CHANGE,
|
||
pdTRUE,
|
||
pdFALSE,
|
||
portMAX_DELAY
|
||
);
|
||
|
||
if (bits & SYNC_PRECISION_CHANGE) {
|
||
// Acknowledge
|
||
xEventGroupSetBits(precision_sync, SYNC_ALL_ACK);
|
||
|
||
// Wait at barrier
|
||
xSemaphoreTake(precision_barrier, portMAX_DELAY);
|
||
|
||
// Apply new precision
|
||
apply_precision(current_precision);
|
||
|
||
// Release barrier
|
||
xSemaphoreGive(precision_barrier);
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 4.2 精度切换开销建模
|
||
|
||
```
|
||
精度切换开销 = Preparing new precision + Swapping weights + Syncing
|
||
|
||
Preparation: 50-200μs (量化参数准备)
|
||
Weight Swap: 100-500μs (内存拷贝, 取决于模型大小)
|
||
Sync: 50-100μs (barrier同步)
|
||
Total: 200-800μs
|
||
|
||
Impact on Scheduling:
|
||
- 精度切换期间所有推理task暂停
|
||
- 切换时间需要计入RT调度分析
|
||
- 切换频率需控制 (避免频繁切换导致的调度抖动)
|
||
|
||
Scheduling Adjustment:
|
||
- 将切换时间计入task的overhead
|
||
- 切换期间的task挂起不消耗调度时间片
|
||
- 切换完成后的task恢复需保持原有优先级
|
||
```
|
||
|
||
## 5. MoE (Mixture of Experts) 调度
|
||
|
||
### 5.1 MoE架构分析
|
||
|
||
```
|
||
MoE Layer Structure:
|
||
Input (d_model) → Router → Expert Selection → Expert Computation → Combine
|
||
|
||
Router: TopK routing (e.g., Top2 = 2 experts per token)
|
||
Expert: FFN with shared weights
|
||
Load Balance: Each expert has capacity limit
|
||
|
||
Example (Qwen2.5-72B-MoE):
|
||
- 64 experts, each FFN = 8B params
|
||
- Top2 routing → 16B active params per token
|
||
- Effective compute: 16B / 72B = 22% of full model
|
||
|
||
MoE Scheduling Challenge:
|
||
- Router decision is dynamic → task graph changes at runtime
|
||
- Different experts have different execution times
|
||
- Expert capacity limits → queuing when overloaded
|
||
```
|
||
|
||
### 5.2 MoE的RTOS调度
|
||
|
||
```
|
||
MoE Layer Scheduling:
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Input Token → Router Task (LOW priority) │
|
||
│ ↓ │
|
||
│ Expert Tasks (MEDIUM priority) × K_experts │
|
||
│ [Expert A] [Expert B] [Expert C] [Expert D] │
|
||
│ ↓ ↓ ↓ ↓ │
|
||
│ Combine Task (HIGH priority) │
|
||
└──────────────────────────────────────────────────────┘
|
||
|
||
Scheduling Strategy:
|
||
1. Router: 最低优先级, 可延迟, 不影响关键路径
|
||
2. Expert: 中优先级, 并行执行, 独立task
|
||
3. Combine: 高优先级, 等待所有expert完成
|
||
|
||
RTOS Implementation:
|
||
// Expert调度使用parallel task pool
|
||
void schedule_moe_layer(uint32_t *expert_ids, int k) {
|
||
for (int i = 0; i < k; i++) {
|
||
xTaskNotify(expert_tasks[expert_ids[i]],
|
||
token_data, eIncrement);
|
||
}
|
||
|
||
// Wait for all experts
|
||
for (int i = 0; i < k; i++) {
|
||
xEventGroupWaitBits(complete_bits,
|
||
(1 << expert_ids[i]),
|
||
pdTRUE, pdFALSE, portMAX_DELAY);
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.3 Expert负载均衡
|
||
|
||
```
|
||
问题: 某些expert可能被过多token激活, 造成排队
|
||
|
||
负载不均衡:
|
||
Expert A: ████████████████████ (loaded)
|
||
Expert B: ████ (unloaded)
|
||
Expert C: ████████ (half-loaded)
|
||
Expert D: █ (nearly idle)
|
||
|
||
RTOS负载均衡策略:
|
||
1. Capacity Tracking: 每个expert维护capacity counter
|
||
2. Priority Adjustment: 排队超限时, 临时提升priority
|
||
3. Queue Migration: 将排队中的task迁移到空闲expert
|
||
4. Load-aware Routing: Router感知负载, 调整TopK选择
|
||
|
||
RTOS实现:
|
||
typedef struct {
|
||
uint32_t capacity; // 总容量
|
||
uint32_t queued; // 当前排队数
|
||
uint32_t executing; // 当前执行数
|
||
uint32_t max_queue; // 最大排队数
|
||
uint32_t priority_base; // 基础优先级
|
||
} expert_load_t;
|
||
|
||
void adjust_export_priority(expert_load_t *load) {
|
||
// 排队越多, 优先级越高 (防止饿死)
|
||
uint32_t priority_boost = load->queued / load->max_queue;
|
||
task_priority = load->priority_base + priority_boost;
|
||
}
|
||
```
|
||
|
||
## 6. 精度与调度的联合优化
|
||
|
||
### 6.1 优化目标
|
||
|
||
```
|
||
Maximize: Throughput (tokens/sec)
|
||
Subject to:
|
||
- WCL ≤ 200ms (硬实时约束)
|
||
- Accuracy ≥ 95% (质量约束)
|
||
- Power ≤ 5W (功耗约束)
|
||
|
||
Decision Variables:
|
||
- Precision per layer (FP16/Int8/Int4)
|
||
- Scheduling strategy (FP/EDF/Hybrid)
|
||
- Batch size (concurrent requests)
|
||
- KV Cache size
|
||
|
||
Trade-off:
|
||
Lower precision → Lower latency → Higher throughput
|
||
BUT → Lower accuracy → May violate quality constraint
|
||
|
||
Higher precision → Higher accuracy → Lower throughput
|
||
BUT → May violate latency constraint
|
||
|
||
Solution: Multi-objective optimization
|
||
- Find Pareto frontier of (latency, accuracy, throughput)
|
||
- Select operating point based on system mode
|
||
```
|
||
|
||
### 6.2 运行时模式切换
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ System Modes │
|
||
├──────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ MODE PERFORMANCE (性能模式): │
|
||
│ - Precision: Int4 (lowest precision) │
|
||
│ - Scheduling: Max throughput, aggressive batching │
|
||
│ - KV Cache: Large pool, aggressive eviction │
|
||
│ - Power: Max freq, no thermal limit │
|
||
│ - Use: Real-time response, latency-critical │
|
||
│ │
|
||
│ MODE BALANCED (均衡模式): │
|
||
│ - Precision: Int8 (balanced) │
|
||
│ - Scheduling: Balanced throughput + latency │
|
||
│ - KV Cache: Moderate pool, moderate eviction │
|
||
│ - Power: Medium freq, thermal-aware │
|
||
│ - Use: General purpose │
|
||
│ │
|
||
│ MODE ACCURACY (精度模式): │
|
||
│ - Precision: FP16 (highest precision) │
|
||
│ - Scheduling: Lower throughput, prioritize quality │
|
||
│ - KV Cache: Large pool, conservative eviction │
|
||
│ - Power: May throttle for stability │
|
||
│ - Use: High-quality output required │
|
||
│ │
|
||
│ MODE POWER (省电模式): │
|
||
│ - Precision: Int4 + DVFS low freq │
|
||
│ - Scheduling: Aggressive idle, power-saving │
|
||
│ - KV Cache: Minimal pool, aggressive eviction │
|
||
│ - Power: Min freq, aggressive sleep │
|
||
│ - Use: Battery-powered, standby │
|
||
│ │
|
||
│ Mode Transitions (RTOS-aware): │
|
||
│ 1. Mode change signaled by control task │
|
||
│ 2. All tasks synchronize at barrier │
|
||
│ 3. Parameters updated (precision, priority, etc.) │
|
||
│ 4. Resume with new parameters │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## 7. 各硬件平台的精度-调度差异
|
||
|
||
### 7.1 MCU级
|
||
|
||
```
|
||
约束:
|
||
- 仅支持INT8 (硬件限制, 无FPunit)
|
||
- 无动态精度切换
|
||
- 精度固定, 调度简单
|
||
|
||
策略:
|
||
- 静态精度(Int8), 固定优先级调度
|
||
- 关注: 精度校准对WCET的影响
|
||
- 量化开销: 固定, 可预先计算
|
||
```
|
||
|
||
### 7.2 SoC级
|
||
|
||
```
|
||
约束:
|
||
- 支持INT8/INT4/FP16 (取决于NPU)
|
||
- 动态精度切换可能有限
|
||
- NPU驱动可能不支持运行时精度调整
|
||
|
||
策略:
|
||
- 静态精度选择, 启动时配置
|
||
- 精度影响调度参数(WCET)
|
||
- 可能支持精度切换但非无缝
|
||
```
|
||
|
||
### 7.3 Edge盒子
|
||
|
||
```
|
||
约束:
|
||
- GPU支持FP16/FP32/INT8动态切换
|
||
- 丰富的精度选项
|
||
- 成熟软件栈支持
|
||
|
||
策略:
|
||
- 运行时动态精度调整
|
||
- 基于负载的精度自适应
|
||
- 精度感知调度优化
|
||
```
|
||
|
||
### 7.4 Server级
|
||
|
||
```
|
||
约束:
|
||
- 全精度支持, 动态调整
|
||
- 丰富的软件生态
|
||
- 调度算法成熟
|
||
|
||
策略:
|
||
- 精细化精度调度
|
||
- 各层不同精度
|
||
- 量化感知 serving (vLLM FP8 support)
|
||
```
|
||
|
||
## 8. 本方向的验证关注点
|
||
|
||
为了让本方向与总课题的双目标评价框架对齐，量化与精度感知调度至少要回答下面三个问题：
|
||
|
||
1. 不同精度策略下，人工智能目标负载的 `TTFT`、`TPOT`、成功率和输出质量如何共同变化；
|
||
2. 精度切换、同步和校准开销是否会把关键保障负载推过实时边界；
|
||
3. 精度模式切换是否可以被准入控制和运行模式管理，并保持可预测的抖动边界。
|
||
|
||
## 9. 关键设计决策
|
||
|
||
| 决策点 | 选项 | 推荐 | 理由 |
|
||
|-------|------|-----|------|
|
||
| 精度选择 | 静态 / 动态 | **Hybrid** | 启动静态+运行时微调 |
|
||
| 精度粒度 | Layer-level / Token-level | **Layer-level** | 实现简单+精度可接受 |
|
||
| 切换开销 | 计入WCET / 不计入 | **计入WCET** | 实时性分析准确 |
|
||
| 负载均衡 | 静态 / 动态 | **动态** | MoE负载不均衡常见 |
|
||
| 模式切换 | 手动 / 自动 | **自动** | 自适应系统负载 |
|
||
| 校准策略 | Offline / Online | **Offline** | 在线校准开销大 |
|
||
|
||
## 10. 开放研究问题
|
||
|
||
1. **Layer-specific Precision**: 每层不同精度对调度有何影响？
|
||
2. **Precision Prediction**: 预测最佳精度, 避免频繁切换？
|
||
3. **MoE Load Balancing in RT**: MoE的负载均衡如何满足实时约束？
|
||
4. **Precision-aware Memory**: 精度变化时的内存自动伸缩？
|
||
5. **Cross-model Precision**: 多模型共存时的精度分配？
|
||
|
||
---
|
||
|
||
*最后更新: 2026-09-17*
|