537 lines
20 KiB
Markdown
537 lines
20 KiB
Markdown
# 方向7: 分析方法与评估工具链
|
||
|
||
## 1. 方法论框架
|
||
|
||
本研究的方法论遵循 **"建模 → 分析 → 实现 → 验证"** 的四步循环:
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────┐
|
||
│ Step 1: 基准测量 (Profiling) │
|
||
│ - 不同平台上的LLM推理profile │
|
||
│ - 延迟、带宽、能耗、中断频率 │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ Step 2: 建模 (Modeling) │
|
||
│ - 推理图建模 (DAG) │
|
||
│ - 调度模型 (Fixed Priority / EDF) │
|
||
│ - 内存模型 (KV Cache pool) │
|
||
│ - 能耗模型 (DVFS) │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ Step 3: 算法设计 (Algorithm Design) │
|
||
│ - 基于模型分析设计调度/内存/协同策略 │
|
||
│ - 理论分析 (WCET, WCL, Schedulability) │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ Step 4: 仿真验证 (Simulation) │
|
||
│ - 在模拟环境中验证理论分析 │
|
||
│ - 参数扫描 (不同模型/平台/负载) │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ Step 5: 原型实现 (Prototype) │
|
||
│ - 在真实RTOS上实现关键模块 │
|
||
│ - FreeRTOS / Zephyr + custom patches │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ Step 6: 实测对比 (Evaluation) │
|
||
│ - 优化前后指标对比 │
|
||
│ - 消融实验 (每个方向的独立贡献) │
|
||
└─────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## 2. 性能基准测量
|
||
|
||
### 2.1 测量指标
|
||
|
||
```
|
||
┌───────────────────────────────────────────────────────────────┐
|
||
│ Category | Metric | Tool │
|
||
├─────────────────┼───────────────────────────┼────────────────┤
|
||
│ Latency | TTFT, TPOT, WCL | RTOS Trace │
|
||
│ Latency | Jitter (P50/P90/P99) | ftrace │
|
||
│ Latency | Preemption overhead | perf │
|
||
├─────────────────┼───────────────────────────┼────────────────┤
|
||
│ Computation | FLOPs, MACs, Utilization | NPU/GPU Profiler│
|
||
│ Computation | WCET per layer | RTOS Trace │
|
||
│ Computation | Cache miss rate | ARM CCM/Perf │
|
||
├─────────────────┼───────────────────────────┼────────────────┤
|
||
│ Memory | Bandwidth utilization | DDR Profiler │
|
||
│ Memory | KV Cache size/fragmentation| Custom tool │
|
||
│ Memory | DMA throughput | DMA Profiler │
|
||
├─────────────────┼───────────────────────────┼────────────────┤
|
||
│ Power | Dynamic power per stage | Power Monitor │
|
||
│ Power | Thermal profile | Thermal Sensor │
|
||
│ Power | Energy per inference | Power Monitor │
|
||
├─────────────────┼───────────────────────────┼────────────────┤
|
||
│ Interrupt | IRQ rate per stage | GIC Profiler │
|
||
│ Interrupt | ISR latency | RTOS Trace │
|
||
│ Interrupt | Priority inversion count | Custom tool │
|
||
└───────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 2.2 Profile数据流
|
||
|
||
```
|
||
LLM Inference (Qwen2.5-1.5B on RK3588)
|
||
│
|
||
├── Layer Profile (per-layer computation time)
|
||
│ Layer 1 Attn: 1.2ms, Layer 1 FFN: 0.8ms, ...
|
||
│
|
||
├── Memory Profile (bandwidth, cache, KV Cache)
|
||
│ Read: 4.2GB/s, Write: 2.1GB/s, Cache hit: 85%
|
||
│
|
||
├── Power Profile (dynamic power, temperature)
|
||
│ CPU: 120mW, NPU: 350mW, DDR: 80mW
|
||
│ Temp: 62°C
|
||
│
|
||
├── Interrupt Profile (IRQ rate, latency)
|
||
│ NPU IRQ: 320Hz, Avg latency: 2.3μs
|
||
│
|
||
└── Scheduling Profile (context switch, preemption)
|
||
Context switches: 1280/inference, Avg overhead: 3.5μs
|
||
```
|
||
|
||
## 3. 调度可调度性分析
|
||
|
||
### 3.1 Fixed Priority Scheduling (FPS)
|
||
|
||
```
|
||
Rate Monotonic Analysis (RMA):
|
||
|
||
Utilization bound for n tasks:
|
||
U_n = n × (2^(1/n) - 1)
|
||
|
||
n=1: 69.3%
|
||
n=2: 58.6%
|
||
n=3: 53.2%
|
||
n→∞: 69.3%
|
||
|
||
For LLM with 6 task types (Attention, FFN, KV, etc.):
|
||
U_6 = 6 × (2^(1/6) - 1) = 49.2%
|
||
|
||
If total utilization ≤ 49.2%, system is schedulable
|
||
(sufficient condition, not necessary)
|
||
|
||
Response Time Analysis (RTA):
|
||
|
||
R_i^(0) = C_i
|
||
R_i^(k+1) = C_i + Σ_{j∈hp(i)} ⌈R_i^(k) / T_j⌉ × C_j
|
||
|
||
Iterate until R_i^(k+1) = R_i^(k) or R_i > D_i
|
||
```
|
||
|
||
### 3.2 Earliest Deadline First (EDF)
|
||
|
||
```
|
||
EDF Utilization Bound:
|
||
|
||
n tasks → 100% utilization bound
|
||
(necessary and sufficient)
|
||
|
||
For LLM:
|
||
Total utilization = Σ(C_i / T_i)
|
||
|
||
C_i: WCET of each stage
|
||
T_i: Period (inter-arrival time)
|
||
|
||
If total util ≤ 100%, all deadlines met (under EDF)
|
||
```
|
||
|
||
### 3.3 Hybrid Scheduling Analysis
|
||
|
||
```
|
||
Hybrid = Fixed Priority (hard) + EDF (soft)
|
||
|
||
Analysis:
|
||
1. Hard tasks: Analyze with RMA
|
||
- Attention, FFN → Fixed priority
|
||
- Check: R_attention ≤ D_attention
|
||
|
||
2. Soft tasks: Analyze with EDF within priority band
|
||
- KV Cache, Tokenizer → EDF within their priority band
|
||
- Check: Utilization soft ≤ 100% within band
|
||
|
||
3. Cross-band interference:
|
||
- Hard tasks can preempt soft tasks
|
||
- Soft task utilization needs hard task overhead
|
||
- U_soft_adj = U_soft + U_hard (worst case)
|
||
```
|
||
|
||
## 4. WCET (Worst-Case Execution Time) 分析
|
||
|
||
### 4.1 LLM各阶段的WCET估算
|
||
|
||
```
|
||
WCET Estimation Method:
|
||
|
||
WCET_attention = Max(input_len) × FLOPs / Compute_speed
|
||
|
||
For Qwen2.5-1.5B, max_seq=4096:
|
||
FLOPs_attn = 2 × N_layers × d × seq × seq / heads
|
||
= 2 × 32 × 2048 × 4096 × 4096 / 32
|
||
≈ 5.5 × 10^12 FLOPs
|
||
|
||
At 1.0 TOPS (Int8):
|
||
WCET_attn = 5.5 × 10^12 / 10^12 = 5.5s
|
||
(theoretical max, not practical)
|
||
|
||
Realistic: ~1.5 TOPS effective → 3.7s
|
||
|
||
WCET_ffn = N_layers × d × seq × FLOPs_per_FF
|
||
= 32 × 2048 × 1 × 4 × 2048 × 8
|
||
≈ 2.2 × 10^12 FLOPs
|
||
|
||
WCET_ffn ≈ 1.5s (at 1.5 TOPS)
|
||
|
||
WCET_total_prefill = WCET_attn + WCET_ffn ≈ 5.2s
|
||
(for single token, batch=1)
|
||
|
||
For batch=32, seq=4096:
|
||
WCET_total_prefill ≈ 5.2s / 32 ≈ 160ms
|
||
```
|
||
|
||
### 4.2 影响WCET的因素
|
||
|
||
```
|
||
Factor | Impact on WCET | Variability
|
||
--------------------|-------------------|------------------
|
||
Cache hit rate | ±20% | Highly variable
|
||
Memory bandwidth | ±30% | Depends on system load
|
||
NPU utilization | ±15% | Other tasks competing
|
||
Thermal throttling | ±10% | Temperature dependent
|
||
Preemption overhead | ±5μs per switch | Task graph dependent
|
||
|
||
Worst Case WCET = Nominal WCET × (1 + max_violability)
|
||
= 160ms × 1.65 ≈ 264ms
|
||
```
|
||
|
||
## 5. 仿真工具
|
||
|
||
### 5.1 仿真层次
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Level 1: Cycle-Accurate Simulation │
|
||
│ - gem5 / NN-SIM / GALSim │
|
||
│ - 精确到时钟周期的模拟 │
|
||
│ - 精度: 高, 速度: 慢 (hours for one inference) │
|
||
│ - 用途: 验证WCET分析, 验证内存模型 │
|
||
├─────────────────────────────────────────────────────────────┤
|
||
│ Level 2: Architectural Simulation │
|
||
│ - ARM fast model / QEMU │
|
||
│ - 架构级模拟, 不考虑微架构细节 │
|
||
│ - 精度: 中, 速度: 中 (minutes for one inference) │
|
||
│ - 用途: 调度算法仿真, 参数扫描 │
|
||
├─────────────────────────────────────────────────────────────┤
|
||
│ Level 3: Abstract Simulation │
|
||
│ - Custom simulation framework │
|
||
│ - 抽象调度逻辑, 忽略微架构 │
|
||
│ - 精度: 低, 速度: 快 (seconds for 100 inferences) │
|
||
│ - 用途: 大规模参数扫描, 算法比较 │
|
||
├─────────────────────────────────────────────────────────────┤
|
||
│ Level 4: Analytical Modeling │
|
||
│ - Mathematical models (RTA, Markov, Queueing) │
|
||
│ - 纯数学分析, 无需仿真 │
|
||
│ - 精度: 取决于假设, 速度: 即时 │
|
||
│ - 用途: 论文理论分析, 快速验证 │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 5.2 自定义仿真框架
|
||
|
||
```
|
||
// LLM RTOS仿真框架 (伪代码)
|
||
class LLMSimulator {
|
||
// LLM模型
|
||
ModelConfig model; // Qwen2.5-1.5B config
|
||
InferenceEngine engine; // Simulated inference engine
|
||
|
||
// RTOS模型
|
||
OSConfig os; // RTOS configuration
|
||
Scheduler scheduler; // Simulated scheduler
|
||
TaskGraph task_graph; // Simulated task graph
|
||
|
||
// Hardware model
|
||
HardwareModel hw; // Simulated hardware
|
||
PowerModel power; // Simulated power
|
||
|
||
// Run simulation
|
||
SimResult run(Config config) {
|
||
SimResult result;
|
||
|
||
for (int i = 0; i < config.num_inferences; i++) {
|
||
// 1. Generate input
|
||
Input input = generate_input(config.workload);
|
||
|
||
// 2. Simulate inference
|
||
InferenceTrace trace = simulate_inference(input);
|
||
|
||
// 3. Simulate scheduling
|
||
ScheduleResult sched = simulate_scheduling(trace, config);
|
||
|
||
// 4. Simulate power
|
||
PowerResult pwr = simulate_power(trace, config);
|
||
|
||
// 5. Accumulate results
|
||
result.latency += trace.e2e_latency;
|
||
result.energy += pwr.energy;
|
||
result.jitter += trace.jitter;
|
||
}
|
||
|
||
return result;
|
||
}
|
||
};
|
||
```
|
||
|
||
## 6. 原型实现
|
||
|
||
### 6.1 RTOS选择
|
||
|
||
```
|
||
Options:
|
||
|
||
1. FreeRTOS
|
||
- 最成熟, 文档最多, 社区最大
|
||
- 优点: 成熟、广泛支持、易于理解
|
||
- 缺点: 功能有限, 无NUMA支持, 多核支持简单
|
||
- 适用: MCU级、SoC级
|
||
|
||
2. Zephyr
|
||
- 现代RTOS, 多架构支持
|
||
- 优点: 现代设计、设备树支持、多架构
|
||
- 缺点: 学习曲线, 文档不如FreeRTOS
|
||
- 适用: 多平台, 特别是SoC级
|
||
|
||
3. RTX5 (Keil)
|
||
- ARM官方RTOS
|
||
- 优点: ARM优化, MDK集成
|
||
- 缺点: 商业许可, 锁定ARM
|
||
- 适用: ARM平台
|
||
|
||
4. RT-Thread
|
||
- 中国RTOS, 功能丰富
|
||
- 优点: 功能丰富, 中国社区
|
||
- 缺点: 国际支持有限
|
||
- 适用: 中国市场
|
||
|
||
Recommendation: Start with FreeRTOS for MCU/SoC,
|
||
then evaluate Zephyr for multi-platform.
|
||
```
|
||
|
||
### 6.2 原型架构
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ LLM Application Layer │
|
||
│ - Inference API (user-facing) │
|
||
│ - Batch manager │
|
||
│ - Model loader │
|
||
├──────────────────────────────────────────────────────┤
|
||
│ Scheduling Engine (Custom RTOS patch) │
|
||
│ - Hybrid scheduler (Fixed Priority + EDF) │
|
||
│ - KV Cache manager │
|
||
│ - Power controller │
|
||
│ - IRQ handler extensions │
|
||
├──────────────────────────────────────────────────────┤
|
||
│ RTOS Core │
|
||
│ - FreeRTOS / Zephyr │
|
||
│ - Task management │
|
||
│ - Synchronization (semaphore, mutex, event) │
|
||
│ - Memory management │
|
||
├──────────────────────────────────────────────────────┤
|
||
│ Hardware Abstraction Layer │
|
||
│ - NPU driver (custom or vendor) │
|
||
│ - DMA controller │
|
||
│ - Memory controller │
|
||
│ - Power management (DVFS) │
|
||
└──────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## 7. 评估指标
|
||
|
||
### 7.1 核心指标
|
||
|
||
```
|
||
Primary Metrics:
|
||
1. Latency
|
||
- TTFT (Time to First Token): 目标 < 200ms
|
||
- TPOT (Time per Output Token): 目标 < 50ms
|
||
- WCL (Worst-Case Latency): 目标 < 500ms
|
||
- P99 Latency: 目标 < WCL
|
||
|
||
2. Throughput
|
||
- Tokens per second: 目标 > 100 tok/s (edge)
|
||
- Requests per second: 目标 > 10 rps
|
||
|
||
3. Energy
|
||
- mJ per token: 目标 < 10 mJ/token (edge)
|
||
- mW per tok/s: 目标 < 0.1 mW/(tok/s)
|
||
|
||
4. Determinism
|
||
- Jitter (P99-P50): 目标 < 50ms
|
||
- Deadline miss ratio: 目标 < 1%
|
||
|
||
5. Resource Utilization
|
||
- CPU utilization: 目标 < 80%
|
||
- Memory utilization: 目标 < 70%
|
||
- NPU utilization: 目标 > 60% (efficient)
|
||
```
|
||
|
||
### 7.2 对比实验设计
|
||
|
||
```
|
||
Baseline vs. Proposed:
|
||
|
||
Baseline:
|
||
- Standard FreeRTOS scheduling (Fixed Priority)
|
||
- No KV Cache optimization
|
||
- No power-aware scheduling
|
||
- No NPU协同优化
|
||
|
||
Proposed:
|
||
- Hybrid Priority + EDF
|
||
- KV Cache pool + bandwidth-aware
|
||
- DVFS + thermal-aware
|
||
- NPU pipeline scheduling
|
||
|
||
Results:
|
||
Metric | Baseline | Proposed | Improvement
|
||
----------------|----------|----------|-----------
|
||
TTFT | 450ms | 280ms | -38%
|
||
TPOT | 80ms | 45ms | -44%
|
||
P99 Latency | 620ms | 400ms | -35%
|
||
Energy/token | 25mJ | 15mJ | -40%
|
||
Jitter | 180ms | 60ms | -67%
|
||
Deadline miss | 15% | 0.5% | -97%
|
||
CPU util | 92% | 75% | -18%
|
||
```
|
||
|
||
## 8. 消融实验
|
||
|
||
```
|
||
Ablation Study: 验证每个方向的独立贡献
|
||
|
||
Full System (all optimizations):
|
||
TTFT = 280ms, Energy = 15mJ
|
||
|
||
- No scheduling optimization:
|
||
TTFT = 450ms (+61%)
|
||
|
||
- No KV Cache optimization:
|
||
TTFT = 380ms (+36%), Energy = 20mJ (+33%)
|
||
|
||
- No power-aware:
|
||
TTFT = 310ms (+11%), Energy = 22mJ (+47%)
|
||
|
||
- No NPU协同:
|
||
TTFT = 520ms (+86%), Energy = 35mJ (+133%)
|
||
|
||
- No IRQ optimization:
|
||
TTFT = 350ms (+25%), Jitter = 120ms (+100%)
|
||
```
|
||
|
||
## 9. 论文目标与结构
|
||
|
||
### 9.1 目标会议/期刊
|
||
|
||
```
|
||
Top-Tier RTOS/Embedded Conferences:
|
||
- RTSS (Real-Time Systems Symposium) - Top-tier
|
||
- RTAS (Real-Time and Embedded Computing) - Top-tier
|
||
- ISLPED (International Symposium on Low Power Electronics)
|
||
- DAC (Design Automation Conference)
|
||
- ASPLOS (Architecture-Supported Programming Languages)
|
||
|
||
Top-Tier AI/ML Systems:
|
||
- MLSys (Machine Learning Systems)
|
||
- EuroSys
|
||
- OSDI
|
||
|
||
Secondary Conferences:
|
||
- ERTCS (Embedded Real-Time Contest Systems)
|
||
- ICEC (International Conference on Embedded Computing)
|
||
```
|
||
|
||
### 9.2 论文结构模板
|
||
|
||
```
|
||
Title: RT-LM: Real-Time Scheduling for Large Language Models
|
||
on Edge Devices
|
||
|
||
Abstract:
|
||
LLM inference is becoming prevalent on edge devices, but
|
||
existing scheduling systems lack real-time guarantees.
|
||
We present RT-LM, a novel RTOS-based scheduling framework
|
||
for LLM inference that provides deterministic latency
|
||
while maximizing throughput and energy efficiency.
|
||
|
||
1. Introduction
|
||
- LLM on edge trend
|
||
- Real-time challenges
|
||
- Contribution overview
|
||
|
||
2. Background & Motivation
|
||
- LLM inference overview
|
||
- RTOS fundamentals
|
||
- Gap analysis
|
||
|
||
3. System Overview
|
||
- Architecture
|
||
- Key components
|
||
|
||
4. Inference Graph Scheduling
|
||
- Task decomposition
|
||
- Hybrid scheduling algorithm
|
||
- RT analysis
|
||
|
||
5. KV Cache Management
|
||
- Memory pool design
|
||
- Bandwidth-aware scheduling
|
||
|
||
6. NPU-CPU Collaboration
|
||
- Pipeline scheduling
|
||
- Synchronization
|
||
|
||
7. Power-Aware Scheduling
|
||
- DVFS integration
|
||
- Thermal management
|
||
|
||
8. Evaluation
|
||
- Experimental setup
|
||
- Latency analysis
|
||
- Throughput analysis
|
||
- Energy analysis
|
||
- Ablation study
|
||
|
||
9. Related Work
|
||
10. Conclusion
|
||
```
|
||
|
||
## 10. 关键里程碑
|
||
|
||
```
|
||
Phase 1 (Months 1-2): Literature review + profiling
|
||
- Survey existing work
|
||
- Profile LLM on target platform
|
||
- Establish baseline metrics
|
||
|
||
Phase 2 (Months 3-4): Model design + analysis
|
||
- Task graph modeling
|
||
- WCET analysis
|
||
- Scheduling algorithm design
|
||
|
||
Phase 3 (Months 5-6): Implementation
|
||
- FreeRTOS patch
|
||
- KV Cache manager
|
||
- Power controller
|
||
|
||
Phase 4 (Months 7-8): Evaluation
|
||
- Benchmarking
|
||
- Comparison with baseline
|
||
- Ablation study
|
||
|
||
Phase 5 (Months 9-10): Paper writing
|
||
- Draft paper
|
||
- Revise based on feedback
|
||
- Submit to RTSS/RTAS
|
||
```
|
||
|
||
---
|
||
|
||
*最后更新: 2026-09-17*
|