Solo and MoA Benchmarking on multiple tasks
I created the tasks. used Hermes Agent as the harness (v0.18)
Scored by ChatGPT 5.5 and Claude opus 4.8
Evaluation Rubric
Created by ChatGPT 5.5 and Claude opus 4.8
| Dimension |
Weight |
Evaluation Lens |
Focus |
| Correctness |
25% |
AI-expert |
Factuality, accurate diagnosis, correct code logic |
| Completeness |
20% |
AI-expert |
Edge-case handling, full instruction coverage |
| Depth |
15% |
AI-expert |
Architectural trade-offs, root-cause resolution |
| Actionability |
15% |
User |
Runnable code blocks, directly applicable commands |
| Clarity |
15% |
User |
Structural hierarchy, conciseness, readability |
| Trust |
10% |
User |
Honesty, calibration, safe failure modes |
Final Overall Ranking
| Rank |
Model Configuration |
Avg |
Bug |
Tool |
Arch |
Clinical |
DLQ |
| 1 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
86.7 |
79.9 |
86.2 |
88.1 |
90.3 |
88.9 |
| 2 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
85.9 |
88.1 |
88.1 |
88.1 |
81.1 |
84.2 |
| 3 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
85.2 |
88.1 |
83.1 |
88.1 |
77.6 |
88.9 |
| 4 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
85.1 |
80.5 |
87.5 |
88.1 |
80.5 |
88.9 |
| 5 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
84.8 |
81.1 |
88.1 |
88.1 |
77.6 |
88.9 |
| 6 |
Qwen-3.6-27B-4bit(vLLM) |
84.6 |
84.7 |
84.7 |
84.7 |
79.9 |
88.9 |
| 7 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
84.3 |
88.1 |
88.1 |
86.6 |
81.1 |
77.6 |
| 8 |
DeepSeek-v4-Flash |
82.8 |
79.5 |
88.1 |
88.1 |
73.6 |
84.9 |
| 9 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
82.7 |
80.5 |
82.4 |
84.7 |
81.1 |
84.9 |
| 10 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
82.4 |
73.0 |
88.1 |
84.7 |
81.1 |
84.9 |
| 11 |
DeepSeek-v4-Pro (cloud) |
81.3 |
93.2 |
88.1 |
88.1 |
52.0 |
85.2 |
| 12 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
66.7 |
59.1 |
61.5 |
78.2 |
61.9 |
72.9 |
| 13 |
Gemma-4-12B-4bit(vLLM) |
54.1 |
69.5 |
51.5 |
56.1 |
43.9 |
49.5 |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.8 |
5.8 |
5.8 |
5.8 |
5.8 |
Category Marks By Model
| Rank |
Model Configuration |
Avg |
Correctness |
Completeness |
Depth |
Actionability |
Clarity |
Trust |
| 1 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
86.7 |
88.3 |
95.8 |
90.8 |
85.5 |
77.2 |
74.4 |
| 2 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
85.9 |
86.3 |
97.2 |
90.3 |
82.5 |
78.0 |
73.0 |
| 3 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
85.2 |
86.3 |
93.8 |
89.7 |
82.6 |
78.0 |
73.0 |
| 4 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
85.1 |
86.6 |
94.4 |
87.3 |
82.9 |
78.0 |
73.0 |
| 5 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
84.8 |
85.8 |
93.0 |
89.3 |
82.1 |
78.0 |
73.0 |
| 6 |
Qwen-3.6-27B-4bit(vLLM) |
84.6 |
88.3 |
97.2 |
88.5 |
84.5 |
68.6 |
68.2 |
| 7 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
84.3 |
81.1 |
97.2 |
91.2 |
79.5 |
78.0 |
73.0 |
| 8 |
DeepSeek-v4-Flash |
82.8 |
83.5 |
91.4 |
89.4 |
79.3 |
74.4 |
72.4 |
| 9 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
82.7 |
83.4 |
90.8 |
85.7 |
79.0 |
78.0 |
73.0 |
| 10 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
82.4 |
82.8 |
91.4 |
86.8 |
79.3 |
75.4 |
71.4 |
| 11 |
DeepSeek-v4-Pro (cloud) |
81.3 |
84.8 |
85.0 |
85.0 |
77.0 |
78.0 |
71.6 |
| 12 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
66.7 |
68.2 |
78.0 |
64.1 |
69.0 |
54.0 |
59.8 |
| 13 |
Gemma-4-12B-4bit(vLLM) |
54.1 |
57.0 |
60.6 |
52.3 |
57.8 |
48.0 |
40.0 |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.0 |
5.0 |
5.0 |
5.0 |
10.0 |
5.0 |
Detailed Task Rankings
Task 1: Bug Investigation
Evaluation of system state discrepancies, race conditions, and root-cause logic mapping.
| Rank |
Model Configuration |
Score |
Correctness |
Completeness |
Depth |
Actionability |
Clarity |
Trust |
Notes |
| 1 |
DeepSeek-v4-Pro (cloud) |
93.2 |
95.0 |
100.0 |
95.0 |
85.0 |
90.0 |
90.0 |
All root causes correct, precise KeyError explanation, SETNX+expire edge case handled |
| 2 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 3 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 4 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 5 |
Qwen-3.6-27B-4bit(vLLM) |
84.7 |
90.0 |
100.0 |
88.0 |
85.0 |
65.0 |
65.0 |
|
| 6 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
81.1 |
81.6 |
86.0 |
86.2 |
77.3 |
78.0 |
73.0 |
|
| 7 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
80.5 |
81.6 |
86.0 |
81.7 |
77.3 |
78.0 |
73.0 |
|
| 8 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
80.5 |
81.6 |
86.0 |
81.7 |
77.3 |
78.0 |
73.0 |
|
| 9 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
79.9 |
81.6 |
86.0 |
86.2 |
77.3 |
70.0 |
73.0 |
|
| 10 |
DeepSeek-v4-Flash |
79.5 |
85.0 |
86.0 |
90.0 |
77.3 |
60.0 |
70.0 |
Correct/deep, but low clarity due to verbose thinking-aloud |
| 11 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
73.0 |
72.6 |
71.0 |
75.0 |
69.0 |
78.0 |
73.0 |
|
| 12 |
Gemma-4-12B-4bit(vLLM) |
69.5 |
81.6 |
86.0 |
63.7 |
77.3 |
45.0 |
40.0 |
Terse (187 words) but technically focused |
| 13 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
59.1 |
65.0 |
71.0 |
50.0 |
69.0 |
45.0 |
40.0 |
Technically correct base concepts, heavily truncated |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.0 |
5.0 |
5.0 |
5.0 |
10.0 |
5.0 |
TEMPLATE FAILURE: boilerplate output |
Task 2: Tool Orchestration
Evaluation of interactive debugging workflow steps: logs \\rightarrow sockets \\rightarrow config checks.
| Rank |
Model Configuration |
Score |
Correctness |
Completeness |
Depth |
Actionability |
Clarity |
Trust |
Notes |
| 1 |
DeepSeek-v4-Flash |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 2 |
DeepSeek-v4-Pro (cloud) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 3 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 4 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 5 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 6 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 7 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
87.5 |
90.0 |
100.0 |
88.0 |
85.0 |
78.0 |
73.0 |
|
| 8 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
86.2 |
90.0 |
100.0 |
88.0 |
85.0 |
70.0 |
73.0 |
|
| 9 |
Qwen-3.6-27B-4bit(vLLM) |
84.7 |
90.0 |
100.0 |
88.0 |
85.0 |
65.0 |
65.0 |
|
| 10 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
83.1 |
84.0 |
90.0 |
88.0 |
79.5 |
78.0 |
73.0 |
|
| 11 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
82.4 |
84.0 |
90.0 |
83.5 |
79.5 |
78.0 |
73.0 |
|
| 12 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
61.5 |
72.0 |
70.0 |
56.5 |
68.5 |
45.0 |
40.0 |
|
| 13 |
Gemma-4-12B-4bit(vLLM) |
51.5 |
60.0 |
50.0 |
47.5 |
57.5 |
45.0 |
40.0 |
Minimal command output |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.0 |
5.0 |
5.0 |
5.0 |
10.0 |
5.0 |
TEMPLATE FAILURE |
Task 3: Code Architecture
Evaluation of class structures, state machines, encryption handling, and DLQ config mapping.
| Rank |
Model Configuration |
Score |
Correctness |
Completeness |
Depth |
Actionability |
Clarity |
Trust |
Notes |
| 1 |
DeepSeek-v4-Flash |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 2 |
DeepSeek-v4-Pro (cloud) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 3 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 4 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 5 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 6 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 7 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.1 |
90.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 8 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
86.6 |
84.0 |
100.0 |
92.5 |
85.0 |
78.0 |
73.0 |
|
| 9 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
84.7 |
90.0 |
92.0 |
84.4 |
81.0 |
78.0 |
73.0 |
|
| 10 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
84.7 |
90.0 |
100.0 |
88.0 |
85.0 |
65.0 |
65.0 |
|
| 11 |
Qwen-3.6-27B-4bit(vLLM) |
84.7 |
90.0 |
100.0 |
88.0 |
85.0 |
65.0 |
65.0 |
|
| 12 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
78.2 |
80.0 |
92.0 |
75.4 |
81.0 |
60.0 |
73.0 |
|
| 13 |
Gemma-4-12B-4bit(vLLM) |
56.1 |
53.5 |
67.0 |
55.1 |
68.5 |
45.0 |
40.0 |
Severely truncated stub architecture |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.0 |
5.0 |
5.0 |
5.0 |
10.0 |
5.0 |
TEMPLATE FAILURE |
Task 4: Clinical Decision
Evaluation of differential diagnoses, IPAF exclusion, anti-fibrotic mapping, and transplant parameters.
| Rank |
Model Configuration |
Score |
Correctness |
Completeness |
Depth |
Actionability |
Clarity |
Trust |
Notes |
| 1 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
90.3 |
90.0 |
93.0 |
95.0 |
90.0 |
90.0 |
80.0 |
Best diagnostic exclusions and management |
| 2 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
81.1 |
81.6 |
86.0 |
86.2 |
77.3 |
78.0 |
73.0 |
|
| 3 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
81.1 |
81.6 |
86.0 |
86.2 |
77.3 |
78.0 |
73.0 |
|
| 4 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
81.1 |
81.6 |
86.0 |
86.2 |
77.3 |
78.0 |
73.0 |
|
| 5 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
81.1 |
81.6 |
86.0 |
86.2 |
77.3 |
78.0 |
73.0 |
|
| 6 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
80.5 |
81.6 |
86.0 |
81.7 |
77.3 |
78.0 |
73.0 |
|
| 7 |
Qwen-3.6-27B-4bit(vLLM) |
79.9 |
81.6 |
86.0 |
86.2 |
77.3 |
70.0 |
73.0 |
|
| 8 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
77.6 |
77.4 |
79.0 |
83.0 |
73.4 |
78.0 |
73.0 |
|
| 9 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
77.6 |
77.4 |
79.0 |
83.0 |
73.4 |
78.0 |
73.0 |
|
| 10 |
DeepSeek-v4-Flash |
73.6 |
72.6 |
71.0 |
79.5 |
69.0 |
78.0 |
73.0 |
|
| 11 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
61.9 |
64.2 |
57.0 |
59.6 |
61.3 |
60.0 |
73.0 |
|
| 12 |
DeepSeek-v4-Pro (cloud) |
52.0 |
62.0 |
42.0 |
63.0 |
40.0 |
58.0 |
40.0 |
Audited penalty: Falsely claimed prompt was truncated |
| 13 |
Gemma-4-12B-4bit(vLLM) |
43.9 |
50.0 |
29.0 |
38.0 |
45.9 |
60.0 |
40.0 |
Only completed 2/5 sub-questions |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.0 |
5.0 |
5.0 |
5.0 |
10.0 |
5.0 |
TEMPLATE FAILURE |
Task 5: DLQ
Evaluation of run-time dead letter queue execution, error auditing, and retry integrity.
| Rank |
Model Configuration |
Score |
Correctness |
Completeness |
Depth |
Actionability |
Clarity |
Trust |
Notes |
| 1 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.9 |
90.0 |
100.0 |
92.5 |
90.0 |
78.0 |
73.0 |
|
| 2 |
Qwen-3.6-27B-4bit(vLLM) |
88.9 |
90.0 |
100.0 |
92.5 |
90.0 |
78.0 |
73.0 |
|
| 3 |
Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) |
88.9 |
90.0 |
100.0 |
92.5 |
90.0 |
78.0 |
73.0 |
|
| 4 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.9 |
90.0 |
100.0 |
92.5 |
90.0 |
78.0 |
73.0 |
|
| 5 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) |
88.9 |
90.0 |
100.0 |
92.5 |
90.0 |
78.0 |
73.0 |
|
| 6 |
DeepSeek-v4-Pro (cloud) |
85.2 |
87.0 |
83.0 |
82.0 |
90.0 |
86.0 |
82.0 |
Audited penalty: Loop fails to store fresh error metrics on retries |
| 7 |
DeepSeek-v4-Flash |
84.9 |
80.0 |
100.0 |
92.5 |
80.0 |
78.0 |
73.0 |
|
| 8 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) |
84.9 |
80.0 |
100.0 |
92.5 |
80.0 |
78.0 |
73.0 |
|
| 9 |
Ornith1.0-35B-Q4_K_M(llama.cpp) |
84.9 |
80.0 |
100.0 |
92.5 |
80.0 |
78.0 |
73.0 |
|
| 10 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) |
84.2 |
80.0 |
100.0 |
88.0 |
80.0 |
78.0 |
73.0 |
|
| 11 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) |
77.6 |
60.0 |
100.0 |
92.5 |
65.0 |
78.0 |
73.0 |
Code executes but demo implementation crashes |
| 12 |
MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) |
72.9 |
60.0 |
100.0 |
79.0 |
65.0 |
60.0 |
73.0 |
Crashes with ZeroDivisionError |
| 13 |
Gemma-4-12B-4bit(vLLM) |
49.5 |
40.0 |
71.0 |
57.0 |
40.0 |
45.0 |
40.0 |
Partial implementation |
| 14 |
Nemotron 2 Cascade Q4_K_M(llama.cpp) |
5.8 |
5.0 |
5.0 |
5.0 |
5.0 |
10.0 |
5.0 |
TEMPLATE FAILURE |
1 Like