For now, I ran a small comparison on Colab:
I tried ZY-S2 on four short Python function-generation tasks, which seemed close to your stated focus. For practical reference I also tested the official Qwen2.5-Coder-1.5B-Instruct and the larger Qwen2.5-Coder-3B-Instruct, all as Q5_K_M GGUFs.
My main impression: ZY-S2 passed every selected check on two of the four functions: run-length encoding and ragged-column sums. The other two exposed specific, testable gaps involving duplicates, input mutation, per-kind cooldown state and repeated output events. A small regression suite around those contracts could be useful without changing the model’s Python-first direction. This is an exploratory four-function test, not a reproduction of SLMFrontier or an overall model ranking.
Direct answers to your four questions
- How does ZY-S2 do on coding tasks? It produced a complete, syntactically valid function for each of my four short Python prompts. Two passed all selected behavior checks. The
pack_integer_spans and collapse_alerts results missed specific requirements shown below. Both Qwen comparators also had identifiable, but not identical, failures.
- What to prioritize next? For useful 1.5B Python generation, I would start with a few explicit edge-case and input-contract checks:
[], duplicates, unchanged inputs, independent per-key state and a later event becoming eligible again. They are inexpensive release checks, not a suggestion to redirect the project toward agentic features.
- Which benchmarks next? Since you already report HumanEval+/MBPP+, I would first look for useful task-level diagnostics in those existing results, alongside a small, separately labeled contract-stress set. If code editing becomes a release goal, a lightweight CodeEditorBench-style subset could test it separately from synthesis.
- What helps real developers? A clear local-run recipe, exact model/quantization identity, small executable contract examples and, when available, real CPU memory and latency measurements on named hardware. My Colab run used CPU inference but did not measure consumer-laptop responsiveness.
What the small experiment measured
These are my four function specifications and nineteen fixed checks per model, not the author’s rubric, HumanEval+ tasks or a broad random sample. One greedy completion was generated for each model/task pair.
| Python function |
ZY-S2 1.5B |
Qwen 1.5B-Instruct |
Qwen 3B-Instruct |
encode_runs (5 checks) |
5/5 |
5/5 |
5/5 |
pack_integer_spans (5) |
1/5 |
4/5 |
1/5 |
ragged_column_sums (5) |
5/5 |
4/5 |
5/5 |
collapse_alerts from specification (4) |
1/4 |
3/4 |
1/4 |
| Individual checks passed |
12/19 |
16/19 |
12/19 |
| Whole functions passing all selected checks |
2/4 |
1/4 |
2/4 |
The aggregation matters: Qwen 1.5B leads on individual checks, while ZY-S2 and Qwen 3B pass more whole functions. Neither view establishes an overall winner from four functions and one completion each. Also, three of the nineteen test inputs exactly repeated examples visible in their prompts. Excluding those exact repeats leaves 10/16, 13/16 and 10/16 passed by ZY-S2, Qwen 1.5B and Qwen 3B, respectively. These sixteen inputs are merely not identical to prompt examples; they are not proven unseen training data.
The 3B model is intentionally a larger practical reference, not a parameter-matched fine-tuning control. Qwen 1.5B-Instruct is a similar-size practical assistant comparator; it is not automatically the exact checkpoint from which ZY-S2 was fine-tuned.
A small default next step
I would keep a compact, visible regression set next to the broader benchmark. For example:
assert encode_runs([3, 3, 1, 3, 3, 3]) == [[3, 2], [1, 1], [3, 3]]
assert ragged_column_sums([[1, 2, 3], [4], [], [5, 6]]) == [10, 8, 3]
assert pack_integer_spans([]) == []
assert pack_integer_spans([4, 4]) == [[4, 4]]
values = [3, 1, 2, 2, 7, 9, 8]
original = values.copy()
assert pack_integer_spans(values) == [[1, 3], [7, 9]]
assert values == original
The first two were successful ZY-S2 examples in this run; the span cases expose specific missed requirements. Some examples here were visible in the corresponding prompts, as noted above. These assertions are illustrative local checks that I wrote, not a request to rerun a large evaluation or treat them as your published benchmark. The stateful alert example follows in the details because its specification matters.
The expandable sections cover exact setup, scoring boundaries, function behavior, the separate repair probe and optional next steps.
Run setup, model provenance and reproduction limits
Why this setup
Your announcement emphasizes compact parameter count, Python code generation/problem solving and local CPU access. I therefore chose four short, standalone Python functions rather than repository-agent tasks or general reasoning prompts. Three were newly written for this run; the fourth (collapse_alerts) reused the behavioral specification and four tests from an earlier, separately labeled repair experiment, but asked for a new function without buggy starter code.
What Colab actually did
The host was Google Colab Linux x86_64 with Python 3.13.16 and a Tesla T4 allocated. Importantly, the inference configuration was CPU-only, n_gpu_layers=0; the T4 was not accelerating model layers in this experiment. I used a pinned, prebuilt JamePeng llama-cpp-python wheel (0.4.3+cu126) and did not compile llama_cpp_python in Colab.
All three models were pinned Q5_K_M GGUF files. The settings were greedy decoding (temperature=0, top_k=1, top_p=1.0), n_ctx=2048 and at most 512 new tokens per completion. The run requested one complete top-level Python function with the exact signature, preferably in a python fence; the prompt allowed normal built-ins but excluded external packages, imports, filesystem and network operations. All models received the same four task specifications, each containing one visible example.
The notebook recorded 12/12 successful model-task worker results, all with finish_reason=stop. Each response contained an extractable, syntactically parseable Python function. Those facts confirm that model generation completed; they do not make the code semantically correct. The Colab notebook preserved the raw replies, prompts, task/test definitions and model identity reports. It did not execute model-generated Python. I inspected and evaluated that code separately afterward.
What was pinned, and what was verified
| Model |
Pinned revision |
Exact GGUF filename |
| ZY-S2 |
49a0f83754f84fd6d1e0a711e318a75fdd1ccd9d |
ZY-S2-Q5_K_M.gguf |
| Qwen 1.5B-Instruct |
f86cb2c1fa58255f8052cc32aeede1b7482d4361 |
qwen2.5-coder-1.5b-instruct-q5_k_m.gguf |
| Qwen 3B-Instruct |
f74adce6aa16316c625447af059dbebe4983757c |
qwen2.5-coder-3b-instruct-q5_k_m.gguf |
The notebook downloaded each pinned file and reported matching SHA-256 values before inference. As one example, its 3B file record had 2,438,740,416 bytes and SHA-256 eb863f2a1a9b67e33bbf2dad98ea09c03b71c8052aeb4835171cf6f7a7a12db4, matching the official pinned 3B file page. This is the Colab runtime’s reported verification; its small result ZIP does not contain multi-gigabyte weights for independent rehashing afterward.
The four task descriptions and tests were fixed before inference. Trusted reference solutions passed their nineteen corresponding tests during setup, which checks the fixture definitions, not the models. The same prompt text and saved test cases were then used for each model. After raw code capture, the twelve functions were inspected and given the same locally controlled execution and scoring policy. This prevents mixing setup failures, syntax errors and actual behavior mismatches.
For a reader trying a similar test, the important reproducibility inputs are the specific model revision, GGUF quantization, actual prompt (including its example), decoding settings and the exact contract checks. “Colab T4” alone would obscure the fact that inference here was on CPU. Also, neither model file size nor a Colab wall-time report is enough to predict laptop throughput, memory peaks or IDE responsiveness; those need target-device measurements.
Scoring provenance, limitations and fair comparison
My tests and the published numbers answer different questions
The four specifications, nineteen fixed checks, and independent scoring policy in my table were written for this experiment. They were not supplied by ZYR AI and do not reproduce the author’s 400-task SLMFrontier evaluation. In the announcement, the author reports 68.0% overall coding for ZY-S2 versus 63.5% for the 1.5B Qwen comparison, with 74.2% versus 63.2% on HumanEval+ and 63.7% versus 63.7% on MBPP+. The post notes overlapping confidence intervals. Those are author-reported results under its own evaluation, whereas my 12/19 and 2/4 figures are independently observed on a different tiny fixture. Neither validates or refutes the other.
The short Qwen label in the forum table does not spell out -Instruct. The available model-card snapshot uses Qwen2.5-Coder-1.5B-Instruct; my comparison explicitly used the official Instruct GGUF. These names are worth keeping exact, especially before calling any model a pre-fine-tuning baseline. A practical similarly sized competitor is a useful user choice, but a controlled before/after test requires the actual starting checkpoint and matched conditions.
What was fixed before generation
The task statements, one visible example per task and all tests were fixed before generation; reference implementations passed them. I scored the saved model functions afterward under one policy, requiring both expected returns and input preservation where specified.
Three of nineteen test inputs exactly repeated prompt-visible examples (runs, spans and ragged sums). Excluding them yields 10/16, 13/16, 10/16 respectively. The other sixteen inputs are not proven unseen or statistically independent; one code bug can fail multiple checks.
The 2/4 whole-function figure means all selected checks passed, not universal correctness. Four chosen functions and one greedy completion cannot support statistical significance or predictions for a different task mix.
Separate source-generation and execution outcomes
I would distinguish four kinds of result in any follow-up table:
| Outcome |
Example in these runs |
Interpretation |
| Generated and syntactically valid |
All 12 new functions |
Python source was extractable, not necessarily correct |
| Correct on chosen cases |
ZY’s ragged_column_sums |
Positive observation under this contract |
| Returned value / side effect mismatch |
Span duplicates; input sorted in place |
Model output violates an explicit behavior |
| Exception on valid input |
Qwen 1.5B on [] in spans or ragged sums |
Missing boundary guard |
Colab generated but did not execute the Python. Reviewed outputs were scored later under a common restricted process. An AST gate is not a general security sandbox; broader automatic evaluation should isolate untrusted code. EvalPlus documents generation/evaluation separation and container execution.
Reusable metadata: GGUF revision, prompt/example, decoding, extraction, test provenance and per-function outcomes. Label why a run failed (syntax, exception, mismatch, mutation or evaluator refusal) rather than conflating causes.
The EvalPlus CLI guide explains these protocol boundaries. Keep my hand-authored cases separate from HumanEval+/MBPP+ totals; inspect existing task-level logs first if available.
Function-by-function behavior and small reproducible checks
encode_runs: a straightforward positive result
This task encodes each consecutive run of equal integers as [value, count]. A value that reappears after another value starts a new run. The function must handle [], preserve order and leave the input list unchanged. All three models passed all five checks, including a singleton, separated repeated values and negative integers.
ZY-S2’s generated code kept a current value/run length, committed a run on each change, and guarded empty input before accessing element zero. It passed encode_runs([-1, -1, 0, -1]) == [[-1, 2], [0, 1], [-1, 1]], correctly distinguishing consecutive runs from global counts, without changing the input. All three models earned the same 5/5 on this task.
ragged_column_sums: another ZY-S2 success, with an instructive rival edge case
The input is a list of integer rows that may have different lengths or be empty. The output contains the sums of the columns that actually exist, with absent cells contributing zero; no list may be modified. For example:
assert ragged_column_sums([[1, 2, 3], [4], [], [5, 6]]) == [10, 8, 3]
assert ragged_column_sums([]) == []
assert ragged_column_sums([[], []]) == []
ZY-S2 returned an empty-list guard followed by a maximum-row-length calculation and a zero-initialized output list. It added each available input entry to its corresponding column. That implementation passed all five checks, including empty input, empty inner rows, negative values and uneven shapes. Qwen 3B also passed all five.
Qwen 1.5B-Instruct used a concise comprehension with max(len(row) for row in rows) but omitted an outer if not rows check. It passed the four checks with a nonempty outer list, including [[], []]; it raised ValueError for rows=[] because max() received an empty iterator. So its 4/5 is explained by one concrete boundary condition, not an inability to compute ragged sums generally. This shows why a single empty-input check can produce useful release feedback without a large benchmark.
The nontrivial ragged example appeared in the prompt and in one check, so I would not present it as unseen. The empty and other fixed checks are separate observations. Here the successful algorithm also respects input ownership: it reads rows and builds a new output instead of altering the nested data.
pack_integer_spans: duplicates and input mutation are different requirements
The specification asked for inclusive [start, end] ranges formed by sorting unique input integers and joining consecutive values. Duplicates are allowed in the input but should be ignored. Importantly, the caller’s list must remain unchanged.
The ZY-S2 answer guarded empty input, then used values.sort() followed by this consecutive-run logic (an excerpt, with initialization and final return omitted):
values.sort()
for value in values[1:]:
if value == end + 1:
end = value
else:
result.append([start, end])
start = value
end = value
Two things happen here. values.sort() mutates the actual input list, contrary to the visible specification. And since the code does not remove duplicates, a repeated value falls into the else branch instead of extending a range. With [4, 4], it emits two copies of [4, 4], rather than the single expected span. With the example [3, 1, 2, 2, 7, 9, 8], the duplicate 2 breaks the intended [1, 3] range. ZY-S2 passed only the empty-input case in this task: 1/5.
Qwen 3B-Instruct also sorted the caller’s input in place and did not deduplicate. Its implementation used a slightly different loop, but it failed the same class of output and side-effect contracts in our five tests, again 1/5. That is not evidence that size has no benefit in general; it simply shows that a bigger reference model did not automatically fix these particular details.
Qwen 1.5B-Instruct took a different approach (also excerpted):
unique_values = sorted(set(values))
spans = []
start = unique_values[0]
sorted(set(values)) deduplicates and does not modify the input. Its solution passed the duplicate, sorted-range, negative-number and non-mutation checks. But unique_values[0] raised IndexError when values=[], leaving it at 4/5. The test is narrowly informative: it separates deduplication/purity from handling the empty boundary.
The earlier short regression block checks both the returned spans and whether the caller’s list stayed unchanged. A correct return value alone cannot detect a mutating implementation. When the specification says inputs must remain unchanged, that is part of correctness, not a style preference.
collapse_alerts: time state belongs to each kind, while output keeps every eligible occurrence
This function takes chronological dictionaries with kind (a string) and minute (an integer), plus a nonnegative cooldown. For each kind independently, it must retain the first event and any later event at least cooldown minutes after the last kept event of that kind. Suppressed events do not reset the clock. The result is the kept kinds in input order, so the same kind may occur multiple times.
One of the fixed cases was:
events = [
{"kind": "A", "minute": 0},
{"kind": "A", "minute": 2},
{"kind": "A", "minute": 4},
{"kind": "A", "minute": 6},
]
assert collapse_alerts(events, cooldown=5) == ["A", "A"]
At minute 0, A is kept. Minutes 2 and 4 are suppressed without moving the timer. Minute 6 is eligible again, so the output needs two A entries.
ZY-S2’s returned code checked an elapsed time per kind, but stored the result in a dictionary with only one slot for each kind:
kept_events = {}
for event in events:
kind = event['kind']
minute = event['minute']
if kind not in kept_events or minute - kept_events[kind]['minute'] >= cooldown:
kept_events[kind] = event
return [event['kind'] for event in kept_events.values()]
That dictionary can track an event for each kind, but overwriting an entry loses earlier kept occurrences. Its final values can contain a kind only once, even if the specification requires it twice. Qwen 3B-Instruct had the same structural limitation: a dictionary indexed by kind and a final list of its keys. Both passed only the empty input case, 1/4, in the saved alert tests.
Qwen 1.5B-Instruct kept an output list and could therefore return repeated A entries. Its mistake was different: it tracked one global last_event, not the last kept event for each kind. The difference is visible when kinds are interleaved:
events = [
{"kind": "A", "minute": 0},
{"kind": "B", "minute": 1},
{"kind": "A", "minute": 4},
{"kind": "A", "minute": 5},
{"kind": "B", "minute": 5},
]
assert collapse_alerts(events, cooldown=5) == ["A", "B", "A"]
A at 0 and B at 1 are both first events. A at 4 is too early, so it must not reset A’s clock. A at 5 is eligible relative to the kept A at 0, while B at 5 is still too early relative to kept B at 1. A single last_event for the entire stream suppresses an eligible A. Qwen 1.5B therefore passed 3/4, failing that interleaved case but handling the simpler repeated-kind and zero-cooldown cases.
These results illustrate two separate invariants. A timestamp map should be keyed by kind, and the output list must preserve repeated eligible occurrences in order. One possible reference algorithm for the specified task is:
def collapse_alerts(events, cooldown):
out = []
last_kept_at = {}
for event in events:
kind, minute = event["kind"], event["minute"]
if kind not in last_kept_at or minute - last_kept_at[kind] >= cooldown:
out.append(kind)
last_kept_at[kind] = minute
return out
This is illustrative reference code for our test specification, not a completion returned by any compared model. It keeps output history in the list and updates per-kind time only when an event is retained. Other implementations could satisfy the same contract; the test should not demand one coding style.
What these cases are good for
The small test suite exposed correctness dimensions that a visually plausible function can miss: empty-list guards, output duplicates, in-place modification, key-specific state and repeated events. The goal is not to burden a compact model with every imaginable corner case. It is to distinguish a few explicit requirements in the prompt from things the generated code merely appears to handle.
The positive observations deserve equal weight. Every model produced a correct run-length function on these cases, and ZY-S2 handled all five selected ragged-column cases. The failed examples do not erase those successes. Conversely, a function that passes one visible example is not guaranteed correct on other permitted inputs. Both sides are useful for an honest, developer-oriented release check.
Earlier code-repair probe: useful, but not the same task
Before this three-model generation comparison, I also tested repairing existing buggy Python with ZY-S2 and Qwen 1.5B-Instruct. That probe supplied starter code. Its three tasks were parsing sensor lines (parse_sensor_lines), fixing per-kind alert cooldowns (collapse_alerts) and recursively merging configuration dictionaries (merge_config), with four checks each.
| Earlier repair results |
ZY-S2 |
Qwen 1.5B-Instruct |
parse_sensor_lines |
1/4 |
1/4 |
collapse_alerts |
2/4 |
2/4 |
merge_config |
0/4 |
1/4 |
| Cases / whole functions |
3/12; 0/3 |
4/12; 0/3 |
The original repair scorer had my own gate error: it rejected a legal del result[key] statement in one Qwen merge_config completion before running any semantic checks. I reviewed that AST-policy issue and reran the original fixed cases with an adjusted, symmetric policy for both models. The table above is the corrected result. The Qwen function then ran but still failed several requirements involving nested data ownership. An evaluator refusal should not be reported as a demonstrated model logic failure; it is important to separate the two.
The alert specification and four checks were shared across the old repair task and the new function-generation task, but the prompts differed:
| Same alert tests |
Repair buggy starter |
Generate new function |
| ZY-S2 |
2/4 |
1/4 |
| Qwen 1.5B-Instruct |
2/4 |
3/4 |
In the repair run, both models had trouble updating cooldown timers only on kept events. In the generation run, the observed defects were different: ZY lost repeated output occurrences; Qwen 1.5B used a global timestamp. This is an interesting task-form comparison, not proof that supplying buggy code caused either score to change. It was not a controlled paired intervention with repeated attempts, and the 3B model was not included in the earlier repair probe.
I would keep the two experiments as separate rows or tabs, never pool 12 repair cases and 19 generation cases into a single accuracy percentage. If code repair becomes a focus, pairing the same specifications under fresh-function and buggy-starter prompts, with matched generation settings, would be a compact next experiment. If not, the current Python generation feedback stands on its own.
Conditional benchmark choices and lightweight roadmap
Start with what the current goal already needs
For a compact Python function generator, I would not begin by adding a long list of agent benchmarks. The lowest-cost useful evidence is a few frozen contract-invariant tests beside the benchmark the project already uses: an empty input, a duplicate, a non-mutation requirement, independent per-key state and a repeated event after the cooldown. Those are exactly the categories that separated outputs in this small experiment. Tracking whether a subsequent version fixes one of them is informative even without a new headline score.
If the existing evaluation already stores per-task results, a short failure-category breakdown may be even cheaper than rerunning all 400 tasks. It can distinguish wrong output, exception, mutation, invalid source, model setup failure and an evaluator restriction. Reports need not reveal training internals. This would complement, not replace, the HumanEval+/MBPP+ results already in the announcement.
Different goals justify different optional tests
| If the next goal is… |
Smallest useful evidence |
Boundary of the conclusion |
| Stronger standalone Python functions |
Fixed regression cases for empty, duplicates, ownership, state |
Does not estimate broad accuracy |
| Clearer published coding scores |
Same harness plus task-level outcomes and prompt metadata |
Does not by itself attribute gains to training |
| Debugging or requirement changes |
A small, separate CodeEditorBench-style subset |
Editing is not synthesis; no need for the full large harness |
| Fine-tuning uplift |
Actual pre-FT checkpoint, matched prompts/runtime/quantization |
A practical Instruct rival and a 3B model are not causal controls |
| Local CPU adoption |
Named-device load, RAM, prompt and generation timings |
A file-size or Colab result is not a laptop-speed benchmark |
| Prompt robustness |
Few frozen functions under labeled prompt variants or repeated samples |
One greedy output cannot estimate variability |
For pure Python synthesis, EvalPlus and its generation/evaluation guide are useful methodological references. Prompt formatting matters: a signature-completion prompt, a chat instruction and a prompt with an example need not elicit equivalent behavior, even on the same test set. When comparing releases, pin the instruction template, decode settings and extraction rules along with the GGUF. Likewise, pass@1 from a single completion and success after selecting among several attempts are different protocols.
If editing joins the roadmap, CodeEditorBench covers debugging, polishing, translation and requirement switching. A small subset is enough for initial feedback; the repair probe does not establish broad editing reliability.
If the question instead becomes what the fine-tuning specifically improved, the starting checkpoint has to be identified precisely. The official Qwen 1.5B-Instruct GGUF in my table is a practical same-size alternative, not necessarily the model’s direct parent. Adding an unconditioned Base model merely to fill a comparison column could confound usable instruction-following assistant performance with training attribution; keep those as separate evaluations. The 3B Instruct comparison is useful as a stronger-capacity reference, not a baseline for a 1.5B fine-tuning effect.
My default is the first row, plus any task-level reporting already available from the second. All other rows are optional branches, not homework for this release.
Practical developer use and safe reproduction
For a compact Python-first model, a useful workflow is simple: run locally, generate a self-contained utility, inspect it and test before use. This is valuable even if repository agents remain a separate goal. Start with the ZY-S2 model card.
The author gives this local example:
ollama run hf.co/zyr-AGENT/ZY-S2:Q5_K_M
That is the author’s suggested command; I used pinned GGUF weights through llama_cpp_python for my own comparison and did not independently benchmark this Ollama path. The advertised roughly 1.1 GB Q5_K_M weight size describes storage, not peak RAM, load time or tokens/second. For CPU-first users, a named-device table of cold load, peak memory, prompt-processing time and generation throughput, with prompt/context lengths, would be more actionable than a hardware-free timing claim. My T4-hosted but CPU-only Colab run cannot substitute for such a device measurement.
The reproducible portion of this feedback is deliberately small: an exact function specification, a pinned model variant, generated source, and assertions about return values and input mutation. Those are enough for a reader to check the duplicate-span or repeated-alert behavior without obtaining my full evaluation environment. Any further automatically executed model code should be reviewed and isolated; a syntactically valid response is not automatically safe or correct. Nothing in these observations suggests malicious behavior from ZY-S2; this is the usual boundary for any coding assistant.
So the practical takeaway is not to change the project’s identity. I would keep the accessible Python model and make progress easier to see with a few compact positive and negative examples, clear scoring provenance and optional CPU measurements on named hardware.
Thanks for sharing a compact model that is approachable enough for small independent experiments. I hope these examples are useful for future releases.