Best model to help to keep system in line

[mrkeepsmiling]

I have been working on an AI assistant for a long time, and have tried many times to find a system for it to run on. I am a self-taught amateur, but have come a long way so far.
My question is how do I keep track of it all, because it has grown to over 1200 lines of script, and is far from finished,
how do I still manage to be the leader and have control of the entire design with all the codes for tools and other things?
I am facing an imminent division of the Python file.
I can feel it starting to slip out of my hands now.
What is the best thing to do now?

[John6666]

Well… From what I can tell, this may be less about the model not being capable enough, and more about your codebase reaching a size where an AI assistant—or the model behind it—can no longer reliably move forward by inferring everything from the code alone. The things that need to be kept in mind while developing a system are much broader than the source code itself, so I think the assistant will behave more consistently if you make that wider context explicit and share it with the AI as well. The same principle applies outside software development:

---

My direct answer would be: do not begin with a complete rewrite or with searching for the single “best” model. First make the project’s intent, rules, interfaces, and validation process recoverable without relying on one chat session. Then split the Python file one responsibility at a time.

A 1,200-line file is not automatically a failed design. Python is explicitly designed to let longer programs be divided into modules. The more useful warning sign is that you are finding it difficult to predict what a change will affect or to keep the whole design in view.

A coding model can inspect what the program currently does. It cannot reliably know, from source code alone:

• which behaviors are intentional and which are temporary;
• what the assistant is ultimately meant to become;
• what it must never do;
• which interfaces and stored data must remain compatible;
• which tools may perform side effects;
• which earlier approaches were rejected;
• what would count as a successful refactor.

Those are project decisions, not facts that can always be reverse-engineered from Python.

The default route I would recommend

1. Save the current working state in Git.
2. Create one short project guide.
3. Record three to five repeatable behavior checks.
4. Ask the model to map the code without editing it.
5. Correct that map yourself.
6. Extract one low-coupling responsibility.
7. Run the same checks, review the diff, and commit.
8. Only after that, compare models or add heavier retrieval tooling.

The first guide can be very small:

# Purpose

What this assistant is intended to do.

# Non-goals

What it is deliberately not intended to do.

# Main components and data flow

Where a request enters, where the model is called, how tools are selected,
where tools execute, and where persistent state is stored.

# Rules that must remain true

Public interfaces, permissions, data compatibility, and other invariants.

# Tool contracts

For each tool: purpose, inputs, output, side effects, permissions, and failures.

# How to run and check the project

Exact setup, run, lint, and test commands.

# Temporary or uncertain parts

Anything provisional, inferred, or still undecided.

You can let an LLM draft this from the existing code, but ask it to separate:

Observed directly from the code
Inferred intent — requires your confirmation
Unknown from the code

Only the parts you confirm should become authoritative guidance.

You also do not need to build RAG immediately. For a project of this size, directly giving the coding assistant:

project guide
+ target code
+ relevant caller/dependency
+ relevant test

may be simpler and more reliable than adding embeddings, chunking, a vector database, index updates, reranking, and stale-document management.

Minimal project knowledge to keep outside the chat

Knowledge	Useful place to keep it
Current implementation	Source code
Expected behavior	Tests and examples
Local function contracts	Docstrings, types, schemas
Project purpose and non-goals	README or project guide
Component boundaries	Architecture note
Accepted design decisions	Short decision notes
Current work and uncertainties	Issue or current-state note
What actually happened during execution	Logs and test output

The purpose is not to create a large bureaucracy. It is to stop making yourself and the model reconstruct the same project knowledge from scratch in every session.

Why the code alone is not enough

Source code tells a model a great deal:

• what functions and classes exist;
• what imports are used;
• what data structures currently exist;
• how control flows through the present implementation;
• which external APIs are called.

But code usually does not uniquely identify the reason behind the implementation.

For example, if UI code directly writes to a database, that might be:

• an intentional architectural choice;
• a temporary shortcut;
• an old design that should be removed;
• a workaround for a framework limitation;
• an unnoticed coupling problem.

All five interpretations can be consistent with the same source code.

This is why asking a model to recover the intended design from code alone is partly a reverse-engineering task. The model may produce a plausible explanation, but “plausible” is not the same as “the design you intended.”

Comments and docstrings are already a useful starting point

You do not necessarily need a separate document for every detail.

A practical division is:

• module docstring: what this file owns and what it must not own;
• class docstring: responsibility, lifecycle, and owned state;
• function docstring: behavior, parameters, return value, side effects, exceptions, and restrictions;
• inline comment: why a surprising piece of code is necessary;
• project guide: rules that apply across several modules.

PEP 257 recommends documenting a function’s behavior, arguments, return values, side effects, exceptions, and restrictions where applicable.

The important part is to document why and what must remain true, rather than paraphrasing every obvious line.

For example:

def execute_tool(tool_name: str, arguments: dict) -> ToolResult:
    """Execute one registered tool after validating its arguments.

    The model may propose a tool call, but this function is the only
    component allowed to execute it.

    Side effects:
        May access external APIs or the filesystem.

    Restrictions:
        Unknown tools are rejected.
        Model-generated arguments must be validated before execution.
    """

That communicates much more architectural intent than:

# Execute the tool.

Comments also have a maintenance cost. PEP 8 explicitly warns that comments which contradict the code are worse than no comments. Documentation changes should therefore be reviewed alongside code changes, not treated as a one-time generation task.

Comments, types, schemas, and tests have different jobs

Mechanism	Main purpose	What it does not guarantee
Comment	Explain a non-obvious reason	That the statement is still current
Docstring	Describe a local contract	That the implementation follows it
Type hint	Describe expected value shapes	Runtime validity by itself
Schema validation	Reject malformed runtime data	Correct business behavior
Test	Check an observable result	Unwritten design intent
Project guide	Explain project-wide rules	Automatic enforcement
Dependency rule	Restrict architecture	Functional correctness

None of these should be expected to replace all the others.

Separate permanent principles from temporary work notes

A single guide can become confusing if it mixes:

• permanent project principles;
• one specific architectural decision;
• today’s unfinished task.

A lightweight separation can help:

PROJECT_GUIDE.md
    Stable purpose, non-goals, and project-wide rules.

docs/decisions/
    Short explanations of important accepted decisions.

CURRENT_STATE.md or an issue
    Current task, temporary assumptions, and open questions.

An architectural decision does not need a formal essay:

# Keep tool execution outside the model

Status: accepted

Context:
Model output cannot be treated as trusted executable input.

Decision:
Every tool call passes through one validated executor.

Consequences:
Unknown tools are rejected.
Arguments are validated before execution.
UI code cannot invoke tools directly.

The main value is that a future session does not have to invent a new explanation for the same boundary.

Make authority visible

An external document is not automatically correct merely because it exists.

A possible priority order is:

1. observed runtime results and test output;
2. current source code and schemas;
3. author-approved decisions;
4. reviewed project guidance;
5. generated documentation;
6. old chat history.

The exact order is your project policy, but generated explanations and confirmed decisions should not be treated as the same category.

How to split the file without losing control

Before deciding on folder names, create a one-page map of one normal request:

User / UI
    ↓
Conversation or task state
    ↓
Prompt and context construction
    ↓
Model call
    ↓
Tool-call parsing
    ↓
Schema and permission validation
    ↓
Tool execution
    ↓
Persistent state / external service
    ↓
Final response

For each part, note:

Responsibility
Inputs
Outputs
Owned state
External side effects
Failure modes

This does not need to be a formal UML diagram. The purpose is to reveal hidden state, mixed responsibilities, and unclear dependency directions.

Identify who owns each kind of state

AI applications often use the words “memory,” “history,” “context,” and “state” for several different things.

A useful table is:

Data	Lifetime	Owner	Source of truth
Current prompt context	One model call	Prompt builder
Conversation history	One thread/session	Conversation store
User preferences	Across sessions	Persistent memory component
Tool registry	One application release	Code or configuration
Accepted design rules	Long-term	Reviewed project guide
Model-generated summary	Temporary or derived	Not authoritative by default

For each item, it may help to know:

Who may write it?
Who may read it?
Can it be regenerated?
When should it be deleted?

Sometimes what looks like a “large file problem” is partly a state-ownership problem.

Define success before requesting a refactor

“Split this file cleanly” gives the model too much freedom.

A safer acceptance definition is:

- Existing public function signatures remain unchanged.
- Tool names and argument schemas remain unchanged.
- Existing persistent data remains readable.
- No new feature is added.
- The same checks pass before and after.
- Only the planned files are changed.
- The project guide remains accurate.
- The new dependency direction can be explained.

This tells the model what “better” means for this particular change.

Test the deterministic shell around the model

You do not need to make the model’s natural-language output fully deterministic before testing the program.

Start with the parts around it:

• unknown tools are rejected;
• malformed arguments are rejected;
• the expected tool receives the expected input structure;
• filesystem writes stay inside an allowed location;
• state is saved in the expected format;
• stored data remains readable after the refactor;
• exceptions are surfaced or logged;
• the program starts with the documented configuration.

With pytest:

• monkeypatch can replace a model client, network call, or environment variable;
• tmp_path provides an isolated filesystem location for tests.

This makes it possible to test tool dispatch, storage, parsing, and error handling without calling a live model every time.

If you do not yet have automated tests, a written smoke-test checklist is still useful:

1. Start the assistant.
2. Send one normal request.
3. Trigger one representative tool.
4. Try one invalid tool argument.
5. Restart and confirm that required state is still readable.

The checklist can be converted into pytest tests later.

Choose the first extraction by coupling, not by line count

A good first candidate usually has:

• clear inputs and outputs;
• few callers;
• little global state;
• limited side effects;
• a contract that can be tested independently.

Possible examples include:

• a formatter;
• a parser;
• a configuration loader;
• a result type;
• one self-contained tool wrapper.

These are only examples. The correct first component depends on your actual dependencies.

If nothing can be moved safely, create a boundary first.

For example, instead of constructing dependencies inside a function:

def answer(text):
    client = ModelClient(...)
    storage = Database(...)
    result = client.generate(text)
    storage.save(result)

move toward explicitly supplied dependencies:

def answer(text, model_client, storage):
    result = model_client.generate(text)
    storage.save(result)

You may still keep it in the same file initially. The important step is making the dependency visible and replaceable. Extraction becomes easier afterward.

Use a small change loop

map
→ define acceptance criteria
→ add or record checks
→ create one boundary
→ move one responsibility
→ run checks
→ inspect the diff
→ update relevant documentation
→ commit

Avoid combining:

• file splitting;
• framework migration;
• prompt redesign;
• naming every component again;
• replacing the model;
• adding new features.

Each may be reasonable separately, but combining them makes it difficult to know which change caused a regression.

Make tool contracts explicit

The smolagents tool documentation is a useful example even if you do not use that framework. It describes a tool through an API containing a name, description, input types and descriptions, and output type.

A useful inventory is:

Field	Meaning
Name	Stable identifier shown to the model
Purpose	When the tool should be used
Inputs	Names, types, and constraints
Output	Stable return structure
Side effects	Filesystem, network, database, subprocess
Permissions	Resources the tool may access
Failures	Invalid input, missing data, timeout
Owner	Module responsible for implementation

This gives both you and the model an explicit boundary to preserve during the split.

Keep a small record of actual executions

The source code describes what should happen. A run log describes what actually happened.

Even a basic structured record can include:

{
  "run_id": "example-run",
  "model": "provider/model-version",
  "instruction_version": "project-guide-v3",
  "context_files": [
    "PROJECT_GUIDE.md",
    "src/tools/search.py"
  ],
  "tool_calls": [
    {
      "name": "search",
      "status": "success"
    }
  ],
  "state_writes": [
    "conversation_history"
  ],
  "errors": []
}

Do not log secrets or unnecessary private content. Model version, selected files, tool names, state changes, and error information may already make debugging much easier.

Environment and tool options

You can start with ordinary repository files:

README.md
PROJECT_GUIDE.md
pyproject.toml or another dependency record
src/ or the current source location
tests/

A pyproject.toml can hold project metadata and configuration used by packaging tools, linters, type checkers, and other development tools.

It is also useful to record:

Python version
dependency versions
run command
test command
required environment variables
model/provider configuration
persistent-data or schema version

A fresh session should not need to guess how to install, run, or validate the application.

If you only have a normal chat interface

That is enough to begin.

Supply:

PROJECT_GUIDE.md
+ relevant Python file
+ relevant caller/dependency
+ related test or smoke checklist

Ask for a map and plan before requesting edits.

File upload alone does not necessarily mean that a RAG system exists, but that is not a problem at this scale. Directly selected context is often easier to inspect than an opaque retrieval pipeline.

If your coding assistant supports persistent instructions

Use its existing mechanism.

GitHub Copilot supports repository custom instructions, including information about how the project should be understood, built, tested, and validated.

A vendor-neutral AGENTS.md is another possible convention for tools that support it.

Useful content includes:

• exact setup and test commands;
• unusual directory conventions;
• stable public interfaces;
• forbidden dependency directions;
• tool permissions;
• storage compatibility rules;
• checks required before a change is complete.

Less useful content includes:

• generic Python advice;
• a complete copy of the README;
• obvious descriptions of every file;
• outdated TODO lists;
• unreviewed model guesses.

Keep persistent instructions short, specific, current, and focused on information that cannot be reliably recovered from the code itself.

If a repository-aware coding tool would help

Aider is one example rather than a required answer.

Its documentation describes:

• a repository map for supplying structural context;
• automatic linting and testing after edits;
• Git-based workflows that make changes inspectable and reversible.

The important capabilities are:

• repository navigation;
• visible diffs;
• Git integration;
• persistent project guidance;
• configurable lint and test commands;
• control over which files may be changed.

A repository map can help the model find functions and classes. It still cannot determine your purpose, non-goals, or forbidden changes unless you write them down.

If you already have RAG

Include more than source code in the knowledge base:

• reviewed project guidance;
• architecture notes;
• tool contracts;
• tests and examples;
• accepted decisions;
• current task information.

Useful metadata might include:

type: principle | architecture | code | test | decision | current_state
scope: project | module | tool
status: accepted | hypothesis | deprecated
authority: author | code | test_result | generated
last_verified: 2026-07-07

Try to make retrieved chunks visible during debugging. Otherwise:

• a retrieval failure can look like a reasoning failure;
• an old document can repeatedly override current code;
• a generated hypothesis can be retrieved as though it were an accepted rule.

RAG solves the problem of selecting relevant information from a larger collection. It does not create missing intent or decide which source is authoritative.

If you do not have RAG

I would not add it only because the file reached 1,200 lines.

Consider retrieval when several of these become true:

• manually selecting relevant files is difficult;
• the project spans many modules or repositories;
• issues, decisions, tests, and documentation must be searched together;
• direct context is consistently too large;
• different tasks require very different subsets of material;
• you can maintain and inspect the index.

Before that point, a small guide plus directly selected source files is often the easier system to understand.

Optional supporting tools

These are optional, not requirements:

• Ruff: linting and formatting;
• Pyright/Pylance: gradual type checking;
• pytest: behavior checks;
• GitHub Desktop: visual Git workflow if command-line Git is uncomfortable;
• pre-commit: run stable checks before commits;
• GitHub Actions: run established checks automatically after pushing.

Introduce tools one at a time, after the underlying manual process is understood.

How to compare models after the project is easier to inspect

I would compare models in two stages.

Stage 1: mapping without editing

Give every model the same code, guide, and task.

Ask it to identify:

• the relevant component;
• callers and dependencies;
• shared state;
• side effects;
• relevant tests;
• possible risks;
• facts it cannot determine.

This reveals whether the model can form a grounded plan before it starts rewriting files.

Stage 2: one constrained edit

For example:

Move one tool behind a separate module or interface.

Do not change:
- its public name;
- argument schema;
- output format;
- permissions;
- error behavior;
- persistent data.

List the files you plan to change before editing.
Run the specified checks afterward.

Evaluate:

• whether the planned files match the actual diff;
• whether it invented APIs or assumptions;
• whether unrelated code changed;
• whether the interface remained compatible;
• whether tests were actually run;
• whether uncertainty was reported clearly;
• whether the resulting code is easier for you to explain.

You can also separate implementation and review:

Session or model A:
Create one constrained patch.

Fresh session or model B:
Review only the diff against the project guide,
acceptance criteria, and relevant tests.

The second model is an additional reviewer, not a replacement for deterministic checks or your own review.

Use failures from your own project

Public coding benchmarks may be informative, but your previous failures are more directly relevant.

Useful regression cases might include:

• the wrong tool was selected;
• malformed arguments were accepted;
• a file was modified unexpectedly;
• state became unreadable;
• old interfaces were used;
• a side effect happened without permission;
• a refactor silently changed behavior.

Even five or ten cases can provide a more meaningful model comparison than asking which model is generally “best.”

For this kind of maintenance work, the most valuable model may be the one that:

• limits its scope;
• identifies uncertainty;
• produces small diffs;
• preserves contracts;
• uses the provided validation process;
• makes changes you can still understand.

Important pitfalls

Do not replace one monolith with another

It is possible to replace one large Python file with:

• dozens of tiny modules;
• a large documentation hierarchy;
• a vector database;
• several agents;
• an orchestration framework;
• a complex evaluation system.

That can increase rather than reduce the amount you must manage.

Start with the smallest artifact that solves the immediate problem.

Do not promote generated documentation directly into truth

A model can draft:

• architecture maps;
• docstrings;
• tool descriptions;
• inferred design principles.

But code often supports several plausible explanations. Generated intent should remain marked as inferred until you approve it.

Do not let documentation become stale

Whenever a change is made, briefly check:

Did a public interface change?
Did a tool contract change?
Did the data flow change?
Did a project rule change?
Did the run or test command change?
Did an old note become false?

If not, no documentation update is needed. If yes, update it in the same change rather than postponing it indefinitely.

Do not treat test success as proof of good architecture

Tests show that tested behavior still works. They do not prove that the dependency structure is understandable or that the next feature will be easy to add.

After each extraction, also ask:

• Is state ownership clearer?
• Is the dependency direction easier to explain?
• Can the component be tested independently?
• Did the change reduce or merely move complexity?
• Will the next change be easier?

Do not use prompts as a security boundary

If tools can:

• execute Python or shell commands;
• write files;
• access credentials;
• call external services;
• modify persistent state;

then permissions and validation should exist in deterministic code.

Hugging Face’s secure code execution guidance notes that executing LLM-generated code in the local environment is inherently risky. Model instructions are not a sandbox.

Do not combine every improvement into one refactor

Keep these separate when possible:

module extraction
framework migration
model change
prompt redesign
storage migration
new features

Small, understandable changes are easier to verify and easier to reverse.

Short version

If very little project infrastructure exists yet, this would be enough to begin:

1. Save the working version in Git.
2. Write one short PROJECT_GUIDE.md.
3. Record three repeatable behavior checks.
4. Ask the AI for a read-only system map.
5. Correct the map yourself.
6. Move one small responsibility.
7. Re-run the checks and inspect the diff.

If Git and tests already exist, start with the project guide, explicit tool contracts, and state ownership.

If those already exist, better repository context, execution logs, and project-specific model evaluations are likely the next useful layer.

The objective is not to memorize every line or personally perform every edit. It is to keep the project’s purpose, boundaries, evidence, and acceptance decisions under your control while letting the model help with mapping, drafting, and implementation.