Oh. After trying a few things in Colab, I think I may have identified at least some of the weak points in the current implementation:
The overall architecture you ended up with looks like the right one:
Ali TTS
→ Ali mouth-analysis loop
→ Ali VTS client
→ Ali VTube Studio port/session
→ Ali model
Hara TTS
→ Hara mouth-analysis loop
→ Hara VTS client
→ Hara VTube Studio port/session
→ Hara model
In other words:
- one WebSocket client per VTube Studio instance;
- one configured port per client;
- separate token storage so the clients cannot accidentally overwrite each other’s local token state;
- each TTS stream sends
MouthOpen only through its own VTS client.
Both models can use the parameter name MouthOpen. Because they are running in separate VTube Studio instances, those are separate API contexts. The VTS rule that only one plugin can normally "set" a parameter applies within one VTS instance, not across two separate instances.
I do not think the remaining weak points are mainly about needing another mouth parameter. They look more like authentication-session state, reconnection state, and cleanup state.
The four changes I would make first are:
- Read the actual
AuthenticationResponse.data.authenticated value.
- Store a newly received token in
self.token as well as in the token file.
- Clear the local authenticated/ready state as soon as the socket disconnects.
- Put TTS/mouth cleanup in a
finally block.
Recommended default lifecycle
For each character, I would use this sequence:
Connect to the configured VTS port
→ optionally request APIStateRequest for diagnostics
→ load an existing token, or request a new token
→ store a new token in memory and on disk
→ send AuthenticationRequest
→ check data.authenticated and data.reason
→ request CurrentModelRequest
→ verify that the expected model is loaded
→ mark this client READY
→ allow mouth/hotkey/parameter requests
After a disconnection:
Socket closes
→ immediately clear READY/authenticated
→ stop or ignore outgoing parameter updates
→ create a new connection
→ authenticate the new session with the stored token
→ verify the model again
→ return to READY
This follows the separation already described in the official VTube Studio API authentication documentation: obtaining a token and authenticating the current WebSocket session are two different steps.
A token normally only needs to be obtained once. The same stored token can be used to authenticate later sessions, such as after VTube Studio restarts or the plugin reconnects.
Minimal changes
1. Treat the response data as the authentication result
Receiving a message whose type is AuthenticationResponse does not by itself mean authentication succeeded.
The official API returns the same response type when a token is invalid or has been revoked, but sets:
{
"authenticated": false,
"reason": "..."
}
A minimal pattern would be:
elif mtype == "AuthenticationResponse":
data = msg.get("data", {})
self.authenticated = bool(
data.get("authenticated", False)
)
if self.authenticated:
print("Authenticated with VTube Studio.")
else:
reason = data.get(
"reason",
"Unknown authentication failure"
)
print(f"VTube Studio authentication rejected: {reason}")
This is probably the most important small change.
At the moment, both client variants set:
self.authenticated = True
whenever an AuthenticationResponse arrives, without checking whether VTS actually returned true or false.
2. Keep the newly issued token in memory
When the token response arrives, update both the file and the current client object:
elif mtype == "AuthenticationTokenResponse":
data = msg.get("data", {})
token = data.get("authenticationToken")
if token:
self.token = token
self.save_token(token)
# Send AuthenticationRequest here using this token.
self.authenticate(token)
The important line is:
self.token = token
Without that, the token may be correctly written to disk and used in the immediate authentication request, while the current object still thinks it has no token.
That becomes relevant if the socket disconnects and reconnects during the same Python process. The reconnection code may enter the “no token” branch again even though a valid token was already obtained and saved.
For Ali, there are two reasonable options:
- keep the “restart after receiving the token” flow, but do not set
authenticated=True before the next session is authenticated; or
- use the same immediate authentication flow as Hara.
The second option makes both clients follow the same lifecycle and is probably easier to maintain.
3. Clear session state on disconnect
A valid token can survive reconnects, but the authenticated state belongs to the current WebSocket session.
At minimum:
def on_close(
self,
ws,
close_status_code,
close_msg
):
self.authenticated = False
print(
"VTS connection closed. "
"Retrying in 5 seconds..."
)
time.sleep(5)
self._connect()
It may be even clearer to use a separate state such as:
DISCONNECTED
CONNECTING
CONNECTED
AUTHENTICATING
READY
RECONNECTING
Then mouth and hotkey requests can be allowed only in READY.
A small guard is already useful even without a full state machine:
def set_mouth(self, value):
if not self.authenticated:
return False
# Send InjectParameterDataRequest here.
That prevents a stale local True value from allowing parameter requests before the new session has authenticated.
4. Make TTS cleanup exception-safe
The mouth-reset and mute-reset logic should run even if audio playback, WebSocket sending, interruption handling, or sd.wait() fails.
Conceptually:
tts_muted = True
try:
sd.play(audio, sample_rate)
# Analyse audio blocks.
# Send MouthOpen values.
# Wait for playback completion.
finally:
try:
vtube.set_mouth(0.0)
except Exception as cleanup_error:
print(
"Could not reset mouth during cleanup:",
cleanup_error
)
tts_muted = False
The inner try around set_mouth(0.0) is useful because the cleanup send can also fail if the socket disconnected. It should not hide the original playback exception.
A small verification matrix
These checks would cover most of the practical separation contract:
| Test |
Expected result |
| Run Ali only |
Only Ali’s mouth moves |
| Run Hara only |
Only Hara’s mouth moves |
| Run both, play Ali TTS only |
Hara remains closed |
| Run both, play Hara TTS only |
Ali remains closed |
| Alternate Ali and Hara speech |
The active speaker alone receives mouth updates |
| Restart only the Python scripts |
Stored tokens authenticate the new sessions |
| Restart only one VTS instance |
Only that client loses READY and re-authenticates |
| Revoke permission for one plugin |
The other client continues working |
| Reverse the VTS startup order |
Model verification catches an incorrect port/model pairing |
| Disconnect during TTS |
Mouth and local mute/playback flags return to their safe state |
For useful logs, I would include:
character
port
absolute token-file path
connection generation
requestID
messageType
authenticated
authentication reason
modelName
modelID
API error ID
API error message
Do not log the token itself.
Using an absolute token path in the log is helpful because a relative filename such as vts_token.json is resolved from the process’s current working directory. Two terminals may therefore read different files—or unexpectedly the same file—depending on how they were launched.
What I tested in Colab, and what the tests do not prove
I ran several small probes against the published code paths:
- direct callback tests using mocked VTS responses;
- an
authenticated: false response;
- token receipt followed by reconnect;
- disconnect while the local authenticated flag was true;
- delayed responses associated with an older mock socket;
- repeated/overlapping close events;
- exceptions during audio start, mouth streaming, and audio wait;
- seeded randomized event-order traces;
- the same local regression matrix before and after a candidate hardening patch.
The public probes are here:
The first notebook checks whether the published state transitions can enter problematic local states when authentication, disconnect, reconnect, and callback events are reordered.
The second notebook generates a candidate hardening patch and applies the same regression checks before and after the candidate changes.
Within that local harness, the candidate changes moved the regression matrix from 5/15 passing cases to 15/15. That result is useful as a consistency check for the proposed state-management changes, but it is not a measurement of real-world failure probability.
The tests reproduced these reachable states:
AuthenticationResponse with authenticated: false being treated locally as success;- a newly issued token being written to disk but not assigned to
self.token;
- the local authenticated flag remaining true after disconnect;
- mouth requests being locally allowed before a new session had authenticated;
- cleanup not being reached after an injected TTS/WebSocket exception.
However, these probes do not reproduce the full VTube Studio environment.
They do not prove:
- the historical root cause of the original instability;
- that old socket callbacks actually arrive in the normal setup;
- that duplicate close callbacks occur during normal operation;
- the real port/model pairing on the machine;
- the real Live2D parameter mappings;
- the real audio-device timing;
- that the candidate patch is production-ready.
I would therefore describe the results as:
reproducible state-management edge cases and hardening opportunities
rather than:
proof of the original root cause.
Authentication, tokens, and reconnect lifecycle
There are several related but different states:
WebSocket object exists
WebSocket transport is connected
A token is available
AuthenticationRequest was sent
The current session is authenticated
The expected model is loaded
The client is ready to inject parameters
Those should not all be represented by one early authenticated=True.
Token acquisition versus session authentication
The VTube Studio authentication documentation describes two operations:
AuthenticationTokenRequestAuthenticationRequest
The first obtains a reusable token after the user approves the plugin.
The second authenticates the current WebSocket session with that token.
The official documentation specifically says that:
- a token usually only needs to be obtained once;
- it can be reused when VTS restarts or the plugin reconnects;
pluginName and pluginDeveloper used during authentication must match the values used when the token was requested;- an invalid or revoked token can produce an
AuthenticationResponse whose authenticated field is false.
Separate pluginName values for Ali and Hara are useful for human-readable permissions and logs, but they are not inherently required merely because there are two VTS instances.
The hard requirements are:
- keep each client’s connection/session state separate;
- keep the token accepted for that client paired with its authentication identity;
- use matching
pluginName and pluginDeveloper values when reusing a token;
- do not let one client’s mutable token state overwrite the other’s.
Checking the current session
The official API provides APIStateRequest, whose response includes:
{
"active": true,
"currentSessionAuthenticated": false
}
This is useful because it asks VTube Studio about the current session instead of relying only on a local Python boolean.
The API documentation also recommends assigning a requestID. VTS returns it with the response and includes it in its logs when errors occur.
For two clients, request IDs such as these are easier to trace:
ali-g3-auth
ali-g3-model
ali-g3-mouth-0042
hara-g2-auth
hara-g2-model
hara-g2-mouth-0017
Here, g3 could mean “connection generation 3.”
Verifying the loaded model
After authentication, CurrentModelRequest can return:
modelLoaded
modelName
modelID
timeSinceModelLoaded
See Getting the currently loaded model.
This provides a stronger check than assuming:
port 8001 always means Ali
port 8003 always means Hara
A better rule is:
The Ali client is READY only if:
- its session authenticated, and
- the expected Ali modelID is loaded.
The same applies to Hara.
VTS also has UDP API discovery, which broadcasts the active port, instanceID, and window title.
One limitation is that instanceID is generated when a VTS instance starts. It remains stable while that instance is running, but it should not be treated as a permanent character ID across restarts.
For persistent character routing, the loaded modelID is the more useful final check.
Old callbacks and connection generations
A defensive reconnect design can assign each new connection a generation number:
self.connection_generation += 1
generation = self.connection_generation
Callbacks capture that generation and ignore messages if they do not belong to the current socket.
Conceptually:
def is_current_connection(self, ws, generation):
return (
ws is self.ws
and generation == self.connection_generation
)
This can prevent an old callback from modifying the state of a newer session.
I could reproduce this problem only by deliberately reordering mock events. I do not know whether that exact event order occurs in the normal setup, so I would treat generation checks as defensive hardening, not as a confirmed fix for the original problem.
The same applies to a reconnect guard. It is reasonable to prevent two close paths from scheduling two reconnects, but the tests do not prove that normal websocket-client operation is currently creating duplicate connections.
The websocket-client long-lived connection examples also show dispatcher-based reconnection using the reconnect argument. That is another design reference, not necessarily a requirement to replace the current implementation.
Diagnostics if one mouth still does not behave correctly
A useful diagnostic order is:
Can the WebSocket connect?
├─ No
│ ├─ verify the configured port
│ ├─ verify "Allow Plugin API access"
│ └─ check firewall/antivirus restrictions
│
└─ Yes
└─ Is the current session authenticated?
├─ No
│ ├─ check stored token
│ ├─ check pluginName/pluginDeveloper pairing
│ ├─ check whether permission was revoked
│ └─ inspect AuthenticationResponse.reason
│
└─ Yes
└─ Is the expected modelID loaded?
├─ No
│ └─ wrong VTS instance, wrong startup order,
│ or model still loading
│
└─ Yes
└─ Does InjectParameterDataRequest succeed?
├─ APIError 453
│ └─ parameter does not exist
│
├─ APIError 454
│ └─ another plugin controls the parameter
│ in this VTS instance
│
└─ InjectParameterDataResponse
└─ Does the VTS input value move?
├─ No
│ └─ inspect the injection/update path
│
└─ Yes, but the visible mouth does not move
└─ inspect model mapping, smoothing,
expressions, animations, and physics
API errors worth logging
The official VTube Studio error list includes:
| Error |
Meaning |
8 |
Request requires authentication |
50 |
User denied the token request |
453 |
Parameter name does not exist |
454 |
Parameter is controlled by another plugin |
455 |
Unknown injection mode |
A simple handler would make troubleshooting much easier:
elif mtype == "APIError":
data = msg.get("data", {})
print(
"VTS API error:",
data.get("errorID"),
data.get("message"),
"requestID=",
msg.get("requestID"),
)
Parameter refresh timing
The parameter injection documentation says that a plugin must resend a parameter it wants to control at least once every second.
If it stops sending for longer than that, VTS considers the parameter “lost” and returns control to the previous source or the default value.
The existing mouth loop appears comfortably above that documented minimum.
I did not find an official maximum injection frequency in the API documentation, so I would not treat the current observed send rate as the root cause without measuring behavior against a real VTS instance.
Parameter ownership
For the normal "set" mode, only one plugin can control a given parameter at a time inside one VTS instance.
VTS also has an "add" mode that can combine inputs from multiple plugins, but I do not think that is needed for this case. A direct mouth controller is easier to reason about in "set" mode.
Because Ali and Hara use separate VTS instances, both instances can independently have a MouthOpen parameter controlled by their corresponding client.
If VTS accepts the parameter but the visible mouth does not move
At that point the problem is probably no longer token routing.
The next layer is the model configuration:
- Is
MouthOpen mapped to the correct Live2D output parameter?
- Is smoothing delaying the visible response?
- Is an active expression overriding the output?
- Is an animation overriding it?
- Is physics affecting the same output?
- Is VTS’s built-in microphone lip-sync also active?
The VTube Studio model settings guide is the relevant reference for that branch.
TTS interruption and mouth cleanup
The playback path has several operations that can fail independently:
TTS generation
audio buffer conversion
sd.play()
mouth-analysis loop
WebSocket parameter send
sd.wait()
user interruption
device disconnect
If cleanup only exists at the normal end of the function, an exception can skip it.
Possible leftover state includes:
tts_muted = True
tts_playing = True
the final local mouth value is nonzero
the speech lock remains held longer than expected
A safer structure is:
def play_tts_with_mouth(audio, sample_rate):
global tts_muted
tts_muted = True
try:
sd.play(audio, sample_rate)
# Process blocks and send mouth values.
# Wait for playback completion.
finally:
try:
vtube.set_mouth(0.0)
except Exception as cleanup_error:
print(
"Mouth cleanup failed:",
cleanup_error
)
tts_muted = False
If there are more flags, reset all of them in the same cleanup section:
finally:
best_effort_reset_mouth()
tts_muted = False
tts_playing = False
tts_generating = False
If an audio_lock is acquired manually, it should also be released reliably. Using:
with audio_lock:
...
is safer than separate acquire() and release() calls.
This is not evidence that TTS cleanup caused the original token behavior. It is a separate hardening point that can prevent a disconnect or playback exception from leaving the character visually or locally stuck.
Optional cleanup for a reusable public example
The Ali and Hara clients currently follow slightly different authentication flows.
That is understandable if they developed from different versions of the code, but keeping two nearly identical classes can make future fixes drift apart.
A reusable example could use one client class with per-character configuration:
@dataclass(frozen=True)
class VTSClientConfig:
character: str
port: int
token_path: str
plugin_name: str
plugin_developer: str
expected_model_id: str | None = None
Then:
ali_client = VTSClient(
VTSClientConfig(
character="Ali",
port=8001,
token_path="tokens/ali_vts_token.json",
plugin_name="Local AI VTuber - Ali",
plugin_developer="Bro77xp",
expected_model_id="ALI_MODEL_ID",
)
)
hara_client = VTSClient(
VTSClientConfig(
character="Hara",
port=8003,
token_path="tokens/hara_vts_token.json",
plugin_name="Local AI VTuber - Hara",
plugin_developer="Bro77xp",
expected_model_id="HARA_MODEL_ID",
)
)
Using different plugin names is useful for human-readable permission screens and logs, but it is not the essential separation mechanism.
The essential separation is:
client object
connection/session
port
token state
expected model
audio source
mouth destination
Other useful public-example improvements would be:
- log the absolute token-file path;
- use character-specific request IDs;
- handle
APIError;
- document the first-run authorization flow;
- document normal restart behavior;
- document recovery after permission revocation;
- use a stop event or graceful shutdown path;
- prevent multiple reconnect workers from being scheduled;
- add dependency versions;
- place forum code inside fenced code blocks so copied quotes and indentation remain valid.
The pyvts project is also useful as a reference for how an existing Python wrapper separates connection and authentication operations. I do not think migration to a wrapper is required, but comparing lifecycle boundaries can be helpful.
Short version
Your two-instance solution is fundamentally the correct direction.
I would keep:
separate VTS instance
separate client object
separate configured port
separate token storage
separate TTS-to-mouth route
Then harden these points first:
1. Read AuthenticationResponse.data.authenticated.
2. Save a newly issued token into self.token.
3. Clear authenticated/READY on disconnect.
4. Re-authenticate every new WebSocket session.
5. Verify the loaded model before enabling output.
6. Reset mouth and TTS state in finally.
The connection-generation and reconnect-guard ideas are useful additional defenses, but I would treat them as the next layer rather than the first required changes.
With those additions, this becomes much easier for another person to copy, debug, and extend without confusing:
token obtained
session authenticated
correct model connected
mouth updates active
as if they were the same state.