Enigma Sound : Multi-Modal Emotion-to-Music Architecture Layout (Gradio + CNN/LSTM Walkthrough)

Show and Tell
1

Hey everyone,

I wanted to share a UI case study layout I put together for a research project mapping text, vocal frequencies (Bi-LSTM), and facial micro-expressions (CNN) into dynamic audio layers via Music21.

Because the underlying models are too heavy for basic free tiers, I built a lightweight Gradio interface to act as a 0-click visual production walkthrough and tech-stack overview.

Would love any feedback on the layout structure or optimization tips for multi-stream pipelines!

Link: Enigma Sound Ai - a Hugging Face Space by ApurvaDev111

2

Update: I’ve migrated to a new, more stable environment for this project. Please use this updated link for the live demo: Enigma Sound Ai - a Hugging Face Space by ApurvaDev111

3

Hmm… assuming we are not trying to change the architecture too drastically, maybe something like this:

Nice walkthrough — the updated Space/video makes the mobile user flow much easier to understand. I would probably frame this as a modular multimodal emotion-to-music pipeline, not as something that needs to become a large end-to-end fusion system immediately.

My short answer to your two questions would be:

Area Suggestion
Layout structure Show the three detectors as parallel streams first: text, audio/voice, and face. Then show one explicit handoff layer before music generation/recommendation.
Multi-stream optimization Make the detector-to-music handoff richer than just a top emotion label. Each stream can return label_distribution, confidence, input_quality, and fallback/status.

So the main architectural suggestion is:

Keep the modular detector-based design,
but make the fusion / detector-to-music contract more explicit.

A compact layout could be:

Text input  -> text preprocessing  -> NLP emotion model      -> stream result
Voice/audio -> MFCC/features        -> CNN/LSTM audio model   -> stream result
Face/video  -> face crop/48x48 img  -> CNN facial model       -> stream result
                                                                  |
                                                                  v
                                                quality/confidence-aware fusion
                                                                  |
                                                                  v
                                      music-control state / recommendation hints
                                                                  |
                                                                  v
                                          Music21 + FluidSynth / Spotify layer

The important part is the shape of stream result. Instead of each detector only returning:

{ "emotion": "happy" }

I would make each stream return something closer to:

{
  "modality": "audio",
  "status": "ok",
  "top_label": "happy",
  "label_distribution": {
    "happy": 0.52,
    "neutral": 0.28,
    "sad": 0.08,
    "angry": 0.07,
    "fear": 0.05
  },
  "confidence": 0.52,
  "input_quality": {
    "score": 0.74,
    "flags": []
  },
  "fallback_used": false
}

That small contract upgrade helps both layout and optimization:

Problem Why the richer contract helps
Different detectors may use different emotion classes A distribution is easier to map than one hard label.
One modality is weak input_quality can reduce its weight.
One modality is missing status / fallback_used makes the behavior explicit.
Streams disagree The fusion layer can arbitrate instead of treating every modality equally.
Music output feels too coarse The final emotion can be converted into a richer music-control state.

For the music side, I would avoid treating the detected emotion label as the whole control signal. A cleaner intermediate layer could be:

emotion distribution
-> valence / arousal
-> tempo, mode, note density, instrumentation, recommendation hints
-> generated or recommended music

For example:

Emotion-side signal Music-control use
Valence mode, harmonic brightness, consonance/dissonance
Arousal tempo, rhythmic activity, note density, loudness
Uncertainty safer fallback, neutral blend, avoid extreme stylistic jumps
Top label choose a preset/template
Label distribution blend adjacent moods instead of hard-switching

A fused output could look like:

{
  "fused_emotion": {
    "top_label": "happy",
    "distribution": {
      "happy": 0.48,
      "neutral": 0.31,
      "surprise": 0.12,
      "sad": 0.09
    },
    "uncertainty": 0.35
  },
  "music_control": {
    "valence": 0.72,
    "arousal": 0.64,
    "tempo_range": [110, 140],
    "density": "medium_high",
    "mode_hint": "major_or_bright",
    "recommendation_mode": "uplifting"
  }
}

That keeps the system explainable and practical, especially for a mobile/edge-style walkthrough. A fully end-to-end multimodal fusion model may be a long-term research ideal, but this smaller contract upgrade gives many of the practical benefits without requiring a completely different training/data pipeline.

Why I would keep the modular stream design for now

A larger end-to-end fusion model could be attractive in theory because it may learn interactions between text, face, and audio directly. But it also raises the cost of data collection, training, evaluation, debugging, and missing-modality handling.

For this project, the modular approach has several practical advantages:

Dimension Modular stream design Large end-to-end fusion
Implementation cost Lower Higher
Debugging Easier; each stream can be tested alone Harder; errors are more entangled
Demo clarity Strong; good for walkthroughs Often more black-box
Missing inputs Easier to handle with fallbacks Needs careful training/masking
Model updates One stream can be replaced independently Changes may affect the full model
Runtime cost Easier to keep light Usually heavier
Explainability High Lower unless carefully designed
Practical fit here Strong Probably future work

So I would not frame the next step as “replace the architecture.” I would frame it as:

keep modular detectors
-> standardize stream outputs
-> add quality/confidence-aware fusion
-> map fused emotion into music controls

This is close to a late fusion or decision-level fusion strategy, but with a better interface between streams. The key improvement is not necessarily a more complex model. It is a better contract between small models.

A practical way to think about it:

Current abstraction Slightly stronger abstraction
text -> emotion text -> distribution + confidence + quality
audio -> emotion audio -> distribution + confidence + quality
face -> emotion face -> distribution + confidence + quality
emotion -> music fused emotion state -> music-control state -> music

This is a useful middle ground between:

very simple:
one detected label -> one music rule

and:

very complex:
all raw modalities -> one large end-to-end model -> generated music

The middle ground is:

modality-specific detectors
-> standardized emotional evidence
-> lightweight fusion
-> controllable music mapping

That feels like the most practical path for this project.

Suggested detector-to-music contract

I would define a small schema for every detector output. The goal is to make every modality speak the same language.

Suggested fields:

Field Meaning
modality text, audio, or face
status Whether this stream should participate in fusion
top_label The highest-scoring emotion
label_distribution Scores over the project’s normalized emotion vocabulary
confidence How confident the model appears to be
input_quality.score How usable the raw input was
input_quality.flags Human-readable quality notes
fallback_used Whether the stream used fallback behavior
timing Optional timestamp/window info for audio/video

Example:

{
  "modality": "face",
  "status": "ok",
  "top_label": "surprise",
  "label_distribution": {
    "happy": 0.05,
    "sad": 0.02,
    "angry": 0.03,
    "fear": 0.09,
    "surprise": 0.68,
    "neutral": 0.11,
    "disgust": 0.02
  },
  "confidence": 0.68,
  "input_quality": {
    "score": 0.81,
    "flags": ["single_face_detected", "face_crop_ok"]
  },
  "fallback_used": false,
  "timing": {
    "start_ms": 0,
    "end_ms": 1000
  }
}

For a weak input:

{
  "modality": "audio",
  "status": "low_quality",
  "top_label": "sad",
  "label_distribution": {
    "happy": 0.14,
    "sad": 0.31,
    "neutral": 0.29,
    "angry": 0.10,
    "fear": 0.16
  },
  "confidence": 0.31,
  "input_quality": {
    "score": 0.38,
    "flags": ["short_clip", "low_volume", "high_silence_ratio"]
  },
  "fallback_used": false
}

For a missing stream:

{
  "modality": "face",
  "status": "missing",
  "top_label": null,
  "label_distribution": {},
  "confidence": 0.0,
  "input_quality": {
    "score": 0.0,
    "flags": ["no_face_detected"]
  },
  "fallback_used": true
}

This keeps the fusion layer simple:

Do not ask the fusion layer to inspect raw images, raw audio, or raw text.
Ask it to combine standardized emotional evidence.

That makes the architecture easier to test.

Recommended emotion vocabulary normalization

A hidden issue in multimodal emotion projects is that each model often uses a different label set.

For example, one model may return:

text: sadness, joy, love, anger, fear, surprise

while another may return:

face: angry, disgust, fear, happy, neutral, sad, surprise

and audio may use something like:

audio: neutral, calm, happy, sad, angry, fearful, disgust, surprised

If these go directly into music generation, the mapping becomes brittle.

A safer design is to create a project-level normalized vocabulary:

happy
sad
angry
fear
surprise
neutral
calm
disgust

Then map each detector’s native labels into that shared vocabulary.

Example:

Native label Normalized label Notes
joy happy text model may use joy instead of happy
sadness sad common NLP label
fearful fear common speech label
surprised surprise speech/face naming difference
calm calm or neutral depends on desired music mapping
love happy / positive may be better handled through valence
disgust disgust or negative_high_tension may need music-specific handling

For fusion, I would usually prefer distributions over hard labels:

{
  "native_distribution": {
    "joy": 0.62,
    "love": 0.18,
    "surprise": 0.10,
    "sadness": 0.05,
    "anger": 0.03,
    "fear": 0.02
  },
  "normalized_distribution": {
    "happy": 0.80,
    "surprise": 0.10,
    "sad": 0.05,
    "angry": 0.03,
    "fear": 0.02
  }
}

This gives the music layer a more stable control signal.

Lightweight fusion rule

A simple weighted late-fusion rule may be enough:

reliability = modality_weight * input_quality * calibrated_confidence

final_score[label] =
    sum_over_modalities(reliability * label_distribution[label])

Example pseudocode:

def fuse_streams(stream_results, project_labels):
    final_scores = {label: 0.0 for label in project_labels}
    total_reliability = 0.0
    debug_rows = []

    for stream in stream_results:
        if stream["status"] not in ["ok", "low_quality"]:
            debug_rows.append({
                "modality": stream["modality"],
                "used": False,
                "reason": stream["status"]
            })
            continue

        modality_weight = stream.get("modality_weight", 1.0)
        quality = stream["input_quality"]["score"]
        confidence = stream.get("calibrated_confidence", stream["confidence"])

        reliability = modality_weight * quality * confidence

        if stream["status"] == "low_quality":
            reliability *= 0.5

        for label in project_labels:
            final_scores[label] += reliability * stream["label_distribution"].get(label, 0.0)

        total_reliability += reliability

        debug_rows.append({
            "modality": stream["modality"],
            "used": True,
            "quality": quality,
            "confidence": confidence,
            "reliability": reliability,
            "top_label": stream["top_label"]
        })

    if total_reliability <= 0:
        return {
            "status": "fallback",
            "distribution": {"neutral": 1.0},
            "top_label": "neutral",
            "debug": debug_rows
        }

    distribution = {
        label: score / total_reliability
        for label, score in final_scores.items()
    }

    top_label = max(distribution, key=distribution.get)
    uncertainty = 1.0 - distribution[top_label]

    return {
        "status": "ok",
        "distribution": distribution,
        "top_label": top_label,
        "uncertainty": uncertainty,
        "debug": debug_rows
    }

This is intentionally simple. The benefit is that every design decision is visible:

Design question Where it appears
Should audio matter more than text? modality_weight
Was the audio too short/noisy? input_quality.score
Was the model uncertain? confidence
Was the stream unusable? status
Why did the final result choose this label? debug_rows

A debug table in the UI could be useful:

Modality Top label Confidence Quality Used? Reliability
Text happy 0.72 0.95 yes 0.684
Audio neutral 0.41 0.60 yes 0.246
Face surprise 0.68 0.30 yes, downweighted 0.204

This would make the walkthrough easier to reason about without adding much complexity.

Confidence vs input quality

I would keep confidence and input_quality separate.

They sound similar, but they mean different things.

Concept Meaning Example
confidence How confident the model appears to be The face CNN gives happy = 0.88
input_quality Whether the input was usable The face crop is blurry or dark
status Whether the stream should participate ok, low_quality, missing, failed
fallback_used Whether a fallback branch was used No face detected, so use text/audio only

Why this matters:

A model can be very confident on a bad input.

For example:

Case Model confidence Input quality Fusion behavior
Clear face, strong prediction high high use strongly
Blurry face, strong prediction high low downweight
Good audio, uncertain prediction low high use weakly
Missing text zero zero skip/fallback

Possible quality checks:

Modality Quality checks
Text empty text, very short text, language mismatch, repeated characters, low emotional content
Audio clip duration, silence ratio, volume/RMS, clipping, noise level, MFCC extraction success
Face face detected, number of faces, bounding-box size, blur, brightness, crop success

For text:

def text_quality(text):
    if not text or not text.strip():
        return 0.0, ["empty_text"]

    flags = []
    score = 1.0

    if len(text.strip()) < 8:
        score *= 0.5
        flags.append("very_short_text")

    if len(text.split()) < 3:
        score *= 0.7
        flags.append("few_words")

    return score, flags

For audio:

def audio_quality(y, sr):
    flags = []
    duration = len(y) / sr

    if duration < 1.0:
        flags.append("short_clip")

    rms = float((y ** 2).mean() ** 0.5)
    if rms < 0.01:
        flags.append("low_volume")

    silence_ratio = float((abs(y) < 0.005).mean())
    if silence_ratio > 0.7:
        flags.append("high_silence_ratio")

    score = 1.0
    if duration < 1.0:
        score *= 0.5
    if rms < 0.01:
        score *= 0.6
    if silence_ratio > 0.7:
        score *= 0.6

    return max(0.0, min(score, 1.0)), flags

For face:

def face_quality(face_box, frame_width, frame_height, blur_score=None):
    flags = []
    score = 1.0

    if face_box is None:
        return 0.0, ["no_face_detected"]

    x, y, w, h = face_box
    area_ratio = (w * h) / (frame_width * frame_height)

    if area_ratio < 0.02:
        score *= 0.5
        flags.append("small_face")

    if blur_score is not None and blur_score < 50:
        score *= 0.6
        flags.append("blurry_face")

    return max(0.0, min(score, 1.0)), flags

These are only heuristic examples, but even simple heuristics can make the system easier to debug.

Calibration note

One caution: softmax probability is not always a reliable confidence estimate.

A model may output:

{ "happy": 0.92 }

but that does not always mean it is correct 92% of the time.

A useful reference is On Calibration of Modern Neural Networks, which discusses how modern neural networks can be poorly calibrated and shows that temperature scaling is a simple post-hoc calibration method.

For this project, I would not overcomplicate the first version. A practical roadmap could be:

Level Method Cost
Basic Use max probability as confidence low
Better Track confidence vs actual correctness on validation samples medium
Better Apply temperature scaling per detector medium
Advanced Use uncertainty-aware fusion higher

Simple calibration workflow:

1. Keep a validation set for each detector.
2. Record predicted probabilities and true labels.
3. Fit a temperature parameter on validation logits.
4. Use calibrated probabilities in the fusion layer.

Pseudocode shape:

# conceptual only
calibrated_probs = softmax(logits / temperature)
confidence = max(calibrated_probs)

This matters because the fusion layer may use confidence as weight. If one model is systematically overconfident, it can dominate the fused result even when its input is weak.

So a good practical rule is:

Use confidence, but do not blindly trust it.
Combine it with input quality and fallback status.
Missing or weak modality behavior

For a real mobile demo, all modalities will not always be available.

Possible cases:

Situation Good behavior
Text only Run text stream, skip audio/face
Audio only Run audio stream, use audio reliability
Face only Run face stream, but maybe require good crop quality
Text + audio disagree Use reliability-weighted fusion
Face is dark/blurry Downweight face
Audio is too short Downweight or skip audio
All streams weak Neutral/safe fallback
One stream crashes Do not break the full flow

A useful behavior table:

status Meaning Fusion action
ok Stream is usable include normally
low_quality Stream is usable but weak include with penalty
missing No input skip
failed_preprocessing Preprocessing failed skip, log
model_error Inference failed skip, log
fallback_used Fallback branch used include cautiously

Example:

{
  "fusion_policy": {
    "ok": 1.0,
    "low_quality": 0.5,
    "missing": 0.0,
    "failed_preprocessing": 0.0,
    "model_error": 0.0,
    "fallback_used": 0.4
  }
}

This is useful for UX too. The app can display:

Detected emotion: Happy
Primary evidence: text
Audio was downweighted because the clip was short.
Face was skipped because no face was detected.

That kind of explanation can make a multimodal app feel more trustworthy.

Emotion-to-music mapping

For the music layer, I would add a small intermediate music-control state.

Instead of:

happy -> generate happy music

use:

emotion distribution
-> valence/arousal
-> musical controls
-> generated/recommended music

This helps because music does not map cleanly from one emotion label to one sound. A label like happy can mean bright, energetic, peaceful, playful, or triumphant depending on arousal and context.

A useful control table:

Emotional dimension Music parameter examples
Valence mode, harmony brightness, consonance/dissonance, melodic contour
Arousal tempo, rhythmic density, note density, loudness, attack strength
Dominance/tension register, dissonance, percussion intensity, harmonic tension
Uncertainty safer blend, neutral fallback, avoid extreme tempo/mode changes
Top label preset/template selection
Distribution blend between nearby presets

Example mapping:

Fused state Music-control state
high valence + high arousal faster tempo, major/bright mode, higher density
high valence + low arousal major/bright mode, slow tempo, soft texture
low valence + low arousal minor/darker mode, slow tempo, sparse density
low valence + high arousal minor/darker mode, faster tempo, tension/percussion
high uncertainty moderate tempo, neutral harmony, conservative recommendation

A compact JSON state:

{
  "music_control": {
    "valence": 0.72,
    "arousal": 0.64,
    "tempo_bpm": 126,
    "mode": "major",
    "note_density": "medium_high",
    "rhythm_activity": "medium_high",
    "register": "mid_high",
    "instrumentation": ["piano", "light_synth"],
    "recommendation_tags": ["uplifting", "bright", "energetic"]
  }
}

For a sad/low-arousal case:

{
  "music_control": {
    "valence": 0.22,
    "arousal": 0.28,
    "tempo_bpm": 72,
    "mode": "minor",
    "note_density": "low",
    "rhythm_activity": "low",
    "register": "mid_low",
    "instrumentation": ["soft_piano", "strings"],
    "recommendation_tags": ["calm", "melancholic", "slow"]
  }
}

Why this is useful:

Benefit Explanation
More controllable generation Music21-style symbolic generation can use explicit parameters.
Better recommendations The same state can be translated into search/recommendation hints.
Better UI explanation Users can see why the system chose a certain mood.
Better debugging If the output feels wrong, inspect the music-control state.
Provider independence The app is not locked to one recommendation API.

A relevant design reference is EmotionBox, which uses music elements such as pitch histogram and note density as emotion-control handles. The exact implementation does not need to be copied, but the design principle is useful: bridge emotion and music through controllable musical attributes.

Music21 / FluidSynth angle

Because the project uses Music21/FluidSynth-style symbolic generation, this intermediate control layer is especially natural.

Music21 is useful when the system wants to construct or manipulate symbolic musical objects such as notes, durations, streams, chords, and scores. See the music21 documentation.

A symbolic-control approach can expose parameters like:

Control Music21-style implementation idea
tempo set BPM / metronome mark
mode choose major/minor scale or pitch collection
note density choose number of notes per time window
register choose pitch range
rhythm activity choose duration patterns
chord brightness choose chord templates
instrumentation choose MIDI program / SoundFont instrument
texture single melody vs chords vs arpeggios

Example conceptual mapping:

def emotion_to_music_control(valence, arousal, uncertainty):
    if valence >= 0.6:
        mode = "major"
    else:
        mode = "minor"

    if arousal >= 0.7:
        tempo = 140
        density = "high"
    elif arousal >= 0.4:
        tempo = 110
        density = "medium"
    else:
        tempo = 75
        density = "low"

    if uncertainty >= 0.5:
        tempo = int((tempo + 100) / 2)
        density = "medium"

    return {
        "mode": mode,
        "tempo_bpm": tempo,
        "note_density": density
    }

Then the generator can translate this into actual notes/chords.

The value is not that every mapping is musically perfect. The value is that the mapping is inspectable.

If emotion detection is correct but the music feels wrong,
debug the music-control mapping.

That is much easier than debugging an opaque direct label-to-music function.

Spotify / recommendation layer note

If Spotify recommendations are part of the app, I would keep the internal music_control state independent from Spotify-specific fields.

Reason: third-party APIs change. Spotify announced Web API access changes in 2024 that affected new use cases for several endpoints/features, including Recommendations, Audio Features, and Audio Analysis: Spotify developer blog.

So I would avoid designing the project around:

emotion -> Spotify-specific target fields only

Instead, keep a provider-neutral state:

{
  "music_control": {
    "valence": 0.72,
    "arousal": 0.64,
    "tempo_range": [110, 140],
    "energy_hint": "medium_high",
    "mood_tags": ["uplifting", "bright", "energetic"]
  }
}

Then adapters can translate it:

music_control -> Music21 generation
music_control -> Spotify recommendation, where available
music_control -> local song database search
music_control -> future recommendation backend

This prevents the rest of the architecture from depending too strongly on one provider.

Evaluation / debugging checklist

For a project like this, I would evaluate more than classifier accuracy.

Useful checks:

Layer What to check
Text detector Does it handle short/ambiguous text?
Audio detector Does it fail gracefully on short/noisy audio?
Face detector Does it handle no-face / multiple-face / low-light cases?
Fusion layer Does it behave sensibly when streams disagree?
Music-control layer Does the control state match the fused emotion?
Generation layer Does the generated music reflect the intended controls?
Recommendation layer Do recommended tracks match the intended mood?
UX layer Can the user understand why the app chose that result?

A useful debug log per run:

{
  "run_id": "demo_001",
  "inputs": {
    "text_available": true,
    "audio_available": true,
    "face_available": false
  },
  "stream_results": [
    {
      "modality": "text",
      "top_label": "happy",
      "confidence": 0.72,
      "quality": 0.95,
      "used": true
    },
    {
      "modality": "audio",
      "top_label": "neutral",
      "confidence": 0.41,
      "quality": 0.60,
      "used": true
    },
    {
      "modality": "face",
      "status": "missing",
      "used": false
    }
  ],
  "fused_emotion": {
    "top_label": "happy",
    "uncertainty": 0.32
  },
  "music_control": {
    "valence": 0.70,
    "arousal": 0.58,
    "tempo_bpm": 118,
    "mode": "major"
  }
}

This kind of log helps answer:

Was the problem detection, fusion, mapping, generation, or recommendation?

That is the main operational benefit of the richer contract.

Possible UI layout for the walkthrough

For the Gradio/Space walkthrough, I would show the architecture in four visual blocks.

Block 1: Inputs
Text | Audio | Face

Block 2: Detectors
DistilBERT/text model | MFCC + CNN/LSTM | CNN face model

Block 3: Fusion / Handoff
label distribution | confidence | quality | fallback

Block 4: Music Output
music-control state | generated music | Spotify/recommendations

Possible table:

Stream Preprocessing Model Output Quality signal
Text clean text / tokenize NLP emotion classifier emotion distribution length / ambiguity
Audio MFCC / audio features CNN/LSTM emotion distribution duration / silence / volume
Face face crop / grayscale / resize CNN emotion distribution face detection / blur / lighting

Then:

Handoff field Why it matters
top_label easy UI display
label_distribution preserves ambiguity
confidence helps fusion weighting
input_quality prevents bad inputs dominating
status handles missing/failed streams
music_control bridges emotion to generated/recommended music

This would make the Space useful even if the heavy models are not all running inside the demo. It becomes a clear architecture walkthrough rather than just a feature list.

Practical roadmap

A practical roadmap could be:

Step Change Cost Benefit
1 Normalize emotion labels across text/audio/face low cleaner fusion
2 Return distributions instead of only top labels low preserves uncertainty
3 Add status and fallback handling low robust missing-modality behavior
4 Add input-quality heuristics low-medium prevents weak inputs dominating
5 Add weighted late fusion medium better multi-stream behavior
6 Add music-control state medium cleaner emotion-to-music mapping
7 Add debug table/logs medium easier troubleshooting
8 Calibrate confidence on validation data medium better reliability weighting
9 Try learned fusion later higher possible performance gains

A good “next PR” size might be only Steps 1–4:

standard output schema
+ quality flags
+ normalized labels
+ simple fusion debug table

Then later:

weighted fusion
+ valence/arousal mapping
+ music-control state

This is small enough to fit the current architecture but useful enough to make the pipeline more understandable.

References / useful reading

Some references that may be useful for the design direction:

Overall, I think the strongest next abstraction is:

parallel detectors
-> standardized stream result
-> quality/confidence-aware fusion
-> music-control state
-> generation/recommendation

That preserves the current architecture while making the multi-stream behavior easier to explain, test, debug, and extend.

4

Hey John, Thank you so much for taking the time to write out such a detailed breakdown. Honestly, I didn’t expect anyone to look at my architecture so deeply, and your feedback is incredibly helpful.

Right now, passing hard string labels like ‘happy’ or ‘sad’ makes the music transitions feel a bit robotic. Your suggestion to use a richer data contract with probability distributions, confidence scores, and quality flags makes so much sense, especially for real-world edge cases where the camera lighting is bad or the mic catches background noise.

Mapping everything into a valence/arousal layer before hitting Music21 and FluidSynth is exactly the right way to make the music sound fluid and natural instead of jumping abruptly between tracks.

I’m saving your flowchart for my next code refactor. I am definitely going to work on this and implement these improvements soon!