The short version
Analysis of a track now runs on the background worker when it is uploaded, and the results are saved with the track. The song is no longer cut into equal chunks, keys are worked out per section, and chords are steadier across the whole track.
Shipped to production backend analysis version 4, the Song Structure display, and the existing-track refresh.
Not proven on real music every check so far used generated audio. Real-song accuracy is the open question.
Where the work stands
| Area | v1 (before 19 Oct) | v2 (now) | Status |
|---|---|---|---|
| Section count | Fixed at 4 | Set by song length, 2 to 12 sections | Built real music unverified |
| Section boundaries | Equal slices | Placed where the sound changes | Built real music unverified |
| Key per section | One guess per equal slice | One key per section, same-key neighbours joined, silence marked as no key | Built Synthetic check passed |
| Chords | Best guess per beat (flips on noise) | Smoothed sequence with a switch cost | Built real music unverified |
| Length analysed | First 120 seconds only | Whole track | Built |
| Display (Song Structure lane) | Section bands, key chips, chord lane (fed by the fixed-4 analysis) | Same display, now fed by the new section, key and chord analysis | Deployed browser check on v2 not run |
| Existing tracks | Never refreshed (stamp bug) | One free automatic refresh per track | Built |
| Worker cost per track | Not measured on a worker | Whole-track path, not measured on a worker either | Open |
How to read this report
- v1 is the code as it stood at commit
fc5a7093(18 Oct, 23:32), when the Song Structure lane first showed sections, keys and chords. - v2 is the code as it stands on this branch today (head
5df09d01). - Build ledger lists every commit that moved the work, so the next session can pick up from a hash, not from memory.
- Open items lists what is not proven, with what would prove it.
v1: the baseline (commit fc5a7093)
In v1, the Song Structure lane reused the existing free track analysis and drew coloured section bands, key chips and a chord lane. The analysis underneath was simple:
| Part | How v1 worked | Where (v1 code) |
|---|---|---|
| Sections | Fixed count of 4 equal-length spans, whatever the song | detect_per_section_keys(audio, sr, sections=4), afcut_engine/harmonic.py |
| Key per section | One key scored per equal span | same function |
| Chords | Best chord label per beat (argmax), no memory between beats | chord_timeline, _chord_events_argmax |
| Length | Only the first 120 seconds were analysed | MAX_ANALYSIS_SECONDS = 120 |
| Chord output | Capped at 200 entries (cap was not the problem) | max_chords default of 200 |
| Silence | Scored anyway, could be given a made-up key | no silence handling |
| Refresh | Old rows with heatmaps counted as current, so they never refreshed | stamp check in webapp/app.py |
What v1 got wrong (the reason for v2)
- A 3-minute song always became 4 slices, even when the real change was somewhere else.
- A single noisy beat could flip the chord label, so one held chord showed up as several.
- Anything after 2 minutes was never analysed, so long songs showed a truncated structure.
- Existing tracks with the old data never picked up any fix.
v1 display
The lane at webapp/static/js/song-structure.js showed what the worker returned. It did not compute anything in the browser. Numbers came from the saved analysis on the track.
v2 backend (analysis engine)
Analysis still runs on the auto-scaled worker, not on the API gateway. Nothing was added to request handlers. No new job type, no new charge, no database schema change.
| Stage | v1 | v2 |
|---|---|---|
| Section count | Fixed 4 (argument) | K = clamp(floor(duration / 40 + 0.5), 2, 12) |
| Boundaries | Equal spans | Novelty curve (16 s look-behind and look-ahead), greedy peak pick |
| Chroma | Per-span scorer | Pure numpy, streamed, grid of max(0.5 s, duration / 300); librosa not required |
| Key per section | Per equal span | One energy-weighted chroma per section, through the shared key ranking |
| Post-merge | None | Same-key neighbours merged; sections under 20 s folded into the left neighbour |
| Silence | Scored anyway | Key set to None (marked, not invented) |
| Tiling | Not guaranteed | Sections are contiguous, non-empty, and cover [0, duration] exactly |
| Chord labels | Per-beat argmax (old path, kept as use_decoder=False) | Per-window scores through a Viterbi decoder, switch cost TRANSITION_PENALTY = 0.12, explicit silence label N |
| Chord confidence | Raw | Margin-scaled |
| Length | 120 s | Whole track (MAX_ANALYSIS_SECONDS = None) |
| Chord cap | 200 (unchanged) | 200, stated as a bound, not a truncation |
| Analysis stamp | Older stamp, with a check that also accepted legacy rows | _ANALYSIS_VERSION_CURRENT = "4", exact equality |
| Legacy repair | None | One free refresh per legacy track, under the existing dedupe and repair cap |
Kept for comparison: chord_timeline(use_decoder=False) still runs the old per-beat path. Beat times are read-only input: the beat grid (SANCT) is never changed by this work.
v2 display (Song Structure lane)
| Display change | Commit |
|---|---|
| Sections, key chips and chord lane added to the full-song scout player (v1 display) | fc5a7093 |
| Analysis-version guard and cache-busters for the lane | 29df4aeb |
Lane opens from ?track= for both pack states | 9e22305d |
| Pack manifest carries the saved structure (sections, chords, key) | 0ee0165e |
| Legacy structure stays renderable during the refresh; v4 refresh polling | cf046e64, 923ef6c2 |
The browser only draws what the worker saved. Chord lane, key chips and section bands read from the same stored analysis.
Not verified a fresh browser pass on a real track in the Song Structure view. The last recorded browser pass was on v1.
v2 upload flow (touches the same path)
Uploads are the entry point for analysis, so the upload fixes from the same day are part of this path:
- Client-compress and import concurrency raised to 3 lanes (
8172a55e). - Rows claimed synchronously to stop duplicate upload POSTs (
6436b9fc). - Import meter and row percentage driven from one clamped byte source (
9ceadf4a).
Real-track check (generated fixture, not a real song)
| Measure | Result |
|---|---|
| Fixture | 180 s generated stereo audio |
| Sections | 5, keys C, A, F, G, D (about 36 s each); matches floor(180/40 + 0.5) = 5 |
| Chord entries | 86 |
| Wall clock (local machine, not a worker) | 113.9 s |
Python's round(4.5) is 4, so an earlier expectation was wrong. The code's floor(x + 0.5) gives 5 and is correct.
Build ledger (analysis and display)
| Date (UTC) | Commit | What it did |
|---|---|---|
| 18 Oct 23:32 | fc5a7093 | v1 display: sections, keys and chords in the full-song player (baseline) |
| 19 Oct 06:13 | 29df4aeb | Analysis-version guard, mirror-before-enqueue order, cache-busters |
| 19 Oct 09:30 | 9e22305d | Lane opens from ?track= for both pack states |
| 19 Oct 10:55 | 9bcb7684 | Library snapshot marked dirty on completion, so refreshed tracks rank first |
| 19 Oct 12:58 | 40d10fc7 | Stamp only when section keys and chords exist; empty rows refresh |
| 19 Oct 13:16 | 43b5eeef | Quote lane retries non-current section analysis |
| 19 Oct 16:49 | 6fd963a4 | Sections generated during manual pack processing |
| 19 Oct 19:36 | e982aa17 | Remove the silent 120 s analysis cap (whole track) |
| 19 Oct 20:09 | 40056e6e | Chord decoder with transition cost |
| 19 Oct 20:48 | 8706918f | Duration-derived section tiling (novelty curve, aggregated-chroma keys) and verifier |
| 19 Oct 21:03 | 1d4a76d3 | Stop over-splitting: same-key merge, 20 s minimum section length |
| 19 Oct 22:41 | cf046e64 | Analysis v4 stamp-equality refresh, free one-time legacy repair |
| 19 Oct 22:41 | 923ef6c2 | Poll predicate no longer pins v3; cache-buster; test stub |
| 19 Oct 22:50 | 9ceadf4a | Upload import meter from one clamped byte source |
| 20 Oct 00:11 | 4f6ecdd3 | Song analysis algorithm progress report (first HTML report) |
| 20 Oct | 5df09d01 | Merge of origin/main into the session branch (head of this report) |
Ledger source: git log on session/20261019-220540-cx. Commits outside the analysis and display path are not listed.
Receipts that record each decision
docs/work-dispositions/section-tiling.mdandsection-tiling-blocked-disk.mddocs/work-dispositions/harmonic-chord-decoder-r1.mddocs/work-dispositions/songchop-section-chord-upgrade-retro.md(lessons and friction)docs/reports/song-analysis-algorithm-progress.html(the earlier report this one extends)
Not proven yet
| Item | Why it matters | What would prove it |
|---|---|---|
| Real-music accuracy of sections | All checks used generated audio | A rights-safe real track set, checked by ear and against a reference |
| Real-music accuracy of chords | Same | Same set; check held chords stay one label |
| Browser display of v2 | Last browser pass was on v1 | Load a track in Song Structure, screenshot at desktop and phone width |
| Post-merge test rerun | Tests not rerun since the last merge | Run scripts/verify_section_tiling.py and scripts/verify_chord_decoder.py on the merged head |
| Worker cost | Whole track takes longer than 120 s | Measure job duration on an auto-scaled worker, then price it |
| Extra cuts | Boundary search can add cuts that are not key changes | Count false cuts on the real set; the merge pass reduces but does not remove them |
Cleanup (not blocking)
webapp/app.pyis already clean:app.py:32641callsdetect_per_section_keys(audio_data, sr)with nosections=4. The stalesections=4site istools/song_intel_demo.py:119(ignored argument, harmless). The othersections=4calls are intentional compatibility checks inscripts/verify_track12_harmonic.pyandscripts/verify_section_tiling.py.- Some older receipts still describe the chord decoder as unwired or unverified.
scripts/verify_track12_harmonic.pyexpected-count fix from the old fixed4.
Next steps (in order)
- Run the Song Structure view on a real track and screenshot it (display check).
- Measure worker time for a 3 to 5 minute real track (cost check).
- Build a small rights-safe real-music set and run the verifiers on it (accuracy check).
- Clean up the stale call sites once the above is in.
v3 plan: what to build next
Plan only Nothing in this tab is MEASURED
v3 is a proposal for the section boundary detector, per-section keys, chord decoding, the display, and the evidence needed to trust them. No v3 code has shipped. Every effort, cost and risk figure below is an ESTIMATE. No accuracy number is claimed, because no labelled data or evaluation harness exists yet.
(a) Inventory: what is live and what is legacy
| Path | file:symbol:line | Status | Note |
|---|---|---|---|
afcut_engine/harmonic.py | detect_per_section_keys :911 | LIVE | The only boundary detector: chroma cells, novelty curve, greedy peak pick of K-1 cuts, K = clamp(round(dur/40), 2, 12), same-key merge, sections under 20 s folded. The sections argument is deprecated and ignored. Returns [] for audio under 4 s. |
webapp/app.py | _build_bpm_key_analysis :32669, call at :32641 | LIVE | Calls detect_per_section_keys(audio_data, sr) inside _harmonic_layer (:32622). No sections=4. |
webapp/app.py | _ANALYSIS_VERSION_CURRENT :463 | LIVE | Value "4", exact-equality refresh gate. Any new segmenter needs a version bump. |
afcut_engine/harmonic.py | chord_timeline :1460 | LIVE | Default path is Viterbi via decode_chord_sequence (afcut_engine/chord_sequence.py:178). Reads beat_times read-only. Callers: app.py:32642, harmonic.py:1555, tools/song_intel_demo.py:120. |
afcut_engine/chord_sequence.py | TRANSITION_PENALTY :52 | LIVE | 0.12, a module constant, not a keyword argument. |
afcut_engine/harmonic.py | _chord_events_argmax :1420 | LEGACY | Per-beat argmax. Deliberately kept for use_decoder=False and the verifier --legacy path. Only caller: harmonic.py:1543. |
afcut_engine/harmonic.py | analyze_harmonics :1549 | LEGACY | Wrapper that passes sections=4 to an argument that is ignored. Re-exported by afcut_engine/detection.py:43,56. No repo caller found. |
tools/song_intel_demo.py | line 119 | LEGACY | Stale sections=4. Harmless. |
scripts/verify_track12_harmonic.py, scripts/verify_section_tiling.py | :359; :107, :258 | LEGACY | sections=4 kept on purpose as compatibility checks. |
shared/models.py | ProcessedSong :197, audio_hash :208, analysis JSONB :218 | LIVE | Cache key is audio_hash (unique, indexed). analysis holds per_section_keys, chord_timeline, segments, heatmaps, character, analysis_version. No sections table, no per-section confidence column, no grid-revision column. |
(b) Proposals, one row per aspect
| Aspect | What changes | Why | Acceptance check | Risk | Tag |
|---|---|---|---|---|---|
| Boundary detection | One detection path only. Calibrate against labelled data. Compare the current novelty curve with a self-similarity matrix and with SongFormer-style and allin1-style structure models, as candidates only. Each candidate: licence not checked, gated on a licence review. | Cuts are placed by a heuristic that has never been scored on real music. | On the labelled set, boundary F-measure from mir_eval.segment at a stated tolerance, reported with the baseline. Any candidate must beat the baseline by a margin fixed before the test. | Candidate models may carry licences that forbid this use. Extra false cuts. | ESTIMATE |
| Key per section | Keep the shared key ranking. Add a per-section confidence and an uncertain mark. | A wrong key on a short or ambiguous section is shown as fact today. | Key accuracy on the labelled set per section, with uncertain sections listed separately. | Modulations and borrowed chords get marked uncertain too often. | ESTIMATE |
| Chord decoding | HMM / Viterbi refinement. Chord-change regression test. | Keeps held chords as one label. The v2 synthetic chord regression FAILED (chords_distinct_in_path: 1), so the decoder is not yet proven to behave. | Chord-change regression test passes on synthetic and labelled audio before any display change. | Smoothing hides real short chord changes. | ESTIMATE |
| Silence and abstain | Per-section confidence. Sections that cannot be scored are marked uncertain, not given a key. | Silence and noise can still get an invented key. | Silent and noise-only fixtures return no key and an uncertain mark. | More sections marked uncertain than users want to see. | ESTIMATE |
| Display | Show per-section confidence and uncertain marks in the Song Structure lane. | The browser check on v2 has never been run. | Browser check on a real track at desktop width and at 390 px, with screenshots. | Mobile layout overflow with more labels. | ESTIMATE |
| Backend and worker cost | Measure job duration on an auto-scaled worker before any price is set. No price is set here. | Whole-track analysis is longer than the old 120 s path. Cost is unmeasured. | Measured job durations for a 3 to 5 minute real track on an auto-scaled worker, recorded in a receipt. | Cost guard priced from a local machine number. | ESTIMATE |
| Verification | A rights-safe real-music set, kept outside the repo, with a manifest. mir_eval metrics for boundaries, chords and keys. | There is no labelled data, no eval harness and no tests/ directory. mir_eval==0.8.2 is pinned in requirements.txt:58 but not used in code. | Harness exits non-zero when labels are missing. Metrics reported for baseline and candidate. | Set too small to say anything. | ESTIMATE |
| Data and schema | Store per-section confidence and the analysis version in the analysis JSON. Schema changes need a plan and a db-migration; none is done here. | Confidence has nowhere to live today. | Additive JSON fields, old rows still readable. Schema plan reviewed before any migration. | Version bump forces a refresh of every cached track. | ESTIMATE |
| Cleanup | Remove the stale sections=4 call in tools/song_intel_demo.py:119. Leave the intentional compatibility checks. | Misleading call sites. | Grep for sections=4 shows only the intentional checks. | Low. | ESTIMATE |
Nothing here is MEASURED. No accuracy figure is claimed for any aspect.
(c) Roadmap (song-structure ids)
tab-impl: i-verdict: decision summary.tab-impl: i-laws: SANCT and storage constraints.tab-impl: i-cost: CPU cost.tab-impl: i-road: phased roadmap. Phase 0: benchmark and ablation gate, own segmenter within about 10 points of HR3F. Phase 1: sections sidecar job, table, migration, touch points T1 to T4, GET route. Phase 2: bar-level chord Viterbi. Phase 3: per-section key and modulation. Phase 4: per-stem, later. Backfill only after the Phase 1 gate.tab-order: order-root: ship a thin display on a versioned, additive JSON contract. Sections are not ready for display until Phase 0.tab-evolution: e-v3: evidence-gated drift and meter, keyed on (audio_hash, grid_revision_hash, analyzer_version). Downbeat F within ±70 ms gate. NOT MEASURED.tab-evolution: e-ablation: v2 synthetic chord regression FAILED (chords_distinct_in_path: 1).tab-evolution: e-cost: CPU first, no GPU.tab-evolution: e-answer, e-boundaries, e-gates, e-priority, e-risks, e-unverified: decision, boundary, gate, priority, risk and unverified notes.
Ids only. Line numbers beyond the ids have not been rechecked.
(d) Ordered build plan
| Unit | Owned files | Check that can fail |
|---|---|---|
| U1 | scripts/eval_sections.py | Exits non-zero when labels are missing. |
| U2 | Rights-safe labelled set, kept outside the repo, plus a manifest | Manifest check fails on a missing or unlicensed track. |
| U3 | Baseline run of detect_per_section_keys through mir_eval.segment | Baseline score is recorded; a missing label set fails the run. |
| U4 | Candidate segmenter behind a flag, new module in afcut_engine/ | Test that beat_times is not mutated and beat cut points are unchanged. |
| U5 | Chord-change regression test | Must catch the e-ablation failure (chords_distinct_in_path: 1). |
| U6 | Per-section confidence and abstain fields in the analysis JSON; _ANALYSIS_VERSION_CURRENT bump plan | Old rows still read; version gate test. |
| U7 | Schema plan document, then db-migration (separate, NOT done here) | Plan reviewed before any migration. |
| U8 | Song Structure display | Browser check at desktop width and at 390 px. |
| U9 | Worker duration measurement on an auto-scaled worker, then cost-guard pricing | Measured durations exist before any price. |
| U10 | tools/song_intel_demo.py:119 | Grep shows no stale sections=4. |
(e) Hard constraints
- SANCT beat grid is sacred. Never alter beat cut points. No zero-crossing snapping on loops or segments. Sections are analysis metadata;
beat_timesis read-only. - The API gateway only enqueues. Processing runs on auto-scaled workers.
- No new charge and no schema change without a plan.
- Whole-track analysis stays cache-keyed by
audio_hash.
(f) Not verified
- No labelled data and no eval harness exist.
- Real-music segmenter accuracy is unmeasured.
- Worker job duration is unmeasured.
- The v2 display has never been browser-checked.
- Model licences are unchecked.
- Roadmap line numbers beyond the ids were not rechecked.
- No browser is installed in this environment, so no page rendering or nav click was tested.
v4 plan: labelling song sections the way competitors do, and better
Step 1 and first-pass Step 2 committed Not deployed, not wired into any job or worker Step 3 not built
VERIFIED-ON-PAGE ESTIMATE / UNVERIFIED PLAN INFERENCE VENDOR CLAIM
Competitors that publish anything label sections with a short fixed vocabulary (intro, verse, chorus, bridge, outro) over boundaries found by analysing the audio. None of the pages we could open publish how, and only one (rekordbox) names a sub-section event (Fill in). The method the research literature supports is repetition first: compare bars with each other to find which bars belong together (repetition), use novelty (sudden change) to place the boundary, and let repetition choose the label while novelty sets the boundary. We plan to read labels off the SANCT bar grid, label only above a confidence threshold (otherwise "Section" or "Variation"), and treat fills, turnarounds, buildups, breaks, risers and stops as deviations inside a repeating section. Status: Step 1 (a labelled test set and eval harness) is committed on session/20261020-014245-cx. A first-pass Step 2 segmenter (shared/section_segmenter.py, analysis-only, reads beat_times and never changes it) is committed as 065a8fc4 on session/20261020-014245-cx. Neither is deployed or wired into any job or worker. Step 3 (sub-section events) and a real-music labelled set are not built. The eval currently exits non-zero for the untuned segmenter. No accuracy is claimed.
(a) Competitor table
| Product | Technique family | Labels offered | Sub-section events | Source / tag |
|---|---|---|---|---|
| Moises Sections | Not published. Likely repetition-based: INFERENCE UNVERIFIED | Detects and isolates intros, verses, choruses, bridges (page says "such as"). Default labels intro, chorus, verse come from a search snippet of the help article only (help article bot-blocked, UNVERIFIED). Loops are synchronised to the downbeat; Premium loops sections across a library. | None stated. | moises.ai/newsroom/product-announcements/new-song-sections-feature/ VERIFIED-ON-PAGE. "Accurate and intelligent" is user impression only; no method, dataset or accuracy published. |
| rekordbox 7 (AlphaTheta / Pioneer DJ) Phrase Analysis | Not published. Guide says track structure is analysed and each part defined as a phrase, editable by the user. | Intro, Up, Down, Chorus, Bridge, Verse, Outro (rekordbox 7.0.7 lighting guide, section 2 terms). Separate Mood High / Mid / Low from tempo, rhythm, kick drum and sound density. | "Fill in": Phrase Edit guide section 1.3, detected at the end of Intro, Up and Chorus (up to 4 beats), display-only on the enlarged waveform. No turnaround, buildup, break, riser or stop stated. | cdn.rekordbox.com/files/20241216130922/rekordbox7.0.7_lighting_operation_guide_EN.pdf and cdn.rekordbox.com/files/20241203210634/rekordbox7.0.5_Phrase_Edit_operation_guide_EN.pdf VERIFIED-ON-PAGE. AlphaTheta help article bot-blocked UNVERIFIED; full Instruction Manual not opened. |
| allin1 (Kim and Nam, WASPAA 2023; arXiv 2307.16425; github.com/mir-aidj/all-in-one, MIT licence) | Learned neural model (dilated neighbourhood attention over source-separated spectrograms) doing beats, downbeats, boundaries and labels jointly. | Harmonix labels: intro, outro, break, bridge, inst, solo, verse, chorus (plus start and end markers). | "break" is a section label, not a deviation event. No fill, turnaround, buildup, riser or stop. | VERIFIED-ON-PAGE |
| SongFormer (arXiv 2510.02797; github.com/ASLP-lab/SongFormer) | Learned model on self-supervised embeddings (MuQ and MusicFM) at short and long windows. | intro, verse, chorus, bridge, inst, outro, silence, pre-chorus. | None. | Code licence CC-BY-4.0. Model and dataset terms NOT verified. VERIFIED-ON-PAGE for the rest. |
| Cyanite SegmentationV1 (docs.cyanite.ai/docs/guides/model-outputs/) | Not published. | Consecutive non-overlapping segments with start and end seconds. Segments carry NO section names. | None. | VERIFIED-ON-PAGE |
| Mixed In Key 11 | Not published. | Automatic cue points; marketing names intros, breakdowns, drops, mix-out sections; energy level per segment. "Up to eight" cue points and "bridges" are snippet-only UNVERIFIED. | None stated. | mixedinkey.com/workflows/how-to-use-mixed-in-key/ and /learn-more/ VERIFIED-ON-PAGE for broad claims. |
| Serato DJ Pro, Traktor Pro 4, Beatport DJ, Ableton Live 12, LALAL.AI, AudioShake | Not published. | None stated on the pages opened. Serato: beatgrid analysis and stems. Traktor: flexible beatgrids, hotcues, stem separation. Beatport DJ: a Cue button only. Ableton: none on the Live 12 page. LALAL.AI: stem and noise separation. AudioShake: docs landing page only. | None stated. | "None stated" on pages opened, not proof none exists. Serato support pages bot-blocked UNVERIFIED. Songstitch: the site found is a Last.fm collage tool, not a structure product, so not applicable. |
Moises chord note (not a table row): the chord finder page says chords are synced to audio. VENDOR CLAIM. Paraphrase only; no Moises text or UI assets are copied.
What this tells us
- No competitor page we could open publishes a method, dataset or accuracy.
- Of everything opened, only rekordbox names a sub-section event (Fill in), and only allin1 has a "break" label. None states turnaround, buildup, riser or stop detection. This is scoped to pages opened, not the whole market.
- Section names are vendor-defined, not a standard. Cyanite returns unnamed segments.
- Licences differ. allin1 is MIT. SongFormer code is CC-BY-4.0 with model and dataset terms unchecked. Both are candidates only, gated on licence review.
(b) Method: repetition first, novelty for boundaries
Card 1: Repetition
Paulus, Muller and Klapuri (ISMIR 2010, ismir2010.ismir.net/proceedings/ismir2010-107.pdf) identify three approaches: repetition-based, novelty-based and homogeneity-based. VERIFIED-ON-PAGE
Serra, Muller, Grosche and Arcos (AAAI 2012, ojs.aaai.org/index.php/AAAI/article/download/8328/8187) combine novelty, homogeneity and repetition. VERIFIED-ON-PAGE
PLAN Summarise audio per SANCT bar (chroma and harmony, drum-onset density, energy, timbre), build a bar-by-bar self-similarity matrix, and find stretches of bars that repeat.
Goto's RefraiD (IEEE TASLP 14(5), 2006, staff.aist.go.jp/m.goto/PAPER/IEEETASLP200609goto.pdf) uses a time-lag view of chroma similarity to find repeated sections and group them to pick the chorus. The paper reports the chorus found in 80 of 100 songs on one 2001 corpus. That is their number, not ours, and not evidence for our system. VERIFIED-ON-PAGE
SIA / SIATEC (Meredith, Lemstrom, Wiggins, J. New Music Research 31(4):321-345, 2002; author draft www.titanmusic.com/papers/public/siajnmr_submit_2.pdf) discover maximal repeated patterns in point-set (symbolic) data. For audio it is an idea (repeated-pattern discovery over detected onsets or notes), not a drop-in method. INFERENCE
Card 2: Novelty
Foote's 2000 novelty approach (ICME 2000, pp. 452-455) slides a checkerboard-style kernel along the diagonal of a self-similarity matrix to get a novelty curve. Peaks mark boundaries.
UNVERIFIED primary Foote's own paper was NOT opened (bot-blocked). This description comes from Paulus et al. section 5.1 and Serra et al., which are secondary sources.
Reconciliation rule
Repetition assigns the label: which bars belong together, and which group is chorus-like. Novelty sets the boundary: the strongest novelty peak within one bar of the repeat edge, on the SANCT bar grid, never off-grid. PLAN ESTIMATE thresholds, set on the labelled set.
- Boundary: if the repeat edge and the novelty peak differ by 1 bar or less, use the peak bar. If they differ by more than 1 bar, keep the repetition edge only when the repeat is strong; otherwise emit a boundary labelled "Section".
- Label: a novelty boundary with no repeat evidence gets "Section", never a guessed name.
- Repeat without a peak: a repeat with no novelty peak stays as "Variation" inside the parent section, not as a new section.
INFERENCE Moises' method is likely repetition-based. No Moises page says so, and this is UNVERIFIED.
Illustration only. No data claims. Fill marker shown inside a Chorus bar.
Fill marker: a sub-section event sitting inside the parent Chorus bar, not a new section.
(c) Taxonomy
| Label | Meaning | Detection signal | Confidence rule |
|---|---|---|---|
| Intro | Opening before the first strong repeating group. Sparse or building, once only, at track start. | Stretch before the first repeat group begins. | Label only if the position rule clears a threshold set on the labelled set; otherwise show "Section". ESTIMATE |
| Verse | Repeating group with changing content, lower energy than the chorus. Recurs 2 or more times. | Repeat score above threshold, energy below the chorus group. | Repeat score and position both clear thresholds; otherwise "Section". ESTIMATE |
| Pre-chorus | Short stretch of 2 to 8 bars just before a chorus group, rising density, recurs before each chorus. | Short segment directly before a chorus group, density rising. | Needs the chorus label first and a repeat before each chorus. ESTIMATE |
| Chorus | Most-repeated, highest-energy group. Same harmony each time. | Largest repeat group with the highest energy. | Needs 2 or more occurrences; otherwise "Section". ESTIMATE |
| Bridge | Contrasting once-only stretch with different harmony, usually late. Low repetition. | Low repeat score, late in the track, harmony differs from the chorus. | Label only when both repeat and position rules clear; otherwise "Section". ESTIMATE |
| Breakdown | Long sparse stretch where drums drop out or thin, bar-aligned, then return. | Long low-energy stretch on bar lines. | Needs a return on a bar line; allin1 "inst" and "solo" folded in here or into "Section" until evidence. ESTIMATE |
| Outro | Closing stretch after the last repeating group. Thins, fades, or repeats the intro. | Stretch after the last repeat group. | Label only if at track end; otherwise "Section". ESTIMATE |
| Variation | Repeat of an earlier section with a small difference, such as a changed last bar or an added layer. Shown as a variant of the parent. | Repeat with a partial match to the parent group. | Always shown as a variant of the parent, never as a new name. ESTIMATE |
Thresholds TBD (ESTIMATE). rekordbox Up and Down are DJ-oriented; ours are song-oriented.
(d) Sub-section events
| Event | What it is | Detection signal (deviation inside a repeating context) | Reliability | Failure mode |
|---|---|---|---|---|
| Fill | Short drum or percussion variation at the end of a phrase, up to about 1 bar. rekordbox "Fill in", up to 4 beats, display-only. | Onset-density spike in the drum band in the last quarter to 1 bar, versus the same bars in earlier repeats. | ESTIMATE medium on drum-forward music | One-off variations, rare short events, low sample counts (needs 2 or more earlier repeats). |
| Turnaround | Non-repeating 1 to 2 bars returning to the pattern, at a phrase end. | Bar-level self-similarity dips for 1 to 2 bars, and the next bar matches the earlier pattern again. | ESTIMATE low to medium | Confused with a real section change if longer. Invisible if the phrase never repeats. |
| Buildup | Rising density or energy into a boundary, 2 to 8 bars before a chorus or drop. | Monotone rise in onset density, energy or brightness over the bars before a novelty peak. | ESTIMATE medium | Slow natural crescendos. Tempo-free intros. |
| Break | Sparse bar-aligned drum drop: kick and snare disappear for whole bars, then return on a bar line. | Drum-band energy below a fraction of the section median for 1 or more whole bars, returning on a bar line. | ESTIMATE medium | Tracks with very sparse drums throughout. A drum stem could help. |
| Riser | Rising noise or synth sweep, 1 to 8 bars into a drop. | Rising high-frequency noise energy or spectral centroid over bars with few new onsets. | ESTIMATE low to medium | Confused with reverse cymbals or vocal swells. |
| Stop | All instruments cut for up to a bar, then resume. | Broadband energy below a floor on or near a bar line. | ESTIMATE medium on clean tracks, low on live or reverb-heavy audio | Reverb tails, fade-outs, real silence at track end. |
All reliability entries are ESTIMATE. No per-event precision or recall exists until measured on the labelled set (see (e)). Events are markers on the parent section, never new sections, and only when there are 2 or more earlier repeats to compare.
(e) What we build better
Step 1: Measure first
- Outcome: a labelled test set with a manifest, plus an eval script in
tests/segmenter/. The set built is SYNTHETIC: six loop/section forms known by construction, generated at eval time bytests/segmenter/synth.py, nothing stored, and scored. Six in-repo demo tracks (3 CC BY 4.0, 3 US public domain pre-1926) are listed with licences but are UNLABELLED and NOT scored. This departs from the earlier plan to keep the set outside the repo. Real-music scored coverage is zero, and six items is too small to say anything about accuracy. Boundary scoring is a pure-numpy greedy matcher at a 0.5 s window (a scratchmir_eval0.8.2 install was used for cross-checks; it is not in the default environment). The harness implements boundary P/R/F and period accuracy only. PLAN Label and clustering metrics (pairwise, vmeasure, nce) and per-event precision and recall for the six events are NOT built. VERIFIED-ON-PAGE metric names: detection, deviation, pairwise, rand_index, ari, nce, mutual_information, vmeasure (mir-eval.readthedocs.io/latest/api/segment.html). - Acceptance: the validator exits non-zero on a missing file, sha mismatch, bad licence or missing labels (
tests/segmenter/test_manifest.py); the current novelty detector's baseline is recorded; no accuracy number is published before this exists. - Risk: a set too small to say anything.
v4 segmenter: gate FAILED — not accepted
Gate FAILED Synthetic only In-sample
The v4 segmenter fails its acceptance gate. It is not accepted and must not be described as accepted. The margin over the 4-bar baseline is 0.093, below the required 0.1.
Measured on the synthetic set only: 6 scored items, known by construction, in-sample. Real-music coverage is zero: no real-music track is scored, the demo tracks are unlabelled, and there is no real-music accuracy claim.
Printed result (eval run 2026-10-20, branch session/20261020-014245-cx, HEAD eed746e2)
| Predictor | Macro P | Macro R | Macro F (0.5 s) |
|---|---|---|---|
| v4 segmenter | 1.000 | 0.737 | 0.778 |
| Trivial every-8-bars | 1.000 | 0.515 | 0.603 |
| Trivial every-4-bars | 0.810 | 0.768 | 0.685 |
| Novelty-only | 0.167 | 0.015 | 0.028 |
| Empty | 0.000 | 0.000 | 0.000 |
Period accuracy: 1.000 (6 of 6).
Gate checks
| Setting / check | Value | Result |
|---|---|---|
| MARGIN_F | 0.1 | threshold |
| MIN_SEGMENTER_F | 0.6 | threshold (v4 0.778 passes) |
| MIN_PERIOD_ACCURACY | 0.66 | threshold (1.000 passes) |
| MIN_TRACK_F | 0.0 | threshold (every track passes) |
| margin_vs_trivial4_F: 0.778 − 0.685 = 0.093 | 0.093 < 0.1 | FAILED CHECK |
The checks that now pass: segmenter F ≥ 0.6, period accuracy ≥ 0.66, and the per-track floor. The margin check against the 8-bar baseline (0.778 − 0.603 = 0.175) also passes. Only the 4-bar margin fails, and that single failure is enough to reject the gate. Exit status: 1.
Per-track boundary F (0.5 s)
| Track | Reference boundaries | Segmenter F |
|---|---|---|
| syn-loop4-120 | 7 | 1.000 |
| syn-loop4-noisy-120 | 7 | 1.000 |
| syn-loop2-90 | 7 | 1.000 |
| syn-aaba-128 | 3 | 1.000 |
| syn-abab-100 | 3 | 0.500 |
| syn-distinct-110 | 11 | 0.167 |
Causes of the misses
- syn-abab-100 (F 0.500): the segmenter misses boundaries at bars 8 and 24. The only repeat evidence is a single lag-16 repeat stripe over bars 0–16. Novelty at bar 8 is 0.12, against a threshold of 0.279, so no novelty peak is placed there. A lag-1 continuity-break idea was tried and refuted; it is not in the code.
- syn-distinct-110 (F 0.167): 1 of 11 reference boundaries is recovered. Bar 6 has novelty 0.28, which clears the 0.21 threshold, but it is suppressed by NOVELTY_MIN_DISTANCE=2 because bar 7 (novelty 0.39) is adjacent.
What changed in v4
- New module
shared/section_segmenter_v4.py. The originalshared/section_segmenter.pyis unchanged. - 32-bar trailing-bar convention; greedy clusters; novelty peaks are used only when no repeat period is found.
- Segmenter tuning is stopped. No further code changes are part of this result.
Tests and cross-checks
- Tests:
tests/segmenter/test_segmenter_v4.py11/11 pass (plain python; v4 copies of the earlier tests). Exit 0. - Boundary metric: greedy one-to-one boundary matcher, tolerance 0.5 s. It can undercount relative to mir_eval on 3 of 300 random stress pairs (random#004, #007, #084; max |dF| 0.222). The greedy matcher is the side that undercounts, so its boundary F is a lower bound on mir_eval's.
- mir_eval status: unreconciled. mir_eval 0.8.2 (scratch install, outside the repo), re-run: 3 of 300 random stress pairs disagree with the greedy matcher, max |dF| 0.222. The earlier "24/24 manifest comparisons equivalent" figure is not reconciled against v4 and is not cited as agreement.
- The cross-check script prints a DISAGREES line because of those three stress pairs. The manifest comparisons are not affected.
Limits and not done
- Synthetic only: 6 items, in-sample, zero real-music tracks scored. No real-music accuracy claim.
- The demo tracks are unlabelled and not scored.
- Not done: browser pass (no headless browser available); real-music scoring.
Step 2: Repetition-first labelling on the SANCT grid
- Outcome: per-bar features, self-similarity matrix, repeated-group discovery, novelty peaks, the reconciliation rule from (b), and the taxonomy with abstain ("Section" or "Variation"). Labels are read off the SANCT grid. Start and end are existing bar boundaries;
beat_timesis read-only; cut points never move; no zero-crossing snapping on loops or segments. - Acceptance: a test that
beat_timesand all cut points are bit-identical before and after; boundary F versus the baseline at a margin fixed before the test (label metrics NOT built); abstain rate reported. - Risk: candidate learned models (allin1 MIT, SongFormer CC-BY-4.0 code) are comparison baselines only, gated on licence and model-terms review.
Step 3: Sub-section events as deviations
- Outcome: fill, turnaround, buildup, break, riser and stop detectors run only inside sections with 2 or more repeats. Output as markers with a confidence.
- Acceptance: per-event precision and recall on the labelled set, reported separately; events below the confidence threshold not shown.
- Risk: false events on sparse or reverb-heavy tracks.
Processing
The API gateway only enqueues. The work runs as a queued job on auto-scaled workers. Measure job duration on a worker before any price is set. Additive JSON fields and an analysis-version bump plan. No schema change without a plan. PLAN
(f) Visual pass
The layout has a tag legend at the top. Tables are wrapped in a horizontal-scroll safety net, and at 390 px they stack as cards with a label on each cell. Not checked in a browser: no headless browser is installed, so the layout and nav click were not tested.
(g) Not verified
- Moises accuracy, method and dataset: none published. The Moises help article is bot-blocked; snippet only.
- rekordbox: AlphaTheta help article bot-blocked; full Instruction Manual not opened.
- Foote 2000 primary not opened (blocked on CiteSeerX, IEEE and Semantic Scholar).
- Meredith et al. publisher page bot-walled; author draft read.
- Goto ICASSP 2003 version not fetched (2006 journal read).
- Muller and Kurth lag-matrix primary not fetched.
- mir_eval ISMIR 2014 paper not fetched.
- Serato support pages bot-blocked.
- Mixed In Key "up to eight cue points" and "bridges": snippet-only.
- Traktor, Ableton, LALAL.AI, AudioShake and Beatport DJ: checked on landing or feature pages only. AudioShake per-model pages not enumerated.
- Songstitch: not applicable.
- SongFormer model and dataset licence not verified.
- Whether rekordbox or Moises use deep learning is unverified.
- Every reliability figure and threshold is an ESTIMATE.
- No headless browser was available, so layout and nav click were not tested.
- No real-music accuracy coverage: zero scored real tracks.
- Boundary metric: greedy one-to-one boundary matcher, tolerance 0.5 s. It can undercount relative to mir_eval on 3 of 300 random stress pairs (random#004, #007, #084; max |dF| 0.222); which side undercounts on those pairs is a hypothesis, not confirmed. mir_eval status: unreconciled. The mir_eval 0.8.2 re-run (scratch install) reproduced 3 of 300 disagreements and max |dF| 0.222; the earlier "24/24" agreement figure is not reconciled against v4 and is not cited as agreement.
- pytest is not installed or run; tests were run with plain python.
- Real music untested: no real track is scored, and the segmenter output on the unlabelled demo tracks is not evaluated.
- Eval is thin. Reproduced: trivial-4bar F 0.685 against the segmenter at 0.778 (margin 0.093, below 0.1), so the gate fails on the margin check.
- Earlier unsourced research notes (e.g. HMM family) are not relied on.
Next detector: harmony, beat-guided sections and lightweight tagging
New research · 20 October 2026. Current source uses the beat grid for named sections, but a separate time-based detector for key regions. Chords have smoothing, not a local-key agreement mechanism. The report explains the mismatch and proposes a resource-bounded upgrade.
Open the full report on its own page — current wiring, weighted voting, libraries, vocal detection, costs and acceptance checks. No detector changes made.
Earlier tabs are historical snapshots. This new source audit corrects their now-stale “not wired” status without changing the recorded results or waiving failed gates.