A signal problem before it is a model problem
Extracting a song's stems - isolating vocals, drums, bass, the rest - amounts to undoing a sum. The mix added sources together; the job is to recover the terms from the result alone. Nothing guarantees the decomposition is unique, and that is exactly what makes the problem interesting.
The raw material does not lend itself to this. One minute of CD-quality audio is more than two and a half million samples per channel: a sequence too long and too weakly structured to be handed to a network as is.
Going through the spectrum
Spectral analysis solves that representation problem. The short-time Fourier transform cuts the signal into overlapping windows and gives, for each one, how energy is distributed across frequency. What you get is an image - a spectrogram - with time on the x axis, frequency on the y axis and intensity as the value.
That change of representation has a direct consequence: the sources become visually separable. A voice, a snare and a bass line occupy neither the same bands nor the same temporal patterns. Separation turns back into a masking problem, which convolutional architectures are comfortable with.
Choosing the window is a trade-off you cannot dodge: a long window gives fine frequency resolution but smears transients; a short window does the opposite. Percussion and sustained harmonic material do not call for the same compromise.
Normalise before inferring
The files entering the pipeline have nothing in common: different sample rates, mono or stereo, levels ranging from a compressed master to a demo recorded far too quietly. A model trained on calibrated inputs degrades as soon as that calibration no longer holds.
So preprocessing levels everything first - resampling, channel handling, level normalisation - before any chunking. It is the least spectacular part of the pipeline, and the part reproducibility depends on.
Serving inference
The model in use is BS-Roformer, a transformer that splits the spectrum into bands and applies attention inside each one: it exploits precisely the frequency structure described above. A spectrogram is a dense tensor and source separation is heavy compute, so the GPU is not an optimisation but the condition for staying under an acceptable response time.
Serverless orchestration answers a different problem: the load is intermittent. Spikes when files arrive, nothing in between. A permanently reserved GPU would be expensive idle time. Inference therefore runs on RunPod workers sized on demand, zero active workers at rest, three at most, five seconds of idling before shutdown. Cost follows actual processing, at the price of a cold start you have to keep in check.
The rest of the chain is decoupled for the same reason: the API validates and stores the file, a separate worker picks up pending jobs and calls the GPU, the stems go back to storage. No component waits on another while holding a connection open, which is what makes the spikes absorbable.
What I take away
On a chain like this, the model is rarely the bottleneck. I/O, signal preparation and how the work is split are what decide the real response time.
And the choice of representation remains the most structural decision in the project: everything that follows - architecture, chunking, post-processing - flows from it.