Speaker-Discriminative Attractors for Robust Continuous Speech Separation

Continuous speech separation (CSS) aims to separate unbounded audio streams under low-latency, chunk-wise streaming constraints. It further requires the separated outputs to remain speaker-consistent over time despite dynamically changing speaker activity. Most existing approaches that achieve strong speaker-attributed performance on long-form recordings rely on modular pipelines that combine separation with external speaker embedding models and offline clustering, which precludes real-time deployment. In this paper, we propose a single-channel CSS frame- work that performs end-to-end speaker tracking within a unified streaming network without requiring any external components, by explicitly enforcing cross-chunk speaker consistency using speaker- discriminative attractors. Our approach adopts a two-stage training strategy. In the first stage, the model is trained on full utterances to learn attractor representations that jointly perform separation and speaker existence modeling. In the second stage, the attractor network is frozen and refined for streaming inference by introducing identity-aware cross-segment alignment. To achieve robust inter-chunk consistency, we introduce Cross-Segment Contrastive Regularization (CSCR), which injects speaker identity supervision by aligning chunk-level embeddings with persistent speaker prototypes stored in an external memory queue. We further propose the Relational Tracking Resolver (RTR), a relational assignment algorithm that robustly matches current speakers to registered identities based on relational geometry, preventing identity overwriting during silence and turn-taking. Experimental results on WSJ0-mix and LibriCSS demonstrate that the proposed framework significantly improves both separation quality and long-term permutation stability under low-latency streaming conditions.