End-To-End Speaker Segmentation for Overlap-Aware Resegmentation

Speaker segmentation consists in partitioning a conversation between one or\nmore speakers into speaker turns. Usually addressed as the late combination of\nthree sub-tasks (voice activity detection, speaker change detection, and\noverlapped speech detection), we propose to train an end-to-end segmentation\nmodel that does it directly. Inspired by the original end-to-end neural speaker\ndiarization approach (EEND), the task is modeled as a multi-label\nclassification problem using permutation-invariant training. The main\ndifference is that our model operates on short audio chunks (5 seconds) but at\na much higher temporal resolution (every 16ms). Experiments on multiple speaker\ndiarization datasets conclude that our model can be used with great success on\nboth voice activity detection and overlapped speech detection. Our proposed\nmodel can also be used as a post-processing step, to detect and correctly\nassign overlapped speech regions. Relative diarization error rate improvement\nover the best considered baseline (VBx) reaches 17% on AMI, 13% on DIHARD 3,\nand 13% on VoxConverse.\n