Repository navigation
(Invalid) Overfitting on Seed-TTS Test Set ZH subset? #9
Description
【重要说明】我在下方最初发表的评论包含不恰当、情绪化、基于误解的指责，并不能代表事实。我保留原文，是为了公开承担错误、接受批评，并提醒自己不要再犯同样的错误。请读者不要将下方原始言论视为真实或可靠的信息。对于作者及项目组，我已在后续评论中正式致歉。
[Important Notice] The original comment below contains inappropriate, emotionally driven, and factually incorrect accusations. I am keeping the original text visible to take responsibility for my mistake, accept all criticism, and remind myself not to repeat such behavior. Please do NOT consider the original statements as accurate or valid. I have issued a formal apology to the authors and maintainers in a later comment.
The paper has presented very impressive evaluation results on its selected test set, which is the ZH subset of Seed-TTS Test Set. As the attached screenshot from paper, in zero-shot voice conversion task with unseen speakers as reference, Mean-VC (14M) has far surpassed baseline systems, which are Seed-VC tiny (25M) and StreamVoice (104M) respectively, in both CER and SSIM.
However, this result is clearly disobeying the scaling law of zero-shot TTS/VC systems, for which larger in parameter size means more timbre diversity is stored in the model as part of its learned knowledge. To dig in more, I ran a evaluation with the released checkpoint of Mean-VC, with default inference setup of 2 NFEs, 200ms chunk size. The evaluation was conducted on both EN and ZH subsets. For fair comparison, I also introduced Vevo and Seed-VC small as baselines. Both of these baseline models are trained on Emilia dataset, which is consistent with Mean-VC as described in its paper. This means, in theory, the three models mentioned above should have a similar distribution of timbre diversity and language coverage.
The table screenshot presented below is the results I obtained:
It is impressive that Mean-VC has achieved very low CER on ZH subset (3.78) with it's compact Fast U2++ content encoder. Although the WER on EN subset (19.65) is quite high, it's understandable as the paper said this Fast U2++ was trained on WenetSpeech only, so it should function well when source speech is Chinese.
However, there is one suspicious fact regarding SSIM metrics presented in the table. Since the three models should have similar distribution on timbre diversity, why does Mean-VC have nearly the highest SSIM (0.73) on ZH subset, while achieving a very poor SSIM (0.4) on EN subset? Comparing to baseline systems, Mean-VC's speaker similarity performance seems strongly biased towards the distribution of ZH subset.
It is hard to believe if the Mean-VC checkpoint participated in this test is the fresh one pretrained on Emilia Dataset only, as there seems no way to explain why it's speaker distribution is so different comparing to other baselines trained on Emilia Dataset. Hence, there exist a possibility that the checkpoint released has gone through some postprocessing or post training to have a nice result on the desired test set.
I hope the author could help give an explanation on the reason behind. Thank you🙏
Activity
yaoxunji commentedon Nov 19, 2025
On the claimed “scaling law” for zero-shot TTS/VC systems.
The assumption that “larger parameter size necessarily leads to better timbre diversity” is generally discussed in the context of large generative speech models. Such scaling behavior has been observed between models on the scale of 1B to 7B or more parameters, but it has not been established for small VC models in the 10M–30M range. To the best of my knowledge, there is no empirical or theoretical evidence showing that a 25M-parameter VC model must outperform a 14M-parameter one. Previous VC works—and even some speech-enhancement studies using parameter-sharing strategies—show that model size alone is not a reliable indicator of performance in this regime. Therefore, the underlying assumption of your argument does not hold.
On the CER/WER discrepancy of Mean-VC and the use of Fast U2++.
It is reasonable that Mean-VC obtains low CER on the Chinese (ZH) subset, given that its Fast U2++ content encoder was trained solely on WenetSpeech. In addition, Mean-VC also trained on the Emilia subset rather than the full corpus, which further limits its cross-lingual robustness. When such a content encoder is applied to English inputs, it may cause substantial spectral distortions during VC inference. These distortions directly affect downstream ASV-based metrics, which are highly sensitive to spectral quality. This could explain the unusually poor ASV results. I would encourage releasing the corresponding English converted samples for inspection, as the spectrograms would likely reveal the degradation.
Given that the argument is based on an unverified assumption and incomplete experimental results, the accusation regarding the integrity of our evaluation dataset or training process is unwarranted. Academic discussion should be grounded in validated reasoning and reproducible analysis. I respectfully request that critiques remain focused on methodological and empirical aspects rather than speculative claims. @Plachtaa
NZqian commentedon Nov 20, 2025
Only a dirty mind sees dirt everywhere.
NZqian commentedon Nov 20, 2025
@Plachtaa
Plachtaa commentedon Nov 20, 2025
@yaoxunji @NZqian @Bing0cv @ASLP-lab
I would like to offer my sincere apology for the remarks I made in this issue.
Upon reflection, I realize that my comments stemmed from impulsiveness and jealousy rather than a measured, constructive dialogue.
I fully accept responsibility for the improper conduct in terms of academic professionalism.
I admit that:
At the same time, I sincerely admire the research achievements and open-source contribution you have made with the MeanVC project.
I would like to take this opportunity to correct my words, and express my appreciation to you and your team: your exploration in zero-shot voice conversion, lightweight streaming models, is a precious advance in the field of speech processing.
Once again, I apologize for my inappropriate behavior at that time.
If you are willing, I am ready to retract or revise my remarks in that issue, and cooperate with you/the community in issuing a further public statement.
You reserves the right to keep this issue open, and I and willing to take all accusations and criticisms regarding any inappropriate remarks I may make. 🙏
对于我在此问题上发表的言论，我深表歉意。
经过反思，我意识到我的言论源于冲动和嫉妒，而非经过深思熟虑的建设性意见。
我完全接受我在学术职业操守方面不当行为的责任。
我承认：
我未能对您和您的团队所付出的努力给予应有的尊重。
我未能以客观、建设性的角度看待此事。
我的行为可能给您、您的团队以及整个社区带来了不便或负面影响。
同时，我由衷地钦佩您在 MeanVC 项目上所取得的研究成果和对开源社区的贡献。
我想借此机会纠正我的言论，并向您和您的团队表达我的感激之情：您在零样本语音转换和轻量级流式模型方面的贡献，是语音处理领域的一项宝贵进步。
再次为我当时的不当行为致歉。
如果您愿意，我愿意撤回或修改我在该问题上的言论，并与您/社区合作发表进一步的公开声明。
您可以继续保持此issue open，我愿意接受后续所有针对我本人以及我所发表的不当言论的指责和批评。🙏
[-]Overfitting on Seed-TTS Test Set ZH subset?[/-][+](Invalid) Overfitting on Seed-TTS Test Set ZH subset?[/+]