Study of the collapse of language models in medical applications during recursive and cross-training on artificial data

Abstract

The purpose of this article is to study the phenomenon of collapse of language models when implementing recursive and cross-sectional approaches to training models of the next generations in ultrasound diagnosis and treatment of the thyroid gland. In the first approach, each new model is trained exclusively on data generated by the previous version of the model, allowing the accumulation of systematic errors and degradation of data diversity to be examined. In the second, data generated by one model is used to train another model, which minimizes the impact of accumulated errors and allows a wider range of information to be stored. In the experiments conducted, Mistral and LLaMA models are trained and changes in data distribution are analyzed using the KL-distance metric, which evaluates the differences between the original data distribution and the data generated by the models. The results show that recursive learning causes a significant reduction in the range of generated text, especially for the LLaMA model, while cross-training exhibits greater resistance to collapse, providing more stable data diversity. The architectural differences of the models are considered, such as optimization of attention and the ability to model long-term dependencies that affect learning outcomes. The influence of different training methods on the ability of medical language models to preserve the variety and quality of the generated text is analyzed.