Accuracy analysis of machine learning models using vectorization methods for heterogeneous text data classification tasks

Abstract

This paper investigates the problem of natural language processing using machine learning techniques, in particular, classification of unstructured heterogeneous text data sets. The paper presents a comparative analysis of some relevant and widely used methods and teacher-assisted machine learning models used for multi-class classification on heterogeneous textual data sources using different feature extraction methods. The dependence of the accuracy of class prediction by classifier models on the quality of the text data corpora used in this paper, applying different vectorization methods on the processed set of source data, is considered. Based on this analysis, a generalized scheme of the software functioning, which implements the algorithm for constructing a model of classification of unstructured texts, in the form of a pipeline for processing text corpus and control of machine learning models is proposed. During the experiment, it was demonstrated that for corpora with different quality of initial text data, the accuracy of classifier predictions differed. This circumstance manifested itself in the fact that the classifiers have lower performance on the corpus of texts of musical compositions and high on the texts of news summaries. It is shown that under certain conditions, the use of solutions to improve the quality of classification, such as stacking and adding additional features of classification, can lead not to improvement, but on the contrary to the deterioration of the results of class prediction, which, ultimately, can have a negative impact on the final accuracy of the obtained model results.