How web interface labeling quality affects ML models predicting users’ subjective impressions

Abstract

Training data quality is widely recognized as the main pre-requisite for constructing successful Machine Learning (ML) models. However, the concrete aspects of the data quality vary for different domains and outcomes to be predicted by the models. In Human-Computer Interaction (HCI), the common knowledge is that labor-intensive manual labeling of graphical user interfaces (UIs), i.e. identification of visual elements in them, allows to predict users’ subjective impressions more accurately. At the same time, computer vision-based services for automated parametrization of UIs gain in popularity. In our paper, we describe an experimental study with over 200 participants and 1000 web UI screenshots, which were assessed on 3 subjective impressions scales: Complexity, Aesthetics and Orderliness. In order to compare the effects of metrics derived from manual vs. automated labeling, as well as of increased data quality in the manual labeling, we calculated in total 16 metrics subsequently used as factors in the predictive ML models. Our results suggest that Pearson correlation of input data quality and the models’ quality was highly significant and negative for Aesthetics (-0.768) and Orderliness (-0.644). Neither could we identify consistent advantages of the “manual” metrics over “automated” ones, except for the number of UI elements, in which the automated services were somehow lacking. Our conclusions regarding the labeling of the UIs and the applicability of the considered metrics might be of interest to both ML-HCI researchers and practicing UI/UX designers.