Investigation of the possibility of automatic data transfer between several data sources with flexible configuration

Abstract

The volumes of processed, analyzed and used data are increasing with the development of information technologies. In this regard, the issue of storing information is becoming more and more urgent.

There are many possibilities for transferring, loading and modifying data, from using handwritten commands to using off-the-shelf ETL systems (Extract, Transform, Load). However, there is often a problem of consistency, in which it is not always possible to constantly monitor the appearance of updated data and their timely transportation. One of the ways to solve this problem is to create a software tool for the automated transfer of data between several sources with the possibility of flexible configuration.

The developed software tool should support the transfer of data between several different sources and be able to work with different dialects - implementations of the SQL language in specific DBMS (MySQL, PostgreSQL, Oracle Database, MS SQL Server). The software is cross-platform; contains a built-in web user interface; contains a built-in database for storing system settings and the last data transferred in the system.

Using a software tool for automated data transfer between multiple sources will allow the user to set the exact time for copying data, the time interval through which new data will be transmitted until the transfer process or an active connection is closed. In addition, the possibility of personal copying of data from individual tables of databases of various implementations is provided, allowing it to be serviced in automatic mode with the publication of the results of operations.