Data version control

Data version control is a method of working with data sets. It is similar to the version control systems used in traditional software development, but is optimized to allow better processing of data and collaboration in the context of data analytics, research, and any other form of data analysis. Data version control may also include specific features and configurations designed to facilitate work with large data sets and data lakes.^[1]

As early as 1985, researchers recognized the need for defining timing attributes in database tables, which would be necessary for tracking changes to databases.^[2] This research continued into the 1990s, and the theory was formalized into practical methods for managing data in relational databases,^[3] providing some of the foundational concepts for what would later become data version control.

In the early 2010s the size of data sets was rapidly expanding, and relational databases were no longer sufficient to manage the amounts of data organizations were accumulating. The rise of the Apache Hadoop eco system, with HDFS as a storage layer, and later object storage had become dominant in big data operations.^[4] Research into data management tools and data version control systems increased sharply, along with demand for such tools from both academia and the private and public sectors.^[5]

The first versioned database was proposed in 2012 for the SciDB database, and demonstrated it was possible to create chains and trees of different versions of the database while decreasing both the overall storage size and access speeds associated with previous methods.^[6] In 2014, a proposal was made to generalize these principles into a platform that could be used for any application.^[7]

In 2016, a prototype for a data version control system was developed during a Kaggle competition. This software was later used internally at an AI firm, and eventually spun off as a startup.^[8] Since then, a number of data version control systems, both open and closed source, have been developed and offered commercially,^[9] with a subset dedicated specifically to machine learning.^[10]

A wide range of scientific disciplines have adopted automated analysis of large quantities of data, including astrophysics, seismology, biology and medicine, social sciences and economics, and many other fields. The principle of reproducibility is an important aspect of formalizing findings in scientific disciplines, and in the context of data science presents a number of challenges. Most datasets are constantly changing, whether due to the addition of more data or changes in the structure and format of the data, and small changes can have significant effects on the outcome of experiments. Data version control allows for recording the exact state of data sets at a particular moment of time, making it easier to reproduce and understand experimental outcomes.^[11] If data practitioners can only know the present state of the data, they may run into a number of challenges such as difficulties in problem debugging or complying with data audits.

Data version control is sometimes used in testing and development of applications that interact with large quantities of data. Some data version control tools allow users to create replicas of their production environment for testing purposes. This approach allows them to test data integration processes such as extract, transform and load (ETL) and understand the changes made to data without having a negative impact on the consumers of the production data.

In the context of machine learning, data version control can be used for optimizing the performance of models. It can allow automating the process of analyzing outcomes with different versions of a data set to continuously improve performance.^[12] It is possible that open source data version control software could eliminate the need for proprietary AI platforms by extending tools like Git and CI/CD for use by machine learning engineers.^[13] Many open-source solutions build on Git-like semantics to provide these capabilities, as Git itself was designed for small text files and doesn't support typical machine learning datasets, which are very large.

CI/CD methodologies can be applied to datasets using data version control.^[14] Version control enables users to integrate with automation servers that allow establishing a CI/CD process for data. By adding testing platforms to the process, they can guarantee high quality of the data product. In this scenario, teams execute Continuous Integration (CI) tests on data and set checks in place to ensure the data is promoted to production only all the set data quality and data governance criteria are met.

To experiment on a dataset without impacting production data, one can use data version control to create replicas of the production environment where tests can be carried out. Such replicas allow testing and understanding of changes safely applied to data.

Data version control tools allow replication environments without the time- and resource-consuming maintenance. Instead, such tools allow objects to be shared using metadata.

Continuous changes in data sets can sometimes cause functionality issues or lead to undesired outcomes, especially when applications are using the data. Data version control tools allow for the possibility to roll back a data set to an earlier state. This can be used to restore or improve functionality of an application or to correct errors or bad data which has been mistakenly included.^[15]

Version controlled data sources:

• Kaggle^[16]
• Quilt^[17]
• Dolt^[18]
• Kamu^[19]

Data version control for data lakes:

• LakeFS^[20]
• Project Nessie^[21]
• Git-LFS^[22]

ML-Ops systems that implement data version control:

• DVC^[23]
• Pachyderm^[24]
• Neptune^[25]
• activeloop^[26]
• graviti^[27]
• dagshub^[28]
• alectio^[29]
• Galileo^[30]
• Voxel51^[31]
• dstack^[32]
• dvid^[33]