Data engineering

Not to be confused with Information Engineering.

Short description: Software engineering approach to designing and developing information systems

Template:Engineering side bar Data engineering is a software engineering approach to the building of data systems, to enable the collection and usage of data. This data is usually used to enable subsequent analysis and data science, which often involves machine learning.[1][2] Making the data usable usually involves substantial computing and storage, as well as data processing.

## Contents

* 1 History
* 2 Tools
* 2.1 Compute
* 2.2 Storage
* 2.2.1 Databases
* 2.2.2 Data warehouses
* 2.2.3 Data lakes
* 2.2.4 Files
* 2.3 Management
* 3 Lifecycle
* 3.1 Business planning
* 3.2 Systems design
* 3.3 Data modeling
* 4 Roles
* 4.1 Data engineer
* 4.2 Data scientist
* 5 See also
* 6 References
* 7 Further reading
* 8 External links

## History

Around the 1970s/1980s the term information engineering methodology (IEM) was created to describe database design and the use of software for data analysis and processing.[3] These techniques were intended to be used by database administrators (DBAs) and by systems analysts based upon an understanding of the operational processing needs of organizations for the 1980s. In particular, these techniques were meant to help bridge the gap between strategic business planning and information systems. A key early contributor (often called the "father" of information engineering methodology) was the Australian Clive Finkelstein, who wrote several articles about it between 1976 and 1980, and also co-authored an influential Savant Institute report on it with James Martin.[4][5][6] Over the next few years, Finkelstein continued work in a more business-driven direction, which was intended to address a rapidly changing business environment; Martin continued work in a more data processing-driven direction. From 1983 to 1987, Charles M. Richter, guided by Clive Finkelstein, played a significant role in revamping IEM as well as helping to design the IEM software product (user data), which helped automate IEM.

In the early 2000s, the data and data tooling was generally held by the information technology (IT) teams in most companies.[7] Other teams then used data for their work (e.g. reporting), and there was usually little overlap in data skillset between these parts of the business.

In the early 2010s, with the rise of the internet, the massive increase in data volumes, velocity, and variety led to the term big data to describe the data itself, and data-driven tech companies like Facebook and Airbnb started using the phrase  data engineer.[3][7] Due to the new scale of the data, major firms like Google, Facebook, Amazon, Apple, Microsoft, and Netflix started to move away from traditional ETL and storage techniques. They started creating data engineering, a type of software engineering focused on data, and in particular infrastructure, warehousing, data protection, cybersecurity, mining, modelling, processing, and metadata management.[3][7] This change in approach was particularly focused on cloud computing.[7] Data started to be handled and used by many parts of the business, such as sales and marketing, and not just IT.[7]

## Tools

### Compute

High-performance computing is critical for the processing and analysis of data. One particularly widespread approach to computing for data engineering is dataflow programming, in which the computation is represented as a directed graph (dataflow graph); nodes are the operations, and edges represent the flow of data.[8] Popular implementations include Apache Spark, and the deep learning specific TensorFlow.[8][9][10] More recent implementations, such as Differential/Timely Dataflow, have used incremental computing for much more efficient data processing.[8][11][12]

### Storage

Data is stored in a variety of ways, one of the key deciding factors is in how the data will be used. Data engineers optimize data storage and processing systems to reduce costs. They use data compression, partitioning, and archiving.

#### Databases

If the data is structured and some form of online transaction processing is required, then databases are generally used.[13] Originally mostly relational databases were used, with strong ACID transaction correctness guarantees; most relational databases use SQL for their queries. However, with the growth of data in the 2010s, NoSQL databases have also become popular since they horizontally scaled more easily than relational databases by giving up the ACID transaction guarantees, as well as reducing the object-relational impedance mismatch.[14] More recently, NewSQL databases — which attempt to allow horizontal scaling while retaining ACID guarantees — have become popular.[15][16][17][18]

#### Data warehouses

Main article: Data warehouse

If the data is structured and online analytical processing is required (but not online transaction processing), then data warehouses are a main choice.[19] They enable data analysis, mining, and artificial intelligence on a much larger scale than databases can allow,[19] and indeed data often flow from databases into data warehouses.[20] Business analysts, data engineers, and data scientists can access data warehouses using tools such as SQL or business intelligence software.[20]

#### Data lakes

A data lake is a centralized repository for storing, processing, and securing large volumes of data. A data lake can contain structured data from relational databases, semi-structured data, unstructured data, and binary data. A data lake can be created on premises or in a cloud-based environment using the services from public cloud vendors such as Amazon, Microsoft, or Google.

#### Files

If the data is less structured, then often they are just stored as files. There are several options:

* File systems represent data hierarchically in nested folders.[21]
* Block storage splits data into regularly sized chunks;[21] this often matches up with (virtual) hard drives or solid state drives.
* Object storage manages data using metadata;[21] often each file is assigned a key such as a UUID.[22]

### Management

The number and variety of different data processes and storage locations can become overwhelming for users. This inspired the usage of a workflow management system (e.g. Airflow) to allow the data tasks to be specified, created, and monitored.[23] The tasks are often specified as a directed acyclic graph (DAG).[23]

## Lifecycle

### Business planning

Business objectives that executives set for what's to come are characterized in key business plans, with their more noteworthy definition in tactical business plans and implementation in operational business plans. Most businesses today recognize the fundamental need to grow a business plan that follows this strategy. It is often difficult to implement these plans because of the lack of transparency at the tactical and operational degrees of organizations. This kind of planning requires feedback to allow for early correction of problems that are due to miscommunication and misinterpretation of the business plan.

### Systems design

The design of data systems involves several components such as architecting data platforms, and designing data stores.[24][25]

### Data modeling

Data modeling is the analysis and representation of data requirements for an organisation. It produces a data model—an abstract representation that organises business concepts and the relationships and constraints between them. The resulting artefacts guide communication between business and technical stakeholders and inform database design.[26][27]

A common convention distinguishes three levels of models:[26]

* Conceptual model – a technology-independent view of the key business concepts and rules.
* Logical model – a detailed representation in a chosen paradigm (most commonly the relational model) specifying entities, attributes, keys, and integrity constraints.[27]
* Physical model – an implementation-oriented design describing tables, indexes, partitioning, and other operational considerations.[27]

Approaches include entity–relationship (ER) modeling for operational systems,[28] dimensional modeling for analytics and data warehousing,[29] and the use of UML class diagrams to express conceptual or logical models in general-purpose modeling tools.[30]

Well-formed data models aim to improve data quality and interoperability by applying clear naming standards, normalisation, and integrity constraints.[27][26]

## Roles

### Data engineer

A  data engineer is a type of software engineer who creates big data ETL pipelines to manage the flow of data through the organization. This makes it possible to take huge amounts of data and translate it into insights.[31] They are focused on the production readiness of data and things like formats, resilience, scaling, and security. Data engineers usually hail from a software engineering background and are proficient in programming languages like Java, Python, Scala, and Rust.[32][3] They will be more familiar with databases, architecture, cloud computing, and Agile software development.[3]

### Data scientist

Data scientists are more focused on the analysis of the data, they will be more familiar with mathematics, algorithms, statistics, and machine learning.[3][33]

## See also

* Big data
* Computer science
* Information technology
* List of data science software
* Software engineering

## References

1. ↑ "What is Data Engineering? | A Quick Glance of Data Engineering". 5 January 2020. https://www.educba.com/what-is-data-engineering/.
2. ↑ "Introduction to Data Engineering". https://www.dremio.com/resources/guides/intro-data-engineering/.
3. ↑ 3.0 3.1 3.2 3.3 3.4 3.5 Black, Nathan (15 January 2020). "What is Data Engineering and Why Is It So Important?". https://quanthub.com/what-is-data-engineering/.
4. ↑ "Information engineering," part 3, part 4, part 5, Part 6" by Clive Finkelstein. In Computerworld, In depths, appendix. May 25 – June 15, 1981.
5. ↑ Christopher Allen, Simon Chatwin, Catherine Creary (2003). Introduction to Relational Databases and SQL Programming.
6. ↑ Terry Halpin, Tony Morgan (2010). Information Modeling and Relational Databases. p. 343
7. ↑ 7.0 7.1 7.2 7.3 7.4 Dodds, Eric. "The History of the Data Engineering and the Megatrends". https://www.rudderstack.com/blog/the-data-engineering-megatrend-a-brief-history.
8. ↑ 8.0 8.1 8.2 Schwarzkopf, Malte (7 March 2020). "The Remarkable Utility of Dataflow Computing". https://www.sigops.org/2020/the-remarkable-utility-of-dataflow-computing/.
9. ↑ "sparkpaper". https://cs.stanford.edu/~matei/papers/2016/cacm_apache_spark.pdf.
10. ↑ Abadi, Martin; Barham, Paul; Chen, Jianmin; Chen, Zhifeng; Davis, Andy; Dean, Jeffrey; Devin, Matthieu; Ghemawat, Sanjay et al. (2016). "TensorFlow: A system for large-scale machine learning". pp. 265–283. https://research.google/pubs/pub45381/.
11. ↑ McSherry, Frank; Murray, Derek; Isaacs, Rebecca; Isard, Michael (5 January 2013). "Differential dataflow". https://www.microsoft.com/en-us/research/publication/differential-dataflow/.
12. ↑ "Differential Dataflow". Timely Dataflow. 30 July 2022. https://github.com/TimelyDataflow/differential-dataflow.
13. ↑ "Lecture Notes | Database Systems | Electrical Engineering and Computer Science | MIT OpenCourseWare". https://ocw.mit.edu/courses/6-830-database-systems-fall-2010/pages/lecture-notes/.
14. ↑ Leavitt, Neal (February 2010). "Will NoSQL Databases Live Up to Their Promise?". Computer 43 (2): 12–14. doi:10.1109/MC.2010.58.
15. ↑ Aslett, Matthew (2011). "How Will The Database Incumbents Respond To NoSQL And NewSQL?". 451 Group. https://cs.brown.edu/courses/cs227/archives/2012/papers/newsql/aslett-newsql.pdf.
16. ↑ Pavlo, Andrew; Aslett, Matthew (2016). "What's Really New with NewSQL?". https://db.cs.cmu.edu/papers/2016/pavlo-newsql-sigmodrec2016.pdf. Retrieved February 22, 2020.
17. ↑ Stonebraker, Michael (June 16, 2011). "NewSQL: An Alternative to NoSQL and Old SQL for New OLTP Apps". Communications of the ACM Blog. https://cacm.acm.org/blogs/blog-cacm/109710-new-sql-an-alternative-to-nosql-and-old-sql-for-new-oltp-apps/fulltext.
18. ↑ Hoff, Todd (September 24, 2012). "Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In". http://highscalability.com/blog/2012/9/24/google-spanners-most-surprising-revelation-nosql-is-out-and.html.
19. ↑ 19.0 19.1 "What is a Data Warehouse?" (in en-us). https://www.ibm.com/cloud/learn/data-warehouse.
20. ↑ 20.0 20.1 "What is a Data Warehouse? | Key Concepts | Amazon Web Services". https://aws.amazon.com/data-warehouse/.
21. ↑ 21.0 21.1 21.2 "File storage, block storage, or object storage?" (in en). https://www.redhat.com/en/topics/data-storage/file-block-object-storage.
22. ↑ "Cloud Object Storage – Amazon S3 – Amazon Web Services". https://aws.amazon.com/s3.
23. ↑ 23.0 23.1 "Home" (in en). https://airflow.apache.org/.
24. ↑ "Introduction to Data Engineering" (in en). https://www.coursera.org/learn/introduction-to-data-engineering.
25. ↑ Finkelstein, Clive. What are The Phases of Information Engineering.
26. ↑ 26.0 26.1 26.2 Simsion, Graeme; Witt, Graham (2015). Data Modeling Essentials (4th ed.). Morgan Kaufmann. ISBN 9780128002025.
27. ↑ 27.0 27.1 27.2 27.3 Date, C. J. (2004). An Introduction to Database Systems (8th ed.). Addison-Wesley. ISBN 9780321197849.
28. ↑ Chen, Peter P. (1976). "The Entity–Relationship Model—Toward a Unified View of Data". ACM Transactions on Database Systems 1 (1): 9–36. doi:10.1145/320434.320440.
29. ↑ Kimball, Ralph; Ross, Margy (2013). The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling (3rd ed.). Wiley. ISBN 9781118530801.
30. ↑ ﻿Unified Modeling Language (UML) Version 2.5.1﻿ (Report). Object Management Group. 2017. https://www.omg.org/spec/UML/2.5.1/.
31. ↑ Tamir, Mike; Miller, Steven; Gagliardi, Alessandro (11 December 2015). ﻿The Data Engineer﻿ (Report).
32. ↑ "Data Engineer vs. Data Scientist" (in en-US). 2019-02-07. https://www.springboard.com/blog/data-engineer-vs-data-scientist/.
33. ↑ "What is Data Science and Why it's Important". Edureka. Jan 5, 2017. https://www.edureka.co/blog/what-is-data-science/.

## Further reading

* Hares, John S. (1992). Information Engineering for the Advanced Practitioner. Wiley. ISBN 978-0-471-92810-2.
* Finkelstein, Clive (1989). An Introduction to Information Engineering: From Strategic Planning to Information Systems. Addison-Wesley. ISBN 978-0-201-41654-1.
* Finkelstein, Clive (1992). Information Engineering: Strategic Systems Development. Addison-Wesley. ISBN 978-0-201-50988-5.
* Ian Macdonald (1986). "Information engineering". in: Information Systems Design Methodologies. T.W. Olle et al. (ed.). North-Holland.
* Ian Macdonald (1988). "Automating the Information engineering methodology with the Information Engineering Facility". In: Computerized Assistance during the Information Systems Life Cycle. T.W. Olle et al. (ed.). North-Holland.
* James Martin and Clive Finkelstein. (1981). Information engineering. Technical Report (2 volumes), Savant Institute, Carnforth, Lancs, UK.
* James Martin (1989). Information engineering. (3 volumes), Prentice-Hall Inc.
* Finkelstein, Clive (2006). Enterprise Architecture for Integration: Rapid Delivery Methods and Technologies. Artech House. ISBN 978-1-58053-713-1.
* Reis, Joe; Housley, Matt (2022). Fundamentals of Data Engineering. O'Reilly Media. ISBN 978-1-0981-0827-4.

## External links

* The Complex Method IEM
* Rapid Application Development
* Enterprise Engineering and Rapid Delivery of Enterprise Architecture

* v
* t
* e

Engineering

Civil|

* Architectural
* Construction
* Environmental
* Earthquake
* Geotechnical
* Hydraulic
* Mining
* Structural
* Transportation

Mechanical|

* Acoustical
* Aerospace
* Automotive
* Marine
* Railway
* Thermal

Electrical|

* Computer
* Control
* Electromechanics
* Electronics
* Microwaves
* Power
* Radio Frequency
* Telecommunications

Chemical|

* Biochemical
* Biological
* Molecular
* Petroleum
* Process
* Reaction

Interdisciplinary|

* Agricultural
* Audio
* Biomedical
* Engineering mathematics
* Engineering mechanics
* Engineering physics
* Engineering science
* Industrial
* Information
* Materials science
* Ceramics
* Metallurgy
* Polymer
* Mechatronics
* Military
* Nanotechnology
* Nuclear
* Optical
* Photonics
* Privacy
* Robotics
* Sanitary
* Security
* Software
* Systems

Glossaries|

* Engineering
* Aerospace engineering
* Civil engineering
* Electrical and electronics engineering
* Mechanical engineering
* Structural engineering

* List of engineering branches
* Category:Engineering
* '

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Data engineering. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Data_engineering&oldid=4116521"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template