Data Catalog
A data catalog is an organized inventory of data assets in your organization. It helps people find, understand, trust, and use data correctly. Your document should answer: What data exists? Where is it? Who owns it? What does it mean? How good is it? Who can use it?
Now, to write good documentation, we have to have some preface and explanation sections to show the user what we are going to do and why we have this documentation, who can read it, what type of data the document contains, the target audience, and how to use the documentation.
We also have to give the user a list of databases and tables. Sometimes, we have huge differences between database versions, and we have Production, Stage, and Dev environments. So, all this information should be given to the user to help them understand everything when they are reading the documentation.
The main and core part of our work is the Data Asset Documentation. Here, we are going to talk about each table separately and provide the user with some metadata and core information about how the table is populated with data (sources), including how and by which methods the data is filled.
The main and core part of our work is the Data Asset Documentation. Here, we are going to talk about each table separately and provide the user with some metadata and core information about how the table is populated with data (sources), including how and by which methods the data is filled.
We will also talk about the pipelines we use (ETL/ELT) and provide all the necessary information for users who need to understand the data and tables. This structure will be repeated for each table/dataset.
| Item | Description |
|---|---|
| Table name & description | What the table contains |
| Owner / Data Steward | Person or team responsible |
| Schema & data types | Columns, types, constraints |
| Column descriptions | Meaning of every column |
| Primary & foreign keys | Relationships between tables |
| Row count & data volume | Size of the dataset |
| Update frequency | Real-time, daily, weekly |
| Last updated | Freshness indicator |
| Sensitive data flag | PII / confidential classification |
One of the key things that I did not care about on my first day of working as a DE was how frequently the data was updated and when the last update occurred. These two measurements were actually very important for BI and some other teams that could use the documentation and rely on this information to make decisions.
Some other topics, such as Data Lineage, Business Glossary, Data Governance & Ownership, Data Quality, Data Models & Diagrams, and Access & Security, are also very important, and we have to discuss them in another section.
Sources to Study 📚 Books Fundamentals of Data Engineering — Joe Reis & Matt Housley (best overall starting point) The Data Catalog: Sherlock Holmes Data Sleuthing for Analytics — Bonnie O'Neil & Lowell Fryman (directly about your task) DAMA-DMBOK (Data Management Body of Knowledge) — the industry standard reference, especially the Metadata Management and Data Governance chapters Data Management at Scale — Piethein Strengholt