One approach to organizing data pipelines is medallion architecture, where data flows through various stages depending on how much it has been processed. Raw data is retained close to original, cleansed and standardized prior to being converted to business-ready datasets. This approach simplifies pipeline understanding, testing and maintenance. It will also assist teams to trace back if something appears wrong, to understand the source of the data. Datalance can be helpful for comprehending the structure's role in an overall enterprise data strategy without adding to the overall complexity of the architecture.
