A declarative approach to data pipelines

Mutinex offers a marketing analytics & econometrics platform that helps marketers make better investment decisions faster.

Over the last year, Altis collaborated with Mutinex to establish an automated data ingestion and validation pipeline.

The Problem

Mutinex’s analysis and insights platform, GrowthOS, requires customer data to be ingested, validated and structured. However, the existing process leant on human handling and manually run ingestion scripts.

The process was time consuming for Mutinex and customers which slowed the delivery of insights.

– Dan Gooden,
Head of Engineering at Mutinex

The Solution

A combined team of Mutinex and Altis Data Engineers designed and developed a scalable process to automate the ingestion of data into GrowthOS. The solution leverages Google Cloud for file storage, Dagster for file management and declarative pipelines and Dbt for modelling and validation. The solution was also designed to scale and be maintainable over time with support for schema evolution and the collection of additional data. An emphasis on data quality meant designing for the addition, refinement, and enhancement of validation rules to meet the requirements of downstream processes. Any failure in validation is able to be easily surfaced back to the customer for rectification.

Tangible Outcomes

The solution enables Mutinex’s customers to provide data, receive near-real-time feedback on any issues, and have the data quickly stored and ready for processing by GrowthOS in the secure Mutinex data lake.

Declarative Data Pipelines

Declarative data pipelines have emerged as a powerful tool to streamline data workflows and data processes. These pipelines, built on the foundations of declarative programming, offer a more efficient and intuitive way to manage complex data processes.

What Are Declarative Pipelines?

Declarative pipelines are a paradigm in which you define the desired state of your data workflow without specifying the exact steps to reach that state. In essence, you declare what you want to achieve, and the underlying system takes care of the how to generate the result. This approach stands in contrast to imperative pipelines, where you explicitly list each step in the process.

Key Benefits of Declarative Pipelines:

• Abstraction of Complexity: Declarative pipelines abstract away much of the complexity associated with data workflow management.
• Enhanced Clarity: Declarative pipelines provide a clear and high-level overview of the entire workflow. This transparency aids in understanding and maintaining complex data pipelines.
• Scalability and Adaptability: Declarative pipelines are inherently scalable. As data volumes grow or requirements change, you can modify the declaration to accommodate new data sources, transformations, or validations. This adaptability is crucial in a dynamic data environment.
• Easier Error Handling: Declarative pipelines often come with built-in error handling and retries. When an unexpected issue arises, the system can attempt to recover automatically or provide clear notifications, reducing the need for manual intervention.

Do you want to find out more about automating your data ingestion and validation pipeline?

Connect with Altis today to find out how we can help maximise your business performance.