What is a data pipeline? From ETL to modern data platforms.

Most organizations generate data in dozens of different systems. Customer records sit in a CRM, financial data lives in accounting software, operational data comes from internal applications, and new information is constantly being created through APIs and third-party platforms.

A data pipeline is the process that brings all of that together. It extracts data from its source, prepares it for analysis, and makes it available to the people and systems that need it, whether that’s a dashboard, an AI model, or another business application.

Every pipeline looks a little different, but they all follow the same general flow.

The goal isn’t simply to move data from one place to another. It’s to turn information collected by different systems into something people can use to make decisions or automate work.

You’ll often hear two terms when people talk about data pipelines: ETL and ELT.

ETL stands for Extract, Transform, Load. Traditionally, data was transformed before it was loaded into a reporting database.

Modern platforms more commonly use ELT. Data is extracted from the source, loaded into a data lake or warehouse, and transformed afterwards.

The change reflects how modern data platforms work. Storage has become relatively inexpensive, making it practical to keep the raw data first and apply transformations later as reporting requirements evolve.

Connecting to a data source is no longer the difficult part.

Most common applications already have APIs, and tools such as Airbyte, Fivetran, or even AI coding assistants can generate much of the extraction code with relatively little effort.

The work that still requires engineering is the business logic.

Those rules are specific to each organization, and they’re where most of the customization happens.

This is also where concepts such as bronze, silver, and gold data layers become useful. Raw data is preserved, cleaned into a more reliable form, and eventually transformed into reporting-ready datasets that the business can work from.

Not every pipeline needs to update in real time.

One of the most common design decisions is choosing how frequently new data should be processed.

For many organizations, a daily batch is more than enough.

If reports are only reviewed once each morning, updating the data every hour usually adds cost without providing much additional value.

Streaming becomes useful when data needs to be processed continuously, perhaps every few minutes or after a certain number of events. Real-time processing goes one step further, updating information as each event happens.

A bank account is a good example. When you use your debit card, your balance updates immediately. Waiting until the end of the day wouldn’t be practical because the system needs to know how much money is available before the next transaction.

Some businesses also adjust how quickly data is processed during busy periods. An e-commerce company might rely on daily reporting for most of the year, then increase processing frequency during events such as Black Friday when sales activity changes much more quickly.

There isn’t a single “best” pipeline architecture.

The right design depends on the business, the systems involved, and how quickly people need the information.

For many organizations, the biggest challenge isn’t moving data. It’s deciding how that data should be structured, which business rules should be applied, and how often it needs to be updated.

Those decisions are what turn a collection of disconnected systems into a reporting platform that people can rely on.

Jinka provides data engineering and analytics infrastructure to clients across North America, Europe, and APAC. This piece is intended as a general overview and does not represent specific implementation advice.