Data Pipeline · ETL
Data Pipeline / Extract · Transform · Load
What is a data pipeline?
- Collect, clean, and store data (ETL)
- The “plumbing” that gets data ready for analysis
- Building the automated flow
What actually changes once a data pipeline is in place is 'who produces the numbers.' Before that, a marketing manager downloads a CSV from the ad platform, sales support pulls a report from the ERP, and each person pastes and reconciles them in Excel, week after week. That is also why the same 'last month's revenue' comes out differently in each department. A pipeline locks this manual work into code so that a scheduler, not a person, produces the numbers. The 'which number is right?' debates that ate the start of every meeting shrink, and there is one place to fix things when they're wrong.
Three design decisions matter most. First, ELT is now used more often than ETL. As cloud warehouse compute costs have fallen, it makes more sense to load the raw data as is and transform it with SQL, which makes recalculation and recovery easier. Second, frequency. If you don’t distinguish metrics that are fine with one daily batch from those that need minute-level updates, you only add cost. Finance and HR reports are fine with an overnight batch; only things like inventory and fraud detection call for near-real-time. Third, the choice between a full load, which reloads everything each time, and an incremental load, which loads only what changed, can change processing time by a factor of dozens.
The most common misconception is that adopting a tool makes your data clean. A pipeline is transport equipment, not a quality guarantee. If a salesperson enters a client name as 'Ganada Co., Ltd.' in one place and 'Co., Ltd. Ganada' in another, those two records are still counted as different clients after passing through the pipeline. Silent failure is even more dangerous. If the data owner doesn't know the dev team renamed a column in the orders table, loading stops while the dashboard keeps showing yesterday's numbers. Without failure alerts and count/total reconciliation, you only find out at month-end close two weeks later. An agreement to be notified of source changes in advance and a named owner come before tool selection.
Jaryong KooMarketing PhD and authority in AI, data analytics, and personal branding
John KimAuthor of B2B Marketing Design and CEO of Nanum ConsultingThe data pipeline grew out of ETL (extract-transform-load), a concept that evolved alongside data warehousing, and expanded with cloud and real-time processing into its modern form.
Consider a hypothetical case. Company A makes household goods and sells them online and in stores, with 320 employees and annual revenue of around KRW 180 billion (≈ US$134,670,100). Sales data was scattered across its own web store, 4 online marketplaces, the ERP, and the CRM, so producing one weekly sales report took two staff members two days. On top of that, sales figures differed by department in every meeting, so the first 30 minutes went to verifying the numbers.
- When Kim, a manager on the data analytics team, traced one weekly report back to its sources, it involved 11 Excel files and 34 manual copy-and-paste steps.
- When the finance lead asked, “Why do our sales and marketing numbers always differ by 3%?”, it turned out each team counted returns at a different point, so they rewrote the metric definitions first.
- Working with an outside development firm, they built a batch job that incrementally loads data from 4 online marketplaces, the ERP, and the company store every day at 3 a.m. They also added phone alerts to the owner whenever a load fails.
- In the second week of operation, one online marketplace changed its API spec and loading stopped, but alerts let the team fix it that same morning. They then agreed to include source changes in the dev team’s release notices.
- Six months later, weekly report preparation had dropped from 16 hours (2 people) to 40 minutes, and the gap in sales figures between departments had narrowed to within 0.2%. Merchandisers now decide on inventory orders on Monday mornings.