Data Pipeline
Everything on SkillVeris tagged Data Pipeline — collected across the glossary, study notes, blog, and cheat sheets.
12 resources across 1 library
Interview Questions(12)
ETL vs ELT Pipelines: What is the Difference?
ETL (Extract, Transform, Load) transforms data in a separate processing engine before loading it into the target system, while ELT (Extract, Load, Transform) l…
What is Change Data Capture (CDC)?
Change Data Capture (CDC) is a technique for identifying and streaming only the rows that changed — inserts, updates, and deletes — in a source database, typic…
What is Data Lineage Tracking and Why Does It Matter?
Data lineage tracking is the practice of recording where each piece of data originated, every transformation it passed through, and every downstream table, rep…
How Do You Implement Data Quality Validation in Pipelines?
Data quality validation in pipelines means running automated checks — for null rates, uniqueness, referential integrity, value ranges, and row-count anomalies…
How Would You Design a Distributed Logging System?
Design a distributed logging system as a pipeline: lightweight agents on each host tail and buffer log lines locally, ship them to a durable, partitioned inges…
How to Design a Log Aggregation System
A log aggregation system collects logs from many hosts via lightweight shippers, ships them through a durable buffer to decouple producers from indexing speed,…
How to Design an Ad Serving System?
An ad serving system is designed around an ultra-low-latency ad-selection path (typically under 100ms) that ranks pre-indexed, targeted candidate ads from an i…
How to Design a Real-Time Analytics Dashboard?
A real-time analytics dashboard is designed around a streaming ingestion pipeline that continuously aggregates raw events into pre-computed rollups at multiple…
What is Lambda Architecture in System Design?
Lambda architecture is a data-processing pattern that runs a slow, accurate batch layer over the complete historical dataset alongside a fast speed layer over…
What is Kappa Architecture and How Does It Differ from Lambda?
Kappa architecture simplifies Lambda architecture by processing all data — historical and real-time — through a single stream-processing pipeline, treating the…
How Would You Design an ETL Pipeline?
An ETL pipeline extracts data from source systems, transforms it into a clean, validated, and correctly shaped form, and loads it into a target store like a wa…
What is Log Sampling and Why Use It?
Log sampling means deliberately recording only a representative subset of log events instead of every single one, in order to control storage cost and ingestio…