Data Engineering · Coursework
SSIS ETL Pipeline
A small pipeline that made the movement of data visible—from raw files, through business rules, into reliable database tables.
01 · Purpose
Understanding what happens before analysis begins
This project focused on a part of data work that users rarely see: reliably moving records from files into structured database tables.
The task used election-style CSV data with country, city, party, vote count, and percentage fields.
02 · Pipeline design
Routing records through explicit rules
I created SSIS data flows that loaded CSV files into SQL Server and separated records into different destination tables according to PartyName.
A Foreach Loop Container automated the processing of multiple files, while an Execute SQL task truncated destination tables before each run so the pipeline could be repeated cleanly.
03 · Reflection
Reliable analytics begins upstream
The assignment showed me how orchestration, control flow, data flow, and database design work together. It also made idempotency practical: a repeatable pipeline needs a clear approach to previous data and reruns.
That foundation connects directly to the larger Airflow, PySpark, and production-pipeline work I later handled professionally.
04 · The next question
From scheduled ETL to observable data products
A modern extension could add incremental loading, schema validation, data-quality checks, lineage, and alerts.
The same core idea remains important even in cloud and lakehouse environments: data should arrive predictably, be validated, and be easy to trace.