Big Data Analytics · Coursework
Spark Weather Analytics
Using distributed SQL to ask practical questions of a weather dataset—and learning how analysis changes when data is treated as a system.
01 · Purpose
Learning to query data beyond a local spreadsheet
This project used a weather dataset to explore temperatures, wind, precipitation, cities, states, and dates through Spark SQL.
The questions were straightforward, but the purpose was broader: learn how structured analysis works in a distributed data environment.
02 · Approach
Turning a CSV into a queryable dataset
I loaded the CSV into Spark, created a temporary view, and used SQL queries to answer questions about maximum and minimum temperatures, wind speed and direction, precipitation, locations, and year-based records.
Several questions were tested with alternative query forms, which helped build confidence in translating an analytical requirement into SQL logic.
03 · Reflection
Simple questions still require precise definitions
The exercise showed how easily a natural-language question can be translated incorrectly. Conditions such as “between,” “least,” or “highest” need careful treatment in code.
That attention to definitions later became important in my work with production data and business metrics.
04 · The next question
From historical queries to environmental signals
A larger version could use streaming weather feeds, geospatial features, and forecasting models to study severe events or long-term patterns.
It could also become a useful foundation for understanding how distributed data pipelines support climate and environmental analysis.