Data Turbulence — Big Data Flight Analytics Platform

Overview

Data Turbulence is a team big data project for exploring a very large set of airline ticket itineraries. It puts a modern web experience on top of a Hadoop and Hive warehouse, so users can search flights, explore pricing trends and ask questions in plain English without writing HiveQL.

How it works

More than 82 million itineraries are stored in Apache Hive on a three-node Hadoop cluster. The raw CSV is converted into an optimised ORC table for faster scans. A Flask backend running under Spark exposes 50+ REST endpoints that run SparkSQL and HiveQL against the warehouse. Several external datasets are joined in for enrichment: airports and runways, navigation aids, FAA delay causes, DOT complaint rates, airline customer-satisfaction scores, Amtrak alternative routes and airline sentiment from Reddit.

The React 18 single-page app has eight routes:

• A landing page with a video hero.
• Flight search that shows the HiveQL behind each result.
• A 22-card analytics dashboard.
• "Where and when to fly" visual suggestions.
• A route-optimisation engine.
• A data-enrichment explorer.
• An AI chatbot that turns natural-language questions into SQL using an LLM and runs them on the cluster.

Engineering notes

The project includes HiveQL DDL for every core and external table, a table-optimisation utility, an automatic analytics report generator, and HDFS maintenance helpers. The frontend uses a custom dark theme and includes a live backend health indicator.