DS2002 Data Project 2 Capstone – Titanic Data Lakehouse

Project Overview

This project implements a dimensional Data Lakehouse using PySpark and Delta Lake based on the Titanic dataset. The solution demonstrates batch and streaming data integration using multiple source systems and follows the Bronze/Silver/Gold data engineering architecture discussed in class and Lab 6.

The project models Titanic passenger survival events as the business process being analyzed.

---

Technologies Used

• PySpark
• Delta Lake
• MySQL
• MongoDB Atlas
• Pandas
• Jupyter Notebook

---

Data Sources

This project integrates data from multiple source systems as required by the assignment.

1. MySQL Relational Database

Used as the primary OLTP source system.

Tables created in MySQL:

• dim_date source data
• passenger dimension source data
• embarkation dimension source data
• ticket dimension source data
• survival fact source data

---

2. MongoDB Atlas

Embarkation dimension data is exported from MySQL as JSON and uploaded into MongoDB Atlas before being read into PySpark.

Flow: MySQL → JSON → MongoDB Atlas → PySpark

---

3. Local CSV Files

Ticket dimension data is exported from MySQL as CSV and read from local disk into Spark.

Flow: MySQL → CSV → Local File System → PySpark

---

4. Streaming JSON Files

Fact survival event data is exported from MySQL into 3 separate JSON files to simulate streaming mini-batches.

Flow: MySQL → 3 JSON Files → PySpark Structured Streaming

---

Data Lakehouse Architecture

The project implements the Bronze/Silver/Gold architecture.

Bronze Layer

Raw streaming JSON survival event data is ingested into Delta tables.

Silver Layer

Bronze data is cleaned and integrated with dimension tables using joins.

Gold Layer

Business analytics and aggregated reporting queries are generated.

Examples:

• Survival rate by passenger class
• Survival rate by gender
• Survival statistics by embarkation location

---

Dimensional Model

Dimension Tables

• dim_date
• dim_passenger
• dim_ticket
• dim_embarkation

Fact Table

• fact_survival_event

---

Project Folder Structure

DS2002_Titanic_Capstone_LOCAL_Project/
├── DS2002_Project2_Titanic_Capstone_LOCAL_PySpark.ipynb
├── README.md
├── titanic_capstone/
│   ├── source/
│   ├── batch/
│   ├── streaming/
│   ├── bronze/
│   ├── silver/
│   └── gold/
└── spark-warehouse/