Data Storage & Processing

Data storage and processing is about the lifecycle of data. Mental model: data is born in operational systems, flows through pipelines, stored in appropriate stores (hot/warm/cold), processed (batch or stream), and served for analytics or ML. Each step has trade-offs between cost, latency, and consistency. Data Storage & Processing Storage Types Example EBS, local SSD S3, GCS EFS, NFS Object Storage (S3) Data Pipeline Patterns ETL vs ELT ETL: Extract → Transform → Load (transform before storing) ELT: Extract → Load → Transform (store raw, transform on demand) Modern trend: ELT with data warehouses (BigQuery, Snowflake) that handle transformation Stream Processing Batch vs Stream Batch Minutes to hours Very high Lower Reports, ML training Spark, Hadoop Data Lake vs Data Warehouse Data Lake: Raw data, any format, schema-on-read (S3 Athena) Data Warehouse: Structured data, schema-on-write (BigQuery, Redshift) Interview Tip "For file uploads, I'd use S3 with pre-signed URLs (secure, scalable). For analytics, ELT into a data warehouse. For real-time processing, Kafka stream processor." Java Implementation JavaScript/TypeScript Implementation