Data Engineering and Best practices

Data and types

• Data at rest (e.g. batch data pipelines / data stored in warehouses or object stores)
• Data in motion (e.g. streaming pipelines / real-time use cases).

Hadoop ecosystem

https://www.upgrad.com/blog/hadoop-ecosystem-components/

https://saiparvathaneni.medium.com/hadoop-ecosystem-explained-like-youre-10-the-kitchen-analogy-595ff624165c

https://medium.com/towards-data-science/end-to-end-data-engineering-system-on-real-data-with-kafka-spark-airflow-postgres-and-docker-a70e18df4090

CI/CD

https://towardsdev.com/ci-cd-for-modern-data-engineering-e2e7d2d0a694

Big Data

https://bigdataschools.com/

Arcihtecture

https://mahmoudai.medium.com/the-backbone-of-data-engineering-5-key-architectural-patterns-explained-2327de58d99a

https://a16z.com/emerging-architectures-for-modern-data-infrastructure/

CDC

https://medium.com/@luishrsoares/the-change-data-capture-cdc-design-pattern-fa8d3adc964f

Data Streaming Architecture

https://medium.com/@techattu/modern-data-streaming-architecture-9d518c371066

Apache Spark link

• RDD vs. DataFrame vs. Datasetlink
• Creating Job in Spark link
• Oozie Workflow with Spark Job:
• RDD (resilient distributed dataset):
• Spark context internal working
• ETL pipeline with spark link

Areas to focus for Data Engineer

Prioritise understanding these core concepts first: These principles are timeless and transferable. New frameworks will emerge, some will fade, but these fundamentals will remain crucial:
🔹SQL: This is the bedrock. Master it. Understand joins, aggregations, window functions, and query optimisation.
🔹NoSQL Databases: Learn about different NoSQL models and when to use them. Understand their trade-offs.
🔹Database Internals: Grasp the difference between row/columnar databases, indexing, and transactions.
🔹Distributed Systems: Understand distributed computing, partitioning, consistency, and fault tolerance.
🔹Data Modeling: Learn different modeling techniques and how to design efficient schemas.
🔹ETL/ELT Concepts: Understand data processing, transformation, and data quality.

Once you have a solid grasp of these fundamentals, learning specific tools becomes much easier. You’ll understand why they work the way they do.

Regarding the modern data stack and big data tools, including cloud data warehouses and query engines:

Be aware of popular tools like dbt for transformations, Airflow/Prefect/Dagster for orchestration, Spark/Flink for processing, Kafka/Pulsar for streaming, and the evolving data lakehouse landscape with Iceberg/Delta Lake/Hudi. It's also important to understand the landscape of cloud data warehouses and high-performance query engines:
🔹Cloud Data Warehouses (Snowflake, BigQuery, AWS Redshift): These offer scalable and managed solutions for analytical workloads. Understand their strengths, weaknesses, and use cases.
🔹High-Performance Query Engines (ClickHouse, StarRocks): These are designed for real-time analytics and often used for specific use cases like dashboards and reporting.