Course Outline

🔹 Day 1: Introduction to Apache Airflow and Workflow Basics

Session 1: Overview of Airflow and ETL Concepts

• What is Apache Airflow? Architecture and components
• Understanding ETL vs ELT pipelines
• Airflow use cases and ecosystem overview

Session 2: Airflow Installation and Setup

• Installing Airflow locally and configuring the environment
• Airflow UI walkthrough and core concepts: DAGs, tasks, scheduler, executor
• Writing your first DAG and running tasks

Session 3: Designing DAGs and Tasks

• DAG structure and best practices
• Operators: BashOperator, PythonOperator, DummyOperator
• Task dependencies and execution order

Lab Activities:

• Install and configure Apache Airflow locally
• Create and run a simple ETL DAG with basic operators
• Visualize DAG runs and task status in the Airflow UI

🔹 Day 2: Advanced Workflow Design and Integration

Session 1: Airflow Sensors and Hooks

• Using sensors for external event detection
• Hooks for connecting to databases, cloud storage, APIs
• Building modular and reusable components

Session 2: Parameterization and Dynamic Pipelines

• Using Variables, Connections, and XCom for data passing
• Dynamic DAGs and templating with Jinja
• Handling retries, SLA, and failure notifications

Session 3: Integrating Airflow with Data Sources and Cloud

• Connecting Airflow to databases (Postgres, MySQL), APIs, and cloud storage (S3, GCS)
• Using Airflow with cloud providers: AWS, GCP, Azure
• Triggering external workflows and sensors

Lab Activities:

• Build a DAG with sensors and hooks to pull data from an API
• Create parameterized workflows with retries and SLA alerts
• Integrate Airflow with AWS S3 or GCP Storage for data ingestion

🔹 Day 3: Scaling, Monitoring & Best Practices

Session 1: Airflow Executors and Scalability

• Executor types: Sequential, Local, Celery, Kubernetes
• Setting up distributed execution and scaling workers
• Using KubernetesExecutor for cloud-native orchestration

Session 2: Monitoring, Logging & Alerting

• Airflow logs and monitoring strategies
• Using external monitoring tools (Prometheus, Grafana)
• Alerting and notifications (email, Slack)

Session 3: Best Practices and Security

• Organizing and versioning DAGs
• Managing secrets and connections securely
• Performance tuning and upgrade strategies

Lab Activities:

• Configure CeleryExecutor with multiple workers
• Set up alerts for failed DAG runs via email or Slack
• Secure connections and manage secrets in Airflow