The Roadmap

Important Prerequisites

Who I am

Why data orchestration

The Core Concepts

The Core Components

Why Airflow

How does Airflow work

Airflow limitations

Practice Quick tour of Airflow CLI

Practice The Rest API

Practice Installing Airflow

Practice Quick tour of Airflow UI

Introduction

The Project! What you will build

Running the new environment

Create the DAG with the dag decorator

Project materials

The new way of authoring DAGs with Taskflow

Fetching stock prices with the PythonOperator

Checking API availability with the Sensor decorator

The best way to load files into data warehouses with Postgres and Astro SDK

Fetching formatted prices from MinIO (AWS S3 like)

Storing stock prices in MinIO (AWS S3 like)

Formatting stock prices with Spark and the DockerOperator

Creating the dashboard to track Apple stock with Metabase

The pipeline in action!

Getting alerts on Slack with the new Notifiers

Set up the new Airflow environment

The best way to create your DAGs

The parameters your DAGs need

DAG scheduling the basics

Backfill and Catchup

The most important rule to follow when creating tasks

Dealing with timezones in Airflow

Conditional Dataset scheduling

Scheduling DAGs based on data with Datasets

Play by scheduling your DAGs

Datasets in action!

Organize your DAGs folder and clean your DAGs

Sharing data between task with XComs

Manage task and DAG failures

Test your tasks and DAGs

Choosing tasks with branching and conditions

Changing execution behaviours with Trigger Rules

The right way of grouping tasks

Templating your tasks

The smart way of storing data with Custom XCOM backends

Using variables to avoid hardcoding values

Executing tasks sequentially with the SequentialExecutor and SQLite

Executing tasks in parallel with the LocalExecutor and Postgres

Concurrency settings to control how tasks and dags run in parallel

Track your tasks using Flower with the CeleryExecutor

Start scaling Airflow with the CeleryExecutor

Introduction to the KubernetesExecutor

Quick introduction to Kubernetes

Add new workers and configure queues to distribute your tasks

Installing Airflow on a Kubernetes cluster

How to configure Airflow on Kubernetes

Deploying DAGs with Airflow on Kubernetes using GitSync

declarations.d

index

useCalendarData

useClearRun

useClearTask

useClearTaskDryRun

useCreateDatasetEvent

useDag

useDagCode

useDagDetails

useDataset

useDatasetDependencies

useDatasetEvents

useDatasets

useDatasetsSummary

useExtraLinks

useGridData

useGridData.test

useGraphData

useHealth

useMarkFailedRun

useHistoricalMetricsData

useMarkFailedTask

useMappedInstances

useMarkSuccessRun

useMarkSuccessTask

useMarkTaskDryRun

usePools

useQueueRun

useRenderedK8s

useSetDagRunNote

useSetTaskInstanceNote

useTaskInstance

useTaskFailedDependency

useTaskLog

useTaskXcom

useUpstreamDatasetEvents

useTIHistory

searchParams.test

searchParams

useTableUrlState

utils.test

utils

keyboardShortcutIdentifier

useSelection

useToggleGroups

types

index.d

theme

api-generated

react-table-config.d

graph

handleError.test

index.test

handleError

URLSearchParamWrapper

useErrorToast

useKeysPress

useOffsetTop

Practice Create an IAM User with permissions

Quick overview of AWS EKS

Practice Set up an EC2 instance for Rancher

How to access your applications from the outside

Practice Create an EKS cluster with Rancher

Practice Deploy Nginx Ingress with Catalogs (Helm)

Practice Create an ECR repository

Practice Cleaning your AWS services

Practice Deploy and run Airflow with the Kubernetes Executor on EKS

How the logging system works in Airflow

Elasticsearch Reminder

Practice Monitoring your DAGs with Elasticsearch

Practice Storing your logs in AWS S3

Introduction to metrics

Practice Configuring Airflow with Elasticsearch

Airflow maintenance DAGs

Practice Monitoring Airflow with TIG stack

Practice Setting up custom logging

Practice Triggering alerts for Airflow with Grafana

Practice Hiding variables

Practice Rotating the Fernet Key

Practice Encrypting sensitive data with Fernet

What to expect from Airflow 2.0

Practice Password authentication and filter by owner

Practice RBAC UI

VIDEO The DockerOperator The basics and more

Backfill your DAGs in Airflow like a PRO