Installing & Setting Up Apache Airflow (Local & Cloud)

1
Installing Setting Up Apache Airflow (Local
Cloud)
2
Introduction to Apache Airflow

• What is Apache Airflow?
• An open-source platform to programmatically
  author, schedule, and monitor workflows.
• Key Features
• Dynamic pipeline generation using Python.
  Extensible architecture with plugins.
• Rich user interface for monitoring.

3
Installation Prerequisites
System Requirements Python 3.6 or later. pip
(Python package installer). Recommended Virtual
environment (venv or virtualenv) for isolated
Python environments.
4
Installing Airflow Locally
Step 1 Set up a virtual environment.
Step 2 Set the AIRFLOW_HOME environment variable
5
Installing Airflow Locally
step 3 pip install apache-airflow
6
Initializing and Starting Airflow
Initialize the database
Create an admin user
7
Initializing and Starting Airflow
Start the web server
Start the scheduler
8
Accessing the Airflow UI
Web Interface Navigate to http//localhost8080
in your browser. Log in using the admin
credentials created earlier. Features Monitor
DAGs (Directed Acyclic Graphs). Trigger and
manage tasks. View logs and task statuses.
9
Installing Airflow on the Cloud
Option 1 Using Cloud Providers (e.g., AWS, GCP,
Azure). Provision virtual machines or
containers. Install Airflow following similar
steps as local installation. Option 2 Managed
Services (e.g., Google Cloud Composer,
Astronomer). Simplify deployment and management.
Scalable and integrated with cloud services.
10
Best Practices

• Security
• Use strong passwords and secure connections.
• Scalability
• Use CeleryExecutor or KubernetesExecutor for
  distributed task execution.
• Monitoring
• Integrate with monitoring tools for alerts and
  metrics.
• Version Control
• Maintain DAGs in a version-controlled repository.

11
Troubleshooting Tips
Common Issues Port conflicts Change the default
port if 8080 is in use. Database errors Ensure
the database is initialized and
running. Scheduler not picking up DAGs Check the
DAGs folder path and file syntax. Resources Airfl
ow logs for detailed error messages. Community
forums and official documentation.
12
Conclusion
Recap Apache Airflow is a powerful tool for
workflow orchestration. Can be set up locally or
on the cloud based on requirements. Next
Steps Explore creating custom DAGs. Integrate
Airflow with other tools and services. Stay
updated with the latest Airflow features and best
practices.
13
Contact Online Training

• We Provide Online Training on Databricks and Big
  Data Technologies!
• ? Hands-on Training with Real-World Use Cases
• ? Live Sessions with Industry Experts
• ? Certification Guidance
• Website https//www.accentfuture.com/
• Contact us at contact_at_accentfuture.com
• Call US 91-9640001789
• Apache Airflow Course