Airflow server setup

The following instructions will install Airflow on a fresh Digital Ocean droplet with Ubuntu 10 as the operating system

log into the droplet

ssh [email protected]

Get latest versions of packages

sudo apt-get update
sudo apt-get upgrade

install requirements

sudo apt-get install python3-pip
sudo apt-get install postgresql
sudo apt-get install redis
sudo apt-get install nginx

Enter postgres command line

sudo -u postgres psql

Create a database

CREATE DATABASE airflow;

Create a user

create user airflow with password 'BF3CGZVSm923SI3prMwn';

Exit the postgres promt

exit

Create user for OS

sudo adduser airflow

When prompted add the password: "BF3CGZVSm923SI3prMwn"

Change to airflow OS user

su airflow

Navigate to the home directory

cd

Install airflow with pip3

AIRFLOW_VERSION=2.0.1 

PYTHON_VERSION="$(python3 --version | cut -d " " -f 2 | cut -d "." -f 1-2)" 

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt" 

pip3 install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Also install psycopg2

pip3 install psycopg2-binary

Add .local/bin to $PATH so the airflow executable can be run

echo "export PATH=$PATH:$HOME/.local/bin" >> ~/.bashrc 
source ~/.bashrc

Test airflow is installed and what version we have

airflow version

Open the file with nano

nano airflow/airflow.cfg

Change the sql_alchemy_conn line to the following.

sql_alchemy_conn = postgresql://airflow:BF3CGZVSm923SI3prMwn@localhost:5432/airflow

Initialise metedata Postgres database

airflow db init

Create a user to log into Airflow web UI

airflow users create --username dexter --firstname donotpushthat --lastname button --role Admin --email [email protected]

When prompted use the password: "zV076wyMfF2f6xr6NtQR"

Start the airflow Webserver

airflow webserver -D

Start the airflow scheduler

airflow scheduler -D

Switch back to the root user

exit

Create a new config file

sudo nano /etc/nginx/sites-available/airflow

Enter the following details in the file

server { 
  listen 80; 
  listen [::]:80; 

  server_name airflow.marquinsmith.com 139.59.235.244; 

  location / {
  proxy_pass http://0.0.0.0:8080; 
    } 
  }

To allow Nginx to use that site's configuration, link the file into the /etc/nginx/sites-enabled directory.

sudo ln -s /etc/nginx/sites-available/airflow /etc/nginx/sites-enabled

Restart nginx to apply new settings

sudo systemctl restart nginx

In the /etc/nginx/nginx.conf file change server_names_hash_bucket_size to 256

nano /etc/nginx/nginx.conf