Skip to content

Chapter 2: Building Our Data Engineering Infrastructure

Installing and configuring the following tools:

  • Apache NiFi
  • Apache Airflow
  • ElasticSearch
  • Kibana
  • PostgreSQL

Installing Apache NiFi

# Download NiFi
curl '' --output 'nifi.tar.gz'
# Extract files from downloaded archive
tar xvzf 'nifi.tar.gz'
# if necessary install Java
sudo apt install openjdk-11-jre-headless

Setting up postgres database

Hosting a postgres database on digital ocean using one click install. Link

Insert iframe to instructions

Access droplet

ssh [email protected]

Access postgress

sudo -u postgres psql postgres

Set up password (password to use: jdW9IuFFztFJhHdMEuI3)

\password postgres

Create a new database

CREATE DATABASE dataengineering;

Then exit SSH.

A connection to the database can be made with the following details:

Field Detail
Port 5432
User postgres
Password jdW9IuFFztFJhHdMEuI3
Database 1 postgres
Database 2 dataengineering

Check if database is up

pg_isready -h -p 5432