Chapter 2: Building Our Data Engineering Infrastructure
Installing and configuring the following tools:
- Apache NiFi
- Apache Airflow
- ElasticSearch
- Kibana
- PostgreSQL
Installing Apache NiFi
# Download NiFi
curl 'https://archive.apache.org/dist/nifi/1.12.1/nifi-1.12.1-bin.tar.gz' --output 'nifi.tar.gz'
# Extract files from downloaded archive
tar xvzf 'nifi.tar.gz'
# if necessary install Java
sudo apt install openjdk-11-jre-headless
Setting up postgres database
Hosting a postgres database on digital ocean using one click install. Link
Insert iframe to instructions
Access droplet
ssh [email protected]
Access postgress
sudo -u postgres psql postgres
Set up password (password to use: jdW9IuFFztFJhHdMEuI3)
\password postgres
Create a new database
CREATE DATABASE dataengineering;
Then exit SSH.
A connection to the database can be made with the following details:
| Field | Detail |
|---|---|
| Host | 206.189.46.156 |
| Port | 5432 |
| User | postgres |
| Password | jdW9IuFFztFJhHdMEuI3 |
| Database 1 | postgres |
| Database 2 | dataengineering |
Check if database is up
pg_isready -h 206.189.46.156 -p 5432