Skip to content

Road to a data engineer

Creating extensible workflows

Road to a data engineer

Current situation and learning path
Data Engineering Landscape
Databases
Databases
- Data warehouse toolkit notes
- Database Design video course and notes
Decision Journal
Decision Journal
- Decision: Choice of cloud platform to learn
- Decision Journal Template
Google Cloud Platform
Google Cloud Platform
- Common GCP code snippets
- Data Engineering Certificate
  Data Engineering Certificate
- Qwiklabs
  Qwiklabs
Jobs and Roles
Jobs and Roles
Linux
Linux
- Data Analytics at the terminal
  Data Analytics at the terminal
  - Crash Course
    Crash Course
    
    Introduction and Motivation
    
    Fundamental Bash Commands
    
    Expanding the fundamentals for Data Analysis
    
    Downloading files with Bash
    
    Working with and manipulating text files
    
    BONUS - Common text file cleaning operations
    
    Setup data tools
    
    csvkit - a brief introduction
    
    08 getting data into sqlite
    
    Interacting with SQLite databases
    
    Common data cleaning and transforming tasks in SQL
    
    Replicating common spreadsheet tasks in SQL
    
    Building a pipeline as a bash script
    
    Additional resources
  - Deepnote Notebooks
    Deepnote Notebooks
    
    Environment setup
    
    Exploring terminal functionality for interacting with data
    
    GREP basics
    
    Getting a feel for the data
    
    Saving data to SQLite at the terminal
    
    CSV and SQLite tools overview
Projects
Projects
- Bitcoin Price Dashboard
  Bitcoin Price Dashboard
- fivethirtyeight data to BigQuery
  fivethirtyeight data to BigQuery
Python
Python
- Apache Airflow
- Data Engineering with python
  Data Engineering with python
- Google Colab
  Google Colab
  - Converting between video types
  - Backup of Domo Cards
  - Intro to Google Colab Notebooks
    Intro to Google Colab Notebooks
    
    What is Google Colab
    
    Using Google Drive as a filesystem in Google Colab
    
    Reading from and saving to Google Sheets
    
    Accessing Adobe Analytics data through the API
    
    Query BigQuery from within Google Colab
- Pycon US 2021
  Pycon US 2021
Resources
Resources

Creating extensible workflows

Designing a task API

A non trivial workflow system requires task API.

Made with Material for MkDocs