Know your customer pipeline in apache air flow

Know your customer pipeline in apache air flow

For a successful pipeline run take these steps:

  1. Run you Airflow server
  2. Admin -> connection -> create
  3. Triger the input_dag
  4. Before triggering the File_ process dag, move one of the JSON files into the tmp folder (for example the request_1411.json). In the program we must give the right name of the JSON file to load.
  5. Triger the File_process dag


For our data, when I read the assignment and what I understood was that I had to create a bunch of dummy data as JSON files with random dates. This can be done with this (FourthlinePython codesGenerating data) which I turned into a Dag in Airflow as


These where the next steps I took for transforming my data: • Read JSON file from a file • Transform the data into an integer • Check the fraud of the request • Agent checks the request • We create the SQLite database • Fill the database with the decision • Create a decision json file with the final output

Read JSON file: …

Check Fraud: …

Agent Check: …


Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

GIPHY App Key not set. Please check settings

WaterDo — The most satisfying and fun to-do list app

LAAVAN LYRICS – Loud | Ranjit Bawa