Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Overview

中文版文档

Airflow Extended API Plugin

Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

This plugin is available for airflow 2.x Version and extensible, as you can easily define your own API to execute any Airflow CLI command so that it fits your demand.

Current Supported Commands

The following commands are supported now, and more is coming.

  • airflow dags backfill
  • airflow tasks run
  • airflow tasks clear

Plugin Install

  1. Install the plugin via pip
  pip install airflow-extended-api
  1. Restart the Airflow Web Server

  2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

Usage

Examples

curl request example:

curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

Response Schema:

{
  "executed_command": "string",
  "exit_code": 0,
  "output_info": [
    "string"
  ],
  "error_info": [
    "string"
  ]
}

curl without Credentials data

Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "detail": null,
  "status": 401,
  "title": "Unauthorized",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
}

curl with wrong CLI Command

curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "error_info": [
    "Traceback (most recent call last):",
    "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
    "    sys.exit(main())",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
    "    args.func(args)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
    "    return func(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
    "    return f(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
    "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
    "    return [get_dag(subdir, dag_id)]",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
    "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
    "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
    ""
  ],
  "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
  "exit_code": 1,
  "output_info": [
    "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
    ""
  ]
}

Project Plan

  • support custom configuration

Links and References

You might also like...
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Streamify A data pipeline with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more! Description Objective The project will stre

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

✨ Udemy Coupon Finder For Discord. Supports Turkish & English Language.

Udemy Course Finder Bot | Udemy Kupon Bulucu Botu This bot finds new udemy coupons and sends to the channel. Before Setup You must have python = 3.6

An esoteric programming language that supports concurrency, regex, and web requests.

The Hofstadter Esoteric Programming Language Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's La

Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries.

Zeus Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries. Any flight worldwide, at your fing

Aero is an open source airplane intelligence tool. Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips.

Aero Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips. Features Main : Flight lookup Aircraft lookup

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.
Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.

MetaCall Polyglot REPL Description This repository implements a Polyglot REPL which shares the state of the meta-object protocol between the REPLs. Us

A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.
A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.

December Welcome to the December blogging system's code repository! Introduction December is a minimalist personal blogging system that natively suppo

Releases(v1.1.2)
  • v1.1.2(Apr 22, 2022)

    中文版文档

    Airflow Extended API Plugin

    Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

    Features

    • 👏Available: Probably the only available CLI command plugin on Github that supports the Airflow 2.x version.
    • 🎉Extensible: Easily define your own API to execute any Airflow CLI command so that it fits your demand.

    Plugin Install

    1. Install the plugin via pip
      pip install airflow-extended-api
    
    1. Restart the Airflow Web Server

    2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

    Usage

    Examples

    curl request example:

    curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    Response Schema:

    {
      "executed_command": "string",
      "exit_code": 0,
      "output_info": [
        "string"
      ],
      "error_info": [
        "string"
      ]
    }
    

    curl without Credentials data

    Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

    curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "detail": null,
      "status": 401,
      "title": "Unauthorized",
      "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
    }
    

    curl with wrong CLI Command

    curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "error_info": [
        "Traceback (most recent call last):",
        "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
        "    sys.exit(main())",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
        "    args.func(args)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
        "    return func(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
        "    return f(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
        "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
        "    return [get_dag(subdir, dag_id)]",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
        "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
        "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
        ""
      ],
      "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
      "exit_code": 1,
      "output_info": [
        "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
        ""
      ]
    }
    

    Project Plan

    • [ ] support dags backfill cli command
    • [ ] support custom configuration

    Links and References

    Source code(tar.gz)
    Source code(zip)
Owner
Eric Cao
Stay hungry, Stay foolish。
Eric Cao
Provide Prometheus url_sd compatible API Endpoint with data from Netbox

netbox-plugin-prometheus-sd Provide Prometheus http_sd compatible API Endpoint with data from Netbox. HTTP SD is a new feature in Prometheus and not a

Felix Peters 66 Dec 19, 2022
An AI-powered device to stop people from stealing my packages.

Package Theft Prevention Device An AI-powered device to stop people from stealing my packages. Installation To install on a raspberry pi, clone the re

rydercalmdown 157 Nov 24, 2022
Software for visualization of RTStruct structures on CT images

This script is responsible for the operation of the program, it is responsible for both creating the GUI and the process of processing images from dicom files. The program is based on the use of the

Adam Piszczek 0 Jun 29, 2022
Stopmagic gives you the power of creating amazing Stop Motion animations faster and easier than ever before.

Stopmagic gives you the power of creating amazing Stop Motion animations faster and easier than ever before. This project is maintained by Aldrin Mathew.

Aldrin's Art Factory 67 Dec 31, 2022
World Happiness Report is a publication of the Sustainable Development Solutions Network

World-Happiness-Report We are going to visualise what are the factors and which

Shubh Almal 1 Jan 03, 2023
Automated Content Feed Curator

Gathers posts from content feeds, filters, formats, delivers to you.

Alper S. Soylu 2 Jan 22, 2022
A python library what works with numbers.

pynum A python library what works with numbers. Prime Prime class have everithing you want about prime numbers. check_prime The check_prime method is

Mohammad Mahdi Paydar Puya 1 Jan 07, 2022
A free website that keeps the people informed about housing and evictions.

Eviction Tracker Currently helping verify detainer warrant data for middle Tennessee - via Middle TN DSA - Red Door Collective Features Presents data

Red Door Collective 7 Dec 14, 2022
Convert Beat Saber maps to Tesla light shows!

Tesla x Beat Saber - Light Show Converter Convert Beat Saber maps to Tesla light shows! This project requires FFMPEG and all packages from requirement

HLVM 20 Dec 21, 2022
A promo calculator for sports betting odds.

Sportbetter Calculation Toolkit Parlay Calculator This is a quick parlay calculator that considers some of the common promos offered. It is used to id

Luke Bhan 1 Sep 08, 2022
Build a grocery store management application.

python_projects_grocery_webapp In this python project, we will build a grocery store management application. It will be 3 tier application, Front end:

codebasics 54 Dec 29, 2022
Code for ML, domain generation, graph generation of ABC dataset

This is the repository for codes for ML, domain generation, graph generation of Asymmetric Buckling Columns (ABC) dataset in the paper "Learning Mechanically Driven Emergent Behavior with Message Pas

Peerasait Prachaseree (Jeffrey) 0 Jan 28, 2022
The Zig programming language, packaged for PyPI

Zig PyPI distribution This repository contains the script used to repackage the releases of the Zig programming language as Python binary wheels. This

Zig Programming Language 100 Nov 04, 2022
Participants of Bertelsmann Technology Scholarship created an awesome list of resources and they want to share it with the world, if you find illegal resources please report to us and we will remove.

Participants of Bertelsmann Technology Scholarship created an awesome list of resources and they want to share it with the world, if you find illegal

Wissem Marzouki 29 Nov 28, 2022
An open-source systems and controls toolbox for Python3

harold A control systems package for Python=3.6. Introduction This package is written with the ambition of providing a full-fledged control systems s

Ilhan Polat 157 Dec 05, 2022
Herramienta para poder automatizar reuniones en Zoom.

Crear Reunión Zoom con Python Herramienta para poder automatizar reuniones en Zoom. Librerías Requeridas Nombre Comando PyAutoGui pip install pyautogu

JkDev 3 Nov 12, 2022
BOHB tune library template (included example)

BOHB-template 실행 방법 python main.py 2021-10-10 기준 tf keras 버전 (tunecallback 방식) 완료 tf gradienttape 버전 (train_iteration 방식) 완료 pytorch 버전은 구현 준비중 방법 소개

Seungwoo Han 5 Mar 24, 2022
solsim is the Solana complex systems simulator. It simulates behavior of dynamical systems—DeFi protocols, DAO governance, cryptocurrencies, and more—built on the Solana blockchain

solsim is the Solana complex systems simulator. It simulates behavior of dynamical systems—DeFi protocols, DAO governance, cryptocurrencies, and more—built on the Solana blockchain

William Wolf 12 Jul 13, 2022
Demo content - Automate your automation!

Automate-AAP2 Demo Content - Automate your automation! A fully automated Ansible Automation Platform. Context Installing and configuring Ansible Autom

0 Oct 27, 2022
A repository containing useful resources needed to complete the SUSE Scholarship Challenge #UdacitySUSEScholars #poweredbySUSE

SUSE-udacity-cloud-native-scholarship A repository containing useful resources needed to complete the SUSE Scholarship Challenge #UdacitySUSEScholars

Nandini Proothi 11 Dec 02, 2021