One-stop-shop for docs and test coverage of dbt projects.

Overview

dbt-coverage

PyPI Downloads GitHub last commit PyPI - Python Version PyPI - Format License: MIT FOSSA Status

One-stop-shop for docs and test coverage of dbt projects.

Why do I need something like this?

dbt-coverage is to dbt what coverage.py and interrogate are to Python.

It is a single CLI tool which checks your dbt project for missing documentation and tests.

Keeping documentation and tests close to the actual SQL code that generates the final model is one of the best design choices of dbt. It ensures documentation is actually useful and tests are actually used. But how do you make adding those a habit in your dbt project?

That is exactly where dbt-coverage comes in. It will

  • Give you a better sense of the level of documentation and test coverage in your project;
  • Help your CI/CD pipeline make sure new changes include documentation and tests;
  • Let you quickly assess the documentation and tests of a new dbt project you get your hands on.

Still not convinced? Here are some more features:

  • zero-config: just install it and run it, there is nothing to set up
  • 🏁 minimal dependences: the only dependencies are click (already installed with dbt) and typer
  • 📦 very small: at ~480 SLOC, you can easily validate it works as advertised

Installation

pip install dbt-coverage

Usage

dbt-coverage comes with two basic commands: compute and compare. The documentation for the individual commands can be shown by using the --help option.

Compute

Compute coverage from target/catalog.json and target/manifest.json files found in a dbt project, e.g. jaffle_shop.

To choose between documentation and test coverage, pass doc or test as the CLI argument.

$ cd jaffle_shop
$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report coverage-doc.json  # Compute doc coverage, print it and write it to coverage-doc.json file

Coverage report
=====================================================================
jaffle_shop.customers                                  6/7      85.7%
jaffle_shop.orders                                     9/9     100.0%
jaffle_shop.raw_customers                              0/3       0.0%
jaffle_shop.raw_orders                                 0/4       0.0%
jaffle_shop.raw_payments                               0/4       0.0%
jaffle_shop.stg_customers                              0/3       0.0%
jaffle_shop.stg_orders                                 0/4       0.0%
jaffle_shop.stg_payments                               0/4       0.0%
=====================================================================
Total                                                 15/38     39.5%

$ dbt-coverage compute test --cov-report coverage-test.json  # Compute test coverage, print it and write it to coverage-test.json file

Coverage report
=====================================================================
jaffle_shop.customers                                  1/7      14.3%
jaffle_shop.orders                                     8/9      88.9%
jaffle_shop.raw_customers                              0/3       0.0%
jaffle_shop.raw_orders                                 0/4       0.0%
jaffle_shop.raw_payments                               0/4       0.0%
jaffle_shop.stg_customers                              1/3      33.3%
jaffle_shop.stg_orders                                 2/4      50.0%
jaffle_shop.stg_payments                               2/4      50.0%
=====================================================================
Total                                                 14/38     36.8%

Compare

Compare two coverage.json files generated by the compute command. This is useful to ensure that the coverage does not drop while making changes to the project.

$ dbt-coverage compare coverage-after.json coverage-before.json

# Coverage delta summary
              before     after            +/-
=============================================
Coverage      39.47%    38.46%         -1.01%
=============================================
Tables             8         8          +0/+0
Columns           38        39          +1/+0
=============================================
Hits              15        15          +0/+0
Misses            23        24          +1/+0
=============================================

# New misses
=========================================================================
Catalog                         15/38   (39.47%)  ->    15/39   (38.46%) 
=========================================================================
- jaffle_shop.customers          6/7    (85.71%)  ->     6/8    (75.00%) 
-- new_col                       -/-       (-)    ->     0/1     (0.00%) 
=========================================================================

Combined use-case

$ cd my-dbt-project

$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report before.json --cov-fail-under 0.5  # Fail if coverage is lower than 50%

# Make changes to the dbt project, e.g. add some columns to the DWH, document some columns, etc.

$ dbt run  # Materialize the changed models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report after.json --cov-fail-compare before.json  # Fail if the current coverage is lower than coverage in before.json
$ dbt-coverage compare after.json before.json  # Generate a detailed coverage delta report

Related packages

License

Licensed under the MIT license (see LICENSE.md file for more details).

FOSSA Status

Comments
  • Allow model filtering

    Allow model filtering

    Issue

    • Closes https://github.com/slidoapp/dbt-coverage/issues/24:

    "Please add options in the CLI to include and exclude models to filter out the checks in some of the models or a path."

    Summary

    I added the ability to perform compute commands on only a subset of tables by adding a --model-path-filter option. This means that a subset of models can be selected based upon their original_file_path value (taken from the manifest.json artifact).

    This functionality means that dbt-coverage can now be used in monolithic dbt projects which contain sub-projects owned by different teams. Before adding model selection functionality, using dbt-coverage would not have been useful/advisable in such a structure because another, unrelated team may decrease the overall coverage, which can then block PR merging (should dbt-coverage have been integrated as part of a CI/CD pipeline for example).

    • See example of added functionality from updated README.md below:
    $ cd jaffle_shop
    $ dbt run  # Materialize models
    $ dbt docs generate  # Generate catalog.json and manifest.json
    $ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file
    
    Coverage report
    ======================================================
    jaffle_shop.stg_customers              0/3       0.0%
    jaffle_shop.stg_orders                 0/4       0.0%
    jaffle_shop.stg_payments               0/4       0.0%
    ======================================================
    Total                                  0/11      0.0%
    
    $ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/orders.sql --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file
    
    Coverage report
    ======================================================
    jaffle_shop.orders                     0/9       0.0%
    jaffle_shop.stg_customers              0/3       0.0%
    jaffle_shop.stg_orders                 0/4       0.0%
    jaffle_shop.stg_payments               0/4       0.0%
    ======================================================
    Total                                  0/20      0.0%
    

    Note: this is a relatively 'rough' solution and there are likely many improvements that could be made to my code / far more elegant implementations that would achieve the same functionality. Please, feel free to suggest changes!

    Testing

    • I have tested these changes on dbt's jaffle_shop 'testing project' and have not encountered issues so far.
    opened by followingell 5
  • Custom run artifacts path

    Custom run artifacts path

    Goal

    Give more flexibility to user who have their manifest.json & catalog.json files not stored in /target, which is mainly the case for user using dbt cloud (need to retrieve the file by api call), adding a new parameter to use a custom path allow user not to create an unnecessary targetdir in their cicd pipeline

    Description

    • Add run_artifacts_dir : Path parameter to load_manifest and load_catalog
    • Add optional run_artifacts_dir parameter to cli
    opened by Fszta 5
  • Support multiple coverage doc/test save formats

    Support multiple coverage doc/test save formats

    From my understanding we can only save .json files as a result of dbt-coverage compute.

    It would be nice if we could pick other formats, for example .csv That way we can put those files as seeds and write to the db.

    opened by rloredo 5
  • dbt 1.0

    dbt 1.0

    • Support dbt 1.0 and drop support for dbt<1
    • Warn when using with unsupported manifest.json version

    @mrshu, I acknowledged the changes in the new v4 manifest.json format introduced by dbt 1.0 and by that the support for v3 dropped. Since the difference between the formats is only tiny, do you think it would make sense to support also v3 manifest.json (used in dbt<1) by having 2 methods for parsing the manifest, one for v3 and one for v4?

    Closes #26.

    opened by sweco 3
  • Tabulate doc coverage output using markdown

    Tabulate doc coverage output using markdown

    Resolves #46

    What has changed?

    Introduces a new argument called cov_format that accepts string (default) or markdown for the compute doc command.

    Outputs the results to the terminal in either the default string format, or in a markdown friendly format

    Example

    dbt-coverage compute doc --cov-format markdown
    

    image

    results in:

    image

    opened by vvvito 2
  • Make error more informative if target/manifest.json doesn't exist

    Make error more informative if target/manifest.json doesn't exist

    Planning to submit a PR, just wanted to tee this up first to document the need

    Steps to reproduce:

    pip install dbt-coverage==0.2.2
    git clone https://github.com/dbt-labs/jaffle_shop
    git checkout b0b77aa
    dbt-coverage compute doc
    

    Line of code throwing the error: https://github.com/slidoapp/dbt-coverage/blob/9f97e6c8fc24e83babc51c2b0584afc88e45308c/dbt_coverage/init.py#L507

    Error message:

    INFO:root:Loading catalog and manifest files from project dir: .
    Traceback (most recent call last):
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/bin/dbt-coverage", line 8, in <module>
        sys.exit(app())
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
        return callback(**use_params)  # type: ignore
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 660, in compute
        return do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 616, in do_compute
        catalog = load_files(project_dir)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 533, in load_files
        catalog = load_catalog(project_dir)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 507, in load_catalog
        with open(project_dir / 'target/catalog.json') as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'target/catalog.json'
    
    opened by bbrewington 2
  • Add support for Manifest v6

    Add support for Manifest v6

    opened by reegs 2
  • Take into account seeds

    Take into account seeds

    It seems that seeds are not considered while reading manifest.json. As a result, the following error pops while running dbt-coverage compute doc

    (venv_dbt) ➜  myCodeFolder git:(master) dbt-coverage compute doc                                
    INFO:root:Loading catalog and manifest files from project dir: .
    INFO:root:Successfully loaded 200 tables from catalog
    Traceback (most recent call last):
      File "/Users/victor/myCodeFolder/venv_dbt/bin/dbt-coverage", line 8, in <module>
        sys.exit(app())
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
        return self.main(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1062, in main
        rv = self.invoke(ctx)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 763, in invoke
        return __callback(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
        return callback(**use_params)  # type: ignore
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 604, in compute
        do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 562, in do_compute
        catalog = load_files(project_dir)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
        manifest = load_manifest(project_dir)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
        manifest = Manifest.from_nodes(manifest_nodes)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 121, in from_nodes
        table_name = id_to_table_name[table_id]
    KeyError: 'seed.dbt_project.browsers'
    

    Versions:

    dbt-core==0.21.0
    dbt-coverage==0.1.7
    

    Python 3.8

    opened by vdeb 2
  • KeyError: 'column_name' ?

    KeyError: 'column_name' ?

    # dbt build
    ...
    Done. PASS=23 WARN=0 ERROR=1 SKIP=1 TOTAL=25
    # dbt docs generate
    Running with dbt=0.21.1-rc1
    ...
    $ pip install dbt-coverage
    Successfully installed dbt-coverage-0.1.7 typer-0.4.0
    $ dbt-coverage compute doc 
    INFO:root:Loading catalog and manifest files from project dir: .
    INFO:root:Successfully loaded 17 tables from catalog
    
        catalog = load_files(project_dir)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
        manifest = load_manifest(project_dir)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
        manifest = Manifest.from_nodes(manifest_nodes)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 122, in from_nodes
        column_name = node['column_name'] or node['test_metadata']['kwargs']['column_name'] \
    KeyError: 'column_name'
    
    opened by chapmanjacobd 2
  • Fix matching of entities between manifest and catalog

    Fix matching of entities between manifest and catalog

    This fixes two issues:

    1. Some systems (e.g. Snowflake) uppercase all table and column names. In accordance with the dbt-docs official source code, we lowercase all table and column names before matching them.
    2. There can be multiple tables with the same name but in different schemas. We therefore use schema_name.table_name to distinguish tables.
    opened by sweco 1
  • Add contents

    Add contents

    What was done

    • Copy contents of dbt_docs_coverage.py verbatim to __init__.py.
    • Add README with basic documentation.
    • Create Python package using poetry.
    • Register dbt_coverage.__init__:app as a CLI script with name dbt-coverage.

    Review objectives

    • Please review README.
    • Please review the poetry files.
    • Ignore __init__.py, it was copied verbatim.
    opened by sweco 1
  • Display coverage doc/test result on gitlab badges

    Display coverage doc/test result on gitlab badges

    Hi team,

    I am wondering whether it is already possible for me to display the coverage doc/test result as a percentage on gitlab badges. Please see similar example for python unit test below image

    Thank you

    opened by kelvinheng92 3
  • Clean up `len(model_path_filter) >= 1` once a Typer issue gets resolved

    Clean up `len(model_path_filter) >= 1` once a Typer issue gets resolved

    Originally I had used is not None as you say, however, this doesn't work due to a known issue with typer: https://github.com/tiangolo/typer/issues/410 whereby [an] "Empty tuple [is] returned for Option instead of expected None when no values passed". As such, I have to use len() since until the issue is resolved model_path_filter will never be None.

    ~~Since you asked the question it's obvious that I could make the reason for my choice clearer, WDYT to the below suggestion?~~

    Originally posted by @followingell in https://github.com/slidoapp/dbt-coverage/pull/45#discussion_r1012694530

    good first issue 
    opened by mrshu 2
  • Support name / identifier format within sources

    Support name / identifier format within sources

    The script does not currently recognize tests applied to a source that is set-up using an identifier rather than just the raw as being associated with the source.

    I think, but am not certain, that this might be related to #30

    opened by reegs 1
  • could you add ability produce a coverage report in cobertura format

    could you add ability produce a coverage report in cobertura format

    To be able to retrieve dbt-coverage output in gitlab, a cobertura format is needed.

    See: https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscoverage_report

    Could you add a command line argument to choose output format ? (json or xml-cobertura)

    opened by cedric-orange 1
  • support generic tests from dbt_utils when calculating coverage

    support generic tests from dbt_utils when calculating coverage

    Our project is relying on tests defined in dbt_utils - is there a way of adding support for these as well?

    https://github.com/dbt-labs/dbt-utils#unique_combination_of_columns-source

    opened by aa-matthias 5
Releases(v0.3.2)
Owner
Slido
Slido is an audience interaction platform for meetings and events. Public repositories
Slido
The fundamentals of Python!

The fundamentals of Python Author: Mohamed NIANG, Staff ML Scientist Presentation This repository contains notebooks on the fundamentals of Python. Th

Mohamed NIANG 1 Mar 15, 2022
FBChecker Account using python , package requests and web old facebook

fbcek FBChecker Account using python , package requests and web old facebook using python 3.x apt upgrade -y apt update -y pkg install bash -y pkg ins

XnuxersXploitXen 5 Dec 24, 2022
Digitales Raumbuch

Helios Digitales Raumbuch Settings Moved to settings. Basic Commands Setting Up Your Users To create a normal user account, just go to Sign Up and fil

1 Nov 19, 2021
Voldemort's Python import helper

importmagician Voldemort's Python import helper pip install importmagician Import from uninstalled Python directories Say you have a directory (relat

Zhengyang Feng 4 Mar 09, 2022
A self contained invitation management system for gatekeeping.

Invitease Description A self contained invitation management system for gatekeeping. Purpose Serves as a focal point for inviting guests to a venue pr

מעגן מיכאל 7 Jul 19, 2022
Architecture example simulator

SCADA architecture Example of a SCADA-like console application, used to serve as a minimal example of a standard architecture of an IIoT system. Insta

1 Nov 06, 2021
Margin Calculator - Personally tailored investment tool

Margin Calculator - Personally tailored investment tool

1 Jul 19, 2022
Keyboard Layout Change - Extension for Ulauncher

Keyboard Layout Change - Extension for Ulauncher

Marco Borchi 4 Aug 26, 2022
Desafio Final do Mod1 do Bootcamp EDC - v2 usando a RAIS

IGTI - Bootcamp Engenheiro de Dados Cloud Módulo 1 - Desafio Final - RAIS 2020 Código do Desafio Final V2 do Bootcamp Engenheiro de Dados Cloud do IGT

Neylson Crepalde 17 Nov 02, 2022
tidevice can be used to communicate with iPhone device

h 该工具能够用于与iOS设备进行通信, 提供以下功能 截图 获取手机信息 ipa包的安装和卸载 根据bundleID 启动和停止应用 列出安装应用信息 模拟Xcode运行XCTest,常用的如启动WebDriverAgent测试

Alibaba 1.8k Dec 30, 2022
Ballcone is a fast and lightweight server-side Web analytics solution.

Ballcone Ballcone is a fast and lightweight server-side Web analytics solution. It requires no JavaScript on your website. Screenshots Design Goals Si

Dmitry Ustalov 49 Dec 11, 2022
Plugin to generate BOM + CPL files for JLCPCB

KiCAD JLCPCB tools Plugin to generate all files necessary for JLCPCB board fabrication and assembly Gerber files Excellon files BOM file CPL file Furt

bouni 566 Dec 29, 2022
The Blinker Herald includes helpers to easily emit signals using the excellent blinker library.

Blinker Herald The Blinker Herald includes helpers to easily emit signals using the excelent blinker library. Decorate a function or method with @blin

SatelliteQE 7 Nov 03, 2022
A class to draw curves expressed as L-System production rules

A class to draw curves expressed as L-System production rules

Juna Salviati 6 Sep 09, 2022
A price calculator for multiple things

Price Calculator A price calculator for multiple things Example I have 0.0567kg diamond. The price of diamond in kg is: $4500. Then it says: The price

Abel 1 Nov 26, 2021
App to decide weekly winners in H2H 1 Win (9 Cat)

Fantasy Weekly Winner for H2H 1 Win (9 Cat) Yahoo Fantasy API Read

Sai Atmakuri 1 Dec 31, 2021
Random Turkish name generator with realistic probabilities.

trnames Random Turkish name generator with realistic probabilities. Based on Trey Hunner's names package. Installation The package can be installed us

Kaan Öztürk 20 Jan 02, 2023
NORETURN is an esoteric programming language, based around the idea of not going back

NORETURN NORETURN is an esoteric programming language, based around the idea of not going back Concept Program coded in noreturn runs over one array,

1 Dec 15, 2021
Python script to commit to your github for a perfect commit streak. This is purely for education purposes, please don't use this script to do bad stuff.

Daily-Git-Commit Commit to repo every day for the perfect commit streak Requirments pip install -r requirements.txt Setup Download this repository. Cr

JareBear 34 Dec 14, 2022
Tethered downgrade 64-bit iDevices vulnerable to checkm8

ra1nstorm Tethered downgrade 64-bit iDevices vulnerable to checkm8 Since the purpose of this tool is to tethered downgrade a device, after restoring p

mini_exploit 65 Nov 08, 2022