A tool for checking if the external data used in Flatpak manifests is still up to date

Overview

Flatpak External Data Checker

Tests Coverage Status Total alerts Language grade: Python CodeFactor

This is a tool for checking for outdated or broken links of external data in Flatpak manifests.


Motivation

Flatpak apps are built using external data (git checkouts, tarballs, simple files, etc.). A very specific case of this is the use of extra data, which works as a way to download third party binaries at installation time.

Of course, the links pointing to external data can become obsolete, so it is very important to account for and correct such issues. This is especially critical for the extra data, in which a broken link impedes the installation of the application.

This project offers ways to easily check or monitor the state of such links, as well as the suggestion of new versions.

It works by extracting all the external data of a Flatpak manifest and giving it to a collection of checkers, which will set up the right state and, possibly, new versions of each external data.

Use

The simplest use of this tool is by calling:

flatpak-external-data-checker MANIFEST_FILE

it should display messages about any broken or outdated external data.

Installation

This tool itself is available in flatpak format from Flathub. Install with

flatpak install --from https://dl.flathub.org/repo/appstream/org.flathub.flatpak-external-data-checker.flatpakref

And run with

flatpak run org.flathub.flatpak-external-data-checker MANIFEST_FILE

Running in a container

flatpak-external-data-checker is also avaiable as an OCI image from GitHub Container Registry.

You can use the run-in-container.sh helper script to set up needed CLI options for you and run the image using podman:

~/src/endlessm/flatpak-external-data-checker/run-in-container.sh \
    [ARGS …] \
    ~/src/flathub/com.example.App/com.example.App.json

Automatically submitting PRs

When run with the --update flag, this tool can commit any necessary changes to Git and open a GitHub pull request. In order to do this, it requires a GitHub access token, specified in the GITHUB_TOKEN environment variable.

Automatically merging PRs

The tool will also automatically merge previously opened pull request for unavailable (BROKEN) sources if the change has successfully passed CI checks and the token has sufficient privileges.

Automatically merging all submitted PRs, not just unavailable sources, from the tool can be forcefully enabled by setting automerge-flathubbot-prs to true in flathub.json, or it can be completely disabled by setting automerge-flathubbot-prs to false.

Changes to Flatpak manifests

For simple checks to see if a URL is broken, no changes are needed. However, you can add additional metadata to the manifest to allow the checker to discover new versions.

Some of the following checkers are able to determine upstream version number, and automatically add it to releases list in metainfo. To specify which source is the app upstream source, set property is-main-source to true in the checker metadata for that source.

URL checker

If the upstream vendor has an URL that redirects to the latest version of the application, you can add something like the following to check and update the URL for the latest version:

"x-checker-data": {
    "type": "rotating-url",
    "url": "http://example.com/last-version",
    "pattern": "http://example.com/foo-v([0-9.]+).tar.gz"
}

The version number for the latest version can be detected in two ways:

  • If the filename ends with .AppImage, the version number is extracted from the AppImage. (It is run in a bwrap sandbox.)
  • Otherwise, if "pattern" is specified in "x-checker-data", the given regular expression is matched against the full URL for the latest version, and the first match group is taken to be the version. (This follows the convention used by debian/watch files.)

HTML checker

Both the version number and the download URL will be gathered from a static HTML page which contains this information:

"x-checker-data": {
    "type": "html",
    "url": "https://www.example.com/download.html",
    "version-pattern": "The latest version is ([\\d\\.-]+)",
    "url-pattern": "(https://www.example.com/pub/foo/v([\\d\\.-]+)/foo.tar.gz)"
}

If the HTML page contains multiple versions with download links, set single pattern containing two nested match groups for both url and version:

(https://sourceforge.net/.+/qrupdate-([\\d\\.]+\\d).tar.gz)/download" } ">
"x-checker-data": {
    "type": "html",
    "url": "https://sourceforge.net/projects/qrupdate/rss",
    "pattern": "
   (https://sourceforge.net/.+/qrupdate-([\\d\\.]+\\d).tar.gz)/download"
}

To disable sorting and get first matched version/url, set sort-matches to false.

URL templates

The HTML checker also supports building the download URL using the retrieved version string, its components according to the Python LooseVersion class and semantic versioning fields:

"x-checker-data": {
    "type": "html",
    "url": "https://www.example.com/download.html",
    "version-pattern": "The latest version is ([\\d\\.-]*)",
    "url-template": "https://www.example.com/$version/v$version.tar.gz"
}
"x-checker-data": {
    "type": "html",
    "url": "https://www.example.com/download.html",
    "version-pattern": "The latest version is ([\\d\\.-]*)",
    "url-template": "https://www.example.com/$major.$minor/v$version.tar.gz"
}

If the placeholder is immediately followed by an underscore, you need to add braces:

"x-checker-data": {
    "type": "html",
    "url": "https://www.example.com/download.html",
    "version-pattern": "The latest version is ([\\d\\.-]*)",
    "url-template": "https://www.example.com/$version0.$version1/v${version0}_${version1}_version2.tar.gz"
}

Git checker

To check for latest git tag in corresponding git source repo, add checker metadata with type git and set tag-pattern to a regular expression with exactly one match group (the pattern will be used to extract version from tag):

"x-checker-data": {
    "type": "git",
    "tag-pattern": "^v([\\d.]+)$"
}

By default tags are sorted based on version number extracted from tag. To disable sorting and keep order from git ls-remote, set sort-tags to false.

If the project follows semver specification, you can set version-scheme property to semantic in order to use semantic version scheme for sorting. In this case, make sure that tag-pattern extracts only valid semver strings.

JSON checker

The JSON checker allows using jq to query JSON data with arbitrary schema to get version and download url.

To use the JSONChecker, specify JSON data URL, version query and url query (you can use $version variable got from the version query in url query):

{
    "type": "json",
    "url": "https://api.github.com/repos/stedolan/jq/releases/latest",
    "version-query": ".tag_name | sub(\"^jq-\"; \"\")",
    "url-query": ".assets[] | select(.name==\"jq-\" + $version + \".tar.gz\") | .browser_download_url"
}

for git type sources, specify tag query and, optionaly, commit and version queries:

{
    "type": "json",
    "url": "https://api.github.com/repos/stedolan/jq/releases/latest",
    "tag-query": ".tag_name",
    "version-query": "$tag | sub(\"^jq-\"; \"\")",
    "timestamp-query": ".published_at"
}

timestamp-query is optional, but if provided - must return a string with timestamp in ISO format.

See the jq manual for complete information about writing queries.

Debian repo checker

For the DebianRepoChecker, which deals only with deb packages, it can read the following metadata (add it to manifest element it refers to, e.g. where "type": "extra-data" is declared):

"x-checker-data": {
    "type": "debian-repo",
    "package-name": "YOUR_PACKAGE_NAME",
    "root": "ROOT_URL_TO_THE_DEBIAN_REPO",
    "dist": "DEBIAN_DIST",
    "component": "DEBIAN_COMPONENT"
}

Anitya (release-monitoring) checker

Anitya is an upstream release monitoring project by Fedora. It supports multiple backends for retrieving version information from different services, including GitHub, GitLab, Sourceforge, etc. To use the AnityaChecker, specify numeric project ID on release-monitoring.org and add a template for source download URL. Template syntax is the same as for the HTMLChecker:

"x-checker-data": {
    "type": "anitya",
    "project-id": 6377,
    "stable-only": false,
    "url-template": "https://github.com/flatpak/flatpak/releases/download/$version/flatpak-$version.tar.xz"
}

Set stable-only to true to retrieve latest stable version (as recognized by Anitya).

For git type sources, instead of url-template, set tag-template to derive git tag from version.

GNOME checker

Check for latest source tarball for a GNOME project.

"x-checker-data": {
    "type": "gnome",
    "name": "pygobject",
    "versions": {
        "<": "3.38.0"
    },
    "stable-only": true
}

Set stable-only to false to check for pre-releases, too.

You can also set version constraints in versions property (optional).
It should contain key-value pairs where key is the comparison operator (one of <, >, <=, >=, ==, !=), and the value is the version to compare with. So, {"<": "3.38.0", "!=": "3.37.1"} means "any version less than 3.38.0 except 3.37.1". All constraints must match simultaneously, i.e. if one doesn't match - version is rejected.

PyPI checker

Check for Python package updates on PyPI.

"x-checker-data": {
    "type": "pypi",
    "name": "Pillow"
}

By default it will check for source package (sdist package type). To check for binary package instead, set packagetype to bdist_wheel (only noarch wheels are supported currently).

JetBrains checker

Special checker that will check for available updates for JetBrains products:

"x-checker-data": {
    "type": "jetbrains",
    "code": "PRODUCT-CODE",
    "release-type": "release or eap (defaults to release)"
}

Snapcraft checker

Special checker that will check for available updates for Snapcraft packages:

"x-checker-data": {
    "type": "snapcraft",
    "name": "PACKAGE-NAME",
    "channel": "stable, beta, or any other tag the project uses"
}

Rust checker

Special checker that will check for available updates for Rust:

"x-checker-data": {
    "type": "rust",
    "package": "package name, for example: rust",
    "channel": "nightly, stable or beta",
    "target": "target triple, for example: x86_64-unknown-linux-gnu"
}

Chromium checker

Special checker that will check for available updates to the Chromium tarballs, as well as the toolchain binaries or sources used to build it.

"x-checker-data": {
    "type": "chromium",
    "component": "chromium, llvm-prebuilt, or llvm-git; defaults to chromium"
}

The following components are supported:

  • chromium: updates the Chromium tarball itself, used on URL-based sources (e.g. type: archive).
  • llvm-prebuilt: updates a tarball pointing to the official LLVM prebuilt toolchain archive matching the latest Chromium version. Used on URL-based sources.
  • llvm-git: updates a type: git source for its commit to point to the LLVM sources for the toolchain used by the latest Chromium version.

License and Copyright

License: GPLv2

Copyright © 2018–2019 Endless Mobile, Inc.

Owner
Flathub
Flathub is a build and distribution service for Flatpak applications. Its goal is to act as a central hub for making desktop applications available to users.
Flathub
3x - This Is 3x Friendlist Cloner Tools

3X FRIENDLIST CLONER TOOLS COMMAND $ apt update $ apt upgrade $ apt install pyth

MAHADI HASAN AFRIDI 2 Jan 17, 2022
For when you really need to rank things

Comparisonator For when you really need to rank things. Do you know that feeling when there's this urge deep within you that tells you to compare thin

Maciej Wilczyński 1 Nov 01, 2021
Painel simples com consulta de cep,CNPJ,placa e ip

Painel mpm Um painel simples com consultas de IP, CNPJ, CEP e PLACA Início 🌐 apt update && apt upgrade -y pkg i python git pip install requests Insta

8 Feb 27, 2022
RFDesign - Protein hallucination and inpainting with RoseTTAFold

RFDesign: Protein hallucination and inpainting with RoseTTAFold Jue Wang (juewan

139 Jan 06, 2023
Helps to arrange nodes

Relax brush for nodes, helps to arrange nodes easier.

336 Dec 15, 2022
Batch generate asset browser previews

When dealing with hundreds of library files it becomes tedious to mark their contents as assets. Using python to automate the process is a perfect fit

54 Dec 24, 2022
Python NZ COVID Pass Verifier/Generator

Python NZ COVID Pass Verifier/Generator This is quick proof of concept verifier I coded up in a few hours using various libraries to parse and generat

NZ COVID Pass Community 12 Jan 03, 2023
The next generation Canto RSS daemon

Canto Daemon This is the RSS backend for Canto clients. Canto-curses is the default client at: http://github.com/themoken/canto-curses Requirements De

Jack Miller 155 Dec 28, 2022
Скрипт позволяет заводить задачи в Панель мониторинга YouTrack на основе парсинга сайта safe-surf.ru

Скрипт позволяет заводить задачи в Панель мониторинга YouTrack на основе парсинга сайта safe-surf.ru

Bad_karma 3 Feb 12, 2022
Experimental proxy for dumping the unencrypted packet data from Brawl Stars (WIP)

Brawl Stars Proxy Experimental proxy for version 39.99 of Brawl Stars. It allows you to capture the packets being sent between the Brawl Stars client

4 Oct 29, 2021
Izy - Python functions and classes that make python even easier than it is

izy Python functions and classes that make it even easier! You will wonder why t

5 Jul 04, 2022
Collection of tools to be more productive in your work environment and to avoid certain repetitive tasks. 💛💙💚

Collection of tools to be more productive in your work environment and to avoid certain repetitive tasks. 💛💙💚

Raja Rakotonirina 2 Jan 10, 2022
🎅🏻 Helping santa understand ✨ python ✨

☃️ Advent of code 2021 ☃️ Helping santa understand ✨ python ✨

Fluffy 2 Dec 25, 2021
The repository is about 100+ python programming exercise problem discussed, explained, and solved in different ways

Break The Ice With Python A journey of 100+ simple yet interesting problems which are explained, solved, discussed in different pythonic ways Introduc

Abdullah Al Masud Tushar 2.2k Jan 04, 2023
Dot Browser is a privacy-conscious web browser with smarts built-in for protection against trackers and advertisments online.

🌍 Take back your privacy with Dot Browser, the privacy-conscious web browser that protects you from being tracked and monitored online.

Dot HQ 1k Jan 07, 2023
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.

Robots.txt tester With this script, you can enumerate all URLs present in robots.txt files, and test whether you can access them or not. Setup Clone t

Podalirius 32 Oct 10, 2022
In this project , I play with the YouTube data API and extract trending videos in Nigeria on a particular day

YouTubeTrendingVideosAnalysis In this project , I played with the YouTube data API and extracted trending videos in Nigeria on a particular day. This

1 Jan 11, 2022
Chess bot can play automatically as white or black on lichess.com, chess.com and any website using drag and drop to move pieces

Chessbot "Why create another chessbot ?" The explanation is simple : I did not find a free bot I liked online : all the bots I saw on internet are par

Dhimas Bagus Prayoga 2 Nov 11, 2021
Turn your IPad into a Screen-Slaver with 1 simple Pythonista script

ScreenSlaver Turn your IPad into a Screen-Slaver with 1 simple Pythonista script

6 Jul 09, 2022
python scripts - mostly automation scripts

python python scripts - mostly automation scripts You can set your environment in various ways bash #!/bin/bash python - locally on remote host #!/bi

Enyi 1 Jan 05, 2022