web application for flight log analysis & review

Overview

Flight Review

Build Status

This is a web application for flight log analysis. It allows users to upload ULog flight logs, and analyze them through the browser.

It uses the bokeh library for plotting and the Tornado Web Server.

Flight Review is deployed at https://review.px4.io.

Plot View

3D View

3D View

Installation and Setup

Requirements

Ubuntu

sudo apt-get install sqlite3 fftw3 libfftw3-dev

Note: Under some Ubuntu and Debian environments you might have to install ATLAS

sudo apt-get install libatlas3-base

macOS

macOS already provides SQLite3. Use Homebrew to install fftw:

brew install fftw

Installation

# After git clone, enter the directory
git clone --recursive https://github.com/PX4/flight_review.git
cd flight_review/app
pip install -r requirements.txt
# Note: preferably use a virtualenv

Setup

  • By default the app will load config_default.ini configuration file
  • You can override any setting from config_default.ini with a user config file config_user.ini (untracked)
  • Any setting on config_user.ini has priority over config_default.ini
  • Run setup_db.py to initialize the database.

Note: setup_db.py can also be used to upgrade the database tables, for instance when new entries are added (it automatically detects that).

Usage

For local usage, the server can be started directly with a log file name, without having to upload it first:

cd app
./serve.py -f <file.ulg>

To start the whole web application:

cd app
./serve.py --show

The plot_app directory contains a bokeh server application for plotting. It can be run stand-alone with bokeh serve --show plot_app (or with cd plot_app; bokeh serve --show main.py, to start without the html template).

The whole web application is run with the serve.py script. Run ./serve.py -h for further details.

Interactive Usage

The plotting can also be used interative using a Jupyter Notebook. It requires python knowledge, but provides full control over what and how to plot with immediate feedback.

  • Start the notebook
  • Locate and open the test notebook file testing_notebook.ipynb.
# Launch jupyter notebook
jupyter notebook testing_notebook.ipynb

Implementation

The web site is structured around a bokeh application in app/plot_app (app/plot_app/configured_plots.py contains all the configured plots). This application also handles the statistics page, as it contains bokeh plots as well. The other pages (upload, browse, ...) are implemented as tornado handlers in app/tornado_handlers/.

plot_app/helper.py additionally contains a list of log topics that the plot application can subscribe to. A topic must live in this list in order to be plotted.

Tornado uses a single-threaded event loop. This means all operations should be non-blocking (see also http://www.tornadoweb.org/en/stable/guide/async.html). (This is currently not the case for sending emails).

Reading ULog files is expensive and thus should be avoided if not really necessary. There are two mechanisms helping with that:

  • Loaded ULog files are kept in RAM using an LRU cache with configurable size (when using the helper method). This works from different requests and sessions and from all source contexts.
  • There's a LogsGenerated DB table, which contains extracted data from ULog for faster access.

Caching

In addition to in-memory caching there is also some on-disk caching: KML files are stored on disk. Also the parameters and airframes are cached and downloaded every 24 hours. It is safe to delete these files (but not the cache directory).

Notes about python imports

Bokeh uses dynamic code loading and the plot_app/main.py gets loaded on each session (page load) to isolate requests. This also means we cannot use relative imports. We have to use sys.path.append to include modules in plot_app from the root directory (Eg tornado_handlers.py). Then to make sure the same module is only loaded once, we use import xy instead of import plot_app.xy. It's useful to look at print('\n'.join(sys.modules.keys())) to check this.

Docker usage

This section explains how to work with docker.

Arguments

Edit the .env file according to your setup:

  • PORT - The number of port, what listen service in docker, default 5006
  • USE_PROXY - The set his, if you use reverse proxy (Nginx, ...)
  • DOMAIN - The address domain name for origin, default = *
  • CERT_PATH - The SSL certificate volume path
  • EMAIL - Email for challenging Let's Encrypt DNS

Paths

  • /opt/service/config_user.ini - Path for config
  • /opt/service/data - Folder where stored database
  • .env - Environment variables for nginx and app docker container

Build Docker Image

cd app
docker build -t px4flightreview -f Dockerfile .

Work with docker-compose

Run the following command to start docker container. Please modify the .env and add app/config_user.ini with respective stages.

Uncomment the BOKEH_ALLOW_WS_ORIGIN with your local IP Address when developing, this is for the bokeh application's websocket to work.

Development

docker-compose -f docker-compose.dev.yml up

Test Locally

Test locally with nginx:

docker-compose up

Remember to Change NGINX_CONF to use default_ssl.conf and add the EMAIL for production.

Production

htpasswd -c ./nginx/.htpasswd username
# here to create a .htpasswd for nginx basic authentication
chmod u+x init-letsencrypt.sh
./init-letsencrypt.sh

Contributing

Contributions are welcome! Just open a pull request with detailed description why the changes are needed, or open an issue for bugs, feature requests, etc...

Owner
PX4 Drone Autopilot
Professional Open Source Autopilot Stack
PX4 Drone Autopilot
script to generate HeN ipfs app exports of GLSL shaders

HeNerator A simple script to generate HeN ipfs app exports from any frag shader created with: GlslViewer GlslEditor The Book of Shaders glslCanvas VS

Patricio Gonzalez Vivo 22 Dec 21, 2022
An interactive UMAP visualization of the MNIST data set.

Code for an interactive UMAP visualization of the MNIST data set. Demo at https://grantcuster.github.io/umap-explorer/. You can read more about the de

grant 70 Dec 27, 2022
HW_02 Data visualisation task

HW_02 Data visualisation and Matplotlib practice Instructions for HW_02 Idea for data analysis As I was brainstorming ideas and running through databa

9 Dec 13, 2022
JupyterHub extension for ContainDS Dashboards

ContainDS Dashboards for JupyterHub A Dashboard publishing solution for Data Science teams to share results with decision makers. Run a private on-pre

Ideonate 179 Nov 29, 2022
CONTRIBUTIONS ONLY: Voluptuous, despite the name, is a Python data validation library.

CONTRIBUTIONS ONLY What does this mean? I do not have time to fix issues myself. The only way fixes or new features will be added is by people submitt

Alec Thomas 1.8k Dec 31, 2022
daily report of @arkinvest ETF activity + data collection

ark_invest daily weekday report of @arkinvest ETF activity + data collection This script was created to: Extract and save daily csv's from ARKInvest's

T D 27 Jan 02, 2023
A high-level plotting API for pandas, dask, xarray, and networkx built on HoloViews

hvPlot A high-level plotting API for the PyData ecosystem built on HoloViews. Build Status Coverage Latest dev release Latest release Docs What is it?

HoloViz 694 Jan 04, 2023
A guide for using Bootstrap 5 classes in Dash Bootstrap Components V1

dash-bootstrap-cheatsheet This handy interactive cheatsheet makes it easy to use the Bootstrap 5 classes with your Dash app made with the latest versi

10 Dec 22, 2022
Plot and save the ground truth and predicted results of human 3.6 M and CMU mocap dataset.

Visualization-of-Human3.6M-Dataset Plot and save the ground truth and predicted results of human 3.6 M and CMU mocap dataset. human-motion-prediction

Gaurav Kumar Yadav 5 Nov 18, 2022
BrowZen correlates your emotional states with the web sites you visit to give you actionable insights about how you spend your time browsing the web.

BrowZen BrowZen correlates your emotional states with the web sites you visit to give you actionable insights about how you spend your time browsing t

Nick Bild 36 Sep 28, 2022
Cartopy - a cartographic python library with matplotlib support

Cartopy is a Python package designed to make drawing maps for data analysis and visualisation easy. Table of contents Overview Get in touch License an

1.2k Jan 01, 2023
The implementation of the paper "HIST: A Graph-based Framework for Stock Trend Forecasting via Mining Concept-Oriented Shared Information".

The HIST framework for stock trend forecasting The implementation of the paper "HIST: A Graph-based Framework for Stock Trend Forecasting via Mining C

Wentao Xu 111 Jan 03, 2023
Generate the report for OCULTest.

Sample report generated in this function Usage example from utils.gen_report import generate_report if __name__ == '__main__': # def generate_rep

Philip Guo 1 Mar 10, 2022
Here are my graphs for hw_02

Let's Have A Look At Some Graphs! Graph 1: State Mentions in Congressperson's Tweets on 10/01/2017 The graph below uses this data set to demonstrate h

7 Sep 02, 2022
Dimensionality reduction in very large datasets using Siamese Networks

ivis Implementation of the ivis algorithm as described in the paper Structure-preserving visualisation of high dimensional single-cell datasets. Ivis

beringresearch 284 Jan 01, 2023
Python script for writing text on github contribution chart.

Github Contribution Drawer Python script for writing text on github contribution chart. Requirements Python 3.X Getting Started Create repository Put

Steven 0 May 27, 2022
With Holoviews, your data visualizes itself.

HoloViews Stop plotting your data - annotate your data and let it visualize itself. HoloViews is an open-source Python library designed to make data a

HoloViz 2.3k Jan 02, 2023
A Python function that makes flower plots.

Flower plot A Python 3.9+ function that makes flower plots. Installation This package requires at least Python 3.9. pip install

Thomas Roder 4 Jun 12, 2022
Sentiment Analysis application created with Python and Dash, hosted at socialsentiment.net

Social Sentiment Dash Application Live-streaming sentiment analysis application created with Python and Dash, hosted at SocialSentiment.net. Dash Tuto

Harrison 456 Dec 25, 2022
This component provides a wrapper to display SHAP plots in Streamlit.

streamlit-shap This component provides a wrapper to display SHAP plots in Streamlit.

Snehan Kekre 30 Dec 10, 2022