A framework for using LSTMs to detect anomalies in multivariate time series data. Includes spacecraft anomaly data and experiments from the Mars Science Laboratory and SMAP missions.

Overview

Telemanom (v2.0)

v2.0 updates:

  • Vectorized operations via numpy
  • Object-oriented restructure, improved organization
  • Merge branches into single branch for both processing modes (with/without labels)
  • Update requirements.txt and Dockerfile
  • Updated result output for both modes
  • PEP8 cleanup

Anomaly Detection in Time Series Data Using LSTMs and Automatic Thresholding

License

Telemanom employs vanilla LSTMs using Keras/Tensorflow to identify anomalies in multivariate sensor data. LSTMs are trained to learn normal system behaviors using encoded command information and prior telemetry values. Predictions are generated at each time step and the errors in predictions represent deviations from expected behavior. Telemanom then uses a novel nonparametric, unsupervised approach for thresholding these errors and identifying anomalous sequences of errors.

This repo along with the linked data can be used to re-create the experiments in our 2018 KDD paper, "Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding", which describes the background, methodologies, and experiments in more detail. While the system was originally deployed to monitor spacecraft telemetry, it can be easily adapted to similar problems.

Getting Started

Clone the repo (only available from source currently):

git clone https://github.com/khundman/telemanom.git && cd telemanom

Configure system/modeling parameters in config.yaml file (to recreate experiment from paper, leave as is). For example:

  • train: True if True, a new model will be trained for each input stream. If False (default) existing trained model will be loaded and used to generate predictions
  • predict: True Generate new predictions using models. If False (default), use existing saved predictions in evaluation (useful for tuning error thresholding and skipping prior processing steps)
  • l_s: 250 Determines the number of previous timesteps input to the model at each timestep t (used to generate predictions)

To run via Docker:

docker build -t telemanom .

# rerun experiment detailed in paper or run with your own set of labeled anomlies in 'labeled_anomalies.csv'
docker run telemanom -l labeled_anomalies.csv

# run without labeled anomalies
docker run telemanom

To run with local or virtual environment

From root of repo, curl and unzip data:

curl -O https://s3-us-west-2.amazonaws.com/telemanom/data.zip && unzip data.zip && rm data.zip

Install dependencies using python 3.6+ (recommend using a virtualenv):

pip install -r requirements.txt

Begin processing (from root of repo):

# rerun experiment detailed in paper or run with your own set of labeled anomlies
python example.py -l labeled_anomalies.csv

# run without labeled anomalies
python example.py

A jupyter notebook for evaluating results for a run is at telemanom/result_viewer.ipynb. To launch notebook:

jupyter notebook telemanom/result-viewer.ipynb

Plotly is used to generate interactive inline plots, e.g.:

drawing2

Data

Using your own data

Pre-split training and test sets must be placed in directories named data/train/ and data/test. One .npy file should be generated for each channel or stream (for both train and test) with shape (n_timesteps, n_inputs). The filename should be a unique channel name or ID. The telemetry values being predicted in the test data must be the first feature in the input.

For example, a channel T-1 should have train/test sets named T-1.npy with shapes akin to (4900,61) and (3925, 61), where the number of input dimensions are matching (61). The actual telemetry values should be along the first dimension (4900,1) and (3925,1).

Raw experiment data

The raw data available for download represents real spacecraft telemetry data and anomalies from the Soil Moisture Active Passive satellite (SMAP) and the Curiosity Rover on Mars (MSL). All data has been anonymized with regard to time and all telemetry values are pre-scaled between (-1,1) according to the min/max in the test set. Channel IDs are also anonymized, but the first letter gives indicates the type of channel (P = power, R = radiation, etc.). Model input data also includes one-hot encoded information about commands that were sent or received by specific spacecraft modules in a given time window. No identifying information related to the timing or nature of commands is included in the data. For example:

drawing

This data also includes pre-split test and training data, pre-trained models, predictions, and smoothed errors generated using the default settings in config.yaml. When getting familiar with the repo, running the result-viewer.ipynb notebook to visualize results is useful for developing intuition. The included data also is useful for isolating portions of the system. For example, if you wish to see the effects of changes to the thresholding parameters without having to train new models, you can set Train and Predict to False in config.yaml to use previously generated predictions from prior models.

Anomaly labels and metadata

The anomaly labels and metadata are available in labeled_anomalies.csv, which includes:

  • channel id: anonymized channel id - first letter represents nature of channel (P = power, R = radiation, etc.)
  • spacecraft: spacecraft that generated telemetry stream
  • anomaly_sequences: start and end indices of true anomalies in stream
  • class: the class of anomaly (see paper for discussion)
  • num values: number of telemetry values in each stream

To provide your own labels, use the labeled_anomalies.csv file as a template. The only required fields/columns are channel_id and anomaly_sequences. anomaly_sequences is a list of lists that contain start and end indices of anomalous regions in the test dataset for a channel.

Dataset and performance statistics:

Data

SMAP MSL Total
Total anomaly sequences 69 36 105
Point anomalies (% tot.) 43 (62%) 19 (53%) 62 (59%)
Contextual anomalies (% tot.) 26 (38%) 17 (47%) 43 (41%)
Unique telemetry channels 55 27 82
Unique ISAs 28 19 47
Telemetry values evaluated 429,735 66,709 496,444

Performance (with default params specified in paper)

Spacecraft Precision Recall F_0.5 Score
SMAP 85.5% 85.5% 0.71
Curiosity (MSL) 92.6% 69.4% 0.69
Total 87.5% 80.0% 0.71

Processing

Each time the system is started a unique datetime ID (ex. 2018-05-17_16.28.00) will be used to create the following

  • a results file (in results/) that extends labeled_anomalies.csv to include identified anomalous sequences and related info
  • a data subdirectory containing data files for created models, predictions, and smoothed errors for each channel. A file called params.log is also created that contains parameter settings and logging output during processing.

As mentioned, the jupyter notebook telemanom/result-viewer.ipynb can be used to visualize results for each stream.

Citation

If you use this work, please cite:

  title={Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding},
  author={Hundman, Kyle and Constantinou, Valentino and Laporte, Christopher and Colwell, Ian and Soderstrom, Tom},
  journal={arXiv preprint arXiv:1802.04431},
  year={2018}
}

License

Telemanom is distributed under Apache 2.0 license.

Contact: Kyle Hundman ([email protected])

Contributors

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Segmentation Transformer Implementation of Segmentation Transformer in PyTorch, a new model to achieve SOTA in semantic segmentation while using trans

Abhay Gupta 161 Dec 08, 2022
Source code for Task-Aware Variational Adversarial Active Learning

Contrastive Coding for Active Learning under Class Distribution Mismatch Official PyTorch implementation of ["Contrastive Coding for Active Learning u

27 Nov 23, 2022
An index of algorithms for learning causality with data

awesome-causality-algorithms An index of algorithms for learning causality with data. Please cite our survey paper if this index is helpful. @article{

Ruocheng Guo 2.3k Jan 08, 2023
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation in TensorFlow 2

Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation in TensorFlow 2 Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan

Phan Nguyen 1 Dec 16, 2021
PyTorch Implement for Path Attention Graph Network

SPAGAN in PyTorch This is a PyTorch implementation of the paper "SPAGAN: Shortest Path Graph Attention Network" Prerequisites We prefer to create a ne

Yang Yiding 38 Dec 28, 2022
This repository provides the code for MedViLL(Medical Vision Language Learner).

MedViLL This repository provides the code for MedViLL(Medical Vision Language Learner). Our proposed architecture MedViLL is a single BERT-based model

SuperSuperMoon 39 Jan 05, 2023
Real-Time Social Distance Monitoring tool using Computer Vision

Social Distance Detector A Real-Time Social Distance Monitoring Tool Table of Contents Motivation YOLO Theory Detection Output Tech Stack Functionalit

Pranav B 13 Oct 14, 2022
Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

Sidd Karamcheti 50 Nov 16, 2022
Doing the asl sign language classification on static images using graph neural networks.

SignLangGNN When GNNs 💜 MediaPipe. This is a starter project where I tried to implement some traditional image classification problem i.e. the ASL si

10 Nov 09, 2022
Source code for paper "ATP: AMRize Than Parse! Enhancing AMR Parsing with PseudoAMRs" @NAACL-2022

ATP: AMRize Then Parse! Enhancing AMR Parsing with PseudoAMRs Hi this is the source code of our paper "ATP: AMRize Then Parse! Enhancing AMR Parsing w

Chen Liang 13 Nov 23, 2022
Taming Transformers for High-Resolution Image Synthesis

Taming Transformers for High-Resolution Image Synthesis CVPR 2021 (Oral) Taming Transformers for High-Resolution Image Synthesis Patrick Esser*, Robin

CompVis Heidelberg 3.5k Jan 03, 2023
Keras-tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation(Unfinished)

Keras-FCN Fully convolutional networks and semantic segmentation with Keras. Models Models are found in models.py, and include ResNet and DenseNet bas

645 Dec 29, 2022
Dense Passage Retriever - is a set of tools and models for open domain Q&A task.

Dense Passage Retrieval Dense Passage Retrieval (DPR) - is a set of tools and models for state-of-the-art open-domain Q&A research. It is based on the

Meta Research 1.1k Jan 03, 2023
Code of Puregaze: Purifying gaze feature for generalizable gaze estimation, AAAI 2022.

PureGaze: Purifying Gaze Feature for Generalizable Gaze Estimation Description Our work is accpeted by AAAI 2022. Picture: We propose a domain-general

39 Dec 05, 2022
SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

Datasets | Website | Raw Data | OpenReview SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning Christopher

67 Dec 17, 2022
[CVPR 2022] TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing

TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing (CVPR 2022) This repository provides the official PyTorch impleme

Billy XU 128 Jan 03, 2023
September-Assistant - Open-source Windows Voice Assistant

September - Windows Assistant September is an open-source Windows personal assis

The Nithin Balaji 9 Nov 22, 2022
A MatConvNet-based implementation of the Fully-Convolutional Networks for image segmentation

MatConvNet implementation of the FCN models for semantic segmentation This package contains an implementation of the FCN models (training and evaluati

VLFeat.org 175 Feb 18, 2022
DockStream: A Docking Wrapper to Enhance De Novo Molecular Design

DockStream Description DockStream is a docking wrapper providing access to a collection of ligand embedders and docking backends. Docking execution an

AstraZeneca - Molecular AI 72 Jan 02, 2023