Python parser for DTED data.

Related tags

Deep Learningdted
Overview

DTED Parser

This is a package written in pure python (with help from numpy) to parse and investigate Digital Terrain Elevation Data (DTED) files. This package is tested to work on Shuttle Radar Topography Mission (SRTM) DTED files (as far as I can tell these are the only publicly available DTED files). This can be used as a library to parse these files into numpy arrays and additionally exposes a CLI that can be used to investigate individual DTED files.

For more information and resources about the DTED file format see the end of the README.

How to install

You can install this as a normal python package using pip

pip install dted

How to use

The following example code will parse DTED file checked into this repository for testing.

As a library

Parsing a DTED file into a numpy array is as simple as:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
assert isinstance(tile.data, np.ndarray)

Additionally you can access the metadata of the DTED file (the User Header Label, Data Set Identification, and Accuracy Description records) easily.

from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
print(tile.dsi.south_west_corner)

Parsing entire DTED files has been heavily optimized, but does still take a little bit of time. On my machine (2014 MacbookPro) parsing the 25MB example file take about 120 ms. However, if you only need to look up specific terrain elevations within a DTED file you do not need to parse the entire file. Doing the following takes <1ms on my machine:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
print(tile.get_elevation(LatLon(latitude=41.5, longitude=-70.5)))

If for some reason you really need to eek out every bit of performance and you thoroughly trust your DTED data, you speed up the data parsing by skipping the checksum verification. Doing the following takes about 75 ms on my machine:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
tile.load_data(perform_checksum=False)

assert isinstance(tile.data, np.ndarray)

The final functionality the dted.Tile class offers is to easily check if a coordinate location is contained within the DTED file. This also does not require that the DTED data is fully loaded into memory:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)

assert LatLon(latitude=41.5, longitude=-70.25) in tile

As a CLI

Installing this package into an activated virtual environment also exposes the dted terminal command. This provides three pieces of functionality:

  1. See report of the metadata of the DTED file.
  2. Lookup terrain elevation at a specific point within the DTED file.
  3. Display and ASCII representation of the DTED file in your terminal.

To get a report of the file metadata:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 
File Path:          test/data/n41_w071_1arc_v3.dt2 (24 MB)
Product Level:      DTED2
Security Code:      U
Compilation Date:   02/2000
Maintenance Date:   
Datums (V/H):       E96/WGS84

    (42.0N,71.0W)      (42.0N,70.0W)
          NW --------------- NE     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          SW --------------- SE     
    (41.0N,71.0W)      (41.0N,70.0W)

Origin:                 (41.0N,71.0W)
Resolution (lat/lon):   1.0"/1.0"
Accuracy (V/H):         6m/13m

To lookup terrain elevation at a specific point:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --location 41.7 -70.4
51.0 meters

To display the DTED file in your terminal:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --display

This will attempt to create an ASCII representation of the DTED file within your terminal at the best resolution possible. Increasing the size of your terminal window or zooming out your terminal window will increase the resolution of the chart:

Normal Resolution Image

High Resolution Image

Why did I add this feature? Why not?

If you want to plot this data like a sane person, you can use the following example code with the matplotlib package (not included)

import matplotlib.pyplot as plt
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
plt.imshow(tile.data.T[::-1], cmap="hot")

The DTED file format

This parser was created using the specification provided here:

https://www.dlr.de/eoc/Portaldata/60/Resources/dokumente/7_sat_miss/SRTM-XSAR-DEM-DTED-1.1.pdf

Some things to be aware of with the DTED file format:

  1. Some DTED files contain "void" values for data points where elevation data is not known (such as over bodies of water). An example of such a file can be found at test/data/n00_e006_3arc_v2.dt1. This package will emit a warning if void data is found, and the definition of the void value can be found in dted.definitions.VOID_DATA_VALUE.
  2. The DTED data is structured along longitudinal lines. Therefore, when accessing the data within the numpy array the rows correspond to longitude and the columns correspond to latitude. This may seem backwards to your intuition, i.e. you would access the elevation at a coordinate point with tile.data[longitude_index, latitude_index].
  3. Elevation within the DTED file is encoded using "signed magnitude" notation. This has no effect on a user of this package interacting with the parsed terrain elevation data, but it does slow down the parsing of this data as I do not know of an optimized method of parsing signed magnitude data in python. If someone knows how to do this this parsing library could become even faster.

Where to find DTED data

Publicly available DTED data is relatively hard to find and access, but it can be done. The DTED files I used for testing and developing this package come from https://earthexplorer.usgs.gov/.

This EarthExplorer app provided by the USGS provides an interface to download many types of terrain data, including the SRTM DTED data. However, you need to make an account with them in order to perform and I'm unsure of a way to use their machine-to-machine API to automate downloading data.

Contributing

Contributions are absolutely encouraged! To develop on this project you need to install the poetry package manager.

Clone the repo:

[email protected]$ git clone https://github.com/bbonenfant/dted

Create and activate the virtual environment:

[email protected]$ poetry install && source .venv/bin/activate

To run the tests:

(.venv) [email protected]$ pytest .

If you are getting BLACK errors from pytest, run the black code formatter:

(.venv) [email protected]$ black .
Comments
  • Areas above 50° or below -50° Latitude

    Areas above 50° or below -50° Latitude

    Hello,

    I am trying to use the Tile(dted_file) on SRTM 1arc DTED files. The publicly available SRTM 1arc data is actually not 1arc by 1arc but 2arc by 1arc, as soon as you are working with areas above 50° Latitude.

    This causes the Tile() method to fail with the error: dted.errors.InvalidFileError: Checksum failed for data block

    When I try to ignore the checksum with load_data(perform_checksum=False), it fails with the error: dted.errors.InvalidFileError: All data blocks within a DTED file must begin with 170. Found: 1

    Do I just have to do something differently or is this something you could fix? It works great otherwise!

    Thank you for the help.

    opened by StefanBregenzer 10
  • Error in calculating latitude and longitude indices

    Error in calculating latitude and longitude indices

    I receive drastically different elevation values for the same location when using DTED levels 0, 1, and 2. An area that I know to be approximately 735m MSL reports the following:

    ~/Desktop/dted/w117$ dted n34.dt0 --location 34.353932 -116.295523
    1035.0 meters
    ~/Desktop/dted/w117$ dted n34.dt1 --location 34.353932 -116.295523
    2557.0 meters
    ~/Desktop/dted/w117$ dted n34.dt2 --location 34.353932 -116.295523
    733.0 meters
    

    In looking at the source code, I believe the calculations for the latitude and longitude indices in the DTED data are wrong:

    I believe this:

            lat_interval, lon_interval = self.dsi.latitude_interval, self.dsi.longitude_interval
            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) / lat_interval
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1) / lon_interval
            )
    

    Should be this:

            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) 
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1)
            )
    
    opened by rickpresley 2
  • Adding in the Tiles class.

    Adding in the Tiles class.

    New Tiles class added.

    • allows user to pass in directory of dted files, and query that list for elevations
    • functionality works regardless of filenames
    • functionality works recursively in directory
    • works even with a mix of dted 1 and dted 2 files
    • added a description of how to use in the README
    opened by westonCoder 0
  • Import Error

    Import Error

    I am getting an error when importing Tile from dted. I am using a fresh Conda environment but cannot seem to get around this error. I can see the Tile class in dted/dted/tile.py but the import does not work. Error shown below:

    ImportError: cannot import name 'Tile' from partially initialized module 'dted' (most likely due to a circular import) (/<PATH TO FILE>/dted.py)
    
    Python Version: Python 3.9.12
    

    Thanks for your help in advance!

    opened by OliverHeilmann 4
  • Query a DTED Directory instead of a specific File

    Query a DTED Directory instead of a specific File

    It'd be very useful to have a directory full of dted files, and be able to query the pool of tiles instead of checking each individual one. It could get fancy with loading tiles as needed, tracking frequency of use to free up older ones, etc.

    enhancement 
    opened by KPB3rd 2
Releases(v1.0.3)
Owner
Ben Bonenfant
Ben Bonenfant
A very lightweight monitoring system for Raspberry Pi clusters running Kubernetes.

OMNI A very lightweight monitoring system for Raspberry Pi clusters running Kubernetes. Why? When I finished my Kubernetes cluster using a few Raspber

Matias Godoy 148 Dec 29, 2022
vit for few-shot classification

Few-Shot ViT Requirements PyTorch (= 1.9) TorchVision timm (latest) einops tqdm numpy scikit-learn scipy argparse tensorboardx Pretrained Checkpoints

Martin Dong 26 Nov 30, 2022
Have you ever wondered how cool it would be to have your own A.I

Have you ever wondered how cool it would be to have your own A.I. assistant Imagine how easier it would be to send emails without typing a single word, doing Wikipedia searches without opening web br

Harsh Gupta 1 Nov 09, 2021
基于Pytorch实现优秀的自然图像分割框架!(包括FCN、U-Net和Deeplab)

语义分割学习实验-基于VOC数据集 usage: 下载VOC数据集,将JPEGImages SegmentationClass两个文件夹放入到data文件夹下。 终端切换到目标目录,运行python train.py -h查看训练 (torch) Li Xiang 28 Dec 21, 2022

Extremely simple and fast extreme multi-class and multi-label classifiers.

napkinXC napkinXC is an extremely simple and fast library for extreme multi-class and multi-label classification, that focus of implementing various m

Marek Wydmuch 43 Nov 14, 2022
Global Filter Networks for Image Classification

Global Filter Networks for Image Classification Created by Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, Jie Zhou This repository contains PyTorch

Yongming Rao 273 Dec 26, 2022
Galileo library for large scale graph training by JD

近年来,图计算在搜索、推荐和风控等场景中获得显著的效果,但也面临超大规模异构图训练,与现有的深度学习框架Tensorflow和PyTorch结合等难题。 Galileo(伽利略)是一个图深度学习框架,具备超大规模、易使用、易扩展、高性能、双后端等优点,旨在解决超大规模图算法在工业级场景的落地难题,提

JD Galileo Team 128 Nov 29, 2022
Source code of AAAI 2022 paper "Towards End-to-End Image Compression and Analysis with Transformers".

Towards End-to-End Image Compression and Analysis with Transformers Source code of our AAAI 2022 paper "Towards End-to-End Image Compression and Analy

37 Dec 21, 2022
A Lightweight Face Recognition and Facial Attribute Analysis (Age, Gender, Emotion and Race) Library for Python

deepface Deepface is a lightweight face recognition and facial attribute analysis (age, gender, emotion and race) framework for python. It is a hybrid

Sefik Ilkin Serengil 5.2k Jan 02, 2023
Pose estimation with MoveNet Lightning

Pose Estimation With MoveNet Lightning MoveNet is the TensorFlow pre-trained model that identifies 17 different key points of the human body. It is th

Yash Vora 2 Jan 04, 2022
Prometheus Exporter for data scraped from datenplattform.darmstadt.de

darmstadt-opendata-exporter Scrapes data from https://datenplattform.darmstadt.de and presents it in the Prometheus Exposition format. Pull requests w

Martin Weinelt 2 Apr 12, 2022
RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

RMNet: Equivalently Removing Residual Connection from Networks This repository is the official implementation of "RMNet: Equivalently Removing Residua

184 Jan 04, 2023
EmoTag helps you train emotion detection model for Chinese audios

emoTag emoTag helps you train emotion detection model for Chinese audios. Environment pip install -r requirement.txt Data We used Emotional Speech Dat

_zza 4 Sep 07, 2022
TensorFlow ROCm port

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

ROCm Software Platform 622 Jan 09, 2023
Code of TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation

TVT Code of TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation Datasets: Digit: MNIST, SVHN, USPS Object: Office, Office-Home, Vi

37 Dec 15, 2022
This implements one of result networks from Large-scale evolution of image classifiers

Exotic structured image classifier This implements one of result networks from Large-scale evolution of image classifiers by Esteban Real, et. al. Req

54 Nov 25, 2022
CS550 Machine Learning course project on CNN Detection.

CNN Detection (CS550 Machine Learning Project) Team Members (Tensor) : Yadava Kishore Chodipilli (11940310) Thashmitha BS (11941250) This is a work do

yaadava_kishore 2 Jan 30, 2022
(NeurIPS 2021) Realistic Evaluation of Transductive Few-Shot Learning

Realistic evaluation of transductive few-shot learning Introduction This repo contains the code for our NeurIPS 2021 submitted paper "Realistic evalua

Olivier Veilleux 14 Dec 13, 2022
RoboDesk A Multi-Task Reinforcement Learning Benchmark

RoboDesk A Multi-Task Reinforcement Learning Benchmark If you find this open source release useful, please reference in your paper: @misc{kannan2021ro

Google Research 66 Oct 07, 2022
Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach

Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach This is the implementation of traffic prediction code in DTMP based on PyTo

chenxin 1 Dec 19, 2021