Fast (simple) spectral synthesis and emission-line fitting of DESI spectra.

Overview

FastSpecFit

Documentation Status

Introduction

This repository contains code and documentation to perform fast, simple spectral synthesis and emission-line fitting of DESI spectra.

Installation

To be written.

License

FastSpecFit is free software licensed under a 3-clause BSD-style license. For details see the LICENSE.rst file.

John Moustakas
Siena College
Winter 2020
Comments
  • photsys can be empty for some secondary targets

    photsys can be empty for some secondary targets

    Summarizing an off-list conversation with Raga, @geordie666, and @stephjuneau:

    To get PHOTSYS from TARGETID (thanks to @geordie666):

    from desitarget.io import releasedict
    from desitarget.targets import decode_targetid  
    _, _, releases, _, _, _ = decode_targetid(targets["TARGETID"])  
    photsys = [releasedict[release] if release >= 9000 else None for release in releases]
    

    This should return "N"/"S" as expected, or None if the target has no Legacy Surveys provenance (by which I mean that no valid TARGETID was propagated from dr9 for the target).

    But then Raga found:

    Adam's code works if the release is not equal to 11 - so, I am assuming it works for 
    the targets that have DR9 photometry. 
    
    Here are two targets for testing - 
    1. TARGETID: 39633489506601735, TILEID: 80696, RELEASE: 9011, PHOTSYS: 'N'
    2. TARGETID: 39632956502838508, TILEID: 80895, RELEASE: 9011, PHOTSYS: 'N'
    
    There is another thing that I have found in the fibermaps. In some cases, the flux_* values 
    are 0.0 - even when they have valid DR9 phtometry. And this is not always the case for just 
    secondary targets. I have seen ~10 BGS targets with FLUX_G = 0.0 - I still need to investigate 
    this further.  From the above two targets, the first target have FLUX_G, R, Z = 0 from the FIBERMAP - 
    but, it has valid values in DR9 photometry. 
    
    help wanted 
    opened by moustakas 42
  • Broad+Narrow Balmer line components

    Broad+Narrow Balmer line components

    Some Seyfert galaxies will likely have both a broad and a narrow Balmer line components. This example shows such a case with broad+narrow components on H-beta and H-alpha. However, the linewidth reported by fastspecfit is not larger for Halpha than for [OIII]5007, as I would have expected if fitting a single broad component to Halpha but a single narrow component to forbidden lines such as [OIII]5007.

         TARGETID     HPXPIXEL SURVEY FAPRGRM HALPHA_SIGMA OIII_5007_SIGMA
    ----------------- -------- ------ ------- ------------ ---------------
    39628517884825354     9730    sv1    dark    126.67458       139.04639
    
    enhancement 
    opened by stephjuneau 10
  • the unit 'nmgy' could not be saved in native FITS format

    the unit 'nmgy' could not be saved in native FITS format

    #69 propagates units through all the astropy tables using the QTable object, but when writing the files in the last step, the following warning is issued. Not sure how to deal with this at the moment but tagging @weaverba137 @geordie666 @dstndstn @sbailey as there's a similar discussion (I think?) going on with the imaging / targeting files.

    WARNING: The unit 'nmgy' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
    WARNING: The unit '1 / nmgy2' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
    
    opened by moustakas 7
  • band_shift the absolute magnitudes

    band_shift the absolute magnitudes

    In fastphot, consider band-shifting the ugriz photometry to z=0.1 so the results can be compared with SDSS more easily. The UBVW1 bands can be band-shifted to z=0.

    help wanted 
    opened by moustakas 5
  • Provide wrapper to instantiate model spectrum

    Provide wrapper to instantiate model spectrum

    It will be helpful to provide a wrapper to take a data table with the continuum coefficients as inputs and instantiate the best fit model spectrum for both photometry only (FASTPHOT) and spectra only (FASTSPEC) fits.

    This can help with science cases like (but not limited to):

    • Synthesize photometry in any arbitrary band
    • apply aperture corrections
    • change distance modulus to any arbitrary cosmology
    opened by biprateep 5
  • profile the code and see if there are some simple optimizations to be had

    profile the code and see if there are some simple optimizations to be had

    The most expensive part of the code is in the emission-line fitting which, unfortunately, relies on the astropy.fitting module. Do a profiling pass and see if there are some simple optimizations and speed-ups to be had.

    https://docs.nersc.gov/development/languages/python/profiling-debugging-python/

    opened by moustakas 5
  • fit for the doublet ratio as a free parameter (with physical priors)

    fit for the doublet ratio as a free parameter (with physical priors)

    The [OII] and [SII] doublet ratios depend on the electron density of the medium producing those lines, while the MgII doublet ratio can vary over a not-arbitrary range (for example, if one line is positive, the other line must be positive, too).

    To integrate these physical priors into fastspecfit, consider using the ratio of these lines as one of the free parameters (with a sensible prior range) rather than either fixing the ratio or letting the individual line-amplitudes to be optimized independently.

    Credit for this idea to Ben Weiner.

    enhancement 
    opened by moustakas 4
  • fixed spectroscopic DN4000 calculation bug

    fixed spectroscopic DN4000 calculation bug

    This pull request fixes issue #31 . The d4000 function was getting the incorrect keyword argument. Also added a fix to prevent unintentional modification of input array by the same function. This shows a comparison of calculations based on using the get_d4000 function directly on camera added spectra vs the output of fastspec. zoomed This shows a comparison between the fastphot catalog and the fastspec values. I think this fixes the bug. download (8)

    opened by biprateep 4
  • refactor fitting engine to not use fnnls or astropy.modeling

    refactor fitting engine to not use fnnls or astropy.modeling

    [WIP]

    Still WIP but opening a PR to track related issues.

    This PR is a fairly major rewrite of the emission-line fitting engine with an eye toward porting the code to the Perlmutter/GPUs (e.g., https://jaxopt.github.io/stable/constrained.html). Specifically, all the astropy.modeling routines have been replaced with a simpler table-based linemodel of parameters (including tied parameters), resulting in about 300 fewer lines of code and notable speed-ups.

    However, the speed-ups have been used to include one additional (third) round of fitting where emission lines are minimally constrained to one another, which leads to notable improvements in the line-fitting results for systems with complex line-kinematics (and to account for any cross-camera wavelength-calibration issues).

    Second, I've moved away from the home-grown fnnls algorithm for continuum fitting and am now just using scipy.optimize.nnls which I also hope to be able to port to a GPU.

    opened by moustakas 3
  • Fujilupe release candidate

    Fujilupe release candidate

    Backwards incompatible updates in preparation of generating fastspec and fastphot catalogs for Fuji+Guadalupe. Together with #55, #61, and #64, this PR contains many significant improvements, including:

    • All non-sky spectra at z>0.001 are now fit by default, irrespective of ZWARN or COADD_FIBERSTATUS (#45).
    • Balmer and Helium lines now optionally include both broad and narrow components (#47).
    • The [OII] 3726,29, [SII] 6716,31, and MgII 2796,2803 doublets are now fit using a more robust model which optimizes the ratio of the doublet lines (#39).
    • The velocity dispersion is now estimated for spectra with enough signal-to-noise (#67).
    • More robust initialization of parameters to help ensure numerical convergence (#52).
    • A simple stellar mass estimate is now computed (#70).
    • Minimum photometric uncertainties included when fitting photometry (#22). In addition, W3 and W4 fluxes and ivar fluxes have been propagated and now included in the fastphot QA.
    • Numerous bug fixes and many data model improvements and changes.

    Still on the critical path at the moment:

    • [ ] Profile the code to identify speed-ups.
    • [ ] Test the feasibility of writing out the best-fitting model spectra.
    • [ ] Consider gzipping the output catalogs, in order to save disk space.
    • [ ] Fix the failing unit tests.
    • [ ] Update the documentation and installation instructions.
    opened by moustakas 3
  • Unreal fits because the fit does not converge to a final solution

    Unreal fits because the fit does not converge to a final solution

    I found one galaxy (low-mass AGN from Reines+2013) where the fit does not converge to a solution leading to wrong emission line fluxes. The fastspecfit-qa for this sources is as follows -

    image

    Source Information: image

    SDSS Information if needed: image

    opened by Ragadeepika-Pucha 3
  • Stacked spectra input option

    Stacked spectra input option

    An option to input spectra for which we just have a flux + ivar + wl_bins would be really useful! For stacking mostly( but maybe someone wants to throw some other spectra in as well).

    In this case MW-foreground correction etc would already be done before entering fastspecfit and cameras coadded into a single spectrum. Once the input format is determined we can write an output option in the stacking code that can write out the stacked spectrum in that format.

    I know you already pointed me to some code for this in the templates folder - I had a bit of a struggle getting it to work. 😅

    opened by dirkscholte 0
  • Biases of model magnitude for red galaxies

    Biases of model magnitude for red galaxies

    @moustakas It seems that the colors based on the best-fit model are biased blue for red galaxies. This bias becomes more pronounced for low redshift galaxies.

    The following plot, based on the fastphot-fuji-sv1-dark.fits catalog, demonstrates this finding. I first cut out galaxies with DES>0 to make sure DECAM filter is used for calculations. The y-axis is the fractional error of color based on FLUX_SYNTH_MODEL_band and color based on Flux_band. Triangles with the error bar show the median and error of the median. The blue band shows the histogram of the samples going into the median calculation. The top panel shows low redshift galaxies while the bottom panel shows the full samples.

    Since the width of the redsequence is ~0.025, it would be great to have biases smaller than this value.

    download - 2022-11-22T153850 701

    templates 
    opened by chto 1
  • nb for k-correction calculation

    nb for k-correction calculation

    Following our meeting, it would be useful to have a notebook that could give an example of how to use the fastspecfit code to calculate k-corrections given a spectrum.

    documentation 
    opened by SgmAstro 0
  • consider adopting the fiducial DESI cosmology

    consider adopting the fiducial DESI cosmology

    fastphot currently uses a vanilla LCDM cosmology with h=0.7 to compute luminosity distances. Consider adopting the fiducial DESI cosmology to make it easier to map between data and mocks-- https://desi.lbl.gov/trac/wiki/keyprojects#Fiducialcosmology

    opened by moustakas 1
  • Objects with odd ABSMAG_SDSS_R

    Objects with odd ABSMAG_SDSS_R

    A handful of objects from the LOW-Z program with unexpectedly bright ABSMAG_SDSS_R

    TARGETID

    39633481701000832 39627862432552794 39633141308069648 39633343209279504 39633428097794736 39633507869262245 39627717636787810 39628023342826572 39628071015284948 39628159053726478 39628182306947825 39628135825672502 39628211436393100 39628199897861563 39628245884208887 39628188023785809 39628357876318744 39632941596281448 39628512897797887 39633072903160512

    HEALPIX

    11615 12320 15025 15279 16006 16106 19075 31655 31685 31725 31738 32298 32311 32311 32358 8521 8575 9397 9411 9868

    PROGRAM | SURVEY -- | -- bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main

    opened by edarragh 9
Releases(v1.0.1)
Owner
Public code associated with the Dark Energy Spectroscopic Instrument (DESI).
Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021)

Transferable Semantic Augmentation for Domain Adaptation Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021) Paper

66 Dec 16, 2022
Official code repository for the EMNLP 2021 paper

Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization PyTorch code for the EMNLP 2021 paper "Integrating Visuospatia

Adyasha Maharana 23 Dec 19, 2022
A Kernel fuzzer focusing on race bugs

Razzer: Finding kernel race bugs through fuzzing Environment setup $ source scripts/envsetup.sh scripts/envsetup.sh sets up necessary environment var

Systems and Software Security Lab at Seoul National University (SNU) 328 Dec 26, 2022
Selfplay In MultiPlayer Environments

This project allows you to train AI agents on custom-built multiplayer environments, through self-play reinforcement learning.

200 Jan 08, 2023
Buffon’s needle: one of the oldest problems in geometric probability

Buffon-s-Needle Buffon’s needle is one of the oldest problems in geometric proba

3 Feb 18, 2022
The ICS Chat System project for NYU Shanghai Fall 2021

ICS_Chat_System [Catenger] This is the ICS Chat System project for NYU Shanghai Fall 2021 Creators: Shavarsh Melikyan, Skyler Chen and Arghya Sarkar,

1 Dec 20, 2021
I tried to apply the CAM algorithm to YOLOv4 and it worked.

YOLOV4:You Only Look Once目标检测模型在pytorch当中的实现 2021年2月7日更新: 加入letterbox_image的选项,关闭letterbox_image后网络的map得到大幅度提升。 目录 性能情况 Performance 实现的内容 Achievement

55 Dec 05, 2022
Regulatory Instruments for Fair Personalized Pricing.

Fair pricing Source code for WWW 2022 paper Regulatory Instruments for Fair Personalized Pricing. Installation Requirements Linux with Python = 3.6 p

Renzhe Xu 6 Oct 26, 2022
Learning High-Speed Flight in the Wild

Learning High-Speed Flight in the Wild This repo contains the code associated to the paper Learning Agile Flight in the Wild. For more information, pl

Robotics and Perception Group 391 Dec 29, 2022
Most popular metrics used to evaluate object detection algorithms.

Most popular metrics used to evaluate object detection algorithms.

Rafael Padilla 4.4k Dec 25, 2022
免费获取http代理并生成proxifier配置文件

freeproxy 免费获取http代理并生成proxifier配置文件 公众号:台下言书 工具说明:https://mp.weixin.qq.com/s?__biz=MzIyNDkwNjQ5Ng==&mid=2247484425&idx=1&sn=56ccbe130822aa35038095317

说书人 32 Mar 25, 2022
Tello Drone Trajectory Tracking

With this library you can track the trajectory of your tello drone or swarm of drones in real time.

Kamran Asgarov 2 Oct 12, 2022
Large Scale Multi-Illuminant (LSMI) Dataset for Developing White Balance Algorithm under Mixed Illumination

Large Scale Multi-Illuminant (LSMI) Dataset for Developing White Balance Algorithm under Mixed Illumination (ICCV 2021) Dataset License This work is l

DongYoung Kim 33 Jan 04, 2023
Main Results on ImageNet with Pretrained Models

This repository contains Pytorch evaluation code, training code and pretrained models for the following projects: SPACH (A Battle of Network Structure

Microsoft 151 Dec 14, 2022
The Fundamental Clustering Problems Suite (FCPS) summaries 54 state-of-the-art clustering algorithms, common cluster challenges and estimations of the number of clusters as well as the testing for cluster tendency.

FCPS Fundamental Clustering Problems Suite The package provides over sixty state-of-the-art clustering algorithms for unsupervised machine learning pu

9 Nov 27, 2022
Official code for 'Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentationon Complex Urban Driving Scenes'

PEBAL This repo contains the Pytorch implementation of our paper: Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urb

Yu Tian 117 Jan 03, 2023
A minimalist tool to display a network graph.

A tool to get a minimalist view of any architecture This tool has only be tested with the models included in this repo. Therefore, I can't guarantee t

Thibault Castells 1 Feb 11, 2022
Training data extraction on GPT-2

Training data extraction from GPT-2 This repository contains code for extracting training data from GPT-2, following the approach outlined in the foll

Florian Tramer 62 Dec 07, 2022
nanodet_plus,yolov5_v6.0

OAK_Detection OAK设备上适配nanodet_plus,yolov5_v6.0 Environment pytorch = 1.7.0

炼丹去了 1 Feb 18, 2022
Multi-task yolov5 with detection and segmentation based on yolov5

YOLOv5DS Multi-task yolov5 with detection and segmentation based on yolov5(branch v6.0) decoupled head anchor free segmentation head README中文 Ablation

150 Dec 30, 2022