Summarizing an off-list conversation with Raga, @geordie666, and @stephjuneau:

To get PHOTSYS from TARGETID (thanks to @geordie666):

from desitarget.io import releasedict
from desitarget.targets import decode_targetid  
_, _, releases, _, _, _ = decode_targetid(targets["TARGETID"])  
photsys = [releasedict[release] if release >= 9000 else None for release in releases]

This should return "N"/"S" as expected, or None if the target has no Legacy Surveys provenance (by which I mean that no valid TARGETID was propagated from dr9 for the target).

But then Raga found:

Adam's code works if the release is not equal to 11 - so, I am assuming it works for 
the targets that have DR9 photometry. 

Here are two targets for testing - 
1. TARGETID: 39633489506601735, TILEID: 80696, RELEASE: 9011, PHOTSYS: 'N'
2. TARGETID: 39632956502838508, TILEID: 80895, RELEASE: 9011, PHOTSYS: 'N'

There is another thing that I have found in the fibermaps. In some cases, the flux_* values 
are 0.0 - even when they have valid DR9 phtometry. And this is not always the case for just 
secondary targets. I have seen ~10 BGS targets with FLUX_G = 0.0 - I still need to investigate 
this further.  From the above two targets, the first target have FLUX_G, R, Z = 0 from the FIBERMAP - 
but, it has valid values in DR9 photometry. 

Some Seyfert galaxies will likely have both a broad and a narrow Balmer line components. This example shows such a case with broad+narrow components on H-beta and H-alpha. However, the linewidth reported by fastspecfit is not larger for Halpha than for [OIII]5007, as I would have expected if fitting a single broad component to Halpha but a single narrow component to forbidden lines such as [OIII]5007.

     TARGETID     HPXPIXEL SURVEY FAPRGRM HALPHA_SIGMA OIII_5007_SIGMA
----------------- -------- ------ ------- ------------ ---------------
39628517884825354     9730    sv1    dark    126.67458       139.04639

#69 propagates units through all the astropy tables using the QTable object, but when writing the files in the last step, the following warning is issued. Not sure how to deal with this at the moment but tagging @weaverba137 @geordie666 @dstndstn @sbailey as there's a similar discussion (I think?) going on with the imaging / targeting files.

WARNING: The unit 'nmgy' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
WARNING: The unit '1 / nmgy2' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]

In fastphot, consider band-shifting the ugriz photometry to z=0.1 so the results can be compared with SDSS more easily. The UBVW1 bands can be band-shifted to z=0.

It will be helpful to provide a wrapper to take a data table with the continuum coefficients as inputs and instantiate the best fit model spectrum for both photometry only (FASTPHOT) and spectra only (FASTSPEC) fits.

This can help with science cases like (but not limited to):

• Synthesize photometry in any arbitrary band
• apply aperture corrections
• change distance modulus to any arbitrary cosmology

The most expensive part of the code is in the emission-line fitting which, unfortunately, relies on the astropy.fitting module. Do a profiling pass and see if there are some simple optimizations and speed-ups to be had.

https://docs.nersc.gov/development/languages/python/profiling-debugging-python/

The [OII] and [SII] doublet ratios depend on the electron density of the medium producing those lines, while the MgII doublet ratio can vary over a not-arbitrary range (for example, if one line is positive, the other line must be positive, too).

To integrate these physical priors into fastspecfit, consider using the ratio of these lines as one of the free parameters (with a sensible prior range) rather than either fixing the ratio or letting the individual line-amplitudes to be optimized independently.

Credit for this idea to Ben Weiner.

This pull request fixes issue #31 . The d4000 function was getting the incorrect keyword argument. Also added a fix to prevent unintentional modification of input array by the same function. This shows a comparison of calculations based on using the get_d4000 function directly on camera added spectra vs the output of fastspec. This shows a comparison between the fastphot catalog and the fastspec values. I think this fixes the bug.

[WIP]

Still WIP but opening a PR to track related issues.

This PR is a fairly major rewrite of the emission-line fitting engine with an eye toward porting the code to the Perlmutter/GPUs (e.g., https://jaxopt.github.io/stable/constrained.html). Specifically, all the astropy.modeling routines have been replaced with a simpler table-based linemodel of parameters (including tied parameters), resulting in about 300 fewer lines of code and notable speed-ups.

However, the speed-ups have been used to include one additional (third) round of fitting where emission lines are minimally constrained to one another, which leads to notable improvements in the line-fitting results for systems with complex line-kinematics (and to account for any cross-camera wavelength-calibration issues).

Second, I've moved away from the home-grown fnnls algorithm for continuum fitting and am now just using scipy.optimize.nnls which I also hope to be able to port to a GPU.

Backwards incompatible updates in preparation of generating fastspec and fastphot catalogs for Fuji+Guadalupe. Together with #55, #61, and #64, this PR contains many significant improvements, including:

• All non-sky spectra at z>0.001 are now fit by default, irrespective of ZWARN or COADD_FIBERSTATUS (#45).
• Balmer and Helium lines now optionally include both broad and narrow components (#47).
• The [OII] 3726,29, [SII] 6716,31, and MgII 2796,2803 doublets are now fit using a more robust model which optimizes the ratio of the doublet lines (#39).
• The velocity dispersion is now estimated for spectra with enough signal-to-noise (#67).
• More robust initialization of parameters to help ensure numerical convergence (#52).
• A simple stellar mass estimate is now computed (#70).
• Minimum photometric uncertainties included when fitting photometry (#22). In addition, W3 and W4 fluxes and ivar fluxes have been propagated and now included in the fastphot QA.
• Numerous bug fixes and many data model improvements and changes.

Still on the critical path at the moment:

• [ ] Profile the code to identify speed-ups.
• [ ] Test the feasibility of writing out the best-fitting model spectra.
• [ ] Consider gzipping the output catalogs, in order to save disk space.
• [ ] Fix the failing unit tests.
• [ ] Update the documentation and installation instructions.

I found one galaxy (low-mass AGN from Reines+2013) where the fit does not converge to a solution leading to wrong emission line fluxes. The fastspecfit-qa for this sources is as follows -

Source Information:

SDSS Information if needed:

An option to input spectra for which we just have a flux + ivar + wl_bins would be really useful! For stacking mostly( but maybe someone wants to throw some other spectra in as well).

In this case MW-foreground correction etc would already be done before entering fastspecfit and cameras coadded into a single spectrum. Once the input format is determined we can write an output option in the stacking code that can write out the stacked spectrum in that format.

I know you already pointed me to some code for this in the templates folder - I had a bit of a struggle getting it to work. 😅

@moustakas It seems that the colors based on the best-fit model are biased blue for red galaxies. This bias becomes more pronounced for low redshift galaxies.

The following plot, based on the fastphot-fuji-sv1-dark.fits catalog, demonstrates this finding. I first cut out galaxies with DES>0 to make sure DECAM filter is used for calculations. The y-axis is the fractional error of color based on FLUX_SYNTH_MODEL_band and color based on Flux_band. Triangles with the error bar show the median and error of the median. The blue band shows the histogram of the samples going into the median calculation. The top panel shows low redshift galaxies while the bottom panel shows the full samples.

Since the width of the redsequence is ~0.025, it would be great to have biases smaller than this value.

Following our meeting, it would be useful to have a notebook that could give an example of how to use the fastspecfit code to calculate k-corrections given a spectrum.

fastphot currently uses a vanilla LCDM cosmology with h=0.7 to compute luminosity distances. Consider adopting the fiducial DESI cosmology to make it easier to map between data and mocks-- https://desi.lbl.gov/trac/wiki/keyprojects#Fiducialcosmology

A handful of objects from the LOW-Z program with unexpectedly bright ABSMAG_SDSS_R

TARGETID

39633481701000832 39627862432552794 39633141308069648 39633343209279504 39633428097794736 39633507869262245 39627717636787810 39628023342826572 39628071015284948 39628159053726478 39628182306947825 39628135825672502 39628211436393100 39628199897861563 39628245884208887 39628188023785809 39628357876318744 39632941596281448 39628512897797887 39633072903160512

HEALPIX

11615 12320 15025 15279 16006 16106 19075 31655 31685 31725 31738 32298 32311 32311 32358 8521 8575 9397 9411 9868

PROGRAM | SURVEY -- | -- bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main