Hi everyone,

following #394, this is a first start on integrating the multirate filterbank which is used in the chroma toolbox.

I have another notebook where I show the whole processing chain: https://github.com/stefan-balke/mpa-exc/blob/master/02_fourier_transform/pitch_filterbank.ipynb

As for implementation, it could live as another "spectral representation" in spectrum.py, although its more a CQT–but I guess we leave cqt.py reserved for the actual transform. Parameter-wise, this one is pretty much fixed, although one could add a parameter for "detuning", meaning another reference frequency than 440 Hz @ A4.

But that's up for discussion.

As next steps, I would add some unit tests comparing the filter coefficients to the Chroma Toolbox and a function which actually calls this filterbank...should be straight-forward!

This change is 

Hey all,

I mainly took @craffel dijtw source code and merged it with mine. As dicussed in #298, we want the following features:

• [x] Arbitrary step sizes
• [x] Additive or multiplicative local weights for the steps
• [x] Subsequence (so it can be used for matching)
• [x] Global path constraints (e.g. Sakoe-Chiba band etc.)
• [x] make numba optional (cf. Brian's comment)
• [x] test backtracking explicitly
• [x] plot D + wp (in the examples)
• ~~Gullying~~

After finishing the features (which are in dijtw), we need more tests and some final notebook with benchmarking the implementation against "vanilla-vanilla" dtw.

This change is

Reference Issue

This PR begins the work of #1130, extending librosa to support multi-channel wherever possible.

What does this implement/fix? Explain your changes.

This branch will serve as the target for smaller pull requests implementing multi-channel support throughout the package.

As a first step, I've implemented multi-channel stft. It's numerically equivalent to our previous implementation, and comparably efficient for single-channel inputs. For multi-channel inputs, it's not as efficient as I think is possible, but optimization will take a bit of sleuthing.

Any other comments?

This PR will not be merged until #1130 is complete.

Progress tracker

beat

• [x] tempo : yes, one tempo per channel?
• [x] plp : yes

decompose

• [x] decompose : maybe, could require some clever reshaping. not obvious how to do it.
• [x] hpss : yes
• [x] nn_filter : features: probably, self-similarity: ~probably not~ no

effects

• [x] hpss : yes
• [x] harmonic : yes
• [x] percussive : yes
• [x] time_stretch : yes
• [x] pitch_shift : yes (issue #1085)
• [x] preemphasis: yes
• [x] deemphasis: yes
• [x] trim
• [x] split
• [x] remix : yes update: zc alignment won't be feasible here, but we can only support zc alignment for mono signals.

filters

~- [ ] diagonal_filter: could require some thought~ - not necessary; this is mainly used for building smoothing filters for self-similarity matrices, which in the multi-channel case, would still operate independently across channels. Keeping diagonal filters 2d seems appropriate.

onset

• [x] onset_strength: yes
• [x] onset_strength_multi: yes ~- [ ] onset_backtrack: probably not~ no

segment

• [x] cross_similarity : and
• [x] recurrence_matrix : yes-ish: channels dimensions are flattened so output integrates over channels ~- [ ] recurrence_to_lag~ : no ~- [ ] lag_to_recurrence~ : no ~- [ ] timelag_filter : maybe~ nothing to do here, as it depends entirely on lag<->recurrence conversion
• [x] path_enhance: yes

sequence

• [x] dtw : probably best to treat similarly to segment modules: flatten channel dimensions ~- [ ] rqa : not really; ragged output. otherwise, treat like dtw~ rqa takes a square matrix as input; nothing to do here
• [x] viterbi : yes
• [x] viterbi_discriminative : yes
• [x] viterbi_binary : yes

core

• [x] stream: n/a, I think?
• [x] to_mono: yes
• [x] resample: yes, n/a
• [x] get_duration: yes
• [x] stft: yes
• [x] istft: yes
• [x] magphase : yes
• [x] griffinlim: yes
• [x] zero_crossings: yes
• [x] autocorrelate: yes
• [x] fmt: yes
• [x] pcen: yes
• [x] cqt: yes
• [x] vqt: yes
• [x] pseudo_cqt: yes
• [x] hybrid_cqt: yes
• [x] icqt: yes
• [x] griffinlim_cqt: yes
• [x] reassigned_spectrogram: yes
• [x] phase_vocoder: yes
• [x] interp_harmonics: yes ~- [ ] harmonics_1d: yes~ -> remove, fold functionality into interp_harmonics ~- [ ] harmonics_2d: yes~ -> remove
• [x] salience: yes
• [x] iirt: yes
• [x] lpc: yes
• [x] yin: yes
• [x] piptrack: yes
• [x] estimate_tuning: yes ~should produce one tuning estimate per channel~ one tuning estimate across all channels ~- [ ] pitch_tuning: yes~ no, this works on a set of frequencies, not the signal directly. Nothing to do here.
• [x] pyin: yes
• [x] clicks: yes, but only if a multichannel click sample is provided ~- [ ] tone~: no

feature

• [x] chroma_stft: yes
• [x] chroma_cqt: yes
• [x] chroma_cens: yes
• [x] melspectrogram: yes
• [x] mfcc:yes
• [x] rms: yes
• [x] spectral_centroid: yes
• [x] spectral_bandwidth: yes
• [x] spectral_flatness: yes
• [x] spectral_rolloff: yes
• [x] poly_features : yes
• [x] tonnetz : yes
• [x] zero_crossing_rate : yes
• [x] delta : yes
• [x] stack_memory : yes
• [x] tempogram : yes
• [x] fourier_tempogram : yes
• [x] mel_to_stft: yes
• [x] mel_to_audio : yes
• [x] mfcc_to_mel :yes
• [x] mfcc_to_audio: yes

util

• [x] frame: yes
• [x] pad_center: yes
• [x] fix_length: yes
• [x] softmask: yes
• [x] normalize: yes
• [x] localmax: yes
• [x] localmin: yes
• [x] valid_audio: yes
• [x] stack: yes ~- [ ] fix_frames: probably not / nothing to do?~ ~- [ ] index_to_slice~: no - this is an inherently one-dimensional operation
• [x] sync: yes, maybe already works ~- [ ] axis_sort: ~maybe, not clear what it would mean~ probably not; possible rename to axis_sort2d~ no ~- [ ] shear: maybe; dense only? but probably not~ no
• [x] nnls: yes

Maintenance tasks

• [x] audit docstrings of all functions extended for multichannel
• [x] expand_to helper function
• [x] relax valid_audio shape checks
• [x] audit for use of valid_audio

Reference Issue

Fixes #527

What does this implement/fix? Explain your changes.

This pull request implements the YIN and pYIN algorithms for pitch tracking. The YIN function is based on @lostanlen's PR #974 (with a few modifications) while the pYIN function is based on this paper.

Any other comments?

Both functions work well but some refactoring is definately needed. I compared the outputs of pYIN to the official vamp plugin and the results are comparable. I haven't added any special treatment of low amplitude frames yet so silent frames with periodic noice occasionally give wrong results.

Also, note that I haven't used the librosa.core.autocorrelate function. The librosa.core.autocorrelate computes equation (2) in the YIN paper which seems to perform worse than the one used here (which computes equation (1)). I tried scaling by the window size (as suggested in the paper) but it didn't improve things much.

I haven't implemented any tests yet so it would be good to get some guidance there.

Hey there,

I recently did a DTW implementation based on Meinard's book with some Cython speed up for the dynamic programming. As this is not yet reflected in librosa, I wondered if we could do a module for music syncronization.

Best Stefan

This PR implements #412, and a couple of bug-fixes. Summary of contents:

• Added a scale boolean option to all CQT methods. If enabled, CQT bands are normalized by sqrt(n[i]) where n[i] is the length of the ith filter. This is analogous to norm='ortho' mode in np.fft.fft.
• Early downsampling is less aggressive by one octave. Previously, it was too aggressive, and the top-end of each octave was close to nyquist, which resulted in attenuation.
• Magnitude is now continuous and approximately equal for full, hybrid, and pseudo CQT. This fixes some of the unresolved continuity errors noted in #347.
• Expanded the CQT unit tests to include a white noise continuity test.

Using scale=True means that white noise input will look like white noise output in CQT space (flat spectrum). With scale=False (current default), white noise in looks like 1/sqrt(f).

scale=True makes an impulse look like sqrt(f) as opposed to constant for scale=False.

This change is

I wanted to make a separate issue to consolidate the discussion around a number of related points that have popped up in the implementation of VQT #1018, inverse CQT #165, and so on. @lostanlen and I have discussed these things in various places (often offline), but it would be helpful to have a more permanent record.

Frequency band definitions

In the CQT implementation, we follow the SK2010 definition of the frequency bands, which are essentially left-aligned: [f, a * f] (for some constant a). This works well enough, but it makes some calculations awkward, eg in the VQT. Later on in the CQT toolbox, the definition shifted to a centered representation, eg [f / sqrt(a), f * sqrt(a)] (again for some constant a), so that the frequency of each filter is centered within the band.

Some questions:

• Should we add support for centered frequency bands? I don't think it would change much, except possibly some boundary effects at the top end of octaves.
• Should we make centering the default? Should we even bother with left-aligned bands anymore?

Energy preservation

Our CQT implementation has historically had a bunch of headaches around normalization and energy preservation. Of course we can't exactly preserve energy with a lossy frequency representation, but our current cqt/icqt round trip gets pretty close. However, other feature representations (notably Mel) are not so forgiving, and it would be nice if we could strive for consistency here.

Currently, the default sample rate is 22050 Hz when loading an audio file. In order to use the native sample rate, sr must be set to None. I often forget to set this to None when I want the native sample rate, and I have noticed many students do the same.

Have others experienced confusion over this? Should this change in a later release so that the native sample rate is the default?

I would like to have code completions and type checking for code that uses librosa. Adding type annotations will significantly improve the coding experience for users who use type checkers / typing-based completion engines.

Currently some type checkers like Pylance are able to infer some types, however this is often wrong and leads to false positives (errors on working code) or false negatives (missing errors on invalid code). Other type checkers like Mypy cannot do anything with code that uses librosa, and must treat it as 'Any'.

It's been mentioned on numerous occasions that our current example track, catchy as it may be, is not great for demonstrating many of the functions of librosa. In offline discussions, @lostanlen and I have talked about extending the included audio content to have several tracks, which could be used to demonstrate different functionality. So I want to kick this out to the community: what do you want to see in our examples? By this I mean: please recommend specific recordings.

To prevent this discussion from becoming an infinite bike shed, I'm going to lay down some ground rules for inclusion:

1. Content must be CC-licensed or public domain.
2. Total content should not bloat the package too much; I think 10MB is a reasonable upper bound, and this functionally limits us to between 5 and 10 total recordings.
3. No single track should be too long. It's okay to have some very short examples.
4. The total collection should be diverse in terms of style, instrumentation, polyphony, etc. If possible, I'd like to include some non-western recordings as well.
5. Any lyrical content should not be offensive, for some reasonable definition of offensive which is compatible with our CoC. I don't expect any problems here, but I'll reserve executive privilege here to veto anything that could be problematic.
6. Familiarity would be a bonus, for making the examples and documentation more immediately accessible.

With all of that out of the way, let's talk about things not currently demonstrated by our current example. We don't have to hit all of these, and I'm sure I'm missing some, but we should aim to hit most of them.

• Monophonic audio: we should have at least one solo instrument recording that can be used to demonstrate things like pitch tracking. Maybe a raga or makam could be good here? This could also be good for demonstrating onsets.
• Interesting harmony: the current example is pretty boring, harmonically speaking. Maybe a jazz recording would be appropriate here? Maybe something with some key changes as well.
• Non-percussive rhythmic elements: something classical (strings) would be nice to have for demonstrating onset and beat tracking with soft attacks.
• Different time signatures: examples with 4/4, 3/4, and maybe a 5/4 or 7/8 would be nice for demonstrating some of the rhythmic features (tempogram, fmt)
• Vocals and instrumentals: we should have at least one track with vocals.
• Non-musical audio: do we need/want this? Speech? Environmental sound? Librosa gets used for these things, so it might be worth considering their inclusion.

Some other discussions that we can have around this:

• Should we have a multi-track / stem set as one of the examples?
• What kind of genre coverage should we strive for?
• How does this issue interact with #641 (non-western systems in display)? Is #641 a pre-requisite for including non-western examples? (I'd argue that it should be, and that this would be a good motivating factor for finally doing it.)

Finally, the examples gallery already includes a few candidate options here. We can take some, all, or none of these, but whatever we decide on including should serve as plausible replacements for them and the example notebooks should be revised afterward.

As proposed in https://github.com/librosa/librosa/issues/353 scipy.fftpack is a terrible bottleneck and supporting FFTW would be immensely useful. This pull request adds an optional wrapper for FFTW. If pyFFTW is installed librosa will prefer that over scipy.fftpack. Everything will work as normal if the user is missing pyFFTW.

@bmcfee, thoughts? The speedup is substantial, and as librosa depends on STFT all over the place (even for rmse) I feel this is a necessary addition, and a lot nicer than having users monkey patch outside of librosa.

This change is

Is your feature request related to a problem? Please describe. This came up in implementing #1632 - the threshold parameter in zero_crossings is currently optional, but it doesn't need to be. If threshold is None, it is converted to 0 automatically.

Describe the solution you'd like Threshold=0 is equivalent behavior, and dropping the None support would simplify some type annotations and a bit of the implementation.

BEFORE POSTING A BUG REPORT Please look through existing issues (both open and closed) to see if it's already been reported or fixed!

Describe the bug A clear and concise description of what the bug is. Recently I am working with some audio data . But the issue I am facing that while I am trying to load the audio file by librosa.load().

To Reproduce

Expected behavior I am guessing that this error is may be happening as the x which should be 1-D numpy array , it is empty in this case. but why?? When I am playing the sound by Ipython ,It is working fine .

Screenshots If applicable, add screenshots to help explain your problem.

Software versions*

Windows-10-10.0.22621-SP0 Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] NumPy 1.22.3 SciPy 1.8.0 librosa 0.9.2 INSTALLED VERSIONS

python: 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)]

librosa: 0.9.2

audioread: 3.0.0 numpy: 1.22.3 scipy: 1.8.0 sklearn: 1.0.2 joblib: 1.1.0 decorator: 5.1.1 soundfile: 0.11.0 resampy: 0.4.2 numba: 0.56.4 pooch: v1.6.0 packaging: 21.3

numpydoc: None sphinx: None sphinx_rtd_theme: None sphinx_multiversion: None sphinx_gallery: None mir_eval: None ipython: None sphinxcontrib-svg2pdfconverter: None pytest: None pytest-mpl: None pytest-cov: None matplotlib: 3.5.1 samplerate: None soxr: None contextlib2: None presets: None Additional context Add any other context about the problem here. Please help me to fix this issue . As the x is getting store an empty numpy 1-D array I cann't able to proceed further .I am not getting it's solution no where .

This is just a general place-holder note to look through the scipy 1.10 release notes for functionality that we should make use of (if not depend on).

There appear to be relevant additions to the interpolate and signal modules, but we should investigate closely before the 0.10 release over here.

I import librosa in my program using the following line import librosa

and when I run my program it just stops there and does nothing, there is no error message so I can't give you any

here you see a screen shot of the beginning of my program and the output window and as you can see the import librosa stop everything

if you need more info just ask them

thank you

In the documentation of librosa.feature.chroma_cqt the tuning parameter is described as follows:

tuning : float
    Deviation (in fractions of a CQT bin) from A440 tuning

This left me somewhat wondering about the automatic tuning estimation offered in librosa.cqt. After inspecting the code it seems that this parameter is only used for passing it to the librosa.cqt or librosa.hybrid_cqt functions. There this parameter is described as follows:

tuning : None or float
        Tuning offset in fractions of a bin.
        If ``None``, tuning will be automatically estimated from the signal.
        The minimum frequency of the resulting CQT will be modified to
        ``fmin * 2**(tuning / bins_per_octave)``.

In addition, the default parameter for tuning in chroma_cqt is None while the default parameter for it in cqt and hybrid_cqt is 0.0.

From my point of view, the description of the tuning parameter in the chroma_cqt function should be the same and therefore be extend with the information about what None actually means. I'm however not sure about the default values, since it would make sense to me to make those equal, but this may introduce some problems with old code.