Hello everyone,

I have a dataset , like this where Q0 is the feature value and TS is the time stamp , and I would like to apply shapelet transform on this csv file. and I have written code for this, but it is throwing an error saying

ValueError: could not convert string to float: '2018-03-02 00:58:19.202450' Q0 TS 0.012364804744720459, 2018-03-02 00:44:51.303082 0.012344598770141602, 2018-03-02 00:44:51.375207 0.012604951858520508, 2018-03-02 00:44:51.475198 0.012307226657867432, 2018-03-02 00:44:51.575189 0.012397348880767822, 2018-03-02 00:44:51.675180 0.013141036033630371, 2018-03-02 00:44:51.775171 0.012811839580535889, 2018-03-02 00:44:51.875162 0.012950420379638672, 2018-03-02 00:44:51.975153 0.013257980346679688, 2018-03-02 00:44:52.075144 ######################################## Code:

from sklearn.linear_model import LogisticRegression import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from pyts.datasets import load_gunpoint from pyts.transformation import ShapeletTransform from datetime import time

Toy dataset

data=pd.read_csv('dataset11.csv') pf=data.head(10)

y=data[['Q0']] X=data[['TS']]

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=10)

print(X_train)

as columns.

dataframe = pd.DataFrame( pf,columns=['TS', 'Q0'])

Changing the datatype of Date, from

Object to datetime64

#dataframe["Sample2"] = Sample2.time.strptime("%T")

Setting the Date as index

dataframe = dataframe.set_index("TS") dataframe

setting figure size to 12, 10

plt.figure(figsize=(12, 6))

Labelling the axes and setting

a title

plt.xlabel("Time") plt.ylabel("Values") plt.title("Vibration")

plotting the "A" column alone

plt.plot(dataframe["Q0"]) plt.legend(loc='best', fontsize=8) plt.show()

st = ShapeletTransform(window_sizes='auto', sort=True) X_new = st.fit_transform(X_train, y_train)

print(X_new)

Visualize the four most discriminative shapelets

plt.figure(figsize=(6, 4)) for i, index in enumerate(st.indices_[:4]): idx, start, end = index plt.plot(X_train[idx], color='C{}'.format(i), label='Sample {}'.format(idx)) plt.plot(np.arange(start, end), X_train[idx, start:end], lw=5, color='C{}'.format(i))

plt.xlabel('Time', fontsize=12) plt.title('The four most discriminative shapelets', fontsize=14) plt.legend(loc='best', fontsize=8) plt.show()

######################################

Can anyone help me with this to run this code and visualize the shapelet transform shapelet.txt

One of the features of the DTW metric is that it can compare time series of different lengths; which is done in this PR

All region methods are updated following the same logic by rescaling to the relative diagonal (n_timestamps_2 / n_timestamps_1).

Here is the updated output of the plot_dtw example with x = x[::2]

This PR adds a "precomputed" option for the dist arg when computingdtw. The user can choose whether to provide x and y or a precomputed cost matrix through a new precomputed_cost arg.

• [x] Add the new args + adapt computation of region + cost_mat.
• [x] Add tests with precomputed cost matrices (test against square for e.g)
• [x] A large portion of the code in the different dtw functions is redundant: shorten it

Hello,

because pyts comes with all the necessary requirements for time series clustering, I set up an example regarding different metrics in this context. Even though pyts focuses on classification, I think a concrete example of using BOSS in an unsupervised way is quite useful. What do you think?

The example refers to the original BOSS paper. Since a slightly different form of the data set was used there, I shifted each time series in a second attempt.

The normalization step when generating the compound images is really useful. However, if the time series data being turned into a compound image is an observation of a longer series of data with a larger range of values, the normalization removes valuable information.

I'm part of a research team at Queen's University and we are using your library to predict who will experience delirium in the ICU. While developing our model we realized the normalization step made it impossible for the model to learn. To give a practical example, one of the features we have is a patient's heart rate. One patient may have a peak heart rate of 200 whereas another one may have a peak heart rate of 140. The normalization treats both peak values as 1 because the smaller values belongs to a different patient and a different training case. To resolve this we, normalize the data across all the patients before creating the compound images without the normalization step. This way, the GASF and GADF have values within a limited range as required, but we keep information regarding what observations are greater than others to keep the observation in context.

Hi,

I am creating MTF matrices from the Lightning2 time-series dataset using the pyts module. When entering a high number for the quantile bins (n_bins) the MTF transformation does not succeed:

from scipy.io import arff
import pandas as pd
from pyts.image import GASF, GADF, MTF

lighttrain = arff.loadarff('Datasets/Lightning2/Lightning2_TRAIN.arff')
dftrain0 = pd.DataFrame(lighttrain[0])
dftrain = dftrain0.drop('target', axis=1) #taking away the class labels

matsize = 16
qsize = 40

dfmtftr = MTF(matsize,n_bins=qsize).fit_transform(dftrain)

Error in console:

File "/anaconda3/lib/python3.6/site-packages/sklearn/base.py", line 462, in fit_transform return self.fit(X, **fit_params).transform(X) File "/anaconda3/lib/python3.6/site-packages/pyts/image/image.py", line 310, in transform for i in range(n_samples)]) File "/anaconda3/lib/python3.6/site-packages/pyts/image/image.py", line 310, in for i in range(n_samples)]) ValueError: bins must be monotonically increasing or decreasing

theoretically I should be able to sort the data into as many bins as I want, shouldn't I? I cannot see a reason in the image.py source-code why the error should occur at n_bins=40 but not at n_bins=8. Are there specific boundaries for image_size and n_bins? The source time-series has a length of 637.

This PR deprecates the functions for specific versions of Dynamic Time Warping. Similarly to scipy.optimize.minimize, only the main function is in the public API, while all the functions for the specific methods are private.

Notable changes:

• All dtw_ functions are deprecated in 0.11 and will be removed in 0.12.
• The corresponding privates functions, _dtw_*, are kept and their docstrings are updated to describe only the method-specific parameters.
• Each version has its own page in the documentation, thanks to a custom Sphinx directive that is heavily inspired from the scipy-optimize custom directive.
• A new function show_options has been added to show the options for each method.

Some stuff in this PR is not related to this topic, just maintenance updates.

Hi,

I am very interested in your efficient implementation of the Singular Spectrum Analysis. Do you ever think of an advanced grouping for the components? You mentioned in the SSA-exmaple that

The first subseries consists of the trend of the original time series. The second and third subseries consist of noise.

But should the trend not rather be a zero-like signal?

And wouldn't it be nicer to show the possibility of decomposing an additive signal into three components (trend, seasonal, residual)? Packages like JULIAs-SSA explicitly return trend and seasonal. They group by a specific consideration of the eigenvalues, which in my opinion also disregard a few special cases.

My commit is only a short demo and tries to demonstrate the potential of the SSA with a grouping inspired by A METHOD OF TREND EXTRACTION USING SINGULAR SPECTRUM ANALYSIS. Also saw that the R package uses a similar approach.

Are you interested in the topic of grouping SSA-components? Then I could add tests and generalize the process.

Best regards, Lucas

The documentation commonly uses this tuple: (samples, timestamps). That doesn't make any sense in my brain as I've always thought of those being the same thing. If I'm sampling something, I'm reading a sensor value periodically. I could create a timestamp for that sample, but I also have the sensor's value at that time. My input data is (samples, sensor values). It has one row for each time I read the sensors, and a column for the value of each sensor. I think this is called the "wide" data format. Is pyts compatible with the wide data format? Or is there an easy way to transform my data into something compatible with pyts?

This PR adds a new algorithm: ShapeletTransform. This algorithm extracts the most discriminative shapelets from a dataset of time series and builds features representing the distances between these shapelets and the time series. This algorithm is in transformation module.

• pyts/transformation/shapelet_transform.py: code for the algorithm
• pyts/transformation/tests/test_shapelet_transform.py: tests
• examples/transformation/plot_shapelet_transform.py: one example illustrating the algorithm

Hi,

I was running the for Markov Transition Field example and getting the following error

X_mtf = mtf.transform(X) D:\Anaconda3\envs\tf-gpu4\lib\site-packages\pyts\image\image.py:321: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result. MTF[np.meshgrid(list_values[i], list_values[j])] = MTM[i, j] Traceback (most recent call last):

IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (4,) (5,)

GASF and GADF example worked fine. Kindly assist.

Regards, Avi

Description

Hi!

I am in doubt about the application of SymbolicAggregateApproximation() in comparison of its describition in the article "Experiencing SAX: a novel symbolic representation of time series". In the article, in section "3.2 Discretization", it is described that the data follows a Gaussian Distribution and the "breakpoints" are created to produce equal-sized areas under the curve of a Gaussian. So, I understand that the parameterer strategy='normal' uses the same strategy as the article, right? So, what a about the uniform and quantile strategies? Are they a change from the article?

Thank you for your help! Have a nice day!

Description

Hello, I am new to python. I am trying to use the SSA decomposition method for rainfall prediction with a dataset of 21500 rows and 5 columns (21500, 5). I used the source codes below. But I do not know how to fix it for my dataset. I have an error when changing the value of window size, n_sample, and n_timestamps. Anyone can help me? How can I use the main step of SSA including embedding, SVD, and reconstruction?

Steps/Code to Reproduce

<< import numpy as np import pandas as pd import matplotlib.pyplot as plt from pyts.decomposition import SingularSpectrumAnalysis

Parameters

n_samples, n_timestamps = 100, 48 df = pd.read_csv('C:/Users/PC2/Desktop/passenger.csv', index_col=0)

Toy dataset

rng = np.random.RandomState(41) X = rng.randn(n_samples, n_timestamps)

We decompose the time series into three subseries

window_size = 15 groups = [np.arange(i, i + 5) for i in range(0, 11, 5)]

Singular Spectrum Analysis

ssa = SingularSpectrumAnalysis(window_size=15, groups=groups) X_ssa = ssa.fit_transform(X)

Show the results for the first time series and its subseries

plt.figure(figsize=(16, 6))

ax1 = plt.subplot(121) ax1.plot(X[0], 'o-', label='Original') ax1.legend(loc='best', fontsize=14)

ax2 = plt.subplot(122) for i in range(len(groups)): ax2.plot(X_ssa[0, i], 'o--', label='SSA {0}'.format(i + 1)) ax2.legend(loc='best', fontsize=14)

plt.suptitle('Singular Spectrum Analysis', fontsize=20)

plt.tight_layout() plt.subplots_adjust(top=0.88) plt.show()>>

<Thank you>

The Markov Transition Matrix contained in the MTF transformer could be useful in many cases. It shouldnt be too much work to expose it (as well as the quantile boundaries) by storing it in the transformer once its fitted. This also means that the computation of the Markov Transition Matrix should be done in the fit pass instead of each time the transform is called. i.e. move: in pyts/pyts/image/mtf.py

    def transform(self, X):
...
        X = check_array(X)
        n_samples, n_timestamps = X.shape
        image_size = self._check_params(n_timestamps)

        discretizer = KBinsDiscretizer(n_bins=self.n_bins,
                                       strategy=self.strategy)
        X_binned = discretizer.fit_transform(X)

        X_mtm = _markov_transition_matrix(X_binned, n_samples,
                                          n_timestamps, self.n_bins)
        sum_mtm = X_mtm.sum(axis=2)
        np.place(sum_mtm, sum_mtm == 0, 1)
        X_mtm /= sum_mtm[:, :, None]
...

into

    def fit(self, X=None, y=None):
        """Pass.
        Parameters
        ----------
        X
            Ignored
        y
            Ignored
        Returns
        -------
        self : object
        """
        return self

Is it possible to implement this?

Description

When running SAX-VSM on my timeseries I get the following error: At least one sample is constant.

I tried filtering out all the constant time series with X = X[np.where(~(np.var(X, axis=1) == 0))[0]] to no avail

I tried fitting the model on 1 non-constant array and still got the error. I think that the issue is that this error is thrown when the SAX approximation would give the same symbol for the window, thus meaning that the window is constant. E.g. for a wordsize of 3 if the SAX transform would yield 'aaa' then this error appears. Could it be the case?

Steps/Code to Reproduce

<< 
from pyts.classification import SAXVSM

X_train = np.array([[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2]])
y_train = np.array([1]) 

clf = SAXVSM(window_size=0.5, word_size=0.5, n_bins=3, strategy='normal')
clf.fit(X_train, y_train)
>>

Versions

NumPy 1.20.1 SciPy 1.6.1 Scikit-Learn 0.24.1 Numba 0.53.1 Pyts 0.11.0

Additionally an error: 'n_bins' must be greater than or equal to 2 and lower than or equal to min(word_size, 26)

If n_bins represents the alphabet size/the paa approximation then why should it be lower than the word size? Doesn't that mean that situations like alphabet = {a,b,c,d,e} and wordsize = 3, are impossible? (which shouldn't be the case)