• nannyml version: 0.4.0
• Python version: 3.8.0
• Operating System: 5.10.102.1-microsoft-standard-WSL2 ; Ubuntu 18.04.6 LTS

Description

I'm trying to walk through the Quickstart guide and getting the following error: module 'nannyml' has no attribute 'load_synthetic_binary_classification_dataset'

What I Did

import pandas as pd
import nannyml as nml
from IPython.display import display
reference, analysis, analysis_target =nml.load_synthetic_binary_classification_dataset()
display(analysis.head())
display(reference.head())

Some minor corrections / suggestions for the documentation:

(0) A general comment: as far as I can tell, nowhere in the documentation is it explicitly stated that the model not required to do the performance prediction (although it is implicit). Also, it's always assumed that the model is a machine learning one, but in theory the software is model-agnostic, as long as the model outputs conform to the expected formats, right?

(1) https://docs.nannyml.com/latest/quick.html

• replace it's with its (2 occurrences)
• "This is why on the synthetic dataset it is provided in a separate object." --> "This is why in the synthetic dataset it is provided in a separate object."
• Some words / phrases are capitalized in the middle of a sentence, seemingly at random. i,e, "Model Monitoring" or "Machine Learning".
• In the first plot, I feel it would be important to also show the actual model performance (ROC AUC in this case). This is probably THE most crucial thing a potential user wants to see here: are the predictions correct?
• Final sentence: "This drift is responsible for the potential negative impact in performance that we observed." The actual ROC AUC is never actually shown, so the reader has no idea what the change in performance really is (just has to trust it is true).

(2) Glossary (https://docs.nannyml.com/latest/glossary.html#glossary)

• In the entry for concept drift, it should probably be stated that the term is sometimes used (by others) with a broader definition that also includes things like label shift (?).
• In the predicted scores entry, "calues" should be "values".
• Univariate Drift Detection and Multivariate Drift Detection entries: "of our model" is superfluous and probably misleading in this context.
• The entry for Model "Definition of a model." is a circular definition.
• CBPE (Confidence-Based Perofmance Estimation): change "Perofmance" to "Performance"

(3) https://docs.nannyml.com/latest/guides/data_drift.html

• "instannce" --> "instance"
• "consice" --> "concise"
• The plot "Distribution over time for y_pred_proba" has the x and y axis labels swapped.
• "occurance" --> "occurrence"
• The section "Drift detection for model targets" seems to just end without much of a conclusion.

(4) https://docs.nannyml.com/latest/guides/performance_estimation.html

• The y-axis title on last plot is partially cropped; it would also be useful to change the x-axis to time, and add the analysis period line to make this plot easier to compare to the previous one that has the performance prediction.

It would be nice if you'd allow bootstrapping (resampling with replacement) instead of non-overlapping chunks for the CBPE estimate. Ideally, something like this: https://stats.stackexchange.com/questions/96739/what-is-the-632-rule-in-bootstrapping

• nannyml version: 0.2.0

Description The confidence bounds of the CBPE plot go above 1.0 when ROC-AUC is one. They should be cut off at 1.0 as a ROC-AUC of over 1.0 is impossible.

Describe the bug Running the Quickstart results in an error

To Reproduce Steps to reproduce the behavior: Runing:

import pandas as pd
import nannyml as nml
from IPython.display import display

# Load synthetic data
reference, analysis, analysis_target = nml.load_synthetic_binary_classification_dataset()
display(reference.head())
display(analysis.head())

# Choose a chunker or set a chunk size
chunk_size = 5000

# initialize, specify required data columns, fit estimator and estimate
estimator = nml.CBPE(
   y_pred_proba='y_pred_proba',
   y_pred='y_pred',
   y_true='work_home_actual',
   timestamp_column_name='timestamp',
   metrics=['roc_auc'],
   chunk_size=chunk_size,
)
estimator = estimator.fit(reference)
estimated_performance = estimator.estimate(analysis)

# Show results
figure = estimated_performance.plot(kind='performance', metric='roc_auc', plot_reference=True)
figure.show()

# Define feature columns
feature_column_names = [
    col for col in reference.columns if col not in [
        'timestamp', 'y_pred_proba', 'period', 'y_pred', 'work_home_actual', 'identifier'
    ]]

# Let's initialize the object that will perform the Univariate Drift calculations
univariate_calculator = nml.UnivariateStatisticalDriftCalculator(
    feature_column_names=feature_column_names,
    timestamp_column_name='timestamp',
    chunk_size=chunk_size
)
univariate_calculator = univariate_calculator.fit(reference)
univariate_results = univariate_calculator.calculate(analysis)
# Plot drift results for all model inputs
for feature in univariate_calculator.feature_column_names:
    figure = univariate_results.plot(
        kind='feature_drift',
        metric='statistic',
        feature_column_name=feature,
        plot_reference=True
    )
    figure.show()

# Rank features based on number of alerts
ranker = nml.Ranker.by('alert_count')
ranked_features = ranker.rank(univariate_results, only_drifting = False)
display(ranked_features)

calc = nml.StatisticalOutputDriftCalculator(
    y_pred='y_pred',
    y_pred_proba='y_pred_proba',
    timestamp_column_name='timestamp'
)
calc.fit(reference)
results = calc.calculate(analysis)

figure = results.plot(kind='prediction_drift', plot_reference=True)
figure.show()

# Let's initialize the object that will perform Data Reconstruction with PCA
rcerror_calculator = nml.DataReconstructionDriftCalculator(feature_column_names=feature_column_names, timestamp_column_name='timestamp', chunk_size=chunk_size).fit(reference_data=reference)
# let's see Reconstruction error statistics for all available data
rcerror_results = rcerror_calculator.calculate(analysis)
figure = rcerror_results.plot(kind='drift', plot_reference=True)
figure.show()

Gives the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~\anaconda3\lib\site-packages\nannyml\base.py in fit(self, reference_data, *args, **kwargs)
     94             self._logger.debug(f"fitting {str(self)}")
---> 95             return self._fit(reference_data, *args, **kwargs)
     96         except InvalidArgumentsException:

~\anaconda3\lib\site-packages\nannyml\drift\model_inputs\univariate\statistical\calculator.py in _fit(self, reference_data, *args, **kwargs)
    105         self.previous_reference_data = reference_data.copy()
--> 106         self.previous_reference_results = self._calculate(self.previous_reference_data).data
    107 

~\anaconda3\lib\site-packages\nannyml\drift\model_inputs\univariate\statistical\calculator.py in _calculate(self, data, *args, **kwargs)
    116 
--> 117         self.continuous_column_names, self.categorical_column_names = _split_features_by_type(
    118             data, self.feature_column_names

~\anaconda3\lib\site-packages\nannyml\base.py in _split_features_by_type(data, feature_column_names)
    229 
--> 230     categorical_column_names = [col for col in feature_column_names if _column_is_categorical(data[col])]
    231 

~\anaconda3\lib\site-packages\nannyml\base.py in <listcomp>(.0)
    229 
--> 230     categorical_column_names = [col for col in feature_column_names if _column_is_categorical(data[col])]
    231 

~\anaconda3\lib\site-packages\nannyml\base.py in _column_is_categorical(column)
    235 def _column_is_categorical(column: pd.Series) -> bool:
--> 236     return column.dtype in ['object', 'string', 'category', 'bool']
    237 

TypeError: data type 'string' not understood

During handling of the above exception, another exception occurred:

CalculatorException                       Traceback (most recent call last)
<ipython-input-1-9ae82d7fa4d4> in <module>
     39     chunk_size=chunk_size
     40 )
---> 41 univariate_calculator = univariate_calculator.fit(reference)
     42 univariate_results = univariate_calculator.calculate(analysis)
     43 # Plot drift results for all model inputs

~\anaconda3\lib\site-packages\nannyml\base.py in fit(self, reference_data, *args, **kwargs)
     99             raise
    100         except Exception as exc:
--> 101             raise CalculatorException(f"failed while fitting {str(self)}.\n{exc}")
    102 
    103     def calculate(self, data: pd.DataFrame, *args, **kwargs) -> AbstractCalculatorResult:

CalculatorException: failed while fitting <nannyml.drift.model_inputs.univariate.statistical.calculator.UnivariateStatisticalDriftCalculator object at 0x0000022BBF196A30>.
data type 'string' not understood

Expected behavior The quickstart code runs without a problem.

Additional context

The user who had that issue was running python 3.8 on windows through a pycharm environment.

I couldn't reproduce the error when I tried on my machine. Moreover when I guided the user to set up a new conda environment the error went away.

However maybe the way string type is defined here could be changed similar to suggestions such as these to cover more cases? I 'd hold of on that until we see more users having the issue, since in this case a misconfigured environment is more likely the problem than a library compatibility issue.

• nannyml version: 0.4.1/ 0.4.0
• Python version: python:3.8
• Operating System: python:3.8-slim-bullseye Docker image (Debian based)

Description

I was trying to reproduce the example on the main readme. Plots are not showing. See below:

I tried this with both versions 0.4.0/0.4.1

If it helps, I'm using:

jupyterlab==3.4.2
pandas==1.4.1

What I Did

Example on readme, as is:

import pandas as pd
import nannyml as nml

# Load synthetic data
reference, analysis, analysis_target = nml.load_synthetic_binary_classification_dataset()
data = pd.concat([reference, analysis], ignore_index=True)

# Extract meta data
metadata = nml.extract_metadata(data = reference, model_name='wfh_predictor', model_type='classification_binary', exclude_columns=['identifier'])
metadata.target_column_name = 'work_home_actual'

# Choose a chunker or set a chunk size
chunk_size = 5000

# Estimate model performance
estimator = nml.CBPE(model_metadata=metadata, metrics=['roc_auc'], chunk_size=chunk_size)
estimator.fit(reference)
estimated_performance = estimator.estimate(data=data)

figure = estimated_performance.plot(metric='roc_auc', kind='performance')
figure.show()

# Detect multivariate feature drift
multivariate_calculator = nml.DataReconstructionDriftCalculator(model_metadata=metadata, chunk_size=chunk_size)
multivariate_calculator.fit(reference_data=reference)
multivariate_results = multivariate_calculator.calculate(data=data)

figure = multivariate_results.plot(kind='drift')
figure.show()

# Detect univariate feature drift
univariate_calculator = nml.UnivariateStatisticalDriftCalculator(model_metadata=metadata, chunk_size=chunk_size)
univariate_calculator.fit(reference_data=reference)
univariate_results = univariate_calculator.calculate(data=data)

# Rank features based on number of alerts
ranker = nml.Ranker.by('alert_count')
ranked_features = ranker.rank(univariate_results, model_metadata=metadata, only_drifting = False)

for feature in ranked_features.feature:
    figure = univariate_results.plot(kind='feature_distribution', feature_label=feature)
    figure.show()

Thank you!

Hi!

When following the instructions and running poetry install -E test -E doc -E dev, extra packages such as tox are not installed. I think this is related to closed issue 67.

This is related to this Poetry behaviour. Extra packages should come from dependencies and not dev-dependencies, otherwise they are not installed.

I have tried to move the dev-dependencies packages within the dependencies section in pyproject.toml and after having updated the poetry.lock it works fine.

Do you want me to send a PR with this? I you think it would be better to somewhat separate "dev" dependencies from other dependencies, an option would be to use Poetry dependency groups with something such as a [tool.poetry.group.test.dependencies] section. I can do that in the PR if you think it is better.

• Changed confidence intervals to 95% instead of 99.99%
• Changed figure sizes and font sizes to make the figures more readable
• Added plots that zoom into the behaviour of methods on smaller shifts for the shifting mean and shifting SD experiments
• Split apart uniform distribution in the categorical section

I got TypeError new() missing 1 required positional argument: 'model_metadata' when running this chunk of the codes:

estimator = nml.CBPE( y_pred_proba='y_pred_proba', y_pred='y_pred', y_true='y', timestamp_column_name='timestamp', metrics=['roc_auc', 'f1'], chunk_size=5000, problem_type='classification_binary', )

nannyML version is 0.4.1

What does the error describe here? Thanks for help

Describe the bug NannyML gets the month and the date values wrong and can make a date of 01-09-2018 (1st September 2018) be thought of 09-01-2018 (9th January 2018).

To Reproduce Install NannyML and run the following code in a jupyter notebook:

import wget
from pathlib import Path
import pandas as pd
import nannyml as nml


url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00618/Steel_industry_data.csv'
download_foler = Path.home().joinpath("Downloads")
filename = wget.download(url,out = str(download_foler))

data = pd.read_csv(filename, header=0)
display(data.head())

features_selected = list(data.columns)[2:]

data['partition'] = 'train'
ind_ = data.shape[0]//3
data.loc[ind_:2*ind_, 'partition'] = 'reference'
data.loc[2*ind_: , 'partition'] = 'analysis'

reference = data.loc[data.partition == 'reference', :].reset_index(drop=True)
analysis = data.loc[data.partition == 'analysis', :].reset_index(drop=True)

calc = nml.UnivariateStatisticalDriftCalculator(
    feature_column_names=[features_selected[0]],
    timestamp_column_name='date'
)
calc.fit(reference)
results = calc.calculate(analysis)

drift_fig = results.plot(kind='feature_drift', feature_column_name=features_selected[0], plot_reference=True)
drift_fig.show()

display(results.data)
display(analysis)

A partial screenshot of the results.data can be seen below:

A partial screensgot of the first and last line of the analysis dataframe can be seen below:

By comparing the two screenshots you can see that some dates have been altered.

Expected behavior The dates would be recognized correctly. The misidentification of dates can be seen by comparing the start and end dates of the results data and the start and end dates of the analysis dataset.

Screenshots & scripts The problem is also visible on the following univariate drift plot:

• nannyml version: 0.3.2
• Python version: 3.10.4
• Operating System: macos 12.0.1

Hello

1. reference, analysis, analysis_target = nml.load_synthetic_binary_classification_dataset() does not work. error msg: AttributeError: module 'nannyml' has no attribute 'load_synthetic_binary_classification_dataset' Looks like instead the method load_synthetic_sample() must be used

2. metadata = nml.extract_metadata(data = reference, model_name='wfh_predictor', model_type='classification_binary', exclude_columns=['identifier']) also does not work. error msg: TypeError: extract_metadata() got an unexpected keyword argument 'model_type'

Issue here

Summary

• Improve readability of sequence checking conditionals by making it simpler

Testing Approach

• Since this is basically a refactor, I just ran poetry run tox to check if there we're any regressions.

Note

Trying to make contributing to open source stick. Feel free to disregard.

Motivation

• Was poking around the codebase when I noticed sequence checking could use improvement in terms of code readability

Solution

• We can use Truth Value Testing in checking the truthy-ness of sequences to make it simpler

Recommended Changes

• We can simplify the following code fragments:

- if len(some_sequence) > 0:
+ if some_sequence:
      do_something()

- if len(some_other_sequence) == 0:
+ if not some_other_sequence:
      do_something_else()

Additional Context

• Caveat; have excluded the ones in joy_plot.py as those are numpy sequences and is a different checking altogether

Note

Trying to make contributing to open source stick. Feel free to disregard.

What is this PR for ?

This PR is intended at improving the binning used in the calibration step by using np.histogram_bin_edges instead of the current _get_bin_index_edges function.

It also adds a new calibration_bin_count parameter to CBPE to be able to tune the number of bins to use in needs_calibration function.

Please note that the default values for bin_count are still 10. However it would probably better to switch to auto or fd as it would be generally better.

How was it tested?

New tests added in tests/test_calibration.py to validate the new behaviour. poetry run tox is passing.

Describe the bug

When we have MANY features into univariate drift and use code trying to plot them all it fails silently. The machine still computes but does, presumably?, nothing.

To Reproduce

Posting example code since a full reproducible example relies on internal compute resources.

# load data and create reference and analysis dataframes
# in this cases fabert openml dataset

chunker = nml.SizeBasedChunker(chunk_size=_suggested_chunk_size)


univariate_calculator = nml.UnivariateDriftCalculator(
    column_names=feature_column_names,
    continuous_methods=['jensen_shannon'],
    categorical_methods=['jensen_shannon'],
    chunker=chunker,
)
univariate_calculator = univariate_calculator.fit(reference)
univariate_results = univariate_calculator.calculate(analysis)

In our case the selected features number is 800. Trying to create one plot with:

figure2 = univariate_results.filter(
    period='all',
    column_names=univariate_results.column_names, 
    methods=['jensen_shannon']).plot(kind='drift')

will fail and the Jupyterlab notebook kept running doing, presumably, nothing.

Expected behavior A plot would be created.

Additional context If instead we try to create plots one by one, things work fine:

for ftr in univariate_results.column_names:
    _fgr = univariate_results.filter(
        period='all',
        column_names=[ftr], 
        methods=['jensen_shannon']
    ).plot(kind='drift')
    _fgr.write_image(f"{_figure_folder}/drift-{ftr}.svg")

# missing code that creates relevant dataframes
feature_column_names = train.drop(['fraud_reported', 'timestamp'],axis = 1).columns

univariate_calculator = nml.UnivariateDriftCalculator(
    column_names=list(feature_column_names),
    continuous_methods='jensen_shannon')

univariate_calculator = univariate_calculator.fit(test)
univariate_results = univariate_calculator.calculate(val)

---------------------------------------------------------------------------
InvalidArgumentsException                 Traceback (most recent call last)
Cell In[23], line 7
      1 feature_column_names = train.drop(['fraud_reported', 'timestamp'],axis = 1).columns
      3 univariate_calculator = nml.UnivariateDriftCalculator(
      4     column_names=list(feature_column_names),
      5     continuous_methods='kolmogorov_smirnov')
----> 7 univariate_calculator = univariate_calculator.fit(test)
      8 univariate_results = univariate_calculator.calculate(val)
     10 # for column_name in univariate_calculator.continuous_column_names:
     11 #     figure = univariate_results.plot(
     12 #         kind='drift',
   (...)
     16 #     )
     17 #     figure.show()

File ~/.conda/envs/nanny/lib/python3.10/site-packages/nannyml/base.py:138, in AbstractCalculator.fit(self, reference_data, *args, **kwargs)
    136 try:
    137     self._logger.debug(f"fitting {str(self)}")
--> 138     return self._fit(reference_data, *args, **kwargs)
    139 except InvalidArgumentsException:
    140     raise

File ~/.conda/envs/nanny/lib/python3.10/site-packages/nannyml/usage_logging.py:189, in log_usage.<locals>.logging_decorator.<locals>.logging_wrapper(*args, **kwargs)
    187 finally:
    188     if runtime_exception is not None:
--> 189         raise runtime_exception
    190     else:
    191         return res

File ~/.conda/envs/nanny/lib/python3.10/site-packages/nannyml/usage_logging.py:142, in log_usage.<locals>.logging_decorator.<locals>.logging_wrapper(*args, **kwargs)
    139 runtime_exception, res = None, None
    140 try:
    141     # run original function
--> 142     res = func(*args, **kwargs)
    143 except BaseException as exc:
    144     runtime_exception = exc

File ~/.conda/envs/nanny/lib/python3.10/site-packages/nannyml/drift/univariate/calculator.py:109, in UnivariateDriftCalculator._fit(self, reference_data, *args, **kwargs)
    104 self.continuous_column_names, self.categorical_column_names = _split_features_by_type(
    105     reference_data, self.column_names
    106 )
    108 for column_name in self.continuous_column_names:
--> 109     self._column_to_models_mapping[column_name] += [
    110         MethodFactory.create(key=method, feature_type=FeatureType.CONTINUOUS, chunker=self.chunker).fit(
    111             reference_data[column_name]
    112         )
    113         for method in self.continuous_method_names
    114     ]
    116 for column_name in self.categorical_column_names:
    117     self._column_to_models_mapping[column_name] += [
    118         MethodFactory.create(key=method, feature_type=FeatureType.CATEGORICAL, chunker=self.chunker).fit(
    119             reference_data[column_name]
    120         )
    121         for method in self.categorical_method_names
    122     ]

File ~/.conda/envs/nanny/lib/python3.10/site-packages/nannyml/drift/univariate/calculator.py:110, in <listcomp>(.0)
    104 self.continuous_column_names, self.categorical_column_names = _split_features_by_type(
    105     reference_data, self.column_names
    106 )
    108 for column_name in self.continuous_column_names:
    109     self._column_to_models_mapping[column_name] += [
--> 110         MethodFactory.create(key=method, feature_type=FeatureType.CONTINUOUS, chunker=self.chunker).fit(
    111             reference_data[column_name]
    112         )
    113         for method in self.continuous_method_names
    114     ]
    116 for column_name in self.categorical_column_names:
    117     self._column_to_models_mapping[column_name] += [
    118         MethodFactory.create(key=method, feature_type=FeatureType.CATEGORICAL, chunker=self.chunker).fit(
    119             reference_data[column_name]
    120         )
    121         for method in self.categorical_method_names
    122     ]

File ~/.conda/envs/nanny/lib/python3.10/site-packages/nannyml/drift/univariate/methods.py:152, in MethodFactory.create(cls, key, feature_type, **kwargs)
    149     raise InvalidArgumentsException(f"cannot create method given a '{type(key)}'. Please provide a string.")
    151 if key not in cls.registry:
--> 152     raise InvalidArgumentsException(
    153         f"unknown method key '{key}' given. "
    154         "Should be one of ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein', 'chi2', "
    155         "'jensen_shannon', 'l_infinity', 'hellinger']."
    156     )
    158 if feature_type not in cls.registry[key]:
    159     raise InvalidArgumentsException(f"method {key} does not support {feature_type.value} features.")

InvalidArgumentsException: unknown method key 'k' given. Should be one of ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein', 'chi2', 'jensen_shannon', 'l_infinity', 'hellinger'].

The problem seems to be that the user incorrectly gave a string instead of a list of strings. However instead of getting a proper error that he should give a list, he gets an error of unknown method k.

I m assuming (because I didn't dig into the code) the string is turned into a list and then the first letter is attempted to be matched against our string options.

P.S. Alternatively we can also accept a string here and handle it appropriately. Nice to have but less important than an error that correctly points out what went wrong.