Add Random Forest and Gradient Boosting from sklearn to the single table detection tests. Being able to fool these classifiers would be a great improvement for generative models.

Environment Details

Please indicate the following details about the environment in which you found the bug:

• SDMetrics version: 0.7.0
• Python version: 3.8
• Operating System: Linux VM
• pandas version: 1.2.4

Error Description

Following the README.md example of calculating the BoundaryAdherence:

Running the code gives the following error: AttributeError: 'Series' object has no attribute 'columns'

I would expect the code to generate the BoundaryAdherence for the start_date column of the real_data pandas dataframe and synthetic_Data pandas dataframe

Steps to reproduce

Follow this snippet from the README.md that shows the usage of BoundaryAdherence

# calculate whether the synthetic data respects the min/max bounds
# set by the real data
from sdmetrics.single_table import BoundaryAdherence

BoundaryAdherence.compute(
    real_data['start_date'],
    synthetic_data['start_date']
)

Solution

I will open a PR for this later today or tomorrow.

type(real_data['start_date']) 

Returns a pandas.core.series.Series

type(real_data[['start_date']])

Returns a pandas.core.frame.DataFrame

BoundaryAdherence.compute() expects DataFrames for both the real_data and synthetic_data arguments.

# calculate whether the synthetic data respects the min/max bounds
# set by the real data
from sdmetrics.single_table import BoundaryAdherence

BoundaryAdherence.compute(
    real_data[['start_date']],
    synthetic_data[['start_date']]
)

Environment Details

Please indicate the following details about the environment in which you found the bug:

• SDMetrics version:'0.7.0'
• Python version: 3.8.8
• Operating System: Windows server 2016 dataserver

Error Description

Steps to reproduce

I tried to use sdmetrics to compare the Synthetic data. Example of the data set

Customer | State | Customer Lifetime Value | Response | Coverage | Education | Effective To Date | EmploymentStatus | Gender | Income | ... | Months Since Policy Inception | Number of Open Complaints | Number of Policies | Policy Type | Policy | Renew Offer Type | Sales Channel | Total Claim Amount | Vehicle Class | Vehicle Size -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- BU79786 | Washington | 2763.519279 | No | Basic | Bachelor | 2/24/11 | Employed | F | 56274 | ... | 5 | 0 | 1 | Corporate Auto | Corporate L3 | Offer1 | Agent | 384.811147 | Two-Door Car | Medsize QZ44356 | Arizona | 6979.535903 | No | Extended | Bachelor | 1/31/11 | Unemployed | F | 0 | ... | 42 | 0 | 8 | Personal Auto | Personal L3 | Offer3 | Agent | 1131.464935 | Four-Door Car | Medsiz

loaded the Gaussian copula pkl model to generate the synthetic data and created a metadata on the original dataset to use it in quality report. I have pasted the Metadata created for the data above: {'fields': {'Customer': {'type': 'id', 'subtype': 'string'}, 'State': {'type': 'categorical'}, 'Customer Lifetime Value': {'type': 'numerical', 'subtype': 'float'}, 'Response': {'type': 'categorical'}, 'Coverage': {'type': 'categorical'}, 'Education': {'type': 'categorical'}, 'Effective To Date': {'type': 'categorical'}, 'EmploymentStatus': {'type': 'categorical'}, 'Gender': {'type': 'categorical'}, 'Income': {'type': 'numerical', 'subtype': 'integer'}, 'Location Code': {'type': 'categorical'}, 'Marital Status': {'type': 'categorical'}, 'Monthly Premium Auto': {'type': 'numerical', 'subtype': 'integer'}, 'Months Since Last Claim': {'type': 'numerical', 'subtype': 'integer'}, 'Months Since Policy Inception': {'type': 'numerical', 'subtype': 'integer'}, 'Number of Open Complaints': {'type': 'numerical', 'subtype': 'integer'}, 'Number of Policies': {'type': 'numerical', 'subtype': 'integer'}, 'Policy Type': {'type': 'categorical'}, 'Policy': {'type': 'categorical'}, 'Renew Offer Type': {'type': 'categorical'}, 'Sales Channel': {'type': 'categorical'}, 'Total Claim Amount': {'type': 'numerical', 'subtype': 'float'}, 'Vehicle Class': {'type': 'categorical'}, 'Vehicle Size': {'type': 'categorical'}}, 'primary_key': 'Customer'}.-->

#creating metadata
metadata.add_table(
    name='INS',
    data=data2,
    primary_key='Customer'
     )


report = QualityReport()
report.generate(data2, synthetic_data,metadata)


error:
Creating report:   0%|          | 0/4 [00:00<?, ?it/s]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [54], in <cell line: 2>()
      1 report = QualityReport()
----> 2 report.generate(data2, synthetic_data,metadata)

File ~\.conda\envs\SDVENV\lib\site-packages\sdmetrics\reports\single_table\quality_report.py:72, in QualityReport.generate(self, real_data, synthetic_data, metadata)
     70 for metric in tqdm.tqdm(metrics, desc='Creating report'):
     71     try:
---> 72         self._metric_results[metric.__name__] = metric.compute_breakdown(
     73             real_data, synthetic_data, metadata)
     74     except IncomputableMetricError:
     75         # Metric is not compatible with this dataset.
     76         self._metric_results[metric.__name__] = {}

File ~\.conda\envs\SDVENV\lib\site-packages\sdmetrics\single_table\multi_single_column.py:147, in MultiSingleColumnMetric.compute_breakdown(cls, real_data, synthetic_data, metadata, **kwargs)
    123 @classmethod
    124 def compute_breakdown(cls, real_data, synthetic_data, metadata=None, **kwargs):
    125     """Compute this metric broken down by column.
    126 
    127     This is done by computing the underlying SingleColumn metric to all the
   (...)
    145             A mapping of column name to metric output.
    146     """
--> 147     return cls._compute(
    148         cls, real_data, synthetic_data, metadata, store_errors=True, **kwargs)

File ~\.conda\envs\SDVENV\lib\site-packages\sdmetrics\single_table\multi_single_column.py:68, in MultiSingleColumnMetric._compute(self, real_data, synthetic_data, metadata, store_errors, **kwargs)
     43 def _compute(self, real_data, synthetic_data, metadata=None, store_errors=False, **kwargs):
     44     """Compute this metric for all columns.
     45 
     46     This is done by computing the underlying SingleColumn metric to all the
   (...)
     66             A mapping of column name to metric output.
     67     """
---> 68     real_data, synthetic_data, metadata = self._validate_inputs(
     69         real_data, synthetic_data, metadata)
     71     fields = self._select_fields(metadata, self.field_types)
     72     invalid_cols = set(metadata['fields'].keys()) - set(fields)

File ~\.conda\envs\SDVENV\lib\site-packages\sdmetrics\single_table\base.py:119, in SingleTableMetric._validate_inputs(cls, real_data, synthetic_data, metadata)
    116 if not isinstance(metadata, dict):
    117     metadata = metadata.to_dict()
--> 119 fields = metadata['fields']
    120 for column in real_data.columns:
    121     if column not in fields:

KeyError: 'fields'

Environment Details

Please indicate the following details about the environment in which you found the bug:

• SDMetrics version: '0.7.0'
• Python version: 3.9.13
• Operating System: Windows under WSL2--Unbuntu

Error Description

Attepmting to generate report from the following command report = QualityReport() report.generate(x_y_df, synth_data, meta_data)

Steps to reproduce

x_y_df.to_json(reports_path + '/' + 'xy_df.json')

    with open(reports_path + '/' + 'xy_df.json') as f:
        meta_data = json.load(f)

    
    # Initialize report
    report = QualityReport()
    report.generate(x_y_df, synth_data, meta_data)

Traceback:

File "/home/bdeck8317/miniconda3/lib/python3.9/site-packages/sdmetrics/reports/single_table/quality_report.py", line 72, in generate
    self._metric_results[metric.__name__] = metric.compute_breakdown(
  File "/home/bdeck8317/miniconda3/lib/python3.9/site-packages/sdmetrics/single_table/multi_single_column.py", line 147, in compute_breakdown
    return cls._compute(
  File "/home/bdeck8317/miniconda3/lib/python3.9/site-packages/sdmetrics/single_table/multi_single_column.py", line 68, in _compute
    real_data, synthetic_data, metadata = self._validate_inputs(
  File "/home/bdeck8317/miniconda3/lib/python3.9/site-packages/sdmetrics/single_table/base.py", line 119, in _validate_inputs
    fields = metadata['fields']
KeyError: 'fields'

Thanks, Ben

Environment Details

• SDV version: sdv==0.17.1
• Python version: Python 3.9.13
• Operating System: Linux

Error Description

pandas==1.4.3

ValueError when running NewRowSynthesis

Steps to reproduce

from sdmetrics.single_table import NewRowSynthesis

metadata_obj, real_data = load_tabular_demo("student_placements_pii", metadata=True)

model = GaussianCopula(
    primary_key="student_id"
)
model.fit(real_data)
synthetic_data = model.sample(250)

new_row_synthesis_score = NewRowSynthesis.compute(
    real_data=real_data, synthetic_data=synthetic_data, metadata=metadata_obj.to_dict()
)

ValueError                                Traceback (most recent call last)
Cell In [43], line 11
      8 model.fit(real_data)
      9 synthetic_data = model.sample(250)
---> 11 new_row_synthesis_score = NewRowSynthesis.compute(
     12     real_data=real_data, synthetic_data=synthetic_data, metadata=metadata_obj.to_dict()
     13 )

File ~/miniconda3/envs/mloperator/lib/python3.9/site-packages/sdmetrics/single_table/new_row_synthesis.py:104, in NewRowSynthesis.compute(cls, real_data, synthetic_data, metadata, numerical_match_tolerance, synthetic_sample_size)
    101     row_filter.append(field_filter)
    103 try:
--> 104     matches = real_data.query(' and '.join(row_filter))
    105 except TypeError:
    106     if len(real_data) > 10000:

File ~/miniconda3/envs/mloperator/lib/python3.9/site-packages/pandas/core/frame.py:4111, in DataFrame.query(self, expr, inplace, **kwargs)
   4109 kwargs["level"] = kwargs.pop("level", 0) + 1
   4110 kwargs["target"] = None
-> 4111 res = self.eval(expr, **kwargs)
   4113 try:
   4114     result = self.loc[res]

File ~/miniconda3/envs/mloperator/lib/python3.9/site-packages/pandas/core/frame.py:4240, in DataFrame.eval(self, expr, inplace, **kwargs)
   4237     kwargs["target"] = self
...
    328     )
    329 engine = _check_engine(engine)
    330 _check_parser(parser)

ValueError: multi-line expressions are only valid in the context of data, use DataFrame.eval

Environment Details

• SDV version: sdv==0.17.1
• Python version: Python 3.9.13
• Operating System: Linux

Error Description

pandas==1.4.3

ValueError when running BoundaryAdherence

Steps to reproduce

from sdmetrics.single_column import BoundaryAdherence

metadata_obj, real_data = load_tabular_demo("student_placements_pii", metadata=True)

model = GaussianCopula(
    primary_key="student_id"
)
model.fit(real_data)
synthetic_data = model.sample(250)

BoundaryAdherence.compute(
    real_data=real_data, synthetic_data=synthetic_data
)

ValueError                                Traceback (most recent call last)
Cell In [42], line 11
      8 model.fit(real_data)
      9 synthetic_data = model.sample(250)
---> 11 BoundaryAdherence.compute(
     12     real_data=real_data, synthetic_data=synthetic_data
     13 )

File ~/miniconda3/envs/mloperator/lib/python3.9/site-packages/sdmetrics/single_column/statistical/boundary_adherence.py:46, in BoundaryAdherence.compute(cls, real_data, synthetic_data)
     32 @classmethod
     33 def compute(cls, real_data, synthetic_data):
     34     """Compute the boundary adherence of two continuous columns.
     35 
     36     Args:
   (...)
     44             The boundary adherence of the two columns.
     45     """
---> 46     real_data = pd.Series(real_data).dropna()
     47     synthetic_data = pd.Series(synthetic_data).dropna()
     49     if is_datetime(real_data):

File ~/miniconda3/envs/mloperator/lib/python3.9/site-packages/pandas/core/series.py:367, in __init__(self, data, index, dtype, name, copy, fastpath)
    364         self.name = name
    365     return
...
   1528         f"The truth value of a {type(self).__name__} is ambiguous. "
   1529         "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
   1530     )

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

BoundaryAdherence expects DataFrames for the real and synthetic dataframe arguments, currently it is receiving series due to accessing date_range with single square brackets instead of double square brackets.

Is there any paper to cite related to this library? I am evaluating some synthetic data for a publication so I wanted to include your contribution among the citations.

Thanks Andrea Galloni

Environment Details

Please indicate the following details about the environment in which you found the bug:

• SDMetrics version: 0.4.2
• SDV version: 0.14.1
• Python version: 3.8.10
• Operating System: ubuntu server lts 20.4.04

Error Description

The latest SDMetrics version (which is installed by default when installing SDV) has incompatible copula requirements with downstream SDV.

Steps to reproduce

On a fresh virtual environment, install pip-tools.

Place the following on a file named requirements.in

sdv
#sdmetrics==0.4.1

Type the following commands

pip install -r requirements.in
pip-compile requirements.in

pip-compile reports:

Could not find a version that matches copulas<0.7,<0.8,>=0.6.1,>=0.7.0 (from sdv==0.14.1->-r requirements.txt (line 1))
Tried: 0.0.0, 0.0.0, 0.1.0, 0.1.0, 0.1.1, 0.1.1, 0.2.0, 0.2.0, 0.2.1, 0.2.1, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.3.0, 0.3.0, 0.3.2, 0.3.2, 0.3.3, 0.3.3, 0.4.0, 0.4.0, 0.5.0, 0.5.0, 0.5.1, 0.5.1, 0.6.0, 0.6.0, 0.6.1, 0.6.1, 0.7.0, 0.7.0
Skipped pre-versions: 0.3.0.dev0, 0.3.0.dev0, 0.3.2.dev1, 0.3.2.dev1, 0.3.3.dev0, 0.3.3.dev0, 0.4.0.dev0, 0.4.0.dev0, 0.5.0.dev0, 0.5.0.dev0, 0.5.0.dev1, 0.5.0.dev1, 0.5.1.dev0, 0.5.1.dev0, 0.5.1.dev1, 0.5.1.dev1, 0.5.2.dev0, 0.5.2.dev0, 0.5.2.dev1, 0.5.2.dev1, 0.6.0.dev0, 0.6.0.dev0, 0.6.1.dev0, 0.6.1.dev0, 0.7.0.dev0, 0.7.0.dev0
There are incompatible versions in the resolved dependencies:
  copulas<0.8,>=0.7.0 (from sdmetrics==0.4.2->sdv==0.14.1->-r requirements.txt (line 1))
  copulas<0.7,>=0.6.1 (from sdv==0.14.1->-r requirements.txt (line 1))

From the setup.py of both projects, we can verify the above requirements.

pip install works correctly, but we get the following (snippet):

Collecting llvmlite<0.39,>=0.38.0rc1
  Using cached llvmlite-0.38.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
Collecting charset-normalizer~=2.0.0; python_version >= "3"
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5; python_version >= "3"
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
ERROR: numba 0.55.1 has requirement numpy<1.22,>=1.18, but you'll have numpy 1.22.3 which is incompatible.
ERROR: rdt 0.6.4 has requirement scipy<1.8,>=1.5.4, but you'll have scipy 1.8.0 which is incompatible.
ERROR: sdmetrics 0.4.2 has requirement copulas<0.8,>=0.7.0, but you'll have copulas 0.6.1 which is incompatible.
Installing collected packages: tqdm, typing-extensions, torch, numpy, six, python-dateutil, pytz, pandas, deepecho, scipy, threadpoolctl, joblib, scikit-learn, llvmlite, numba, pyts, pyyaml, psutil, rdt, fonttools, cycler, pyparsing, packaging, pillow, kiwisolver, matplotlib, copulas, sdmetrics, charset-normalizer, idna, certifi, urllib3, requests, torchvision, ctgan, graphviz, text-unidecode, Faker, sdv

When uncommenting sdmetrics from requirements.in, both commands run "correctly".

Furthermore, when pip-compile and pip have cached sdmetrics==0.4.1, they both select that version instead and no error is shown.

The following file never compiles:

sdv==0.14.1
sdmetrics==0.4.2

I don't know what the appropriate solution to something like this would be. I'm not a library developer.

The current README doesn't print the latest output. More specifically, the command sdmetrics.compute_metrics(metrics, real_data, synthetic_data, metadata=metadata) currently doesn't print the same as what the README prints (e.g. the current code produces a column named error containing None values which the README doesn't have, as well as other changes).

Environment Details

• SDMetrics version: 0.4.1
• Python version: 3.7

Error Description

The relational KSTest is supposed to run the KSTest on all numerical columns in all tables and return the average score.

However, this test crashes if it encounters a table that has no numerical columns. I expect this test to succeed as long as there is at least 1 numerical column in any of the tables.

Steps to reproduce

Use the relational demo dataset and pass it in with the metadata.

from sdv.metrics.demos import load_multi_table_demo
from sdv.metrics.relational import KSTest

real_data, synthetic_data, metadata = load_multi_table_demo()
KSTest.compute(real_data, synthetic_data, metadata)

Output:

/usr/local/lib/python3.7/dist-packages/sdmetrics/single_table/base.py in _select_fields(cls, metadata, types)
     78 
     79         if len(fields) == 0:
---> 80             raise IncomputableMetricError(f'Cannot find fields of types {types}')
     81 
     82         return fields

IncomputableMetricError: Cannot find fields of types ('numerical',)

I believe this is happening because table sessions has no numerical columns. Interestingly, it does work if I exclude the metadata object -- because then it starts assuming that the id field is a numerical column.

KSTest.compute(real_data, synthetic_data)

0.8555555555555556

Problem Description

The Detection metrics use machine learning to determine whether the real vs. synthetic data can be detected. For this to work, we should only be using columns that are statistically modeled.

Expected behavior

When running any of the detection metrics, the following columns should be ignored:

• Primary keys
• Foreign keys
• Any other kinds of IDs
• PII or sensitive data
• Text data (or data created by RegEx)

None of these columns provide any useful information for detection.

The remaining data types are statistically modeled and should be included: numerical, datetime, categorical (non-PII), boolean

Additional context

We already filtered out primary keys in #119. The issue of foreign keys is discussed in #285.

Environment details

If you are already running SDMetrics, please indicate the following details about the environment in which you are running it:

• SDMetrics version: 0.8.2
• Python version:
• Operating System:

Problem description

Nice correction for DetectionMetric (Solving primary_key use for detection metrics #251, https://github.com/sdv-dev/SDMetrics/pull/251) Removing the primary key from the table makes more sense for the evaluation. But, what about foreign keys present in a table of a relational database. Does not the same problem also occur for the foreign keys too and they need to be deleted? Also, for the parent-child logistic detection metric, the foreign keys (referencing parent rows) in the child tables are no longer required when the denormalized tables are used.

I'm filing this issue on behalf of a user request on our Slack.

Problem Description

Currently, the single column visualization function (utils.get_column_plot) only supports columns that are numerical, categorical, boolean or datetime. It would be nice to support foreign keys as well.

Expected behavior

If I provide a foreign key name into the utils.get_column_plot function, then I expect to see a plot of real vs. synthetic data:

1. Compute the cardinality (# of children with the same parent) for the synthetic and real data. This will form 2 distributions for real & synthetic data
2. Plot those distributions similar to how we plot numerical data. The x-axis label should read "Cardinality".

This can be done from get_column_plot.

from sdmetrics.reports import utils

fig = utils.get_column_plot(
    real_data=real_table,
    synthetic_data=synthetic_table,
    column_name='foreign_key_column',
    metadata=my_table_metadata_dict
)

fig.show()

Environment Details

• SDMetrics version: 0.8.0
• Python version: 3.7
• Operating System: Linux

Error Description

A numerical column in the real data may contain missing values. Sometimes, the synthetic data may only produce these missing values and fail to create any numerical values. In such cases, the software crashes when I try to produce a quality report.

Expected Behavior: Certain metrics may not be computable if there are only NaN values. But instead of crashing the report, the error should be noted in the detailed breakdowns, and the report should still produce a score while ignoring the values (along with details, visualizations, etc.)

Steps to reproduce

import pandas as pd
from sdmetrics.reports.single_table import QualityReport

real_data = pd.DataFrame(data={
    'col1': [1, 2, 1, 3, 4],
    'col2': [2, 4, 1, 7, 1]
})

# the 'col2' only contain NaN values
synthetic_data = pd.DataFrame(data={
    'col1': [1, 3, 2, 2, 1],
    'col2': [np.nan]*5
})

metadata = {
    'fields': {
        'col1': { 'type': 'numerical', 'subtype': 'integer' },
        'col2': { 'type': 'numerical', 'subtype': 'integer' }
    }
}

report = QualityReport()
report.generate(real_data, synthetic_data, metadata)

Output

Creating report:  50%|█████     | 2/4 [00:00<00:00, 106.06it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-55-a6be2bd142ce> in <module>
     17 
     18 report = QualityReport()
---> 19 report.generate(real_data, synthetic_data, metadata)

3 frames
/usr/local/lib/python3.7/dist-packages/sdmetrics/reports/single_table/quality_report.py in generate(self, real_data, synthetic_data, metadata)
     71             try:
     72                 self._metric_results[metric.__name__] = metric.compute_breakdown(
---> 73                     real_data, synthetic_data, metadata)
     74             except IncomputableMetricError:
     75                 # Metric is not compatible with this dataset.

/usr/local/lib/python3.7/dist-packages/sdmetrics/single_table/multi_column_pairs.py in compute_breakdown(cls, real_data, synthetic_data, metadata, **kwargs)
    128             synthetic = synthetic_data[list(sorted_columns)]
    129             breakdown[sorted_columns] = cls.column_pairs_metric.compute_breakdown(
--> 130                 real, synthetic, **kwargs)
    131 
    132         return breakdown

/usr/local/lib/python3.7/dist-packages/sdmetrics/column_pairs/statistical/correlation_similarity.py in compute_breakdown(cls, real_data, synthetic_data, coefficient)
     83 
     84         correlation_real, _ = correlation_fn(real_data[column1], real_data[column2])
---> 85         correlation_synthetic, _ = correlation_fn(synthetic_data[column1], synthetic_data[column2])
     86 
     87         if np.isnan(correlation_real) or np.isnan(correlation_synthetic):

/usr/local/lib/python3.7/dist-packages/scipy/stats/stats.py in pearsonr(x, y)
   4014 
   4015     if n < 2:
-> 4016         raise ValueError('x and y must have length at least 2.')
   4017 
   4018     x = np.asarray(x)

ValueError: x and y must have length at least 2.

Note: It is OK that the correlation metric is crashing (correlation is undefined if there are no values). But the report should not crash.