Summary

This PR adds optional static type checking using mypy, which uses the type hints to check for errors that don't show up from unit tests.

• [x] Code passes all tests
• [x] Unit tests provided for these changes
• [x] Documentation and docstrings added for these changes

Changes

1. Add mypy environment to tox.ini
2. Add call to mypy via tox in the GHA configuration for CI
3. Make a few improvements to code based on suggestions from type checker
4. Add additional tests

Summary

As a follow-up to #33, this PR adds a duplicate_of annotation to the rexmex.utils.Annotator class and begins annotating which functions are duplicate of each other.

~Caveat I'm not really happy with the direction of which is the "duplicate" in many cases, e.g., where miss_rate is the "canonical" one and "false_negative_rate" is the duplicate~

• [x] Code passes all tests
• [x] Unit tests provided for these changes
• [x] Documentation and docstrings added for these changes
• [x] Check all functions are annotated, and correctly

Changes

• Add new annotation duplicate_of to rexmex.utils.Annotator
• Annotate duplicate functions (e.g., precision_score is a duplicate of positive_predictive_value)
• Pins pandas<=1.3.5 since they just put the 1.4 release candidate up on PyPI this morning and it doesn't work with the conda env in the GHA workflow

I noticed that there are duplicate functions such as miss_rate()/false_negative_rate() and fall_out()/false_positive_rate(). What's the reason for the duplication?

Summary

Fixes #56 by removing deprecated sklearn dependency.

• [x] Code passes all tests
• [x] Unit tests provided for these changes (does not apply)
• [x] Documentation and docstrings added for these changes (does not apply)

Changes

• Remove deprecated sklearn dependency

Hey rexmex team,

I wanted to ask you if there is a specific reason why your installation requires pandas to be of this version?

Since this package gets installed with the latest version of PyKEEN, I am facing some versioning issues due to the Pandas version restriction of your package.

The related line of code: https://github.com/AstraZeneca/rexmex/blob/44f453ff20e92569270b9e1cfcb75b44b7839128/setup.py#L3

Apologies if it's a silly question but: is it indeed a strict requirement for rexmex to have Pandas<1.3.5 or is this something we can modify in rexmex setup.py? ( if so I can open a related PR for it )

Thank you in advance! Best, Dimitris

Summary

Please provide a high-level summary of the changes for the changes and notes for the reviewers

• [X] Code passes all tests
• [X] Unit tests provided for these changes
• [X] Documentation and docstrings added for these changes

Changes

• changed signature for the Coverage metrics, it now requires supplying the relevant user and item spaces plus a list of tuples (user, item) as final predictions
• added the item coverage and user coverage metrics to CoverageMetricSet
• created a ScoreCard for coverage metrics (they need a different signature than classification and regression)

Summary

Similarly to #35, this PR adds an additional annotation for functions that need to be binarized.

• [x] Code passes all tests
• [x] Unit tests provided for these changes
• [x] Documentation and docstrings added for these changes

Changes

• Add additional keyword argument binarize to rexmex.utils.Annotator.annotate. This has a default of False, since most functions do not need to be binarized.
• Annotate binarize=True on the functions that were binarized in the rexmex.metricset.ClassificationMetricSet.__init__

Future outlook

This improvement will enable the later implementation of automated collection and processing of metric functions to improve the rexmex.metricset.ClassificationMetricSet class.

Summary

Closes #28

This PR adds four new annotations to classification functions:

• lower bound inclusive
• upper bound inclusive
• a description
• a URL link to more information / citation

Checks:

• [x] Code passes all tests
• [x] Unit tests provided for these changes
• [x] Documentation and docstrings added for these changes

Changes

This PR adds new annotation requirements and applies them to all classification functions

Summary

Added a basic coverage statistic + some tests. The naming convention might need to be refactored though, not sure if item_coverage() is the right name (or perhaps catalogue_coverage() instead?)

• [X] Code passes all tests
• [X] Unit tests provided for these changes
• [X] Documentation and docstrings added for these changes

Changes

• added new metric (coverage) which check how many of the possible objects/items get recommended at least once, expressed as a fraction (if 1 out of 5 items never got recommended, it will be 80%)

Summary

This PR demonstrates a potential solution to #28 in a pilot applied to classification metrics. It could be extended in a future pull request to other branches of the package.

• [x] Code passes all tests
• [x] Unit tests provided for these changes
• [x] Documentation and docstrings added for these changes

Changes

• [x] Add an annotate decorator that adds some structured information to classification functions
• [x] Add tests to make sure the data is accessible
• [x] Add unit test to ensure all classification functions are annotated (1afd68b)
• [x] Annotate all classification functions

Future

Before finalizing this PR, I had also used the annotate function to make a registry of functions. This could be used to make the generation/maintenance of the ClassificationMetricSet much easier, but I'd save that for a different PR.

Right now it's a bit round-about to get a scorecard for a given dataset since it expect a pandas format. I'd suggest exposing ScoreCard._get_performance_metrics as a public user interface and also encourage people to use that directly in case they're generating their own y_true and y_score and don't want to write their own code to generate a pandas dataframe from it, just for rexmex to need to unpack it.

My example is in PyKEEN, where we do just that: https://github.com/pykeen/pykeen/blob/799e224e772176703d796a9247bfcc179d343c6c/src/pykeen/evaluation/sklearn.py#L129-L142

I'd also say it would be worth adding a second introductory example based on this in the README.

Summary

This PR uses the rexmex.utils.Annotator to annotate information about ranking metrics (e.g., MR, MRR, Hits @ K). I'm not familiar with all of the rankings, so help would be great on this one. This would especially be good for first-time contributors, since a lot of it is busy work of looking up metrics, finding out about their properties, etc.. A potential contributor could make a branch off of mine and then either PR it directly, or PR it into my fork (or just post the curation as a comment in this PR, and I can make the code updates while crediting them as a co-author on the relevant commits)

• [ ] Code passes all tests
• [ ] Unit tests provided for these changes
• [ ] Documentation and docstrings added for these changes
• [ ] https://github.com/AstraZeneca/rexmex/pull/43, since it would be good to re-use its generalized testing framework

Changes

• [x] Add annotator to rexmex.metrics.ratings and annotate its functions
• [ ] Switch construction of metric set to use the annotator's registry

The current binarize function uses a cutoff of 0.5 for binarization: https://github.com/AstraZeneca/rexmex/blob/3e266529761281ae832e49736e48d3e46f3b4af4/rexmex/utils.py#L28-L34

This is an issue for PyKEEN, where the scores that come from a model could all be on the range of [-5,-2]. The current TODO text says to use https://en.wikipedia.org/wiki/Youden%27s_J_statistic, but it's not clear how that would be used.

As an alternative, the NetMF package implements the following code for constructing an indicator that might be more applicable (though I don't personally recognize what method this is, and unfortunately it's not documented):

def construct_indicator(y_score, y):
    # rank the labels by the scores directly
    num_label = np.sum(y, axis=1, dtype=np.int)
    y_sort = np.fliplr(np.argsort(y_score, axis=1))
    y_pred = np.zeros_like(y, dtype=np.int)
    for i in range(y.shape[0]):
        for j in range(num_label[i]):
            y_pred[i, y_sort[i, j]] = 1
    return y_pred

Summary

This PR streamlines generating metric sets and annotates more functions.

• [x] Code passes all tests
• [x] Unit tests provided for these changes
• [x] Documentation and docstrings added for these changes

Changes

• [x] Update the rexmex.utils.Annotator class to include a key. If it's not given, this defaults to the function's name. The registry now uses the key instead of the function's name
• [x] Update the metric sets to load the function names directly from the keys in the annotator's dictionary
• [x] Annotate functions in the ratings module
• [x] Generalizes tests to make it easier to test the existence of annotations for ratings, coverage, and rankings

@mberr's adjusted mean rank address some of the problems with the mean rank, including its size dependence. Reference: https://arxiv.org/abs/2002.06914