scikit-learn models hyperparameters tuning and feature selection, using evolutionary algorithms.

Is your feature request related to a problem? Please describe. NA

Describe the solution you'd like Implement in the sklearn_genetic.plots module a function named plot_parallel_coordinates to inspect the results of the learning process

Describe alternatives you've considered The function should take two arguments:

• estimator: A fitted estimator from sklearn_genetic.GASearchCV
• features: list, default=None. Subset of features to plot, if None it plots all the features by default

The function should return an object to plot parallel coordinates according the pandas.plotting.parallel_coordinates function

The data to plot is available on the estimator.logbook object, look the implementation of the plot_search_space function to see how to convert this data to a pandas data frame

The function must select only the non categorical variables, this can be done by inspecting the estimator.space object and comparing against the data types defined in sklearn_genetic.space, i.e Categorical, Continuous and Integer and color against the "score" column. In the same way, it must validate and make a warning if in the features parameter a Categorial one is passed

Additional context Links of some implementations:

• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.plotting.parallel_coordinates.html
• https://www.mathworks.com/help/stats/feature-selection-and-feature-transformation.html#buwh6hc-1

Hello,

I have been looking into your package and it is really cool. Thank you for putting a lot of effort in developing such an amazing tool.

Is your feature request related to a problem? Please describe. GASearchCV, unlike GridSearchCV, only accepts one scoring metric. Obviously, the algorithm can only use one metric to decide which models will carry over to the next generation. However, I think it would be useful to view different scoring metrics for the best models (e.g. R², MAE, RMSE), which intrinsically may provide a slightly different idea of model performance to the user. Of course we would still be able to decide which metric should be used to select the best models within each generation.

Describe the solution you'd expect I think the implementation of multiple scoring metrics in GASearchCV could be similar to the one implemented in GridSearchCV regarding this specific matter. I show below some examples of this implementation in GridSearchCV:

import numpy as np
from sklearn.svm import SVR
from sklearn.model_selection import GridSearchCV

#generate input
X = np.random.normal(75, 10, (1000, 2))
y = np.random.normal(200, 20, 1000)
params = {"degree": [2, 3], "C": [10, 20, 50]}

#calculate both R2 and MAE for each tested model, but model refit is performed based on the combination of hyperparameters with the best R2
grid = GridSearchCV(SVR(), param_grid=params, scoring=["neg_mean_absolute_error",  "r2"], refit="r2")

#another way of doing the above, but this time using aliases for the scorers
grid = GridSearchCV(SVR(), param_grid=params, scoring={"MAE": "neg_mean_absolute_error",  "R2": "r2"], refit="R2")

#perform grid search
grid.fit(X, y)

If you call grid.cv_results_ in this example, you will see the output dict will have a mean_test_MAE and mean_test_R2 keys (in the case of the second example).

System information Windows 10 Sklearn-genetic-opt version: 0.6.1 Scikit-learn version: 0.24.2 Python version: Python 3.7

Describe the bug When using the GASearchCV class with MLPClassifier as the estimator, I get the error in the title. In my param_grid, I simply have it set to Categorical([True, False]), but it doesn't seem to play well. Wondering what could be causing it?

To Reproduce Could recreate it by creating a binary classification dataset from sklearn, then implementing this:

    curr_params = {"shuffle": Categorical([True, False])}

    evolved_estimator = GASearchCV(estimator=MLPClassifier(),
                                   cv=StratifiedKFold(n_splits=2, shuffle=True, random_state=42),
                                   scoring='balanced_accuracy',
                                   population_size=30,
                                   generations=30,
                                   tournament_size=3,
                                   elitism=True,
                                   crossover_probability=0.8,
                                   mutation_probability=0.1,
                                   param_grid=curr_params,
                                   criteria='max',
                                   algorithm='eaMuPlusLambda',
                                   n_jobs=1,
                                   verbose=True,
                                   keep_top_k=1)

Expected behavior Seems to only be an issue with MLPClassifier so far, but should set the parameter shuffle to True or False.

Screenshots

Additional context

System information OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10.0.19044 Sklearn-genetic-opt version: 0.8.0 Scikit-learn version: 1.0.1 Python version: 3.8.11

Describe the bug A clear and concise description of what the bug is.

Firstly, great work!!

When using :

clf_RF    = RandomForestClassifier(random_state=0)

evolved_estimator = GAFeatureSelectionCV(
    estimator   = clf_RF,
    cv          = 5,
    population_size=20, 
    generations =40,
    crossover_probability=0.8,
    mutation_probability = 0.075,
    n_jobs      = -1,
    scoring     = "accuracy",
    max_features = 300
    )

# Train and select the features
evolved_estimator.fit(X, y)

the Output looks like this;

gen	nevals	fitness	fitness_std	fitness_max	fitness_min
0  	20    	-10000 	0          	-10000     	-10000     
1  	32    	-10000 	0          	-10000     	-10000     
2  	33    	-10000 	0          	-10000     	-10000     
3  	37    	-10000 	0          	-10000     	-10000     
4  	36    	-10000 	0          	-10000     	-10000     
5  	37    	-10000 	0          	-10000     	-10000     
6  	36    	-10000 	0          	-10000     	-10000     
7  	36    	-10000 	0          	-10000     	-10000     
8  	36    	-10000 	0          	-10000     	-10000     
9  	33    	-10000 	0          	-10000     	-10000     
10 	33    	-10000 	0          	-10000     	-10000     
11 	34    	-10000 	0          	-10000     	-10000     
12 	34    	-10000 	0          	-10000     	-10000     
13 	36    	-10000 	0          	-10000     	-10000     
14 	33    	-10000 	0          	-10000     	-10000     
15 	34    	-10000 	0          	-10000     	-10000     
16 	33    	-10000 	0          	-10000     	-10000     
17 	37    	-10000 	0          	-10000     	-10000     
18 	35    	-10000 	0          	-10000     	-10000     
19 	37    	-10000 	0          	-10000     	-10000     
20 	34    	-10000 	0          	-10000     	-10000     
21 	35    	-10000 	0          	-10000     	-10000     
22 	35    	-10000 	0          	-10000     	-10000 

This doesn't happen when removing the max_feature() parameter.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Add this code '....'
3. Run with this command '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

Main Features: GASearchCV: Principal class of the package, holds the evolutionary cross-validation optimization routine. Algorithms: Set of different evolutionary algorithms to use as an optimization procedure. Callbacks: Custom evaluation strategies to generate early stopping rules, logging (into TensorBoard, .pkl files, etc) or your custom logic. Plots: Generate pre-defined plots to understand the optimization process. MLflow: Build-in integration with mlflow to log all the hyperparameters, cv-scores and the fitted models

Is your feature request related to a problem? Please describe. Nope.

• A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] None.

• What is the use case for this feature? When a user wants to stop the optimization process manually (without using a callback), they could press CTRL + C to stop. The best evolved_estimator at the time of pressing CTRL + C will be returned and optimization will stop, allowing the rest of the script to continue.

Describe the solution you'd expect See above

• A clear and concise description of what you want to happen. TPOT is a good reference for this. The user presses CTRL + C after at least 1 pipeline has been fitted, and the best pipeline found until that point is used. The rest of the script can continue after that, like the evolved_estimator.predict() function.

• Describe the workflow you want to enable See above.

Additional context Love the tool! Would be cool to see this implemented :D

I used the default mlruns file store and I created a test to ensure the mlruns folder is removed. I tried to cover all the tests you requested. If there's anything I should change, just give me a shout.

I'm trying to get a docker container running with the mlflow server and a backend. I do think it's going to take a bit of time to figure out though because I'm not too clued up with docker. So hopefully these tests are ok for now.

The idea is to take in predictions from an arbitrary number of models, and find optimal weights that maximize the accuracy of the ensembled model.

Here's the estimator that I wrote:

from typing import List, Optional
import numpy as np
from sklearn.base import BaseEstimator, RegressorMixin
from sklearn.utils import check_X_y, check_array
from sklearn.utils.estimator_checks import check_estimator, check_is_fitted
from sklearn.metrics import mean_absolute_error


class WeightedAverageEnsemble(BaseEstimator, RegressorMixin):
    """
    
    >>> wae = WeightedAverageEnsemble()
    >>> X = np.random.rand(20, 5)
    >>> y = np.random.rand(20, 1)
    >>> wae.fit(X, y)
    >>> wae.predict(X)
    
    >>> wae = WeightedAverageEnsemble(weights=[0.25, 0.75])
    >>> X = np.random.rand(20, 2)
    >>> y = np.random.rand(20, 1)
    >>> wae.fit(X, y)
    >>> wae.predict(X)

    Parameters
    ----------
    BaseEstimator : _type_
        _description_
    RegressorMixin : _type_
        _description_
    """

    def __init__(self, weights: Optional[List[float]] = None):
        if weights is not None:
            assert np.isclose(sum(weights), 1.0)
        self.weights = weights

    def fit(self, X, y):
        # TODO: deal with sparse inputs (i.e. mask `W` and convert to sparse)
        X, y = check_X_y(X, y, accept_sparse=False)
        self.is_fitted_ = True
        self.n_features_in_ = X.shape[1]
        if self.weights is None:
            self._mod_weights = np.ones(self.n_features_in_) / self.n_features_in_
            # equivalent to:
            # w = np.ones(self.n_features_in_).reshape(1, -1)
            # w = sklearn.preprocessing.normalize(w, norm="l1", axis=1)
        else:
            self._mod_weights = self.weights
        return self

    def predict(self, X):
        # TODO: deal with sparse inputs (i.e. mask `W` and convert to sparse)
        X = check_array(X, accept_sparse=False)
        check_is_fitted(self, "is_fitted_")
        W = np.tile(self._mod_weights, (X.shape[0], 1))
        y = np.einsum("ij, ij->i", W, X)
        # should be equivalent to: y = np.sum(W * X)
        # loop with np.dot might also be fast due to BLAS compatibility
        # https://stackoverflow.com/a/26168677/13697228
        # https://stackoverflow.com/a/39657770/13697228
        return y

    def score(self, X, y, **kwargs):
        y_pred = self.predict(X)
        return mean_absolute_error(y, y_pred, **kwargs)


check_estimator(WeightedAverageEnsemble())

Related: https://machinelearningmastery.com/weighted-average-ensemble-with-python

How would you suggest optimizing weights since it's a vector that can change in size based on the size of the input data?

Is your feature request related to a problem? Please describe. I ran GASearchCV with a callback that stopped the optimization if the fitness was no greater than at least one value of fitness from the last 5 generations.

callback = ConsecutiveStopping(generations=5, metric='fitness')

Checking the log information while the algorithm was running, I have noticed that the reported fitness (-12.7893) was the same for more than 5 consecutive generations (please see the attached image). Under these circumstances, I would have expected the algorithm to have stopped much earlier (in generation 8).

I assume the algorithm did not stop because the logbook only shows 4 decimal places. However, given that fitness improved very little after generation 8, I think in some situations the user could have the option to provide a threshold value to ConsecutiveStopping, which would make the algorithm to stop after N consecutive generations if the improvement in fitness (or any other metric) was no greater than a specific threshold (e.g. 0.0001). This could make the algorithm to finish much faster in some occasions.

Describe the solution you'd expect I have made a custom callback (which hopefully is correct) to achieve what I want (the documentation was quite helpful). Please feel free to make any comments regarding my code:

from sklearn_genetic.callbacks.base import BaseCallback

class ConsecutiveStoppingThreshold(BaseCallback):
    def __init__(self, threshold, N, metric='fitness'):
        self.threshold = threshold
        self.N = N
        self.metric = metric
        
    def on_step(self, record, logbook, estimator=None):
        #not enough data points
        if len(logbook) <= self.N:
            return False
        
        #get the last N metrics
        stats = logbook.select(self.metric)[-self.N :]
        
        #find the difference between max and min fitness in the last metrics
        diff = max(stats) - min(stats)
        
        if self.threshold > diff:
            return True
        return False

I have tested this code and it appears to work fine. In my perspective, such type of callback is very useful and, therefore, I think it should be more easily accessible to users. In my opinion, you could do one of the following:

1. Show an explicit example, in the section "Custom callbacks" in the package's homepage, where you demonstrate how to achieve the above.
2. Or have a threshold argument in ConsecutiveStopping where the user can provide a float to determine how much improvement is allowed after N consecutive generations.

I updated the understand_cv documents grammar and spelling a bit. This is for #43 and I'll slowly go over all the documentation because I want to better understand the package and work on the inner workings eventually.

Is your feature request related to a problem? Please describe. This feature will make the package extend it's functionalities to include feature selection using evolutionary algorithms. Currently, only hyperparameters tuning is being done.

Describe the solution you'd expect

Implement the class GAFeaturesSelectionCV inside sklearn_genetic.genetic_search with the following functionalities:

• This function should take the same parameters as GASearchCV except for param_grid, the estimator should have it's own defined parameters.
• Perform cross-validation over different set of features that are selected using evolutionay algorithms. The same sklean_genetic.algorithms options must be available as optimization routine.
• The class should be able to work with the existing features of the package, such as Callbacks, plot fitness evolution.
• All the documentation must be updated, indicating which functionallity of the package is compatible only with GASearchCV (e.g most likely plot_search_space won't be compatible with feature selection).
• It must accepts a GASearchCV instance as the estimator.
• There must be an attribute called best_features_ that has the final selected features by the model.

Additional context The evolutionary algorithm can be defined by assigning a gen to each parameter, if the gen is 1, it means the parameters is selected, 0 otherwise.

Note: I'll be working on this feature, but as always, new ideas and contributions to this is welcome

It seems to me that the best approach to optimizing an estimator would be to run both feature selection AND hyperparameter optimization simultaneously within the same evolution process. It would be complex but probably yield better results instead of using one after the other.

Is this something I can do within the current framework, or does this require new code?

Also, do you think it is even a good idea in the first place?

Is your feature request related to a problem? Please describe.

• A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
• What is the use case for this feature?

Describe the solution you'd expect

• A clear and concise description of what you want to happen.
• Describe the workflow you want to enable

Additional context Add any other context or screenshots about the feature request here.

Hello, as part of our studies, me and some friends have to contribute to an opensource project related to optimization. Your project seems particularly interesting.

Do you need help on a particular feature? On which subject do you advise us to work?

Have a nice day, Pierre C.

Is your feature request related to a problem? Please describe. May I ask if there are plans to release a conda package in the near future?

I want to use this package within a project whose virtual environment is created with conda and all installed packages are also from conda/conda-forge. I have pip installed in the environment and tried to install sklearn-genetic-opt via pip as stated in the docs (pip install sklearn-genetic-opt). pip identified the dependencies and installed them (deap, numpy, etc.). The problem though is that it doesn't integrate well with the environment. For instance, I have pandas 1.5.0 installed in the conda environment, but when I open a Python session and run import sklearn_genetic, the interpreter returns me an error claiming that pandas is not installed.

Describe the solution you'd expect The package would be easier to use if it were possible to install it within conda.

Additional context Everything I reported refers to a Windows 10 21H2 machine.

System information OS Platform and Distribution: Windows 11 Home Sklearn-genetic-opt version: 0.9.0 deap version: 1.3.3 Scikit-learn version: 1.1.2 Python version: 3.8.13

Describe the bug I have fitted an instance of GAFeatureSelectionCV using LGBMClassifier

clf_dim = LGBMClassifier()
gen_opt = GAFeatureSelectionCV(
                               clf_dim, cv=5, scoring='avg_prec', refit=True, 
                               generations=20, population_size=50, tournament_size=3,
                               mutation_probability=0.8, crossover_probability=0.2, elitism=True, keep_top_k=1,
                               n_jobs=1, verbose=True, 
                              )

and got the expected results in the various output attributes such as .best_estimator_ and n_features_in_

However, unlike the example provided in the documentation, I am not attempting to use the selected features and the estimator directly to predict results on test data.

Instead, I am trying to follow the traditional scikit-learn approach of incorporating this estimator to select features as step 'dim' in the following pipeline, before passing them on to another classifier at the end of the pipeline

This requires that the 'transformer' based on GAFeatureSelectionCV supports a transform() method, which it does. However, when I try to use the transform method of the fitted estimator standalone, as in:

gen_opt.transform(X_t)

I get an error suggesting that

'LGBMClassifier' object has no attribute 'transform'

I went on to define a pipeline with the estimator as below:

pipe_dim_full = Pipeline(
    steps=[
        ('enc', encode), 
        ('dim', gen_opt), 
        ('clf', clf), 
    ], 
)

and upon trying to fit it, I get a somewhat contradictory error:

TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'GAFeatureSelectionCV(cv=StratifiedKFold(n_splits=5, random_state=0, shuffle=True), estimator=LGBMClassifier(n_jobs=1, random_state=0, verbose=-1), generations=20, n_jobs=18, return_train_score=True, scoring=make_scorer(average_precision_score, needs_proba=True, pos_label=1))' (type <class 'sklearn_genetic.genetic_search.GAFeatureSelectionCV'>) doesn't

As it stands, GAFeatureSelectionCV can't be used in a pipeline without the transform() method being fixed, which is unfortunate as I really like it and was looking forward to using GA across my pipeline.

To Reproduce Steps to reproduce the behavior: As described above. Please reach out if you need more detail.

Expected behavior The transform method should product a matrix with n_features_in_ columns of the input matrix

Additional context There is another module based on deap that successfully offers feature selection by genetic algorithm. Here is a link for reference https://sklearn-genetic.readthedocs.io/en/latest/api.html

Thanks for such a cool package.

I'm using GASearchCV to hypertune an xgboost model. However, it is failing if I use early stopping an fit() Can early stopping (and the additional xgbosst fitting params) be used with GASearchCV().fit()?

Thanks, Hayden