Miscellaneous fixes and updates.

Miscellaneous fixes and updates.

We're happy to announce that carefree-learn released v0.3.x, which made it much more light-weight!

Miscellaneous fixes and updates.

Miscellaneous fixes and updates.

Miscellaneous fixes and updates.

Miscellaneous fixes and updates.

Release Notes

We're happy to announce that carefree-learn released v0.2.x, which made it capable of solving not only tabular tasks, but also other general deep learning tasks!

Introduction

Deep Learning with PyTorch made easy 🚀!

Like many similar projects, carefree-learn can be treated as a high-level library to help with training neural networks in PyTorch. However, carefree-learn does more than that.

• carefree-learn is highly customizable for developers. We have already wrapped (almost) every single functionality / process into a single module (a Python class), and they can be replaced or enhanced either directly from source codes or from local codes with the help of some pre-defined functions provided by carefree-learn (see Register Mechanism).
• carefree-learn supports easy-to-use saving and loading. By default, everything will be wrapped into a .zip file, and onnx format is natively supported!
• carefree-learn supports Distributed Training.

Apart from these, carefree-learn also has quite a few specific advantages in each area:

Machine Learning 📈

• carefree-learn provides an end-to-end pipeline for tabular tasks, including AUTOMATICALLY deal with (this part is mainly handled by carefree-data, though):
  • Detection of redundant feature columns which can be excluded (all SAME, all DIFFERENT, etc).
  • Detection of feature columns types (whether a feature column is string column / numerical column / categorical column).
  • Imputation of missing values.
  • Encoding of string columns and categorical columns (Embedding or One Hot Encoding).
  • Pre-processing of numerical columns (Normalize, Min Max, etc.).
  • And much more...
• carefree-learn can help you deal with almost ANY kind of tabular datasets, no matter how dirty and messy it is. It can be either trained directly with some numpy arrays, or trained indirectly with some files locate on your machine. This makes carefree-learn stand out from similar projects.

When we say ANY, it means that carefree-learn can even train on one single sample.

import cflearn

toy = cflearn.ml.make_toy_model()
data = toy.data.cf_data.converted
print(f"x={data.x}, y={data.y}")  # x=[[0.]], y=[[1.]]

This is especially useful when we need to do unittests or to verify whether our custom modules (e.g. custom pre-processes) are correctly integrated into carefree-learn.

import cflearn
import numpy as np

# here we implement a custom processor
@cflearn.register_processor("plus_one")
class PlusOne(cflearn.Processor):
    @property
    def input_dim(self) -> int:
        return 1

    @property
    def output_dim(self) -> int:
        return 1

    def fit(self, columns: np.ndarray) -> cflearn.Processor:
        return self

    def _process(self, columns: np.ndarray) -> np.ndarray:
        return columns + 1

    def _recover(self, processed_columns: np.ndarray) -> np.ndarray:
        return processed_columns - 1

# we need to specify that we use the custom process method to process our labels
toy = cflearn.ml.make_toy_model(cf_data_config={"label_process_method": "plus_one"})
data = toy.data.cf_data
y = data.converted.y
processed_y = data.processed.y
print(f"y={y}, new_y={processed_y}")  # y=[[1.]], new_y=[[2.]]

There is one more thing we'd like to mention: carefree-learn is Pandas-free. The reasons why we excluded Pandas are listed in carefree-data.

Computer Vision 🖼️

• carefree-learn also provides an end-to-end pipeline for computer vision tasks, and:

  • Supports native torchvision datasets.

    data = cflearn.cv.MNISTData(transform="to_tensor")
    

    Currently only mnist is supported, but will add more in the future (if needed) !

  • Focuses on the ImageFolderDataset for customization, which:

    • Automatically splits the dataset into train & valid.
    • Supports generating labels in parallel, which is very useful when calculating labels is time consuming.

    See IFD introduction for more details.

• carefree-learn supports various kinds of Callbacks, which can be used for saving intermediate visualizations / results.
  • For instance, carefree-learn implements an ArtifactCallback, which can dump artifacts to disk elaborately during training.

Examples

import cflearn
import numpy as np
x = np.random.random([1000, 10])
y = np.random.random([1000, 1])
m = cflearn.api.fit_ml(x, y, carefree=True)

import cflearn
data = cflearn.cv.MNISTData(batch_size=16, transform="to_tensor")
m = cflearn.api.resnet18_gray(10).fit(data)

Please refer to Quick Start and Developer Guides for detailed information.

Migration Guide

From 0.1.x to v0.2.x, the design principle of carefree-learn changed in two aspects:

• The DataLayer in v0.1.x has changed to the more general DataModule in v0.2.x.
• The Model in v0.1.x, which is constructed by pipes, has changed to general Model.

These changes are made because we want to make carefree-learn compatible with general deep learning tasks (e.g. computer vision tasks).

Data Module

Internally, the Pipeline will train & predict on DataModule in v0.2.x, but carefree-learn also provided useful APIs to make user experiences as identical to v0.1.x as possible:

Train

import cflearn
import numpy as np
x = np.random.random([1000, 10])
y = np.random.random([1000, 1])
m = cflearn.make().fit(x, y)

import cflearn
import numpy as np
x = np.random.random([1000, 10])
y = np.random.random([1000, 1])
m = cflearn.api.fit_ml(x, y, carefree=True)

Predict

predictions = m.predict(x)

predictions = m.predict(cflearn.MLInferenceData(x))

Evaluate

cflearn.evaluate(x, y, metrics=["mae", "mse"], pipelines=m)

cflearn.ml.evaluate(cflearn.MLInferenceData(x, y), metrics=["mae", "mse"], pipelines=m)

Model

It's not very straight forward to migrate models from v0.1.x to v0.2.x, so if you require such migration, feel free to submit an issue and we will analyze the problems case by case!

Release Notes

carefree-learn 0.1.16 improved overall performances.

Optimizer

MADGRAD (4466c9f) & Ranger (acdeec4) are now introduced.

Reference: Best-Deep-Learning-Optimizers.

Misc

• Fixed ddp when np.ndarray is provided (969a6c8).
• Fixed RNN when bidirectional is True (be974df) (6ef49f7).

• Optimized Transformer (00bd2c4) (dc6abc4) (aec1846).
• Re-designed the reduce part (b99d4a2).
• Summary will now be written to disk (d5435e9).
• batch_indices will be injected to forward_results (7a40dcc).

Release Notes

carefree-learn 0.1.15 improved overall performances.

DDP

Since PyTorch is introducing ZeRO optimizer, we decided to remove deepspeed dependency and use native DDP from PyTorch.

results = cflearn.ddp(tr_file, world_size=2)
predictions = results.m.predict(te_file)

JitLSTM

Since native RNNs of PyTorch do not support dropouts on w_ih and w_hh, we followed the official implementation of jit version LSTM and implemented these dropouts.

m = cflearn.make(
    "rnn",
    model_config={
        "pipe_configs": {
            "rnn": {
                "extractor": {
                    "cell": "JitLSTM"
                }
            }
        }
    }
)

Misc

• Fixed NNB when std is 0 (177363e).
• Fixed summary in some edge cases (945ca15, f95f667, 2768153).
• Introduced ONNXWrapper for more general ONNX exports (226de5b).

• Optimized Transformer (b09916b).
• Upgraded PyTorch dependency (a596031).
• Supported reuse_extractor in PipeInfo (149aa49).
• Implemented HighwayBlock (3dad99e, 436ebab) and Introduced *FCNNConfig (e0670f7).
• Implemented Initializer.orthogonal (1019114) and Optimized initializations of RNN (2193706).

Release Notes

carefree-learn 0.1.14 improved overall performances.

Summary

For non-distributed trainings, carefree-learn will print out model summaries now by default (inspired by torchsummary):

========================================================================================================================
Layer (type)                             Input Shape                             Output Shape    Trainable Param #
------------------------------------------------------------------------------------------------------------------------
RNN                                       [-1, 5, 1]                                [-1, 256]              198,912
    GRU                                   [-1, 5, 1]           [[-1, 5, 256], [-1, 128, 256]]              198,912
FCNNHead                                   [-1, 256]                                  [-1, 1]              395,777
  MLP                                      [-1, 256]                                  [-1, 1]              395,777
      Mapping-0                            [-1, 256]                                [-1, 512]              132,096
        Linear                             [-1, 256]                                [-1, 512]              131,072
        BN                                 [-1, 512]                                [-1, 512]                1,024
        ReLU                               [-1, 512]                                [-1, 512]                    0
        Dropout                            [-1, 512]                                [-1, 512]                    0
      Mapping-1                            [-1, 512]                                [-1, 512]              263,168
        Linear                             [-1, 512]                                [-1, 512]              262,144
        BN                                 [-1, 512]                                [-1, 512]                1,024
        ReLU                               [-1, 512]                                [-1, 512]                    0
        Dropout                            [-1, 512]                                [-1, 512]                    0
      Linear                               [-1, 512]                                  [-1, 1]                  513
========================================================================================================================
Total params: 594,689
Trainable params: 594,689
Non-trainable params: 0
------------------------------------------------------------------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.30
Params size (MB): 2.27
Estimated Total Size (MB): 2.57
------------------------------------------------------------------------------------------------------------------------

Zoo.search

Now carefree-learn supports empirical HPO via Zoo.search (2c19505), which can achieve good performances without searching in a large search space.

Misc

• Upgraded numpy dependency in pyproject.toml to try to avoid building issues.
• Optimized default Zoo settings (9e051d5).
• Fixed & Optimized Trainer when max_epoch is specified (ea44d88).

Weekly patch with miscellaneous fixes and updates.

Weekly patch with miscellaneous fixes and updates.

carefree-learn 0.1.11 is mainly a patch release which supported more customizations. However, these features are still at early stage and are likely to be changed in the future. If you want to customize carefree-learn, it is still highly recommended to clone this repo and install it with edit mode (pip install -e .).

Release Notes

carefree-learn 0.1.10 improved overall performances and deepspeed accessibility.

Versioning

carefree-learn now supports checking its version via __version__:

import cflearn

cflearn.__version__  # '0.1.10'

Distributed Training

carefree-learn now provides out-of-the-box API for distributed training with deepspeed:

import cflearn
import numpy as np

x = np.random.random([1000000, 10])
y = np.random.random([1000000, 1])
m = cflearn.deepspeed(x, y, cuda="0,1,2,3").m

⚠️⚠️⚠️ However it is not recommended to use this API unless you have to (e.g. when training some very large models). ⚠️⚠️⚠️

Misc

• Supported use_final_bn in FCNNHead (#75).
• Ensured that models are always in eval mode in inference (#67).
• Supported specifying resource_config of Parallel in Experiment (#68).
• Implemented profile_forward for Pipeline.

• Fixed other bugs.
• Accelerated DNDF with Function (#69).
• Patience of TrainMonitor will now depend on dataset size (#43).
• Checkpoints will be logged earlier now when using warmup (#74).

Release Notes

carefree-learn 0.1.9 improved overall performances and accessibilities.

ModelConfig

carefree-learn now introduces ModelConfig to manage configurations more easily.

Modify extractor_config, head_config, etc

head_config = {...}
cflearn.make(
    model_config={
        "pipe_configs": {
            "fcnn": {"head": head_config},
        },
    },
)

head_config = {...}
cflearn.ModelConfig("fcnn").switch().head_config = head_config

Switch to a preset config

# Not accessible, must register a new model 
#  with the corresponding config:
cflearn.register_model(
    "pruned_fcnn",
    pipes=[
        cflearn.PipeInfo(
            "fcnn",
            head_config="pruned",
        )
    ],
)
cflearn.make("pruned_fcnn")

cflearn.ModelConfig("fcnn").switch().replace(head_config="pruned")
cflearn.make("fcnn")

Misc

• Enhanced LossBase (#66).
• Introduced callbacks to Trainer (#65).
• Enhanced Auto and support specifying extra_config with json file path (752f419).

• Fixed other bugs.
• Optimized Transformer (adce2f9).
• Optimized performance of TrainMonitor (91dfc43).
• Optimized performance of Auto (47caa48, 9dfa204, 274b28d and #61, #63, #64).

Release Notes

carefree-learn 0.1.8 mainly registered all PyTorch schedulers and enhanced mlflow integration.

Backward Compatible Breaking

carefree-learn now keeps a copy of the orignal user defined configs (#48), which changes the saved config file:

{
    "data_config": {
        "label_name": "Survived"
    },
    "cuda": 0,
    "model": "tree_dnn"
    // the `binary_config` was injected into `config.json`
    "binary_config": {
        "binary_metric": "acc",
        "binary_threshold": 0.49170631170272827
    }
}

{
    "config": {
        "data_config": {
            "label_name": "Survived"
        },
        "cuda": 0,
        "model": "tree_dnn"
    },
    "increment_config": {},
    "binary_config": {
        "binary_metric": "acc",
        "binary_threshold": 0.49170631170272827
    }
}

New Schedulers

carefree-learn newly supports the following schedulers based on PyTorch schedulers:

• step: the StepLR with lr_floor supported.
• exponential: the ExponentialLR with lr_floor supported.
• cyclic: the CyclicLR.
• cosine: the CosineAnnealingLR.
• cosine_restarts: the CosineAnnealingWarmRestarts.

These schedulers could be utilized easily with scheduler=... specified in any high-level API in carefree-learn, e.g.:

m = cflearn.make(scheduler="cyclic").fit(x, y)

Better mlflow Integration

In order to utilize mlflow better, carefree-learn now handles some better practices for you under the hood, e.g.:

• Makes the initialization of mlflow multi-thread safe in distributed training.
• Automatically handles the run_name in distributed training.
• Automatically handles the parameters for log_params.
• Updates the artifacts in periodically.

The (brief) documentation for mlflow Integration could be found here.

Release Notes

carefree-learn 0.1.7 integrated mlflow and cleaned up Experiment API, which completes the machine learning lifecycle.

• v0.1.7.1: Hotfixed a critical bug which will load the worst checkpoint saved.

mlflow

mlflow can help us visualizing, reproducing, and serving our models. In carefree-learn, we can quickly play with mlflow by specifying mlflow_config to an empty dict:

import cflearn
import numpy as np

x = np.random.random([1000, 10])
y = np.random.random([1000, 1])
m = cflearn.make(mlflow_config={}).fit(x, y)

After which, we can execute mlflow ui in the current working directory to inspect the tracking results (e.g. loss curve, metric curve, etc.).

We're planning to add documentation for the mlflow integration and it should be available at v0.1.8.

Experiment

Experiment API was embarrassingly user unfriendly before, but has been cleaned up and is ready to use since v0.1.7. Please refer to the documentation for more details.

Misc

• Integrated DeepSpeed for distributed training on one single model (experimental).
• Enhanced Protocol for downstream usages (e.g. Quantitative Trading, Computer Vision, etc.) (experimental).

• Fixed other bugs.
• Optimized TrainMonitor (#39)
• Optimized some default settings.

Release Notes

carefree-learn 0.1.6 is mainly a hot-fix version for 0.1.5.

Misc

• Simplified Pipeline.load (0033bda).
• Generalized input_sample (828e985).
• Implemented register_aggregator.

Release Notes

⚠️⚠️⚠️ This release is broken and could hardly perform customizations. We'll release v0.1.6 ASAP ⚠️⚠️⚠️

draw

We can visualize every model built by carefree-learn with draw API now (see here).

Aggregator

We can now customize how to aggregate the results from each head now.

We plan to add documentation for Aggregator in v0.1.6.

Protocol

carefree-learn now supports Protocol. With Protocols it is possible to port other frameworks' models (e.g. models from scikit-learn) to carefree-learn, as well as utilize carefree-learn on other forms of input data.

We plan to add documentation for Protocols in v0.1.6.

Updated 2020.12.13: documentation for Protocol is delayed because it may lack of users. Please feel free to contact me if you are interested in this set of features!

Misc

• Implemented PrefetchLoader.

• Fixed QuantileFCNN.
• Fixed other bugs.
• Optimized some default settings.

Release Notes

carefree-learn 0.1.4 fixed some critical bugs in 0.1.3, as well as introduced some new features (supported evaluating multiple models, customizing losses, etc.).

Backward Compatible Breaking

carefree-learn now deals with list of pipelines instead of a single pipeline in most APIs (#27)

cflearn.save(m)        # -> cflearn^_^fcnn.zip
print(cflearn.load())  # -> {'fcnn': FCNN()}

cflearn.save(m)        # -> cflearn^_^fcnn^_^0000.zip
print(cflearn.load())  # -> {'fcnn': [FCNN()]}

Misc

• Supported customizing new losses.
• Enhanced cflearn.evaluate, it can now evaluate on multiple pipelines.
• Changed default parallel settings to non-parallel.
• Supported specify loss and loss_config in Elements.
• Optimized auto metric settings. It will now depend itself on loss.
• Implemented QuantileFCNN for quantile regression (experimental).

• Fixed Pipeline.load.
• Fixed the configuration stuffs.
• Fixed other bugs.
• Optimized some default settings.

Release Notes

carefree-learn 0.1.3 focuses on performances and accessability.

pipe

A new abstraction, pipe, was introduced to carefre-learn which significantly improved accessability for developers. Now we can introduce a new model to carefree-learn with only one line of code:

cflearn.register_model("wnd_full", pipes=[cflearn.PipeInfo("fcnn"), cflearn.PipeInfo("linear")])
m = cflearn.make("wnd_full")

Please refer to Build Your Own Models for detailed information.

Auto

carefree-learn now provides a high level AutoML API:

import cflearn
from cfdata.tabular import TabularDataset

x, y = TabularDataset.iris().xy
auto = cflearn.Auto("clf").fit(x, y)
predictions = auto.predict(x)

Production

carefree-learn now supports onnx export, and provides a high level API Pack to pack everything up for production:

import cflearn
from cfdata.tabular import TabularDataset

x, y = TabularDataset.iris().xy
m = cflearn.make().fit(x, y)
cflearn.Pack.pack(m, "onnx")

This piece of code will generate an onnx.zip in the working directory with following file structure:

|--- preprocessor
   |-- ...
|--- binary_config.json
|--- m.onnx
|--- output_names.json
|--- output_probabilities.txt

With onnx.zip we can make predictions (inference) on the fly:

predictor = cflearn.Pack.get_predictor("onnx")
predictions = predictor.predict(x)

Misc

• carefree-learn should be ~3x faster than before on small datasets thanks to optimizations on categorical encodings.
• DNDF in carefre-learn is highly optimized and should be ~3x faster than before.
• APIs have been re-designed and are much easier to use now.
• Much better documented than before (documentations).
• Implemented more models (Linear, TreeLinear, Wide and Deep, RNN, Transformer, etc.).
• Implemented more modules (CrossBlock, ConditionalBlocks, MonotonousMapping, MLP.funnel, etc.).

• Fixed some bugs.
• Optimized some default settings.

WARNING: Requires carefree-data == 0.1.5!

Miscellaneous fixes and updates.

Release Notes

Experiments

Experiments is much more powerful and much easier to use now:

Updated 2020.12.13: Experiment is more useful now in v0.1.7! Please refer to the documentation for more details.

import cflearn
import numpy as np

from cfdata.tabular import *

def main():
    x, y = TabularDataset.iris().xy
    experiments = cflearn.Experiments()
    experiments.add_task(x, y, model="fcnn")
    experiments.add_task(x, y, model="fcnn")
    experiments.add_task(x, y, model="tree_dnn")
    experiments.add_task(x, y, model="tree_dnn")
    results = experiments.run_tasks(num_jobs=2)
    # {'fcnn': [Task(fcnn_0), Task(fcnn_1)], 'tree_dnn': [Task(tree_dnn_0), Task(tree_dnn_1)]}
    print(results)
    ms = {k: list(map(cflearn.load_task, v)) for k, v in results.items()}
    # {'fcnn': [FCNN(), FCNN()], 'tree_dnn': [TreeDNN(), TreeDNN()]}
    print(ms)
    # experiments could be saved & loaded easily
    saving_folder = "__temp__"
    experiments.save(saving_folder)
    loaded = cflearn.Experiments.load(saving_folder)
    ms_loaded = {k: list(map(cflearn.load_task, v)) for k, v in loaded.tasks.items()}
    # {'fcnn': [FCNN(), FCNN()], 'tree_dnn': [TreeDNN(), TreeDNN()]}
    print(ms_loaded)
    assert np.allclose(ms["fcnn"][1].predict(x), ms_loaded["fcnn"][1].predict(x))

if __name__ == '__main__':
    main()

We can see that experiments.run_tasks returns a bunch of Tasks, which can be easily transfered to models through cflearn.load_task.

It is important to wrap the codes with main() on some platforms (e.g. Windows), because running codes in parallel will cause some issues if we don't do so. Here's an explaination.

Benchmark

Benchmark class is implemented for easier benchmark testing:

Updated 2020.12.13: Benchmark was moved to a separated repo (carefree-learn-benchmark).

import cflearn
import numpy as np

from cfdata.tabular import *

def main():
    x, y = TabularDataset.iris().xy
    benchmark = cflearn.Benchmark(
        "foo",
        TaskTypes.CLASSIFICATION,
        models=["fcnn", "tree_dnn"]
    )
    benchmarks = {
        "fcnn": {"default": {}, "sgd": {"optimizer": "sgd"}},
        "tree_dnn": {"default": {}, "adamw": {"optimizer": "adamw"}}
    }
    msg1 = benchmark.k_fold(3, x, y, num_jobs=2, benchmarks=benchmarks).comparer.log_statistics()
    """
    ~~~  [ info ] Results
    ===============================================================================================================================
    |        metrics         |                       acc                        |                       auc                        |
    --------------------------------------------------------------------------------------------------------------------------------
    |                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
    --------------------------------------------------------------------------------------------------------------------------------
    |    fcnn_foo_default    |    0.780000    | -- 0.032660 -- |    0.747340    |    0.914408    |    0.040008    |    0.874400    |
    --------------------------------------------------------------------------------------------------------------------------------
    |      fcnn_foo_sgd      |    0.113333    |    0.080554    |    0.032780    |    0.460903    |    0.061548    |    0.399355    |
    --------------------------------------------------------------------------------------------------------------------------------
    |   tree_dnn_foo_adamw   | -- 0.833333 -- |    0.077172    | -- 0.756161 -- | -- 0.944698 -- | -- 0.034248 -- | -- 0.910451 -- |
    --------------------------------------------------------------------------------------------------------------------------------
    |  tree_dnn_foo_default  |    0.706667    |    0.253684    |    0.452983    |    0.924830    |    0.060007    |    0.864824    |
    ================================================================================================================================
    """
    # save & load
    saving_folder = "__temp__"
    benchmark.save(saving_folder)
    loaded_benchmark, loaded_results = cflearn.Benchmark.load(saving_folder)
    msg2 = loaded_results.comparer.log_statistics()
    assert msg1 == msg2

if __name__ == '__main__':
    main()

Misc

• Integrated trains.
• Integrated Tracker from carefree-toolkit.
• Integrated native amp from PyTorch.
• Implemented FocalLoss.
• Implemented cflearn.zoo.

• Introduced CI.
• Fixed some bugs.
• Simplified some APIs.
• Optimized some default settings.

First release