Hi, I've been spending the last few days trying to make a loss_yolo layer for dlib, in particular the loss presented in the YOLOv3 paper.

I think I came up with a pretty straightforward implementation but, as of now, it still does not work.

I wondered if you could have a look. I am quite confident the loss implementation is correct, however I think I might be making some assumptions about the dlib API when the loss layer takes several inputs from the network.

I tried to make the loss similar to the loss_mmod in the way you set the options of the layer, etc.

So, my question is, does this way of coding the loss in dlib make sense for multiple outputs? Or is dlib doing something I don't expect?

There's also a simple example program that takes a path containing a training.xml file (like the one from the face or vehicle detection examples).

Thanks in advance :)

I am trying to define yolov3 using dlib's dnn module. I'm stuck with the darknet53 backbone, as I want it to output the outputs of the last three layers. So far i have this:

using namespace dlib;
                    
template <int outc, int kern, int stride, typename SUBNET> 
using conv_block = leaky_relu<affine<con<outc,kern,kern,stride,stride,SUBNET>>>;

template <int inc, typename SUBNET>
using resblock = add_prev1<conv_block<inc,3,1,conv_block<inc/2,1,1,tag1<SUBNET>>>>;

template<int nblocks, int outc, typename SUBNET>
using conv_resblock = repeat<nblocks, resblock<outc,
                      conv_block<outc, 3, 2, SUBNET>>>;

template<typename SUBNET>
using darknet53 = tag3<conv_resblock<4, 1024,
                  tag2<conv_resblock<8, 512,
                  tag1<conv_resblock<8, 256,
                  conv_resblock<2, 128,
                  conv_resblock<1, 64,
                  conv_block<32, 3, SUBNET
                  >>>>>>>>>;

Is it possible for darknet53 to output tag1, tag2 and tag3?

Enable and optimize support for POWER8 VSX SIMD instructions on PPC64LE Linux to dlib/simd.

$$$$ Financial bounties available. Any reasonable suggested value will be seriously considered.

I welcome contact / replies from developers in the dlib community who are interested to work on this project.

Hi, I would like to contribute a DCGAN example to dlib.

I have implemented a version of Pytorch DCGAN for C++.

However, I would need some guidance with some things I don't know how to do. I am wondering on how I should proceed. Should I attach my current code here (around 150 lines), or make a pull request, even if the code is not able to learn anything? Maybe @edubois can help out, since he stated that he managed to make it work on https://github.com/davisking/dlib/issues/1261

Thanks for your hard work on dlib.

This PR adds a C++ port of kissFFT as the default backend of dlib's fft routines. All the kiss code is inserted into the dlib namespace to avoid conflicts with user code that may or may not be using original (C) kissFFT. All MKL fft code is put into it's own separate header file. Now dlib/matrix_fft.h simply calls the right wrappers depending on a pre-processor macro (as before). I've removed the FFTW wrapper code as per @davisking's request as the fftw planner isn't thread safe. I've also removed the original fft backend code, again, as per @davisking's request.

Note that this PR isn't quite ready yet. Need more unit tests. Not quite sure why MKL wrappers only worked for matrix<complex<double>> before. There's no reason why it couldn't work for matrix<complex<float>> if using DFTI_SINGLE instead of DFTI_DOUBLE. (maybe user code had to do a cast?) So need more units tests for MKL wrappers.

Not quite happy with the number of copies at the moment. When passing const matrix_exp<EXP>& data as input, there are 2 copies: one to evaluate the matrix expression to matrix<typename EXP::type> and one to do either the in-place fft (copy is done in the implementation details of the fft functions) or out-of-place fft (copy is done explicitly in dlib/matrix_fft.h). For example, if you want to do the fft of std::vector<std::complex>, you have to use dlib::mat which returns a matrix expression then dlib::fft will evaluate that to dlib::matrix<std::complex<float>>. So there is a copy of std::vector to dlib::matrix that is unnecessary. Could add some function overloads to support std::vector. Don't know what's the best thing to do without dirtying the API. Could benefit from @davisking's advice.

I think if we've gone this far, we should support real FFTs so the requirement that input should be complex vanishes. Output will be complex though. Both KISS and MKL support this so there shouldn't be a lot of leg work. More unit tests.

Hi, recently I've been interested in self-supervised learning (SSL) methods in deep learning. I noticed that dlib has support for unsupervised loss layers by just leaving out the training_label_type typedef and the truth iterator argument to compute_loss_value_and_gradient().

However, most methods I've read about and tried are quite complicated (hard-negative mining, predictors, stop gradient, exponential moving average of the weights between both architectures) due to the need of breaking the symmetry between both branches to avoid collapse. But recently, I came across a simple method named Barlow Twins: Self-Supervised Learning via Redundancy Reduction. The idea is as follows:

1. forward two augmented versions of an image
2. compute the empirical cross-correlation matrix between both feature representations
3. make that matrix as close to the identity as possible.

That prevents collapse, since it's making each individual dimension of the feature vectors to focus on different tasks, and thus avoiding redundancies between dimensions (it needs a high dimension for it to work, since it relies on sparse representations).

So far, I've implemented:

input_rgb_image_pair

It takes a std::vector<std::pair<matrix<rgb_pixel>, matrix<rgb_pixel>>> and puts the first elements of the pairs in the first half of the batch and the second elements of the batch in the second half of the batch. This allows computing batch normalization on each half efficiently (done on the loss layer).

loss_barlow_twins

I tried to follow the official paper as close as possible, and made use of the awesome Matrix Calculus site for the gradients (having element wise operations, (off-)diagonal terms, and summations was a bit tedious to do manually 😅)

But… I am experiencing some difficulties:

• if I use a dnn_trainer I get a segfault (but if I train manually, like in the code, it works: at least the loss goes down)
• if I use a batch size larger than 64, I also get a segfault (but I have plenty of RAM/VRAM)

So maybe I misunderstood something about how to implement the input layer or the unsupervised loss layer… If you could have a look at some point… You can just run the example by giving a path to a folder containing the CIFAR-10 dataset.

It would be great to have an example of SSL on dlib.

Thanks in advance (all the code is inside the example program, I will make a proper PR if we manage to get this work, and you are interested, since the method is fairly new)

I've been playing a bit with the idea of having fused convolutions (convolution + batch_norm) in dlib. I think the first step would be to move all the operations that are done by the affine_ layer into the convolution, that is, update the bias of the convolution and re-scale the filters.

This PR adds some helper methods that allow doing this. The next step could be adding a new layer that can be constructed from an affine_ layer and it's a no-op, like the tag layers, or add a version of the affine layer that does nothing (just outputs its input, without copying or anything). How would you approach this?

Finally, here's an example that uses a visitor to update the convolutions that are below an affine layer. It can be build from by putting the file into the examples folder and loading the pretrained resnet 50 from the dnn_introduction3_ex.cpp. If we manage to make something interesting out of it, maybe it would be interesting to have this visitor, too.

#include "resnet.h"

#include <dlib/dnn.h>
#include <dlib/image_io.h>

using namespace std;
using namespace dlib;

class visitor_fuse_convolutions
{
    public:
    template <typename T> void fuse_convolutions(T&) const
    {
        // disable other layer types
    }

    // handle the standard case (convolutional layer followed by affine;
    template <long nf, long nr, long nc, int sy, int sx, int py, int px, typename U, typename E>
    void fuse_convolutions(add_layer<affine_, add_layer<con_<nf, nr, nc, sy, sx, py, px>, U>, E>& l)
    {
        // get the parameters from the affine layer as alias_tensor_instance
        auto gamma = l.layer_details().get_gamma();
        auto beta = l.layer_details().get_beta();

        // get the convolution below the affine layer and its paramaters
        auto& conv = l.subnet().layer_details();
        const long num_filters_out = conv.num_filters();
        const long num_rows = conv.nr();
        const long num_cols = conv.nc();
        tensor& params = conv.get_layer_params();
        // guess the number of input filters
        long num_filters_in;
        if (conv.bias_is_disabled())
            num_filters_in = params.size() / num_filters_out / num_rows / num_cols;
        else
            num_filters_in = (params.size() - num_filters_out) / num_filters_out / num_rows / num_cols;

        // set the new number of parameters for this convolution
        const size_t num_params = num_filters_in * num_filters_out * num_rows * num_cols + num_filters_out;
        alias_tensor filters(num_filters_out, num_filters_in, num_rows, num_cols);
        alias_tensor biases(1, num_filters_out);
        if (conv.bias_is_disabled())
        {
            conv.enable_bias();
            resizable_tensor new_params = params;
            new_params.set_size(num_params);
            biases(new_params, filters.size()) = 0;
            params = new_params;
        }

        // update the biases
        auto b = biases(params, filters.size());
        b+= mat(beta);

        // rescale the filters
        DLIB_CASSERT(filters.num_samples() == gamma.k());
        auto t = filters(params, 0);
        float* f = t.host();
        const float* g = gamma.host();
        for (long n = 0; n < filters.num_samples(); ++n)
        {
            for (long k = 0; k < filters.k(); ++k)
            {
                for (long r = 0; r < filters.nr(); ++r)
                {
                    for (long c = 0; c < filters.nc(); ++c)
                    {
                        f[tensor_index(t, n, k, r, c)] *= g[n];
                    }
                }
            }
        }

        // reset the affine layer
        gamma = 1;
        beta = 0;
    }

    template <typename input_layer_type>
    void operator()(size_t , input_layer_type& l) const
    {
        // ignore other layers
    }

    template <typename T, typename U, typename E>
    void operator()(size_t , add_layer<T, U, E>& l)
    {
        fuse_convolutions(l);
    }
};

int main(const int argc, const char** argv)
try
{
    resnet::infer_50 net1, net2;
    std::vector<std::string> labels;
    deserialize("resnet50_1000_imagenet_classifier.dnn") >> net1 >> labels;
    net2 = net1;
    matrix<rgb_pixel> image;
    load_image(image, "elephant.jpg");

    const auto& label1 = labels[net1(image)];
    const auto& out1 = net1.subnet().get_output();
    resizable_tensor probs(out1);
    tt::softmax(probs, out1);
    cout << "pred1: " << label1 << " (" << max(mat(probs)) << ")" << endl;


    // fuse the convolutions in the network
    dlib::visit_layers_backwards(net2, visitor_fuse_convolutions());
    const auto& label2 = labels[net2(image)];
    const auto& out2 = net2.subnet().get_output();
    tt::softmax(probs, out2);
    cout << "pred2: " << label2 << " (" << max(mat(probs)) << ")" << endl;

    cout << "max abs difference: " << max(abs(mat(out1) - mat(out2))) << endl;
    DLIB_CASSERT(max(abs(mat(out1) - mat(out2))) < 1e-2);
}
catch (const exception& e)
{
    cout << e.what() << endl;
    return EXIT_FAILURE;
}

output with this image (elephant.jpg):

pred1: African_elephant (0.962677)
pred2: African_elephant (0.962623)
max abs difference: 0.00436211

UPDATE: make visitor more generic and show a results with a real image

I'm interested in using dlib for semantic segmentation. I think the only necessary features would be:

• Loss function - it doesn't look like loss_multiclass_log would work for this
• "Unpooling" upsampling layer - there are a few ways to do this

Is this something you're interested in supporting? I'm planning to add this functionality regardless, just wanted to check in.

Regarding implementation, would the upsampling be something you would want added to the pooling class or its own upsample class?

This PR covers the first part of items discussed in #34, namely, the installation.

A follow-up PR (hopefully tomorrow) should cover the 2nd part, namely, the generation and installation of dlibConfig.cmake

Environment: Pop-Os (Ubuntu 20.04) Gcc 9 and 8 (tried both) Cmake 3.16.3 dlib 19.20.99 python 3.8.2 NVIDIA Quadro P600

Expected Behavior

Compiling with cuda..

Current Behavior

..... -- Found cuDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so -- Building a CUDA test project to see if your compiler is compatible with CUDA... -- Checking if you have the right version of cuDNN installed. -- *** Found cuDNN, but it looks like the wrong version so dlib will not use it. *** -- *** Dlib requires cuDNN V5.0 OR GREATER. Since cuDNN is not found DLIB WILL NOT USE CUDA. .....

• Where did you get dlib: git clone https://github.com/davisking/dlib.git

I first tried downloading the latest (and suggested) version of cuda from Nvidia site, which was Cuda 11. then downloaded cuDNN for Cuda 11, which was 8.0.0 (runtime and dev .deb packets) installed them following Nvidia method.

when compiling "cudnn_samples_v8" everything works, so I think installations went ok. but no way to get dlib compiled with Cuda.

I've tried to uninstall cuda 11 and cudnn 8 and install cuda 10.1 and cudnn 7.6 (suggested for cuda 10.1) but the result is the same. Every time I erased the build directory, and compiled in 2 ways: as per dlib instructions, using: $ sudo python3 setup.py install or: $ cmake .. -DDLIB_USE_CUDA=1 -DUSE_AVX_INSTRUCTIONS=1

Any suggestion? Thanks.

Feature suggestion: allow the fft routine to do arbitrary sized FFTs. Let FFTW, lapack, BLAS or whatever do the magic of choosing the correct kernels for the input

I'm having another go as I'm in need of this again and would rather do it properly. The previous PR has fallen behind and my git rebase/merge skills are pants. In any case, it was too big. So i've split it into 2 parts:

• decoding/demuxing
• encoding/muxing This PR includes the first part.

Need little guidance about the formation/generation of image from the encoded-face model, my face_recognition.face_encoding() has helped a lot, I want to know if I can convert back the encoded face-model with the person's face image, and what all steps it required to convert an image back from the encoded saved face-model.

Please guide

• Version:
• Where did you get dlib:
• Platform:
• Compiler:

I try to run a program but it returns an error

################################### Traceback (most recent call last): File "/data/data/com.termux/files/home/./faceswap.py", line 47, in import dlib File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/dlib-19.24.99-py3.11-linux-aarch64.egg/dlib/init.py", line 19, in from _dlib_pybind11 import * ImportError: dlopen failed: cannot locate symbol "PyExc_ImportError" referenced by "/data/data/com.termux/files/usr/lib/python3.11/site-packages/dlib-19.24.99-py3.11-linux-aarch64.egg/_dlib_pybind11.cpython-311.so"... ###################################

then,I try to replace "from _dlib_pybind11 import *" with "#from _dlib_pybind11 import".But another error appeared

###################################Traceback (most recent call last): File "/data/data/com.termux/files/home/./faceswap.py", line 47, in import dlib File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/dlib-19.24.99-py3.11-linux-aarch64.egg/dlib/init.py", line 20, in from _dlib_pybind11 import version, time_compiled ImportError: dlopen failed: cannot locate symbol "PyExc_ImportError" referenced by "/data/data/com.termux/files/usr/lib/python3.11/site-packages/dlib-19.24.99-py3.11-linux-aarch64.egg/_dlib_pybind11.cpython-311.so"... ###################################

How to fix it?

dlib version:19.24.99 python version:3.11.1 termux version:0.118.21 android version:12

Dlib build gets stuck forever

Expected Behavior

It should at least print out more details why it gets stuck at those lines of code. Or it should not stuck too long like that.

Current Behavior

Stucks at these lines

 => [stage-1  9/16] RUN pip install --no-cache-dir dlib -vvv                                                                                                                                                                                           506.9s
 => => #                    from /tmp/pip-install-2d_opbew/dlib_93a3736d4a0d4fcaa59f51c92234fb97/dlib/../dlib/python.h:6,
 => => #                    from /tmp/pip-install-2d_opbew/dlib_93a3736d4a0d4fcaa59f51c92234fb97/tools/python/src/opaque_types.h:6,
 => => #                    from /tmp/pip-install-2d_opbew/dlib_93a3736d4a0d4fcaa59f51c92234fb97/tools/python/src/face_recognition.cpp:4:
 => => #   /usr/local/include/python3.9/ceval.h:130:37: note: declared here
 => => #     130 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
 => => #         |

Steps to Reproduce

• Version: Latest (since I run pip install dlib)
• Where did you get dlib: From official source
• Platform: I use img to build a Dockerimage, inside the Dockerfile there is a step to install dlib using pip

FROM python:3.9.14
# Install base packages for Python
RUN pip install --no-cache-dir huggingface_hub flask numpy pandas scipy matplotlib Pillow cython \
    torch torchvision cmake virtualenv ndg-httpsclient ldm h5py datasets
RUN pip install --no-cache-dir dlib -vvv

img tool: https://github.com/genuinetools/img

Hi

Please provide me the example for using the find_max_global with non fixed number of arguyments. The current example present doesn't show the functionality

Thanks, Ankit

As the http://dlib.net/ml.html#test_shape_predictor says, Tests a shape_predictor's ability to correctly predict the part locations of objects. The output is the average distance (measured in pixels) between each part and its true location.

I simply ran print("\Testing accuracy:{0}".format(dlib.test_shape_predictor(testing_xml_path, "modelbk/predictor.dat"))) for my own model and print("\Testing accuracy:{0}".format(dlib.test_shape_predictor(testing_xml_path, "shape_predictor_68_face_landmarks.dat"))) for a pretrained model. and get the Testing accuracy 81.7678 and 322.52788 for my own model and pretrained model, how to understand these numbers? My own model cannot give the correct locations.

What's more, my own model was trained with "options.tree_depth = 2", which in my opinion caused the model size far smaller than the pretrained "shape_predictor_68_face_landmarks.dat", can I test these two model in the above way?