Hi,

I have some questions about the training procedure:

1. In the paper, you've mentioned training Stage 1, for 15 epochs, while in the code you've set the instructions to 20 epochs. Is there a reason for such change? Will the results be similar?
2. I could not get access to Distinctions-646, I had no reply from the authors/maintainers of the dataset. Based on your indicated file structure, I've built a similar dataset, which adds uncertainty to the quality of my training, but it is a risk I am willing to take. To have a comparison parameter (stages 1-3) do not depend on this dataset, would you mind sharing your partial training weights on pytorch (stage1/epoch19.pth, stage2/epoch21.pth, and stage3/epoch22.pth)?
3. What is the min resolution you've used for the background images while training?

For the 3rd time, thank you very much for your contribution to the field. It was a brilliant work. Looking forward to your future work.

I've been trying to export an onnx model replacing the hardsigmoid operator.

I have modified the site-packages/torch/onnx/symbolic_opset9.py file this way:

@parse_args("v") def hardswish(g, self): hardsigmoid = g.op('HardSigmoid', self, alpha_f=1 / 6) return g.op("Mul", self, hardsigmoid)

@parse_args("v") def hardsigmoid(g, self): hardsigmoid = g.op('HardSigmoid', self, alpha_f=1 / 6) return g.op("Mul", self, hardsigmoid)

But I am not sure at all if this is the way to replace them with primitive ops

When I export the onnx with this change I still get and error "OnnxImportException: Unknown type HardSigmoid encountered while parsing layer 396" with the inference engine I am trying to use.

if I'm going to train stage3 and stage4, the VideoMatte-HD data will be used. And is it right to modify the following path？VideoMatte240K_JPEG_SD to VideoMatte240K_JPEG_HD

'videomatte': { 'train': '../matting-data/VideoMatte240K_JPEG_SD/train', 'valid': '../matting-data/VideoMatte240K_JPEG_SD/valid', },

Hey there, I just ported RVM to Unity using NatML, an open-source machine learning runtime. I have questions:

1. Can I make a PR to add a link into the README to a Unity example project demonstrating using RVM?
2. I published the model under my account on NatML Hub. Would you be interested in signing up on Hub, so that I can transfer the model to you?

Here's the model on NatML Hub:

@natsuite/robust-video-matting

1.How to eliminate or reduce edge flickering problem，can i set --seq-length-lr Is it possible to increase the sequence length improvement，Does it work? 2.Only the composite image has no foreground image，Is it possible to remove foreground training，and foreground loss？or is there a better way？ 3.How important is foreground prediction for matting

Looking forward to your reply

First of all thank you for working on this project! it looks much stronger than the BMV2 !

1. Will it work on Anaconda and Windows 10 just like BMV2 works? (not more complicated?)

2. Will it support same hardware, or need a much more powerful CPU / GPU compare to BMV2 ?

3. Can you please tell when will you release it again, I missed it first so I can't test it because it's still offline. It will be very nice to have it this week if possible of course.

Thanks ahead for the answers, please keep up the good work! ❤

Hello!. Thanks for the code. I have had some timing issues between the inferred output in the mask compared to the original video. I made this comparison by transforming my original video and the output video from masks to frames. I have obtained the same amount of frames in both processes, so the difference can be caused by a bad configuration of mine. My original video is 30fps and 1080x1920. If you have a suggestion I would appreciate it.

https://github.com/PeterL1n/RobustVideoMatting/blob/f8a26e27198a93a94bfd06e96b8d5a34d0660f81/inference.py#L127

I changed this line to use Segmentation Pass. (use the pretrained weights rvm_mobilenetv3.pth)

pha, *rec = model(src, *rec, segmentation_pass=True)
fgr = src * pha

But I got weird mask results, something like this, why?

大神辛苦,两个问题请教....... 1.除了更改downsample_ratio的参数值来修正抠图的精度,还可以更改那些参数来更改实现效果? 2.此项目对显卡的要求是否更高?显卡的型号会影响最后结果么? 目前,有执行model的项目,但是效果并不是很理想,再次感谢!

Hello, two questions to consult.......

1. In addition to changing the parameter value of downsample_ratio to correct the accuracy of matting, which other parameters can be changed to change the implementation effect?

2. Does this project have higher requirements for graphics cards? Does the type of graphics card affect the final results?

I have my own project to implement model, but the effect is not very ideal, thank you!(Translation from Youdao Translation)

I use pre-trained ONNX model parameters for inference tests (in Python not C++), only onnxruntime, cv2 and numpy libraries, nothing extra. Parameters downloaded from https://github.com/PeterL1n/RobustVideoMatting/releases/: rvm_mobilenetv3_fp32.onnx and rvm_mobilenetv3_fp16.onnx

Inference on 1080x1920 video，downsample_ratio=0.25. As a result, the speed of FP32 is about 170ms (1 frame), and the speed of FP16 is about 240ms. Why is FP16 so slow?

I have adjusted the input correctly, for src, r1i, r2i, r3i, r4i it is np.array([[[[]]]], dtype=np.float32 or 16) and for downsample_ratio it is always np.array([0.25], dtype= float32)

I use CPU (Intel i5) for inference, Is it so slow because the CPU does not support FP16 operations?

你好，请教一下，关于前景预测，从官方提供的web demo中,我看到模型预测的前景图片中除了前景(人像)外，还存在输入图片的背景细节（非人像像素），但是我自己训练得到的模型（我的模型没有修改官方的任何细节，唯一的不同仅仅是采用我采集而来的背景图片），预测的前景图片只含有人像而不会存在输入图片的背景细节，一开始我怀疑可能是前景loss包含了所有像素（alpha可以是任何值而不仅仅是像论文中所说的大于0）的loss, 但是我查看代码后没有任何问题，和论文一致，请问这是什么原因造成？ 谢谢。

Hey @PeterL1n ! 👋

Thanks for this wonderful project!

This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

This also means we can make a web page where other people can run your model! View it here: https://replicate.com/arielreplicate/robust_video_matting Replicate also have an API, so people can easily run your model from their code:

import replicate
model = replicate.models.get("[arielreplicate/robust_video_matting]")
model.predict(...)

If you'd like to modify the Replicate page, let me know and I can transfer ownership to your account.

In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

Hello How are you? Thanks for contributing to this project. I tried to use NCNN version of this model on Windows C++. Of course, there are several github repos using NCNN model in C++ but I can NOT run them because there are some issue when extracting the output data from model. First, the model are old so it is impossible to run them. Could u support NCNN version here?

I tried both the replicate page and local COG variants. When predicting with alpha-mask, this error is a constant:

FileNotFoundError: [Errno 2] No such file or directory: 'alpha-mask.mp4'

Exporting with green-screen and foreground-mask works however. Maybe this is a issue with mp4 not supporting alpha transparent video, so it fails?

I see "x = normalize(x, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])" in mobilenetv3.py and ''' f1, f2, f3, f4 = self.backbone(src_sm) ... hid, *rec = self.decoder(src_sm, f1, f2, f3, f4, r1, r2, r3, r4) ''' in model.py. This means the input src_sm of the decoder has not been normalized. Is that your intention?

Hey there,

thanks for this amazing tool! Does anybody know how the performance for grayscale images is? I want to use it in the dark and I have an infrared camera.

If retraining is required how much GPU hours do you think are necessary?

:)

The inference.py and its running at ~4.2it/s. It barely loads my RTX2060 (0-13% use) The inference_speed_test script gives me ~33.2it/s on the same model and video settings. Changing the --workers on the convert_video() function did nothing. Am I missing something? How can I run inferences faster using the full hardware potential?

Thanks.