Was training on custom human dataset. Batch Size = 8 No of training images = 3800

No of steps trained before showing error = 75

After 75th step It generated an error:

RuntimeError: Function 'UpsampleBilinear2DBackward1' returned nan values in its 0th output.

The model trained successfully when using BCE loss.

We even checked for NaN values using torch.autograd.set_detect_anamoly(True) But it returned False stating that no NaN values were found

Hello,

I would like to apply your API_loss to my 3_classes semantic segmentation task, but i am not sure it will be work or not after some modification? Do you have any advice? Look forward to your reply~

Thanks, best wish!

Thanks for your great work! How can I use the trained model to test when I got the best model after training on my data？ Because it seems that the pre training model is still used in the inference stage?

Hi, I tried your code to reproduce your result. I used python main.py train --arch 1 --img_size 320 to train the model. I did not change any setting in your code. I used four GPU to train this model. But the MAE I got on DUTS-TE is around 0.088 (it is 0.033 in paper). In picture, it shows the loss and the MAE of train and val set when training. It early stopped at 82th epoch. It kind of converged.

I would like to know did you do any data augmentation off-line (I train on DUTS-TR and the edge gt you provided. I simply change the folder directory to meet your folder structure mentioned in readme)? Should I change any hyper parameter to achieve the same result?

Tried training using the same hyper params of the paper in order to reproduce results on the DUTS-TE split for example.
Apart from batch size which had to be reduced to 8, all other params were the same.
However, I reach inferior results:

Dataset:DUTS
Test Loss:1.457 | MAX_F:0.903 | AVG_F:0.873 | MAE:0.030 | S_Measure:0.891, time: 68938.211s

This is the training command I used:

python main.py train --exp_num 1 --arch 7 --img_size 640 --model_path /data/outputs/TRACER --data_path /data/datasets/TRACER/ --multi_gpu true --num_workers 12 --batch_size 8

The best model was achieved at epoch 14 as stated in the logs:

Best Val Epoch:14 | Val Loss:38.960 | Val MAE:0.019

And the training concluded after epoch 24 due to early stopping mechanism.

I tested the model for some time and witnessed something strange. The same mask doesn't show up if i entered the same input image. It seems there is a random seed generator in the code that affects the output results.

Do you have any idea where something like this can happen ?

Hi, I downloaded pre-trained models of TRACER as you mentioned in README, but I'm confused about the step below Change the model name as 'best_model.pth' and put the weights to the path 'results/DUTS/TEx_0/best_model.pth'

question is, this path is model save path, not pretrained model load path, and I could not find you load "best_model.pth" at start of training anywhere in your code, did I miss something?

Hi, thanks for putting out an interesting paper and the code. I was testing it with the various pretrained weights on DUT-TE, and I'm getting an end-to-end speed of ~11 FPS on an RTX 2070 Super. Is that an expected speed?

Hi This is really an amazing work. I would like to reproduce your result. But I found you did not mentioned how you generate gt edge data in the paper. And I don't have the permission to access the gt edge data link in README.

Of course, it would be wonderful if you can provide the full pre-trained model.

Hi, I tried to replicate your code and managed to replicate the test on the DUTS test set using TE0_0 pretrained model. However I would like to know how to get the actual masks predicted of the images after running the test. Thanks, Malcolm

Hello! And thank you for this work. I was curious if you would be releasing or amending (or if I missed it in your paper) the mask generation part of your testing code? To try against more complicated imagery in visual form?

Bumps certifi from 2021.5.30 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I created a custom dataset and tried training with arch 5 with image size 512 but the loss decreased quite slowly. Is this normal, or is my data problem?

my loss Epoch:[001/200] Train Loss:99.721 | MAE:0.189 Valid Loss:82.747 | MAE:0.125 =============================Epoch:[002/200] Train Loss:91.156 | MAE:0.156 Valid Loss:91.560 | MAE:0.166 =============================Epoch:[003/200] Train Loss:91.532 | MAE:0.155 Valid Loss:78.634 | MAE:0.106 =============================Epoch:[004/200] Train Loss:86.963 | MAE:0.135 Valid Loss:76.247 | MAE:0.095 =============================Epoch:[005/200] Train Loss:85.692 | MAE:0.130 Valid Loss:74.397 | MAE:0.086 =============================Epoch:[006/200] Train Loss:85.188 | MAE:0.127 Valid Loss:74.988 | MAE:0.086 =============================Epoch:[007/200] Train Loss:84.972 | MAE:0.125 Valid Loss:73.414 | MAE:0.082 =============================Epoch:[008/200] Train Loss:83.838 | MAE:0.121 Valid Loss:75.131 | MAE:0.087 =============================Epoch:[009/200] Train Loss:83.403 | MAE:0.119

Hi, I'm getting this error. @Karel911

<---- Training Params ----> Namespace(RFB_aggregated_channel=[32, 64, 128], action='train', arch='7', aug_ver=1, batch_size=8, channels=[24, 40, 112, 320], clipping=2, criterion='API', data_path='/home/nagas/TRACER/data', dataset='DUTS', denoise=0.93, epochs=100, exp_num=0, frequency_radius=16, gamma=0.1, img_size=640, lr=5e-05, lr_factor=0.1, model_path='/home/nagas/TRACER/results', multi_gpu=True, num_workers=4, optimizer='Adam', patience=5, save_map=None, scheduler='Reduce', seed=42, weight_decay=0.0001) train length : 42 val length : 3 Loaded pretrained weights for efficientnet-b7 0%| | 0/5 [00:00<?, ?it/s]ERROR: Unexpected segmentation fault encountered in worker. ERROR: Unexpected segmentation fault encountered in worker. ERROR: Unexpected segmentation fault encountered in worker. ERROR: Unexpected segmentation fault encountered in worker. 0%| | 0/5 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/queues.py", line 104, in get if not self._poll(timeout): File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/connection.py", line 257, in poll return self._poll(timeout) File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/connection.py", line 414, in _poll r = wait([self], timeout) File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/connection.py", line 921, in wait ready = selector.select(timeout) File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/selectors.py", line 415, in select fd_event_list = self._selector.poll(timeout) File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 20604) is killed by signal: Segmentation fault.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "main.py", line 56, in main(args) File "main.py", line 35, in main Trainer(args, save_path) File "/home/nagas/TRACER/trainer.py", line 56, in init train_loss, train_mae = self.training(args) File "/home/nagas/TRACER/trainer.py", line 101, in training for images, masks, edges in tqdm(self.train_loader): File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/tqdm/std.py", line 1185, in iter for obj in iterable: File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data() File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data idx, data = self._get_data() File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data success, data = self._try_get_data() File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 20604) exited unexpectedly

Hello. Thank you for your wonderful research.

I want to extend your methodology to multi-class classification. Is this possible? If possible, could you give me some implementation tips?

hi,thank you for your excellent work，I don't understand the following formula. Can you explain it for me？ 1、Why does equation 8 need to be multiplied by yij? This will cause all background weights to be 0。 2、What is the meaning of adding 1.5 to the denominator in formula 9? Thank you very much and hope to get your reply！！