Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

While training a custom data in intermediate i'm getting error

See original GitHub issue

[ 16] 12400 || B: 5.023 | C: 2.542 | M: 6.272 | S: 0.114 | T: 13.950 || ETA: 3 days, 6:05:47 || timer: 0.358 [ 16] 12410 || B: 5.001 | C: 2.542 | M: 6.508 | S: 0.126 | T: 14.177 || ETA: 3 days, 6:05:50 || timer: 0.355 [ 16] 12420 || B: 5.125 | C: 2.548 | M: 6.596 | S: 0.129 | T: 14.397 || ETA: 3 days, 6:05:45 || timer: 0.357

Computing validation mAP (this may take a while)…

Calculating mAP…

   |  all  |  .50  |  .55  |  .60  |  .65  |  .70  |  .75  |  .80  |  .85  |  .90  |  .95  |

-------±------±------±------±------±------±------±------±------±------±------±------+ box | 6.09 | 20.33 | 20.30 | 16.83 | 3.47 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | mask | 3.22 | 7.92 | 7.92 | 7.92 | 7.92 | 0.50 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -------±------±------±------±------±------±------±------±------±------±------±------+

[ 17] 12430 || B: 4.725 | C: 2.548 | M: 6.641 | S: 0.135 | T: 14.049 || ETA: 3 days, 6:52:38 || timer: 0.356 [ 17] 12440 || B: 4.636 | C: 2.541 | M: 6.520 | S: 0.121 | T: 13.818 || ETA: 3 days, 6:52:41 || timer: 0.357 [ 17] 12450 || B: 4.655 | C: 2.538 | M: 6.710 | S: 0.119 | T: 14.022 || ETA: 3 days, 6:52:40 || timer: 0.357 [ 17] 12460 || B: 4.801 | C: 2.533 | M: 6.707 | S: 0.113 | T: 14.155 || ETA: 3 days, 6:52:29 || timer: 0.359 [ 17] 12470 || B: 4.465 | C: 2.544 | M: 6.497 | S: 0.123 | T: 13.629 || ETA: 3 days, 6:52:34 || timer: 0.357 [ 17] 12480 || B: 4.464 | C: 2.539 | M: 6.670 | S: 0.136 | T: 13.809 || ETA: 3 days, 6:52:32 || timer: 0.356 [ 17] 12490 || B: 4.622 | C: 2.534 | M: 7.012 | S: 0.139 | T: 14.306 || ETA: 3 days, 6:52:29 || timer: 0.355 [ 17] 12500 || B: 4.615 | C: 2.538 | M: 7.111 | S: 0.153 | T: 14.417 || ETA: 3 days, 6:52:25 || timer: 0.358 [ 17] 12510 || B: 4.576 | C: 2.560 | M: 6.934 | S: 0.144 | T: 14.214 || ETA: 3 days, 6:52:24 || timer: 0.359 [ 17] 12520 || B: 4.516 | C: 2.553 | M: 6.547 | S: 0.142 | T: 13.758 || ETA: 3 days, 6:52:19 || timer: 0.357 [ 17] 12530 || B: 4.567 | C: 2.547 | M: 6.474 | S: 0.135 | T: 13.724 || ETA: 3 days, 6:52:20 || timer: 0.355 [ 17] 12540 || B: 4.792 | C: 2.532 | M: 6.801 | S: 0.121 | T: 14.246 || ETA: 3 days, 6:52:16 || timer: 0.355 [ 17] 12550 || B: 4.766 | C: 2.525 | M: 6.734 | S: 0.130 | T: 14.155 || ETA: 3 days, 6:52:20 || timer: 0.358 [ 17] 12560 || B: 4.564 | C: 2.511 | M: 6.705 | S: 0.129 | T: 13.910 || ETA: 3 days, 6:52:25 || timer: 0.357 [ 17] 12570 || B: 4.467 | C: 2.486 | M: 6.713 | S: 0.136 | T: 13.803 || ETA: 3 days, 6:52:25 || timer: 0.355 [ 17] 12580 || B: 4.381 | C: 2.495 | M: 6.505 | S: 0.126 | T: 13.507 || ETA: 3 days, 6:52:31 || timer: 0.356 [ 17] 12590 || B: 3.971 | C: 2.483 | M: 6.208 | S: 0.132 | T: 12.795 || ETA: 3 days, 6:52:30 || timer: 0.356 [ 17] 12600 || B: 3.766 | C: 2.477 | M: 6.069 | S: 0.129 | T: 12.441 || ETA: 3 days, 6:52:29 || timer: 0.358 [ 17] 12610 || B: 3.656 | C: 2.452 | M: 5.964 | S: 0.128 | T: 12.200 || ETA: 3 days, 6:52:27 || timer: 0.356 [ 17] 12620 || B: 3.622 | C: 2.439 | M: 6.141 | S: 0.125 | T: 12.326 || ETA: 3 days, 6:52:33 || timer: 0.356 [ 17] 12630 || B: 3.712 | C: 2.427 | M: 6.246 | S: 0.124 | T: 12.509 || ETA: 3 days, 6:52:38 || timer: 0.359 [ 17] 12640 || B: 3.489 | C: 2.434 | M: 6.113 | S: 0.120 | T: 12.156 || ETA: 3 days, 6:52:37 || timer: 0.358 [ 17] 12650 || B: 3.384 | C: 2.445 | M: 5.906 | S: 0.120 | T: 11.855 || ETA: 3 days, 6:52:33 || timer: 0.355 [ 17] 12660 || B: 3.653 | C: 2.459 | M: 6.166 | S: 0.117 | T: 12.394 || ETA: 3 days, 6:52:36 || timer: 0.353 [ 17] 12670 || B: 3.760 | C: 2.464 | M: 6.224 | S: 0.100 | T: 12.549 || ETA: 3 days, 6:52:38 || timer: 0.357 [ 17] 12680 || B: 3.882 | C: 2.452 | M: 6.390 | S: 0.097 | T: 12.822 || ETA: 3 days, 6:52:39 || timer: 0.359 [ 17] 12690 || B: 4.236 | C: 2.455 | M: 6.754 | S: 0.095 | T: 13.540 || ETA: 3 days, 6:52:50 || timer: 0.354 [ 17] 12700 || B: 4.662 | C: 2.460 | M: 7.379 | S: 0.104 | T: 14.606 || ETA: 3 days, 6:49:43 || timer: 0.357 [ 17] 12710 || B: 4.683 | C: 2.454 | M: 7.404 | S: 0.111 | T: 14.652 || ETA: 3 days, 6:49:45 || timer: 0.359 [ 17] 12720 || B: 4.698 | C: 2.459 | M: 7.431 | S: 0.112 | T: 14.700 || ETA: 3 days, 6:49:39 || timer: 0.357 [ 17] 12730 || B: 4.437 | C: 2.468 | M: 7.125 | S: 0.128 | T: 14.158 || ETA: 3 days, 6:49:43 || timer: 0.356 [ 17] 12740 || B: 4.434 | C: 2.470 | M: 7.025 | S: 0.145 | T: 14.074 || ETA: 3 days, 6:49:40 || timer: 0.354 [ 17] 12750 || B: 4.448 | C: 2.453 | M: 7.122 | S: 0.168 | T: 14.192 || ETA: 3 days, 6:49:34 || timer: 0.357 [ 17] 12760 || B: 4.289 | C: 2.456 | M: 6.763 | S: 0.171 | T: 13.679 || ETA: 3 days, 6:49:26 || timer: 0.361 [ 17] 12770 || B: 4.462 | C: 2.445 | M: 6.904 | S: 0.171 | T: 13.983 || ETA: 3 days, 6:49:18 || timer: 0.354 [ 17] 12780 || B: 4.606 | C: 2.441 | M: 6.875 | S: 0.173 | T: 14.096 || ETA: 3 days, 6:49:05 || timer: 0.354 [ 17] 12790 || B: 4.360 | C: 2.450 | M: 6.649 | S: 0.159 | T: 13.616 || ETA: 3 days, 6:48:55 || timer: 0.356 [ 17] 12800 || B: 4.125 | C: 2.456 | M: 6.210 | S: 0.138 | T: 12.928 || ETA: 3 days, 6:48:58 || timer: 0.360 [ 17] 12810 || B: 4.230 | C: 2.464 | M: 6.044 | S: 0.129 | T: 12.867 || ETA: 3 days, 6:48:49 || timer: 0.357 [ 17] 12820 || B: 3.991 | C: 2.453 | M: 5.686 | S: 0.128 | T: 12.259 || ETA: 3 days, 6:48:46 || timer: 0.358 [ 17] 12830 || B: 4.185 | C: 2.455 | M: 5.814 | S: 0.114 | T: 12.567 || ETA: 3 days, 6:48:32 || timer: 0.354 [ 17] 12840 || B: 4.152 | C: 2.443 | M: 5.731 | S: 0.099 | T: 12.424 || ETA: 3 days, 6:48:17 || timer: 0.358 [ 17] 12850 || B: 4.328 | C: 2.451 | M: 5.711 | S: 0.064 | T: 12.554 || ETA: 3 days, 6:48:10 || timer: 0.358 [ 17] 12860 || B: 4.382 | C: 2.445 | M: 5.515 | S: 0.063 | T: 12.404 || ETA: 3 days, 6:48:15 || timer: 0.357 [ 17] 12870 || B: 4.257 | C: 2.455 | M: 5.234 | S: 0.081 | T: 12.026 || ETA: 3 days, 6:48:15 || timer: 0.353 [ 17] 12880 || B: 4.149 | C: 2.469 | M: 5.083 | S: 0.081 | T: 11.782 || ETA: 3 days, 6:48:12 || timer: 0.357 [ 17] 12890 || B: 4.106 | C: 2.469 | M: 4.845 | S: 0.086 | T: 11.506 || ETA: 3 days, 6:48:12 || timer: 0.359 [ 17] 12900 || B: 4.060 | C: 2.466 | M: 4.778 | S: 0.116 | T: 11.420 || ETA: 3 days, 6:48:11 || timer: 0.355 [ 17] 12910 || B: 3.959 | C: 2.472 | M: 4.964 | S: 0.124 | T: 11.519 || ETA: 3 days, 6:48:11 || timer: 0.355 [ 17] 12920 || B: 4.053 | C: 2.493 | M: 5.045 | S: 0.146 | T: 11.738 || ETA: 3 days, 6:48:03 || timer: 0.355 [ 17] 12930 || B: 4.230 | C: 2.499 | M: 5.152 | S: 0.145 | T: 12.026 || ETA: 3 days, 6:47:59 || timer: 0.360 [ 17] 12940 || B: 4.517 | C: 2.501 | M: 5.243 | S: 0.146 | T: 12.408 || ETA: 3 days, 6:48:02 || timer: 0.357 [ 17] 12950 || B: 4.292 | C: 2.520 | M: 5.012 | S: 0.205 | T: 12.029 || ETA: 3 days, 6:47:58 || timer: 0.357 [ 17] 12960 || B: 4.167 | C: 2.512 | M: 5.189 | S: 0.228 | T: 12.096 || ETA: 3 days, 6:47:54 || timer: 0.358 [ 17] 12970 || B: 4.392 | C: 2.516 | M: 5.241 | S: 0.213 | T: 12.363 || ETA: 3 days, 6:47:48 || timer: 0.357 [ 17] 12980 || B: 4.339 | C: 2.501 | M: 5.229 | S: 0.214 | T: 12.283 || ETA: 3 days, 6:47:57 || timer: 0.358 [ 17] 12990 || B: 4.413 | C: 2.495 | M: 5.140 | S: 0.218 | T: 12.266 || ETA: 3 days, 6:48:00 || timer: 0.356 [ 17] 13000 || B: 4.544 | C: 2.484 | M: 4.857 | S: 0.192 | T: 12.077 || ETA: 3 days, 6:47:55 || timer: 0.355 [ 17] 13010 || B: 4.561 | C: 2.477 | M: 4.672 | S: 0.217 | T: 11.927 || ETA: 3 days, 6:47:55 || timer: 0.357 [ 17] 13020 || B: 4.617 | C: 2.474 | M: 4.626 | S: 0.198 | T: 11.915 || ETA: 3 days, 6:47:59 || timer: 0.358 [ 17] 13030 || B: 4.330 | C: 2.474 | M: 4.454 | S: 0.199 | T: 11.457 || ETA: 3 days, 6:48:02 || timer: 0.354 [ 17] 13040 || B: 4.154 | C: 2.482 | M: 4.343 | S: 0.202 | T: 11.180 || ETA: 3 days, 6:48:02 || timer: 0.355 [ 17] 13050 || B: 4.306 | C: 2.458 | M: 4.562 | S: 0.264 | T: 11.590 || ETA: 3 days, 6:48:10 || timer: 0.358 [ 17] 13060 || B: 4.233 | C: 2.463 | M: 4.518 | S: 0.241 | T: 11.455 || ETA: 3 days, 6:48:01 || timer: 0.358 [ 17] 13070 || B: 3.974 | C: 2.460 | M: 4.445 | S: 0.267 | T: 11.146 || ETA: 3 days, 6:47:55 || timer: 0.361 [ 17] 13080 || B: 3.946 | C: 2.459 | M: 4.613 | S: 0.262 | T: 11.281 || ETA: 3 days, 6:47:59 || timer: 0.357 [ 17] 13090 || B: 3.916 | C: 2.463 | M: 4.699 | S: 0.254 | T: 11.331 || ETA: 3 days, 6:48:01 || timer: 0.356 [ 17] 13100 || B: 3.617 | C: 2.444 | M: 4.736 | S: 0.246 | T: 11.043 || ETA: 3 days, 6:47:56 || timer: 0.357 [ 17] 13110 || B: 3.743 | C: 2.442 | M: 5.006 | S: 0.211 | T: 11.403 || ETA: 3 days, 6:47:58 || timer: 0.356 [ 17] 13120 || B: 3.750 | C: 2.436 | M: 5.360 | S: 0.205 | T: 11.751 || ETA: 3 days, 6:48:03 || timer: 0.357 [ 17] 13130 || B: 3.877 | C: 2.423 | M: 5.334 | S: 0.231 | T: 11.866 || ETA: 3 days, 6:48:04 || timer: 0.356 [ 17] 13140 || B: 3.743 | C: 2.425 | M: 5.388 | S: 0.228 | T: 11.783 || ETA: 3 days, 6:48:00 || timer: 0.358 [ 17] 13150 || B: 3.511 | C: 2.420 | M: 5.206 | S: 0.108 | T: 11.244 || ETA: 3 days, 6:47:55 || timer: 0.354 [ 18] 13160 || B: 3.625 | C: 2.424 | M: 5.388 | S: 0.124 | T: 11.560 || ETA: 3 days, 6:50:44 || timer: 0.359 [ 18] 13170 || B: 3.746 | C: 2.428 | M: 5.483 | S: 0.102 | T: 11.760 || ETA: 3 days, 6:50:49 || timer: 0.358 [ 18] 13180 || B: 3.817 | C: 2.440 | M: 5.298 | S: 0.111 | T: 11.665 || ETA: 3 days, 6:50:39 || timer: 0.356 [ 18] 13190 || B: 3.805 | C: 2.447 | M: 5.404 | S: 0.127 | T: 11.783 || ETA: 3 days, 6:50:39 || timer: 0.360 [ 18] 13200 || B: 3.899 | C: 2.494 | M: 5.348 | S: 0.128 | T: 11.868 || ETA: 3 days, 6:50:53 || timer: 0.357 [ 18] 13210 || B: 3.804 | C: 2.495 | M: 5.139 | S: 0.129 | T: 11.566 || ETA: 3 days, 6:50:54 || timer: 0.357 [ 18] 13220 || B: 3.627 | C: 2.495 | M: 4.718 | S: 0.132 | T: 10.972 || ETA: 3 days, 6:50:40 || timer: 0.356 [ 18] 13230 || B: 3.515 | C: 2.516 | M: 4.471 | S: 0.108 | T: 10.610 || ETA: 3 days, 6:50:37 || timer: 0.359 [ 18] 13240 || B: 3.490 | C: 2.516 | M: 4.434 | S: 0.107 | T: 10.546 || ETA: 3 days, 6:50:41 || timer: 0.358 [ 18] 13250 || B: 3.711 | C: 2.534 | M: 4.487 | S: 0.125 | T: 10.856 || ETA: 3 days, 6:50:41 || timer: 0.358 [ 18] 13260 || B: 3.955 | C: 2.518 | M: 4.463 | S: 0.106 | T: 11.042 || ETA: 3 days, 6:50:42 || timer: 0.354 [ 18] 13270 || B: 4.028 | C: 2.522 | M: 4.476 | S: 0.109 | T: 11.135 || ETA: 3 days, 6:50:51 || timer: 0.359 [ 18] 13280 || B: 3.926 | C: 2.513 | M: 4.480 | S: 0.101 | T: 11.020 || ETA: 3 days, 6:50:48 || timer: 0.358 [ 18] 13290 || B: 3.781 | C: 2.493 | M: 4.353 | S: 0.084 | T: 10.710 || ETA: 3 days, 6:50:50 || timer: 0.356 [ 18] 13300 || B: 3.715 | C: 2.466 | M: 4.472 | S: 0.083 | T: 10.736 || ETA: 3 days, 6:50:44 || timer: 0.356 [ 18] 13310 || B: 3.626 | C: 2.461 | M: 4.440 | S: 0.085 | T: 10.612 || ETA: 3 days, 6:50:38 || timer: 0.357 [ 18] 13320 || B: 3.671 | C: 2.474 | M: 4.495 | S: 0.087 | T: 10.727 || ETA: 3 days, 6:50:29 || timer: 0.358 [ 18] 13330 || B: 3.743 | C: 2.463 | M: 4.461 | S: 0.097 | T: 10.764 || ETA: 3 days, 6:50:21 || timer: 0.357 [ 18] 13340 || B: 3.751 | C: 2.471 | M: 4.291 | S: 0.102 | T: 10.615 || ETA: 3 days, 6:50:13 || timer: 0.356 [ 18] 13350 || B: 3.877 | C: 2.460 | M: 4.501 | S: 0.084 | T: 10.921 || ETA: 3 days, 6:50:03 || timer: 0.355 [ 18] 13360 || B: 3.676 | C: 2.476 | M: 4.548 | S: 0.094 | T: 10.794 || ETA: 3 days, 6:49:55 || timer: 0.356 [ 18] 13370 || B: 3.430 | C: 2.475 | M: 4.317 | S: 0.103 | T: 10.325 || ETA: 3 days, 6:49:57 || timer: 0.358 [ 18] 13380 || B: 3.524 | C: 2.510 | M: 4.092 | S: 0.104 | T: 10.230 || ETA: 3 days, 6:50:03 || timer: 0.354 [ 18] 13390 || B: 3.721 | C: 2.527 | M: 4.112 | S: 0.109 | T: 10.468 || ETA: 3 days, 6:49:57 || timer: 0.357 [ 18] 13400 || B: 3.842 | C: 2.520 | M: 4.090 | S: 0.107 | T: 10.559 || ETA: 3 days, 6:49:57 || timer: 0.360 [ 18] 13410 || B: 4.212 | C: 2.557 | M: 4.283 | S: 0.105 | T: 11.157 || ETA: 3 days, 6:49:55 || timer: 0.358 [ 18] 13420 || B: 4.443 | C: 2.561 | M: 4.393 | S: 0.103 | T: 11.499 || ETA: 3 days, 6:49:58 || timer: 0.358 [ 18] 13430 || B: 4.454 | C: 2.556 | M: 4.536 | S: 0.093 | T: 11.640 || ETA: 3 days, 6:03:02 || timer: 0.355 [ 18] 13440 || B: 4.514 | C: 2.544 | M: 4.703 | S: 0.089 | T: 11.850 || ETA: 3 days, 6:03:04 || timer: 0.359 [ 18] 13450 || B: 4.237 | C: 2.526 | M: 4.542 | S: 0.086 | T: 11.391 || ETA: 3 days, 6:03:06 || timer: 0.358 [ 18] 13460 || B: 4.406 | C: 2.517 | M: 4.432 | S: 0.077 | T: 11.432 || ETA: 3 days, 6:03:03 || timer: 0.357 [ 18] 13470 || B: 4.534 | C: 2.518 | M: 4.628 | S: 0.056 | T: 11.737 || ETA: 3 days, 6:02:57 || timer: 0.356 [ 18] 13480 || B: 4.620 | C: 2.495 | M: 4.871 | S: 0.070 | T: 12.057 || ETA: 3 days, 6:02:55 || timer: 0.359 [ 18] 13490 || B: 4.506 | C: 2.489 | M: 4.917 | S: 0.131 | T: 12.042 || ETA: 3 days, 6:03:01 || timer: 0.357 [ 18] 13500 || B: 4.460 | C: 2.501 | M: 4.790 | S: 0.139 | T: 11.891 || ETA: 3 days, 6:02:55 || timer: 0.357 [ 18] 13510 || B: 4.200 | C: 2.473 | M: 4.525 | S: 0.137 | T: 11.335 || ETA: 3 days, 6:02:55 || timer: 0.356 [ 18] 13520 || B: 4.136 | C: 2.467 | M: 4.478 | S: 0.137 | T: 11.217 || ETA: 3 days, 6:02:52 || timer: 0.356 [ 18] 13530 || B: 4.161 | C: 2.479 | M: 4.594 | S: 0.154 | T: 11.387 || ETA: 3 days, 6:02:52 || timer: 0.358 [ 18] 13540 || B: 3.935 | C: 2.475 | M: 4.498 | S: 0.157 | T: 11.064 || ETA: 3 days, 6:02:52 || timer: 0.355 [ 18] 13550 || B: 4.013 | C: 2.479 | M: 4.456 | S: 0.160 | T: 11.107 || ETA: 3 days, 6:02:43 || timer: 0.354 [ 18] 13560 || B: 3.889 | C: 2.491 | M: 4.388 | S: 0.165 | T: 10.934 || ETA: 3 days, 6:02:35 || timer: 0.357 [ 18] 13570 || B: 3.705 | C: 2.480 | M: 4.153 | S: 0.190 | T: 10.529 || ETA: 3 days, 6:02:33 || timer: 0.356 [ 18] 13580 || B: 3.566 | C: 2.479 | M: 4.146 | S: 0.197 | T: 10.388 || ETA: 3 days, 6:02:29 || timer: 0.356 [ 18] 13590 || B: 3.570 | C: 2.484 | M: 4.070 | S: 0.154 | T: 10.278 || ETA: 3 days, 6:02:29 || timer: 0.355 [ 18] 13600 || B: 3.694 | C: 2.485 | M: 4.107 | S: 0.153 | T: 10.438 || ETA: 3 days, 6:02:29 || timer: 0.360 [ 18] 13610 || B: 3.701 | C: 2.491 | M: 4.319 | S: 0.154 | T: 10.665 || ETA: 3 days, 6:02:35 || timer: 0.358 [ 18] 13620 || B: 3.580 | C: 2.461 | M: 4.316 | S: 0.152 | T: 10.509 || ETA: 3 days, 6:02:32 || timer: 0.358 [ 18] 13630 || B: 3.632 | C: 2.447 | M: 4.437 | S: 0.131 | T: 10.647 || ETA: 3 days, 6:02:19 || timer: 0.354 [ 18] 13640 || B: 3.744 | C: 2.446 | M: 4.378 | S: 0.140 | T: 10.708 || ETA: 3 days, 6:02:26 || timer: 0.358 [ 18] 13650 || B: 3.661 | C: 2.444 | M: 4.271 | S: 0.146 | T: 10.523 || ETA: 3 days, 6:02:24 || timer: 0.358 [ 18] 13660 || B: 3.768 | C: 2.447 | M: 4.720 | S: 0.150 | T: 11.086 || ETA: 3 days, 6:02:28 || timer: 0.357 [ 18] 13670 || B: 4.226 | C: 2.485 | M: 5.642 | S: 0.129 | T: 12.481 || ETA: 3 days, 6:02:25 || timer: 0.354 /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [320,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [321,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [322,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [323,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [224,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [225,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [220,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [221,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [222,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [223,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [448,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [449,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [450,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [451,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [452,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [453,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [454,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [455,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [456,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [457,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [458,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [459,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [460,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [20,0,0], thread: [461,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [19,0,0], thread: [122,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [19,0,0], thread: [123,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [19,0,0], thread: [124,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [19,0,0], thread: [125,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [19,0,0], thread: [126,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [19,0,0], thread: [127,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [352,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [353,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [354,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [355,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [356,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [357,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [358,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [359,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [360,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [361,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [362,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [21,0,0], thread: [363,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [82,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [83,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [84,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [85,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [86,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [87,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [88,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [89,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [90,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [91,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [92,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [93,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [94,0,0] Assertion *input >= 0. && *input <= 1. failed. /pytorch/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [18,0,0], thread: [95,0,0] Assertion *input >= 0. && *input <= 1. failed. THCudaCheck FAIL file=/pytorch/aten/src/THC/THCReduceAll.cuh line=327 error=59 : device-side assert triggered Traceback (most recent call last): File “train.py”, line 382, in <module> train() File “train.py”, line 257, in train losses = criterion(out, wrapper, wrapper.make_mask()) File “/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 547, in call result = self.forward(*input, **kwargs) File “/usr/local/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 150, in forward return self.module(*inputs[0], **kwargs[0]) File “/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 547, in call result = self.forward(*input, **kwargs) File “/home/administrator/Sheet_detection/yoloact_new/yolact/layers/modules/multibox_loss.py”, line 180, in forward losses[‘C’] = self.ohem_conf_loss(conf_data, conf_t, pos, batch_size) File “/home/administrator/Sheet_detection/yoloact_new/yolact/layers/modules/multibox_loss.py”, line 243, in ohem_conf_loss loss_c = log_sum_exp(batch_conf) - batch_conf[:, 0] File “/home/administrator/Sheet_detection/yoloact_new/yolact/layers/box_utils.py”, line 279, in log_sum_exp x_max = x.data.max() RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/THCReduceAll.cuh:327

Issue Analytics

State:
Created 4 years ago
Comments:9 (3 by maintainers)

Top GitHub Comments

1reaction

dbolyacommented, Nov 13, 2019

@abhigoku10 Unless you’re experiencing the same issue as him, I don’t think it matters much. I’ve already said this in the thread he made about it, but I think his particular issue is instability (specifically w.r.t. batch norm) because of a batch size of 1. If you’re using standard batch sizes, even on custom datasets I don’t think this should be an issue.

0reactions

cyrilzakkacommented, Jan 15, 2021

I got this same issue as well! If I understand it correctly, I should increase batch size from 1 to 2?

[01/15 17:00:38 yolact.train]: eta: 6 days, 1:37:48  epoch: 0  iter: 780  B: 1098.858  M: 7.907  C: 461.627  S: 6.542  T: 1574.935  time: 0.671  data_time: 0.004  lr: 0.001000  max_mem: 9152M
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
Warning: Moving average ignored a value of inf
Warning: Moving average ignored a value of nan
[01/15 17:00:44 yolact.train]: eta: 6 days, 1:32:08  epoch: 0  iter: 790  B: 139819971895902803613112598528.000  M: 46.037  C: 15536841114220526695316717568.000  S: 119668369.416  T: 155356813010123321512336293888.000  time: 0.642  data_time: 0.003  lr: 0.001000  max_mem: 9152M
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [0,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [1,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [2,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [3,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [4,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [5,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [6,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [7,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [8,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [9,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [10,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [11,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [12,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [13,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [14,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [15,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [16,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [17,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [18,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [19,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [20,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [21,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [22,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [23,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [24,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [25,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [26,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [27,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [28,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [29,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [30,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [71,0,0], thread: [31,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [0,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [1,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [2,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [3,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [4,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [5,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [6,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [7,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [8,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [9,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [10,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [11,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [12,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [13,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [14,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [15,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [16,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [17,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [18,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [19,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [20,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [21,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [22,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [23,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [24,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [25,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [26,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [27,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [28,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [29,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [30,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [31,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [32,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [33,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [34,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [35,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [36,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [37,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [38,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [39,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [40,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [41,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [42,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [43,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [44,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [45,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [46,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [47,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [48,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [49,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [50,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [51,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [52,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [53,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [54,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [55,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [56,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [57,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [58,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [59,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [60,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [61,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [62,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [63,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [32,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [33,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [34,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [35,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [36,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [37,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [38,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [39,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [40,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [41,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [42,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [43,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [44,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [45,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [46,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [47,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [48,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [49,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [50,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [51,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [52,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [53,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [54,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [55,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [56,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [57,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [58,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [59,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [60,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [61,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [62,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [123,0,0], thread: [63,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [32,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [33,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [34,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [35,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [36,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [37,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [38,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [39,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [40,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [41,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [42,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [43,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [44,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [45,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [46,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [47,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [48,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [49,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [50,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [51,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [52,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [53,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [54,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [55,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [56,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [57,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [58,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [59,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [60,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [61,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [62,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [58,0,0], thread: [63,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [32,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [33,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [34,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [35,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [36,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [37,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [38,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [39,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [40,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [41,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [42,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [43,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [44,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [45,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [46,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [47,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [48,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [49,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [50,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [51,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [52,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [53,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [54,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [55,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [56,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [57,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [58,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [59,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [60,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [61,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [62,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [97,0,0], thread: [63,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [0,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [1,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [2,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [3,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [4,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [5,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [6,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [7,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [8,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [9,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [10,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [11,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [12,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [13,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [14,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [15,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [16,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [17,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [18,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [19,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [20,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [21,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [22,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [23,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [24,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [25,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [26,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [27,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [28,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [29,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [30,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [84,0,0], thread: [31,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [32,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [33,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [34,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [35,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [36,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [37,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [38,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [39,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [40,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [41,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [42,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [43,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [44,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [45,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [46,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [47,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [48,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [49,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [50,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [51,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [52,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [53,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [54,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [55,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [56,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [57,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [58,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [59,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [60,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [61,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [62,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [110,0,0], thread: [63,0,0] Assertion `input_val >= zero && input_val <= one` failed.
Traceback (most recent call last):
  File "train.py", line 744, in <module>
    train(0, args=args)
  File "train.py", line 459, in train
    losses = criterion(out, wrapper, wrapper.make_mask())
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/yolact_edge/layers/modules/multibox_loss.py", line 180, in forward
    losses['C'] = self.ohem_conf_loss(conf_data, conf_t, pos, batch_size)
  File "/content/yolact_edge/layers/modules/multibox_loss.py", line 247, in ohem_conf_loss
    loss_c[pos]        = 0 # filter out pos boxes
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fe016d448b2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fe016f96952 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fe016d2fb7d in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #3: std::vector<c10d::Reducer::Bucket, std::allocator<c10d::Reducer::Bucket> >::~vector() + 0x312 (0x7fe06183dc12 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #4: c10d::Reducer::~Reducer() + 0x2da (0x7fe06183c24a in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x12 (0x7fe06180bf42 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7fe0611e3f16 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x8deefb (0x7fe06180cefb in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x2c1e80 (0x7fe0611efe80 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x2c30ce (0x7fe0611f10ce in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #10: python3() [0x572cb7]
frame #11: python3() [0x54f122]
frame #12: python3() [0x588948]
frame #13: python3() [0x5ad418]
frame #14: python3() [0x5ad42e]
frame #15: python3() [0x56b4c6]
<omitting python frames>
frame #21: __libc_start_main + 0xe7 (0x7fe065a03bf7 in /lib/x86_64-linux-gnu/libc.so.6)