Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

SyncBN?

See original GitHub issue

Hi @HobbitLong

I see you use SyncBN from apex to train with DataParallel, however, SyncBN seems to be designed with DistributedDataParallel. Could you please confirm if SyncBN works in this case?

Best, Jizong

Issue Analytics

State:
Created 3 years ago
Comments:6 (1 by maintainers)

Top GitHub Comments

3reactions

HobbitLongcommented, Jul 2, 2020

@jizongFox, I did not realize it. Maybe you are right then that the code probably only used normal BN.

0reactions

zcy5417commented, Dec 6, 2021

作者你这也太坑了

??? what are you talking about?

我说作者坑啊，又没说你

SyncBN可能都没起到作用，论文的数据不就是有问题的吗

Top Results From Across the Web

SyncBN Explained | Papers With Code

Synchronized Batch Normalization (SyncBN) is a type of batch normalization used for multi-GPU training. Standard batch normalization only normalizes the ...

pytorch-syncbn - GitHub

This is an alternative implementation of "Synchronized Multi-GPU Batch Normalization" which computes global stats across GPUs instead of locally computed.

SyncBatchNorm — PyTorch 1.13 documentation

Applies Batch Normalization over a N-Dimensional input (a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper Batch ...

Implementing Synchronized Multi-GPU Batch Normalization

In this tutorial, we discuss the implementation detail of Multi-GPU Batch Normalization (BN) (classic implementation: encoding.nn.BatchNorm2d . We will provide ...

SyncBN layer for data parallelism - IDRIS

The SyncBN layers enable hardware synchronisation (during the data parallelisation) for the calculation of normalization factors.