CupyMemoryProfileHook crashed
See original GitHub issue- Chainer version: 4.0.0
- CuPy version: 4.0.0
- OS/Platform: Windows 10.0.16299.431
- CUDA/cuDNN version: CUDA 9.1 w/o cuDNN
I want to use CupyMemoryProfileHook
to do some profiling in my network, however the process crashed without any error message when then trainer started. I tried again for several times and once it showed a message Exception in main training loop: Cannot convert int to cupy.cuda.memory._Chunk
without showing a tracing stack.
I tried to debug the Python process using VS and the calling stack shows
> python36.dll!update_refs(_gc_head * containers) 行 346 C
python36.dll!collect(int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) 行 956 C
python36.dll!collect_with_callback(int generation) 行 1129 C
python36.dll!collect_generations() 行 1151 C
[内联框架] python36.dll!_PyObject_GC_Alloc(int) 行 1726 C
[内联框架] python36.dll!_PyObject_GC_Malloc(unsigned __int64) 行 1736 C
python36.dll!PyType_GenericAlloc(_typeobject * type, __int64 nitems) 行 936 C
python36.dll!make_new_set(_typeobject * type, _object * iterable) 行 1050 C
util.cp36-win_amd64.pyd!00007ffda4d715e7() 未知
util.cp36-win_amd64.pyd!00007ffda4d72d83() 未知
util.cp36-win_amd64.pyd!00007ffda4d74700() 未知
core.cp36-win_amd64.pyd!00007ffd78313a02() 未知
core.cp36-win_amd64.pyd!00007ffd78313b19() 未知
core.cp36-win_amd64.pyd!00007ffd78223e2f() 未知
core.cp36-win_amd64.pyd!00007ffd782bbb1f() 未知
core.cp36-win_amd64.pyd!00007ffd782e4290() 未知
I’m using the hook with following code:
with chainer.function_hooks.CupyMemoryProfileHook() as hook:
trainer.run()
hook.print_report()
Besides, TimerHook
is usable in my network.
My network is quite complex that I cannot summarize it into a single reproduce case. How can I provide more information for you to look into this issue?
Issue Analytics
- State:
- Created 5 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
cupy.cuda.MemoryHook — CuPy 11.3.0 documentation
MemoryHook is an callback object. Registered memory hooks are invoked before and after memory is allocated from GPU device, and memory is retrieved...
Read more >CuPy crashes on anything that just isn't memory allocation
My hasty search shows that this pattern shows up 3 times, all in cupy/cuda/nvrtc.pyx. According to the documentation, nvrtcCreateProgram() does ...
Read more >How To Avoid Performance Pitfalls in React with memo ...
You'll look at how different actions can trigger re-renders and how you can use Hooks and memoization to minimize expensive data calculations.
Read more >Possible fix for the Scripthook dilemma : r/PCRedDead - Reddit
Or where you hear the beeps from LML and it crashes. COPY (NOT CUT) the current .exe (the updated one) into the UPDATEEXE...
Read more >Your Jest Tests are Leaking Memory
For many test suites this isn't a problem because even if tests leak memory, the tests don't use enough memory to actually cause...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I used the latest cython and built cupy, now it works! Thanks for your efforts!
I just succeed to reproduce this case (on Linux) with Cython 0.27.3. The error disappeared with Cython 0.28.0. If possible, could you also try updating Cython, then installing cupy from source? (cupy wheels are built with Cython 0.27.3)