question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reuse tensordot memory tricks in matmul

See original GitHub issue

Currently the dask array matmul implemenation is similar to the tensordot implemenation (understandably) but lacks a clever trick that tensordot uses to avoid contractions in blockwise (which can result in large pileups of intermediate results).

https://github.com/dask/dask/blob/ff3ea6c74e8736a5823d648defcf9164f083d266/dask/array/routines.py#L290-L329

Tensordot uses only outer products in blockwise (there are no shared indices in the blockwise call) , and then uses a sum call to reduce these. Because sum uses tree reductions, this results in nicer memory usage. Matmul doesn’t use this trick.

I briefly tried replacing the blockwise call in matmul with a call to our internal dot function, but various tests failed. It would be good to do one of two things:

  1. Find a way to back matmul by tensordot, so that the reduction trick is shared with a common codebase
  2. Replicate the reduction trick in matmul

This was originally reported by @eric-czech and @tomwhite in https://github.com/pystatgen/sgkit/issues/375#issuecomment-731060672

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:11 (11 by maintainers)

github_iconTop GitHub Comments

1reaction
ravwojdylacommented, Jan 12, 2021

@jsignell yes. And if I may add, I believe before the release, #7050 should be fixed/closed as well.

0reactions
jsignellcommented, Jan 12, 2021

Was this closed by #7000?

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to matmul a 2d tensor with a 3d tensor in tensorflow?
Try using tf.tile to match the dimension of the matrix before multiplication. The automatic broadcast feature of numpy doesnt seem to be ...
Read more >
Tensordot — Multidimensional Dot Product — Explained
Tensordot from Scalars to Multidimensional Tensors​​ To get a full grasp of tensordots, it helps to start from simple matrices then slowly build ......
Read more >
Distributed-Memory Tensor Completion for Generalized Loss ...
Tensor computations are increasingly prevalent numerical techniques in data ... Tensor contractions can be reduced to matrix multiplication with matrices ...
Read more >
Investigating Tensors with PyTorch - DataCamp
In this tutorial, you'll learn about Tensors, PyTorch, and how to create a simple neural network with PyTorch.
Read more >
Efficient and Parallel Sparse Tensor-Matrix Multiplication
This paper introduces SPLATT, a C library with shared-memory parallelism for three-mode tensors. SPLATT contains algorithmic improvements ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found