question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Sparse covariance matrix fails

See original GitHub issue

Hi! I am reproducing the tutorial for counts clustering.

Requesting covariance matrix in full shape (sparse = False) mu, cov = jc.angular_cl.gaussian_cl_covariance_and_mean(cosmo, ell, probes, sparse=False)

returns error TypeError: _transpose() got an unexpected keyword argument 'axes' with this line from angular_cl.py: cov_mat = cov_mat.transpose(axes=(0, 2, 1, 3)).(n_ell * n_cls, n_ell * n_cls))

I changed the source line code to cov_mat = np.transpose(cov_mat, axes=(0, 2, 1, 3)).(n_ell * n_cls, n_ell * n_cls)) and the error disappeared. I don’t know, probably it is related to newer versions of jax.numpy.

The returned non-sparse matrix equals to jc.sparse.to_dense(cov_sparse).

Package versions are:

JAX version: 0.2.18
jax-cosmo version: 0.1rc7

By the way: I running the tutorial and the Cls calculations by jax cosmo is ~40 sec in opposition to CCL’s 0.2 sec for 1x1, 1x2 and 2x2 cross correlations. What I am doing wrong?

Thanks.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
EiffLcommented, Aug 19, 2021

Not a lame question 😃 That’s a great question Well, it has to do with how JAX works, when you jit a function, you “compile” it for a specific shape and type of input, and it also wants its inputs to be something JAX internally understands in terms of simple arrays. So, if your function takes as an input some sort of complicated object that doesn’t have a translation into a simple representation JAX understands, it will fail. But it is fine to use that function inside of a larger function, with simple input/outputs. It also takes time to jit a function, so for instance, if you want to apply your cl function to various arrays of \ell, of different sizes, JAX would rejit the code everytime, because everytime your input arrays are of different sizes, it looks to JAX as a different function.

Looong story short, I think the best practice is to leave it to the end user to decide where and what to jit, so there is minimal or no jitting at all in the library itself.

In practice, imagine you want to run an MCMC, you would only jit the entire function that computes the likelihood for a given set of cosmological parameters.

0reactions
SergeiDBykovcommented, Aug 23, 2021

Hi @EiffL I also noted a typo in likelihood module. Instead of include_logdet you should use ignore_logdet as a name of a parameter in the function gaussian_log_likelihood.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Sparse Covariance Matrix Estimation With Eigenvalue ... - NCBI
We propose a new approach for estimating high-dimensional, positive-definite covariance matrices. Our method extends the generalized thresholding operator ...
Read more >
Sparse Covariance Matrix Estimation with ... - Georgia Tech ISyE
Abstract: We propose a new approach for estimating high dimensional positive-definite covariance matrices. Our method extends the generalized thresholding ...
Read more >
Sparse estimation of a covariance matrix - JSTOR
We suggest a method for estimating a covariance matrix on the basis of a sample of vectors drawn from a multivariate normal distribution....
Read more >
Sparse covariance matrix estimation in high-dimensional ...
In this high-dimensional semiparametric deconvolution problem, we propose spectral thresholding estimators that are adaptive to the sparsity of Σ. We establish ...
Read more >
Using sparse inverse covariance matrix in estimating least ...
Bias-Variance Tradeoff · where · Hence we only have an unbiased estimator if W=(XTX)−1, ie. · Considering the optimization problem of the Glasso...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found