MemoryError at algorithms.big_clam

Describe the bug I was unable to run the algorithms.big_clam algorithm in a graph with 2436844 edges, receiving a MemoryError.

To Reproduce I’m instantiating my graph from a data frame with source, target and weight, and than trying to get the communities.

g = nx.from_pandas_edgelist(ds, ‘source’, ‘target’, ‘weight’) coms = algorithms.big_clam(g, 500)

• CDlib version cdlib-0.1.7
• Operating System mac os 10.12.5
• Python version 3.6
• Version(s) of CDlib required libraries

Additional context I’m using jupyter notebook

Exception trace

MemoryError Traceback (most recent call last) <ipython-input-123-1161752e0744> in <module> ----> 1 coms = algorithms.big_clam(g, 500)

/opt/conda/lib/python3.6/site-packages/cdlib/algorithms/overlapping_partition.py in big_clam(g, number_communities) 576 g = convert_graph_formats(g, nx.Graph) 577 –> 578 communities = BIGCLAM.big_Clam(g, number_communities) 579 580 return NodeClustering(communities, g, “BigClam”, method_parameters={“number_communities”: number_communities}, overlap=True)

/opt/conda/lib/python3.6/site-packages/cdlib/algorithms/internal/BIGCLAM.py in big_Clam(graph, number_communities) 76 77 def big_Clam(graph, number_communities): —> 78 adj = nx.to_numpy_matrix(graph) 79 F = train(adj, number_communities) 80 F_argmax = np.argmax(F, 1)

/opt/conda/lib/python3.6/site-packages/networkx/convert_matrix.py in to_numpy_matrix(G, nodelist, dtype, order, multigraph_weight, weight, nonedge) 446 return M 447 –> 448 449 def from_numpy_matrix(A, parallel_edges=False, create_using=None): 450 “”"Returns a graph from numpy matrix.

/opt/conda/lib/python3.6/site-packages/networkx/convert_matrix.py in to_numpy_array(G, nodelist, dtype, order, multigraph_weight, weight, nonedge) 1125 # This occurs when there are fewer desired nodes than 1126 # there are nodes in the graph: len(nodelist) < len(G) -> 1127 pass 1128 1129 A[np.isnan(A)] = nonedge

/opt/conda/lib/python3.6/site-packages/numpy/core/numeric.py in full(shape, fill_value, dtype, order) 300 shape : int or sequence of ints 301 Shape of the new array, e.g., (2, 3) or 2. –> 302 fill_value : scalar 303 Fill value. 304 dtype : data-type, optional

MemoryError:

[GiulioRossetti]

You are right my bad, I didn’t notice the transformation dependency. I have to check but, it is likely that this implementation of bigclam does not accept sparse scipy matrices… I’ll try to figure it out next week.

[Yquetzal]

Since we identified the problem, and we added a new implemention by integrating the karate club library, I close the issue.