Speed benefit of upcoming sparse graph importer

 * JGraphT version: 1.3
 * Java version (java -version)/platform:  13
**Issue**
The bottleneck in JGraphT 1.3 when constructing graphs with hundreds of millions of edges is the single threaded nature of the edge construction process. It can take an hour on a large cluster to build a billion edge graph.

**Solution**
The new sparse graph format and edge list importer (#829) has the potential to resolve this issue as follows:
(1) Java parallel streams (outside JGraphT) can be used to create integer pairs as edges, collected into a raw edge list using methods of the Java Collectors class.
(2) The new importer should be able to import the entire edge list at once, instead of the "one edge at a time" approach in JGraphT 1.3.
(3) The RAM needed in JGraphT for a billion edge graph is about 300Gb. This should come down by an order of magnitude with the sparse graph class, making such graphs constructible on ordinary workstations.

**Other information**
This "issue" is not a complaint, but rather a boost to D. Michail and others who are bringing this vision to fruition.

[probabilist-us]

This is on my agenda for January 2020, Dimitrios. I aim to build some large graphs using SparseIntDirectedGraph i nthe 1.3.2 snapshot on a multicore workstation, and report the results.

[probabilist-us]

P.S. When I say “always wins”, I mean it supplies the best vertex cover for my application, according to a machine learning criterion.

On Fri, Apr 17, 2020 at 12:13 PM R W R Darling [e-mail] wrote:

Dear Dimitrios,

Thanks so much for the reference to your blog post, and the code examples. Also the reference to https://arxiv.org/pdf/1904.08355.pdf which I will find useful in teaching.

Your sparse graph importer is expected to play a crucial role when I refactor my code, which was supposed to happen in the summer. Mainly it should cut down on RAM, when I am building big graphs.

Like everyone else, I am stuck at home for a while, doing various Maths Projects. There is a good chance I will start running sparse graph importers for fun.

Joris: I have run JGraphT *minimum weight vertex covers *hundreds of times, on sparse graphs with millions of vertices. You may be happy to know that your greedy implementation ALWAYS wins!

Best regards

Richard Darling

On Wed, Apr 15, 2020 at 5:12 AM Dimitrios Michail < [e-mail]> wrote:

Hi Richard! Any update on this? Did you find time to experiment with our sparse graph?

I also wrote a simple blog post on medium in order to increase visibility. You can find it here https://medium.com/p/using-sparse-graphs-in-jgrapht-1d517f654c05?source=email-3bcbe0a6fb93--writer.postDistributed&sk=2e3084d78e6b778a205d82cbcfb50ac5. It also discusses how to use GraalVM in order to build a native executable which should also provide some performance benefits.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jgrapht/jgrapht/issues/850#issuecomment-613918848, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADK3EIZGL3EMO5QH6NMP5RTRMV26JANCNFSM4KCY3QLA .