Session expired while scanning aws

Bug Template

Title: *neo4j.exceptions.SessionExpired: Failed to read from defunct connection IPv4Address((‘07be756b.databases.neo4j.io’, 7687)) *

Description: running cartography in ec2 launched instance to observe relationship of my aws resources

To Reproduce:

cartography --neo4j-uri neo4j+s://07be756b.databases.neo4j.io --neo4j-password-prompt --neo4j-user neo4j

Logs: Unhandled exception during sync stage 'aws'Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 57, in _yield_messages buffer.receive(sock, 2) File "/usr/local/lib/python3.7/site-packages/neo4j/packstream.py", line 484, in receive raise OSError("No data")OSError: No dataThe above exception was the direct cause of the following exception:Traceback (most recent call last): File "/home/ec2-user/cartography-ec2-dev/cartography/sync.py", line 86, in run stage_func(neo4j_session, config) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 244, in start_aws_ingestion requested_syncs, File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 168, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 63, in _sync_one_account RESOURCE_FUNCTIONS[func_name](**sync_args) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 118, in sync_ebs_volumes load_volume_relationships(neo4j_session, transformed_data, update_tag) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 95, in load_volume_relationships aws_update_tag=aws_update_tag, File "/home/ec2-user/cartography-ec2-dev/cartography/experimental_neo4j_4x_support.py", line 109, in wrapper return run_func(query, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/work/simple.py", line 218, in run self._bookmarks, **kwparameters File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 117, in _run self._attach() File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 206, in _attach self._connection.fetch_message() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 188, in inner func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_bolt4.py", line 283, in fetch_message details, summary_signature, summary_metadata = next(self.inbox) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 83, in __next__ tag, fields = self.pop() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 77, in pop return next(self._messages) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 74, in _yield_messages self.on_error(error) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 561, in _set_defunct_read self._set_defunct(message, error=error, silent=silent) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 606, in _set_defunct raise SessionExpired(message) from errorneo4j.exceptions.SessionExpired: Failed to read from defunct connection IPv4Address(('07be756b.databases.neo4j.io', 7687)) (IPv4Address(('34.126.171.25', 7687))) | Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 57, in _yield_messages buffer.receive(sock, 2) File "/usr/local/lib/python3.7/site-packages/neo4j/packstream.py", line 484, in receive raise OSError("No data") OSError: No data The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/ec2-user/cartography-ec2-dev/cartography/sync.py", line 86, in run stage_func(neo4j_session, config) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 244, in start_aws_ingestion requested_syncs, File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 168, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 63, in _sync_one_account RESOURCE_FUNCTIONS[func_name](**sync_args) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 118, in sync_ebs_volumes load_volume_relationships(neo4j_session, transformed_data, update_tag) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 95, in load_volume_relationships aws_update_tag=aws_update_tag, File "/home/ec2-user/cartography-ec2-dev/cartography/experimental_neo4j_4x_support.py", line 109, in wrapper return run_func(query, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/work/simple.py", line 218, in run self._bookmarks, **kwparameters File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 117, in _run self._attach() File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 206, in _attach self._connection.fetch_message() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 188, in inner func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_bolt4.py", line 283, in fetch_message details, summary_signature, summary_metadata = next(self.inbox) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 83, in __next__ tag, fields = self.pop() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 77, in pop return next(self._messages) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 74, in _yield_messages self.on_error(error) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 561, in _set_defunct_read self._set_defunct(message, error=error, silent=silent) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 606, in _set_defunct raise SessionExpired(message) from error neo4j.exceptions.SessionExpired: Failed to read from defunct connection IPv4Address(('07be756b.databases.neo4j.io', 7687)) (IPv4Address(('34.126.171.25', 7687)))

Screenshots:

If applicable, add screenshots to help explain your problem.

Please complete the following information::

• Cartography release version or commit hash [1e274e3e1e30e6558a688d70f4f274b5e78f8dc0]
• Python version: 2.7.18
• OS (EC2-Amazon Linux 2)

Additional context:

Add any other context about the problem here.

[achantavy]

In addition to contacting Neo4j support, you may want to play around with our --neo4j-max-connection-lifetime argument. This can help with timed out sessions especially over a load balancer. See https://github.com/lyft/cartography/blob/7122301f65ffe328552631b1b93427c37d11b6e9/cartography/cli.py#L92-L101 and https://neo4j.com/docs/api/python-driver/current/api.html#max-connection-lifetime-ref.

[kushankurdas]

I switched to self hosted version of neo4j https://neo4j.com/docs/operations-manual/current/installation/linux/debian/, worked for me since you can increase the memory limits while deploying container

# Raise memory limits:
- NEO4J_dbms_memory_pagecache_size=1G
- NEO4J_dbms.memory.heap.initial_size=1G
- NEO4J_dbms_memory_heap_max__size=1G