CPU overuse caused by errors in LND and NBXplorer

Environment:

• Lunanode M2
• pruned 60GB
• LND 0.5.2
• BTCPayServer v1.0.3.58

Problem:

CPU overuse in LND and Bitcoind and strange errors in nbxplorer. Server does not crash, but mostly because I have CPU utilization enabled on my VM. But it works noticeably slower and I am unable to operate my Lightning node normally as connection is not always possible to BTCPay.

Notes

I do not have problems on the same setup with c-lightning.

First errors in BTCPay

fail: PayServer:      Error while connecting to WebSocket of NBXplorer (BTC)
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
   at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, Boolean throwOnPrematureClosure)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
   at NBXplorer.WebsocketMessageListener.NextMessageAsync(CancellationToken cancellation)
   at BTCPayServer.Payments.Bitcoin.NBXplorerListener.Listen(BTCPayWallet wallet) in /source/Payments/Bitcoin/NBXplorerListener.cs:line 134

Second Error in BTCPay

fail: PayServer:      BTC (Lightning): Can't connect to the lightning server
System.Net.Http.HttpRequestException: Connection refused ---> System.Net.Sockets.SocketException: Connection refused
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at BTCPayServer.Lightning.LND.LndSwaggerClient.LookupInvoiceAsync(String r_hash_str, Byte[] r_hash, CancellationToken cancellationToken)
   at BTCPayServer.Lightning.LND.LndClient.BTCPayServer.Lightning.ILightningClient.GetInvoice(String invoiceId, CancellationToken cancellation)
   at BTCPayServer.Payments.Lightning.LightningListener.EnsureListening(String invoiceId, Boolean poll) in /source/Payments/Lightning/LightningListener.cs:line 105

nbxplorer

fail: Configuration:  BTC: Error connecting to RPC server Connection refused Connection refused
fail: Configuration:  BTC: Unhandled in Waiter loop
NBXplorer.Configuration.ConfigException
   at NBXplorer.Configuration.RPCArgs.TestRPCAsync(NBXplorerNetwork networkInfo, RPCClient rpcClient, CancellationToken cancellation) in /source/NBXplorer/Configuration/RPCArgs.cs:line 134
   at NBXplorer.BitcoinDWaiter.StepAsync(CancellationToken token) in /source/NBXplorer/BitcoinDWaiter.cs:line 259
   at NBXplorer.BitcoinDWaiter.StartLoop(CancellationToken token, AutoResetEvent tick) in /source/NBXplorer/BitcoinDWaiter.cs:line 197
info: Configuration:  BTC: Testing RPC connection to http://bitcoind:43782/
fail: Configuration:  BTC: Error connecting to RPC server Connection refused Connection refused
fail: Configuration:  BTC: Unhandled in Waiter loop
NBXplorer.Configuration.ConfigException
   at NBXplorer.Configuration.RPCArgs.TestRPCAsync(NBXplorerNetwork networkInfo, RPCClient rpcClient, CancellationToken cancellation) in /source/NBXplorer/Configuration/RPCArgs.cs:line 134
   at NBXplorer.BitcoinDWaiter.StepAsync(CancellationToken token) in /source/NBXplorer/BitcoinDWaiter.cs:line 259
   at NBXplorer.BitcoinDWaiter.StartLoop(CancellationToken token, AutoResetEvent tick) in /source/NBXplorer/BitcoinDWaiter.cs:line 197

ngnix the only error log

2019/02/21 13:25:17 [error] 9#9: *156 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 123.81.123.171, server: myserver.com, request: "POST /lnrpc.Lightning/GetInfo HTTP/2.0", upstream: "grpcs://172.123.23.5:10009", host: "myserver.com

LND 0.52 logs

Pretty much nothing, the only errors I see here and there is below, this and that seems pretty standard.

2019-02-21 18:07:40.183 [ERR] CMGR: Can't accept connection: unable to accept connection from 92.53.65.100:54230: EOF

and

2019-02-21 18:52:07.109 [ERR] CRTR: Attempt to send payment 6dbe2c2f299d76a668be9697910e13e3d62a132c2d1257bb483ffff7b2d561b9 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate)(0xc002a0e210)({

Really nothing odd in LND logs.

bitcoin:0.17.0

No errors, here but it’s important to mention that my node has:

2019-02-21T13:11:05Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1

postgres logs

LOG:  database system was shut down at 2019-02-21 07:16:12 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
LOG:  using stale statistics instead of current ones because stats collector is not responding

Docker Stats for LND

CONTAINER ID        NAME                                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
2c271c938227        letsencrypt-nginx-proxy-companion   0.07%               6.371MiB / 1.947GiB   0.32%               2.39kB / 0B         281MB / 4.1kB       11
211b3c1dd70a        nginx-gen                           0.09%               2.227MiB / 1.947GiB   0.11%               2.39kB / 0B         127MB / 4.1kB       7
ab0044fa253d        generated_btcpayserver_1            0.12%               141.2MiB / 1.947GiB   7.08%               87.9MB / 56.3MB     1.48GB / 3.99MB     27
3ad8eb2a8187        nginx                               0.00%               2.469MiB / 1.947GiB   0.12%               66.3MB / 70.4MB     92.3MB / 0B         2
cb1fa751af73        btcpayserver_lnd_bitcoin            49.07%              564.6MiB / 1.947GiB   28.33%              217GB / 2.67GB      3.26GB / 2.98GB     43
31dace948ead        btcpayserver_bitcoind               17.28%              370.3MiB / 1.947GiB   18.57%              1.11GB / 217GB      43.8GB / 3.16GB     15
5cf361dfbd8e        generated_postgres_1                0.00%               6.492MiB / 1.947GiB   0.33%               7.51MB / 14MB       636MB / 31.8MB      8
b3f2102553f5        generated_nbxplorer_1               0.36%               22.5MiB / 1.947GiB    1.13%               216MB / 38.3MB      1.93GB / 14.2MB     21

c-lightning docker stats (for comparison purposes)

CONTAINER ID        NAME                                    CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
5b1878c5a6d8        letsencrypt-nginx-proxy-companion       0.23%               2.469MiB / 1.953GiB   0.12%               2kB / 0B            147MB / 4.1kB       11
32410b2fd1a5        nginx-gen                               0.33%               1.828MiB / 1.953GiB   0.09%               2kB / 0B            8.81MB / 0B         7
6316c76ea496        generated_btcpayserver_1                0.19%               179.6MiB / 1.953GiB   8.98%               81.4MB / 27.7MB     225MB / 4.87MB      23
b3363e5b6a40        generated_clightning_bitcoin_spark_1    0.00%               66.95MiB / 1.953GiB   3.35%               42.4MB / 14.8MB     972MB / 0B          10
7b3f887f62f6        generated_clightning_bitcoin_charge_1   0.01%               7.145MiB / 1.953GiB   0.36%               460kB / 4.55MB      990MB / 1.27MB      10
10541302e101        btcpayserver_clightning_bitcoin         0.00%               182.9MiB / 1.953GiB   9.14%               31.4GB / 14.6GB     7.55GB / 5.98GB     45
98e5fd520fbc        generated_nbxplorer_1                   0.47%               68.74MiB / 1.953GiB   3.44%               13.6GB / 2.48GB     13.9GB / 836MB      22
5d6be87d8d28        nginx                                   0.00%               1.52MiB / 1.953GiB    0.08%               503MB / 558MB       872MB / 0B          2
1ebe92ba5f24        btcpayserver_bitcoind                   0.71%               481.6MiB / 1.953GiB   24.08%              12.9GB / 33.3GB     984GB / 206GB       13
ed4ae8841943        generated_postgres_1                    0.00%               4.391MiB / 1.953GiB   0.22%               20.3MB / 23.9MB     1.75GB / 3.33GB     8

VM Logs

btcpay login: [  106.760528] cloud-init[2459]: Cloud-init v. 18.3-9-g2e62cb8a-0ubuntu1~18.04.2 running 'modules:final' at Wed, 26 Dec 2018 15:05:55 +0000. Up 106.51 seconds.
[  106.763347] cloud-init[2459]: Cloud-init v. 18.3-9-g2e62cb8a-0ubuntu1~18.04.2 finished at Wed, 26 Dec 2018 15:05:55 +0000. Datasource DataSourceOpenStack [net,ver=2].  Up 106.75 seconds
[4248747.118102] Out of memory: Kill process 6807 (lnd) score 522 or sacrifice child
[4248747.128923] Killed process 6807 (lnd) total-vm:2025436kB, anon-rss:1282292kB, file-rss:0kB, shmem-rss:0kB
[4249091.183184] Out of memory: Kill process 4105 (lnd) score 502 or sacrifice child
[4249091.184485] Killed process 4105 (lnd) total-vm:1847992kB, anon-rss:1544220kB, file-rss:0kB, shmem-rss:0kB
[4836729.665629] Out of memory: Kill process 4621 (lnd) score 396 or sacrifice child
[4836729.670545] Killed process 4621 (lnd) total-vm:1712280kB, anon-rss:998632kB, file-rss:0kB, shmem-rss:0kB

[NicolasDorier]

Probably fixed by https://github.com/lightningnetwork/lnd/pull/2690

[pavlenex]

Some follow up on this.

1. LND logs do not show anything weird.
2. LND still causes huge CPU usage.
3. LND does not apear to restart or crash when checking docker ps
4. The error in BTCPay I’m getting is:

fail: PayServer:      Error while connecting to WebSocket of NBXplorer (BTC)
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.ReadOnlyMemory`1..ctor(T[] array, Int32 start, Int32 length)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameFallbackAsync(MessageOpcode opcode, Boolean endOfMessage, ReadOnlyMemory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameFallbackAsync(MessageOpcode opcode, Boolean endOfMessage, ReadOnlyMemory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.HandleReceivedPingPongAsync(MessageHeader header)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
   at NBXplorer.WebsocketMessageListener.NextMessageAsync(CancellationToken cancellation)
   at BTCPayServer.Payments.Bitcoin.NBXplorerListener.Listen(BTCPayWallet wallet) in /source/Payments/Bitcoin/NBXplorerListener.cs:line 134

1. The error in BTCPay I’m getting in NBXplorer:

fail: Explorer:       BTC: Connection unexpectedly failed: Unexpected exception while connecting to socket
info: Events:         BTC: Node state changed: Ready => NotStarted
info: Configuration:  BTC: Testing RPC connection to http://bitcoind:43782/
fail: Configuration:  BTC: Error connecting to RPC server Connection refused Connection refused
fail: Configuration:  BTC: Unhandled in Waiter loop
NBXplorer.Configuration.ConfigException
   at NBXplorer.Configuration.RPCArgs.TestRPCAsync(NBXplorerNetwork networkInfo, RPCClient rpcClient, CancellationToken cancellation) in /source/NBXplorer/Configuration/RPCArgs.cs:line 134
   at NBXplorer.BitcoinDWaiter.StepAsync(CancellationToken token) in /source/NBXplorer/BitcoinDWaiter.cs:line 259
   at NBXplorer.BitcoinDWaiter.StartLoop(CancellationToken token, AutoResetEvent tick) in /source/NBXplorer/BitcoinDWaiter.cs:line 197
info: Configuration:  BTC: Testing RPC connection to http://bitcoind:43782/
fail: Configuration:  BTC: Error connecting to RPC server Connection refused Connection refused
fail: Configuration:  BTC: Unhandled in Waiter loop
NBXplorer.Configuration.ConfigException
   at NBXplorer.Configuration.RPCArgs.TestRPCAsync(NBXplorerNetwork networkInfo, RPCClient rpcClient, CancellationToken cancellation) in /source/NBXplorer/Configuration/RPCArgs.cs:line 134
   at NBXplorer.BitcoinDWaiter.StepAsync(CancellationToken token) in /source/NBXplorer/BitcoinDWaiter.cs:line 259
   at NBXplorer.BitcoinDWaiter.StartLoop(CancellationToken token, AutoResetEvent tick) in /source/NBXplorer/BitcoinDWaiter.cs:line 197
info: Configuration:  BTC: Testing RPC connection to http://bitcoind:43782/
fail: Configuration:  BTC: Error connecting to RPC server Connection refused Connection refused
fail: Configuration:  BTC: Unhandled in Waiter loop
NBXplorer.Configuration.ConfigException
   at NBXplorer.Configuration.RPCArgs.TestRPCAsync(NBXplorerNetwork networkInfo, RPCClient rpcClient, CancellationToken cancellation) in /source/NBXplorer/Configuration/RPCArgs.cs:line 134
   at NBXplorer.BitcoinDWaiter.StepAsync(CancellationToken token) in /source/NBXplorer/BitcoinDWaiter.cs:line 259
   at NBXplorer.BitcoinDWaiter.StartLoop(CancellationToken token, AutoResetEvent tick) in /source/NBXplorer/BitcoinDWaiter.cs:line 197

1. LND randomly appears to be available at the checkout, then sometimes it disappears. Really odd.