Lightning support in EPS

Lightning is coming to Electrum soon. Also eclair wallet uses the Electrum protocol.

Obviously EPS should aim to support these.

Here are some of my notes and IRC logs on the topic:

They’re a bit unreadable because of the long lines, probably the easiest way to read is to copypaste them into a text editor and enable line wrap

as of 20/10/2019 the electrum master branch contains LN code

electrum wallet connects directly to the LN p2p network
electrum uses servers to query addresses to check
    whether a channel is still open
so in theory needs indexing of all addresses on the blockchain

one way EPS can implement this is to also connect directly to the LN p2p
    network and download all the channels
then (assuming the blockchain is unpruned) can obtain the actual transactions
    and addresses of the channels
 using importprunedfunds
and by checking the utxo set can determine whether the channels are still open
    (unspent) blockchain needs to be unpruned only for this initial sync and small
    amounts unpruned for catching up for when EPS restarts
this shouldnt need rescanning

if a channel UTXO is spent we wont be able to find this spending TX
however maybe we could just return an empty result to the client (i.e. make out
    that the channel never existed)

seems like a good idea to have a seperate wallet.dat file for the LN channels
although presumably the address_history dict will contain both LN and user's
   on-chain funds
 perhaps with an extra field saying whether the address is LN or user (or both)

worth checking the limits of bitcoin core's internal wallet
 that determines how big the list of LN channels could grow
i think gmaxwell wrote somewhere on the dev ml how it should handle up to
    100k addresses or so?
later edit: core can easily monitor at least 1 million addresses (although the
    wallet file gets big)


if the EPS runs constantly, then that same machine is a great place for a user
    to run a private watchtower





= later =
3/12/2019

its worth just investigating what exactly electrum does
requiring EPS to also connect to the LN p2p network is subpar, ideally it would
   only connect to the full node and electrum wallet

querying the utxo set would be ideal to check for unspentness
 that would require a new method in the electrum protocol

for checking whether electrum's own channels have been closed by the other
  side, perhaps importing the channel addresses as watch-only

one option is to create a command line option for electrum which tells it to
  send newly-created channel addresses to the server, and then EPS can
  import those addresses


= =
7/12/2019
tried using electrum LN
 the channel address is indeed queried by the client to the server, but its in
  scripthash form one idea could be to have a new server method
   say `blockchain.script.notify` which gives an address or scriptPubKey, this
  message is sent before the client broadcasts the transaction
although another idea is to just get this address from the broadcasted
   transaction, because that will go through the server too

that would mean the server must keep some persistent state to remember these
   channel addresses
actually if the only thing needed to be stored is the channel address, then that
  can be stored in the bitcoin core wallet file and accessed with getaddressgroupings
  or whatever that RPC is called which returns imported addresses

that would also mean it wouldnt be possible to switch from a regular public server
   to a personal server
 because the personal server wont know your channel addresses
 a remedy there might be to require the user to add all open channels TXIDs and
   block heights or channel address to the config file
 in this it should be made clear that this is only needed for already-open channels in
  the initial move to EPS
 this could be made a bit easier by having a gui option for electrum to export all open channel
  addresses in some kind of serialization
 right now its nontrival in the Electrum GUI to obtain the user's channel addresses

importing already-existing channel addresses into EPS is the best way, better than TXIDs/heights because
 that needs seperate rescanning
importing channels addresses needs a rescan but that rescan can be done at the same time as rescanning
 all the other addresses for regular on-chain addresses

and all this is only needed for importing existing channels
 if new channels are created while connected to EPS then EPS itself can figure out the channel address
 by looking at the broadcast transaction
 it can import every address which is a p2wsh address in each transaction
can confirm that the broadcast RPC happens before the subscribe RPC, so importing-on-broadcast will work
question is, do electrum devs ever plan to support incoming channels, i.e. channels which someone else
 opens by connecting to our client, id guess no
  actually yes, because how are people meant to buy incoming liquidity like from bitrefill's thor
 if so then EPS wouldnt have access to the transaction
dont import addresses if they're already in the address_history dict

another possibility with the import-on-broadcast trick is to note that a `blockchain.scripthash.subscribe`
 arrives immediately after the broadcast, but i dont think that will be useful since when it arrives its
 already too late to import

this would mean that people would be a little locked onto EPS, if they use it they shouldnt
 go back to public servers, or else they'll need to import channels again
 but thats probably fine


iv seen one random 'blockchain.scripthash.subscribe' in the log which wasnt correlated with anything i did


a whole other issue is sync'ing the LN routing table
electrum connects to the LN p2p network and downloads all the public channels
presumably it queries the server to check stuff about them, but i havent seen anything related in the log

after searching the source code a bit
iv found the file lnverifier.py which has verify_channel()
 which from the channel announcement queries the server for the channel opening txid
 EPS should be able to handle all this
 oddly the client only checks that the TXID exists, not that the output is still unspent
 that would be easy to check by querying the UTXO set, admittedly theres no electrum server RPC to
  actually do this

it seems right now electrum doesn't verify the LN routing table at all
 so doesnt query EPS at all

= =
9/12/2019
i deleted the electrum testnet dir and made it re-sync all the LN channels again
but nothing appears in stdout, does it even verify the channels?

ill try pointing it to my EPS and see what happens

the question of other side funded channels is still unanswered


<belcher> i think i found a way to make LN work on EPS, which is that when the channel opening tx is broadcast it is broadcast through EPS, and EPS can obtain the channel address from that and then be able to let the client subscribe to it
<ghost43> or maybe we could centrally host a zip with merkle proofs and funding txns on a fast dedicated server? :P
<belcher> and if the user is moving from a public server to EPS, they need to add their already-open channels to EPS's config files like they already do with mpks
<ThomasV_> belcher: there are more addresses to watch than just the channel funding address
<belcher> i know, but watching addresses isnt very expensive for core
<belcher> oh wait i misread
<ghost43> if you subscribe to everything 2 levels down from the funding, that should be enough
<belcher> which other addresses need to be watched?
<ThomasV_> their outputs, 2 levels down
<ghost43> funding outpoint; that is spent by commitment tx (so outputs of that); which is e.g. spent by htlc tx (so outputs of that)
<belcher> is that in the case of the channel being force-closed? so the addresses that need to be subscribed to are the OP_CTLV and OP_HASH ones ?
<belcher> that makes sense, i havent played around with force closing channels yet
* KiloLima (~[e-mail]) has joined
<belcher> another thing, does electrum LN support opening channels where the other side funds them? for example this is used in bitrefill's thor channel product
<ThomasV_> belcher: the command line support --push
<ThomasV_> supports
<ThomasV_> belcher: it's not just about force-closed cases, it is about attacks
<belcher> yes, i used bad terminology i think (an attack is essentially a force close but with an earlier state)
<ghost43> belcher: yes, the other party can also be the channel initiator
<belcher> then obtaining the channel address from the broadcasted transaction wont always work, because presumably in that case the other party broadcasts instead of electrum
<ghost43> indeed
<belcher> perhaps EPS could search the mempool or recent blocks to find that address 
<belcher> btw what was meant by the --push command line option ? i dont see it in electrum's source code
<ghost43> open_channel --push_amount   see https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#the-open_channel-message


so need to study the channel closing

also try to figure out the best way to extract addresses from the mempool or recent blocks
 and a way to pass that to the client in a graceful way
 perhaps return None to start with and then scan, then can send update to client when found

long term a solution to searching the mempool could be to code a new RPC which searches
 for a scriptpubkey which is in the mempool right now
 since the alternative is dumping the entire mempool over RPC and searching it: slow af
although its worth testing, after all that bug in Core was supposedly fixed
 i should make a script which generates a big mempool in regtest

then theres the complication that you dont know for sure whether a scripthash subscription
 request is actually from LN, it could just be from a brand new wallet that the user just opened
 to make it harder, that brand new wallet could be of a single address
note that importprunedfunds asks for a txoutproof, so it cant work for mempool txes


now try to sync the LN channels with trust manually set to false, so that it uses EPS
 i have done it
 its pretty slow, although my server also prints out every request which is slow
 also i notice electrum is downloading all the block headers even though it has them

actually you CANT search the mempool to find a certain tx that has a certain output
maybe the best solution is to define a new stratum method which tells the channel address to the server

= =
11/12/2019
actually you can search the mempool, with getrawtransaction
although i expect it will be slow af


= =
11/01/2020

conversation with m-schmoock in #electrum

<m-schmoock> belcher: belcher_:  what would be required in order to make LN wallets (i.e. eclair) fully compatible with eps? currently it EPS prints and unknown address hash for each channel and eclair 'seems to be fine' with that. Can we think of adding LN channel script punkeys just-in-time when a 'known' user (that requested valid addresses before) asks for an yet unknown output?
<m-schmoock> in this case the knowledge of the correct pubkey chain would be the authenticator that allows adding of new outputs on the run
<belcher> yeah its not too hard conceptually, i discussed it here before
<m-schmoock> i have not been around here before sorry
<belcher> EPS needs to add the user's channel addresses as watch-only wallets and monitor them
<m-schmoock> yes i figured that
<belcher> if they get spent from (i.e. the channel is closing) then EPS needs to monitor that transaction too, so it can tell the client
<belcher> the only hard part is EPS learning the channel addresses in the first place, i've been thinking the best way is to add some code to electrum wallet which tells the server whenever a new channel address is created
<belcher> and if the EPS user is newly setting up the server then they add already-existing channel addresses to EPS's config.ini
<belcher> eclair seems fine with it but it needs to know about channel closings, so its still the wrong behaviour i believe
<belcher> (iv never used eclair, maybe they use watchtowers instead?)
<m-schmoock> hm, the eps could proactively guess the correct address by looking at the outputs of the wallet, right?
<belcher> yep but only for client-broadcasted transactions, theres also a mode where the other LN peer broadcasts, that happens if the other peer funds the channel (for example if you buy bitrefill's thor product to buy incoming liquidity)
<m-schmoock> belcher: should we make an issue on Github to discuss potential requirements?
<belcher> sure, sec
<m-schmoock> I would really like to use EPS with eclair
<m-schmoock> also I can help implementing it
<m-schmoock> (i work on clightning btw)
<belcher> cool
<belcher> iv got some notes on my hard drive, ill put them on github in a sec
<belcher> in my view the cleanest way is to have the clients (electrum and eclair) tell the server what their channel addresses are, electrum already has a command line flag --skipmerklecheck which is only used for EPS to allow pruning
<m-schmoock> nice, lets bring that feature on then...

[chris-belcher]

The client already tells the Electrum server its addresses if the addresses are used, so new electrum protocol calls won’t degrade privacy any more.

Option 4 seems much better than 3 because 3 won’t work with remote-funded channels.

If its helpful to code the feature in stages, then option 1 could be made first with the plan to create option 4 later. Some hardcore users might not mind editing config.ini and restarting EPS if it means they can use Lightning. (Although a complication I just thought of is Electrum’s GUI doesn’t display channel addresses last time I looked)

[chris-belcher]

One side-effect is that if your Electrum server is exposed to the internet, someone could DoS you by issuing blockchain.outpoint.subscribe requests for outputs belonging to a popular satoshidice-like addresses .

I’d argue that people should not expose their personal servers to the internet, for other reasons as well as this. The way to fix all these attacks is to somehow have a way for the server to authenticate the client.

Hmm, in the case where the txo was already spent when the client issues blockchain.outpoint.subscribe on it, there isn’t really an easy way to tell which transaction spent it, or distinguish whether its spent or non-existent, right?

Good catch. A plausible situation where this happens is if a user opens a lightning channel but sends with a low miner fee. They go to sleep and wake up the next morning expecting their channel to have confirmed overnight. However before they sync their Electrum wallet the other lightning peer has already closed their channel.

The client isn’t coded yet I think, but to me it seems the best way to solves this is having the client request blockchain.outpoint.subscribe immediately after broadcasting the funding transaction (and so before our hypothetical Electrum user from above turns off his Electrum and goes to bed), and that would give the chance for the server to import the channel address. Then if the channel is closed overnight the full node will still save the spending tx.