Commit Graph

10778 Commits

Author SHA1 Message Date
Rusty Russell 6115ed02e8 subdaemons: don't stream gossip_store at all.
We now let gossipd do it.

This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 029d65cf2e connectd: serve gossip_store file for the peer.
We actually intercept the gossip_timestamp_filter, so the gossip_store
mechanism inside the per-peer daemon never kicks off for normal connections.

The gossipwith tool doesn't set OPT_GOSSIP_QUERIES, so it gets both, but
that only effects one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 1f608acd4e common: add routine for absolute timeouts (vs. relative).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell e37a638c0c connectd: do nagle by packet type.
channeld can't do it any more: it's using local sockets.  Connectd
can do it, and simply does it by type.

Amazingly, on my machine the timing change *always* caused
test_channel_receivable() to fail, due to a latent race.

Includes feedback from @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 7a514112ec connectd: do dev_disconnect logic.
As connectd handles more packets itself, or diverts them to/from gossipd,
it's the only place we can implement the dev_disconnect logic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 9c0bb444b7 per_peer_state: remove struct crypto_state
Now that connectd does the crypto, no need to hand around crypto_state.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell ce8b69401c peer_io: replace crypto_sync in daemons, use normal wire messages.
Now connectd is doing the crypto, we can use normal wire io.  We
create helper functions to clearly differentiate between "peer" comms
and intra-daemon comms though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell a2b3d335bb connectd: do decryption for peers.
We temporarily hack to sync_crypto_write/sync_crypto_read functions to
not do any crypto, and do it all in connectd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 71f736678f lightningd: make sure gossipd knows before we update blockheight.
Without this, we can get spurious failures from lnprototest, which
is waiting for lightningd to acknowledge the blockheight in getinfo:

```
2021-12-21T00:56:21.460Z DEBUG   lightningd: Adding block 109: 57a7bd3ade3a88a899e5b442075e9722ccec07e0205f9a913b304a3e2e038e26
2021-12-21T00:56:21.470Z DEBUG   lightningd: Adding block 110: 11a280eb76f588e92e20c39999be9d2baff016c3c6bac1837b649a270570b7dd
2021-12-21T00:56:21.479Z DEBUG   lightningd: Adding block 111: 02977fc9529b2ab4e0a805c4bc1bcfbff5a4e6577a8b31266341d22e204a1d27
2021-12-21T00:56:21.487Z DEBUG   lightningd: Adding block 112: 2402f31c5ddfc9e847e8bbfb7df084d29a5d5d936a4358c109f2f4cf9ea8d828
2021-12-21T00:56:21.496Z DEBUG   lightningd: Adding block 113: 5a561fe9423b4df33f004fc09985ee3ef38364d692a56a8b27ecbc6098a16d39
2021-12-21T00:56:21.505Z DEBUG   lightningd: Adding block 114: 4502f5ec23c89177872846848848322e8fa6c3fb6f5eb361194e4cd47596dfe9
2021-12-21T00:56:21.511Z DEBUG   02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9-gossipd: Ignoring future channel_announcment for 109x1x0 (current block 108)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6bfc3d8692 gossipwith: create our own internal sync_crypto functions.
This avoids changes to crypto_sync which are coming in next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell e683649004 connectd: maintain connection with peer, shuffle data.
Instead of passing the incoming socket to lightningd for the
subdaemon, create a new one and simply shuffle data between them,
keeping connectd in the loop.

For the moment, we don't decrypt at all, just shuffle.  This means our
buffer code is kind of a hack, but that goes away once we start
actually decrypting and understanding message boundaries.

This implementation is naive: it closes the socket to the local daemon
as soon as the peer closes the socket to us.  This is fixed in a
successive patch series (along with many other similar issues).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 8f3b390356 pytest: make test_channel_state_changed_unilateral more robust.
This test started mostly failing (in non-DEVELOPER mode) after the
next patch, due to timing issues.

Handle both cases for now, and we'll add more enhancements later to
things we should be handling more consistently.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 7e7a63a20d connectd: keep timeout timer around so we can disable it.
connectd will be keeping the conn open, so it needs to free this
"conn_timeout" timer.  Pass it through, so we can do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 562e6c4a93 update mocks
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Vincenzo Palazzo 35eaaf8e63 cli: change the redefine logic to read from the stdin
Procedure redefinition is a very cool feature but when other libraries redefine the same function can be very tricky to avoid compilation error.

This PR proposed a change of logic and use a customizzation function definition to read from the stdin, so we can avoid future error at compile time.

However, we could be check also the os or compiler that cause the error and redefine the missing function.

Changelog-None: Fixed compilation error due redefinition procedure.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-01-19 10:36:43 +01:00
johnongit 52b1f5a8f4 fix dockerfile 2022-01-11 12:00:30 +01:00
manreo 73bf4b2c2e Update lightning.py 2022-01-08 14:15:29 +01:00
manreo dded47e332 LightningRpc should accept pathlib Path
Not sure why socket does not accept pathlib, but just added some code to transform the path to string.
It is not the most pretty code, but better than these:
https://stackoverflow.com/questions/58647584/how-to-test-if-object-is-a-pathlib-path
2022-01-08 14:15:29 +01:00
Michael Schmoock 019b31c522 options: only allow one DNS announcement 2022-01-02 16:01:38 +01:00
Rusty Russell b659fbbdf7 pytest: further deflake test_funding_push.
I also got an error under CI; it seems the sleep() was insufficient.
So try adding a sleep inside the check_coin_moves, which should cover
everyone.

```
                acct_moves = acct_moves[number_moves:]
            else:
>               if not move_matches(m, acct_moves[0]):
E               IndexError: list index out of range
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell ae4669f77f common/coin_mvt: make it clear we're using Lightning HRP not BIP 173.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 425a7af512 common/coin_mvt: clean up API a little.
1. tal_strndup(.., str, strlen(str)) == tal_strdup()
2. tal_strdup also takes(), so document that.
3. Avoid passing 'struct sha256' on the stack: use ptr.
4. Generally, structures shouldn't keep pointers to things they don't own.
   In this case, mvt->node_id.
5. Make payment_hash a pointer, since NULL is more natural than an all-zero
   hash.

And add NON_NULL_ARGS() to the functions; it's cumbersome, but make it
fairly clear what params are optional.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 967ffbfbcb global: use tal_dup_or_null().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 8759641b06 ccan: update to get new helpers in ccan/tal
tal_dup_talarr() is simply stolen from common/utils, but tal_dup_or_null is new.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 5aa8663c98 common/socket_close.c: remove obsolete comment.
Put more inline comments, and also preserve errno properly from read.

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 4ef2367c04 connectd: rename struct peer in peer_exchange_initmsg to early_peer.
We want to have a real (persistent) struct peer eventually.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell d287d7d229 gossipd: make request handlers return void.
They all returned the next io_plan, but it was always the same.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 0aad222c2d gossipd: rewrite update generation, remove local_chan.
local_chan was mainly around so we could "soft" disable channels (and
really disable them once we used the channel_update in an error
message).

Instead we introduce the idea of a "deferred_update": it's either
deferred indefinitely (a peer goes offline, if we need to send it in
an error we'll apply it immediatly), or simply delayed to avoid
spamming everyone.

The resulting rewrite is much clearer, IMHO.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 8e37eb0028 gossipd: local_direction helper to generalize is_local_channel.
Increasingly we want to know is it local, and get the direction: it's
more efficient to do both at once.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 70ed47d77a channeld: add dev-disable-commit-after instead of dev-disconnect -nocommit
It was always a hack, but an impossible one once connectd will be
interpreting dev-disconnect!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 888745be16 dev_disconnect: remove @ marker.
Once connectd is doing this, we can't close as soon as we send,
and in fact we can't do 'fail write' either.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 560fa06f42 pytest: disable dev-disconnect blackhole tests.
These would have to be done by connectd, not the local daemon, once it's
intermediating.  Otherwise the remote peer won't see any change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 2fb58772a4 common/socket_close: don't wait forever.
The "read until closed" trick doesn't work if the other end doesn't
close (as found in the next patch, where we use DEV_DISCONNECT_DISABLE_AFTER).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 35e3c1866e common: generalize extract_channel_id().
connectd is going to end up using this do demux; make it fast and complete.

Fixing this reveals a problem in openingd: it now extracts the channel_id
from funding_signed (which is where we transition off the temporary), and
gets upset.  So fix that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 5111f39d2a connectd: clean up lightningd connection handling.
They all returned the same thing anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 6ce7e7ea60 common: is_msg_gossip_broadcast() for putting things in filter.
We don't need the other "gossip" messages in our echo-suppression filter.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 90b669857e lightningd: handle channel cleanups more explicitly.
1. Freeing an unconfirmed channel already releases the subd, so don't
   do that explicitly.
2. Use channel->owner to transfer ownership where possible, using
   channel_set_owner() which handles all the cases.

This simplifies the code and makes it more readable, IMHO.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 18526a3a5b lightningd: close one more fd for subdaemons.
Noticed by stracing for an unrelated problem.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Vincenzo Palazzo 9f02b6eb59 external: update lnprototest dependeces
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-12-28 09:49:09 +10:30
niftynei 70a73928cb balance-snaps: add a `balance_snapshot` event; fires after first catchup
Fire off a snapshot of current account balances (node wallet + every
'active' channel) after we've caught up to the chain tip for the *first*
time (in other words, on start).
2021-12-28 04:42:42 +10:30
niftynei f169597a02 test plugins/coin_moves: clean up/tidy coin moves plugin logic
removes unused in-memory stash of coin-moves; don't write the entire set
of coinmoves out on every update.
2021-12-28 04:42:42 +10:30
niftynei ec991e3af7 balance snap: first pass 2021-12-28 04:42:42 +10:30
niftynei 5570c6da08 coin_moves: cleanup tags list, update docs, remove unused method
We don't use journal_entry any more; remove it and update all of the
remaining tags (plus docs)
2021-12-28 04:42:42 +10:30
niftynei 8225a9decf coin_mvt: log events for pushes/lease_fees for leased channels
We need to stash/save the amount of the lease fees on a leased channel,
we do this by re-using the 'push' amount field on channel (which is
technically correct, since we're essentially pushing the fee amount to
the peer).

Also updates a bit of how the pushes are accounted for (pushed to now
has an event; their channel will open at zero but then they'll
immediately register a push event).

Leases fees are treated exactly the same as pushes, except labeled
differently.

Required adding a 'lease_fee' field to the inflights so we keep track of
the fee for the lease until the open happens.
2021-12-28 04:42:42 +10:30
niftynei bddd3694fa coin_mvt: record fees for an outbound htlc
If we initialized the payment, the fees are the entire fee-chain
(final hop amount - starting hop amount)

If it's a payment we routed, the fees are the diff between the
inbound htlc and the outbound (net gain by this routing)

Added to database so data persists nicely.
2021-12-28 04:42:42 +10:30
niftynei 29c6718297 coin_mvt: record new 'fees' field on htlc channel moves
We record the amount of fees collected for a routed payment. For
simplicity's sake on the data agg side, we record the fee payment on
*BOTH* the incoming htlc and the outgoing htlc. Note that this results
in double counting if you add up the fees from both an in-routed and
out-routed payment.
2021-12-28 04:42:42 +10:30
niftynei b6463174d6 coin moves: turn 'tag' into 'tags' array, add OPENER tag
Channels that the node has hopened will now be tagged with 'opener' in a
list of tags.
2021-12-28 04:42:42 +10:30
niftynei d3d6c09758 coin_moves: update some info about 'blockheight' 2021-12-28 04:42:42 +10:30
niftynei 4506f639fa coin_moves: remove 'index' for moves; bump version
Get rid of the 'movement_idx', since we replay events now.

Since we're removing a field from the 'coin_movement' event emission, we
bump the version type.

Changelog-Updated: `coin_movements` events have been revamped and are now on version 2.
2021-12-28 04:42:42 +10:30
niftynei 8098a4cd21 onchaind: remove 'is_replay' logic
we used this originally to suppress duplicate issuance of coin-move
events; we're assuming that any plugin expects duplicate events though
(and knows how to de-dupe them), so we no longer need this logic.
2021-12-28 04:42:42 +10:30