Commit Graph

1267 Commits

Author SHA1 Message Date
Rusty Russell cf55d40eca psbt: fix dual-funding memleak.
This happened occasionally in tests/test_opening.py::test_rbf_fails_to_broadcast:
we overwrote the witness stack without freeing the old one.

```
lightningd-2: 2022-03-18T02:23:32.113Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1: MEMLEAK: 0x559c375d9df8
lightningd-2: 2022-03-18T02:23:32.113Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   label=wally_tal
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   backtrace:
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     common/setup.c:13 (wally_tal)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/internal.c:285 (wally_calloc)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/transaction.c:234 (wally_tx_witness_stack_init_alloc)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/psbt.c:1119 (pull_psbt_input)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/psbt.c:1411 (wally_psbt_from_bytes)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     bitcoin/psbt.c:722 (psbt_from_bytes)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     bitcoin/psbt.c:753 (fromwire_wally_psbt)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend_wiregen.c:246 (fromwire_dualopend_reinit)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend.c:3855 (main)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   parents:
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     struct wally_psbt
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend.c:3804:struct state
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba932
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 10:19:50 +10:30
Rusty Russell 65be18d355 memleak: handle libwally allocations better.
Things allocated by libwally all get the tal_name "wally_tal",
which cost me a few hours trying to find a leak.

In the case where we're making one of the allocations the parent
of the others (e.g. a wally_psbt), we can do better: supply a name
for the tal_wally_end().

So I add a new tal_wally_end_onto() which does the standard
tal_steal() trick, and also changes the (typechecked!) name.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 10:19:50 +10:30
Rusty Russell 278ea6c8ac memleak: make sure we catch children which are also "notleak".
By not recursing into "notleak" children, we could miss other "notleak"
pointers:

```
2022-03-15T05:25:33.9759500Z lightningd-1: 2022-03-15T05:00:15.998Z **BROKEN** connectd: MEMLEAK: 0x55b29c3d0cc8
2022-03-15T05:25:33.9759901Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:   label=common/timeout.c:22:struct oneshot **NOTLEAK**
2022-03-15T05:25:33.9760195Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:   backtrace:
2022-03-15T05:25:33.9760555Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
2022-03-15T05:25:33.9760905Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     common/timeout.c:22 (new_reltimer_)
2022-03-15T05:25:33.9761272Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     connectd/multiplex.c:558 (set_closing_timer)
2022-03-15T05:25:33.9761647Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     connectd/multiplex.c:586 (write_to_peer)
2022-03-15T05:25:33.9761986Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     ccan/ccan/io/io.c:59 (next_plan)
2022-03-15T05:25:33.9762335Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/io.c:435 (io_do_always)
2022-03-15T05:25:33.9762692Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/poll.c:304 (handle_always)
2022-03-15T05:25:33.9763035Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/poll.c:385 (io_loop)
2022-03-15T05:25:33.9763376Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     connectd/connectd.c:2131 (main)
2022-03-15T05:25:33.9763645Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:   parents:
2022-03-15T05:25:33.9764020Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/io.c:91:struct io_conn **NOTLEAK**
2022-03-15T05:25:33.9764471Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     connectd/connectd.c:2104:struct daemon
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-20 08:42:57 +10:30
Rusty Russell 43a833e405 lightningd: remove support for legacy onion format.
As per proposal in https://github.com/lightning/bolts/pull/962

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
2022-03-18 09:20:11 +10:30
Ken Sedgwick 36466af3eb hsmd: Add fields to hsmd_sign_{,remote_}commitment_tx for validating signers 2022-03-16 12:06:21 +10:30
Michael Schmoock b930b8c548 wireaddr: adds wireaddr_eq_without_port and wireaddr_cmp_type
Adds wireaddr_eq_without_port so it can be used later.
Moves wireaddr_cmp_type from connectd.c to this file, so it can be reused later.
2022-03-11 16:42:45 +10:30
Ken Sedgwick 8f56f96801 hsmd: Add wallet index metadata to existing messages 2022-03-10 10:22:29 +10:30
Ken Sedgwick 8876c03791 hsmd: Add PSBT keypath utility functions 2022-03-10 10:22:29 +10:30
Ken Sedgwick 1e1948bfd2 hsmd: Add redeemscripts to anchor outputs 2022-03-10 10:22:29 +10:30
niftynei b33868f7c2 coin_mvt: only chain moves have a blockheight
channel moves don't have blockheights
2022-03-05 15:12:12 +10:30
niftynei 7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei 24a1c91045 coin_mvt: report the number of outputs on a channel close tx
The bookkeeper needs to know how many outputs to expect before we can
consider a channel resolved onchain.
2022-03-05 15:12:12 +10:30
Rusty Russell dd495b2b21 memleak: fix handling of excluded pointers.
We often hand an exclude pointer (usually the current command) to
memleak.  But when we encountered this we would stop iterating, rather
than just ignore it: this means we would often ignore significant siblings.

In particular, fixing this (which has always been there) reveals many
previously-undetected leaks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell ea71e9fe5b memleak: fix handling of notleak pointers.
We were ignoring the *parent* of the notleak pointers, not the notleak
pointer itself!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell fe86395e7c memleak: restore functionality accidentally removed.
Accidentally removed in 6c9b752751.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell 9bf2aa65cd common/param: don't keep around the temporary array off cmd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell 3218e5332f common/param: fix leak with opt_param_def.
We allocate the default, then callback allocates over the top.  Mark
params with a default, so we can free that when it's called.

(We can't do this generally, since not all param args are actually
pointers to pointers, though opt_param_def has to be).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell f0ea4d60b9 onchaind: fix minor leaks.
Not actually leaks, but they do live longer than they need.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell 4b7d2dc5b8 common/bolt11: make decoded routes hang off routes arr, not b11.
This causes weirdness in pay, which tal_steal's b11->routes and expects
to get it all.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
William Casarin 2b92ac4236 bolt11: mark when expiry is decoded
It looks like the x tag isn't marked when parsed either?
2022-02-22 09:58:48 +10:30
William Casarin af16b9b9f4 bolt11: mark when we decode min_final_cltv_expiry
It looks like decode_c doesn't set have_c unlike the other decode_
methods. At the start of the function, decode_c checks have_c to see if
it's set, but it is never set. It seems like this could allow for
duplicate c tags, which is probably not intended.

Signed-off-by: William Casarin <jb55@jb55.com>
2022-02-22 09:58:48 +10:30
Michael Schmoock 38e2abf68a peer_exchange: set, read and log remote_addr
Changelog-Added: Protocol: set remote_addr on init tlvs
2022-02-22 05:45:47 +10:30
Rusty Russell b47930f395 gossip_store: handle compacted gossip_store correctly.
Don't send EOF marker to peer, e.g. in tests/test_gossip.py::test_gossip_store_compact:

```
lightningd-2: 2022-01-24T03:34:22.925Z DEBUG   connectd: gossip_store at end, new fd moved to 1875
lightningd-2: 2022-01-24T03:34:22.933Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: Sending gossip INVALID 4105
lightningd-2: 2022-01-24T03:34:22.933Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: peer_in WIRE_WARNING
lightningd-2: 2022-01-24T03:34:22.941Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: peer_out INVALID 4105
lightningd-2: 2022-01-24T03:34:22.949Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: billboard perm: Received warning channel 2c7cf1dc9dada7ed14f10c78ade8f0de907c1b70e736c12ff6f7472dc69c3db3: Peer sent unknown message 4105 (INVALID 4105)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 3c5d27e3e9 subdaemons: remove gossipd fd from per-peer daemons.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell f14c0ef76a common: make sure we hand through peer for io logging.
This is mainly useful for connectd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
niftynei 43f1fb4561 coin_moves: an htlc_fulfill claimed by them is a credit to "external"
The money moved into an external account here.
2022-02-07 13:02:09 +10:30
niftynei 39572c1aaf coin_moves: remove unused method
This wasn't used anywhere, so we remove it.
2022-02-07 13:02:09 +10:30
Rusty Russell 8a8d7c4243 elements: unify overhead calculation.
And in particular, fix onchaind grinding code which used the
actual number of inputs and outputs (which already includes the
fee output); that breaks with the next patch which fixes other
calculations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-27 12:22:36 +01:00
niftynei 4dafeede5c coin moves: notify when we make deposits to external accounts
The blockheight is zero though, since these aren't included in a block
yet.

We also don't issue an 'external' deposit event if we can tell that the
address you're sending to actually belongs to our wallet (we'll issue a
deposit event when it gets included in a block)
2022-01-26 13:34:45 +10:30
niftynei 7f511b9ba9 coin moves: external deposits should be *deposits*
Report how much the deposit was for.
2022-01-26 13:34:45 +10:30
niftynei ff4ae8b5f4 coin_moves: add an 'originating_account' field
If a coin move concerns an external account, it's really useful to know
which 'internal' account initiated the transfer.

We're about to add a notification for withdrawals, so we can use this to
track wallet pushes to outside addresses

Changelog-Added: JSONRPC: `coin_movement` to 'external' accounts now include an 'originating_account' field
2022-01-26 13:34:45 +10:30
Rusty Russell 2b5d25c851 common: remove stderr debug in is_valid_witnessprog.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell be299c0d59 common/socket_close: remove now only connectd talks to peer.
connectd does this internally now using ccan/io, with appropriate
credit for ZmnSCPxj who wrote this code in the first place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell f6847f44f6 subds: remove "ignore error" from old LND nodes.
This was put in late 2019, and @t-bast says Eclair doesn't ignore their
errors and has had no issues.

It also conflicts with https://github.com/lightning/bolts/pull/932
which suggests you *should* fail when you receive an error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell 2ab5603624 peer subds: ignore failed writes.
In the case where the peer sends an error (and hangs up) immediately
after init, connectd *doesn't actually read the error* (even after all the
previous fixes so it actually receives the error!).

This is because to tried to first write WIRE_CHANNEL_REESTABLISH, and
that fails, so it never tries to read.  Generally, we should ignore
write failures; we'll find out if the socket is closed when we read
nothing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell d51fb5207a msg_queue: don't allow magic MSG_PASS_FD message for peers.
msg_queue was originally designed for inter-daemon comms, and so it has
a special mechanism to mark that we're trying to send an fd.  Unfortunately,
a peer could also send such a message, confusing us!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell a93c49ca65 connectd: implement @ correctly.
dev_blackhole_fd was a hack, and doesn't work well now we are async
(it worked for sync comms in per-peer daemons, but now we could sneak
through a read before we get to the next write).

So, make explicit flags and use them.  This is much easier now we
have all peer comms in one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 26b9384fd0 various: minor cleanups from Christian's review.
More significant things have been folded.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6d4c56e8b6 connectd: put more stuff into struct gossip_state.
We're the only ones who use it now, so put our fields inside it and
make it local.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6115ed02e8 subdaemons: don't stream gossip_store at all.
We now let gossipd do it.

This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 029d65cf2e connectd: serve gossip_store file for the peer.
We actually intercept the gossip_timestamp_filter, so the gossip_store
mechanism inside the per-peer daemon never kicks off for normal connections.

The gossipwith tool doesn't set OPT_GOSSIP_QUERIES, so it gets both, but
that only effects one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 1f608acd4e common: add routine for absolute timeouts (vs. relative).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell e37a638c0c connectd: do nagle by packet type.
channeld can't do it any more: it's using local sockets.  Connectd
can do it, and simply does it by type.

Amazingly, on my machine the timing change *always* caused
test_channel_receivable() to fail, due to a latent race.

Includes feedback from @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 7a514112ec connectd: do dev_disconnect logic.
As connectd handles more packets itself, or diverts them to/from gossipd,
it's the only place we can implement the dev_disconnect logic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 9c0bb444b7 per_peer_state: remove struct crypto_state
Now that connectd does the crypto, no need to hand around crypto_state.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell ce8b69401c peer_io: replace crypto_sync in daemons, use normal wire messages.
Now connectd is doing the crypto, we can use normal wire io.  We
create helper functions to clearly differentiate between "peer" comms
and intra-daemon comms though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell a2b3d335bb connectd: do decryption for peers.
We temporarily hack to sync_crypto_write/sync_crypto_read functions to
not do any crypto, and do it all in connectd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell ae4669f77f common/coin_mvt: make it clear we're using Lightning HRP not BIP 173.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 425a7af512 common/coin_mvt: clean up API a little.
1. tal_strndup(.., str, strlen(str)) == tal_strdup()
2. tal_strdup also takes(), so document that.
3. Avoid passing 'struct sha256' on the stack: use ptr.
4. Generally, structures shouldn't keep pointers to things they don't own.
   In this case, mvt->node_id.
5. Make payment_hash a pointer, since NULL is more natural than an all-zero
   hash.

And add NON_NULL_ARGS() to the functions; it's cumbersome, but make it
fairly clear what params are optional.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 967ffbfbcb global: use tal_dup_or_null().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30