Commit Graph

1274 Commits

Author SHA1 Message Date
Rusty Russell a770f51d0e tools/generate_wire.py: make functions allocate the TLV.
Requiring the caller to allocate them is ugly, and differs from
other types.

This means we need a context arg if we don't have one already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell fa0c29f959 tools/generate_wire.py: tlvs should start with tlv_
No more "towire_offer", but "towire_tlv_offer".

This means we double-up on the unfortunately-named `tlv_payload` inside
the onion, but we should rename that in the spec when we remove
old payloads.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 7829f2eb06 onion_messages: remove obs2 support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Removed backwards compat with onion messages from v0.10.1.
2022-03-25 13:55:44 +10:30
Rusty Russell 726b6878d1 offers: import latest variant from draft.
In particular, this changes the name of a field in invoice_request:
`payer_signature` becomes simply `signature`.  So we allow both for
now, and send the old one unless deprecated_apis is disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell e36d4d1143 devtools/decodemsg: fix printing of wireaddr.
printwire_ routines are supposed to print!  And they're only needed
inside devtools/.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 6f7d51ee68 common/onion: always fill in failtlvtype and failtlvpos parameters.
When we did fill them in, we filled them in wrong: the offset should be
the offset in the message, not the field number!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Vincenzo Palazzo 53806d1abd cli: make the command line more user friendly.
Also has to fix up tests.

Changelog-Fixed: cli doesn't required anymore to confirm the password if the `hsm_secret` is already encrypted.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-24 09:53:38 +10:30
Rusty Russell cf55d40eca psbt: fix dual-funding memleak.
This happened occasionally in tests/test_opening.py::test_rbf_fails_to_broadcast:
we overwrote the witness stack without freeing the old one.

```
lightningd-2: 2022-03-18T02:23:32.113Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1: MEMLEAK: 0x559c375d9df8
lightningd-2: 2022-03-18T02:23:32.113Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   label=wally_tal
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   backtrace:
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     common/setup.c:13 (wally_tal)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/internal.c:285 (wally_calloc)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/transaction.c:234 (wally_tx_witness_stack_init_alloc)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/psbt.c:1119 (pull_psbt_input)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/psbt.c:1411 (wally_psbt_from_bytes)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     bitcoin/psbt.c:722 (psbt_from_bytes)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     bitcoin/psbt.c:753 (fromwire_wally_psbt)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend_wiregen.c:246 (fromwire_dualopend_reinit)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend.c:3855 (main)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   parents:
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     struct wally_psbt
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend.c:3804:struct state
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba932
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 10:19:50 +10:30
Rusty Russell 65be18d355 memleak: handle libwally allocations better.
Things allocated by libwally all get the tal_name "wally_tal",
which cost me a few hours trying to find a leak.

In the case where we're making one of the allocations the parent
of the others (e.g. a wally_psbt), we can do better: supply a name
for the tal_wally_end().

So I add a new tal_wally_end_onto() which does the standard
tal_steal() trick, and also changes the (typechecked!) name.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 10:19:50 +10:30
Rusty Russell 278ea6c8ac memleak: make sure we catch children which are also "notleak".
By not recursing into "notleak" children, we could miss other "notleak"
pointers:

```
2022-03-15T05:25:33.9759500Z lightningd-1: 2022-03-15T05:00:15.998Z **BROKEN** connectd: MEMLEAK: 0x55b29c3d0cc8
2022-03-15T05:25:33.9759901Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:   label=common/timeout.c:22:struct oneshot **NOTLEAK**
2022-03-15T05:25:33.9760195Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:   backtrace:
2022-03-15T05:25:33.9760555Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
2022-03-15T05:25:33.9760905Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     common/timeout.c:22 (new_reltimer_)
2022-03-15T05:25:33.9761272Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     connectd/multiplex.c:558 (set_closing_timer)
2022-03-15T05:25:33.9761647Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     connectd/multiplex.c:586 (write_to_peer)
2022-03-15T05:25:33.9761986Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     ccan/ccan/io/io.c:59 (next_plan)
2022-03-15T05:25:33.9762335Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/io.c:435 (io_do_always)
2022-03-15T05:25:33.9762692Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/poll.c:304 (handle_always)
2022-03-15T05:25:33.9763035Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/poll.c:385 (io_loop)
2022-03-15T05:25:33.9763376Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     connectd/connectd.c:2131 (main)
2022-03-15T05:25:33.9763645Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:   parents:
2022-03-15T05:25:33.9764020Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/io.c:91:struct io_conn **NOTLEAK**
2022-03-15T05:25:33.9764471Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     connectd/connectd.c:2104:struct daemon
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-20 08:42:57 +10:30
Rusty Russell 43a833e405 lightningd: remove support for legacy onion format.
As per proposal in https://github.com/lightning/bolts/pull/962

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
2022-03-18 09:20:11 +10:30
Ken Sedgwick 36466af3eb hsmd: Add fields to hsmd_sign_{,remote_}commitment_tx for validating signers 2022-03-16 12:06:21 +10:30
Michael Schmoock b930b8c548 wireaddr: adds wireaddr_eq_without_port and wireaddr_cmp_type
Adds wireaddr_eq_without_port so it can be used later.
Moves wireaddr_cmp_type from connectd.c to this file, so it can be reused later.
2022-03-11 16:42:45 +10:30
Ken Sedgwick 8f56f96801 hsmd: Add wallet index metadata to existing messages 2022-03-10 10:22:29 +10:30
Ken Sedgwick 8876c03791 hsmd: Add PSBT keypath utility functions 2022-03-10 10:22:29 +10:30
Ken Sedgwick 1e1948bfd2 hsmd: Add redeemscripts to anchor outputs 2022-03-10 10:22:29 +10:30
niftynei b33868f7c2 coin_mvt: only chain moves have a blockheight
channel moves don't have blockheights
2022-03-05 15:12:12 +10:30
niftynei 7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei 24a1c91045 coin_mvt: report the number of outputs on a channel close tx
The bookkeeper needs to know how many outputs to expect before we can
consider a channel resolved onchain.
2022-03-05 15:12:12 +10:30
Rusty Russell dd495b2b21 memleak: fix handling of excluded pointers.
We often hand an exclude pointer (usually the current command) to
memleak.  But when we encountered this we would stop iterating, rather
than just ignore it: this means we would often ignore significant siblings.

In particular, fixing this (which has always been there) reveals many
previously-undetected leaks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell ea71e9fe5b memleak: fix handling of notleak pointers.
We were ignoring the *parent* of the notleak pointers, not the notleak
pointer itself!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell fe86395e7c memleak: restore functionality accidentally removed.
Accidentally removed in 6c9b752751.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell 9bf2aa65cd common/param: don't keep around the temporary array off cmd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell 3218e5332f common/param: fix leak with opt_param_def.
We allocate the default, then callback allocates over the top.  Mark
params with a default, so we can free that when it's called.

(We can't do this generally, since not all param args are actually
pointers to pointers, though opt_param_def has to be).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell f0ea4d60b9 onchaind: fix minor leaks.
Not actually leaks, but they do live longer than they need.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
Rusty Russell 4b7d2dc5b8 common/bolt11: make decoded routes hang off routes arr, not b11.
This causes weirdness in pay, which tal_steal's b11->routes and expects
to get it all.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-27 07:44:26 +10:30
William Casarin 2b92ac4236 bolt11: mark when expiry is decoded
It looks like the x tag isn't marked when parsed either?
2022-02-22 09:58:48 +10:30
William Casarin af16b9b9f4 bolt11: mark when we decode min_final_cltv_expiry
It looks like decode_c doesn't set have_c unlike the other decode_
methods. At the start of the function, decode_c checks have_c to see if
it's set, but it is never set. It seems like this could allow for
duplicate c tags, which is probably not intended.

Signed-off-by: William Casarin <jb55@jb55.com>
2022-02-22 09:58:48 +10:30
Michael Schmoock 38e2abf68a peer_exchange: set, read and log remote_addr
Changelog-Added: Protocol: set remote_addr on init tlvs
2022-02-22 05:45:47 +10:30
Rusty Russell b47930f395 gossip_store: handle compacted gossip_store correctly.
Don't send EOF marker to peer, e.g. in tests/test_gossip.py::test_gossip_store_compact:

```
lightningd-2: 2022-01-24T03:34:22.925Z DEBUG   connectd: gossip_store at end, new fd moved to 1875
lightningd-2: 2022-01-24T03:34:22.933Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: Sending gossip INVALID 4105
lightningd-2: 2022-01-24T03:34:22.933Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: peer_in WIRE_WARNING
lightningd-2: 2022-01-24T03:34:22.941Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: peer_out INVALID 4105
lightningd-2: 2022-01-24T03:34:22.949Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: billboard perm: Received warning channel 2c7cf1dc9dada7ed14f10c78ade8f0de907c1b70e736c12ff6f7472dc69c3db3: Peer sent unknown message 4105 (INVALID 4105)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 3c5d27e3e9 subdaemons: remove gossipd fd from per-peer daemons.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell f14c0ef76a common: make sure we hand through peer for io logging.
This is mainly useful for connectd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
niftynei 43f1fb4561 coin_moves: an htlc_fulfill claimed by them is a credit to "external"
The money moved into an external account here.
2022-02-07 13:02:09 +10:30
niftynei 39572c1aaf coin_moves: remove unused method
This wasn't used anywhere, so we remove it.
2022-02-07 13:02:09 +10:30
Rusty Russell 8a8d7c4243 elements: unify overhead calculation.
And in particular, fix onchaind grinding code which used the
actual number of inputs and outputs (which already includes the
fee output); that breaks with the next patch which fixes other
calculations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-27 12:22:36 +01:00
niftynei 4dafeede5c coin moves: notify when we make deposits to external accounts
The blockheight is zero though, since these aren't included in a block
yet.

We also don't issue an 'external' deposit event if we can tell that the
address you're sending to actually belongs to our wallet (we'll issue a
deposit event when it gets included in a block)
2022-01-26 13:34:45 +10:30
niftynei 7f511b9ba9 coin moves: external deposits should be *deposits*
Report how much the deposit was for.
2022-01-26 13:34:45 +10:30
niftynei ff4ae8b5f4 coin_moves: add an 'originating_account' field
If a coin move concerns an external account, it's really useful to know
which 'internal' account initiated the transfer.

We're about to add a notification for withdrawals, so we can use this to
track wallet pushes to outside addresses

Changelog-Added: JSONRPC: `coin_movement` to 'external' accounts now include an 'originating_account' field
2022-01-26 13:34:45 +10:30
Rusty Russell 2b5d25c851 common: remove stderr debug in is_valid_witnessprog.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell be299c0d59 common/socket_close: remove now only connectd talks to peer.
connectd does this internally now using ccan/io, with appropriate
credit for ZmnSCPxj who wrote this code in the first place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell f6847f44f6 subds: remove "ignore error" from old LND nodes.
This was put in late 2019, and @t-bast says Eclair doesn't ignore their
errors and has had no issues.

It also conflicts with https://github.com/lightning/bolts/pull/932
which suggests you *should* fail when you receive an error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell 2ab5603624 peer subds: ignore failed writes.
In the case where the peer sends an error (and hangs up) immediately
after init, connectd *doesn't actually read the error* (even after all the
previous fixes so it actually receives the error!).

This is because to tried to first write WIRE_CHANNEL_REESTABLISH, and
that fails, so it never tries to read.  Generally, we should ignore
write failures; we'll find out if the socket is closed when we read
nothing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell d51fb5207a msg_queue: don't allow magic MSG_PASS_FD message for peers.
msg_queue was originally designed for inter-daemon comms, and so it has
a special mechanism to mark that we're trying to send an fd.  Unfortunately,
a peer could also send such a message, confusing us!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell a93c49ca65 connectd: implement @ correctly.
dev_blackhole_fd was a hack, and doesn't work well now we are async
(it worked for sync comms in per-peer daemons, but now we could sneak
through a read before we get to the next write).

So, make explicit flags and use them.  This is much easier now we
have all peer comms in one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 26b9384fd0 various: minor cleanups from Christian's review.
More significant things have been folded.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6d4c56e8b6 connectd: put more stuff into struct gossip_state.
We're the only ones who use it now, so put our fields inside it and
make it local.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6115ed02e8 subdaemons: don't stream gossip_store at all.
We now let gossipd do it.

This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 029d65cf2e connectd: serve gossip_store file for the peer.
We actually intercept the gossip_timestamp_filter, so the gossip_store
mechanism inside the per-peer daemon never kicks off for normal connections.

The gossipwith tool doesn't set OPT_GOSSIP_QUERIES, so it gets both, but
that only effects one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 1f608acd4e common: add routine for absolute timeouts (vs. relative).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell e37a638c0c connectd: do nagle by packet type.
channeld can't do it any more: it's using local sockets.  Connectd
can do it, and simply does it by type.

Amazingly, on my machine the timing change *always* caused
test_channel_receivable() to fail, due to a latent race.

Includes feedback from @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30