Commit Graph

12985 Commits

Author SHA1 Message Date
Christian Decker b42984afe1 pytest: Reproduce #6143 2023-04-10 17:34:54 +09:30
Rusty Russell 75ec1bebee lightningd: use channel_type as we're supposed to for forward descisions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 7acaccfb36 wallet: add channel_type field to db.
This was always the intent, but now we have to reconstruct from the
disparate fields.

This means `option_anchor_outputs` is now redundant.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 355aa8f497 zeroconf: don't accept channel_type with option_zeroconf unless we're really zeroconf.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: we will upfront reject channel_open which asks for a zeroconf channel unless we are going to do a zerconf channel.
2023-04-10 17:26:47 +09:30
Rusty Russell 7e5146ab0c common/channel_type: routines to set known variants, set scid_alias.
I tested this indeed breaks if we don't accept it, then implemented
the code to accept it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: protocol: We now correctly accept the `option_scid_alias` bit in `open_channel` `channel_type`.
Changelog-Deprecated: protocol: Not setting `option_scid_alias` in `option_channel` `channel_type` for unannounced channels.
2023-04-10 17:26:47 +09:30
Rusty Russell f1deeda123 wallet: allow psbt_using_utxos to take a starter psbt.
It will append inputs to this PSBT instead of allocating a new one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell d502a7ecbb bitcoin: bitcoin_tx_remove_output()
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 34f25db435 lightningd: fix parent reporting for memleaks.
This was confusing!  We reported every second one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 89b96e8ac0 pyln-testing: add support to tell bitcoind not to include txs if fee is too low.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 2fb942d21c bitcoin: rename confusing functions.
1. anchor_to_remote_redeem => bitcoin_wscript_to_remote_anchored,
   which matches other witness script producing functions and makes
   it clear that it's a to_remote variant.
2. is_anchor_witness_script => is_to_remote_anchored_witness_script
   makes it clear that it's about a to_remote output (as altered
   when anchors are enabled) not an anchor output!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 57b2cbcb32 lightningd: expose default_locktime for wider usage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 8493ee5e1a db: print nice message and not just backtrace on bad column name.
Happens more than I expected!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell c45eb62b57 lightningd: create small hsm_sync_req() helper for hsm queries.
Commonalizes a small piece of code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell cf80f0520a connectd: dev-report-fds to do file descriptor audit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 09:41:56 +09:30
Rusty Russell e514a5d43c common: lookup function for symnames.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 09:41:56 +09:30
Rusty Russell 3e49cb01bd connectd: don't leak fds if we have both IPv4 and IPv6.
We accept that we will fail to listen if we bind both IPv6 and IPv4 to
the same socket on a dual-stack machine (e.g. normal Linux), but we weren't
closing the fd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 09:41:56 +09:30
Rusty Russell ed58c24bc7 connectd: log broken if TCP_CORK fails.
But not if we're a developer using dev_disconnect, which substitutes the fd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 09:41:56 +09:30
Rusty Russell 295557ac50 connectd: don't try to set TCP_CORK on websocket pipe.
Most of this is piping the flag through so we know it's a websocket!

Reported-by: @ShahanaFarooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 09:41:56 +09:30
Rusty Russell a000ee015a lightningd: do RBF again for all the txs.
Now we've set everything up, the replacement code is quite simple.

Some tests now have to deal with RBF though, and our rbf tests need work
since they look for the old onchaind messages.

In particular, when we can't afford the fee we want, we back off to
the next blockcount estimate, rather than spending all on fees
(necessarily).  So test_penalty_rbf_burn no longer applies.

Changelog-Changed: Protocol: spending unilateral close transactions now use dynamic fees based on deadlines (and RBF), instead of fixed fees.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 62fa91e23b lightningd: rebroadcast all pending txs each 30-60 seconds.
We also do it on every block, but since bitcoind can't always be counted
to rebroadcast for us, we might as well be aggressive!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 3754e283f8 lightningd: remember if they set "allowhighfees" when we rebroadcast.
We would only set it the first time, which was OK for how we were using it
before.  Now we want to also set it for rebroadcast.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 5582970715 lightningd: split the simple onchain tx signing code.
Splitting into onchaind_tx() into onchaind_tx_unsigned() and
sign_and_get_witness() makes it easier to reuse for RBF.

Adding more information in onchain_signing_info is required too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 819d9882aa lightningd: base feerate for onchain txs on deadlines, not fixed fees. 2023-04-10 07:31:12 +09:30
Rusty Russell 3a3370f4c1 feerates: add `floor` field for the current minimum feerate bitcoind will accept
Changelog-Added: JSON-RPC: `feerates`: added `floor` field for current minimum feerate bitcoind will accept
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 812a5a14c0 plugins/bcli: use the new feerate levels, and the floor.
Fixes: #4473
Changelog-Deprecated: Plugins: `estimatefees` returning feerates by name (e.g. "opening"); use `fee_floor` and `feerates`.
Changelog-Fixed: Plugins: `bcli` now tells us the minimal possible feerate, such as with mempool congestion, rather than assuming 1 sat/vbyte.
2023-04-10 07:31:12 +09:30
Rusty Russell 9e2d4240b1 lightningd: handle bcli plugins returning fee_floor and feerates parameters.
Changelog-Added: Plugins: `estimatefees` can return explicit `fee_floor` and `feerates` by block number.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell c46473e615 lightningd: allow "NNblocks" and "minimum" as feerates.
And consolidate descriptions into lightning-feerates().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` now allow "minimum" and NN"blocks" as `feerate` (`feerange` for `close`).
2023-04-10 07:31:12 +09:30
Rusty Russell 64b1ddd761 lightningd: clean up feerate handling, deprecate old terms.
Drop try_get_feerate() in favor of explicit feerate_for_deadline() and
smoothed_feerate_for_deadline().

This shows us everywhere we deal with old-style feerates by names.

`delayed_to_us` and `htlc_resolution` will be moving to dynamic fees,
so deprecate those.

Note that "penalty" is still used for generating penalty txs for
watchtowers, and "unilateral_close" still used until we get zero-fee
anchors.

Changelog-Added: JSON-RPC: `feerates` `estimates` array shows fee estimates by blockcount from underlying plugin (usually *bcli*).
Changelog-Changed: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) value *slow* is now 100 block-estimate, not half of 100-block estimate.
Changelog-Deprecated: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) expressed as, "delayed_to_us", "htlc_resolution", "max_acceptable" or "min_acceptable".  Use explicit block counts or *slow*/*normal*/*urgent*/*minimum*.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell cdb85d5618 lightningd: handle fees as blockcount + range.
Rather than have specific-purpose levels, have an array of
[blockcount, feerate], and rebuild the specific-purpose levels
for now on top.

We also keep a *separate* smoothed feerate, so you can ask for that
explicitly.

Since all the plugins used the same formula to derive the different
named fee levels, we apply the reverse to return to the underlying
estimates: updating the interface comes next.

This is ugly for now, but various specific-purpose levels will be
going away, as we shift to deadline-driven fees.

This temporarily breaks the floor calculation, so that test is
disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell faae44713b lightningd: clarify uses of dynamic (mempool) feerate floor, and static.
We have the FEERATE_FLOOR constant if you don't care, but usually you want
to use the current bitcoind lower limit, so call get_feerate_floor()
(which is currently the same, but coming!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 7aa8c76002 pytest: test parsefeerate explicitly.
Since we're messing with feerates, it's good to test this directly upfront.

Also, fix documentation!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell a2ca34ccf5 common: add tal_arr_insert helper to utils.h
We have tal_arr_remove and tal_arr_append already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 6799cd5d0b plugins/bcli: move commit-fee (dev-max-fee-multiplier) and into core.
Turns out the two bcli replacements I checked (`sauron` and
`trustedcoin`) don't even implement this, and the multiplier makes
more sense in lightningd, especially as we move to bcli just providing
raw feerate estimates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell b53cc69cfd msggen: fix incorrect assertion.
Adding a new field with `added` fails:

```
AssertionError: Field Feerates.perkb.estimates[] does not have an `added` annotation
```

Looks like this assertion is wrong: we should get an added from the field itself or
from the .msggen.json file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Matt Morehouse ca80dee514 doc: add section about improving fuzzing corpora
We want to encourage contributions to the seed corpora that improve
coverage.
2023-04-09 12:41:27 +09:30
Matt Morehouse 6e11a2e416 doc: document "make check-fuzz" 2023-04-09 12:41:27 +09:30
Matt Morehouse 14afa6efe7 make: add check-fuzz target
The target builds and runs each fuzz target on its seed corpus.
2023-04-09 12:41:27 +09:30
Matt Morehouse fa988d2942 fuzz: add check-fuzz.sh
The script runs each fuzz target on its seed corpus and prints any
failures.
2023-04-09 12:41:27 +09:30
Matt Morehouse a1a1373090 fuzz: improve corpus merging
The following arguments were copied from Bitcoin Core's corpus merging
script https://github.com/bitcoin/bitcoin/blob/master/test/fuzz/test_runner.py:

-shuffle=0
-prefer_small=1
-use_value_profile=1
2023-04-09 12:41:27 +09:30
Matt Morehouse 9384692e2a fuzz: add initial seed corpora
These corpora were generated with default libFuzzer flags with 30+ hours
of CPU time, and then minimized with:
  ./fuzz-TARGET -merge=1 -shuffle=0 -prefer_small=1 -use_value_profile=1 corpora/fuzz-TARGET UNMINIMIZED_CORPUS
2023-04-09 12:41:27 +09:30
Alex Myers 2f050621b0 pytest: add blackbox tests for reckless
A canned lightningd/plugins is used to test against. This allows faster
and more deterministic outcomes.

Changelog-None
2023-04-09 12:41:11 +09:30
Alex Myers cf203369bc reckless: use environment variable redirects
This will be used during CI testing in the following commit.
2023-04-09 12:41:11 +09:30
Alex Myers 55cddcd350 reckless: add support for additional networks
This should have been added earlier as @cdecker suggested, but is needed
to enable CI testing.

Changelog-Changed: Reckless - added support for networks beyond bitocoin and regtest
2023-04-09 12:41:11 +09:30
Alex Myers e61401aab9 reckless: don't crash on subprocess calls
They prefer Paths to be explicitly cast as strings
2023-04-09 12:41:11 +09:30
Rusty Russell 88905e8372 tests: split fetchinvoice recurrence tests into separate test.
This is for VLS, which doesn't implement signing for the non-standard
recurrence fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:23:14 +09:30
Rusty Russell d4ffc75691 Makefile: update to latest BOLT text.
In particular:
	- Bolt 4: add route blinding construction
	- Bolt 4: add blinded payments

And this means it's not experimental, so we can turn it on
by default!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: blinded payments are now supported by default (not just with `--experimental-onion-messages`)
2023-04-07 21:22:56 +09:30
Rusty Russell 458a85042b Makefile: update to BOLT 20066dc2aba906f37f3be5a810ae67040f265377
"BOLT 03: fix static-remote same amt and pre-image test vector"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell f26b1166b7 Makefile: update bolts a0bbe47b0278b4f152dbaa4f5fab2562413a217c
"BOLT 04: remove associated data from test vector"

(We actually use merge point).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell dfa6c0ca52 Makefile: bolt version b38156b9510c0562cf50f8758a64602cc0315c19
"Allow nodes to overshoot final htlc amount and expiry (#1032)"

Note that this also renamed `min_final_cltv_expiry` to the more-correct
`min_final_cltv_expiry_delta`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell fdf9b13bdb Makefile: update bolts fc40879995ebc61cc50dfd729512f17afb15b355.
"Allow nodes to overshoot the MPP `total_msat` when paying (#1031)"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Allow slight overpaying, even with MPP, as spec now recommends.
2023-04-07 21:22:56 +09:30