Commit Graph

2885 Commits

Author SHA1 Message Date
niftynei 3190c26bc9 dualfund: error on out of order sigs
We weren't blocking if the tx-sigs arrived before the commitment sigs.

This was causing problems in the openchannel (spender plugin)

spenderp: FATAL SIGNAL 11 (version v23.08.1-404-g62ff475-modded)
0x559836dc98ba send_backtrace
	common/daemon.c:33
0x559836dc9951 crashdump
	common/daemon.c:75
0x7f37f42c351f ???
	./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x7f37f441ac92 ???
	../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S:83
0x559836db7760 bitcoin_txid_eq
	./bitcoin/tx.h:29
0x559836db7760 collect_sigs
	plugins/spender/openchannel.c:509
0x559836db81de check_sigs_ready
	plugins/spender/openchannel.c:531
0x559836db84dd json_peer_sigs
	plugins/spender/openchannel.c:611
0x559836dbcad7 ld_command_handle
	plugins/libplugin.c:1611
0x559836dbcd9d ld_read_json_one
	plugins/libplugin.c:1721
0x559836dbce29 ld_read_json
	plugins/libplugin.c:1741
0x559836ef3bff next_plan
	ccan/ccan/io/io.c:59
0x559836ef40da do_plan
	ccan/ccan/io/io.c:407
0x559836ef4177 io_ready
	ccan/ccan/io/io.c:417
0x559836ef5b14 io_loop
	ccan/ccan/io/poll.c:453
0x559836dbd48d plugin_main
	plugins/libplugin.c:1948
0x559836db22bf main
	plugins/spender/main.c:35
0x7f37f42aad8f __libc_start_call_main
	../sysdeps/nptl/libc_start_call_main.h:58
0x7f37f42aae3f __libc_start_main_impl
	../csu/libc-start.c:392
0x559836da3774 ???
	???:0
0xffffffffffffffff ???
	???:0
2023-10-31T15:15:57.458Z INFO    plugin-spenderp: Killing plugin: exited during normal operation
2023-10-31T15:15:57.458Z **BROKEN** plugin-spenderp: Plugin marked as important, shutting down lightningd!
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_break: lightningd_exit
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_loop_with_timers: main
2023-10-31T15:15:57.458Z DEBUG   connectd: REPLY WIRE_CONNECTD_START_SHUTDOWN_REPLY with 0 fds
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_break: connectd_start_shutdown_reply
2023-10-31T15:15:57.458Z DEBUG   021ccce7bc396996c8f3b7bfeb1e30c6600269517026a74adfe2217b7187879797-dualopend-chan#1: Status closed, but not exited. Killing
2023-10-31T15:15:57.458Z DEBUG   lightningd: Command returned result after jcon close
2023-10-31T15:15:57.458Z INFO    021ccce7bc396996c8f3b7bfeb1e30c6600269517026a74adfe2217b7187879797-chan#1: Unsaved peer failed. Deleting channel.
2023-10-31T15:15:57.464Z DEBUG   lightningd: io_break: destroy_plugin
2023-10-31T15:15:57.464Z DEBUG   connectd: Shutting down
2023-10-31T15:15:57.464Z DEBUG   gossipd: Shutting down
2023-10-31T15:15:57.464Z DEBUG   hsmd: Shutting down

Reported-By: @t-bast
2023-11-02 19:32:05 +10:30
niftynei fa8458c00a dualfund: add test to make sure that tx-sigs sent before commitment
results in an error.
2023-11-02 19:32:05 +10:30
niftynei 30ec8cbf8e dualfund, test: add test for dropping to chain during RBF
Here we make sure we can drop the initial tx to chain, and that an
inflight txid that's missing its commitment sigs is properly ignored.
2023-11-02 19:32:05 +10:30
niftynei da34c369f3 dualfund, tests: break out "peer forgets" test
Now that we save the commitment sigs immediately, we have to drop the
connection elsewhere in the flow to get the state where only one peer
remembers.
2023-11-02 19:32:05 +10:30
niftynei 5417312911 tests: update opening tests for new reconnect behavior
Let's test that things stay together!

One cool thing to note is that now we sort of "magically" recover from
pretty brutal disconnects!

Very nice!
2023-11-02 19:32:05 +10:30
niftynei 20c77419dc dualfund: split 'commit-received' into two parts
Here, we split up what was "commit_received" into two phases:
	- commit-ready, where we're about to send our commitment tx to
	  peer
	- commit-received, when we've gotten the commitment tx from our
          peer

This lets us do the right thing (as far as the spec is concerned) with
returning the correct 'next_funding_txid' on reconnect (later commits).
2023-11-02 19:32:05 +10:30
Rusty Russell f004952442 lightningd: wumbo is now the default, setting has no effect.
"Patrick, I'm sorry I doubted you."

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Config: `large-channels` is now the default, wumbology for all.
2023-11-02 08:16:51 +01:00
Alex Myers 10bac49dac ld: add commit-fee-offset option, update config schema
Changelog-Added: Added option --commit-fee-offset to potentially reduce feerate update disagreements
2023-11-02 09:49:59 +10:30
Alex Myers 4265699fcd lightningd: add a feerate offset when updating feerates as opener
Adding a fee offset as the channel opener reduces the likelihood of a
disconnect by the peer do to slight variation in feerate calculation
between nodes.

Changelog-Fixed: Some peer disconnects due to update_fee disagreements are avoided.
2023-11-02 09:49:59 +10:30
Rusty Russell 28fd70a3d8 lightningd: rewrite anchor spend to use multiple UTXOs if needed.
Closes: #6747
Changelog-EXPERIMENTAL: Fixed anchor spending to be able to use more than one UTXO.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-01 14:11:28 +10:30
Rusty Russell b901f885e0 pytest: force anchor tests to use more than one UTXO.
They fail now, since one UTXO is not enough.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-01 14:11:28 +10:30
Rusty Russell 1d677bcba7 addpsbtoutput: allow command to specify output address.
Default is still to generate it.

Changelog-None: Introduced this release anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-01 14:11:28 +10:30
Chris Guida c6262189b7 cln-plugin: Add test for cln-plugin send_custom_notification
Also fix Makefile for rust plugin examples

Also add in a missing assert in the test_plugin_start test
2023-10-30 18:24:23 +01:00
Rusty Russell af971fd025 wait forwards: add in_htlc_id
Without this, we have no unique identifier for which forward happened.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-28 15:48:26 +10:30
Rusty Russell 0317348479 pytest: add test for wait forwards.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-28 15:48:26 +10:30
Rusty Russell 1d8af90b56 listforwards: add created_index and updated_index fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listforwards` fields `created_index` (old: `id`) and `updated_index`.
2023-10-28 15:48:26 +10:30
Rusty Russell c70a326f1f lightningd: add ordering and pagination to listsendpays.
Changelog-Added: JSON-RPC: `listsendpays` new parameters `index`, `start` and `limit`.
2023-10-28 15:48:26 +10:30
Rusty Russell 00ac1a9475 listsendpays: add created_index and updated_index fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `sendpay`, `listsendpays`, `delpay` new fields `created_index` (old: `id`) and `updated_index`.
2023-10-28 15:48:26 +10:30
Rusty Russell 9ec6ac9922 lightningd: don't log that we're trying to pay if we're not.
Looking through logs I was surprise to see:

```
lightningd-1 2023-10-26T03:42:36.824Z INFO    lightningd: Sending 200000000msat over 1 hops to deliver 200000000msat
```

On a re-payment where we simply returned from sendpay immediately!  Move that log to later.
2023-10-28 15:48:26 +10:30
Rusty Russell f65c3cab75 wallet: use standard-style iterators for payments.
We used to have "unsaved" payments: now we don't we can use
our normal "iterator" pattern rather than returning arrays.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-28 15:48:26 +10:30
Aditya Sharma 34692ec3c8 tests/test_misc.py: Add test_decode to test the decryption of bech32 encoding of emergency.recover 2023-10-27 16:05:28 +10:30
Dusty Daemon 990f4d0dad splice: Update test for new logs
The log messages were changed but the test fields weren’t updated, resulting in some test flakiness. Being more explicit with the log message we’re looking for should help.

Changelog-None
2023-10-27 13:20:33 +10:30
Rusty Russell f8e036b22d channeld: use anchors on peer's commitment(s) if we can't broadcast our own.
This means refactoring out some of the generic anchor info, from the
per-commitment-tx info (we can have at least two, perhaps more with
splicing!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-27 11:19:56 +10:30
Rusty Russell 2bf21daba7 pytest: test (fails!) that we push anchor on peer's commitment tx.
We try to use anchors to CPFP our own commitment, but what if they
get there first?  We also need to use anchors on the commitment
txs they broadcast.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-27 11:19:56 +10:30
Christian Decker 0c32183e8d pytest: Mark `test_plugin_startdir_lol` as `@unittest.skip()`
It isn't that flaky, but when it gets stuck we'll be waiting for 30m
for it to complete.
2023-10-26 15:51:07 +02:00
Christian Decker 23ce9a947d pytest: Mark test_exclude_adjacent_routehint as skip
It has about a 7% flakyness rate.
2023-10-26 15:51:07 +02:00
Christian Decker 8f4c997248 pytest: Mark `test_splice` as falky 2023-10-26 15:51:07 +02:00
Christian Decker 7deeff8404 pytest: Fix a falke in test_gossip_pruning
Fixes cause of https://github.com/ElementsProject/lightning/actions/runs/6518612194/job/17704479411
2023-10-26 15:51:07 +02:00
Christian Decker 5c12d14e11 pytest: Skip `test_commit_crash_splice` as it is too flaky 2023-10-26 15:51:07 +02:00
Chris Guida 88ac2dcb7e add test, update schemas, fix warning_custom_fallbacks 2023-10-26 19:11:17 +10:30
Dusty Daemon dc4e0a400f Splice: Better balance checking
* Regression test added for Issue #6572 (issuecomment-1730808863) w/stuck HTLC
* `check_balance` adjusted to calculate pending HTLCs explicitly
* Test confirmed to fail prior to PR #6713

ChangeLog-Fixed: Issue splicing with pending / stuck HTLCs fixed.
2023-10-26 13:24:04 +10:30
Rusty Russell a50b795bc2 lightningd: implement `recover` command.
Changelog-Added: JSON-RPC: `recover` command to force (unused) lightningd node to restart with `--recover` flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell 7f18ed7743 lightningd: allow --recover to take a 64-char hex string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `--recover` can take a 32-byte hex string, as well as codex32.
2023-10-26 12:59:55 +10:30
Rusty Russell 3b622f06c3 lightningd: use param_check on all commands which do extra checks.
This makes `check` much more thorough, and useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` now does much more checking on every command (not just basic parameter types).
2023-10-26 12:59:55 +10:30
Rusty Russell 25110ff2cc connectd: fix fd leak for --offline.
```
**BROKEN** connectd: dev_report_fds: 5 open but unowned?
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell 35a47c57e3 lightningd: add direct close outputs to listfunds (mutual close).
We had a complaint that you can't CPFP a mutual close, which you
should be able to do.

Fixes: #6692
Changelog-Fixed: wallet: close change outputs show up immediately in `listfunds` so you can CPFP.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:58:34 +10:30
Rusty Russell bddff79bc5 pytest: test that we can CPFP a mutual close.
You can't (yet): it doesn't even show in listfunds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:58:34 +10:30
Peter Neuroth 747725dabd test: Add test for datastoreusage
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Rusty Russell 0550bb0a7d pytest: fix flake in test_remote_addr_port
```
FAILED tests/test_connection.py::test_remote_addr_port - TimeoutError: Unable to find "[re.compile('Update our node_announcement for discovered address')]" in logs.
```

Because it can happen before the "Already have funding locked in" message:

```
lightningd-2 2023-10-24T22:07:02.018Z DEBUG   gossipd: Update our node_announcement for discovered address: 127.0.0.1:1234
lightningd-2 2023-10-24T22:07:02.019Z DEBUG   lightningd: Plugin chanbackup returned from peer_connected hook call
lightningd-2 2023-10-24T22:07:02.019Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#2: Peer has reconnected, state CHANNELD_NORMAL: connecting subd
lightningd-2 2023-10-24T22:07:02.036Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: pid 63142, msgfd 67
lightningd-2 2023-10-24T22:07:02.037Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#2: Already have funding locked in (and ready to announce)
```

Also, wait_for_log() asserts itself, no need to assert on result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 23:11:37 +02:00
Rusty Russell 5f664dac77 pytest: fix flake in test_excluded_adjacent_routehint
Wait to be sure l1->l2 is ready.  And use the same pattern for l2->l3.

```
    def test_excluded_adjacent_routehint(node_factory, bitcoind):
        """Test case where we try have a routehint which leads to an adjacent
        node, but the result exceeds our maxfee; we crashed trying to find
        what part of the path was most expensive in that case
    
        """
        l1, l2, l3 = node_factory.line_graph(3)
    
        # We'll be forced to use routehint, since we don't know about l3.
        wait_for(lambda: len(l3.rpc.listchannels(source=l2.info['id'])['channels']) == 1)
        inv = l3.rpc.invoice(10**3, "lbl", "desc", exposeprivatechannels=l2.get_channel_scid(l3))
    
        # This will make it reject the routehint.
        err = r'Fee exceeds our fee budget: 1msat > 0msat, discarding route'
        with pytest.raises(RpcError, match=err):
>           l1.rpc.pay(bolt11=inv['bolt11'], maxfeepercent=0, exemptfee=0)

tests/test_pay.py:3420: 
...
>           raise RpcError(method, payload, resp['error'])
E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt10n1pjntczasp59x0weqkg4u9amd364yaeyw6rmgnmf9qtra6epylcntvt65yalpzspp5x8wgtmjhq33qruk6mmhutyr7w74xxjhct7v9tppel0t9p4rtautsdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqgqqqqqqqqpqqqqqzsqqc9qxpqysgq4euy2qyzl2nufpxv6ahf0s5zry0h5dgrpa5adwu4swrdvjw7qe48cj8kp5fl7k20ex0x3dnk6e8xk5jp82snrdcr6he7eyqd0wrmvlgqwe5nma', 'maxfeepercent': 0, 'exemptfee': 0}, error: {'code': 210, 'message': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 1000msat}]}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 10:02:52 +02:00
Rusty Russell b0ab350927 pytest: fix flake in test_listforwards_and_listhtlcs
Make sure l1 sees chanell to l4.

```
   def test_listforwards_and_listhtlcs(node_factory, bitcoind):
        """Test listforwards and listhtlcs commands."""
        l1, l2, l3, l4 = node_factory.get_nodes(4, opts=[{}, {}, {}, {}])
    
        l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
        l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
        l2.rpc.connect(l4.info['id'], 'localhost', l4.port)
    
        c12, c12res = l1.fundchannel(l2, 10**5)
        c23, _ = l2.fundchannel(l3, 10**5)
        c24, _ = l2.fundchannel(l4, 10**5)
    
        # Wait until channels are active
        mine_funding_to_announce(bitcoind, [l1, l2, l3, l4])
        l1.wait_channel_active(c23)
    
        # successful payments
        i31 = l3.rpc.invoice(1000, 'i31', 'desc')
        l1.rpc.pay(i31['bolt11'])
    
        i41 = l4.rpc.invoice(2000, 'i41', 'desc')
>       l1.rpc.pay(i41['bolt11'])

tests/test_misc.py:2871: 
...
>           raise RpcError(method, payload, resp['error'])
E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnert20n1pjnd7xlsp5lcwwuxw90nanl7cpfzp4jf7w5tats5aalkksswpps6cqrwpu0l3spp5pzng94efstxjhdc2ya8hhgrch33qavzgv4gahga3y6dxqshezuqsdq8v3jhxccxqyjw5qcqp99qxpqysgqys70jhf8aj4zpwrvfrse54c5kfaqxj98xl4ddxp7jkm75e4p59gy6l2ryvz4fejg5reqkxw44k90rp0l7c8vxut424m4ek743k0lpwcqzg2a60'}, error: {'code': 210, 'message': 'Destination 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199 is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199 is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 2000msat}]}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 10:02:52 +02:00
Rusty Russell 043c4ae5eb pytest: fix flake in test_even_sendcustommsg
Make sure plugin has got message to connectd before sending!

```
    def test_even_sendcustommsg(node_factory):
        l1, l2 = node_factory.get_nodes(2, opts={'log-level': 'io',
                                                 'allow_warning': True})
        l1.connect(l2)
    
        # Even-numbered message
        msg = hex(43690)[2:] + ('ff' * 30) + 'bb'
    
        # l2 will hang up when it gets this.
        l1.rpc.sendcustommsg(l2.info['id'], msg)
        l2.daemon.wait_for_log(r'\[IN\] {}'.format(msg))
        l1.daemon.wait_for_log('Invalid unknown even msg')
        wait_for(lambda: l1.rpc.listpeers(l2.info['id'])['peers'] == [])
    
        # Now with a plugin which allows it
        l1.connect(l2)
        l2.rpc.plugin_start(os.path.join(os.getcwd(), "tests/plugins/allow_even_msgs.py"))
    
        l1.rpc.sendcustommsg(l2.info['id'], msg)
        l2.daemon.wait_for_log(r'\[IN\] {}'.format(msg))
>       l2.daemon.wait_for_log(r'allow_even_msgs.*Got message 43690')

tests/test_misc.py:3623: 
...
>                   raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E                   TimeoutError: Unable to find "[re.compile('allow_even_msgs.*Got message 43690')]" in logs.

contrib/pyln-testing/pyln/testing/utils.py:327: TimeoutError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 09:51:43 +02:00
Rusty Russell 6f1bb6fa41 lightningd: don't process request twice if plugin dies.
We remove it from the pending_requests strmap before calling it,
so it doesn't get called again by destroy_plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 15:07:08 +10:30
Rusty Russell db7f59eeba lightningd: simplify plugin hook handling.
Now the internal code will generate a "PLUGIN_TERMINATED" response
when the plugin dies, we can handle it in plugin_hook.

But we can also simplify it by turning the snapshot of hooks into
a simple array: this means we are robust against any combination of plugins
exiting at any time.

Note: this reveals an issue with test_rpc_command_hook where we run
the request hook again (unexpectedly), so we disable that for the next
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 15:07:08 +10:30
Rusty Russell 22ea202ae3 lightningd: generically fail requests when plugin dies.
We had special code to fail a forwarded request, but not for an
internally-generated request.  Instead, we should pretend the (dead)
plugin responded with a PLUGIN_TERMINATED error, and handle the
request through the normal paths.

This breaks the case where a plugin crashes (or stops itself) in a
hook, so we handle that next.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 15:07:08 +10:30
Matt Morehouse a59a8557d9 fuzz: new bolt11 seeds 2023-10-24 12:17:18 +10:30
Matt Morehouse 3a7a1fad4e fuzz: add bolt11 cross-over mutator
This mutator helped the fuzzer find new coverage and detect the bug
fixed in 87f4907bb4.
2023-10-24 12:17:18 +10:30
Matt Morehouse 38e31d6034 fuzz: add custom cross-over functions
These can be used in custom mutators for libFuzzer targets.
2023-10-24 12:17:18 +10:30
Matt Morehouse 58f16c2146 fuzz: set minimum bolt11 size to 9
This prevents a buffer overflow in the custom mutator that can occur
when data_maxlen is 0.
2023-10-24 12:17:18 +10:30
Rusty Russell dfe207d503 pytest: test for accepting even unknown messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 11:50:57 +10:30