Commit Graph

2707 Commits

Author SHA1 Message Date
Rusty Russell 34b6731b82 lightningd: don't return to a default filter level if there are no per-file filters.
In this case, the user's default was info, but they specifically asked for debug
from one plugin.  Since there were no per-file filters, it set filtering to the
default level, info, and rejected it.  Since it's been explicitly filtered in,
we need to pass it at this point.

Reported-by: @wtogami
Fixes: #6503
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-06 14:37:49 +09:30
Rusty Russell be2cbe784f pytest: test for filtering bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-06 14:37:49 +09:30
Dusty Daemon fb179f2d5e
splicing: Remove dependency on experimental_dual_fund
Splicing should work automatically on v1 or v2 channels so this requirement isn’t needed.

Changelog-None
[ Squashed fixup into a single commit --RR ]
2023-08-06 12:19:42 +09:30
Rusty Russell 0f6687ec7b codex32: use "cl" instead of "ms" as our HRP.
This was strongly recommended by Russell O'Connor: the "ms" implies that
it's a BIP-32 master secret, and this is CLN specific.

If we changed the hrp to "cln" it would be better, but apparently that
means we no longer fit in a "standard billfold metal wallet" (and
our code assumes a 2-byte prefix anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 16:12:32 +09:30
Rusty Russell 5900742b3a hsmtool: remove threshold arg.
It really has to be 0, since it's the complete secret.  And we didn't handle
it well, (`a` would be treated as 0, for example!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 08:42:50 +09:30
Rusty Russell 4dcd712d6e hsmtool: tighten checks for validity on getcodexsecret
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 08:42:50 +09:30
Rusty Russell e981a4f14a lightningd: pass signed tx through to close callback.
Thread the signed tx through so close's JSON return contains that,
rather than the unsigned channel->last_tx.

We have to split the "get cmd_id" from "resolve the close commands" though;
and of course, as before, we don't actually print the txids of multiple
transactions even though we may have multi in flight due to splice!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `close` returns a `tx` field with witness data populated (i.e. signed).
Fixes: #6440
2023-08-02 07:09:28 +09:30
Rusty Russell 673d81adde pytest: test for bad 'tx' result from close.
We print it without the witness!

Reported-by: https://github.com/wtogami
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 07:09:28 +09:30
Dusty Daemon 4628e3ace8 channeld: Code to implement splicing
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.

Implement the channeld splicing protocol leveraging the interactivetx protocol.

Implement lightningd’s channel_control to support channeld in its splicing efforts.

Changelog-Added: Added the features to enable splicing & resizing of active channels.
2023-07-31 21:00:22 +09:30
Aditya Sharma 6a734d8b4d tests/test_misc.py: Add a subprocess to make sure getcodexsecret fetches out the hsm_secret in bip93 format. 2023-07-31 20:57:40 +09:30
Rusty Russell 2c2592fe36 lightningd: fix crash on startup expirations.
The wait code assumes ld->wallet is populated, but it's not.  Start the
expiration cycle later.

```
0x7f271a18d08f ??? /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x5581a27dc082 wait_index_increment lightningd/wait.c:112
0x5581a27e331a invoice_index_inc wallet/invoices.c:738
0x5581a27e3dfe invoice_index_update_status wallet/invoices.c:775
0x5581a27e3ea3 trigger_expiration wallet/invoices.c:185
0x5581a27e3f47 invoices_new wallet/invoices.c:134
0x5581a27e8a2c wallet_new wallet/wallet.c:121
0x5581a27b08b5 main lightningd/lightningd.c:1082
```

Fixes: #6457
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 20:42:31 +09:30
Rusty Russell 4b173eec3b pytest: add test for expiration-on-startup crash.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 20:42:31 +09:30
Rusty Russell 4a1d4f4f46 pytest: correctly annotate invoice tests needing sqlite3.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 20:42:31 +09:30
Rusty Russell 97af40c545 pytest: fix the invoice typo migration if you ran and created both.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 13:25:22 +09:30
Rusty Russell 81b30c9483 pytest: test for grubles' screwed node.
If you actually ran your node with the botched "last_invoice_created_index" typo migration
(fortunately, not release, just master) you can get a db with both the real "last_invoices_created_index" and the bad "last_invoice_created_index" entries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 13:25:22 +09:30
Lagrang3 e5695b3c23 renepay: add test
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Lagrang3 3024afe524 renepay: more cleanups
- adopt "const <type> *"convention
- remove use_shadow option for some pyln tests
- show prob. information of flows into paynotes
- show prob. of success of entire payment flow in paynotes
- minflow: We were not releasing the memory of flow arrays when replacing
  them with a new canditate.
- use memleak_scan_obj in memleak_check
- replace u64 with size_t

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Lagrang3 79486c1e3b renepay: fixups after comments
- remove internal gheap checks
- add check for arc_t.chanidx overflow
- remove outdated comments
- check the delta flow bounds before augmenting along a path
- get_flow_paths uses a dynamic tal array instead of a list.
- fix a unit test that depended on the order of returned flows
- fix bug: lightnind doesn't like if I reuse the partid of a failed
  flow, therefore use a higher partid than any of the previous attempts.
- plugin_err instead of LOG_BROKEN if sendpay fails and we cannot get a
  an error code.
- fix wrong comments.
- remove the background timer.
- This is a bugfix. Previous to this the MCF network was built using the
knowledge of the min and max liquidity but it didn't take into account
pending HTLCs.
- Also remove the min_prob_success option but hardcode a 90% value.

Removing some options that are not relevant to the user, they're kept
for developer mode only:
- base_fee_penalty
- min_prob_success
- prob_cost_factor
- remove heap.h, not used

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell b8ca193606 renepay: an experimental payment plugin
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
Changelog-Added: Plugins: `renepay`: an experimental pay plugin implementing Pickhardt payments (`renepay` and `renepaystatus`).
2023-07-31 12:58:56 +09:30
adi2011 82c11529ab tests: Added test_recover to test --recover flag. 2023-07-31 11:29:59 +09:30
niftynei 9b8909e507 dual-fund: keep track of aborted requests, seamlessly restart daemon
Clean restart of daemon after a tx-abort is a nice way to work around
the 'persistent' disconnect that we t-bast noticed.

Changelog-Fixed: `dualopend`: Fix behavior for tx-aborts. No longer hangs, appropriately continues re-init of RBF requests without reconnction msg exchange.
2023-07-30 15:20:04 +09:30
niftynei c807db4b7f dual-fund tests: add tests for reported incompat with Eclair
Bug Report:
- initiate a channel open eclair -> cln
- wait for the transaction to be published
- eclair initiates rbf, and cancels it by sending tx_abort before exchanging commit_sig
- at that point everything looks good, cln echoes the tx_abort and stays connected
- eclair initiates another RBF attempt and sends tx_init_rbf: for some unknown reason,
  cln answers with channel_reestablish (??) followed by an error saying
  "Bad reestablish message: WIRE_TX_INIT_RBF"

Diagnosis:
  CLN is doing a reconnect after a tx-abort is sent.

Extra Test:
  Realized that if we abort, we won't correctly advanced to NORMAL if
  blocks are mined while we're in hanging state. CLN should advance
  after block containing channel open is mined.

Reported-By: @t-bast
2023-07-30 15:20:04 +09:30
Rusty Russell e39fe41ff0 pytest: fix flake in penalty tests
Make sure we've completely processed htlc, so we will definitely consider it an old spend.  If we're too fast, l2 might consider it a legitimate unilateral close:

```
        # Make sure both sides got revoke_and_ack for final.
        l1.daemon.wait_for_log('peer_in WIRE_REVOKE_AND_ACK')
        l2.daemon.wait_for_log('peer_in WIRE_REVOKE_AND_ACK')
    
        # Now we really mess things up!
        bitcoind.rpc.sendrawtransaction(tx)
        bitcoind.generate_block(1)
    
        l2.daemon.wait_for_log(' to ONCHAIN')
        # FIXME: l1 should try to stumble along!
    
        # l2 should spend all of the outputs (except to-us).
        # Could happen in any order, depending on commitment tx.
        needle = l2.daemon.logsearch_start
        ((_, txid1, blocks1), (_, txid2, blocks2)) = \
>           l2.wait_for_onchaind_txs(('OUR_PENALTY_TX',
                                      'THEIR_REVOKED_UNILATERAL/DELAYED_CHEAT_OUTPUT_TO_THEM'),
                                     ('OUR_PENALTY_TX',
                                      'THEIR_REVOKED_UNILATERAL/OUR_HTLC'))

tests/test_closing.py:687: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
contrib/pyln-testing/pyln/testing/utils.py:1264: in wait_for_onchaind_txs
    r = self.daemon.wait_for_log('Telling lightningd about {} to resolve {}'
contrib/pyln-testing/pyln/testing/utils.py:346: in wait_for_log
    return self.wait_for_logs([regex], timeout)
```

You can see l2 here:

```
lightningd-2 2023-07-27T03:34:24.533Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-onchaind-chan#1: Their unilateral tx, old commit point
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 09:44:58 +09:30
Rusty Russell f15e2506ca pytest: fix flake in test_gossip_store_compact_on_load.
We can have almost arbitrary re-transmissions of private updates, looking at the decode: in this case, instead of the expected:

```
  DELETED: private channel_announcement (scid23)
  DELETED: private channel_update (scid23/0)
  DELETED: private channel_update (scid23/1)
delete channel (scid23)
channel_announcement (scid23)
channel_amount
  DELETED: channel_update (scid23/0)
  DELETED: channel_update (scid23/1)
node_announcement
node_announcement
channel_update (scid23)
private channel_announcement (scid12)
  DELETED: private channel_update (scid12/0)
  DELETED: private channel_update (scid12/1)
channel_update (scid23)
private_channel_update (scid12)
private_channel_update (scid12)
```

We had:

```
  DELETED: private channel_announcement
  DELETED: private channel_update
  DELETED: private channel_update
  DELETED: private channel_update
  DELETED: private channel_update
channel_announcement
channel_amount
  DELETED: channel_update
  DELETED: channel_update
node_announcement
node_announcement
channel_update
private channel_announcement
  DELETED: private channel_update
  DELETED: private channel_update
channel_update
private channel_update
private channel_update
```

Which means we deleted 9, and hence failed:

```
    @pytest.mark.developer("gossip without DEVELOPER=1 is slow")
    def test_gossip_store_compact_on_load(node_factory, bitcoind):
        l2 = setup_gossip_store_test(node_factory, bitcoind)
    
        gs_path = os.path.join(l2.daemon.lightning_dir, TEST_NETWORK, 'gossip_store')
        gs = subprocess.run(['devtools/dump-gossipstore', '--print-deleted', gs_path],
                            check=True, timeout=TIMEOUT, stdout=subprocess.PIPE)
        print(gs.stdout.decode())
    
        l2.restart()
    
>       wait_for(lambda: l2.daemon.is_in_log(r'gossip_store_compact_offline: [5-8] deleted, 9 copied'))

tests/test_gossip.py:1776: 
```
2023-07-28 09:44:58 +09:30
Rusty Russell 435a4623ad wallet: fix migration of existing invoices's id variable.
Rusty can't type.  Rusty can't test.  Bad Rusty.  Bad.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-27 15:54:08 +09:30
Rusty Russell 909bed6a30 pytest: add test for db crash on conflicting invoice.id
```
2023-07-26T14:26:37.504Z BROKEN lightningd: Error executing statement: wallet/invoices.c:282: INSERT INTO invoices ( id, payment_hash, payment_key, state , msatoshi, label, expiry_time , pay_index, msatoshi_received , paid_timestamp, bolt11, description, features, local_offer_id) VALUES ( ?, ?, ?, ? , ?, ?, ? , NULL, NULL , NULL, ?, ?, ?, ?);: UNIQUE constraint failed: invoices.id
```

Reported-by: https://github.com/hMsats
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-27 15:54:08 +09:30
Alex Myers 7915a9d678 pytest: add reckless local install test 2023-07-27 06:53:49 +09:30
Alex Myers 8704a4b499 reckless: add installation capability for additional sources
Abstracts search and directory traversal. Adds support for installing
from a local git repository, a local directory, or a web hosted git repo
without relying on an api.

Changelog-Changed: Reckless can now install directly from local sources.
2023-07-27 06:53:49 +09:30
Rusty Russell 978c1699ea lightningd: fail incoming HTLCs if peer would close channel.
This cause of cascading failure was pointed out by @t-bast: if fees spike and
you don't timeout an outgoing onchain HTLC, you should nonetheless fail the incoming htlc
because otherwise the incoming peer will close on you.

Of course, there's a risk of losing funds, but this only happens if you weren't going to get the HTLC spend in time anyway.  And it would also catch any other reason that the downstream onchain goes wrong, containing the damage.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @t-bast
Changelog-Fixed: Protocol: We will close incoming HTLCs early if the outgoing HTLC is stuck onchain long enough, to avoid cascating failure.
2023-07-26 06:58:38 +09:30
Rusty Russell fd445aa467 pytest: test that we proactively close incoming HTLCs to avoid them closing on us.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30
Vincenzo Palazzo f4fd12cc15 channeld: Verify the signature sent by the counterparty
This commit addresses an issue to enhance the resilience of core
lightning when receiving node announcements.

According to BOLT 7 (The announcement_signatures Message),
if the node_signature OR the bitcoin_signature is NOT correct,
it is recommended to either send a warning and close the connection or send an error and fail the channel.

In this commit, we take a strict approach. If any error is detected, we
send an error and fail the open channel operation.
This is because the announcement_signatures operation is optional,
and we assume that it must be correct.

lnprototest at commit dea47c29b5541dbfe7fe53cc2598330e897fa4f4 report
the following error now.

```
2023-07-06T21:03:20.930Z DEBUG   hsmd: Shutting down

ERROR    root:helpers.py:170 Traceback (most recent call last):
  File "/home/vincent/Github/lightning/external/lnprototest/tests/helpers.py", line 167, in run_runner
    runner.run(test)
  File "/home/vincent/Github/lightning/external/lnprototest/lnprototest/runner.py", line 99, in run
    all_done = sequence.action(self)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vincent/Github/lightning/external/lnprototest/lnprototest/structure.py", line 55, in action
    all_done &= e.action(runner)
                ^^^^^^^^^^^^^^^^
  File "/home/vincent/Github/lightning/external/lnprototest/lnprototest/event.py", line 365, in action
    raise EventError(self, "{}: message was {}".format(err, msg.to_str()))
lnprototest.errors.EventError: `Expected msgtype-warning, got msgtype-error: message was error channel_id=a37362839b13f61cfe82d35bd397b1264c389b245847cfb6111b38892546dc77 data=4661696c656420746f20766572696679206e6f64655f7369676e61747572652e` on event [{"event": "ExpectMsg", "file": "test_bolt2-01-close_channel.py", "pos": "157"},]
============================================================================================================================================================== short test summary info ===============================================================================================================================================================
FAILED tests/test_bolt2-01-close_channel.py::test_close_channel_shutdown_msg_normal_case_receiver_side - AssertionError: `Expected msgtype-shutdown, got msgtype-error: message was error channel_id=a37362839b13f61cfe82d35bd397b1264c389b245847cfb6111b38892546dc77 data=4661696c656420746f20766572696679206e6f64655f7369676e61747572652e` on event [{"event": "ExpectMsg", "file": "test_bolt2-01-close_channel.py", "pos": "75"},]
FAILED tests/test_bolt2-01-close_channel.py::test_close_channel_shutdown_msg_wrong_script_pubkey_receiver_side - AssertionError: `Expected msgtype-warning, got msgtype-error: message was error channel_id=a37362839b13f61cfe82d35bd397b1264c389b245847cfb6111b38892546dc77 data=4661696c656420746f20766572696679206e6f64655f7369676e61747572652e` on event [{"event": "ExpectMsg", "file": "test_bolt2-01-close_channel.py", "pos": "157"},]

```

Changelog-Fixes: channeld: Verify the signature sent in announcement_signatures by the counterparty
Reported-by: lnprototest
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-07-26 06:57:15 +09:30
Rusty Russell 620135335d libplugin/pay: allow shortcut for self-pay.
This is the simplest solution, not the best, but there's significant risk in try to remove the "we have a path" assumption in the code pay code.

Includes removing a `tal_steal` which was incorrect: the buffer has the same lifetime as the plugin, so if we steal it then things get messy when we free the  struct payment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `pay` will now pay your own invoices if you try.
2023-07-26 06:55:48 +09:30
Rusty Russell 18760db66d lightningd: sendpay implement zero-length path == self-pay.
Previously, the "payment" and "invoice" paths were completely separate, but this now calls both.  It bypasses htlc_sets (and thus, cannot do MPP), and bypasses the hook too: the former is tied closely to HTLCs, and the hook is also very htlc-centric.

Includes finishing unfinished sentence in sendpay man page, as a bonus.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sendpay` now allows self-payment of invoices, by specifying an empty route.
2023-07-26 06:55:48 +09:30
Rusty Russell 07e00d50c6 invoice: return error string from invoice_check_payment.
Clean these up: they were debug logs, but we want to pass this information
back for self-payments.

Also fixes "Attept" typo which altered tests!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:55:48 +09:30
Rusty Russell adee07125a pytest: test self-payments using sendpay.
We want to allow this, so write a test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:55:48 +09:30
Rusty Russell 11b5f31034 lightningd: clean up commando rune_counter from datastore on migration.
Pointed out by @ShahanaFarooqui, we leave a single unused entry in the datastore,
so we should clean that up too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell 925f9fcce5 lightningd: migrate (and delete) old commando blacklists.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell dccbccf8f2 lightningd: migrate (and delete) old commando runes.
If they have invalid runes, we bail, but if they have runes which used
a different master secret (old commando.py allowed you to override
secret), we just complain and delete them.

Note that this requires more mocks in wallet/test/run-db.c...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell a219c99691 commando: pass through commands to lightningd.
The in-core commands are much more helpful with details on why runes failed, too!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell 38075a95d4 commando: use CLN's checkrune() instead of our own for for rune validation.
This means (temporarily) that blacklisting won't work (fix later), and
means that old-style (commando.py) master-secret-override doesn't work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: `commando` no longer allows datastore ['commando', 'secret'] to override master secret (re-issue runes if you were using that!).
2023-07-25 14:47:12 +09:30
Rusty Russell cd16690af8 commando: always active.
We used to activate on the first rune creation, but we're no longer in charge
of runes, so we can't make that call.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell 8d64e6e277 pytest: use native l1 commando secret.
We're going to remove the ability to override the master secret, so fix
up our tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Shahana Farooqui 49fdc0df99 runes: Renamed `listrunes` to `showrunes` 2023-07-25 11:07:43 +09:30
Rusty Russell d25a8ca0fd lightningd: expose created_index and updated_index fields.
If you miss a wait event, you can catch up by doing listinvoices and
getting the max of these fields.  It's also a good debugging clue.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-23 13:48:37 +09:30
Rusty Russell 284262d006 listinvoices: add limit param.
Changelog-Added: JSON-RPC: `listinvoices` has `limit` parameter for listing control.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-23 13:48:37 +09:30
Rusty Russell 16c133746b listinvoices: add index and start params.
Now we have defined ordering, we can add a start param.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listinvoices` has `index` and `start` parameters for listing control.
2023-07-23 13:48:37 +09:30
Rusty Russell bbf4f312a4 invoice: include invoice label to wait subsystem when it's paid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-23 13:48:37 +09:30
Rusty Russell 6782c2fef5 lightningd: trigger changed wait when delinvoice desconly used.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-23 13:48:37 +09:30
Rusty Russell cbdfc75bde invoices: pass through info to delete function, call invoice_index_deleted.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-23 13:48:37 +09:30
Rusty Russell 5a76eaab06 wallet: hook up updated_index for invoices.
We do expirations inside the loop, so we can set updated_index
and trigger the callback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-23 13:48:37 +09:30