Commit Graph

13529 Commits

Author SHA1 Message Date
Lagrang3 79486c1e3b renepay: fixups after comments
- remove internal gheap checks
- add check for arc_t.chanidx overflow
- remove outdated comments
- check the delta flow bounds before augmenting along a path
- get_flow_paths uses a dynamic tal array instead of a list.
- fix a unit test that depended on the order of returned flows
- fix bug: lightnind doesn't like if I reuse the partid of a failed
  flow, therefore use a higher partid than any of the previous attempts.
- plugin_err instead of LOG_BROKEN if sendpay fails and we cannot get a
  an error code.
- fix wrong comments.
- remove the background timer.
- This is a bugfix. Previous to this the MCF network was built using the
knowledge of the min and max liquidity but it didn't take into account
pending HTLCs.
- Also remove the min_prob_success option but hardcode a 90% value.

Removing some options that are not relevant to the user, they're kept
for developer mode only:
- base_fee_penalty
- min_prob_success
- prob_cost_factor
- remove heap.h, not used

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Lagrang3 d46990d301 renepay: add information decay linear in time
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell b8ca193606 renepay: an experimental payment plugin
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
Changelog-Added: Plugins: `renepay`: an experimental pay plugin implementing Pickhardt payments (`renepay` and `renepaystatus`).
2023-07-31 12:58:56 +09:30
Lagrang3 c02f175a75 Import lqueue from CCAN
[ Split into separate commit --RR ]
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell 58327a594f codex32: rework.
Firstly, I wanted the results easier to use:
1. Make them always lower case, even if the string was UPPER.
2. Decode the payload for them.
3. Don't give the user any fields they don't need, and make
   the field sizes explicit.

Secondly, I wanted to avoid the pattern of "check in one place, assume
in another", in favour of "check on use".

So, I changed the code to lower the string if it needs to at the start,
and then changed the pull functions so we always use them to get data:
this way we should fail clearly and gracefully if we don't have enough data.

I made all the checks explicit, where we assign the fields.

I also addressed the FIXME: I think the array is *often* one shorter,
but not always, so I trim the last byte at the end if needed.

[ Aditya modified the tests to work ]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 11:29:59 +09:30
Rusty Russell bf2d9b19c6 codex32: document the complete representations from BIP, adjust numbers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 11:29:59 +09:30
Rusty Russell 9721ddd2ab codex32: minor cleanups.
Nothing major here:
1. size_t for lengths.
2. pass engine to checksum_verify, as caller wants ->len (avoid repeating 13/15 magic numbers).
3. Use x.member instesad of (&x)->member.
4. Return memcmp result directly instead of if.
5. Spacing removal, `;;` removal.
6. codexl is a bool `true`/`false` not 0/1 (it's the same, but clearer)
7. Make sanity_check assign *fail directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 11:29:59 +09:30
adi2011 97bc3ae20a doc: Add documentation for --recover flag. 2023-07-31 11:29:59 +09:30
adi2011 51f72c1d1d lightningd: Added plugin hook to notify whenever started in rocover mode. 2023-07-31 11:29:59 +09:30
adi2011 88a2c0d17e lightningd/options: Always start in offline mode when run using --recover flag. 2023-07-31 11:29:59 +09:30
adi2011 82c11529ab tests: Added test_recover to test --recover flag. 2023-07-31 11:29:59 +09:30
adi2011 581677691d lightningd/common: Added --recover flag to populate HSM_secret with appropriate secret through a valid codex32 secret. 2023-07-31 11:29:59 +09:30
adi2011 3b848d84d1 common/test: Added run-codex32 to test codex32.c decoding function. 2023-07-31 11:29:59 +09:30
adi2011 135a739023 common/codex32: Added checksum verification for codex32 encoding. 2023-07-31 11:29:59 +09:30
adi2011 efc783422f common/codex32: Added codex32_decode() without checksum verification. 2023-07-31 11:29:59 +09:30
Shahana Farooqui 637f314bc6 plugin/clnrest: Improved readme and comments
- Adding runes info in readme
- Added extra comments in the code
- System exit on shutdown
- Replaced single quotes with double quotes

Changelog-None
2023-07-30 16:50:04 +09:30
Shahana Farooqui 3fab4cc4f3 plugin/clnrest: Updating package
- Adding package initializer for future proofing
- Updated Pyln version
2023-07-30 16:50:04 +09:30
niftynei 9b8909e507 dual-fund: keep track of aborted requests, seamlessly restart daemon
Clean restart of daemon after a tx-abort is a nice way to work around
the 'persistent' disconnect that we t-bast noticed.

Changelog-Fixed: `dualopend`: Fix behavior for tx-aborts. No longer hangs, appropriately continues re-init of RBF requests without reconnction msg exchange.
2023-07-30 15:20:04 +09:30
niftynei c807db4b7f dual-fund tests: add tests for reported incompat with Eclair
Bug Report:
- initiate a channel open eclair -> cln
- wait for the transaction to be published
- eclair initiates rbf, and cancels it by sending tx_abort before exchanging commit_sig
- at that point everything looks good, cln echoes the tx_abort and stays connected
- eclair initiates another RBF attempt and sends tx_init_rbf: for some unknown reason,
  cln answers with channel_reestablish (??) followed by an error saying
  "Bad reestablish message: WIRE_TX_INIT_RBF"

Diagnosis:
  CLN is doing a reconnect after a tx-abort is sent.

Extra Test:
  Realized that if we abort, we won't correctly advanced to NORMAL if
  blocks are mined while we're in hanging state. CLN should advance
  after block containing channel open is mined.

Reported-By: @t-bast
2023-07-30 15:20:04 +09:30
niftynei 73a67fbb3e funder: don't re-reserve utxos on retries
This way unreserving the PSBT will work as intended, and we don't have
to keep track of how many times we've called reserved for any one input.

Technically we're supposed to not reserve inputs at *all* while doing
opens, this moves us slightly closer to that.
2023-07-30 15:20:04 +09:30
Alex Myers 955afdeea8 gossipd: set timer for node_announcement regen
Also as update_own_node_announcement is called nearly continuously
under normal operation by maybe_send_own_node_announce, the timer should
not be freed continuously - better to only free before actually
refreshing.
2023-07-29 15:31:26 +09:30
Alex Myers 8f196b2d7b gossipd: actually update own node announcement if needed
When an outdated own node announcement is present, it fails the
nannounce_different test and also fails to kick off the forced regen
timer.

Changelog-Fixed: Node announcements are refreshed more reliably.
2023-07-29 15:31:26 +09:30
Adi Shankara a9565f5589 added frontmatter and fix github workflow 2023-07-29 10:30:25 +09:30
Rusty Russell 4b4937b9bd gossipd: tell lightningd about all our previous channel_updates at startup.
This will at least *help* the case where these were not populated, causing us
to send errors without channel_updated appended.

It's not perfect: we can still send such errors if the gossip store is
corrupted, and we still send them for private channels, but it should
help.

(The much better fix is far more invasive, so slips to next release!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-29 09:44:20 +09:30
Rusty Russell c98711ad28 lightningd: don't crash if peer manages to spend onchain HTLC after we've abandoned upstream.
Since we now abandon HTLCs to avoid breaking a channel (in limited circumstances), the
peer can take the funds if they beat us to spend the HTLC before we timeout.  This is extremely
unusual, but finally happend in CI.

First it fails our internal sanity checks, but then it would fail when we tried to fulfill
an already-failed HTLC:

```
**BROKEN** lightningd: fulfill_our_htlc_out:Output preimage, input failonion?
**BROKEN** lightningd: FATAL SIGNAL 6 (version f82fedb-modded)
**BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x55c09b04f0b5
**BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x55c09b04f247
**BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7f607463551f
**BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:44 (__pthread_kill_implementation) 0x7f6074689a7c
**BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:78 (__pthread_kill_internal) 0x7f6074689a7c
**BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:89 (__GI___pthread_kill) 0x7f6074689a7c
**BROKEN** lightningd: backtrace: ../sysdeps/posix/raise.c:26 (__GI_raise) 0x7f6074635475
**BROKEN** lightningd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7f607461b7f2
**BROKEN** lightningd: backtrace: lightningd/log.c:1016 (fatal_vfmt) 0x55c09afdb7cb
**BROKEN** lightningd: backtrace: lightningd/log.c:1026 (fatal) 0x55c09afdb880
**BROKEN** lightningd: backtrace: lightningd/htlc_end.c:87 (corrupt) 0x55c09afc9472
**BROKEN** lightningd: backtrace: lightningd/htlc_end.c:207 (htlc_out_check) 0x55c09afc9c6b
**BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1451 (fulfill_our_htlc_out) 0x55c09b004dd7
**BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1526 (onchain_fulfilled_htlc) 0x55c09b0050fe
**BROKEN** lightningd: backtrace: lightningd/onchain_control.c:313 (handle_extracted_preimage) 0x55c09afdf9f8
**BROKEN** lightningd: backtrace: lightningd/onchain_control.c:1423 (onchain_msg) 0x55c09afe2da9
**BROKEN** lightningd: backtrace: lightningd/subd.c:557 (sd_msg_read) 0x55c09b019ac8
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 10:18:32 +09:30
Christian Decker 5f2072869a pay: Fix a null-dereference introduced in #6428
The alias may not be set for non-alias channels after they
confirm. The other branch is safe because we only consider active
channels.

Changelog-None
Fixes #6450
2023-07-28 10:06:43 +09:30
Rusty Russell f51ce7be85 pytest: follow RBF txids properly, but ignoring identical "RBF" txs.
`mine_txid_or_rbf` uses is_in_log, which grabs the first line.  Thus it doesn't track when a txid gets "RBF"ed by itself, such as:

```
DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: RBF onchain txid 5df8474399e43c58cc45efb6acf1ae08fe678bea9a27d131628394a009cda361 (fee 122sat) with txid 5df8474399e43c58cc45efb6acf1ae08fe678bea9a27d131628394a009cda361 (fee 122sat)
...
INFO    022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: RBF onchain txid 5df8474399e43c58cc45efb6acf1ae08fe678bea9a27d131628394a009cda361 (fee 122sat) with txid 044a564a2b6f8c7c212246e4973a303d24ce0dcd31c470a9ea272f314cf6a4ce (fee 3630sat)
...
DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: RBF onchain txid 044a564a2b6f8c7c212246e4973a303d24ce0dcd31c470a9ea272f314cf6a4ce (fee 3630sat) with txid 044a564a2b6f8c7c212246e4973a303d24ce0dcd31c470a9ea272f314cf6a4ce (fee 3630sat)
...
DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: RBF onchain txid 044a564a2b6f8c7c212246e4973a303d24ce0dcd31c470a9ea272f314cf6a4ce (fee 3630sat) with txid 044a564a2b6f8c7c212246e4973a303d24ce0dcd31c470a9ea272f314cf6a4ce (fee 3630sat)
```

The simplest fix is to only use INFO lines, which are printed when we actually increase fee.

```
    def test_onchaind_replay(node_factory, bitcoind):
...
        # l1 should still notice that the funding was spent and that we should react to it
        _, txid, blocks = l1.wait_for_onchaind_tx('OUR_DELAYED_RETURN_TO_WALLET',
                                                  'OUR_UNILATERAL/DELAYED_OUTPUT_TO_US')
        assert blocks == 200
        bitcoind.generate_block(200)
        # Could be RBF!
>       l1.mine_txid_or_rbf(txid)

tests/test_closing.py:1860: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
contrib/pyln-testing/pyln/testing/utils.py:1292: in mine_txid_or_rbf
    wait_for(lambda: rbf_or_txid_broadcast(txids))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

success = <function LightningNode.mine_txid_or_rbf.<locals>.<lambda> at 0x7f5d55f770d0>
timeout = 180

    def wait_for(success, timeout=TIMEOUT):
        start_time = time.time()
        interval = 0.25
        while not success():
            time_left = start_time + timeout - time.time()
            if time_left <= 0:
>               raise ValueError("Timeout while waiting for {}".format(success))
E               ValueError: Timeout while waiting for <function LightningNode.mine_txid_or_rbf.<locals>.<lambda> at 0x7f5d55f770d0>
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 09:44:58 +09:30
Rusty Russell e39fe41ff0 pytest: fix flake in penalty tests
Make sure we've completely processed htlc, so we will definitely consider it an old spend.  If we're too fast, l2 might consider it a legitimate unilateral close:

```
        # Make sure both sides got revoke_and_ack for final.
        l1.daemon.wait_for_log('peer_in WIRE_REVOKE_AND_ACK')
        l2.daemon.wait_for_log('peer_in WIRE_REVOKE_AND_ACK')
    
        # Now we really mess things up!
        bitcoind.rpc.sendrawtransaction(tx)
        bitcoind.generate_block(1)
    
        l2.daemon.wait_for_log(' to ONCHAIN')
        # FIXME: l1 should try to stumble along!
    
        # l2 should spend all of the outputs (except to-us).
        # Could happen in any order, depending on commitment tx.
        needle = l2.daemon.logsearch_start
        ((_, txid1, blocks1), (_, txid2, blocks2)) = \
>           l2.wait_for_onchaind_txs(('OUR_PENALTY_TX',
                                      'THEIR_REVOKED_UNILATERAL/DELAYED_CHEAT_OUTPUT_TO_THEM'),
                                     ('OUR_PENALTY_TX',
                                      'THEIR_REVOKED_UNILATERAL/OUR_HTLC'))

tests/test_closing.py:687: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
contrib/pyln-testing/pyln/testing/utils.py:1264: in wait_for_onchaind_txs
    r = self.daemon.wait_for_log('Telling lightningd about {} to resolve {}'
contrib/pyln-testing/pyln/testing/utils.py:346: in wait_for_log
    return self.wait_for_logs([regex], timeout)
```

You can see l2 here:

```
lightningd-2 2023-07-27T03:34:24.533Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-onchaind-chan#1: Their unilateral tx, old commit point
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 09:44:58 +09:30
Rusty Russell b2c1ef73cb build: avoid printing PLUGIN_BASES on every build.
Debugging left over in 94205c094f.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 09:44:58 +09:30
Rusty Russell f15e2506ca pytest: fix flake in test_gossip_store_compact_on_load.
We can have almost arbitrary re-transmissions of private updates, looking at the decode: in this case, instead of the expected:

```
  DELETED: private channel_announcement (scid23)
  DELETED: private channel_update (scid23/0)
  DELETED: private channel_update (scid23/1)
delete channel (scid23)
channel_announcement (scid23)
channel_amount
  DELETED: channel_update (scid23/0)
  DELETED: channel_update (scid23/1)
node_announcement
node_announcement
channel_update (scid23)
private channel_announcement (scid12)
  DELETED: private channel_update (scid12/0)
  DELETED: private channel_update (scid12/1)
channel_update (scid23)
private_channel_update (scid12)
private_channel_update (scid12)
```

We had:

```
  DELETED: private channel_announcement
  DELETED: private channel_update
  DELETED: private channel_update
  DELETED: private channel_update
  DELETED: private channel_update
channel_announcement
channel_amount
  DELETED: channel_update
  DELETED: channel_update
node_announcement
node_announcement
channel_update
private channel_announcement
  DELETED: private channel_update
  DELETED: private channel_update
channel_update
private channel_update
private channel_update
```

Which means we deleted 9, and hence failed:

```
    @pytest.mark.developer("gossip without DEVELOPER=1 is slow")
    def test_gossip_store_compact_on_load(node_factory, bitcoind):
        l2 = setup_gossip_store_test(node_factory, bitcoind)
    
        gs_path = os.path.join(l2.daemon.lightning_dir, TEST_NETWORK, 'gossip_store')
        gs = subprocess.run(['devtools/dump-gossipstore', '--print-deleted', gs_path],
                            check=True, timeout=TIMEOUT, stdout=subprocess.PIPE)
        print(gs.stdout.decode())
    
        l2.restart()
    
>       wait_for(lambda: l2.daemon.is_in_log(r'gossip_store_compact_offline: [5-8] deleted, 9 copied'))

tests/test_gossip.py:1776: 
```
2023-07-28 09:44:58 +09:30
Adi Shankara 7281c448ab update dead links 2023-07-28 09:24:21 +09:30
Christian Decker 6ee98ae74c doc: Add documentation for the tracing subsystem
Changelog-Added: ops: A new tracing system with minimal performance impact was added allowing performance tracing in productive systems
2023-07-28 09:21:42 +09:30
Christian Decker ca808b8476 trace: Use User Statically-Defined Tracepoints to emit tracing spans 2023-07-28 09:21:42 +09:30
Christian Decker 039aaaf777 trace: Instrument topology functions 2023-07-28 09:21:42 +09:30
Christian Decker 3da974ca19 trace: Instrument the wallet startup functions 2023-07-28 09:21:42 +09:30
Christian Decker 7743062928 trace: Instrument lightningd startup sequence
Should tell us where we're spending time.
2023-07-28 09:21:42 +09:30
Christian Decker ed8d3336c9 trace: First draft of API 2023-07-28 09:21:42 +09:30
Christian Decker df10a7ec9b config: Detect presence of USDT header in configure 2023-07-28 09:21:42 +09:30
Matt Morehouse 693d510992 SECURITY.md: fix email address 2023-07-28 09:20:22 +09:30
Rusty Russell 435a4623ad wallet: fix migration of existing invoices's id variable.
Rusty can't type.  Rusty can't test.  Bad Rusty.  Bad.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-27 15:54:08 +09:30
Rusty Russell 909bed6a30 pytest: add test for db crash on conflicting invoice.id
```
2023-07-26T14:26:37.504Z BROKEN lightningd: Error executing statement: wallet/invoices.c:282: INSERT INTO invoices ( id, payment_hash, payment_key, state , msatoshi, label, expiry_time , pay_index, msatoshi_received , paid_timestamp, bolt11, description, features, local_offer_id) VALUES ( ?, ?, ?, ? , ?, ?, ? , NULL, NULL , NULL, ?, ?, ?, ?);: UNIQUE constraint failed: invoices.id
```

Reported-by: https://github.com/hMsats
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-27 15:54:08 +09:30
Christian Decker 7b8ea7f60a msggen: Add two missing fields to the `listpays` schema
The fields were missing because they weren't annotated with a type and
a description. Adding those fixes them.

Changelog-Fixed: msggen: `listpays` now includes the missing `amount_msat` and `amount_sent_msat` fields
No-schema-diff-check: fields were always there, just undocumented!
2023-07-27 12:40:28 +09:30
Alex Myers 7915a9d678 pytest: add reckless local install test 2023-07-27 06:53:49 +09:30
Alex Myers 8704a4b499 reckless: add installation capability for additional sources
Abstracts search and directory traversal. Adds support for installing
from a local git repository, a local directory, or a web hosted git repo
without relying on an api.

Changelog-Changed: Reckless can now install directly from local sources.
2023-07-27 06:53:49 +09:30
Alex Myers 30b873de31 reckless: style cleanups, type hints, simplify create_dir 2023-07-27 06:53:49 +09:30
Alex Myers 4a4da00d28 gossipd: handle upgrade from version 11 gossip_store 2023-07-27 06:41:44 +09:30
Christian Decker bc59bb71fa pay: Fix a non-rebased PR merge error
Looks like there was a PR that was merged without being rebased on top of
master first. Don't to that!

Changelog-None
2023-07-26 13:50:03 +02:00
Rusty Russell 978c1699ea lightningd: fail incoming HTLCs if peer would close channel.
This cause of cascading failure was pointed out by @t-bast: if fees spike and
you don't timeout an outgoing onchain HTLC, you should nonetheless fail the incoming htlc
because otherwise the incoming peer will close on you.

Of course, there's a risk of losing funds, but this only happens if you weren't going to get the HTLC spend in time anyway.  And it would also catch any other reason that the downstream onchain goes wrong, containing the damage.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @t-bast
Changelog-Fixed: Protocol: We will close incoming HTLCs early if the outgoing HTLC is stuck onchain long enough, to avoid cascating failure.
2023-07-26 06:58:38 +09:30
Rusty Russell af71fd689a onchaind: keep trying to collect HTLC even if we're past timeout.
The test actually triggers this:
1. We don't get our commitment tx mined at all (we block it).
2. By the time the peer does, the HTLC is expired.
3. We have the preimage but we don't even try, since it's expired.

We should at least *try* to collect the HTLC in this case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30
Rusty Russell fd445aa467 pytest: test that we proactively close incoming HTLCs to avoid them closing on us.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30