Commit Graph

13130 Commits

Author SHA1 Message Date
Rusty Russell cba310b983 bitcoin: set PSBT amount inside tx for bitcoin_tx_output_set_amount.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell a5232659bf wallet: fix psbt_using_utxos when used with base PSBT.
We were setting the wrong input number: don't assume it's the
same as the UTXO number, but simply the last-appended input.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell 3b1652842e msggen: add new version string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell 7f265300c7 lightningd: ignore any new options we add in deprecated output.
It's weird to add new options into the deprecated section.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell 5c6e706026 lightningd: initialize channel_type field on dualopend channel creation.
Otherwise a badly-timed listpeerchannels will crash.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell 336dcef541 wallet: don't try to set a timer past 2038 on 32-bit platforms.
It'll wrap, probably be in the past, and infinite loop.  This was caused by an invoice
with expiry set at 2076.  This wrap caused us to think the expiry has already
passed, and keep looping!

Reported-by: @telelvis
Fixes: #6339
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightnind: don't infinite loop on 32 bit platforms if only invoices are expiring after 2038.
2023-06-29 06:54:58 -04:00
Bitkarrot 910630c497
add m1 native lib paths to Makefile
add m1 native lib paths to Makefile (also need PKG_CONFIG_PATH)

[ Squashed into a single commit --RR ]
2023-06-26 08:25:21 +09:30
avatar4d 856808aa2d Correct INSTALL.md requirements for OpenBSD
GNU xgettext is required to compile. It is found in the gettext-tools package in OpenBSD. Tested on OpenBSD 7.3
2023-06-26 08:18:28 +09:30
Christian Decker 6507d34b50 cln-grpc: Add some missing states to HtlcState 2023-06-23 14:38:23 +09:30
Rusty Russell b5845afd43 pytest: fix another flake in test_restorefrompeer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-23 14:36:23 +09:30
Rusty Russell 71adeccfbf pytest: fix reconnect flake in test_plugin_connected_hook_chaining
```
         l1.rpc.reject(l3.info['id'])
     
         l2.connect(l1)
         l1.daemon.wait_for_logs([
             f"peer_connected_logger_a {l2id}",
             f"{l2id} is allowed",
             f"peer_connected_logger_b {l2id}"
         ])
         assert len(l1.rpc.listpeers(l2id)['peers']) == 1
     
 >       l3.connect(l1)
 tests/test_plugin.py:468: 
... 
 >           raise RpcError(method, payload, resp['error'])
 E           pyln.client.lightning.RpcError: RPC call failed: method: connect, payload: {'id': '0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518', 'host': '127.0.0.1', 'port': 42391}, error: {'code': 402, 'message': 'disconnected during connection'}
 
 contrib/pyln-client/pyln/client/lightning.py:422: RpcError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-23 14:36:23 +09:30
Rusty Russell ffb0f03e1a lightningd: fix crash on shutdown while if channel being cancelled at same time.
```
 DEBUG   lightningd: Command returned result after jcon close
 DEBUG   connectd: Shutting down
 DEBUG   gossipd: Shutting down
 DEBUG   hsmd: Shutting down
 **BROKEN** lightningd: FATAL SIGNAL 6 (version b29955a-modded)
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/common/daemon.c:38 (send_backtrace) 0x55fea4a13a08
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/common/daemon.c:75 (crashdump) 0x55fea4a140a1
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9ea680651f
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9ea685aa7c
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9ea6806475
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9ea67ec7f2
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:95 (call_error) 0x55fea4bbef68
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:169 (check_bounds) 0x55fea4bbfa68
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:180 (to_tal_hdr) 0x55fea4bbdefe
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:193 (to_tal_hdr_or_null) 0x55fea4bbd9cf
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:461 (tal_alloc_) 0x55fea4bbd893
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:506 (tal_alloc_arr_) 0x55fea4bbdce6
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/str/str.c:91 (tal_vfmt_) 0x55fea4bbc95e
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/lightningd/jsonrpc.c:502 (command_fail) 0x55fea499c427
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/lightningd/channel.c:81 (destroy_channel) 0x55fea4976a31
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:246 (notify) 0x55fea4bbdc18
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:437 (del_tree) 0x55fea4bbe028
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:521 (tal_free) 0x55fea4bbde84
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/lightningd/lightningd.c:577 (free_all_channels) 0x55fea49a2660
 **BROKEN** lightningd: backtrace: /home/runner/work/lightning/lightning/lightningd/lightningd.c:1280 (main) 0x55fea49a1530
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9ea67edd8f
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9ea67ede3f
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x55fea496ef04
 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-23 14:36:23 +09:30
Rusty Russell f0a9f1100a poetry, CI: insist in protobuf v21.12 in both Python and CI.
And re-ran poetry update which updated the lock file ofc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-23 13:58:31 +09:30
Rusty Russell 69fc9c28bc Makefile: correctly erase generated contrib/pyln-testing/pyln/testing/ in distclean, and rebuild by default.
And add contrib/pyln-testing/pyln/testing/grpc2py.py since we didn't previously know
how to build it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-23 13:58:31 +09:30
Matt Whitlock 8d737cc4bf Makefile: use grouped targets for recipes with multiple fixed outputs
See the section headed "Rules with Grouped Targets" on the Texinfo page
`(make)Multiple Targets`.

Without this fix, Make does not know that these recipes unconditionally
make *all* of their named targets regardless of which target triggers
their execution, and Make will blissfully execute multiple instances of
any such recipe in parallel, not only wasting CPU cycles but potentially
producing incorrect results if the recipe is not atomic in its effects
on the file system.  With this fix, Make understands that it need only
execute such a recipe once to make all of its targets.

In pursuit of the above, move and combine two redundant msggen recipes
into the top-level Makefile, and populate its grouped targets from the
subordinate Makefiles.

Changelog-None
2023-06-23 13:58:31 +09:30
Vincenzo Palazzo fdb676491f pyln-proto: expose the tlv types
Exposing the tlv types to allow public access to it.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-06-23 13:32:25 +09:30
Matt Morehouse 8991f27497 fuzz: initial fuzz-cryptomsg seeds
Corpus generated by 300+ CPU hours of fuzzing and then minimized.
2023-06-23 13:12:42 +09:30
Matt Morehouse 1d94d2cd5d fuzz: test message encryption and decryption
Add a fuzz test for BOLT 8 message encryption and decryption. The fuzz
test is based on the unit test at common/test/run-cryptomsg.c and uses a
static initial state with fuzzer-generated messages to encrypt or
decrypt.
2023-06-23 13:12:42 +09:30
Matt Morehouse fca50eaebc common: make hdr parameter const
cryptomsg_decrypt_header doesn't modify hdr, so we should make it const.
2023-06-23 13:12:42 +09:30
Matt Morehouse 8d9f033954 fuzz: initial handshake target seeds
Corpora generated for fuzz-connectd-handshake-act{1,2,3} from 300+ CPU
hours of fuzzing and then minimized.
2023-06-22 13:40:45 +09:30
Matt Morehouse 0e2173f1fd fuzz: add handshake act 3 target
The fuzz target uses static keys with a fuzzer-generated Act 3 packet.
2023-06-22 13:40:45 +09:30
Matt Morehouse a203db00ca fuzz: add handshake act 2 target
The fuzz target uses static keys with a fuzzer-generated Act 2 packet.
2023-06-22 13:40:45 +09:30
Matt Morehouse 4bb216a611 fuzz: add handshake act 1 target
The fuzz target uses static keys with a fuzzer-generated Act 1 packet.
2023-06-22 13:40:45 +09:30
Matt Morehouse 2b629b2509 fuzz: add connectd_handshake.h
This header will be used by multiple fuzz targets to fuzz Acts 1, 2, and
3 of the BOLT 8 handshake.

We could make this header into a full library, but considering its
narrow use let's try not to over-engineer things.
2023-06-22 13:40:45 +09:30
Rusty Russell 55d6eb6762 CI: run 2, not 3 UBSAN/ASAN pytests at once.
It seems to reliably be getting a SIGTERM after 17-18 minutes:

```
[gw1] [ 91%] PASSED tests/test_plugin.py::test_forward_event_notification
Error: Process completed with exit code 143.
```

Perhaps this is out of mem?  So, try -n2.  We also make the configure
line explicit, rather than relying on environment vars (we should probably
do this for the other cases, too)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-22 13:23:21 +09:30
Rusty Russell ca9a6b15b5 pay: don't require description for hashdesc invoices (i.e. undeprecate).
Since we didn't hash the descriptions properly (see previous commit), we
cannot immediately deprecate omitting the descriptions (since you'd
have to omit them for backwards compat!).

And move the "must have description or hash" test into bolt11.c core.
Changelog-Deprecated: `pay` has *undeprecated* paying a description-hash invoice without providing the description.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-22 13:23:21 +09:30
Rusty Russell 92ff0fd8c0 pay, decodepay: handle descriptions with " inside them where we use hashed descriptions.
This means we need to push off requring this for another full deprecation cycle!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `pay` and `decodepay` with description now correctly handle JSON escapes (e.g " inside description)
2023-06-22 13:23:21 +09:30
Bryan Ramos 7a57f7024c Added sample config file 2023-06-21 14:57:34 +09:30
Rusty Russell 6287f3e3b1 pytest: test_pay flake fix.
This is almost certainly because the HTLCs are not fully settled, so wait for that:

```
2023-06-20T11:37:56.2332158Z         assert apys_2[0]['our_start_balance_msat'] == Millisatoshi(0)
2023-06-20T11:37:56.2332443Z >       assert apys_1[0]['routed_out_msat'] == apys_2[0]['routed_in_msat']
2023-06-20T11:37:56.2332571Z E       assert 1892216msat == 2810170msat
2023-06-20T11:37:56.2332580Z 
2023-06-20T11:37:56.2332717Z tests/test_pay.py:81: AssertionError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-21 13:28:49 +09:30
Rusty Russell 35011337eb pytest: fix unexpected reconnection check in test_setconfig()
If we reconnect before the channel is completely closed, we might get
a "reconnected" message, so mine a block after and make sure it's
processed.

```
2023-06-20T11:37:56.1302058Z         if errors.has_errors():
2023-06-20T11:37:56.1302648Z             # Format a nice list of everything that went wrong and raise an exception
2023-06-20T11:37:56.1303781Z             request.node.has_errors = True
2023-06-20T11:37:56.1304091Z >           raise ValueError(str(errors))
2023-06-20T11:37:56.1304370Z E           ValueError: 
2023-06-20T11:37:56.1304624Z E           Node errors:
2023-06-20T11:37:56.1305042Z E            - lightningd-2: had unexpected reconnections
2023-06-20T11:37:56.1305340Z E           Global errors:
```
...
```
2023-06-20T11:37:56.1960525Z lightningd-2 2023-06-20T11:21:28.638Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Peer has reconnected, state CLOSINGD_SIGEXCHANGE: connecting subd
```
2023-06-21 13:28:49 +09:30
Rusty Russell 263625872e autoclean: allow dynamic changes to autoclean-cycle.
Slightly less trivial: reset timer unless it's currently running callback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell f8f8045a06 autoclean: various configuration options now dynamic.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `autoclean` configuration variables now settable with `setconfig`.
2023-06-20 20:08:25 +09:30
Rusty Russell a65f2ef22a plugins: libplugin support for dynamic config options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell aa5c7e763f libplugin: make set callback for options take plugin ptr, check correct type.
I added a plugin arg and was surprised that compile didn't break.
This is because typesafe_cb et al are conditional casts: if the type
isn't as expected it has no effect, but we're passing plugin_option() through
varargs, so everything is accepted!

Add a noop inline to check type, and fix up the two cases where we
used `const char *` instead of `char *`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell 630dba8840 lightningd: call setconfig on plugins' dynamic options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell b0c07dff2a plugin: allow plugins to set `dynamic` on options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell e50e57712a setconfig: hook into plugin infrastructure for setconfig.
If it's a plugin opt, we'll need a callback, so reshuffle logic.  Also
add infra to map option name to plugin and option.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell 703ffdf59b setconfig: comment out now-overridden lines.
Do it slightly intelligently, so if we had set previously using setconfig
we don't keep appending new ones, but replace it in-place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell 6546be9757 lightningd: setconfig command.
Currently only implemented for min-capacity-sat.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: new command `setconfig` allows a limited number of configuration settings to be changed without restart.
2023-06-20 20:08:25 +09:30
Rusty Russell ecc030f12d lightningd, libplugins: allocate opt strings from tmpctx, not NULL.
Previously, if these failed we always exited; once we have dymamic
configs this would be a (tiny) memory leak, so use tmpctx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell aca893f1eb common: allow configvars to be marked dynamic.
To test, we do min-capacity-sat which is simple.  We also update the
listconfigs man page which contained some obsolete information.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell 9f9f2f1493 lightningd: move listconfigs into configs.c
It mainly clutters up options.c.  The deprecated handling was very tied
to it, so it stays there.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell d09d0112f4 pytest: handle v fast disconnect during test_funding_external_wallet_corners()
We get "disconnected during connection" if we haven't finished processing
the connection when the peer sends error and hangs up.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30
Rusty Russell 771dbca790 ccan: update to fix hang in ./configure with clang.
Also pulls in runes update, but that's harmless.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30
Rusty Russell c1d35cc0c8 Revert "CI: allow postgres and ASAN longer to run."
This reverts commit 2123a057ac9ad37dfc36f003974bf8238de842f2.
2023-06-20 17:24:48 +09:30
Rusty Russell 2abcd8108e pytest: fix race in test_penalty_rbf_normal
We re-enable sendrawtransaction then mine a block to kick off RBF, but there's
a race where it can get a tx in that block, and then we timeout waiting for
both txs to get into the next block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30
Rusty Russell c1d3da889c tools/gossipwith: shutdown, don't close, TCP socket.
This isn't the cause of the test_gossip_ratelimit flake I saw (since
the final gossip msg clearly was received), but it's still good to fix
since it means we might not send the final messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30
Rusty Russell 7820dfdb35 lightningd: tell connectd to disconnect even if it's connecting now.
We were seeing hangs in disconnect in
tests/test_connection.py::test_feerate_stress, and looking at the logs
it's because we're not actually telling connectd to disconnect.

These days, we have a connect counter, so connectd knows to ignore it
if we simply haven't read the message about it already disconnecting.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30
Rusty Russell 0c3757f3a8 CI: allow postgres and ASAN longer to run.
It seems to time out on tests/test_connection.py::test_feerate_stress.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30
Rusty Russell 34b9786076 CI: suppress Postgres vacuuming during tests.
build.sh did this, but we didn't carry it across to the new test
matrix, leading to spurious CI failures:

```
lightningd-1 2023-06-06T16:15:34.931Z DEBUG   plugin-bookkeeper: Setting up database at ***localhost:39061/accounts_0_7ccmg745
lightningd-1 2023-06-06T16:15:34.931Z INFO    plugin-bookkeeper: Creating database
lightningd-1 2023-06-06T16:15:34.931Z **BROKEN** plugin-bookkeeper: Error vacuuming db: VACUUM command failed: ERROR:  deadlock detected\nDETAIL:  Process 77248 waits for AccessShareLock on relation 1260 of database 0; blocked by process 77414.\nProcess 77414 waits for RowExclusiveLock on relation 1214 of database 0; blocked by process 77248.\nHINT:  See server log for query details.\n
lightningd-1 2023-06-06T16:15:34.931Z INFO    plugin-bookkeeper: Killing plugin: exited before replying to init
lightningd-1 2023-06-06T16:15:34.931Z **BROKEN** plugin-bookkeeper: Plugin marked as important, shutting down lightningd!
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 17:24:48 +09:30