Commit Graph

14266 Commits

Author SHA1 Message Date
niftynei df71f2ffa0 coin_mvt: use the `lightning_hrp` for *all* coin movement currency
We were (dumbly?) using the `onchain_hrp` for 'chain_mvts' and the
`lightning_hrp` for 'channel_mvts'.

This works fine everywhere *except* for on a signet, where we use
different prefixes.

Since the lightning-hrp set is more diversified (testnet btc
+ signet btc use the same HRP 'onchain'), let's use that.

Should have zero impact on anything other than nodes running on signet.

To preserve your current accounts database without needing to delete,
restart, execute the following: (note preferrably when your node isn't
running).

```
UPDATE chain_events SET currency = 'tbs' WHERE currency = 'tb';
```

Fixes #6534

Changelog-Fixed: `bkpr-listbalances` would crash for nodes on signet with payments in channels, because onchain events were using a different currency than inchannel events.
2023-12-05 20:30:09 -06:00
niftynei 5a29baafdd startup-regtest: fund_nodes make balanced, funded channels
Changelog-Changed: startup_regtest.sh: `fund_nodes` will now make balanced channels
2023-12-05 01:02:43 -06:00
niftynei a1cecf044c startup-regtest: flag every use as developer
Now that 'developer' isn't a compile time flag, let's always use it
when using the startup regtest nodes.

Should always make gossip + polling bitcoind fast
2023-12-05 01:02:43 -06:00
niftynei 8476a15a44 startup-regtest: add ability to set bitcoin-bin + lightning-dir
We were using similar variables for different things (directories vs binary).

Update this to separate them out, while also adding ability to use a different directory
for lightning nodes

Changelog-Changed: startup_regtest.sh PATH_TO_LIGHTNING + PATH_TO_BITCOIN are no more. Use LIGHTNING_BIN and BITCOIN_DIR
2023-12-05 01:02:43 -06:00
Christian Decker be52f2f21e ci: Force flaky of tests
We are tracking the flakiness of tests [here][1], so no need to suffer
through fail-looping CI while trying to fix a PR or get it merged.

Changelog-None

[1]: http://35.239.136.52:3170/run
2023-12-04 20:22:04 +01:00
Christian Decker a96e64ed56 chore(CI): Add manually triggerable PyPI publication
We used to run the publication on each PR, which could cause conflicts
due to the naming convention being based on the commit history length,
and multiple PRs potentially hitting the same name.

Adding a `workflow_dispatch` trigger allows us to still test the
workflow in PRs but only do so manually (when we're watching anyway).

Changelog-None
2023-12-04 20:22:04 +01:00
Christian Decker 66bf0a7bbb chore(CI): Increase the timeout to publish to test.pypi.org 2023-12-04 20:22:04 +01:00
Peter Neuroth 84f22066f2 misc: Add peter's gpg key
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-None
2023-11-29 12:56:45 +01:00
Peter Neuroth 5887be000c CHANGELOG.md: Update for 23.11 final
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-28 19:01:39 +01:00
Peter Neuroth 9ace27e77b doc: Update example to be valid codex32 string
I guess this was to honor Aditya, but it was also a wrong example.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-28 11:01:26 +01:00
Peter Neuroth 3d51f00147 doc: Add `getemergencyrecover` to the hsmtool doc
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-28 11:01:26 +01:00
Peter Neuroth 9d40a165a0 hsmtool: Fix argc check on getcodexsecret
This caused a segmentation fault when calling `getcodexsecret` without
id.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-Fixed: Hsmtool: Fix segmentation fault when calling `getcodexsecret` without id.
2023-11-28 11:01:26 +01:00
Peter Neuroth 194dd2bb7d CHANGELOG.md: Update for 23.11rc3
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-21 19:14:21 +01:00
Peter Neuroth eb3b1b8134 clnrest: Import sys in except clause explicitly
This fixes a crash on startup of core-lightning where gevent could not
be imported. This happens before sys is imported and throws us into the
except clause which calls sys.
By importing it explicitly in the except clause we are not dependend of
the order of imports in the try bracket.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-Fixes: Plugin: clnrest crashed on startup when gevent was
missing.
2023-11-21 08:24:09 +01:00
Rusty Russell 0b23133ab2 lightningd: don't print out notification msat fields as strings.
Reported-by: Shahana Farooqui
Changelog-Fixed: JSON-RPC: Plugin notification `msat` fields in `invoice_payment` and `invoice_created` hooks now a number, not a string with "msat" suffix.
Changelog-Fixed: JSON-RPC: Plugin hook `payment` `msat` field is now a number, not a string with "msat" suffix.
2023-11-21 08:23:39 +01:00
ShahanaFarooqui fdc7fcbf34 tests/clnrest: Test to confirm that `msat` suffix has been removed from notifications 2023-11-21 08:23:39 +01:00
ShahanaFarooqui 2a7a3d2371 tests/clnrest: Added rpc method name and params to use the same `notifications_received_via_websocket` function for other rpc requests too 2023-11-21 08:23:39 +01:00
Peter Neuroth 8aff6f788f CHANGELOG.md: Update for 23.11rc2
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-20 08:26:58 +01:00
Dusty Daemon a6a9e5b1e3 splice: Reestablish when commit or sig sends fail
Adds tests for when the connection fails during
1) splice tx_signature
2) splice commitment_signed

Fleshed out the reestablish flow for these two cases and implemented the fixes to make these reestablish flows work.

Part of this work required changing commit process for splices: Now we send a single commit_part for the splice where previously we sent all commits, and accordingly, we no longer revoke in response.

Changelog-Fixed: Implemented splicing restart logic for tx_signature and commitment_signed. Splice commitments are reworked in a manner incompatible with the last version.
2023-11-20 07:35:22 +01:00
Rusty Russell eae0d1da1e lightningd: fix up deprecated `rest-port`, `rest-protocol`, `rest-host` and `rest-certs` option if we would otherwise fail.
Since these worked in v23.08, we can't just rename them.  So if they are
used and unclaimed, we should rename them internally (if they're claimed,
it's probably clightning-rest, and we should *NOT* touch them!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `clnrest` parameters `rest-port`, `rest-protocol`, `rest-host` and `rest-certs`: prefix `cln` to them
2023-11-17 10:51:41 +01:00
Peter Neuroth 8f8202daa7 meta: Update release checklist
Add a version bump for clnrest and the main poetry.lock to the makefile
and update the release checklist to include the new target.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-16 15:20:33 +01:00
ShahanaFarooqui cc2665da7f clnrest: prefixing all rest config options with `cln`
This will allow users to use clnrest with c-lightning-REST without conflicts.
It was required for applications to have enough time for migrating from c-lightning-REST to clnrest.

Changelog-Changed:
config option `rest-certs` changed to `clnrest-certs`
config option `rest-protocol` changed to `clnrest-protocol`
config option `rest-host` changed to `clnrest-host`
config option `rest-port` changed to `clnrest-port`
config option `rest-cors-origins` changed to `clnrest-cors-origins`
config option `rest-csp` changed to `clnrest-csp`
2023-11-16 11:21:04 +01:00
ShahanaFarooqui 017581cd79 doc: update decode instruction for emergency.recover
Related to:
https://github.com/ElementsProject/lightning/issues/6711
https://github.com/ElementsProject/lightning/pull/6773

Changelog-None
2023-11-16 09:13:34 +01:00
ShahanaFarooqui 9e9c3dbc00 clnrest: add gevent ssl monkey patch to remove warning log 2023-11-16 09:07:07 +01:00
Rusty Russell 8b84c8945b clnrest: don't convert *_msat fields to strings.
We have a global JSON encoder hack, which means that any field ending in msat gets special treatment (so we can safely talk to lightningd, even if a field expects satoshi amounts, we are explicit).  However, requests uses the JSON parser and DOES NOT want this conversion when sending it out as an HTTP response!

The simplest local fix we could find was Shahana's suggestion to iterate and covert away from Millisatoshi(): the reverse of what our JSON encoder does.

Fixes: https://github.com/ElementsProject/lightning/issues/6848
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-16 09:07:07 +01:00
Rusty Russell f2f6c73977 clnrest: don't try to handle plugin.rpc.call results.
It already throws an exception or error, or decodes the JSON response:
we can simply pass it through.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-16 09:07:07 +01:00
Tony Aldon b3f122a2c5 pytest: adds test suite for clnrest plugin
- cln-grpc certificate reuse
- new certificate generation
- `GET` and `POST` requests
- websocket server
- config options and HTTP headers

Link to #6436.
2023-11-16 09:07:07 +01:00
Tony Aldon 672477c95c poetry: add `requests` dependency for clnrest tests in contrib/pyln-testing
`poetry update` did not change the poetry.lock file for contrib/pyln-testing
because it already has `requests`.
2023-11-16 09:07:07 +01:00
Christian Decker 94f4d75006 msggen: Add `Wait` method to managed API
Fixes #6792

Changelog-Added: Cln-RPC: Add `wait` system to cln-rpc and cln-grpc.
2023-11-16 09:02:33 +01:00
Christian Decker bdfd2d5fda msggen: Add `FetchInvoice` method to Rust and grpc bindings
Closes #6844

Changelog-Added: Cln-RPC: Add `fetchinvoice` method to cln-rpc and cln-grpc.
2023-11-16 09:02:33 +01:00
Ken Sedgwick 9e0b244058 splice: fix WIRE_HSMD_SIGN_SPLICE_TX capability check
Fixes #6866

Changelog-Fixed: The WIRE_HSMD_SIGN_SPLICE_TX HSM capability is now correctly checked.
2023-11-15 18:42:08 +10:30
Tony Aldon cce4f68f7a doc: fix cln-grpc example.
- Add instruction to generate `primitives_pb2.py` file,
- Import `grpc` in the Python script.

Changelog-None
2023-11-14 10:48:41 +01:00
Peter Neuroth 02ca226f88 CHANGELOG.md: Update for 23.11rc1
Update changelog, pyln versions, clnrest version and poetry.lock.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-03 14:49:12 +01:00
Peter Neuroth b135ba2224 reprobuild: Upgrade rust version
We need to upgrade the rust version of the ubuntu docker images as
time-core v0.1.2 needs a version newer than 1.67.0

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-02 19:52:46 +01:00
Peter Neuroth 1546127b23 docker: Add fix for broken setuptools installation
The fedora image installs a setuptools version that can not be upgraded
via pip. A fix is to remove the files manually and reinstall it via pip.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-02 19:52:46 +01:00
Ken Sedgwick 88a536f3eb Possible fix for CI_SERVER issue in VLS CI
Code using `CI_SERVER` was added recently to track something.  It
attempts to short-cut if `CI_SERVER` is not set, but on gitlab it
is set to `yes`.  https://docs.gitlab.com/ee/ci/variables/

Instead use `CI_SERVER_URL`, maybe that is what is intended?
2023-11-02 13:53:03 +01:00
niftynei 3190c26bc9 dualfund: error on out of order sigs
We weren't blocking if the tx-sigs arrived before the commitment sigs.

This was causing problems in the openchannel (spender plugin)

spenderp: FATAL SIGNAL 11 (version v23.08.1-404-g62ff475-modded)
0x559836dc98ba send_backtrace
	common/daemon.c:33
0x559836dc9951 crashdump
	common/daemon.c:75
0x7f37f42c351f ???
	./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x7f37f441ac92 ???
	../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S:83
0x559836db7760 bitcoin_txid_eq
	./bitcoin/tx.h:29
0x559836db7760 collect_sigs
	plugins/spender/openchannel.c:509
0x559836db81de check_sigs_ready
	plugins/spender/openchannel.c:531
0x559836db84dd json_peer_sigs
	plugins/spender/openchannel.c:611
0x559836dbcad7 ld_command_handle
	plugins/libplugin.c:1611
0x559836dbcd9d ld_read_json_one
	plugins/libplugin.c:1721
0x559836dbce29 ld_read_json
	plugins/libplugin.c:1741
0x559836ef3bff next_plan
	ccan/ccan/io/io.c:59
0x559836ef40da do_plan
	ccan/ccan/io/io.c:407
0x559836ef4177 io_ready
	ccan/ccan/io/io.c:417
0x559836ef5b14 io_loop
	ccan/ccan/io/poll.c:453
0x559836dbd48d plugin_main
	plugins/libplugin.c:1948
0x559836db22bf main
	plugins/spender/main.c:35
0x7f37f42aad8f __libc_start_call_main
	../sysdeps/nptl/libc_start_call_main.h:58
0x7f37f42aae3f __libc_start_main_impl
	../csu/libc-start.c:392
0x559836da3774 ???
	???:0
0xffffffffffffffff ???
	???:0
2023-10-31T15:15:57.458Z INFO    plugin-spenderp: Killing plugin: exited during normal operation
2023-10-31T15:15:57.458Z **BROKEN** plugin-spenderp: Plugin marked as important, shutting down lightningd!
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_break: lightningd_exit
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_loop_with_timers: main
2023-10-31T15:15:57.458Z DEBUG   connectd: REPLY WIRE_CONNECTD_START_SHUTDOWN_REPLY with 0 fds
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_break: connectd_start_shutdown_reply
2023-10-31T15:15:57.458Z DEBUG   021ccce7bc396996c8f3b7bfeb1e30c6600269517026a74adfe2217b7187879797-dualopend-chan#1: Status closed, but not exited. Killing
2023-10-31T15:15:57.458Z DEBUG   lightningd: Command returned result after jcon close
2023-10-31T15:15:57.458Z INFO    021ccce7bc396996c8f3b7bfeb1e30c6600269517026a74adfe2217b7187879797-chan#1: Unsaved peer failed. Deleting channel.
2023-10-31T15:15:57.464Z DEBUG   lightningd: io_break: destroy_plugin
2023-10-31T15:15:57.464Z DEBUG   connectd: Shutting down
2023-10-31T15:15:57.464Z DEBUG   gossipd: Shutting down
2023-10-31T15:15:57.464Z DEBUG   hsmd: Shutting down

Reported-By: @t-bast
2023-11-02 19:32:05 +10:30
niftynei fa8458c00a dualfund: add test to make sure that tx-sigs sent before commitment
results in an error.
2023-11-02 19:32:05 +10:30
niftynei 30ec8cbf8e dualfund, test: add test for dropping to chain during RBF
Here we make sure we can drop the initial tx to chain, and that an
inflight txid that's missing its commitment sigs is properly ignored.
2023-11-02 19:32:05 +10:30
niftynei f4b4f772f3 dualfund, bump: when bumping a channel make sure it's in ok state
If we disconnect, we lose the open_attempt record. Which is fine, but we
should prevent the user from starting another RBF if the last one isn't
done yet!
2023-11-02 19:32:05 +10:30
niftynei dbcdfd7d66 dualfund, memleak: don't leak the msg on error
We don't let go of the `msg` on error, which triggers a memleak warning!

lightningd-2 2023-10-31T19:54:06.582Z **BROKEN** lightningd: MEMLEAK: 0x55ae3615b498
lightningd-2 2023-10-31T19:54:06.582Z **BROKEN** lightningd:   label=openingd/dualopend_wiregen.c:919:u8[]
lightningd-2 2023-10-31T19:54:06.582Z **BROKEN** lightningd:   alloc:
lightningd-2 2023-10-31T19:54:06.685Z **BROKEN** lightningd:     ccan/ccan/tal/tal.c:477 (tal_alloc_)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     ccan/ccan/tal/tal.c:506 (tal_alloc_arr_)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     openingd/dualopend_wiregen.c:919 (towire_dualopend_send_tx_sigs)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     lightningd/dual_open_control.c:1122 (openchannel2_sign_hook_cb)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     lightningd/plugin_hook.c:194 (plugin_hook_call_next)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin_hook.c:169 (plugin_hook_callback)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin.c:660 (plugin_response_handle)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin.c:772 (plugin_read_json_one)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin.c:823 (plugin_read_json)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/io.c:59 (next_plan)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/io.c:407 (do_plan)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/io.c:417 (io_ready)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/poll.c:453 (io_loop)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:     lightningd/lightningd.c:1333 (main)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:     ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:     ../csu/libc-start.c:392 (__libc_start_main_impl)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:   parents:
2023-11-02 19:32:05 +10:30
niftynei da34c369f3 dualfund, tests: break out "peer forgets" test
Now that we save the commitment sigs immediately, we have to drop the
connection elsewhere in the flow to get the state where only one peer
remembers.
2023-11-02 19:32:05 +10:30
niftynei 89f6fd27e3 dual-fund: have accepter send their commitment sigs asap
Originally the accepter waited for the peer to send us their commitment
sigs before we send ours; this changes things so that the accepter
sends their commitment sigs ASAP.

	This test fails: when cln is not the channel initiator, it waits for the other node to send commit_sig before sending its own commit_sig. There is no reason to do that, both nodes should send commit_sig immediately after exchanging tx_complete? Otherwise it's a missed opportunity to finalize the channel creation on reconnection, because in that case cln hasn't saved the channel and fails it on reconnection.

Reported-By: @t-bast
2023-11-02 19:32:05 +10:30
niftynei 48bb2d831b dual-fund: don't re-notify plugin on arrival of sigs (2nd time)
When we got our peer's sigs, if we were the remote, we would re-notify
the plugin, which in turn would re-send the tx-sigs to use.

In the case of CLN, we'd then
- break, because we'd re-forward the sigs to the `openchannel` plugin,
  which was then in the wrong state (MULTIFUNDCHANNEL_SIGNED)

    spenderp: plugins/spender/openchannel.c:598: json_peer_sigs: Assertion `dest->state == MULTIFUNDCHANNEL_SECURED' failed.
    spenderp: FATAL SIGNAL 6 (version 5880d59-modded)

In the case of eclair, they'd just see our 2nd TX_SIGS message and
@t-bast would complain:

	> This test works, with one minor issue: on reconnection, cln sends its tx_signatures twice (duplicate?).

This commit does two things:
	- has the openchannel / spender plugin log a broken instead of
	  crashing when the state is not what we're expecting
	- stops us from calling the `funder` plugin if this is a
	  replay/second receipt of commit-sigs.
2023-11-02 19:32:05 +10:30
niftynei 5417312911 tests: update opening tests for new reconnect behavior
Let's test that things stay together!

One cool thing to note is that now we sort of "magically" recover from
pretty brutal disconnects!

Very nice!
2023-11-02 19:32:05 +10:30
niftynei 4e63d36e08 mfc, nit: print out the error reason when an open fails
Makes it easier to see why things are failing in the logs.
2023-11-02 19:32:05 +10:30
niftynei 6771518e31 dualfund, reconnects: update dual-fund to use next-funding-id
Here we conform to the specification, which requires that we handle
next-funding-id in a specific way.

Note that we were already sending it, but now we actually correctly
handle its presence.

Changelog-Changed: Spec: dual-funding now follows the next-funding-id rules.
2023-11-02 19:32:05 +10:30
niftynei b2d2796aad dualfund, tx-abort: only check for abort state if we're sending
In the case where you're echoing back a tx-abort, just let it through.

Not doing this causes problems in the case where your node has forgotten
about an in-progress open.

This fixes the following problem:

- you send a tx-abort (even tho you have marked tx-sigs as received)
- peer echos it back (we echo back tx-aborts always)
- you throw an error because you're already in a tx-abort unallowed
  state

In this commit, we allow for echos to come thru no matter our current state and
this fixes things/makes them work as expected.
2023-11-02 19:32:05 +10:30
niftynei 979276386a dualfund: update handling of tx-sigs
If you get the right series of disconnects, it's possible for your peer
to send you a tx-sigs even though the current state of the channel open
is that you've seen the funding open on chain (your channel_ready[LOCAL]
= true)

In this case, if we haven't marked that we've seen the tx sigs yet,
we go ahead and mark them as seen and just ignore this tx-sigs msg.
2023-11-02 19:32:05 +10:30
niftynei 5d195710f6 dualfund: handle commitment signed
If we get a commitment-signed message from a peer, outside of a normal
flow, process it!

We're about to send these during reconnect, so we need to be able to
handle them!
2023-11-02 19:32:05 +10:30