Commit Graph

11000 Commits

Author SHA1 Message Date
Christian Decker 22618a2f94 cln-plugin: Rework the plugin library using a Builder 2022-03-10 10:21:41 +10:30
Christian Decker 4aba119733 pytest: Use valgrind target suppressions instead of skipping tests
Having a list of very targeted suppressions allows us to still run the
majority of tests with valgrind checking, and not fail when Rust does
some trickery. This is for example the case with `std::sync::Once`
which uses `num_procs` calling out to the cgroups subsystem, sometimes
with a null path.

Suggested-by: Rusty Russell <@rustyrussell>
2022-03-10 10:21:41 +10:30
Christian Decker a1555623bc pytest: Mark Rust-dependent tests as skipped with VALGRIND
`valgrind` reports seems to flag some memory accesses that are ok in
the Rust standard library, which we can consider false positives for
our purposes:

```Valgrind error file: valgrind-errors.69147
==69147== Syscall param statx(file_name) points to unaddressable byte(s)
==69147==    at 0x4B049FE: statx (statx.c:29)
==69147==    by 0x2E2DA0: std::sys::unix::fs::try_statx (weak.rs:139)
==69147==    by 0x2D7BD5: <std::fs::File as std::io::Read>::read_to_string (fs.rs:784)
==69147==    by 0x2632CE: num_cpus::linux::Cgroup::param (linux.rs:214)
==69147==    by 0x263179: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==69147==    by 0x263002: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==69147==    by 0x262C01: num_cpus::linux::load_cgroups (linux.rs:149)
==69147==    by 0x26289D: num_cpus::linux::init_cgroups (linux.rs:129)
==69147==    by 0x26BD88: core::ops::function::FnOnce::call_once (function.rs:227)
==69147==    by 0x26B749: std::sync::once::Once::call_once::{{closure}} (once.rs:262)
==69147==    by 0x139717: std::sync::once::Once::call_inner (once.rs:419)
==69147==    by 0x26B6D5: std::sync::once::Once::call_once (once.rs:262)
==69147==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==69147==
==69147== Syscall param statx(buf) points to unaddressable byte(s)
==69147==    at 0x4B049FE: statx (statx.c:29)
==69147==    by 0x2E2DA0: std::sys::unix::fs::try_statx (weak.rs:139)
==69147==    by 0x2D7BD5: <std::fs::File as std::io::Read>::read_to_string (fs.rs:784)
==69147==    by 0x2632CE: num_cpus::linux::Cgroup::param (linux.rs:214)
==69147==    by 0x263179: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==69147==    by 0x263002: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==69147==    by 0x262C01: num_cpus::linux::load_cgroups (linux.rs:149)
==69147==    by 0x26289D: num_cpus::linux::init_cgroups (linux.rs:129)
==69147==    by 0x26BD88: core::ops::function::FnOnce::call_once (function.rs:227)
==69147==    by 0x26B749: std::sync::once::Once::call_once::{{closure}} (once.rs:262)
==69147==    by 0x139717: std::sync::once::Once::call_inner (once.rs:419)
==69147==    by 0x26B6D5: std::sync::once::Once::call_once (once.rs:262)
==69147==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==69147==

```
2022-03-10 10:21:41 +10:30
Christian Decker fbcb4c33ad cln-plugin: Populate the options when we get an `init` call 2022-03-10 10:21:41 +10:30
Christian Decker 249fa8675a cln-plugin: Add options to the `getmanifest` call 2022-03-10 10:21:41 +10:30
Christian Decker fe21b89b56 pytest: Add a test for the cln-plugin logging integration 2022-03-10 10:21:41 +10:30
Christian Decker f5e1829117 cln-plugin: Implement logging facade adapter for cln plugins
We wrap emitted messages into a JSON-RPC notification envelope and
write them to stdout. We use an indirection over an mpsc channel in
order to avoid deadlocks if we emit logs while holding the writer lock
on stdout.
2022-03-10 10:21:41 +10:30
Christian Decker 9ae1f33992 cln-plugin: Get started with the plugin interface 2022-03-10 10:21:41 +10:30
Vincenzo Palazzo c0c826d2ee ci: introduce in ci the compilation testing on different os
Changelog-None: introduce in ci the compilation testing on different os

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-10 09:44:16 +10:30
Rusty Russell d7ffb712e5 dualopend: restore memleak calls.
And implement a timeout (20 seconds) just in case it's not listening.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-10 09:40:09 +10:30
Rusty Russell 84e0e743eb dualopend: fix memleak reports.
Next patch re-enables runtime leak checking for dualopend, so fix those
leak reports.

In some cases, this menas allocating off tmpctx or state->channel
(which gets reset on failure), not state.  The problem with tmpctx is
that there are event loops in the *middle* of some functions, which
free it.  So for RBF functions we use a rbf_ctx temporary (with leak
detection suppressed, like it is for tmpctx), then be careful to free
it on all exits!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-10 09:40:09 +10:30
Rusty Russell 00bb6f07d7 lightningd: simplify memleak code.
Instead of doing this weird chaining, just call them all at once and
use a reference counter.

To make it simpler, we return the subd_req so we can hang a destructor
off it which decrements after the request is complete.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-10 09:40:09 +10:30
Christian Decker 0c4cc782df docker: Clean up the dockerfile to use poetry 2022-03-08 05:00:36 +10:30
Christian Decker 81259bbd67 py: Update mrkd 2022-03-08 05:00:36 +10:30
Christian Decker ad22becb26 pyln: Switch to binary psycopg2 2022-03-08 05:00:36 +10:30
Christian Decker 60135cf9ba pyln: MIgrate the BOLT packages to poetry / PEP 517 2022-03-08 05:00:36 +10:30
Christian Decker 8857789a54 gci: Only test pushes if it's `master`
Should avoid many of the duplicate test runs
2022-03-08 05:00:36 +10:30
Christian Decker 941f217778 gci: Fix up the MacOS test with poetry 2022-03-08 05:00:36 +10:30
Christian Decker fa183b8119 python: Exclude newly detected f-string flake 2022-03-08 05:00:36 +10:30
Christian Decker f4fda04fb6 misc: Make the spell checker just a tiny bit less trigger happy 2022-03-08 05:00:36 +10:30
Christian Decker 978711699a gci: Switch to python 3.7
Since the minimum requirement for pyln-testing is 3.7 we better test
with that version at least.
2022-03-08 05:00:36 +10:30
Christian Decker 70840ef066 gci: Switch docs and CI builder to use `poetry` 2022-03-08 05:00:36 +10:30
Christian Decker d51f97a9e4 gci: Do not generate JSON reports for test runs
We've not been using them, so let's not generate them.
2022-03-08 05:00:36 +10:30
Christian Decker 49d2779078 pyln: Use poetry and PEP 517 locks for the root project
Allows us to just rely on `poetry` to manage our dependencies, should
finally solve all the `mrkd`, `mako` and `mistune` issues we've had
for a while.
2022-03-08 05:00:36 +10:30
Christian Decker 4b9bf22193 pyln: Migrate pyln-testing to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker dd8d2c138c pyln: Migrate pyln-client to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker b0f8a99310 pyln: Migrate pyln-bolt7 to PEP517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker 74fd685219 pyln: Fix the pyln-proto version and migrate to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Rusty Russell a3696f0469 gossipd: allow faster updating, 2 per day, before ratelimiting.
This limit applies to both node_announcements (which we now send 1 per
day), and channel_updates; I've had reports of LND nodes going down
daily for database compation, so they end up ratelimited.

Changelog-Protocol: We now allow two channel_updates or node_announcements per day, up from 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-08 04:54:45 +10:30
Rusty Russell 93c00cc07d gossipd: force a fresh node_announcement every 24 hours.
Even if nothing has changed.  Note that this is different from simply
re-xmitting the old one, in that it has a different timestamp, so others
will re-xmit it too.

This is a side-effect of reports that node_announcement propagation
through the network is poor:

	https://github.com/ElementsProject/lightning/issues/5037

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: We now refresh our ndoe_announcement every 24 hours, due to propagation issues.
2022-03-08 04:54:45 +10:30
Rusty Russell bf8cb640b7 gossipd: always re-sent our own node_announcement on startup.
Even if it hasn't changed, re-send it to everyone.  That should help
propagation a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-08 04:54:45 +10:30
Rusty Russell d991f13516 gossipd: make sure we set the `urgent` bit if we move our own node_announcement.
node_announcement has to follow at least one channel_announcement.
When channels close, if this isn't the case, we remove the old
node_announcement and put it at the end of the gossip_store.

But we lose the "send even if they don't want it" bit in the case it's
our own node_announceent, so keep it.

This only happens if you don't change your node configuration at all
since you opened your first channel, but still worth fixing.

We expose the force_node_announce_rexmit() for later use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-08 04:54:45 +10:30
Rusty Russell b5a1715c2b connectd: also fail without a scary backtrace when listen fails.
For example, if you do:

```
./lightningd/lightningd --network=regtest --experimental-websocket-port=19846
```

Then you're trying to reuse the normal port as the websocket port, but this
only fails at *listen* time, when we activate connectd.  Catch this too.

Fixes incorrect fatal() message, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell 885a6f50ae connectd: make sure we announce websocket addr which succeeded.
By accessing `addr` after the loop, it's possible that it's one which
failed, in complex scenarios.

Also gives us a chance to warn if they specify a websocket but don't
actually end up advertizing it (you *must* advertize a normal addr as
well).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell c075d78431 connectd: use listen_fd array directly, rather than returning binding arr.
We always added to both arrays, might as well just keep one.  

We make mayfail an explicit flag, rather than relying on the presence
of errstr, which is never NULL now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell a62f5e5d82 connectd: hoist find_local_address so we can give more graceful Tor erros.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell 200a8a985b connectd: add is_websocket and wireaddr to struct listen_fd.
This lets us give a better error message if listen fails, and also
moved the callback closer to where it's needed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell f1ed373c97 connectd: be more graceful when we an address is in use.
Aditya had this issue due to a config line, and the result was hard
to diagnose even for me.

It's now:

```
$ ./lightningd/lightningd --network=regtest --addr=:18444
2022-02-26T05:01:28.705Z **BROKEN** connectd: Failed to bind socket for 0.0.0.0:18444: Address already in use
```

Whereas before it doesn't even give the address it's trying to bind:

```
rusty@rusty-XPS-13-9370:~/devel/cvs/lightning (master)$ ./lightningd/lightningd --network=regtest --addr=:18444
lightning_connectd: Failed to bind on 2 socket: Address already in use (version v0.10.2-331-g86b83e4)
0x558a8b8d9a12 send_backtrace
	common/daemon.c:33
0x558a8b8e91e1 status_failed
	common/status.c:221
0x558a8b8c8e4f make_listen_fd
	connectd/connectd.c:1090
0x558a8b8c8f55 handle_wireaddr_listen
	connectd/connectd.c:1129
0x558a8b8c993d setup_listeners
	connectd/connectd.c:1312
0x558a8b8ca344 connect_init
	connectd/connectd.c:1517
0x558a8b8cbb57 recv_req
	connectd/connectd.c:1896
0x558a8b8d9f9f handle_read
	common/daemon_conn.c:31
0x558a8b9247c1 next_plan
	ccan/ccan/io/io.c:59
0x558a8b9253c9 do_plan
	ccan/ccan/io/io.c:407
0x558a8b92540b io_ready
	ccan/ccan/io/io.c:417
0x558a8b9276fe io_loop
	ccan/ccan/io/poll.c:453
0x558a8b8cbf36 main
	connectd/connectd.c:2033
0x7fe4d02940b2 ???
	???:0
0x558a8b8c285d ???
	???:0
0xffffffffffffffff ???
	???:0
2022-02-26T05:02:27.547Z **BROKEN** connectd: Failed to bind on 2 socket: Address already in use (version v0.10.2-331-g86b83e4)
2022-02-26T05:02:27.547Z **BROKEN** connectd: backtrace: common/daemon.c:38 (send_backtrace) 0x558a8b8d9a68
2022-02-26T05:02:27.547Z **BROKEN** connectd: backtrace: common/status.c:221 (status_failed) 0x558a8b8e91e1
2022-02-26T05:02:27.547Z **BROKEN** connectd: backtrace: connectd/connectd.c:1090 (make_listen_fd) 0x558a8b8c8e4f
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1129 (handle_wireaddr_listen) 0x558a8b8c8f55
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1312 (setup_listeners) 0x558a8b8c993d
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1517 (connect_init) 0x558a8b8ca344
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1896 (recv_req) 0x558a8b8cbb57
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: common/daemon_conn.c:31 (handle_read) 0x558a8b8d9f9f
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x558a8b9247c1
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x558a8b9253c9
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x558a8b92540b
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x558a8b9276fe
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:2033 (main) 0x558a8b8cbf36
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fe4d02940b2
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x558a8b8c285d
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2022-02-26T05:02:27.548Z **BROKEN** connectd: STATUS_FAIL_INTERNAL_ERROR: Failed to bind on 2 socket: Address already in use
lightningd: connectd failed (exit status 242), exiting.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
niftynei 9075c74e3d balance_snapshot: don't count unconfirmed utxos
This was causing journal_entries to show up in the accountant plugin,
since we don't emit events for unconfirmed events until they're actually
confirmed onchain.
2022-03-05 15:12:12 +10:30
niftynei b33868f7c2 coin_mvt: only chain moves have a blockheight
channel moves don't have blockheights
2022-03-05 15:12:12 +10:30
niftynei 7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei 590f12145b listpeers: include the `pushed_msat` amount
Useful for accounting for missed/historical channel opens, to figure out
what the actual sat contribution from each peer is at a utxo level

Changelog-Added: JSONRPC: `listpeers` now includes a `pushed_msat` value. For leased channels, is the total lease_fee.
2022-03-05 15:12:12 +10:30
niftynei ecb19ba6f2 coin_mvt: report mutual close outputs also
It's better to report every single utxo on close so we know when
to mark a channel account as definitively closed.
2022-03-05 15:12:12 +10:30
niftynei d246e9c1a2 tx_parts: as a convenience, copy over the value as the 'satoshi'
Reporting coin movements was crashing for liquid-regtest tests because
we were using an un-initialized field (the tx_part output's satoshi
field).

We fill this in 'as a convenience' for other wally_tx_outputs that are
on liquid elsewhere, here we do the same for tx_parts sent over the
wire to onchaind.
2022-03-05 15:12:12 +10:30
niftynei 7a00277b43 coin_mvt: mark coins destined for wallet w/ originating acct
Only shows up on delayed to us outputs, but nice to have anyway.

It's missing for channel index destined deposits, maybe nice to add at
some point in the future; right now you can figure out which close a
wallet deposit comes from via the channel close txid
2022-03-05 15:12:12 +10:30
niftynei 24a1c91045 coin_mvt: report the number of outputs on a channel close tx
The bookkeeper needs to know how many outputs to expect before we can
consider a channel resolved onchain.
2022-03-05 15:12:12 +10:30
niftynei 12cbf614cd coin_mvt: mark every event that already has a destination
Every event that's coming out of here that specifies a different place
to be deposited should be marked as originating from this channel's
 account.
2022-03-05 15:12:12 +10:30
niftynei b30328a91f coin_mvt: rm unncessary if statement
We don't need to switch on this; if it's zero the below ops will be
effectiely no-ops.
2022-03-05 15:12:12 +10:30
niftynei ea36c3a938 db: collapse the db_queries files into the generated ones
There's no reason to have these be independent of the generated files,
all the data contained within them is 'formulaic'
2022-03-05 15:03:34 +10:30
niftynei 453ef7f0ce db: mark that column is unused/ignored
2022-01-25T23:41:45.2994844Z ----------------------------- Captured stderr call -----------------------------
2022-01-25T23:41:45.2995230Z lightningd: Never accessed column 1 in query SELECT 1  FROM offers WHERE offer_id = ?;
2022-03-05 15:03:34 +10:30