Commit Graph

10996 Commits

Author SHA1 Message Date
Rusty Russell 3217dbe17f channeld: get htlc_maximum_msat from lightningd.
We used to calculate it ourselves.  Unfortunately this needs to
be done in several places, since new_channel() isn't used to fully
create a channel in the case of dual funding :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell 6fdcc86f9d lightningd: store htlc_maximum_msat for channel in the db.
We currently don't allow setting it, but it's been requested.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell 953f238bd2 connectd: use closefrom for faster forking, and ignore children
Zombie sighting fom jb55.

Fixes: #5092
Changelog-EXPERIMENTAL: Fixed `experimental-websocket-port` not to leave zombie processes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 21:07:26 +10:30
Rusty Russell cf55d40eca psbt: fix dual-funding memleak.
This happened occasionally in tests/test_opening.py::test_rbf_fails_to_broadcast:
we overwrote the witness stack without freeing the old one.

```
lightningd-2: 2022-03-18T02:23:32.113Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1: MEMLEAK: 0x559c375d9df8
lightningd-2: 2022-03-18T02:23:32.113Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   label=wally_tal
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   backtrace:
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     common/setup.c:13 (wally_tal)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/internal.c:285 (wally_calloc)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/transaction.c:234 (wally_tx_witness_stack_init_alloc)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/psbt.c:1119 (pull_psbt_input)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     ../../../libwally-core/src/psbt.c:1411 (wally_psbt_from_bytes)
lightningd-2: 2022-03-18T02:23:32.114Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     bitcoin/psbt.c:722 (psbt_from_bytes)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     bitcoin/psbt.c:753 (fromwire_wally_psbt)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend_wiregen.c:246 (fromwire_dualopend_reinit)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend.c:3855 (main)
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:   parents:
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     struct wally_psbt
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-dualopend-chan#1:     openingd/dualopend.c:3804:struct state
lightningd-2: 2022-03-18T02:23:32.115Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba932
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 10:19:50 +10:30
Rusty Russell 65be18d355 memleak: handle libwally allocations better.
Things allocated by libwally all get the tal_name "wally_tal",
which cost me a few hours trying to find a leak.

In the case where we're making one of the allocations the parent
of the others (e.g. a wally_psbt), we can do better: supply a name
for the tal_wally_end().

So I add a new tal_wally_end_onto() which does the standard
tal_steal() trick, and also changes the (typechecked!) name.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-21 10:19:50 +10:30
Ken Sedgwick 8994b8dade hsmd: Add validate_commitment_tx 2022-03-20 13:06:16 +10:30
Vincenzo Palazzo 8b04bf1357 lnprototest: upgrade the lnprototest version
Changelog-None: upgrade the lnprototest version

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-20 08:46:32 +10:30
Rusty Russell 278ea6c8ac memleak: make sure we catch children which are also "notleak".
By not recursing into "notleak" children, we could miss other "notleak"
pointers:

```
2022-03-15T05:25:33.9759500Z lightningd-1: 2022-03-15T05:00:15.998Z **BROKEN** connectd: MEMLEAK: 0x55b29c3d0cc8
2022-03-15T05:25:33.9759901Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:   label=common/timeout.c:22:struct oneshot **NOTLEAK**
2022-03-15T05:25:33.9760195Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:   backtrace:
2022-03-15T05:25:33.9760555Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
2022-03-15T05:25:33.9760905Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     common/timeout.c:22 (new_reltimer_)
2022-03-15T05:25:33.9761272Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     connectd/multiplex.c:558 (set_closing_timer)
2022-03-15T05:25:33.9761647Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     connectd/multiplex.c:586 (write_to_peer)
2022-03-15T05:25:33.9761986Z lightningd-1: 2022-03-15T05:00:16.003Z **BROKEN** connectd:     ccan/ccan/io/io.c:59 (next_plan)
2022-03-15T05:25:33.9762335Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/io.c:435 (io_do_always)
2022-03-15T05:25:33.9762692Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/poll.c:304 (handle_always)
2022-03-15T05:25:33.9763035Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/poll.c:385 (io_loop)
2022-03-15T05:25:33.9763376Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     connectd/connectd.c:2131 (main)
2022-03-15T05:25:33.9763645Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:   parents:
2022-03-15T05:25:33.9764020Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     ccan/ccan/io/io.c:91:struct io_conn **NOTLEAK**
2022-03-15T05:25:33.9764471Z lightningd-1: 2022-03-15T05:00:16.004Z **BROKEN** connectd:     connectd/connectd.c:2104:struct daemon
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-20 08:42:57 +10:30
Aaron Dewes 92fd97627d fix: Remove \ from help documentation 2022-03-20 08:42:20 +10:30
Jules Comte 8f0aab7061 doc: update ubuntu install instructions
Changelog-None
2022-03-18 13:09:37 +10:30
Rusty Russell 43a833e405 lightningd: remove support for legacy onion format.
As per proposal in https://github.com/lightning/bolts/pull/962

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
2022-03-18 09:20:11 +10:30
Rusty Russell 072c4711ec pytest: convert test_sendonion_rpc to modern TLV onion.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-18 09:20:11 +10:30
Rusty Russell 45143cc731 pytest: Remove onion test vectors containing legacy onions.
I thought about fixing them up, but really these should be in
lnprototest anyway.  Turns out they're from the spec, so we should
actually fix them up there.

I moved the vector files into contrib/pyln-proto, since that still
needs them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-18 09:20:11 +10:30
Rusty Russell 0db05f6e9c lightningd: opt_var_onion is now a compulsory feature.
We're about to drop support for legacy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-18 09:20:11 +10:30
Ken Sedgwick 36466af3eb hsmd: Add fields to hsmd_sign_{,remote_}commitment_tx for validating signers 2022-03-16 12:06:21 +10:30
Christian Decker 704162f24a docker: Build the docker image with psql support
Changelog-Fixed: docker: The docker image is now built with postgresql support
2022-03-16 11:25:41 +10:30
Rusty Russell 508b8cb54a doc: fix sendonionmessage documentation.
It's a low-level interface now, expecting you to build your own TLVs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @valentinewallace
2022-03-16 11:24:42 +10:30
Christian Decker a8aa9bd5ae cln-rpc: Fixed two minor compiler warnings 2022-03-16 11:11:58 +10:30
Christian Decker 487b5e6169 msggen: Add meta file to keep the field numbers stable
We are inferring the field numbers on the fly, which isn't really
compatible with the way GRPC field numbers work, i.e., they must be
stable while the IDL file evolves. So far when a field was added in
the middle of a struct or removed all subsequent fields would get
renumbered, essentially breaking any client that was using the old
scheme.

We now add a meta file `.msggen.json` that keeps track of the numbers
assigned so far, so they can be reused, and new ones can be generated
not to conflict with existing ones. This file is intentionally kept
generic, so other generators can add more information that has to be
managed across runs.

Changelog-None
2022-03-16 11:11:58 +10:30
Christian Decker e99b11565c pyln: Change the default argument to `exclude` in `pay`
We must use `None` for default arguments since otherwise they aren't
filtered out when serializing the request. In addition default
arguments to functions are initialized once and, if mutable, could
persist internal changes across function calls.

Changelog-None
2022-03-15 12:20:35 -05:00
William Casarin f72a08c802 websocketd: fix random failures by blocking stdin reads
Example request that is dying:

NEW REQUEST! lightning_websocketd:main [1955685] <-- bad request from safari

   read 507
   write_all 1
-> websocket_to_lightningd
-> read_payload_header
   read 2
   read_all 1
   read -11      <--- This tried to read a part of the header, is this -EAGAIN?
   read_all 0         should we be blocking on these reads?
   *dies*

Fixes #5089

Changelog-Fixed: `experimental-websocket` intermittent read errors fixed
Signed-off-by: William Casarin <jb55@jb55.com>
2022-03-14 12:26:46 -05:00
GoofyAF a2a6b8c3ff
Goofy af tor.md systemd syntax update (#5059)
Update TOR.md

Update tor install to represent current systemd syntax for enabling and starting a new system service
Corrected a typo. After tor is installed to refresh /etc/tor/torrc config changes the command should be 'sudo systemctl restart tor' not 'sudo systemctl start tor'
2022-03-13 10:30:11 +10:30
Michael Schmoock 9703ee05bf doc: update faq and docs for IP discovery 2022-03-12 16:35:09 +10:30
Vincenzo Palazzo 6a146bed26 db: small code cleanup
Changelog-None: db: small code cleanup

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-11 16:43:15 +10:30
Michael Schmoock db95893aec lightningd: do not use remote_addr for always_use_proxy 2022-03-11 16:42:45 +10:30
Michael Schmoock ef84d6eec5 chore: remove EXPERIMENTAL for rfc #917 remote_addr 2022-03-11 16:42:45 +10:30
Michael Schmoock 57fb34ed06 test: connectd netaddress
Increases test coverage by adding a testcase for connectd/netaddress.c

Changelog-None
2022-03-11 16:42:45 +10:30
Michael Schmoock 07fbc7ef13 pytest: IP address discovery updates node_announcement 2022-03-11 16:42:45 +10:30
Michael Schmoock 67fdc6f8ad gossipd: send updated node_annoucement remote_addr
This is the cheapest algo I came up with that simply checks that the
same `remote_addr` has been report by two different peers. Can be
improved in many ways:

 - Check by connecting to a radonm peers in the network
 - Check for more than two confirmations or a certain fraction
 - ...

Changelog-Added: Send updated node_annoucement when two peers report the same remote_addr.
2022-03-11 16:42:45 +10:30
Michael Schmoock 28b4e57974 lightningd: store recently reported remote_addr 2022-03-11 16:42:45 +10:30
Michael Schmoock b930b8c548 wireaddr: adds wireaddr_eq_without_port and wireaddr_cmp_type
Adds wireaddr_eq_without_port so it can be used later.
Moves wireaddr_cmp_type from connectd.c to this file, so it can be reused later.
2022-03-11 16:42:45 +10:30
Michael Schmoock f1981461ef connectd: ignore private remote_addr on non-DEVELOPER builds
When compiled without DEVELOPER this will now filter out `remote_addr` that
come from localhost. The testcase checks for DEVELOPER to test for correct
function of `remote_addr`.

Also, I renamed "test_connect" to "test_connect_basic" so it can be started
without all the other tests in that file that start with "test_connect..."
2022-03-11 16:42:45 +10:30
Michael Schmoock a01e2740ef pyln-proto: fix port typo in example script
Changelog-None
2022-03-11 16:42:45 +10:30
Michael Schmoock e92176248e chore: fix typo announcable -> announceable
"announcable" is a common misspelling of "announceable", see:

https://en.wiktionary.org/wiki/announcable
2022-03-11 16:42:45 +10:30
Vincenzo Palazzo 2c55c27ad4 doc: remove ambiguity for functionality enabled after a release version
Changelog-None: remove ambiguity for functionality enabled after a release version

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-10 10:24:34 +10:30
Marnix c357411337 Update LICENSE
Update LICENSE to 2022
2022-03-10 10:23:40 +10:30
Ken Sedgwick 8f56f96801 hsmd: Add wallet index metadata to existing messages 2022-03-10 10:22:29 +10:30
Ken Sedgwick 3abe22213c hsmd: Augment call to hsmd_sign_withdrawal_tx with wallet index metadata 2022-03-10 10:22:29 +10:30
Ken Sedgwick 8876c03791 hsmd: Add PSBT keypath utility functions 2022-03-10 10:22:29 +10:30
Ken Sedgwick 1e1948bfd2 hsmd: Add redeemscripts to anchor outputs 2022-03-10 10:22:29 +10:30
Christian Decker 6706384c50 cln-plugin: Ensure we cleanly shut down when we lose the master conn 2022-03-10 10:21:41 +10:30
Christian Decker 1bd2b8c9f7 cln-plugin: Make notification handlers asynchronous 2022-03-10 10:21:41 +10:30
Christian Decker af4eed3787 cln-plugin: Make hooks asynchronous 2022-03-10 10:21:41 +10:30
Christian Decker a7ef38732f cln-plugin: Make rpcmethod handlers async 2022-03-10 10:21:41 +10:30
Christian Decker 60e773239c cln-plugin: Add notification subscriptions and hooks to the plugins
For now hooks are treated identically to rpcmethods, with the
exception of not being returned in the `getmanifest` call. Later on we
can add typed handlers as well.
2022-03-10 10:21:41 +10:30
Christian Decker 8c6af21169 cln-plugin: Add support for synchronous RPC methods
Changelog-Experimental: cln-plugin: Added support for non-async RPC method passthrough (async support coming soon)
2022-03-10 10:21:41 +10:30
Christian Decker 22618a2f94 cln-plugin: Rework the plugin library using a Builder 2022-03-10 10:21:41 +10:30
Christian Decker 4aba119733 pytest: Use valgrind target suppressions instead of skipping tests
Having a list of very targeted suppressions allows us to still run the
majority of tests with valgrind checking, and not fail when Rust does
some trickery. This is for example the case with `std::sync::Once`
which uses `num_procs` calling out to the cgroups subsystem, sometimes
with a null path.

Suggested-by: Rusty Russell <@rustyrussell>
2022-03-10 10:21:41 +10:30
Christian Decker a1555623bc pytest: Mark Rust-dependent tests as skipped with VALGRIND
`valgrind` reports seems to flag some memory accesses that are ok in
the Rust standard library, which we can consider false positives for
our purposes:

```Valgrind error file: valgrind-errors.69147
==69147== Syscall param statx(file_name) points to unaddressable byte(s)
==69147==    at 0x4B049FE: statx (statx.c:29)
==69147==    by 0x2E2DA0: std::sys::unix::fs::try_statx (weak.rs:139)
==69147==    by 0x2D7BD5: <std::fs::File as std::io::Read>::read_to_string (fs.rs:784)
==69147==    by 0x2632CE: num_cpus::linux::Cgroup::param (linux.rs:214)
==69147==    by 0x263179: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==69147==    by 0x263002: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==69147==    by 0x262C01: num_cpus::linux::load_cgroups (linux.rs:149)
==69147==    by 0x26289D: num_cpus::linux::init_cgroups (linux.rs:129)
==69147==    by 0x26BD88: core::ops::function::FnOnce::call_once (function.rs:227)
==69147==    by 0x26B749: std::sync::once::Once::call_once::{{closure}} (once.rs:262)
==69147==    by 0x139717: std::sync::once::Once::call_inner (once.rs:419)
==69147==    by 0x26B6D5: std::sync::once::Once::call_once (once.rs:262)
==69147==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==69147==
==69147== Syscall param statx(buf) points to unaddressable byte(s)
==69147==    at 0x4B049FE: statx (statx.c:29)
==69147==    by 0x2E2DA0: std::sys::unix::fs::try_statx (weak.rs:139)
==69147==    by 0x2D7BD5: <std::fs::File as std::io::Read>::read_to_string (fs.rs:784)
==69147==    by 0x2632CE: num_cpus::linux::Cgroup::param (linux.rs:214)
==69147==    by 0x263179: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==69147==    by 0x263002: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==69147==    by 0x262C01: num_cpus::linux::load_cgroups (linux.rs:149)
==69147==    by 0x26289D: num_cpus::linux::init_cgroups (linux.rs:129)
==69147==    by 0x26BD88: core::ops::function::FnOnce::call_once (function.rs:227)
==69147==    by 0x26B749: std::sync::once::Once::call_once::{{closure}} (once.rs:262)
==69147==    by 0x139717: std::sync::once::Once::call_inner (once.rs:419)
==69147==    by 0x26B6D5: std::sync::once::Once::call_once (once.rs:262)
==69147==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==69147==

```
2022-03-10 10:21:41 +10:30
Christian Decker fbcb4c33ad cln-plugin: Populate the options when we get an `init` call 2022-03-10 10:21:41 +10:30