This loads up 20MB of plugins temporarily; we seem to be getting OOM
killed under CI and I wonder if this is contributing.
Doesn't significantly reduce runtime here, but I have lots of memory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This surprised me, since the CHANGELOG for [0.8.2] said:
We now announce multiple addresses of the same type, if given. ([3609](https://github.com/ElementsProject/lightning/pull/3609))
But it lied!
Changelog-Fixed: We really do allow providing multiple addresses of the same type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.
I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's both complex and flawed, as ZmnSCPxj points out. Make a generic
fd ordering routine, and use it.
Plus, test it!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we're over the dust limit, we fail it immediatey *after* commiting
it, but we need a way to signal this throughout the lifecycle, so we add
it to htlc_in struct and persist it through to the database.
If it's supposed to be failed, we fail after the commit cycle is
completed.
To reduce the surface area of amount of a channel balance that can be
eaten up as htlc dust, we introduce a new config
'--max-dust-htlc-exposure-msat', which sets the max amount that any
channel's balance can be added as dust
Changelog-Added: config: new option --max-dust-htlc-exposure-msat, which limits the total amount of sats to be allowed as dust on a channel
Fixes: #4868
ChangeLog-Fixed: We now no longer self-limit the number of file descriptors (which limits the number of channels) in sufficiently modern systems, or where we can access `/proc` or `/dev/fd`. We still self-limit on old systems where we cannot find the list of open files on `/proc` or `/dev/fd`, so if you need > ~4000 channels, upgrade or mount `/proc`.
This also inadvertently fixes a latent bug: before this patch, in the
`subd` function in `lightningd/subd.c`, we would close `execfail[1]`
*before* doing an `exec`.
We use an EOF on `execfail[1]` as a signal that `exec` succeeded (the
fd is marked CLOEXEC), and otherwise use it to pump `errno` to the
parent.
The intent is that this fd should be kept open until `exec`, at which
point CLOEXEC triggers and close that fd and sends the EOF, *or* if
`exec` fails we can send the `errno` to the parent process vua that
pipe-end.
However, in the previous version, we end up closing that fd *before*
reaching `exec`, either in the loop which `dup2`s passed-in fds (by
overwriting `execfail[1]` with a `dup2`) or in the "close everything"
loop, which does not guard against `execfail[1]`, only
`dev_disconnect_fd`.
If the port is set, we spawn it (lightning_websocketd) on any
connection to that port. That means websocketd is a per-peer daemon,
but it means every other daemon uses the connection normally (it's
just actually talking to websocketd instead of the client directly).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes init a two-stage, and causes some code hoisting.
And we can now send all the HTLCs in a single message, since we have
an 128MB limit and each HTLC is 37 bytes.
This breaks the onchaind stresstest, which uses canned internal messages.
It's time to finally delete that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This re-establishes the prior behavior where a `sendpay` or
`sendonion` that'd match a prior payment would cause the prior payment
to be deleted. While we no longer delete prior attempts we now avoid a
primary key collision by incrementing once. This helps us not having
to touch all existing tests, and likely avoids breaking other users
too.
One of the fundamental constraints of the payment groups idea is that
there may only ever be one group in flight at any point in time, so if
we find a group that is in flight, any new `sendpay` or `sendonion`
must match its `groupid`.
This was the main cause of the pay states flip-flopping, since we
reset the status on each attempt any final status is not really
final. Let's keep them around, and provide a stable history.
It's probably not worth fixing for the other daemons.
Changelog-Changed: JSON-RPC: `ping` now only works if we have a channel with the peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To minimize the diffs, we #if 0 the code. We'll reenable it once
channeld is ready.
We also temporarily disable the ping tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a new hook: onion_message_ourpath for when we know a message
came in via a blinded path we created. The onion_message_blinded hook
is now called for all other messages, since all messages are now
blinded.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we are calling hooks, we track them via a linked list. As they
execute, we pop them off the list in plugin_hook_killed().
When we kill a plugin, we have a destructor which remove its entry from the linked list: plugin_hook_killed.
If it's at the head of the list, that means the plugin died while
processing the hook, so instead of just deleting it, we call
plugin_hook_killed() which behaves as if it said "result: continue".
But plugin_hook_killed() just returns if we're shutting down; this
leaves the link (then freed) on the list, and the *next* plugin tries
to unlink from the list, accessing the previous free entry.
The fix is simple: unlink from the list in plugin_hook_killed() even
if we're shutting down.
```
Valgrind error file: valgrind-errors.78570
==78570== Invalid write of size 8
==78570== at 0x174B55: list_del_ (list.h:328)
==78570== by 0x174FCC: plugin_hook_killed (plugin_hook.c:135)
==78570== by 0x21DC3F: notify (tal.c:240)
==78570== by 0x21E156: del_tree (tal.c:402)
==78570== by 0x21E1A8: del_tree (tal.c:412)
==78570== by 0x21E4F2: tal_free (tal.c:486)
==78570== by 0x16EBD1: plugin_kill (plugin.c:345)
==78570== by 0x16F9C4: plugin_conn_finish (plugin.c:724)
==78570== by 0x20F1A5: destroy_conn (poll.c:244)
==78570== by 0x20F1C9: destroy_conn_close_fd (poll.c:250)
==78570== by 0x21DC3F: notify (tal.c:240)
==78570== by 0x21E156: del_tree (tal.c:402)
==78570== Address 0x6aee688 is 40 bytes inside a block of size 72 free'd
==78570== at 0x483CA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==78570== by 0x21E224: del_tree (tal.c:421)
==78570== by 0x21E1A8: del_tree (tal.c:412)
==78570== by 0x21E4F2: tal_free (tal.c:486)
==78570== by 0x16EBD1: plugin_kill (plugin.c:345)
==78570== by 0x16F9C4: plugin_conn_finish (plugin.c:724)
==78570== by 0x20F1A5: destroy_conn (poll.c:244)
==78570== by 0x20F1C9: destroy_conn_close_fd (poll.c:250)
==78570== by 0x21DC3F: notify (tal.c:240)
==78570== by 0x21E156: del_tree (tal.c:402)
==78570== by 0x21E4F2: tal_free (tal.c:486)
==78570== by 0x20D7B6: io_close (io.c:450)
==78570== Block was alloc'd at
==78570== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==78570== by 0x21DCAD: allocate (tal.c:250)
==78570== by 0x21E26E: tal_alloc_ (tal.c:428)
==78570== by 0x175599: plugin_hook_call_ (plugin_hook.c:259)
==78570== by 0x13616F: plugin_hook_call_onion_message_blinded (onion_message.c:126)
==78570== by 0x13643B: handle_obs_onionmsg_to_us (onion_message.c:187)
==78570== by 0x138BBD: gossip_msg (gossip_control.c:140)
==78570== by 0x178AEC: sd_msg_read (subd.c:495)
==78570== by 0x20CA00: next_plan (io.c:59)
==78570== by 0x20D608: do_plan (io.c:407)
==78570== by 0x20D64A: io_ready (io.c:417)
==78570== by 0x20F8F1: io_loop (poll.c:445)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was measured as a 95th percentile in our rough testing, thanks to
all the volunteers who monitored my channels.
Fixes: #4761
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setchannelfee` gives a grace period (`enforcedelay`) before rejecting old-fee payments: default 10 minutes.
Currently it will be used for onion replies, but we can use it for offers
and invoices in future, if we want to avoid revealing our node_id.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This expects the caller to create the TLVs to put in each hop; it
simply creates the onion and sends it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We put this in reply paths, so we can tell if they are used. This lets us
avoid responding unless the correct reply path is used.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
One change from the obsolete version handling, gossipd will no longer send
forwarding onion msgs to lightningd, but will forward it directly.
That was the effect before, anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
sendonionmessage is going to be the new one, and do much *less*.
As this is an internal experimental-only API, no deprecation cycle
required.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
offers contain an x-only pubkey: to route to them to need to know the
02 vs 03 prefix. If they're in the gossmap it's easy, but if they're
a directly-connected peer it's harder. We used to have
sendonionmessage tweak the key if it found a peer with the matching
key, but this was always a hack.
It turns out that we try to connect to the node anyway, which is
a noop if it's already connected. So try connecting to the other
parity if the first one fails.
Also, this registers when we fail to connect, and returns an error
rather than waiting for timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
E - Node /tmp/ltests-uf2g_5gd/test_sendinvoice_obsolete_1/lightning-1/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E "ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)",
E "ccan/ccan/tal/str/str.c:91 (tal_vfmt_)",
E "ccan/ccan/tal/str/str.c:44 (tal_fmt_)",
E "common/wireaddr.c:232 (fmt_wireaddr_without_port)",
E "common/wireaddr.c:251 (fmt_wireaddr)",
E "common/wireaddr.c:208 (fmt_wireaddr_internal)",
E "common/wireaddr.c:221 (fmt_wireaddr_internal_)",
E "common/type_to_string.c:32 (type_to_string_)",
E "lightningd/peer_control.c:1433 (json_add_peer)",
E "lightningd/peer_control.c:1481 (json_listpeers)",
E "lightningd/jsonrpc.c:627 (command_exec)",
E "lightningd/jsonrpc.c:762 (rpc_command_hook_final)",
E "lightningd/plugin_hook.c:274 (plugin_hook_call_)",
E "lightningd/jsonrpc.c:850 (plugin_hook_call_rpc_command)",
E "lightningd/jsonrpc.c:949 (parse_request)",
E "lightningd/jsonrpc.c:1040 (read_json)",
E "ccan/ccan/io/io.c:59 (next_plan)",
E "ccan/ccan/io/io.c:435 (io_do_always)",
E "ccan/ccan/io/poll.c:300 (handle_always)",
E "ccan/ccan/io/poll.c:377 (io_loop)",
E "lightningd/io_loop_with_timers.c:21 (io_loop_with_timers)",
E "lightningd/lightningd.c:1112 (main)"
E ],
E "label": "common/wireaddr.c:232:char[]",
E "parents": [
E "common/json_stream.c:22:struct json_stream",
E "ccan/ccan/io/io.c:91:struct io_conn",
E "lightningd/lightningd.c:103:struct lightningd"
E ],
E "value": "0x56041b322a48"
E }
E ]
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rusty Russell
Christian Decker
Vincenzo Palazzo
niftynei
ZmnSCPxj jxPCSnmZ