Commit Graph

11545 Commits

Author SHA1 Message Date
Rusty Russell d58e6fa20b lightningd: don't tell connectd to disconnect peer if it told us.
We allow connectd to tell us a peer has gone away, but now we need
to make sure we don't double-spiderman and tell it to disconnect peer.

This is particularly harmful on reconnect: it (will soon) tell us the
old connection is gone, ready to tell us the new peer has connected.
We would tell it to disconnect the peer, which throws away the new
connection!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell c64ce4bbf3 lightningd: clean up channels when connectd says peer is gone.
This is redundant now, since connectd only sends us this once we tell
it it's OK, but that's changing, so clean up now.  This means that
connectd will be able to make *unsolicited* closes, if it needs to.

We share logic with peer_please_disconnect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 9dc3880360 connectd: put peer into "draining" mode when we want to close it.
This removes it from the hashtable, and forces it to do nothing but
send out any remaining packets, then close.

It is, in effect, reduced to a stub, with no further interactions
with the rest of the system (all subds are freed already).

Also removes the need for an explicit "final_msg" too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 37ff013c2c connectd: fix subd tal parents.
This came out in a later patch: freeing the peer->subds doesn't actually
free the subds, because they're reparented onto subd->conn, which is
a child of peer itself.


This breaks because when the peer is finally freed, destroy_subd is
called, and expects to find itself in peer->subds (but we made that
NULL when we manually freed it!).

Fix this, and make it obvious that we tal_steal it.

```
ightning_connectd: FATAL SIGNAL 11 (version v0.11.0.1-25-gbf025aa-modded)
0x55de2a1b8b94 send_backtrace
	common/daemon.c:33
0x55de2a1b8c3e crashdump
	common/daemon.c:46
0x7fe2be2fc08f ???
	/build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x55de2a1af41e destroy_subd
	connectd/multiplex.c:1119
0x55de2a217686 notify
	ccan/ccan/tal/tal.c:240
0x55de2a217b9d del_tree
	ccan/ccan/tal/tal.c:402
0x55de2a217bef del_tree
	ccan/ccan/tal/tal.c:412
0x55de2a217bef del_tree
	ccan/ccan/tal/tal.c:412
0x55de2a217f39 tal_free
	ccan/ccan/tal/tal.c:486
0x55de2a1aa116 peer_discard
	connectd/connectd.c:1834
0x55de2a1aa38d recv_req
	connectd/connectd.c:1903
0x55de2a1b9121 handle_read
	common/daemon_conn.c:31
0x55de2a205a35 next_plan
	ccan/ccan/io/io.c:59
0x55de2a20663d do_plan
	ccan/ccan/io/io.c:407
0x55de2a20667f io_ready
	ccan/ccan/io/io.c:417
0x55de2a208972 io_loop
	ccan/ccan/io/poll.c:453
0x55de2a1aa736 main
	connectd/connectd.c:2042
0x7fe2be2dd082 __libc_start_main
	../csu/libc-start.c:308
0x55de2a1a085d ???
	???:0
0xffffffffffffffff ???
	???:0
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 912ac25270 lightningd: remove 'connected' flag from channel structure.
It's directly a product of "does it have a current owner subdaemon"
and "does that subdaemon talk to peers", so create a helper function
which just evaluates that instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell b8ed107743 lightningd: fix dev-memleak crash on unown unconfirmed channels.
```
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: FATAL SIGNAL 11 (version e0507aa)
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x56319736e437
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0x56319736e48a
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7fc37721151f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/subd.c:839 (subd_send_msg) 0x5631973425f3
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/subd.c:859 (subd_req_) 0x5631973426e4
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/peer_control.c:2967 (peer_dev_memleak) 0x56319732ca93
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/memdump.c:296 (json_memleak) 0x56319730fc97
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:630 (command_exec) 0x563197306f9d
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:765 (rpc_command_hook_final) 0x5631973075d5
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:278 (plugin_hook_call_) 0x56319733db47
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:853 (plugin_hook_call_rpc_command) 0x5631973079d4
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:957 (parse_request) 0x563197307efb
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1054 (read_json) 0x563197308361
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x5631973de46c
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x5631973df0a1
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x5631973df0e3
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x5631973e147f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x563197305041
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1184 (main) 0x56319730b58d
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7fc3771f8d8f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:392 (__libc_start_main_impl) 0x7fc3771f8e3f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x5631972dec54
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
lightningd-3 2022-07-14T08:18:38.976Z DEBUG   connectd: drain_peer
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 08e3e979c8 lightningd: set cid correctly in peer->uncommitted_channel.
Setting it to 0xfffff... is just confusing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
William Casarin 98185dfc2b startup_regtest: add connect helper
`connect 1 2` to connect from l1 to l2, etc
2022-07-18 20:37:32 -05:00
William Casarin a3f5d31b09 startup_regtest: add experimental-offers 2022-07-18 20:37:32 -05:00
Rusty Russell 73762de18c lightningd: reduce log level for remote address reporting.
It's available in listpeers() if you want to see it, otherwise it's not
really something users want to see in the normal course of operation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 05:32:24 +02:00
Michael Schmoock 65433de05f options: set DNS port to network default if not specified
- set port for a DNS announcement without port to network default
- remove x-fail

Changelog-Fixed: Port of a DNS announcement can be 0 if unspecified
2022-07-17 23:26:07 +02:00
Michael Schmoock 6d4285d7c4 pytest: add xfail test to show DNS w/o port issue
This adds an X-Fail testcase that demonstrates that currently
the port of a DNS announcement is not set to the corresponding
network port (in this case regtest), but it will be set to 0.

Changelog-None
2022-07-17 23:26:07 +02:00
Swapnil 6204d70a37 docs: fix contrib/ docs 2022-07-17 21:40:01 +09:30
grubles af2b863b4a Add instructions for checking out a release tag 2022-07-17 11:45:34 +09:30
zero fee routing 58ae885f48 fix typo in commando documentation 2022-07-17 08:51:02 +09:30
Rusty Russell 8c48eda8c7 decode: support decoding runes.
This is a bit weird since it lives in the offers plugin, but it works
well.  This should make runes much more approachable for people!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 468dff1723 commando: add rate for maximum successful rune use per minute.
I'm assuming that nobody wants a rate slower than 1 per minute; we can
introduce 'drate' if we want a per-day kind of limit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 4ab09f7cfb commando: add support for parameters by array, parameter count.
Awkward to filter, but they're really practical for many commands.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell cf28cff398 doc: document commando and commando-rune.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 8688daf937 commando: require runes for operation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell ae4856df70 commando: don't look at messages *at all* unless they've created a rune.
This means we can leave commando on by default, without an explicit config flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 419cb60b1b commando: add commando-rune command.
Can both mint new runes, and add one or more restrictions to existing ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell cf4374c4ed devtools/rune: simple decode tool.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 0d94530f13 commando: runes infrastructure.
We support the old commando.py plugin, which stores a random secret,
as well as a more modern approach which uses makesecret.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell b49703e279 commando: correctly reflect error data field.
Some JSON error include "data", and we should reflect that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 49df89556b commando: support commands larger than 64k.
This is needed for invoice, which can be asked to commit to giant descriptions
(though that's antisocial!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 3fe246c2e7 plugins/commando: basic commando plugin (no runes yet).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au
Changelog-Added: Plugins: `commando` a new builtin plugin to send/recv peer commands over the lightning network, using runes.
2022-07-17 08:51:02 +09:30
Rusty Russell 3eccf16f98 libplugin: datastore helpers.
Plugins are supposed to store their data in the datastore, and commando does so:
let's make it easier for them by providing convenience APIs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell d3e64c3970 libplugin: jsonrpc_request_whole_object_start() for more custom request handling.
commando wants to see the whole reply object, and also not to assume params is
an object.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell d0a55a62b3 common/json_stream: make json_add_jsonstr take a length.
This is useful when have have a jsmntok_t.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell f65d3bb1fc ccan: upgrade to get ccan/runes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 3e672b784d Makefile: use a library archive for CCAN
The linker discards whole files in an archive if it doesn't need them,
so saves a bit of space (and time).  Also allows us to add more niche
things to CCAN (e.g. runes support!) without bloating all the binaries.

We also had many places which depended on $(CCAN_FILES), but that was
already a dependent of $(ALL_PROGRAMS) and $(ALL_TEST_PROGRAMS).

Before:

```
$ size lightningd/lightning*d
   text	   data	    bss	    dec	    hex	filename
2247683	   8696	  39008	2295387	 23065b	lightningd/lightning_channeld
2086607	   7432	  38880	2132919	 208bb7	lightningd/lightning_closingd
2227916	   8056	  39200	2275172	 22b764	lightningd/lightning_connectd
3369236	 119288	  39240	3527764	 35d454	lightningd/lightningd
2183551	   8352	  38880	2230783	 2209ff	lightningd/lightning_dualopend
2196389	   8024	  39136	2243549	 223bdd	lightningd/lightning_gossipd
2086216	   7488	  39264	2132968	 208be8	lightningd/lightning_hsmd
2134396	   8136	  39424	2181956	 214b44	lightningd/lightning_onchaind
2133391	   8352	  38880	2180623	 21460f	lightningd/lightning_openingd
1512168	   2136	  34384	1548688	 17a190	lightningd/lightning_websocketd
```

After:
```
text	   data	    bss	    dec	    hex	filename
2192065	   8488	  38912	2239465	 222be9	lightningd/lightning_channeld
2030957	   7224	  38816	2076997	 1fb145	lightningd/lightning_closingd
2179571	   7968	  39104	2226643	 21f9d3	lightningd/lightning_connectd
3354296	 119288	  39208	3512792	 3599d8	lightningd/lightningd
2127933	   8144	  38816	2174893	 212fad	lightningd/lightning_dualopend
2141699	   7856	  39072	2188627	 216553	lightningd/lightning_gossipd
2024482	   7288	   5240	2037010	 1f1512	lightningd/lightning_hsmd
2072074	   7920	   5400	2085394	 1fd212	lightningd/lightning_onchaind
2077773	   8144	  38816	2124733	 206bbd	lightningd/lightning_openingd
1408958	   1752	    344	1411054	 1587ee	lightningd/lightning_websocketd
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Simon Vrouwe ad3cbed7c2 plugin: autoclean fix double free when re-enable, remove xfail mark from test_
Fixes a crash when enabling after a disable with cycle_seconds=0.
2022-07-16 14:19:05 +09:30
Simon Vrouwe cc40243399 pyln-testing: print content of errlog file when _some_ node failed unexpected 2022-07-16 14:19:05 +09:30
Simon Vrouwe 94f16f14b7 test: test_invoices add a stress test 2022-07-16 14:19:05 +09:30
Rusty Russell dbae5ae569 common/json_stream.c: provide explicit json_add_primitive_fmt and json_add_str_fmt routines.
Rather than a generic "add member", provide two routines: one which
doesn't quote, and one which does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 36a29fbfbc lightningd/json.h: remove.
There are hardly any lightningd-specific JSON functions: all that's left
are the feerate ones, and there's already a comment that we should have
a lightningd/feerate.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 401f1debc5 common: clean up json routine locations.
We have them split over common/param.c, common/json.c,
common/json_helpers.c, common/json_tok.c and common/json_stream.c.

Change that to:
* common/json_parse (all the json_to_xxx routines)
* common/json_parse_simple (simplest the json parsing routines, for cli too)
* common/json_stream (all the json_add_xxx routines)
* common/json_param (all the param and param_xxx routines)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell f12f0d6929 common: remove json_tok.
Make local copies for tests, and fundchannel plugin (which just passes
params through).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell a9b992ff4a plugins/spender/multifundchannel: remove json_tok.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell e621b8b24e lightningd: remove gratuitous param_tok from help and config.
They predate json_string!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell ec76ba3895 lightningd/connect_control: remove param_tok from connect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 814cde5623 lightningd/closing_control: remove param_tok().
It's bad form; if we do parsing inside params() then the `check`
command is much more effective.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 0236d4e4da common/json_stream: remove useless attempt at oom handling.
We tell membuf to use tal, and tal never returns NULL, so this
code can never be triggered.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 9685c1adaf lightningd: remove getsharedsecret.
This was introduced to allow creating a shared secret, but it's better to use
makesecret which creates unique secrets.  getsharedsecret being a generic ECDH
function allows the caller to initiate conversations as if it was us; this
is generally OK, since we don't allow untrusted API access, but the commando
plugin had to blacklist this for read-only runes explicitly.

Since @ZmnSCPxj never ended up using this after introducing it, simply
remove it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSONRPC: `getsharedsecret` API: use `makesecret`
2022-07-15 22:17:58 +09:30
Rusty Russell c34a0a22ad makesecret: change info_hex arg to simply "hex" to match datastore command.
And fix schema: it wasn't tested as there was no test-by-parameter-name.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 22:17:58 +09:30
niftynei 769efe8d54 tests: massively speed up our wait for enormous feerates
We do smoothing, waiting for this to hit the target (50k+) was taking
longer than my TIMEOUT=15; here we increase the speed at which it hits
exit velocity, so to speak.
2022-07-15 22:16:27 +09:30
niftynei f2e7e9d919 coin-moves: only log htlc_timeout pair for penalty txs
We cleanup our output tracking for timeout txs when the peer's
htlc_timeout self-expiry is hit; we'd also log its spend if happen to
see it get spent.

This is a bit of a race as they can't spend it until the locktime is
available. Hence the flakiness in tests that expected the `htlc_timeout`
to *not* be spent.

Instead, we only log an external's `htlc_timeout` spend in the case
where we also immediately register another output to track for it (only
happens when said htlc is stealable)

Fixes #5405
In-Collab-With: @ddustin
2022-07-15 22:16:27 +09:30
Rusty Russell 6fd8fa4d95 connectd: optimize requests for "recent" gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 21:18:29 +09:30
Rusty Russell 62a5183fb5 common: add ability for gossip_store to track by timestamp.
We'll use this to keep a two-hour-ago "recent gossip offset" as an
optimization.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 21:18:29 +09:30