This cause of cascading failure was pointed out by @t-bast: if fees spike and
you don't timeout an outgoing onchain HTLC, you should nonetheless fail the incoming htlc
because otherwise the incoming peer will close on you.
Of course, there's a risk of losing funds, but this only happens if you weren't going to get the HTLC spend in time anyway. And it would also catch any other reason that the downstream onchain goes wrong, containing the damage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @t-bast
Changelog-Fixed: Protocol: We will close incoming HTLCs early if the outgoing HTLC is stuck onchain long enough, to avoid cascating failure.
This is the simplest solution, not the best, but there's significant risk in try to remove the "we have a path" assumption in the code pay code.
Includes removing a `tal_steal` which was incorrect: the buffer has the same lifetime as the plugin, so if we steal it then things get messy when we free the struct payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `pay` will now pay your own invoices if you try.
Previously, the "payment" and "invoice" paths were completely separate, but this now calls both. It bypasses htlc_sets (and thus, cannot do MPP), and bypasses the hook too: the former is tied closely to HTLCs, and the hook is also very htlc-centric.
Includes finishing unfinished sentence in sendpay man page, as a bonus.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sendpay` now allows self-payment of invoices, by specifying an empty route.
Clean these up: they were debug logs, but we want to pass this information
back for self-payments.
Also fixes "Attept" typo which altered tests!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they have invalid runes, we bail, but if they have runes which used
a different master secret (old commando.py allowed you to override
secret), we just complain and delete them.
Note that this requires more mocks in wallet/test/run-db.c...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The wallet_datastore_first() SELECT statement only iterates from the
given key (if any), relying on the caller to notice when the key no
longer applies. (e.g. startkey = ["foo", "bar"] will return key
["foo", "bar"] then ["foo", "bar", "child" ], then ["foo", "baz"]).
The only caller (listdatastore) would notice the keychange and stop
looping, but reallly wallet_datastore_next() should do this. When I
tried to use it for migrations, I got very confused!
Also, several places want a simple "wallet_datastore_get()" function,
so provide that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to access this in db migrations, which happen very early, but
runes_init needs the db, creating a circular dependency which must be
split.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The uid isn't enough: it could be someone else's rune. This is tested
in the command rune list tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
At cold start, if your node is behind the blocktip and you've sent your
peer a blockheight counter from the future, we shouldn't confuse ourselves
with our rollback/replay.
Should fix flakes in CI that were spotting BROKEN blockheight updates.
Logs below from a previuos CI fail (edited for relative clarity)
The one that sasy "{ SENT_ADD_ACK_REVOCATION:111 }, our current 108` is
the tell; the last line is the node finally catching up to the tip.
In the test we get into this state by stopping and restarting the node.
```
2023-07-22T11:24:28.2754533Z lightningd-1 2023-07-22T11:19:34.188Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#2: Already have funding locked in
2023-07-22T11:24:28.2755486Z lightningd-1 2023-07-22T11:19:34.188Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#2: attempting update blockheight a5b23dff5177badd6df725c
efeb83ceccbfc52dc64a16b38894a41f0ad8fa181
2023-07-22T11:24:28.2755778Z lightningd-1 2023-07-22T11:19:34.188Z DEBUG lightningd: update_blockheight: height = 108
2023-07-22T11:24:28.2766210Z lightningd-1 2023-07-22T11:19:34.210Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: init LOCAL: remote_per_commit = 029563e7c898
5d8b95bdfe19e47e494bb8ec8d53ff4edb93f156be57667bfee8c9, old_remote_per_commit = 02bf3117c149d324361f0b418db8984b1e29af70c773eb2865a41ff7f583c7c9ed next_idx_local = 3 next_idx_remote = 3 revocations_recei
ved = 2 feerates { SENT_ADD_ACK_REVOCATION:3750 } range 253-150000 blockheights { SENT_ADD_ACK_REVOCATION:111 }, our current 108
2023-07-22T11:24:28.2768866Z lightningd-1 2023-07-22T11:19:34.211Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: peer_out WIRE_CHANNEL_REESTABLISH
2023-07-22T11:24:28.2769416Z lightningd-1 2023-07-22T11:19:34.211Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: billboard: Sent reestablish, waiting for the
irs
2023-07-22T11:24:28.2771115Z lightningd-1 2023-07-22T11:19:34.212Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: peer_in WIRE_CHANNEL_REESTABLISH
2023-07-22T11:24:28.2774150Z lightningd-1 2023-07-22T11:19:34.212Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Got reestablish commit=3 revoke=2
2023-07-22T11:24:28.2776056Z lightningd-1 2023-07-22T11:19:34.212Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: next_revocation_number = 2
2023-07-22T11:24:28.2805639Z lightningd-1 2023-07-22T11:19:34.239Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 108 less than last 111
2023-07-22T11:24:28.2823960Z lightningd-1 2023-07-22T11:19:34.240Z DEBUG lightningd: Adding block 109: 5f67b6e110eb3c3457bea4fcf0d04ce9be90efeee5df8e083ed4266074ca911f
2023-07-22T11:24:28.2833154Z lightningd-1 2023-07-22T11:19:34.251Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 108 less than last 111
2023-07-22T11:24:28.2833630Z lightningd-1 2023-07-22T11:19:34.252Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Trying commit
2023-07-22T11:24:28.2834165Z lightningd-1 2023-07-22T11:19:34.252Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 108 less than last 111
2023-07-22T11:24:28.2835070Z lightningd-1 2023-07-22T11:19:34.252Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Can't send commit: nothing to send, feechange not wanted ({ SENT_ADD_ACK_REVOCATION:3750 }) blockheight not wanted ({ SENT_ADD_ACK_REVOCATION:111 })
2023-07-22T11:24:28.2835516Z lightningd-1 2023-07-22T11:19:34.350Z DEBUG lightningd: Adding block 110: 5f43f3ac9d808e3a309720d1b0727a00d5a3d3ddca71d97401e233637e87639c
2023-07-22T11:24:28.2835962Z lightningd-1 2023-07-22T11:19:34.476Z DEBUG lightningd: Adding block 111: 55b0d1e0a08ff6233e186e6735cb1cbec33e2b0a6e7d08f2622e8c1db30b54b9
```
We get intermittant reports of subd->conn being leaked, but I could never find it.
That's because it's actually subd which is not referenced any more: subd->conn
gets reported because it's subd's tal_parent (and, except for the reference in
subd, not referenced either).
The real issue is that the channel->owner is reassigned to the new subdaemon,
and the old one is still exiting. During that time, we can see a "leak".
```
- Node /tmp/ltests-hkr089bp/test_sql_1/lightning-3/ has memory leaks: [
{
"backtrace": [
"ccan/ccan/tal/tal.c:477 (tal_alloc_)",
"ccan/ccan/io/io.c:91 (io_new_conn_)",
"lightningd/subd.c:774 (new_subd)",
"lightningd/subd.c:828 (new_channel_subd_)",
"lightningd/dual_open_control.c:3662 (peer_restart_dualopend)",
"lightningd/peer_control.c:1161 (connect_activate_subd)",
"lightningd/peer_control.c:1273 (peer_connected_hook_final)",
"lightningd/plugin_hook.c:213 (plugin_hook_callback)",
"lightningd/plugin.c:591 (plugin_response_handle)",
"lightningd/plugin.c:702 (plugin_read_json_one)",
"lightningd/plugin.c:747 (plugin_read_json)",
"ccan/ccan/io/io.c:59 (next_plan)",
"ccan/ccan/io/io.c:407 (do_plan)",
"ccan/ccan/io/io.c:417 (io_ready)",
"ccan/ccan/io/poll.c:453 (io_loop)",
"lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
"lightningd/lightningd.c:1249 (main)"
],
"label": "ccan/ccan/io/io.c:91:struct io_conn",
"parents": [
"lightningd/lightningd.c:107:struct lightningd"
],
"value": "0x556c63c859f8"
}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The channel selection did not consider the amounts that we are trying
to transfer, which in a multiplexed channel world could end up always
selecting a channel that is too small for the payment. We also log
which channel was selected based on the selector that is passed in,
allowing us to better follow the decisions.
Changelog-Fixed: pay: `sendonion` and `sendpay` will now consider amounts involved when using picking one channel for a peer
If you miss a wait event, you can catch up by doing listinvoices and
getting the max of these fields. It's also a good debugging clue.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we have defined ordering, we can add a start param.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listinvoices` has `index` and `start` parameters for listing control.
This will initially be for listinvoices, but can be expanded to other
list commands.
It's documented, but it makes promises which currently don't exist:
* listinvoice does not support `index` or `start` yet.
* It doesn't actually fire when invoices change yet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `wait`: new generic command to wait for events.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setchannel` adds a new `ignorefeelimits` parameter to allow peer to set arbitrary commitment transaction fees on a per-channel basis.
This extracts the core checking functionality for a rune, so they can
easily be used more widely than just commando.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this, only per-peer daemons were filtered correctly. For generic
daemons, we need to filter with the actual nodeid they use (if any).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: config: `log-level` filters now apply correctly to messages from `connectd`.
Rather than initializating the "print_level" field on first use, we can
do it in logging_options_parsed(), now we have a linked list of them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`struct log` becomes `struct logger`, and the member which points to the
`struct log_book` becomes `->log_book` not `->lr`.
Also, we don't need to keep the log_book in struct plugin, since it has
access to ld's log_book.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can expose the dbid, rather than pretending we have some "struct
invoice" which is actually just the dbid. And don't have a pile of
"wallet_" wrappers for redirection.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We hand "estimatefees.feerate_floor" as method, for example, and then
crash instead of reporting the plugin which gave us the bad answer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously, our code checked for the presence of the `lightning:`
prefix while decoding a bolt11 string. Although this prefix is valid
and accepted by the core lightning pay command, it was causing issues
with how we managed invoices. Specifically, we were skipping the prefix
when creating a copy of the invoice string and storing the raw invoice
(including the prefix) in the database, which caused inconsistencies
in the user experience.
To address this issue, we need to strip the `lightning:` prefix before
calling each core lightning command. In addition, we should
modify the invstring inside the db with the canonical one.
This commit fixes the issue by stripping the `lightning:` prefix
from the `listsendpays` function, which will improve the
user experience and ensure consistency in our invoice management (see
next commit).
Reported-by: @johngribbin
Link: ElementsProject#6207
Fixes: debbdc0
Changelog-Fixes: trim the `lightning:` prefix from invoice everywhere.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We fixed the others. There are no fields, but this keeps it consistent.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `shutdown` notification contains `shutdown` object (notification consistency)
Requested-by: Shahana Farooqui @Shahana
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: plugins can subscribe to all notifications using "*".
This is just housekeeping that allows up
to do not spam the logs of people with not
useful information.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
`deprecated_apis` is now inside `ld`.
```
lightningd/notification.c: In function ‘connect_notification_serialize’:
lightningd/notification.c:60:13: error: ‘deprecated_apis’ undeclared (first use in this function)
60 | if (deprecated_apis)
| ^~~~~~~~~~~~~~~
lightningd/notification.c:60:13: note: each undeclared identifier is reported only once for each function it appears in
lightningd/notification.c: In function ‘disconnect_notification_serialize’:
lightningd/notification.c:97:13: error: ‘deprecated_apis’ undeclared (first use in this function)
97 | if (deprecated_apis)
| ^~~~~~~~~~~~~~~
lightningd/notification.c: In function ‘block_added_notification_serialize’:
lightningd/notification.c:612:13: error: ‘deprecated_apis’ undeclared (first use in this function)
612 | if (deprecated_apis) {
| ^~~~~~~~~~~~~~~
make: *** [Makefile:299: lightningd/notification.o] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ignore the min fee as specified by the user setting.
We explicitly allow the user to ignore the fee limits, although this comes with inherent risks.
By enabling this option, users are explicitly
I was aware of the potential dangers.
There are situations, such as the one described in [1], where it
becomes necessary to bypass the fee limits to resolve issues like a stuck channel.
BTW experimental-anchors should fix this.
[1] https://github.com/ElementsProject/lightning/issues/6362
Reported-by: @pabpas
Fixes: 64b1ddd761
Link: https://github.com/ElementsProject/lightning/issues/6362
Changelog-Fixes: do not ignore the ignore-fee-limit option during
update_fee
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: JSON-RPC: `connect` and `disconnect` notifications now wrap `id` field in a `connect`/`disconnect` object (consistency with other notifications)
We usually have access to `ld`, so avoid the global.
The only place generic code needs it is for the json command struct,
and that already has accessors: add one for libplugin and lightningd
to tell it if deprecated apis are OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the simple version which always tries to keep some sats if we
have an anchor channel. Turns out that we need something more
sophisticated for multifundchannel, so that's next.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` will refuse to spend funds below `min-emergency-msat` if we have any anchor channels (and `all` will be reduced appropriately).
Changelog-Changed: JSON-RPC: `fundpsbt` and `utxopsbt` will refuse to spend funds below `min-emergency-msat` if we have any anchor channels.
For anchors, we need some sats sitting around in case we need to CPFP
a close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `min-emergency-msat` setting for (currently experimental!) anchor channels, to keep funds in reserve for forced closes.
We disabled experimental support for opening non-zero-fee anchor
channels (though old nodes may still have such channels if they turned
that on!).
So we simply call this `experimental-anchors`, since this is the variant
which we expect to be used widely.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: protocol: added support for zero-fee-htlc anchors (`option_anchors_zero_fee_htlc_tx`), using `--experimental-anchors`.
In most cases, it's the same as option_anchor_outputs, but for
fees it's different. This transformation is the simplest:
pass it as a pair, and test it explicitly.
In future we could rationalize some paths, but this was nice
and mechanical.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to know both, because in theory we could negotiate a
non-anchor channel even if they support it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to know both, because in theory we could negotiate a
non-anchor channel even if they support it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we can CPFP, we don't have to track the feerate as closely. But
it still needs to get in the mempool, so we use 10 sat/byte, or the
100 block estimate if that is higher.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `feerates` has new fields `unilateral_anchor_close` to show the feerate used for anchor channels (currently experimental), and `unilateral_close_nonanchor_satoshis`.
Changelog-Changed: JSON-RPC: `feerates` `unilateral_close_satoshis` now assumes anchor channels if enabled (currently experimental).
We don't actually use it anywhere, but we actually want to now for
CPFP. So give it more parameters and make it return bool so it can
be set without necessarily suppressing rexmit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It depends on whether we negotiated anchors: just document that this
field doesn't apply for anchors (it becomes zero by the end of this
patch series, which is weird).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we need to push off requring this for another full deprecation cycle!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `pay` and `decodepay` with description now correctly handle JSON escapes (e.g " inside description)
If it's a plugin opt, we'll need a callback, so reshuffle logic. Also
add infra to map option name to plugin and option.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Do it slightly intelligently, so if we had set previously using setconfig
we don't keep appending new ones, but replace it in-place.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently only implemented for min-capacity-sat.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: new command `setconfig` allows a limited number of configuration settings to be changed without restart.
Previously, if these failed we always exited; once we have dymamic
configs this would be a (tiny) memory leak, so use tmpctx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To test, we do min-capacity-sat which is simple. We also update the
listconfigs man page which contained some obsolete information.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were seeing hangs in disconnect in
tests/test_connection.py::test_feerate_stress, and looking at the logs
it's because we're not actually telling connectd to disconnect.
These days, we have a connect counter, so connectd knows to ignore it
if we simply haven't read the message about it already disconnecting.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When core lightning is asking the information about
the blockchain with `getchaininfo` command lightningd
know already the information about the min and max block height.
the problem is when we have a smarter Bitcoin backend that is able
to switch between different clients in some cases is helpful
give the information about current known height by lightningd and
pass it down to the plugin.
In this way, the plugin knows what is the correct known height from lightnind, and can
try to fix some problems if any exit.
This is particularly useful when you are syncing a new backend from scratch
like https://github.com/cloudhead/nakamoto and we avoid returning the
lower height from the known, and avoid the crash of core lightning.
With this information, the plugin can start to sync the chain and return
the answer back only when the chain is in sync with the current status of
lightningd.
Another reason to add this field and not wait the correct block in core
lightning itself is because Bitcoin Core is extremely slow to sync up,
so the question here is, how long should we wait? The time depends
on various factors.
With this approach of informing the plugin about the height, in some cases,
you can start the syncing but move the execution to another backend until
the previous one is ready.
The problem I want to solve is that I don't want to be left in the dark when
we run `getchaininfo`, and I want to have the opportunity to wait for
the blockchain sync or decide to dispatch the request elsewhere.
Changelog-Added: Pass the current known block height down to the getchaininfo call.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
option_scid_alias inside a channel_type allows for more private
channels: in particular, it tells the peer that it MUST NOT allow
routing via the real short channel id, and MUST use the alias.
It only makes sense (and is only permitted!) on unannounced channels.
Unfortunately, we didn't set this bit in the channel_type in v12.0
when it was introduced, instead relying on the presence of the feature
bit with the peer. This was fixed in 23.05, but:
1. Prior to 23.05 we didn't allow it to be set at all, and
2. LND has a limited set of features they allow, and this isn't allowed without
option_anchors_zero_fee_htlc_tx.
We could simply drop this channel_type until we merge anchors, *but*
that has nasty privacy implications (you can probe the real channel id).
So, if we don't negotiate anchors (we don't!), we don't set this
channel_type bit even if we want it, and *intuit* it, based on:
1. Is this a non-anchor channel_type?
2. Did we both send channel_type?
3. Is this an unannounced channel?
4. Did both peers announce support for scid aliases?
In addition, while looking at the previous backwards-compat code, I
realized that v23.05 violated the spec and send accept_channel with
OPT_SCID_ALIAS if it intuited it, even if it wasn't offered. Stop
doing this, but allow our peers to.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Fix incompatibility with LND which prevented us opening private channels
Fixes: #6208
defaults were deprecated in 0df97547dd, but that was a bit
harsh as several plugins do that (summary, for example). So allow false, but warn
that we ignore anything else.
Reported-by: @microsatoshi on Discord.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: ...actually, `default` `false` still accepted on `flag` type parameters.
Memory leak detected by ASan:
==880002==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x5039e7 in malloc (lightningd/lightningd+0x5039e7)
#1 0x7f2e8c203884 in __alloc_dir (/lib64/libc.so.6+0xd2884)
The function is tiny and was only used in one location. And that one
location was leaking memory.
Detected by ASan:
==2637667==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 7 byte(s) in 1 object(s) allocated from:
#0 0x4cd758 in __interceptor_strdup
#1 0x64c70c in json_stream_log_suppress_for_cmd lightning/lightningd/jsonrpc.c:597:31
#2 0x68a630 in json_getlog lightning/lightningd/log.c:974:2
...
SUMMARY: AddressSanitizer: 7 byte(s) leaked in 1 allocation(s).
Changelog-Deprecated: JSON-RPC: `listconfigs` direct fields, use `configs` sub-object and `set`, `value_bool`, `value_str`, `value_int`, or `value_msat` fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This integrates them with configvars properly: they almost "just work"
in listconfigs now, and we don't put them in a special sub-object
under their plugin.
Unfortunately, this means `listconfigs` now has a loose schema: any
plugin can add something to it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: reloaded plugins get passed any vars from configuration files.
Changelog-Deprecated: Config: boolean plugin options set to `1` or `0` (use `true` and `false` like non-plugin options).
We use multi-specifiable options elsewhere, this is just another.
Otherwise you can't add, you can only set them all.
Changelog-Added: Config: `accept-htlc-tlv-type` (replaces awkward-to-use `accept-htlc-tlv-types`)
Changelog-Deprecated: Config: `accept-htlc-tlv-types` (use `accept-htlc-tlv-type` multiple times)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `default` no longer accepted on `flag` type parameters (it was silently ignored, so just don't set it).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
listconfigs is convenient, but it doesn't handle multi-options well: it
outputs an object with duplicate fields in this case (e.g. log-file), nor
is it extensible to show more than raw values.
However, listconfigs doesn't do what other list commands do (use a
sub-object "configs") so we can put the new values under that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listconfigs` now has `configs` subobject with more information about each config option.
It literally contained \" to avoid it being interpreted as a literal;
now we have OPT_SHOWINT we no longer need this hack.
It's obscure, so I'm not bothering with a deprecation cycle.
Changelog-Fixed: JSON-RPC: `listconfigs` `rpc-file-mode` no longer has gratuitous quotes (e.g. "0600" not "\"0600\"").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have hacky code to show some listconfigs values as literals; instead
explicitly encode the types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Clearly, listconfigs shouldn't list these.
Also, hoist the opt_hidden check since it's independent of whether
there's an arg or not.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an obscure command, but this we won't see old plugin commands in
listconfigs (once it uses configvars).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we wire in the code which gathers configvars and parses from there;
lightningd keeps the array of configuration variables for future use.
Note that lightning-cli also needs to read the config, but it has its
own options (including short ones!) and doesn't want to use this
configvar mechanism, so we have a different API for that now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a nod towards moving to runtime-developer-mode, and also
quite clear; we currently have all these options prefixed with dev,
but this flags makes handling explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will be used for listconfig, which knows these should be presented
as arrays, not single values.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that this actually changes listconfigs output for three msat
fields, which were not changed with the great msat merge. Since
listconfigs isn't actually used by grpc, and the values are always a
little vague, I simply changed this.
Changelog-Fixed: JSON-RPC: `listconfigs` `htlc-minimum-msat`, `htlc-maximum-msat` and `max-dust-htlc-exposure-msat` fields are now numbers, not strings.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds:
1. ability to search for an option by name.
2. allowance to set our own bits when registering options.
3. show callbacks which can say "don't show", and variable length.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We leave the code in contrib/pyln-client/pyln/client/lightning.py to handle
msat null fields for now, though, for a bit more compatibility.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are only likely to confuse users, by silently changing behavior.
Changelog-Deprecated: Config: bind-addr=xxx.onion and addr=xxx.onion, use announce=xxx.onion (which was always equivalent).
Changelog-Deprecated: Config: addr=/socketpath, use listen=/socketpath (which was always equivalent).
This obsoletes the use of --announce-addr-dns which I know Michael
didn't really like either.
Changelog-Deprecated: Config: `announce-addr-dns`; use `--bind-addr=dns:ADDR` for finer control.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a major cleanup to how we parse addresses.
1. parse_wireaddr now supports the "dns:" prefix to support dns records (type 5).
2. We are less reliant on separate_address_and_port() which gets confused by
that colon.
3. We explicitly test every possible address type we can get back from
parsing, and handle them appropriately.
We update the documentation to use the clearer HOSTNAME vs DNS prefixes now
we also have `dns:` as a prefix.
Changelog-Added: Config: `bind` can now take `dns:` prefix to advertize DNS records.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Make it the standard "return the error" pattern.
2. Rather than flags to indicate what types are allowed, have the callers
check the return explicitly.
3. Document the APIs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an internal type: it has no API guarantees (indeed, I'm about
to change it, which is how I discovered scb was using it).
Fortunately for every case we care about, it is actually a wireaddr
(in theory the peer can connect locally using a local socket, but this
is mostly for testing and is a very strange setup, and so simply don't
do scb for those).
In this case, the wire encoding is a single byte followed by the
wireaddr, so open-code that in scb_wire.csv for compatibility.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CI hit this issue where it would get a tx_abort in fundchannel. This
happens when the dualopend we use to query the feerates has not exited
yet (it waits for the tx_abort reply), and we mistakenly reuse it.
With multi-channel support, this is wrong: just run another one and it
all Just Works.
This means we need to rework our dual_open_control.c logic, since it
would previously create an unsaved channel then not clean up if
something went wrong.
Most people will never try to negotiate opening multiple channels to
the same peer at the same time (vs. having an established channel and
opening a new one), so this case is a bit weird.
```
rates = l1.rpc.dev_queryrates(l2.info['id'], amount, amount)
# l1 leases a channel from l2
l1.rpc.fundchannel(l2.info['id'], amount, request_amt=amount,
feerate='{}perkw'.format(feerate),
> compact_lease=rates['compact_lease'])
tests/test_opening.py:1611:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-client/pyln/client/lightning.py:833: in fundchannel
return self.call("fundchannel", payload)
contrib/pyln-testing/pyln/testing/utils.py:721: in call
res = LightningRpc.call(self, method, payload, cmdprefix, filter)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.testing.utils.PrettyPrintingLightningRpc object at 0x7f6cbcd97950>
method = 'fundchannel'
payload = {'amount': 500000, 'announce': True, 'compact_lease': '029a00640064000000644c4b40', 'feerate': '2000perkw', ...}
cmdprefix = None, filter = None
def call(self, method, payload=None, cmdprefix=None, filter=None):
"""Generic call API: you can set cmdprefix here, or set self.cmdprefix
...
if not isinstance(resp, dict):
raise ValueError("Malformed response, response is not a dictionary %s." % resp)
elif "error" in resp:
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: fundchannel, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'amount': 500000, 'feerate': '2000perkw', 'announce': True, 'request_amt': 500000, 'compact_lease': '029a00640064000000644c4b40'}, error: {'code': -1, 'message': 'Abort requested', 'data': {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'method': 'openchannel_init'}}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Build: all experimental features are now runtime-enabled; no more ./configure --enable-experimental-features
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And no longer insist on opt_quiesce.
Changelog-EXPERIMENTAL: Config: `--experimental-upgrade-protocol` enables simple channel upgrades.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a regression that we introduced in this release
due to some dirty parts of our codebase.
For historical reasons (I think), we were using a `json_add_sat_only`
procedure defined in `peer_control.c`. So when @rustyrussell removed the _msat,
we thought that all the fields were reflecting the new behavior, but
we were wrong.
This PR fixes this bug and also removes the additional function
from `peer_control.c`. This way, we can be sure that there is no other part
of our codebase that uses this method (except for other `json_add` methods).
Link: https://github.com/ElementsProject/lightning/issues/6244
Reported-by: @hMsats
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This finally happened on my test box; tests simply stopped. Turns out
we were spinning here, with waitpid returning -1.
Since this is during shutdown, that also explains why pytest under CI
would hang, since timeouts don't apply during test teardown!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were debugging a number of issues related to the forwarding logic,
when using public scids on private channels, and we noticed that we
are very verbose everywhere, except where it counts, i.e., what
decisions are being taken. So we add a couple of debug logs, and a
final info one that tells the operator which resolution was chosen in
the end.
I tested this indeed breaks if we don't accept it, then implemented
the code to accept it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: protocol: We now correctly accept the `option_scid_alias` bit in `open_channel` `channel_type`.
Changelog-Deprecated: protocol: Not setting `option_scid_alias` in `option_channel` `channel_type` for unannounced channels.
Now we've set everything up, the replacement code is quite simple.
Some tests now have to deal with RBF though, and our rbf tests need work
since they look for the old onchaind messages.
In particular, when we can't afford the fee we want, we back off to
the next blockcount estimate, rather than spending all on fees
(necessarily). So test_penalty_rbf_burn no longer applies.
Changelog-Changed: Protocol: spending unilateral close transactions now use dynamic fees based on deadlines (and RBF), instead of fixed fees.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We also do it on every block, but since bitcoind can't always be counted
to rebroadcast for us, we might as well be aggressive!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We would only set it the first time, which was OK for how we were using it
before. Now we want to also set it for rebroadcast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Splitting into onchaind_tx() into onchaind_tx_unsigned() and
sign_and_get_witness() makes it easier to reuse for RBF.
Adding more information in onchain_signing_info is required too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `feerates`: added `floor` field for current minimum feerate bitcoind will accept
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #4473
Changelog-Deprecated: Plugins: `estimatefees` returning feerates by name (e.g. "opening"); use `fee_floor` and `feerates`.
Changelog-Fixed: Plugins: `bcli` now tells us the minimal possible feerate, such as with mempool congestion, rather than assuming 1 sat/vbyte.
Changelog-Added: Plugins: `estimatefees` can return explicit `fee_floor` and `feerates` by block number.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And consolidate descriptions into lightning-feerates().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` now allow "minimum" and NN"blocks" as `feerate` (`feerange` for `close`).
Drop try_get_feerate() in favor of explicit feerate_for_deadline() and
smoothed_feerate_for_deadline().
This shows us everywhere we deal with old-style feerates by names.
`delayed_to_us` and `htlc_resolution` will be moving to dynamic fees,
so deprecate those.
Note that "penalty" is still used for generating penalty txs for
watchtowers, and "unilateral_close" still used until we get zero-fee
anchors.
Changelog-Added: JSON-RPC: `feerates` `estimates` array shows fee estimates by blockcount from underlying plugin (usually *bcli*).
Changelog-Changed: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) value *slow* is now 100 block-estimate, not half of 100-block estimate.
Changelog-Deprecated: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) expressed as, "delayed_to_us", "htlc_resolution", "max_acceptable" or "min_acceptable". Use explicit block counts or *slow*/*normal*/*urgent*/*minimum*.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than have specific-purpose levels, have an array of
[blockcount, feerate], and rebuild the specific-purpose levels
for now on top.
We also keep a *separate* smoothed feerate, so you can ask for that
explicitly.
Since all the plugins used the same formula to derive the different
named fee levels, we apply the reverse to return to the underlying
estimates: updating the interface comes next.
This is ugly for now, but various specific-purpose levels will be
going away, as we shift to deadline-driven fees.
This temporarily breaks the floor calculation, so that test is
disabled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have the FEERATE_FLOOR constant if you don't care, but usually you want
to use the current bitcoind lower limit, so call get_feerate_floor()
(which is currently the same, but coming!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out the two bcli replacements I checked (`sauron` and
`trustedcoin`) don't even implement this, and the multiplier makes
more sense in lightningd, especially as we move to bcli just providing
raw feerate estimates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular:
- Bolt 4: add route blinding construction
- Bolt 4: add blinded payments
And this means it's not experimental, so we can turn it on
by default!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: blinded payments are now supported by default (not just with `--experimental-onion-messages`)
"Allow nodes to overshoot final htlc amount and expiry (#1032)"
Note that this also renamed `min_final_cltv_expiry` to the more-correct
`min_final_cltv_expiry_delta`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
"Allow nodes to overshoot the MPP `total_msat` when paying (#1031)"
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Allow slight overpaying, even with MPP, as spec now recommends.
"BOLT 4: Remove legacy format, make var_onion_optin compulsory."
This also renamed the redundant "tlv_payload" to "payload", so we
replace "tlv_tlv_payload" with "tlv_payload" everyhere!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is when they closed the channel, we can simply make our own tx to
expire the HTLC. (The other case is where we closed the channel, and
we have a special htlc_timeout tx which we have their signature for).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This breaks tests/test_closing.py::test_onchain_all_dust's accouting
checks.
That test doesn't really test what it claims to test; sure, onchaind
*says* it's going to ignore the output due to high fees, but the tx
still gets mined.
I cannot figure out what the test is supposed to look like, so I
simply disabled the accounting checks :(
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We add code for the case of spending a (timelocked) to-us output of an
HTLC output, so lightningd can do it (rather than onchaind doing all
the work itself).
onchaind still needs to know whether we bothered to create the tx
(fees might have caused it to evaporate, so it should consider it
immediately resolved rather than waiting for it), and what the
witnesses were, and which parts of the witnesses were signatures (as
these parts might change, with RBF or in future, combining other txs).
The inputs (known to onchaind) and the witnesses (told by lightningd)
uniquely identify the spend for the purposes of onchaind. In
particular, they definitely distinguish HTLC-timeout and HTLC-success
cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Importantly, the code in jsonrpc.c which actually does the io_break:
```
/* Once the stop_conn conn is drained, we can shut down. */
if (jcon->ld->stop_conn == conn && jcon->ld->state == LD_STATE_RUNNING) {
/* Return us to toplevel lightningd.c */
log_debug(jcon->ld->log, "io_break: %s", __func__);
io_break(jcon->ld);
```
By having the state not set until later, we avoid running this. Of course,
we need to avoid calling the main loop when we get there, if we've already
been told to shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fun story. We're changing onchaind to hand txs to us, and we will
construct them and do the broadcast for it. lightningd tells onchaind
the witness it used (with flags to indicate which fields were
signatures so should be ignored) so onchaind can recognize the tx
when/if it is mined.
And when onchaind was waiting for a CLTV delay, it wouldn't tell
lightningd yet, but wait until the parent was sufficiently deep
But this caused bugs!
In particular, on replay, onchaind would see transactions which it
hasn't sent yet. This was not a problem before, as onchaind had
created the tx, even if it hadn't told lightningd to broadcast it, so
recognized the variant when it came in. When we're relying on
lightningd to tell us what the tx will look like, this doesn't work
any more.
The cause of this is that we fire off txowatches ("this output was
spent!") while we process blocks, and only fire off txwatches ("this
tx increased depth") once all the current blocks are processed. Often
this didn't matter, since we replay messages to onchaind from the
database, *but* we trim the last few blocks on restart (or, if there's
a small reorg while we're stopped), and we can hit this misordering.
Changing our topology code to only ever process one block at a time
would be a solution, but slows down catchup (and tests, where we often
mine a run of blocks).
So, this seems like a premature optimization, but it's really
required! And in future, lightningd can use this knowledge of pending
transactions to combine them in more clever ways.
Note that if a tx is valid at block N, we broadcast it once we see
block N-1, to get it in the mempool for block N.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was once only called on failure, now it's always called (if set).
It was called different things in different places, so unify it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We always call channel_fail_transient() on all channels when a peer
connects, to clean up any previous connections. However, when
we startup, this channel doesn't have an owner yet, resulting in
a fairly weird INFO level message.
Reported-by: Michael Schmook @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `lightningd`: don't log gratuitous "Peer transient failure" message on first connection after restart.
There are cases (difficult to reproduce with a test) where
a payment will fail one time and succeed later.
As far I understand in this case the groupid field of the payment
is the same, and the only thing that change is the status, so
our logic inside the delpay is ambiguous where it is not
possible to delete a payment as described in https://github.com/ElementsProject/lightning/issues/6114
A sequence of commands that explain the problem is
```
$ lc -k listpays payment_hash=H
{
"pays": [
{
"bolt11": "I",
"destination": "redacted",
"payment_hash": "H",
"status": "complete",
"created_at": redacted,
"completed_at": redacted,
"preimage": "P",
"amount_msat": "redacted",
"amount_sent_msat": "redacted"
}
]
}
$ lc delpay H complete
{
"code": 211,
"message": "Payment with hash H has failed status but it should be complete"
}
```
In this case, the delpay is not able to delete a payment because the
listpays is returning only the succeeded one, so by running the
listsendpays we may see the following result where our delpay logic
will be stuck because it works to ensure that all the payments stored
in the database has the status specified by the user
```
➜ VincentSSD clightning --testnet listsendpays -k payment_hash=7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4
{
"payments": [
{
"id": 322,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 1,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 1664,
"created_at": 1679510203,
"completed_at": 1679510205,
"status": "failed",
"bolt11": "lntb1pjpkj4xsp52trda39rfpe7qtqahx8jjplhnj3tatxy8rh6sc6afgvmdz7n0llspp50lr5hmdm0re0xvcp2hv3nf2wwvx0r8q3h3e7jmqz0awdfg6w206qdp0w3jhxarfdenjqargv5sxgetvwpshjgrzw4njqun9wphhyaqxqyjw5qcqp2rzjqtp28uqy77te96ylt7ek703h4ayldljsf8rnlztgf3p8mg7pd0qzwf8a3yqqpdqqqyqqqqt2qqqqqqgqqc9qxpqysgqgeya2lguaj6sflc4hx2d89jvah8mw9uax4j77d8rzkut3rkm0554x37fc7gy92ws9l76yprdva2lalrs7fqjp9lcx40zuty8gca0g5spme3dup"
},
{
"id": 323,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 2,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 3663,
"created_at": 1679510205,
"completed_at": 1679510207,
"status": "failed"
},
{
"id": 324,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 3,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 3663,
"created_at": 1679510207,
"completed_at": 1679510209,
"status": "failed"
},
{
"id": 325,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 4,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 4663,
"created_at": 1679510209,
"completed_at": 1679510221,
"status": "complete",
"payment_preimage": "43f746f2d28d4902489cbde9b3b8f3d04db5db7e973f8a55b7229ce774bf33a7"
}
]
}
```
This commit solves the problem by forcing the delete query in the
database to specify status too, and work around this kind of
ambiguous case.
Fixes: f52ff07558 (lightningd: allow delpay to delete a specific payment.)
Reported-by: Antoine Poinsot <darosior@protonmail.com>
Link: https://github.com/ElementsProject/lightning/issues/6114
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Co-Developed-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: delpay be more pedantic about delete logic by allowing
delete payments by status directly on the database.
Changelog-Added: JSON-RPC: `listclosedchannels` to show old, dead channels we previously had with peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Libwally update breaks compatibility, so
we do this in one large step.
Changelog-Changed: JSON-RPC: elements network PSET now only supports PSETv2.
Changelog-Added: JSON-RPC: PSBTv2 supported for fundchannel_complete, openchannel_update, reserveinputs, sendpsbt, signpsbt, withdraw and unreserveinputs parameter psbt, openchannel_init and openchannel_bump parameter initialpsbt, openchannel_signed parameter signed_psbt and utxopsbt parameter utxopsbt
At the moment only lightingd needs it, and this avoids missing any
places where we do bip32 derivation.
This uses a hsm capability to mean we're backwards compatible with older
hsmds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: we now always double-check bitcoin addresses are correct (no memory errors!) before issuing them.
It's needed as the db and wallet is being set up (db migrations), so
it's simpler this way to always use ld->bip32_base for the next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Importantly, adds the version number at the *front* to help future
parsing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'fix-hsm-check-pubkey.patch':
fixup! hsmd: capability addition: ability to check pubkeys.
We were handing 3 to hsmd (and Ken added that in 7b2c5617c1,
so I guess he's OK with that being the minimum supported version!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: require the `"jsonrpc": "2.0"` property (requests without this deprecated in v0.10.2).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: `checkmessage` now always returns an error when the pubkey is not specified and it is unknown in the network graph (deprecated v0.12.0)
It works for the trivial case, where groupid and partid are the same,
but silently deletes nothing in the other cases (or worse, deletes the
wrong entry!).
See: #5835
Changelog-Fixed: `delpay`: actually delete the specified payment (mainly found by `autoclean`).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were feeding in the raw JSON, which escapes \". Then we were
escaping *again* to return it.
Reported-by: @m-schmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `datastore` handles escapes in `string` parameter correctly.
This fixes the following compilation error
and allow rebuilding again on 32-bit platform.
```
lightningd/dual_open_control.c: In function 'validate_input_unspent':
lightningd/dual_open_control.c:2627:43: error: format '%llu' expects argument of type 'long long unsigned int', but argument 4 has type 'size_t' {aka 'unsigned int'} [-Werror=format=]
2627 | err = tal_fmt(pv, "PSBT input at index %"PRIu64
| ^~~~~~~~~~~~~~~~~~~~~~~
2628 | " missing serial id", i);
| ~
| |
| size_t {aka unsigned int}
ccan/ccan/tal/str/str.h:43:46: note: in definition of macro 'tal_fmt'
43 | tal_fmt_(ctx, TAL_LABEL(char, "[]"), __VA_ARGS__)
| ^~~~~~~~~~~
```
PS: apparently I'm the only remaining people that ran cln on an old raspberry pi 2?
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
These were deprecated in v0.12.0, hence scheduled for removal next version anyway
(use local_fund_msat and remote_funds_msat).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since it's not spec-final yet (hell, it's not even properly specified
yet!) we need to put it behind an experimental flag.
Unfortunately, we don't have support for doing this in a plugin; a
plugin must present features before parsing options. So we need to do
it in core.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is needed for the next patch, which does this from the peer_connected hook!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `sendcustommsg` can now be called by a plugin from within the `peer_connected` hook.
We keep the node_id, not a pointer to the peer.
This also means that it might have reconnected while we were in the hook, so make
sure we ignore the result if it's in state PEER_CONNECTED.
And remove the `tal_steal(peer, hook_payload)` which doesn't do anything: the
plugin_hook call steals hook_payload anyway!
Fixes: #5944
It's not likely but possible that the node's settings will shift btw a
start and an RBF; we persist the setting to the database so we don't
lose it.
Right now holding onto it forever is kind of extra but maybe we'll
reuse the setting for splices? idk.
Should this be a channel type??
`openchannel_init` takes a psbt, which we pipe over to dualopend
process.
If the peer requests that they'll only accept confirmed inputs, we need
to go validate those before we continue.
This wires up the harness for this (validation check yet tc)
technically we don't need this info after the channel opens, but for any
subsequent RBF (and maybe splice?) we need to remember what the
open/accept peer signaled
not amazing, since we'll probably call openchannel_update multiple
times per open, but this is the simplest way to confirm that we're
not sending unconfirmed outputs to peer.
This will save a lot of RPC ping/pong when plugins still need to iterate
both, `listpeers` and `listpeerchannels`. When `num_channels` is 0 they
can skip additional calls.
Changelog-Added: RPC `listpeers` output now has `num_channels`.
Though there's already a `createinvoice` command, there are usecases where a
user may want to sign an invoice that they don't yet have the preimage to. For
example, they may have an htlc_accepted plugin that pays to obtain the preimage
from someone else and returns a `{ "result": "resolve", ... }`.
This RPC command addresses this usecase without overly complicating the
semantics of the existing `createinvoice` command.
Changelog-Added: JSON-RPC: `signinvoice` new command to sign BOLT11
invoices.
This allows to accept safely long paths as options
and does not truncate them
as https://github.com/ElementsProject/lightning/issues/5576
described
Changelog-Fixed: cli: accepts long paths as options
Suggested-by: @rustyrussell <rusty@rustcorp.com.au>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
In v0.11 (71f736678f) we changed lightningd to wait for gossipd to
acknowledge blocks before updating blockheight: this resolved a problem
which lnprototest had where it wanted to know when we'd fully digested
a block.
However, it broke the syncing case: until then we don't even tell
gossipd, so this stayed at zero. We should use the current blockheight
for that corner case!
Fixes: #5894
Changelog-Fixed: JSON-RPC: `getinfo` `blockheight` no longer sits on 0 while we sync with bitcoind the first time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When a channel open fails, we use tx-abort instead of warning/error.
This means that the peer won't disconnect! And instead when a new
message arrives, we'll need to rebuild the dualopend subd (if missing).
Makes opens a bit easer to retry (no reconnect needed), as well as keeps
the connection alive for other channels we may have with that peer.
Changelog-Changed: Experimental-Dual-Fund: open failures don't disconnect, but instead fail the opening process
```
----------------------------- Captured stderr call -----------------------------
Sending onchaind an invalid message 03ed00000000000000004e52a9129a66619d6809b1024eb9a0159f173a988f3a5d0bdd2447b4fcc24cef
lightningd: FATAL SIGNAL 6 (version 3c57147-modded)
```
The channel state can also be `FUNDING_SPEND_SEEN` if onchaind is still
starting up.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In various circumstances we can start a reconnection while one is
already going on. These can stockpile if the node really is unreachable.
Reported-by: @whitslack
Fixes: #5654
Changelog-Fixed: lightningd: we no longer stack multiple reconnection attempts if connections fail.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The leak-detector can't find unconnected_htlcs_in on the stack and
incorrectly flags this as a leak. However, it is appropriately tal
allocated and freed.
Changelog-None
There were cases where address it's empty, and this cases are not right if the
field is considered optional.
This makes it required and add the field also when `--offline` is set.
Changelog-Changed: JSON-RPC: `getinfo` `address` array is always present (though may be empty)
We need to check if the key parameter is an empty array in
`listdatastore` as we do assume an array of at least length 1 in
`wallet.c:5306`.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
We didn't actually populate them properly, and the real annotations
are on inputs and outputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `listtransactions` `channel` and `type` field removed at top level.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-onion-message" BOLTVERSION=guilt/offers
```
Mainly textual, though I neatened the extra fields check for TLVs with
blinding, and implemented the "no other fields" requirement for
non-final onion message hops.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-route-blinding" BOLTVERSION=guilt/offers
```
Other than textual changes, this does:
1. Ensures we put total_amount_msat in onion final hop (reported by @t-bast).
2. Require that they put total_amount_msat in onion final hop.
3. Return `invalid_onion_blinding` exactly as defined by the spec (i.e. less
aggressive when we're the final hop) (also reported by @t-bast, but I already
knew).
See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (total_amount_sat required, Eclair compat)
You can use rs->nextcase, but we don't always keep that around, so
keep a flag in onion_payload.
We'll use this in the "do we need to return a blinded error code"
patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also, we don't need to pass the total length to the field parsers,
just the length for this field (confusingly, this was called
"data_length").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will give the user an option to set a custom port when using
discovered IPs for node_announcents. Without this, only the selected
networks default port can used.
Changelog-Added: Adds --announce-addr-discovered-port config option to set custom port for IP discovery.
This switch was not doing anything useful anymore.
We deprecate it anyways to notify the user about the new switch.
Changelog-Deprecated: The old --disable-ip-discovery config switch
This adds the option to explicitly enable ip-discovery, which maybe
helpful for example when a user wants TOR announced along with
discovered IPs to improve connectivity and have TOR just as a fallback.
Changelog-Added: Adds config switch 'announce-addr-discovered': on/off/auto
After connecting 100,000 peers with one channel each (not all at
once!), we see various places where we exhibit O(N^2) behaviour.
Fix these by keeping a map of id->peer instead of a simple
linked-list.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
E Global errors:
E - Node /tmp/ltests-adkwu44c/test_logging_1/lightning-2/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E "lightningd/peer_control.c:2203 (load_channels_from_wallet)",
E "lightningd/lightningd.c:1105 (main)"
E ],
E "label": "lightningd/peer_control.c:2203:struct htlc_in_map",
E "parents": [
E "lightningd/lightningd.c:107:struct lightningd"
E ],
E "value": "0x55d920a345e8"
E }
E ]
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: new command `listpeerchannels` now contains information on direct channels with our peers.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We're soon going to call json_add_unsaved_channel and
json_add_uncommitted_channel from a new place, where we want the peer
state directly included.
Based on patch by @vincenzopalazzo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids any confusion between primitive and string ids, and in
particular stops an issue with commando once it starts chaining ids,
that weird ids can be double-escaped and commando will not recognize
the response, leaving the client hanging. It's the client's fault for
using a weird id, but it's still rude (and triggered by our tests!).
It also makes substituting the id in passthrough simpler, FTW.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When a channel is not yet confirmed onchain and only has an alias,
`getroute` will return the alias under the "channel" key. However, if
you supply that to `sendpay`, it will not be able to find the channel
and will fail since it calls `find_channel_for_htlc_add` and that only
looks for channels by their scid and doesn't consider their alias.
This patch makes `find_channel_for_htlc_add` also consider channel
aliases when looking for channels.
Changelog-Fixed: JSON-RPC: `sendpay` now can send to a short-channel-id alias for the first hop.
We were stressing the servers if node cannot be found. Only do lookup
on manual connect commands.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: lightningd: Only use DNS server address lookup on manual `connect` commands, not normal reconnection attempts.
This broke BTCPayServer, so revert. I originally (accidentally!)
implemented this such that it broadcast both DNS and IP entries, but
Michael reported earlier that they still don't propagage well, so
simply suppress them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5795
Changelog-Changeed: Config: `announce-addr-dns` needs to be set to *true* to put DNS names into node announcements, otherwise they are suppressed.
Changelog-Deprecated: Config: `announce-addr-dns` (currently defaults to `false`). This will default to `true` once enough of the network has upgraded to understand DNS entries.
This reverts commit 43b037ab0b.
Nicholas Dorier says BTC Payserver still wants this for another year
or so.
Changelog-Added: JSON-RPC: reverts requirement for "jsonrpc" "2.0" inside requests (still deprecated though, just for a while longer!)
The read_json() call expects len_read to be the amount of *new*
data read. If we call this back without resetting, it will parse
this much random junk in the buffer.
Fixes: #5766
Changelog-Fixed: Plugin: `autoclean` could misperform or get killed due to lightningd's invalid handling of JSON batching.
If we both support large channels, we can actually send giant HTLCs.
This, in turn, fixes pay (which relies on this field!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` now knows it can use locally-connected wumbo channels for large payments.
Fixes: #5250Fixes: #5417
This is a mistake that I introduced while I implemented the lightningd custom error for hsmd.
In particular, when the command line parser check
if the file is encrypted make an additional check regarding the existence of the file.
This can be a not useful check, but due to the delicate nature of the hsm file, it is better to check if exist and if it doesn't exist an informative line inside the log is emitted that notifies the user that the file does not exist.
This log may return useful while debugging disaster
(that can happen) to understand that there is something strange (eg. the user moves the hsm file somewhere else).
Fixes https://github.com/ElementsProject/lightning/issues/5719
Changelog-Fixed: lightningd: do not abort while parsing hsm pwd
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This was found by tests/test_plugin.py::test_important_plugin and
was NOT a flake!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: only just committed
This is a minimal fix: we wait until all plugins reply from init before
continuing. Really large or busy nodes can have other things monopolize
lightningd, then the timer goes off and we blame the plugin (which has
responded, we just haven't read it yet!).
The real answer is to have some timeouts only advance when we're idle,
or have them low-priority so we only activate them when we're idle (this
doesn't apply to all timers: some are probably important!). But
this is a minimal fix for -rc3.
Fixes: https://github.com/ElementsProject/lightning/issues/5736
Changelog-Fixed: plugins: on large/slow nodes we could blame plugins for failing to answer init in time, when we were just slow.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And document support for it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `getmanfest` response can contain `nonnumericids` to indicate support for modern string-based JSON request ids.
Changelog-Deprecated: Plugins: numeric JSON request ids: modern ones will be strings (see doc/lightningd-rpc.7.md!)
We had a quadratic check when processing a block, and this just
reduces it to be linear. Combined with the previous fix of adding txs
multiple times, this should speed up block processing considerably.
We were using a quadratic lookup to locate the transactions we
broadcast, so let's use an `htable` instead, allowing for constant
lookups during block processing.
The list of outgoing transactions was growing with every
rebroadcast. Now we just check whether it is already in the list and
don't add it anymore
Changelog-Fixed: ld: Reduce identification of own transactions to not slow down over time, reducing block processing time
Suggested-by: Matt Whitlock <@whitslack>
On testnet I noticed if we can't reach bitcoind for some reason, we'll
keep RBFing our penalty tx ("it didn't go in, RBF harder!!"). Makes
no sense to grossly exceed the amount needed for next block, so simply
cap penalty at 2x "estimatesmartfee 2 CONSERVATIVE".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is how we put new invoice_requests into the db; this will be used
by a new "invoicerequest" command which replaces "offerout".
The API is now the same as the offers api.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We no longer use offers for "I want to send you money", but we'll use
invoice_requests directly. Create a new table for them, and
associated functions.
The "localofferid" for "pay" and "sendpay" is now "localinvreqid".
This is an experimental-only option, so document the change under
experimental only.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `pay` and `sendpay` `localofferid` is now `localinvreqid`.
I know this is an unforgivably large diff, but the spec has changed so
much that most of this amounts to a rewrite.
Some points:
* We no longer have "offer_id" fields, we generate that locally, as all
offer fields are mirrored into invoice_request and then invoice.
* Because of that mirroring, field names all have explicit offer/invreq/invoice
prefixes.
* The `refund_for` fields have been removed from spec: will re-add locally later.
* quantity_min was removed, max == 0 now mean "must specify a quantity".
* I have put recurrence fields back in locally.
This brings us to 655df03d8729c0918bdacac99eb13fdb0ee93345 ("BOLT 12:
add explicit invoice_node_id.")
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And check it in invoice.c, insead of a hack where we compare against invhash.
Restore checking, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The "path" is just a message to ourselves. This meets the minimal
requirement for bolt12 invoices: that there be a blinded path (at
least so we can use the path_id inside in place of "payment_secret").
We expose the method to make this path_id to a common routine: offers
will need this for generating more sophisticated paths.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of doing command_fail() in the else, do it immediately then
unindent the normal path.
No code changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a scheme where lightningd itself would put a per-node secret in
the blinded path, then we'd tell the caller when it was used. Then it
simply checks the alias to determine if the correct path was used.
But this doesn't work when we start to offer multiple blinded paths.
So go for a far simpler scheme, where the secret is generated (and
stored) by the caller, and hand it back to them.
We keep the split "with secret" or "without secret" API, since I'm
sure callers who don't care about the secret won't check that it
doesn't exist! And without that, someone can use a blinded path for a
different message and get a response which may reveal the node.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We actually want lightningd to create these, since it wants to put the
path_id secret in the last element. So best API is actually a generic
one, rather than separate APIs to create first and last ones.
And really, the more explicit initialization makes the users clearer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This current spec is not strict enough: we might complain that the
next peer is not connected, for example, which leaks information.
So return WIRE_INVALID_ONION_BLINDING even if we're the first hop
on the path, to be safe.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes us match eed2ab0c30ad7f93e3b2641ca9d7ade32f3d121d
("Use `invalid_onion_blinding` everywhere").
1. Numerous typographical changes.
2. Make sure we *always* return WIRE_INVALID_ONION_BLINDING if
we're in a blinded path.
3. Handle p->total_msat correctly (MPP payments).
4. Reorganize blinding handling just like spec order.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `listfunds` now lists coinbase outputs as 'immature' until they're spendable
Changelog-Changed: JSON-RPC: UTXOs aren't spendable while immature