Commit Graph

3872 Commits

Author SHA1 Message Date
Rusty Russell 0d94d2d269 gossipd: batch outpoints spent, add block height.
It's a bit more optimal, and tells gossipd exactly what height the
spend occurred at (with multiple blocks, it's not always the current
height!).  It will need that next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell bfc21cbb55 gossipd: set no_forward bit on channel_update for private channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: We now set the `dont_forward` bit on private channel_update's message_flags (as per latest BOLTs).
2022-09-24 15:22:27 +09:30
Rusty Russell f52ff07558 lightningd: allow delpay to delete a specific payment.
This is actually what the autoclean plugin wants, especially since
you can't otherwise delete a payment which has failed then succeeded.

But insist on neither or both being specified, at least for now.

Changelog-Added: JSON-RPC: `delpay` takes optional `groupid` and `partid` parameters to specify exactly what payment to delete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell fa7d732ba6 lightningd: allow a connection to specify db batching.
Previous commit was a hack which *always* batched where possible, this
is a more sophisticated opt-in varaint, with a timeout sanity check.

Final performance for cleaning up 1M pays/forwards/invoices:

```
$ time l1-cli autoclean-once succeededpays 1
{
   "autoclean": {
      "succeededpays": {
         "cleaned": 1000000,
         "uncleaned": 26895
      }
   }
}

real	6m9.828s
user	0m0.003s
sys	0m0.001s
$ time l2-cli autoclean-once succeededforwards 1
{
   "autoclean": {
      "succeededforwards": {
         "cleaned": 1000000,
         "uncleaned": 40
      }
   }
}

real	3m20.789s
user	0m0.004s
sys	0m0.001s
$ time l3-cli autoclean-once paidinvoices 1
{
   "autoclean": {
      "paidinvoices": {
         "cleaned": 1000000,
         "uncleaned": 0
      }
   }
}

real	6m47.941s
user	0m0.001s
sys	0m0.000s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `batching` command to allow database transactions to cross multiple back-to-back JSON commands.
2022-09-22 15:19:46 +02:00
Rusty Russell 555b8a2f7a lightningd: don't always wrap each command in a db transaction.
This shows the benefits of batching when being slammed by autoclean:


Before:

```
   plugin-autoclean: last 1000 deletes took 6699022 nsec each
   plugin-autoclean: last 1000 deletes took 6734025 nsec each
   plugin-autoclean: last 1000 deletes took 6681189 nsec each
   plugin-autoclean: last 1000 deletes took 6597148 nsec each
   plugin-autoclean: last 1000 deletes took 6637085 nsec each
   plugin-autoclean: last 1000 deletes took 6801425 nsec each
   plugin-autoclean: last 1000 deletes took 6788572 nsec each
   plugin-autoclean: last 1000 deletes took 6603641 nsec each
   plugin-autoclean: last 1000 deletes took 6642947 nsec each
   plugin-autoclean: last 1000 deletes took 6590495 nsec each
   plugin-autoclean: last 1000 deletes took 6695076 nsec each
   plugin-autoclean: last 1000 deletes took 6477981 nsec each
```

After:
```
   plugin-autoclean: last 1000 deletes took 342764 nsec each
   plugin-autoclean: last 1000 deletes took 375031 nsec each
   plugin-autoclean: last 1000 deletes took 357564 nsec each
   plugin-autoclean: last 1000 deletes took 381581 nsec each
   plugin-autoclean: last 1000 deletes took 337989 nsec each
   plugin-autoclean: last 1000 deletes took 329391 nsec each
   plugin-autoclean: last 1000 deletes took 328322 nsec each
   plugin-autoclean: last 1000 deletes took 372810 nsec each
   plugin-autoclean: last 1000 deletes took 351228 nsec each
   plugin-autoclean: last 1000 deletes took 413885 nsec each
   plugin-autoclean: last 1000 deletes took 348317 nsec each
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 3079afb024 lightningd: add `delforward` command.
Changelog-Added: JSON-RPC: `delforward` command to delete listforwards entries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 7420a7021f lightningd: add `listhtlcs` to list all the HTLCs we know about.
Using `listfowards` for this wrong; expose this directly if people
care (and unlike listforwards, which could be deleted, we have to
remember these while the channel is still open!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listhtlcs` new command to list all known HTLCS.
2022-09-22 15:19:46 +02:00
Rusty Russell 311807ff1f lightningd: add in_htlc_id / out_htlc_id to listforwards.
And document that we never know payment_hash.

Changelog-Added: JSON-RPC: `listforwards` now shows `in_htlc_id` and `out_htlc_id`
Changelog-Changed: JSON-RPC: `listforwards` now never shows `payment_hash`; use `listhtlcs`.
2022-09-22 15:19:46 +02:00
Rusty Russell 2022e4a7a9 wallet: simplify payments lookup so sqlite3 uses index.
Filtering by status is rare, so we can do it in the caller; just let sqlite3
filter by payment_hash.

With ~650,000 payments in db:

Before:
```
129/300000 complete 5.60/sec (33078 invs, 169 pays, 0 retries) in 30 seconds. 19 hours-14 hours remaining.
201/300000 complete 7.20/sec (43519 invs, 241 pays, 0 retries) in 40 seconds. 16 hours-11 hours remaining.
257/300000 complete 5.60/sec (54568 invs, 289 pays, 0 retries) in 50 seconds. 16 hours-14 hours remaining.
305/300000 complete 4.80/sec (65772 invs, 337 pays, 0 retries) in 60 seconds. 16 hours-17 hours remaining.
361/300000 complete 5.60/sec (75875 invs, 401 pays, 0 retries) in 70 seconds. 16 hours-14 hours remaining.
```

After:
```
760/300000 complete 40.00/sec (19955 invs, 824 pays, 0 retries) in 20 seconds. 2 hours-2 hours remaining.
1176/300000 complete 41.60/sec (30082 invs, 1224 pays, 0 retries) in 30 seconds. 2 hours-119 minutes remaining.
1584/300000 complete 40.80/sec (40224 invs, 1640 pays, 0 retries) in 40 seconds. 2 hours-2 hours remaining.
1984/300000 complete 40.00/sec (49938 invs, 2048 pays, 0 retries) in 50 seconds. 2 hours-2 hours remaining.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 17858c9490 lightningd: deprecated "delexpiredinvoice", put functionality in autoclean plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `delexpiredinvoice`: use `autoclean-once`.
2022-09-22 15:19:46 +02:00
Rusty Russell 0868fa9f1e lightningd: allow extra tlv types in non-experimental mode.
The old `experimental-accept-extra-tlv-types` is now `accept-htlc-tlv-types`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `accept-htlc-tlv-types` lets us accept unknown even HTLC TLV fields we would normally reject on parsing (was EXPERIMENTAL-only `experimental-accept-extra-tlv-types`).
2022-09-22 17:19:11 +09:30
Christian Decker 67467213cb opening: Add `dev-allowdustreserve` option to opt into dust reserves
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
2022-09-21 11:25:47 +02:00
Christian Decker 5a54f450bd openingd: Pass `reserve` down to openingd when funding 2022-09-21 11:25:47 +02:00
Christian Decker 8d6423389a openingd: Wire `reserve` value through to `openingd` 2022-09-21 11:25:47 +02:00
Christian Decker 9a97f8c154 plugin: Add `reserve` to `openchannel` hook result
Changelog-Added: plugin: The `openchannel` hook may return a custom absolute `reserve` value that the peer must not dip below.
2022-09-21 11:25:47 +02:00
Christian Decker 5c1de8029a openingd: Add `reserve` to `fundchannel` and `multifundchannel`
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel` and `fundchannel_start` now accept a `reserve` parameter to indicate the absolute reserve to impose on the peer.
2022-09-21 11:25:47 +02:00
Greg Sanders 248d60d7bd Don't report redundant feerates to subdaemons 2022-09-19 19:10:35 +02:00
Rusty Russell 701dd3dcef memleak: remove exclusions from memleak_start()
Add memleak_ignore_children() so callers can do exclusions themselves.

Having two exclusions was always such a hack!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell 3380f559f9 memleak: simplify API.
Mainly renaming.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell 2da5244e83 jsonrpc: make error codes an enum.
This allows GDB to print values, but also allows us to use them in
'case' statements.  This wasn't allowed before because they're not
constant terms.

This also made it clear there's a clash between two error codes,
so move one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: Error code from bcli plugin changed from 400 to 500.
2022-09-19 10:18:55 +09:30
Rusty Russell bbe1711e16 lightningd: use jcon logging for commands where possible.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell caecd1ee0a lightningd: don't log JSON ids as debug, use log io.
They are cute, sure, but they do spam the logs.

@Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell e8ef42b741 plugin: wire JSON id for commands which caused hooks to fire.
Most obvious one is the "connect" hook.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell eceb9f4328 lightningd: wire plugin command JSON id through to plugin commands.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell ea7903f69a lightningd: trace JSON id prefixes through sendrawtx.
First, merge the _ahf_ and non-ahf interfaces.
Second, remove the always-NULL txs->cmd field.

Then, add optional id_prefix for bitcoind_sendrawx, so if it's
triggered by a command (e.g. "withdraw") it's shown correctly in logs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell a9557d5194 lightningd: derive JSONRPC ids from incoming id (append /cln:<method>#NNN).
Usually the calls are spontanous, so it's just "cln:<method>#NNN", but
json_invoice() calls listincoming, and json_checkmessage calls
listnodes, so those become "cli:invoice-<pid>/cln:listincoming#NNN".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell 8fcf880e0f lightningd: explicitly remember if JSON id was a string.
This lets us use 'cmd->id' as an unquoted string (for building
new ids!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell ed3f700991 lightningd: use string as json req ids when we create them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell 99f2019a24 lightningd: add jsonrpc_request_start_raw instead of NULL method.
Since we want to use methodname to create id, don't overload it
for a raw request.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Rusty Russell 2d7cf153ad lightningd: log JSON request ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-16 12:31:45 +09:30
Michael Schmoock 532544ce4f gossipd: rename remote_addr to discovered_ip within gossipd
This is cleaner because, the `remote_addr` and `discovered_ip` are
related but two different things.

Within connectd and lightningd we use the peers `remote_addr` feature
to validate (and guess a port) to be used for IP discovery.

Also when a peer reports us a `remote_addr`, this is given to the plugin API
via the `peer_connected` hook. The network port here is not modified for
godd reason! This can be used i.e. to detect if we are behind a NAT.

But once lightningd figures enough peers report the same `remote_addr`,
it sets the port to the selected network and tells gossipd to use that for
`node_announcement` updates.

Hence, within gossipd, there is no (should not be) `remote_addr`.

Changelog-None
2022-09-15 13:30:06 +09:30
Michael Schmoock c8ab8192ca peer_control: getinfo show correct port on discovered IPs
Changelog-Fixed: peer_control: getinfo shows the correct port on discovered IPs
2022-09-15 13:30:06 +09:30
Rusty Russell 4ca6b36439 lightningd: refuse to upgrade db on non-released versions by default.
This is a good sanity check that users understand that if they upgrade
to master mid-cycle they can't go back!

Suggested-by: @wtogami
Changelog-Added: Config: `--database-upgrade=true` required if a non-release version wants to (irrevocably!) upgrade the db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-15 13:25:58 +09:30
fiatjaf 1ef8fb7ef8 rename `block_processed` to `block_added` 2022-09-14 13:50:38 -05:00
fiatjaf 9b33a921f0 Add plugin notification topic "block_processed".
Changelog-Added: Plugins: Added notification topic "block_processed".
2022-09-14 13:50:38 -05:00
Rusty Russell 023a688e3f lightningd: fix spurious leak report.
We can (and probably should!) allocate this off tmpctx rather than
off the response.

Also, json_add_invoice doesn't actually match the normal pattern,
so fix that.


```
E           ValueError: 
E           Node errors:
E           Global errors:
E            - Node /tmp/ltests-_vbj8az8/test_pay_fail_unconfirmed_channel_1/lightning-2/ has memory leaks: [
E               {
E                   "backtrace": [
E                       "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E                       "wallet/invoices.c:81 (wallet_stmt2invoice_details)",
E                       "wallet/invoices.c:697 (invoices_get_details)",
E                       "wallet/wallet.c:2946 (wallet_invoice_details)",
E                       "lightningd/invoice.c:1296 (json_add_invoices)",
E                       "lightningd/invoice.c:1370 (json_listinvoices)",
E                       "lightningd/jsonrpc.c:625 (command_exec)",
E                       "lightningd/jsonrpc.c:753 (rpc_command_hook_final)",
E                       "lightningd/plugin_hook.c:280 (plugin_hook_call_)",
E                       "lightningd/jsonrpc.c:841 (plugin_hook_call_rpc_command)",
E                       "lightningd/jsonrpc.c:938 (parse_request)",
E                       "lightningd/jsonrpc.c:1035 (read_json)",
E                       "ccan/ccan/io/io.c:59 (next_plan)",
E                       "ccan/ccan/io/io.c:407 (do_plan)",
E                       "ccan/ccan/io/io.c:417 (io_ready)",
E                       "ccan/ccan/io/poll.c:453 (io_loop)",
E                       "lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
E                       "lightningd/lightningd.c:1194 (main)",
E                       "../csu/libc-start.c:308 (__libc_start_main)"
E                   ],
E                   "label": "wallet/invoices.c:81:struct invoice_details",
E                   "parents": [
E                       "common/json_stream.c:41:struct json_stream",
E                       "ccan/ccan/io/io.c:91:struct io_conn **NOTLEAK**",
E                       "lightningd/lightningd.c:107:struct lightningd"
E                   ],
E                   "value": "0x55a57a77de08"
E               }
E           ]
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-14 13:43:27 -05:00
Christian Decker cb3ee0ac2e wallet: Load and value `completed_at` timestamp from DB 2022-09-14 13:14:10 +02:00
niftynei 1980ba420b notif: dont send balance snapshot for not yet opened channel
We were double counting channel lease fees because we were double firing
the channel open event sequence (so to speak). If we don't report
balances for unopened channels, we don't have this problem?

Changelog-Changed: Plugins: `balance_snapshot` notification does not send balances for channels that aren't locked-in/opened yet
2022-09-14 15:59:37 +09:30
Michael Schmoock e0d6f3ceb1 connectd: DNS Bolt7 #911 no longer EXPERIMENTAL
Changelog-Changed: Bolt7 #911 DNS annoucenent support is no longer EXPERIMENTAL
2022-09-13 06:42:20 +09:30
Rusty Russell 375215a141 lightningd: more graceful shutdown.
Be more graceful in shutting down: this should fix the issue where
bookkeeper gets upset that its commands are rejected during shutdown,
and generally make things more graceful.

1. Stop any new RPC connections.
2. Stop any per-peer daemons (channeld, etc).
3. Shut down plugins.
4. Stop all existing RPC connections.
5. Stop global daemons.
6. Free up peer, chanen HTLC datastructures.
7. Close database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: RPC operations are now still available during shutdown.
2022-09-12 14:00:41 +02:00
Rusty Russell 4ca1203eb8 doc: include recent BOLT recommendation on grace period.
This includes the recommendation that we use 10 minute grace period,
so add quotes to where we use that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell 5b7f14a7cb channeld/dualopend/lightningd: use channel_ready everywhere.
This alters the billboard, but that's a human-readable thing so not
noted in CHANGELOG.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `listpeers` `status` now refers to "channel ready" rather than "funding locked" (BOLT language change for zeroconf channels)
Changelog-Added: JSON-RPC: `channel_opened` notification `channel_ready` flag.
Changelog-Deprecated: JSON-RPC: `channel_opened` notification `funding_locked` flag (use `channel_ready`: BOLTs namechange).
2022-09-12 09:34:52 +09:30
Rusty Russell 1b30ea4b82 doc: update BOLTs to bc86304b4b0af5fd5ce9d24f74e2ebbceb7e2730
This contains the zeroconf stuff, with funding_locked renamed to
channel_ready.  I change that everywhere, and try to fix up the
comments.

Also the `alias` field is called `short_channel_id`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `funding_locked` is now called `channel_ready` as per latest BOLTs.
2022-09-12 09:34:52 +09:30
Rusty Russell 6cf3d47505 offers: remove backwards-compatiblity invoice_request signatures.
We changed the field name in v0.11.0, so this breaks compat with
v0.10.2.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell 29264e83fb lightningd: remove `use_proxy_always` parameter to plugin init.
Changelog-Removed: Plugins: plugin init `use_proxy_always` (deprecated v0.10.2)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell 733ce81bd4 plugins: require usage for plugin APIs.
Changelog-Removed: JSON-RPC: plugins must supply `usage` parameter (deprecated v0.7)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell 15751ea1b8 lightningd: do inline parsing for listforwards status parameter
We can do this now the function is cleaned up.

Always better to do the work inside param() since then `check`
gets the benefit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell a45ec78c36 lightningd: don't allow old listforwards arg order.
Changelog-Removed: Old order of the `status` parameter in the `listforwards` rpc command (deprecated in v0.10.2)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell 43b037ab0b lightningd: always require "jsonrpc": "2.0" in request.
Changelog-Removed: JSONRPC: RPC framework now requires the `"jsonrpc"` property inside the request (deprecated in v0.10.2)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell bfe342c64b lightningd: remove double-wrapped rpc_command hook.
Somehow we missed this deprecation, found by grep.

Changelog-Removed: JSON API: Removed double wrapping of `rpc_command` payload in `rpc_command` JSON field (deprecated v0.8.2)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell 6c33f7db65 common: remove unused parameter "allow_deprecated" from parse_wireaddr_internal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Michael Schmoock c6858748bb cleanup: fix mixed indentation of json_getinfo
This one got badly messed up over time. I know we usually don't fix
these to have easier git-bisect. I can remove this commit if required.
2022-09-11 18:09:13 +02:00
Michael Schmoock ec95c7c18c peer_control: fix getinfo showing unannounced addr
Currently discovered IPs are only announced when we don't have any
usable addresses detected or configured already. However, the cli
command `getinfo` still showed theses unannounced addr as if they
were announced.

Changelog-Fixed: peer_control: getinfo showing unannounced addresses.
2022-09-11 18:09:13 +02:00
Rusty Russell 44d9e8d9c5 Remove names of parameters of callbacks which confuse gcc.
We annotate them with UNNEEDED, which is legal but weird, but it
makes gcc (at least 11.2.0) complain about shadowing:

	https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106424

I simply removed the names.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-31 12:18:28 +03:00
Rusty Russell 6fe570820e Remove general shadowed variables.
We shadow local variables in several places: generally, these changes
make the code clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-31 12:18:28 +03:00
adi2011 23b675922f lightningd/opening_control.c: Skip over channels which are already stored, and don't create new peer if it already exits. Changelog-None 2022-08-19 10:27:09 -04:00
Christian Decker 65549a2931 ld: Fix a log message assuming that the `channel->scid` was set
This is no longer true after we introduce `zeroconf`, so use the alias
local alias instead if not set.

Signed-off-by: Christian Decker <@cdecker>
2022-08-16 14:52:52 -05:00
Rusty Russell 054339e0cb lightningd: obey first hop channel id.
Reported-by: Warren Togami
Changelog-Changed: `sendpay` and `sendonion` now obey the first hop "channel" short_channel_id, if specified.
Changelog-Deprecated: `sendpay` and `sendonion` could take a bogus first hop "channel" short_channel_id
2022-08-09 16:33:24 -05:00
Rusty Russell 8a9ce55345 lightningd: don't route private channels via real scid.
Again, we should use the real channel_type, but we approximate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: private channels will only route using short-channel-ids if channel opened with option_scid_alias-supporting peer.
2022-08-09 16:31:36 -05:00
Rusty Russell cfe6b06fb5 lightnind: use aliases in routehints for private channels.
We *should* remember the channel type, since this is only required
if they set the channel_type to include option_scid_alias.

However, since we support channel upgrade, channel_type really needs
a new table.  I have a patch for that, from my abandoned original
"fastopen" branch for aliases, but it's too big a chance for rc2 IMHO.

Meanwhile, we allow exposeprivatechannels's scids to be either real or
the aliases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: invoice routehints will use fake short-channel-ids for private channels if channel opened with option_scid_alias-supporting peer.
2022-08-09 16:31:36 -05:00
Rusty Russell 80a6d9b58e lightningd: set the channel_type feature.
AFAICT we should have been doing this since we started sending and
receiving it, but didn't.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: we now advertize the `option_channel_type` feature (which we actually supported since v0.10.2)
2022-08-08 11:49:56 -05:00
Rusty Russell 645b1b505b lightningd: fix funding_locked in channel_opened notification.
Previously, "funding_locked" was always "true"!

(It's actually been wrong since its introduction in v0.7.3!)

Changelog-Fixed: Plugins: `channel_opened` notification `funding_locked` field is now accurate: was always `true`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-04 13:08:25 +09:30
niftynei 4e503f7d0a bkpr/listpeeers: add lease_fees back to funds; separate out in listpeers
First, how we record "our_funds" and then apply pushes vs lease_fees
(for liquidity ad buys/sales) was exactly opposite.

For pushes we were reporting the total funded into the channel, with the
push representing how much we'd later moved to the peer.

For lease_fees we were rerporting the total in the channel, with the
push representing how much was already moved to the peer.

We fix this (from a view perspective) by re-adding lease fees to what's
reported in the channel funding totals. Since this is now new behavior
(for leased channel values), we added new fields so we can take the old
field names thru a deprecation cycle.

We also make it possible to differentiate btw a push and a lease_fee
(before they were all the same), by adding to new fields to `listpeers`:
`fee_paid_msat` and `fee_rcvd_msat`.

This allows us to avoid math in the bookkeeper, instead we just pick
the numbers out directly and record them.

Fixes #5472

Changelog-Added: JSON-RPC: `listpeers` now has a few new fields for `funding` (`remote_funds_msat`, `local_funds_msat`, `fee_paid_msat`, `fee_rcvd_msat`).
Changelog-Deprecated: JSON-RPC: `listpeers`.`funded` fields `local_msat` and `remote_msat` are now deprecated.
2022-07-31 21:53:05 +09:30
niftynei a675f4c24e balance_snapshot: emit balances for channels that are awaiting_lockin
9cad7d6a6a changed the behavior of `channel_active`, which slightly
broke how our balance snapshots work (we need info about channels that
aren't on-chain yet).

This patches adds AWAITING_UNILATERAL back in.
2022-07-31 21:53:05 +09:30
Rusty Russell 22ff007d64 connectd: control connect backoff from lightningd.
We used to tell connectd to remember our connect delay, and hand it
back (increased if necessary).

Instead, simply record when we last tried to connect.  If it was less
than 10 minutes ago, double delay (up to 5 minutes max), otherwise
reset delay to 1 second.

This covers all scenarios: whether we reconnect then immediately
disconnect, or never successfully connect, it doesn't matter.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5453
2022-07-28 15:08:44 +09:30
Rusty Russell 9cad7d6a6a lightningd: don't consider AWAITING_UNILATERAL to be "active".
It's not active: we don't want to connect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 15:08:44 +09:30
niftynei 30aa1d79fb bkpr: for zerconfs, we still wanna know you're opening a channel
We need a record of the channel account before you start sending
payments through it. Normally we don't start allowing payments to be
sent until after the channel has locked in but zeroconf does away with
this assumption.

Instead we push out a "channel_proposed" event, which should only show
up for zeroconfs.
2022-07-28 12:08:18 +09:30
niftynei 0617690981 coin_mvt/bkpr: add "stealable" tag to stealable outputs
If we expect further events for an onchain output (because we can steal
it away from the 'external'/rightful owner), we mark them.

This prevents us from marking a channel as 'onchain-resolved' before
all events that we're interested in have actually hit the chain.

Case that this matters:
Peer publishes a (cheating) unilateral close and a timeout htlc (which
we can steal).
We then steal the timeout htlc.

W/o the stealable flag, we'd have marked the channel as resolved when
the peer published the timeout htlc, which is incorrect as we're still
waiting for the resolution of that timeout htlc (b/c we *can* steal it).
2022-07-28 12:08:18 +09:30
niftynei a45da63280 bkpr: pass the node_id over for channel_opens, add to account
it's nice to know what node your channel was opened with. in theory we
could use listpeers to merge the data after the fact, except that
channels disappear after they've been closed for a bit. it's better to
just save the info.

we print it out in `listbalances`, as that's a great place account level
information
2022-07-28 12:08:18 +09:30
niftynei d87bdeeace coin_mvt: log channel_open for channels that close before they're locked
We don't push out a coin_move for a channel open until it's locked in,
but this causes problems for channels that close before they're locked.

So if we go the "close before locked in" route, we push out a channel
open event.

These will get a blockheight of 0, if we haven't seen the
funding transaction in a block yet.
2022-07-28 12:08:18 +09:30
niftynei 282ab72e2d tests: valgrind barfing on uninitialized value
------------------------------- Valgrind errors --------------------------------
Valgrind error file: valgrind-errors.493330
==493330== Conditional jump or move depends on uninitialised value(s)
==493330==    at 0x154051: opt_add_addr_withtype (options.c:275)
==493330==    by 0x154406: opt_add_announce_addr (options.c:302)
==493330==    by 0x2696E6: parse_one (parse.c:121)
==493330==    by 0x25CFB5: opt_parse (opt.c:228)
==493330==    by 0x155DB6: handle_opts (options.c:1413)
==493330==    by 0x127317: main (lightningd.c:994)
==493330==
{
   <insert_a_suppression_name_here>
   Memcheck:Cond
   fun:opt_add_addr_withtype
   fun:opt_add_announce_addr
   fun:parse_one
   fun:opt_parse
   fun:handle_opts
   fun:main
}
--------------------------------------------------------------------------------
Leaving base_dir /tmp/ltests-iyf2dw3n intact, it still has test sub-directories with failure details: ['test_announce_dns_without_port_1']
====================================== short test summary info ======================================
ERROR tests/test_gossip.py::test_announce_dns_without_port - ValueError:
2022-07-26 15:11:30 -07:00
Rusty Russell 008a59b004 lightningd: ignore default if it's a literal 'null' JSON token.
I wondered how `tests/plugins/dblog.py` worked, since it is supposed to fail
unless the `dblog-file` arg is set:

```
@plugin.init()
def init(configuration, options, plugin):
    if not plugin.get_option('dblog-file'):
        raise RpcError("No dblog-file specified")
```

But it was set to "null".  That's because 'None' in python is turned into a literal
JSON "null", and we take that as the default value.

We also cleanup the popt->description double-assignment (a leftover
from when this was optional).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: plugins: setting the default value of a parameter to `null` is the same as not setting it (pyln plugins did this!).
2022-07-26 09:33:40 -07:00
Rusty Russell 0fd8a6492e lightningd: fix fatal() log message in log.
The one to stderr is fine, the log one gets corrupted, like so:

```
2022-07-24T07:20:08.6250702Z lightningd-2 2022-07-24T06:49:19.494Z **BROKEN** lightningd: Plugin '????UH??SH??8H?}?H?u?H?U?H?M?H?M?H?E?H?????' returned an invalid response to the db_write hook: (F???U
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-25 08:11:09 -07:00
niftynei 7bbfef5054 tests: flake fix; l1 was waiting too long to reconnect
We were waiting too long for the reconnect to happen (60s default),
which caused this test to timeout.

When testing, let's speed up the reconnect.

L2 tried to reconnect but didn't have connection information in its
gossip -- is there a way to ask/save connection data from a node you're
making a channel with that doesn't rely on their node_announcement?
2022-07-25 16:28:09 +09:30
Rusty Russell 5979a7778f lightningd: expand exit codes for various failures.
Most unexpected ones are still 1, but there are a few recognizable error codes
worth documenting.

Rename the HSM ones to put ERRCODE_ at the front, since we have non-HSM ones
too now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-20 19:28:33 +09:30
Rusty Russell e96eb07ef4 lightningd: test that hsm_secret is as expected, at startup.
If you get the wrong hsm_secret, your node_id will change, and
peers won't know who you are, bitcoind will reject your transaction
signatures, and other madness.

Catch this as soon as it happens, by storing our node_id in the db.

Suggested-by: @cdecker, @fiatjaf
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Config: `lightningd` will refuse to start with the wrong node_id (i.e. hsm_secret changes).
2022-07-20 19:28:33 +09:30
Vincenzo Palazzo 7ae616ef60 rpc: improve error format
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-07-19 17:55:31 +02:00
Vincenzo Palazzo 1d671a2380 rpc: checkmessage return an error if pubkey is not found
Returning an warning message when the pub key is not specified and there is no node in the graph.

We try to help people that use core lightning as a signer and nothings else.

Changelog-Deprecated: rpc: checkmessage return an error when the pubkey is not specified and it is unknown in the network graph.
2022-07-19 17:55:31 +02:00
Rusty Russell a08728497b lightningd: reintroduce "slow connect" logic.
Just keep a flag on the peer, and delay connection longer if that is set.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 02e169fd27 lightningd: drive all reconnections out of disconnections.
The only places which should call try_reconnect now are the "connect"
command, and the disconnect path when it decides there's still an
active channel.

This introduces one subtlety: if we disconnect when there's no active
channel, but then the subd makes one, we have to catch that case!

This temporarily reverts "slow" reconnections to fast ones: see next
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell a3c4908f4a lightningd: don't explicitly tell connectd to disconnect, have it do it on sending error/warning.
Connectd already does this when we *receive* an error or warning, but
now do it on send.  This causes some slight behavior change: we don't
disconnect when we close a channel, for example (our behaviour here
has been inconsistent across versions, depending on the code).

When connectd is told to disconnect, it now does so immediately, and
doesn't wait for subds to drain etc.  That simplifies the manual
disconnect case, which now cleans up as it would from any other
disconnection when connectd says it's disconnected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell e59e12dcb6 lightningd: don't forget peer if it's still connected.
In particular, when onchaind finishes with a channel, and we delete it, we
would forget about the peer, even if it's still connected.  That leads to a
surprise if we are activated because of something it sends:

```
2022-07-16T09:07:51.8668176Z lightningd-1 2022-07-16T08:54:32.497Z INFO    022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Peer transient failure in AWAITING_UNILATERAL: Disconnected
2022-07-16T09:07:51.8668717Z lightningd-1 2022-07-16T08:54:32.497Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-lightningd: Will try reconnect in 1 seconds
2022-07-16T09:07:51.8669323Z lightningd-1 2022-07-16T08:54:32.497Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Peer has reconnected, state AWAITING_UNILATERAL: connecting subd
2022-07-16T09:07:51.8671225Z lightningd-1 2022-07-16T08:54:32.497Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Telling connectd to send error 001185a48d443eae6fbcc679accd4d497c4183b711f2cd204c0b50acd3cd76fda08d00936368616e6e656c643a207265636569766564204552524f52206572726f72206368616e6e656c20383561343864343433656165366662636336373961636364346434393763343138336237313166326364323034633062353061636433636437366664613038643a20466f726369626c7920636c6f7365642062792060636c6f73656020636f6d6d616e642074696d656f7574
2022-07-16T09:07:51.8671786Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_in WIRE_GOSSIP_TIMESTAMP_FILTER
2022-07-16T09:07:51.8672270Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-hsmd: Got WIRE_HSMD_ECDH_REQ
2022-07-16T09:07:51.8673027Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-onchaind-chan#1: billboard: All outputs resolved: waiting 0 more blocks before forgetting channel
2022-07-16T09:07:51.8673419Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds
2022-07-16T09:07:51.8673954Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_out WIRE_GOSSIP_TIMESTAMP_FILTER
2022-07-16T09:07:51.8674298Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   hsmd: Client: Received message 1 from client
2022-07-16T09:07:51.8674811Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: seeker: chosen as startup peer
2022-07-16T09:07:51.8675330Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Activating for message WIRE_ERROR
2022-07-16T09:07:51.8675825Z lightningd-2 2022-07-16T08:54:32.497Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-hsmd: Got WIRE_HSMD_ECDH_REQ
...
2022-07-16T09:07:51.9503144Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: FATAL SIGNAL 11 (version 6e6b41d)
2022-07-16T09:07:51.9503563Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x5620882dbffb
2022-07-16T09:07:51.9503970Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0x5620882dc04d
2022-07-16T09:07:51.9504534Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0x7ffb3abdf08f
2022-07-16T09:07:51.9504973Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: lightningd/peer_control.c:1378 (peer_spoke) 0x5620882aedd7
2022-07-16T09:07:51.9505418Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: lightningd/connect_control.c:492 (connectd_msg) 0x56208828e9db
2022-07-16T09:07:51.9505835Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: lightningd/subd.c:557 (sd_msg_read) 0x5620882bd89a
2022-07-16T09:07:51.9506236Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x5620883318d4
2022-07-16T09:07:51.9506618Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x562088331da1
2022-07-16T09:07:51.9507021Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x562088331e3e
2022-07-16T09:07:51.9507428Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x5620883337d3
2022-07-16T09:07:51.9507945Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x5620882969f5
2022-07-16T09:07:51.9508368Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1190 (main) 0x56208829a7bb
2022-07-16T09:07:51.9508804Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0x7ffb3abc0082
2022-07-16T09:07:51.9509172Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x56208827d32d
2022-07-16T09:07:51.9509552Z lightningd-2 2022-07-16T08:54:32.737Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell e15e55190b lightningd: provide peer address for reconnect if connect fails.
It usually works out due to other reconnections, but I noticed this
diagnosing another test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell c415c80d48 connectd: spelling and typo fixes.
From @niftynei.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 671e66490e lightningd: don't kill subds immediately on disconnect.
Give them time to process any final messages!  If there's a reconnect,
then we need to clean them up immediately of course.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell d31420211a connectd: add counters to each peer connection.
This allows us to detect when lightningd hasn't seen our latest
disconnect/reconnect; in particular, we would hit the following pattern:

1. lightningd says to connect a subd.
2. connectd disconnects and reconnects.
3. connectd reads message, connects subd.
4. lightningd reads disconnect and reconnect, sends msg to connect to subd again.
5. connectd asserts because subd is alreacy connected.

This way connectd can tell if lightningd is talking about the previous
connection, and ignoere it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 41b379ed89 lightningd: hand fds to connectd, not receive them from connectd.
Before this patch:
1. connectd says it's connected (peer_connected)
2. we tell connectd we want to talk about each channel (peer_make_active)
3. connectd gives us an fd for each channel, and we connect it to a subd (peer_active)
4. OR, connectd says it sent something about a channel we didn't tell it about, with an fd (peer_active)

Now:
1. connectd says it's connected (peer_connected)
2. we start all appropriate subds and tell connectd to what channels/fds (peer_connect_subd).
3. if connectd says it sent something about a channel we didn't tell it about, we either tell
   it to hang up (peer_final_msg), or connect a new opening daemon (peer_connect_subd).

This is the minimal-size patch, which is why we create socket pairs in
so many places to use the existing functions.  Many cleanups are
possible, since the new flow is so simple.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell eff53495db lightningd: make "is peer connected" a tristate.
First, connectd tells us the peer has connected, and we call the connected hook,
and if it says it's fine, we are actually connected and we fire off notifications.

Of course, we could be disconnected while in the connected hook, and that would
mean we tell people about a connection which is no longer current.

Make this clear with a tristate: if we're not marked disconnected by
the time the hooks finish, we're good.  It also gives us a cleaner
"connect" command return when we connected but disconnected before
processing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 571f0fad1b lightningd: remove delay on succeeding connect.
We used to not return from "connect" until we had connected all the subds,
which introduced more races if something went wrong.

Remove this workaround, since we're going to rework this logic entirely.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 40145e619b connectd: remove the redundant "already connected" logic.
It should now be reliable, so we don't need this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 9b6c97437e connectd: remove reconnection logic.
We don't have to put aside a peer which is reconnecting and wait for
lightningd to remove the old peer, we can now simply free the old
and add the new.

Fixes: #5240
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell d58e6fa20b lightningd: don't tell connectd to disconnect peer if it told us.
We allow connectd to tell us a peer has gone away, but now we need
to make sure we don't double-spiderman and tell it to disconnect peer.

This is particularly harmful on reconnect: it (will soon) tell us the
old connection is gone, ready to tell us the new peer has connected.
We would tell it to disconnect the peer, which throws away the new
connection!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell c64ce4bbf3 lightningd: clean up channels when connectd says peer is gone.
This is redundant now, since connectd only sends us this once we tell
it it's OK, but that's changing, so clean up now.  This means that
connectd will be able to make *unsolicited* closes, if it needs to.

We share logic with peer_please_disconnect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 912ac25270 lightningd: remove 'connected' flag from channel structure.
It's directly a product of "does it have a current owner subdaemon"
and "does that subdaemon talk to peers", so create a helper function
which just evaluates that instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell b8ed107743 lightningd: fix dev-memleak crash on unown unconfirmed channels.
```
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: FATAL SIGNAL 11 (version e0507aa)
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x56319736e437
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0x56319736e48a
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7fc37721151f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/subd.c:839 (subd_send_msg) 0x5631973425f3
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/subd.c:859 (subd_req_) 0x5631973426e4
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/peer_control.c:2967 (peer_dev_memleak) 0x56319732ca93
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/memdump.c:296 (json_memleak) 0x56319730fc97
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:630 (command_exec) 0x563197306f9d
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:765 (rpc_command_hook_final) 0x5631973075d5
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:278 (plugin_hook_call_) 0x56319733db47
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:853 (plugin_hook_call_rpc_command) 0x5631973079d4
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:957 (parse_request) 0x563197307efb
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1054 (read_json) 0x563197308361
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x5631973de46c
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x5631973df0a1
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x5631973df0e3
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x5631973e147f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x563197305041
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1184 (main) 0x56319730b58d
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7fc3771f8d8f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:392 (__libc_start_main_impl) 0x7fc3771f8e3f
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x5631972dec54
lightningd-1 2022-07-14T08:18:38.972Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
lightningd-3 2022-07-14T08:18:38.976Z DEBUG   connectd: drain_peer
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 08e3e979c8 lightningd: set cid correctly in peer->uncommitted_channel.
Setting it to 0xfffff... is just confusing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Rusty Russell 73762de18c lightningd: reduce log level for remote address reporting.
It's available in listpeers() if you want to see it, otherwise it's not
really something users want to see in the normal course of operation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 05:32:24 +02:00
Michael Schmoock 65433de05f options: set DNS port to network default if not specified
- set port for a DNS announcement without port to network default
- remove x-fail

Changelog-Fixed: Port of a DNS announcement can be 0 if unspecified
2022-07-17 23:26:07 +02:00
Rusty Russell d0a55a62b3 common/json_stream: make json_add_jsonstr take a length.
This is useful when have have a jsmntok_t.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell dbae5ae569 common/json_stream.c: provide explicit json_add_primitive_fmt and json_add_str_fmt routines.
Rather than a generic "add member", provide two routines: one which
doesn't quote, and one which does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 36a29fbfbc lightningd/json.h: remove.
There are hardly any lightningd-specific JSON functions: all that's left
are the feerate ones, and there's already a comment that we should have
a lightningd/feerate.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 401f1debc5 common: clean up json routine locations.
We have them split over common/param.c, common/json.c,
common/json_helpers.c, common/json_tok.c and common/json_stream.c.

Change that to:
* common/json_parse (all the json_to_xxx routines)
* common/json_parse_simple (simplest the json parsing routines, for cli too)
* common/json_stream (all the json_add_xxx routines)
* common/json_param (all the param and param_xxx routines)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell e621b8b24e lightningd: remove gratuitous param_tok from help and config.
They predate json_string!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell ec76ba3895 lightningd/connect_control: remove param_tok from connect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 814cde5623 lightningd/closing_control: remove param_tok().
It's bad form; if we do parsing inside params() then the `check`
command is much more effective.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 0236d4e4da common/json_stream: remove useless attempt at oom handling.
We tell membuf to use tal, and tal never returns NULL, so this
code can never be triggered.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell 9685c1adaf lightningd: remove getsharedsecret.
This was introduced to allow creating a shared secret, but it's better to use
makesecret which creates unique secrets.  getsharedsecret being a generic ECDH
function allows the caller to initiate conversations as if it was us; this
is generally OK, since we don't allow untrusted API access, but the commando
plugin had to blacklist this for read-only runes explicitly.

Since @ZmnSCPxj never ended up using this after introducing it, simply
remove it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSONRPC: `getsharedsecret` API: use `makesecret`
2022-07-15 22:17:58 +09:30
Rusty Russell c34a0a22ad makesecret: change info_hex arg to simply "hex" to match datastore command.
And fix schema: it wasn't tested as there was no test-by-parameter-name.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 22:17:58 +09:30
Vincenzo Palazzo b624c53051 plugin: add check on the type json object during the IO message handling 2022-07-14 12:49:23 -05:00
niftynei d284b98911 notify: `channel_state_changed` now receives notice when channel opens
Previously we wouldn't notify when a channel moves into state
"CHANNELD_AWAITING_LOCKIN", as this is the original state (so there's
no movement btw states). This meant that it's impossible to track when a
channel's commitment txs have been exchanged and we're waiting for
onchain confirmation.

It's useful to have notice of this initialization though, all in one
place so that the `channel_state_changed` notification can successfully
track the entire lifecycle of a channel, from inception to close.

Note that for v2 "dual-funded" channels, we already notify at the same place, at
"DUALOPEND_AWAITING_LOCKIN" (the initial state for a dualopend channel
is "DUALOPEND_OPEN_INIT" -- this is the only state we don't get notified
at now...)

Changelog-Added: Plugins: `channel_state_changed` now triggers for a v1 channel's initial "CHANNELD_AWAITING_LOCKIN" state transition (from prior state "unknown")
2022-07-14 12:42:48 -05:00
Rusty Russell bdefbabbef lightningd: re-transmit all closing transactions on startup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-14 12:40:57 -05:00
adi2011 6ba8abb0de lightningd: Add RPC for populating DB with stub channels and set an error on reconnecting 2022-07-14 12:24:48 -05:00
adi2011 1a1be6abd6 lightningd/peer_control: Add RPC for fetching scb for all the in-memory channels. 2022-07-14 12:24:48 -05:00
adi2011 eca844eb36 channel: Add struct scb_chan in channel and making last tx optional. 2022-07-14 12:24:48 -05:00
adi2011 e42ba8366b common: Add scb_wire for serializing the static_chan_backup 2022-07-14 12:24:48 -05:00
adi2011 64c03f8990 hsmd: Create derive_secret and makesecret RPC for deriving pseudorandom keys from HSM 2022-07-14 12:24:48 -05:00
Rusty Russell 1217f479df sendpay: allow route to contain both amount_msat and (deprecated) msatoshi.
Since it's a deprecation, we simply ignore one, rather than properly
checking they match etc.

Fixes: #5386
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-12 12:42:05 -05:00
Rusty Russell 312751075c lightningd: save outgoing information for more forwards.
There's one case where we can present more infomation, so do that, and
fix documentation.

Changelog-Added: JSON-RPC: `listforwards` now shows `out_channel` in more cases: even if it couldn't actually send to it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5329
2022-07-12 06:38:11 +09:30
Rusty Russell 3a1a7eb93f wallet: allow saving forwarding scid even if we don't have amount.
They're not logically connected: we can know where they wanted to
go, but we didn't send it.

Where possible, it's the scid *they asked for*; otherwise, it's the
scid or fallback to the alias, but do this in the *caller*, not by
overriding inside wallet_forwarded_payment_add.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-12 06:38:11 +09:30
Rusty Russell d6afb0cd8d lightningd: allow outgoing_scid without outgoing amount.
This (will) happen if they ask is to forward to an unknown scid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-12 06:38:11 +09:30
Simon Vrouwe 71cd07ea61 json: add "dynamic" field to plugin list
CHANGELOG: add "dynamic" field to plugin list
2022-07-10 21:09:41 -05:00
Simon Vrouwe cdf12d06ba lightningd: Make sure plugins don't register the same option "name"
The extra entry in opt_table would never be called, leaving plugins
clueless why options keep defaulting.

Note that option registration outside startup does nothing.
Instead, dynamic plugins can use `plugin start [second_parameter]` to pass options.
2022-07-10 21:09:41 -05:00
Simon Vrouwe 2fddfe3ffc lightningd: don't increment plugin state to NEEDS_INIT when error in getmanifest
Otherwise we hangs forever in startup when it was the last plugin, we would
miss destroy_plugin --> check_plugins_manifests --> io_break

e.g. when a plugin tries to register a bool option with a string as default value.
2022-07-10 21:09:41 -05:00
Simon Vrouwe 981fd2326a lightningd: convert plugin->cmd to absolute path, fail plugin early when non-existent
Otherwise different relative paths (e.g. /dir/plugin and /dir/../dir/plugin) to same plugin
executable would not be recognized as the same plugin
2022-07-10 21:09:41 -05:00
Rusty Russell 6753675c31 lightningd: fire watches on blocks before telling lightningd.
We noticed bogus behavior where (with 200 blocks added at once)
lightningd didn't see a DF channel open: this is because we told
lightningd about the new blocks (and it timed out channel) before
the watches for the transaction are fired.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-10 20:31:30 -05:00
Rusty Russell 36e5156832 lightningd: fix close of ancient all-dust channel.
libwally asserts() when trying to calculate the txid: we report it as
invalid elsewhere, but then crash:

```
May 19 20:37:03 c-lightning lightningd[26236]: 2022-05-19T20:37:03.689Z DEBUG   lightningd: close_command: timeout = 1
May 19 20:37:04 c-lightning lightningd[26236]: 2022-05-19T20:37:04.689Z UNUSUAL 037f32400c108f8385db94371bfcef08948bc0042069c1ef6f39086cdf23420ee5-chan#91: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by `close` command timeout
May 19 20:37:04 c-lightning lightningd[26236]: 2022-05-19T20:37:04.689Z **BROKEN** 037f32400c108f8385db94371bfcef08948bc0042069c1ef6f39086cdf23420ee5-chan#91: Cannot broadcast our commitment tx: it's invalid! (ancient channel?)
May 19 20:37:04 c-lightning lightningd[26236]: lightningd: bitcoin/tx.c:485: wally_txid: Assertion `len == written' failed.
May 19 20:37:04 c-lightning lightningd[26236]: lightningd: FATAL SIGNAL 6 (version v0.11.1)
May 19 20:37:04 c-lightning lightningd[26236]: 0x45bd01 send_backtrace
May 19 20:37:04 c-lightning lightningd[26236]:         common/daemon.c:33
May 19 20:37:04 c-lightning lightningd[26236]: 0x45bd8b crashdump
May 19 20:37:04 c-lightning lightningd[26236]:         common/daemon.c:46
May 19 20:37:04 c-lightning lightningd[26236]: 0x7f9f877604bf ???
May 19 20:37:04 c-lightning lightningd[26236]:         ???:0
May 19 20:37:04 c-lightning lightningd[26236]: 0x7f9f87760438 ???
May 19 20:37:04 c-lightning lightningd[26236]:         ???:0
May 19 20:37:04 c-lightning lightningd[26236]: 0x7f9f87762039 ???
May 19 20:37:04 c-lightning lightningd[26236]:         ???:0
May 19 20:37:04 c-lightning lightningd[26236]: 0x7f9f87758be6 ???
May 19 20:37:04 c-lightning lightningd[26236]:         ???:0
May 19 20:37:04 c-lightning lightningd[26236]: 0x7f9f87758c91 ???
May 19 20:37:04 c-lightning lightningd[26236]:         ???:0
May 19 20:37:04 c-lightning lightningd[26236]: 0x4751b7 wally_txid
May 19 20:37:04 c-lightning lightningd[26236]:         bitcoin/tx.c:485
May 19 20:37:04 c-lightning lightningd[26236]: 0x4751f4 bitcoin_txid
May 19 20:37:04 c-lightning lightningd[26236]:         bitcoin/tx.c:496
May 19 20:37:04 c-lightning lightningd[26236]: 0x40bad0 resolve_one_close_command
May 19 20:37:04 c-lightning lightningd[26236]:         lightningd/closing_control.c:60
May 19 20:37:04 c-lightning lightningd[26236]: 0x40cef3 resolve_close_command
May 19 20:37:04 c-lightning lightningd[26236]:         lightningd/closing_control.c:82
May 19 20:37:04 c-lightning lightningd[26236]: 0x430355 drop_to_chain
May 19 20:37:04 c-lightning lightningd[26236]:         lightningd/peer_control.c:286
```

Fixes: #5277
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-10 20:24:04 -05:00
Rusty Russell 3a61e0e181 invoice: turn assert failure into more informative log_broken message.
We had this assertion fail, and I can't see a clear reason why.  Remove it
(i.e. don't crash because we're having trouble with creating invoice routehints)
and add logging.

```
Assertion failed: a->c->rr_number < b->c->rr_number (lightningd/invoice.c: cmp_rr_number: 623)
lightningd: FATAL SIGNAL 6 (version v0.10.2-modded)
0x5654f1c40061 send_backtrace
        common/daemon.c:33
0x5654f1c400e9 crashdump
        common/daemon.c:46
0x7efd87da6c8a ???
        ???:0
```

There are several possible causes for this:

1. We have two channels with the same rr_number. A quick audit shows we always set that rr_number to a unique value (and 64 bits, so wrap is not possible between the release and now!).
2. It's theoretically possible that sort() could compare a value with itself, but that would be really dumb: it doesn't that I've ever seen, but then, we've never seen this assert() hit, either.
3. listincoming has given us the same channel twice. I don't see how that is possible: we had a race where channels could momentarily vanish, but never be duplicated (emailed crash.log shows no duplicates!).
4. General corruption/freed memory access. If a channel we've just looked up is gone but still in the hash table, this is possible but would cause lots of random behavior and crashes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-10 19:55:33 -05:00
Rusty Russell 7a7e43c078 channeld: fix uninitializes scid alias for dual-funding.
Caused a crash in CI, reproduced under valgrind by calling
any_channel_by_scid from io_poll_lightningd:

```
==2422524== Conditional jump or move depends on uninitialised value(s)
==2422524==    at 0x12C98D: any_channel_by_scid (channel.c:606)
==2422524==    by 0x14FF75: io_poll_lightningd (lightningd.c:682)
==2422524==    by 0x225FDE: io_loop (poll.c:420)
==2422524==    by 0x14A914: io_loop_with_timers (io_loop_with_timers.c:22)
==2422524==    by 0x150C4E: main (lightningd.c:1193)
==2422524==  Uninitialised value was created by a heap allocation
==2422524==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2422524==    by 0x234F61: allocate (tal.c:250)
==2422524==    by 0x235522: tal_alloc_ (tal.c:428)
==2422524==    by 0x12B500: new_unsaved_channel (channel.c:203)
==2422524==    by 0x13B77A: json_openchannel_init (dual_open_control.c:2610)
==2422524==    by 0x14C78D: command_exec (jsonrpc.c:630)
==2422524==    by 0x14CD9F: rpc_command_hook_final (jsonrpc.c:765)
==2422524==    by 0x181DDA: plugin_hook_call_ (plugin_hook.c:278)
==2422524==    by 0x14D198: plugin_hook_call_rpc_command (jsonrpc.c:853)
==2422524==    by 0x14D6A0: parse_request (jsonrpc.c:957)
==2422524==    by 0x14DAFE: read_json (jsonrpc.c:1054)
==2422524==    by 0x2231C8: next_plan (io.c:59)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 09:59:52 +09:30
Rusty Russell 4633085ffd lightningd: mark subd->conn notleak() properly in transition.
Since subd is temporarily unreferenced, and our leak detection goes up
to the highest unreferenced parent, it complains about openingd->conn.

The correct fix is to catch this where we detach the subd:

```
DEBUG:root:{
  "id": 1,
  "result": {
    "leaks": [
      {
        "value": "0x556e0445d8f8",
        "label": "ccan/ccan/io/io.c:91:struct io_conn",
        "backtrace": [
          "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
          "ccan/ccan/io/io.c:91 (io_new_conn_)",
          "lightningd/subd.c:774 (new_subd)",
          "lightningd/subd.c:828 (new_channel_subd_)",
          "lightningd/opening_control.c:872 (peer_start_openingd)",
          "lightningd/peer_control.c:1311 (peer_active)",
          "lightningd/connect_control.c:458 (connectd_msg)",
          "lightningd/subd.c:557 (sd_msg_read)",
          "lightningd/subd.c:357 (read_fds)",
          "ccan/ccan/io/io.c:59 (next_plan)",
          "ccan/ccan/io/io.c:407 (do_plan)",
          "ccan/ccan/io/io.c:417 (io_ready)",
          "ccan/ccan/io/poll.c:453 (io_loop)",
          "lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
          "lightningd/lightningd.c:1182 (main)",
          "../csu/libc-start.c:308 (__libc_start_main)"
        ],
        "parents": [
          "lightningd/lightningd.c:107:struct lightningd"
        ]
      }
    ]
  }
}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 09:59:52 +09:30
Rusty Russell 6aec374674 lightningd: make log-prefix actually prepend all log messages as expected.
It actually only sets the prefix for the lightningd core log messages;
the other logs have their own prefix.

Make it a real, process-wide prefix which actually goes in front of the timestamp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: options: `log-prefix` now correctly prefixes *all* log messages.
2022-07-09 09:59:52 +09:30
Rusty Russell ba4e9b64b5 options: print empty options properly.
Noticed by log-prefix default (""), which causes bad JSON:

```
    Malformed JSON reply '{"jsonrpc":"2.0","id":0,"result":{...
    "log-prefix":,"log-file":"log"} }
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 09:59:52 +09:30
Rusty Russell f6f1844e15 options: let log-level subsystem filter also cover nodeid.
That's useful for "tell me everything about this node" debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5348
Changelog-Added: lightningd: `log-level=debug:<partial-nodeid>` supported to get debug-level logs for everything about a peer.
2022-07-09 09:59:52 +09:30
Rusty Russell 32af92145b update-mocks: handle missing deprecated_apis.
This expands update-mocks to be able to handle (simple!) missing
symbols which are not functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 09:59:52 +09:30
niftynei d0937a2e97 df: check mempool/block for funding output on broadcast fail
If we can't broadcast the tx, confirm that it didn't end up in the
mempool or the utxo set before throwing an error.

Note that this doesn't protect us in the case where the funding
output has already been *spent*... but that's extremely rare, right?

Fixes #5296

Reported-By: @rustyrussell
Collab-With: @vincenzopalazzo
2022-07-08 19:21:11 +09:30
Dustin Dettmer 9296537edb peer_control: Fix check_funding_details assert
Check funding_outnum validity first to avoid reading invalid outputs

Changelog-Fixed: Fixed a potential issue if the number of outputs decreases in a dualopen RBF or splice.
2022-07-07 22:03:29 -05:00
Christian Decker 669bca4a02 ld: Use the local alias in the `htlc_accepted` hook
If we have no real short-channel-id this is the best we can do. Use
the local one since we can be sure we have assigned one.
2022-07-04 22:14:06 +02:00
Christian Decker 29157735fb channeld: Track the funding depth while awaiting lockin
We used to agree up on the `minimum_depth` with the peer, thus when
they told us that the funding locked we'd be sure we either have a
scid or we'd trigger the state transition when we do. However if we
had a scid, and we got a funding_locked we'd trust them not to have
sent it early. Now we explicitly track the depth in the channel while
waiting for the funding to confirm.

Changelog-Fixed: channeld: Enforce our own `minimum_depth` beyond just confirming
2022-07-04 22:14:06 +02:00
Christian Decker 692a001198 ld: Use the local alias when reporting failures with zeroconf
Ran into this with a zeroconf channel, without confs, that was
disconnected.
2022-07-04 22:14:06 +02:00
Christian Decker 252ccfa7ab db: Store the local alias for forwarded incoming payments
Not only can the outgoing edge be a zeroconf channel, it can also be
the incoming channel. So we revert to the usual trick of using the
local alias if the short_channel_id isn't known yet.

We use the LOCAL alias instead of the REMOTE alias even though the
sender likely used the REMOTE alias to refer to the channel. This is
because we control the LOCAL alias, and we keep it stable during the
lifetime of the channel, whereas the REMOTE one could change or not be
there yet.
2022-07-04 22:14:06 +02:00
Christian Decker 92b891bee3 ld: Add function to retrieve either the scid or the local alias
We use this in a couple of places, when we want to refer to a channel
by its `short_channel_id`, I'm moving this into a separate function
primarily to have a way to mark places where we do that.
2022-07-04 22:14:06 +02:00
Christian Decker 0ce68b26c6 jsonrpc: Include the direction also if we have an alias
The direction only depends on the ordering between node_ids, not the
short_channel_id, so we can include it and it won't change. This was
causing some trouble loading the `channel_hints` in the `pay` plugin.
2022-07-04 22:14:06 +02:00
Christian Decker 22b6e33030 zeroconf: Trigger coin_movement on first real confirmation
We don't trigger on depth=0 since that'd give us bogus blockheights
and pointers into the chain, instead we defer until we get a first
confirmation. This extracts some of the logic from `lockin_complete`,
into the depth change listener (not the remote funding locked since at
that point we're certainly locked in and we don't really care about
that for bookkeeping anyway).
2022-07-04 22:14:06 +02:00
Christian Decker a4e6b58fa4 ld: Consider local aliases when forwarding 2022-07-04 22:14:06 +02:00
Christian Decker 1ae3dba529 invoice: Consider aliases too when selecting routehints 2022-07-04 22:14:06 +02:00
Christian Decker 78c9c6a9e0 ld: Allow lockin despite not having a scid yet
This is needed for us to transition to CHANNELD_NORMAL for zeroconf
channels, i.e., channels where we don't have a short channel ID yet.

We'll have to call lockin_complete a second time, once we learn the
real scid.
2022-07-04 22:14:06 +02:00
Christian Decker cdedd433a4 jsonrpc: Add aliases to `listpeers` result 2022-07-04 22:14:06 +02:00
Christian Decker 5e74048508 gossip: Add both channel directions with their respective alias
We locally generate an update with our local alias, and get one from
the peer with the remote alias, so we need to add them both. We do so
only if using the alias in the first place though.
2022-07-04 22:14:06 +02:00
Christian Decker bf44178047 gossipd: Use the remote alias if no real scid is known
This is for the local channel announcement that'll not leave this
host, as it doesn't have signatures.
2022-07-04 22:14:06 +02:00
Christian Decker c98f011479 channeld: Send a depth=0 notification when channeld starts up
This is used in order to ensure zeroconf doesn't just wait for the
first confirmation despite mindepth being set to 0.
2022-07-04 22:14:06 +02:00
Christian Decker b9817d395f zeroconf: Wire the aliases through `channeld` 2022-07-04 22:14:06 +02:00
Christian Decker de1c0b51f0 zeroconf: Add alias_remote and alias_local to channel and DB
`alias_local` is generated locally and sent to the peer so it knows
what we're calling the channel, while `alias_remote` is received by
the peer so we know what to include in routehints when generating
invoices.
2022-07-04 22:14:06 +02:00
Christian Decker 3fbaac3fdb jsonrpc: Add option_zeroconf handling to `listpeers` 2022-07-04 22:14:06 +02:00
Christian Decker 1477873190 plugin: Allow plugins to customize the mindepth in accept_channel
This is the counterpart of the `mindepth` parameter in `fundchannel`
and friends. Allows dynamic lookups of `node_id` and selectively
opting into `option_zeroconf` being used.

Changelog-Added: plugin: The `openchannel` hook may return a `mindepth` indicating how many confirmations are required.
2022-07-04 22:14:06 +02:00
Christian Decker e4511452ac bolt: Reflect the zeroconf featurebits in code 2022-07-04 22:14:06 +02:00
Christian Decker 185cd81be4 jsonrpc: Add `mindepth` argument to fundchannel and multifundchannel
This will eventually enable us to specify 0 for zeroconf channels.

Changelog-Added: JSON-RPC: Added `mindepth` argument to specify the number of confirmations we require for `fundchannel` and `multifundchannel`
2022-07-04 22:14:06 +02:00
Rusty Russell afbddcf7f3 lightningd: fix crash on rapid reconnect.
Happens occasionally when running
`tests/test_connection.py::test_mutual_reconnect_race` (which is too
flaky to add, without more fixes):


```
lightningd: lightningd/peer_control.c:1252: peer_active: Assertion `!channel->owner' failed.
lightningd: FATAL SIGNAL 6 (version v0.11.0.1-38-g4f167da)
0x5594a41f8f45 send_backtrace
	common/daemon.c:33
0x5594a41f8fef crashdump
	common/daemon.c:46
0x7f7cb585c08f ???
	/build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7f7cb585c00b __GI_raise
	../sysdeps/unix/sysv/linux/raise.c:51
0x7f7cb583b858 __GI_abort
	/build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79
0x7f7cb583b728 __assert_fail_base
	/build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92
0x7f7cb584cfd5 __GI___assert_fail
	/build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101
0x5594a41b45ca peer_active
	lightningd/peer_control.c:1252
0x5594a418794c connectd_msg
	lightningd/connect_control.c:457
0x5594a41cd457 sd_msg_read
	lightningd/subd.c:556
0x5594a41ccbe5 read_fds
	lightningd/subd.c:357
0x5594a4269fc2 next_plan
	ccan/ccan/io/io.c:59
0x5594a426abca do_plan
	ccan/ccan/io/io.c:407
0x5594a426ac0c io_ready
	ccan/ccan/io/io.c:417
0x5594a426ceff io_loop
	ccan/ccan/io/poll.c:453
0x5594a41930d9 io_loop_with_timers
	lightningd/io_loop_with_timers.c:22
0x5594a4199293 main
	lightningd/lightningd.c:1181
0x7f7cb583d082 __libc_start_main
	../csu/libc-start.c:308
0x5594a416e15d ???
	???:0
0xffffffffffffffff ???
	???:0
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-28 13:47:27 +09:30
Vincenzo Palazzo 7ff62b4a00 lightnind: remove`DEFAULT_PORT` global definition
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-06-28 06:09:01 +09:30
Vincenzo Palazzo cc7a405ca4 lightningd: use the standard port derivation in connect command
Complete implementation of BOLT1 port derivation proposal https://github.com/lightning/bolts/pull/968

Changelog-Added: rpc: use the standard port derivation in connect command when the port is not specified.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-06-28 06:09:01 +09:30
Rusty Russell e120b4afd6 lightningd: add more information should subd send wrong message.
I saw this once, but could not track it down.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell 3f98cf3fce lightningd: track weird CI crash in test_important_plugin
Looks like we woke one of the startup io_loops early, and thus
we thought we'd finished connectd_activate and we hadn't.  This
caused us to use an uninitialized ld->announceable array, and
finally caused an assert fail in the main loop.

Make *every* loop assert that it was exited for the correct reason,
so if it happens again, we can maybe figure out what part of
the code to look at.

```
lightningd: lightningd/lightningd.c:1186: main: Assertion `io_loop_ret == ld' failed.
lightningd: FATAL SIGNAL 6 (version 4df66fa)
...
------------------------------- Valgrind errors --------------------------------
Valgrind error file: valgrind-errors.895509
==895509== Conditional jump or move depends on uninitialised value(s)
==895509==    at 0x22C58E: to_tal_hdr_or_null (tal.c:184)
==895509==    by 0x22D531: tal_bytelen (tal.c:637)
==895509==    by 0x1F10B6: towire_gossipd_init (gossipd_wiregen.c:100)
==895509==    by 0x13AC6E: gossip_init (gossip_control.c:254)
==895509==    by 0x1497EC: main (lightningd.c:1090)
==895509== 
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell 70b091d9f6 lightningd: fix transient leak report when openingd shutting down.
```
>           raiseValueError(str(errors))
E           ValueError: 
E           Node errors:
E           Global errors:
E            - Node /tmp/ltests-x5sfpiwp/test_openchannel_hook_chaining_1/lightning-2/ has memory leaks: [
E               {
E                   "backtrace": [
E                       "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E                       "ccan/ccan/io/io.c:91 (io_new_conn_)",
E                       "lightningd/subd.c:773 (new_subd)",
E                       "lightningd/subd.c:827 (new_channel_subd_)",
E                       "lightningd/opening_control.c:870 (peer_start_openingd)",
E                       "lightningd/peer_control.c:1307 (peer_active)",
E                       "lightningd/connect_control.c:457 (connectd_msg)",
E                       "lightningd/subd.c:556 (sd_msg_read)",
E                       "lightningd/subd.c:357 (read_fds)",
E                       "ccan/ccan/io/io.c:59 (next_plan)",
E                       "ccan/ccan/io/io.c:407 (do_plan)",
E                       "ccan/ccan/io/io.c:417 (io_ready)",
E                       "ccan/ccan/io/poll.c:453 (io_loop)",
E                       "lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
E                       "lightningd/lightningd.c:1181 (main)",
E                       "../csu/libc-start.c:308 (__libc_start_main)"
E                   ],
E                   "label": "ccan/ccan/io/io.c:91:struct io_conn",
E                   "parents": [
E                       "lightningd/lightningd.c:107:struct lightningd"
E                   ],
E                   "value": "0x2b5a898"
E               }
E           ]
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell b4820d6706 lightningd: don't run off end of buffer if db_hook returns nonsense.
It shouldn't return nonsense, but it did, and we segfaulted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell d2952576cd pyln-testing: restore proper streaming behaviour for lightningd.
Get it to log direct to stdout, so we see what's happening *as it
happens* rather than as we read it.  We could restore the thread we
were using before, but that added more problems than it solved.

This means that we need the hsm password prompts in the log though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell 517828adb2 lightningd: don't print nasty message when onchaind fails partially-failed HTLC
1. We set an outgoing htlc's `failonion` when we get a commitment_signed.
2. We don't transfer it to the corresponding incoming HTLC until we send
   commitment_signed and receive revoke_and_ack (meaning, outgoing htlc is
   completely dead).
3. If between these steps we go onchain, onchaind (after 3 blocks) tells us
   to fail the HTLC.
4. hout->failonion is set, but hout->hin has not been failed yet.  We
   do a sanity check and print a nasty message, and fail it with
   WIRE_PERMANENT_CHANNEL_FAILURE instead of relaying the error.

So handle this case explicitly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell 56dde2cb77 lightningd: multiple log-file options allow more than one log output.
I've wanted this for a while: the ability to log to multiple places
at once.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: lightningd: `log-file` option specified multiple times opens multiple log files.
2022-06-27 17:21:35 +09:30
Vincenzo Palazzo d4bc4f6460 signmessage: improve the UX of the rpc command when zbase is not a valid one
Changelog-Fixed: signmessage: improve the UX of the rpc command when zbase is not a valid one

Stacktrace generated with a bad `zbase`

```
lightningd: lightningd/signmessage.c:59: from_zbase32: Assertion `len == tal_bytelen(u8arr)' failed
lightningd: FATAL SIGNAL 6 (version v0.11.1)
0x55b9b1b4e617 send_backtrace
[...]
```

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-06-24 14:02:04 +09:30
Rusty Russell a0c6447611 lightningd: remove overzealous assertion.
This is being hit: it's possible if connectd and lightningd get desynchronized,
and we'll handle this later when peer is activated.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-24 10:16:18 +09:30
Rusty Russell 5531de99de lease_rates: prepare for msats fields as raw numbers.
We would otherwise multiply them by 1000.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-21 06:52:35 +09:30
Rusty Russell 08d5776ebc lightningd: deprecate `msatoshi` in `sendpay` `route`.
We should be using amount_msat always.  Many tests were not.  Plus,
deprecating it simplifies the code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSONRPC: `sendpay` `route` elements `msatoshi` (use `amount_msat`)
2022-06-21 06:52:35 +09:30
Rusty Russell 01411d70be common: enforce that msat fields are called "xxx_msat".
Now we've fixed them all, make sure no new ones slip in!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-21 06:52:35 +09:30
Rusty Russell f6b4dbc65a lightningd: use amount_msat not amount in htlc_accepted_hook.
Changelog-Added: Plugins: `htlc_accepted_hook` `amount_msat` field.
Changelog-Deprecated: Plugins: `htlc_accepted_hook` `amount` field (use `amount_msat`)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-21 06:52:35 +09:30
Rusty Russell cd7e784d6f lightningd: change `msatoshi` args to `amount_msat`.
This is consistent with our output changes, and increases consistency.
It also keeps future sanity checks happy, that we only use JSON msat
helpers with '_msat' fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `invoice`, `sendonion`, `sendpay`, `pay`, `keysend`, `fetchinvoice`, `sendinvoice`: `msatoshi` argument is now called `amount_msat` to match other fields.
Changelog-Deprecated: JSON-RPC: `invoice`, `sendonion`, `sendpay`, `pay`, `keysend`, `fetchinvoice`, `sendinvoice` `msatoshi` (use `amount_msat`)
2022-06-21 06:52:35 +09:30
Rusty Russell ca69e293d1 coinmvt: don't use msats in fields not called "_msat".
The new msat fields are turned into Millisatoshi, so handle that correctly
too in tests too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `coin_movement` notification: `balance`, `credit`, `debit` and `fees` (use `balance_msat`, `credit_msat`, `debit_msat` and `fees_msat`)
2022-06-21 06:52:35 +09:30
Rusty Russell e2f0ca9cbe lightningd: don't add null for unset plugin options.
In general, we don't like to use `null` in JSON: simply omit the
field.  I found this one because it broke our 'msat' parsing (made
stricter in followup) which doesn't allow `null`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: `listconfigs` `plugins` `options` which are not set are omitted, not `null`.
2022-06-21 06:52:35 +09:30
Rusty Russell 36a2491a89 json: fix up msat amounts in non-_msat fields.
We had json_add_amount_msat_only(), which was designed to be used to
print out msat fields, if we had sats.

However, we misused it, so split it into the three different cases:
1. json_add_amount_sat_msat: We are using it correctly, with a field called
   xxx_msat.
2. json_add_amount_sats_deprecated: We were using it wrong, so deprecate
   the old field and create a new one which does end in _msat.
3. json_add_sats: we were using it to hand sats as a JSON parameter to an
   interface, where "XXXsat".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `rbf_channel` and `openchannel2` hooks `their_funding` (use `their_funding_msat`)
Changelog-Deprecated: Plugins: `openchannel2` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `funding_satoshis` (use `funding_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `channel_reserve_satoshis` (use `channel_reserve_msat`)
Changelog-Deprecated: Plugins: `channel_opened` notification `amount` (use `funding_msat`)
Changelog-Deprecated: JSON-RPC: `listtransactions` `msat` (use `amount_msat`)
Changelog-Deprecated: Plugins: `htlc_accepted` `forward_amount` (use `forward_msat`)
2022-06-21 06:52:35 +09:30
Rusty Russell c5b032598e lightningd: fix outgoing IO logging for JSONRPC.
Changelog-Fixed: lightningd: `log-level` `io` shows JSONRPC output, as well as input.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-21 06:52:35 +09:30
pn 83c31f548f log: Add termination to log level 2022-06-19 16:42:11 +09:30
pn 47a7b4a55b log: Add termination to prefix log 2022-06-19 16:42:11 +09:30
fiatjaf 1eaec223b7 expose short_channel_id and htlc id to htlc_accepted.
Changelog-Added: Plugins: `htlc_accepted` now exposes the `short_channel_id` for the channel from which that HTLC is coming from and the low-level per-channel HTLC `id`, which are necessary for bridging two different Lightning Networks when MPP is involved.
2022-06-18 14:53:32 +09:30
Michael Schmoock 475e4c9bd9 jsonrpc: adds optional `remote_addr` to listpeers
Changelog-Added: jsonrpc: adds optional `remote_addr` to listpeers
2022-06-17 20:30:16 +09:30
Michael Schmoock 32c4540fc0 jsonrpc: adds dynamicaly detected IP addresses to `getinfo`
Changelog-Fixed: JSON-RPC: Adds dynamically detected public IP addresses to `getinfo`
2022-06-17 20:30:16 +09:30
Michael Schmoock de9bc172de connect: adds nodeid to remote_addr log message 2022-06-17 20:30:16 +09:30
Brian Barto 24b02c33cc
lightning-cli plugin start - Assume default relative path
When starting a plugin, if the plugin path cannot be found in
absolute context, assume it is a relative path to the default
plugins dir. As a result, the following now works when my_plugin.py
is installed in the default plugins dir:

lightning-cli plugin start my_plugin.py

Also update the plugin documentation to reflect that the use of a
relative path is now available.

Changelog-Added: plugin start RPC subcommand now assumes relative path to default plugins dir if the path is not found in absolute context. i.e. lightning-cli plugin start my_plugin.py

[ Squashed two commits into one -- RR ]
2022-06-17 14:18:48 +09:30
Rusty Russell 7c8dc62035 channeld: take over gossip_rcvd_filter.c and is_msg_gossip_broadcast.
channeld is the only user of these functions, since it now streams
all gossip itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-17 14:14:02 +09:30
Jon Griffiths 572942c783 psbt: use DER encoded + sighash byte for PSBT_IN_PARTIAL_SIG items
Per BIP-0171, the signature map is of pubkey to "The signature as would
be pushed to the stack from a scriptSig or witness".

Fixes 5298

Changelog-Fixed: PSBT: Fix signature encoding to comply with BIP-0171.

Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2022-06-09 18:28:35 +02:00
Rusty Russell f078e54e98 lightningd: remove various deprecated JSON fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: removed `listtransactions` `outputs` `satoshis` field (deprecated v0.10.1)
Changelog-Removed: JSON-RPC: removed `listpeers` `channels` deprecated fields (deprecated v0.10.1)
Changelog-Removed: JSON-RPC: removed `listpeers` `channels` `closer` now omitted, rather than `null` (deprecated v0.10.1)
2022-05-18 10:15:36 +09:30
Rusty Russell 8b62e2584f connectd: remove enable-autotor-v2-mode option
Changelog-Removed: lightningd: removed `enable-autotor-v2-mode` option (deprecated v0.10.1)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-05-18 10:15:36 +09:30
Rusty Russell 98f64fb623 lightningd: don't crash listpeers if we're opening DF channel.
We call out to connectd to activate the peer, and while we do that,
channel->owner is NULL.  A better pattern would be to set up the unsaved
channel once connectd has given us the peer, but this works for now.

Fixes: #5204
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-21 20:45:32 +09:30
Rusty Russell 37e8d2fb0f connectd: disable advertizement of WEBSOCKET addresses.
This seems to prevent broad propagation, due to LND not allowing it.  See
	https://github.com/lightningnetwork/lnd/issues/6432

We still announce it if you disable deprecated-apis, so tests still work,
and hopefully we can enable it in future.

Fixes: #5196
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Protocol: disabled websocket announcement due to LND propagation issues
2022-04-21 06:13:55 +09:30
Rusty Russell f0dc028fa9 lightningd: fix overzealous check in htlc_out_check:
```
+11.668971802 lightningdBROKEN: backtrace: lightningd/log.c:821 (fatal_vfmt) 0x558c893c997f
+11.668978165 lightningdBROKEN: backtrace: lightningd/log.c:829 (fatal) 0x558c893c9a30
+11.668984935 lightningdBROKEN: backtrace: lightningd/htlc_end.c:87 (corrupt) 0x558c893b9b7d
+11.668991262 lightningdBROKEN: backtrace: lightningd/htlc_end.c:205 (htlc_out_check) 0x558c893ba352
+11.669016705 lightningdBROKEN: backtrace: lightningd/peer_htlcs.c:1471 (check_already_failed) 0x558c893ea9c9
+11.669023345 lightningdBROKEN: backtrace: lightningd/peer_htlcs.c:1575 (onchain_failed_our_htlc) 0x558c893eb098
+11.669043122 lightningdBROKEN: backtrace: lightningd/onchain_control.c:438 (handle_onchain_htlc_timeout) 0x558c893cd6ec
+11.669049818 lightningdBROKEN: backtrace: lightningd/onchain_control.c:548 (onchain_msg) 0x558c893cdbdc
+11.669056372 lightningdBROKEN: backtrace: lightningd/subd.c:556 (sd_msg_read) 0x558c893fa532
+11.669063030 lightningdBROKEN: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x558c8948e3cd
+11.669069093 lightningdBROKEN: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x558c8948ef9e
+11.669075470 lightningdBROKEN: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x558c8948efdc
+11.669081900 lightningdBROKEN: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x558c894912a8
+11.669087916 lightningdBROKEN: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x558c893c0966
+11.669094531 lightningdBROKEN: backtrace: lightningd/lightningd.c:1181 (main) 0x558c893c6bf9
```

Fixes: #5191
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-20 16:30:05 +09:30
Rusty Russell 151d009435 lightningd: remove over-zealous assert.
This was hit on my node.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-13 10:32:29 +09:30
Michael Dance f067e8c909 Changed external/libwally-core to test_build_fix
Combined with the following commit which is required to
update against changed libsecp256k1 APIs:

Updated deprecated function calls
2022-04-12 15:17:52 +09:30
Rusty Russell 836c1b805b doc: update c-lightning to Core Lightning almost everywhere.
Mostly comments and docs: some places are actually paths, which
I have avoided changing.  We may migrate them slowly, particularly
when they're user-visible.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-07 06:53:26 +09:30
Michael Schmoock 21fd3b22d4 config: adds htlc_minimum_msat htlc_maximum_msat announce_discovered_ip
This adds config and commandline options for htlc_min_msat, htlc_max_msat and
announce_discovered_ip. The default is 0msat for htlc_min_msat, unlimited for
htlc_max_msat and enabled for announce_discovered_ip.

The announce_discovered_ip gets the disable commandline switch --disable-ip-discovery

Changelog-added: Config options for htlc_min_msat, htlc_max_msat and announce_discovered_ip.
2022-04-05 05:45:04 +09:30
Rusty Russell e47786da04 pay/sendpay: also store description in case bolt11 uses description_hash.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-04 20:26:04 +09:30
Rusty Russell 9784fa816f lightningd: deprecate invoice expiry suffixes.
Makes types harder, and I've never personally used them.

Changelog-Deprecated: JSON-RPC: `invoice` `expiry` no longer allowed to be a string with suffix, use an integer number of seconds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:46:01 +10:30
Rusty Russell 7f89763f9e sendpay: add payment_metadata argument.
And document the missing arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell 4718ee076c lightningd: metadata received support (log and decline).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell e01abf0b34 bolt11: support payment_metadata.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell 2526e804f7 doc: big BOLT update to incorporate warnings language.
We do this (send warnings) in almost all cases anyway, so mainly this
is a textual update, but there are some changes:

1. Send ERROR not WARNING if they send a malformed commitment secret.
2. Send WARNING not ERROR if they get the shutdown_scriptpubkey wrong (vs upfront)
3. Send WARNING not ERROR if they send a bad shutdown_scriptpubkey (e.g. p2pkh in future)
4. Rename some vars 'err' to 'warn' to make it clear we send a warning.

This means test_option_upfront_shutdown_script can be made reliable, too,
and it now warns and doesn't automatically close channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell 9f06a59e3c shutdown: don't allow shutdown to p2pkh or p2sh addresses for anchor outputs.
This doesn't have an effect now (except in experimental mode), but it
will when we support anchors.  So we deprecate the use of those in the
close command too.

For experimental mode we have to avoid using p2pkh; adapt that test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `shutdown` no longer allows p2pkh or p2sh addresses.
2022-04-02 09:40:18 +10:30
Rusty Russell 3b536400f9 specs: update CSV files.
Regenerate from current BOLTS via `make extract-bolt-csv`

1. The remote_addr field was added manually into peer_wire.csv: this
   needs to be a patch otherwise it vanishes on regen.
2. We never brought into the channel_disabled fields, because it was
   too much hassle (we never actually generate this!).  Do it now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell 9bddfc2048 connectd: take dev-suppress-gossip from gossipd.
Gossipd didn't actually suppress all gossip, resulting in a flake!
Doing it in connectd now makes much more sense.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 19:38:05 +10:30
Rusty Russell 2f7f7ec255 lightningd: add "style" to listforwards.
Suggested-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listforwards` has new entry `style`, currently "legacy" or "tlv".
2022-03-31 19:38:05 +10:30
Rusty Russell 116a77f1be lightningd: Restore forwarding of legacy onions.
Partial revert of 43a833e405
"lightningd: remove support for legacy onion format."; we restore the
ability to decode legacy onions for forwarding, but not to generate them.
(We don't accept them properly since making payment_secret compulsory
anyway, so no real change there!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: ... but we still forward legacy HTLC onions for now.
2022-03-31 19:38:05 +10:30
Rusty Russell 0b7f789291 lightningd: extra sanity checks and rescue attempts for missing HTLCs.
These trip when anything weird happens; turns out that we tell
onchaind about old htlcs (e.g. for penalties), so in that case we can
actually have it tell us about missing HTLCs which we no longer have
in memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell e616b4fff7 lightningd: add extra debugging for weird onchain htlc interactions.
This doesn't actually fix anything, but may shed more clues if it
happens again.

The broken() logs are overzealous, see next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell ea7120a313 lightningd: add --dev-no-ping-timer to avoid ping response timeouts.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell 5c949e3116 subd: make channel/peer own the subd.
We get some memleak reports because ld owns the subd, but once
the peer/channel is freed, there's no reference for the brief time
until the subd exits.

This happens for both opening and closingd.  For openingd, the
peer owns it, for others (including dualopend) the channel owns it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-30 06:27:52 +10:30
Rusty Russell 20392ae526 connectd: restore obs2 onion support.
I removed these prematurely: we *haven't* had a release since
introducing them!

This consists of reverting d15d629b8b
"plugins/fetchinvoice: remove obsolete string-based API." and
plugins/fetchinvoice: remove obsolete string-based
API. "onion_messages: remove obs2 support."

Some minor changes due to updated fromwire_tlv API since they
were removed, but not much.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: REVERT: Removed backwards compat with onion messages from v0.10.1.
2022-03-29 10:55:12 +10:30
Rusty Russell aad4495f56 delinvoice: allow desconly arg to only remove the description.
Means that field is now optional in JSON output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `delinvoice` has a new parameter `desconly` to remove description.
2022-03-29 10:04:16 +10:30
Rusty Russell ccaf04d268 invoice: add deschashonly parameter.
LNURL wants this so they can include images etc in descriptions.

Replaces: #4892
Changelog-Added: JSON-RPC: `invoice` has a new parameter `deschashonly` to put hash of description in bolt11.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-29 10:04:16 +10:30
Rusty Russell 7abc491f4c pay: fix crash on invalid onion in reply in some cases.
fail can be NULL here.

Reported-by: https://github.com/shafemtol
Fixes: #5053
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:59:46 +10:30
Rusty Russell a770f51d0e tools/generate_wire.py: make functions allocate the TLV.
Requiring the caller to allocate them is ugly, and differs from
other types.

This means we need a context arg if we don't have one already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell fa0c29f959 tools/generate_wire.py: tlvs should start with tlv_
No more "towire_offer", but "towire_tlv_offer".

This means we double-up on the unfortunately-named `tlv_payload` inside
the onion, but we should rename that in the spec when we remove
old payloads.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 7829f2eb06 onion_messages: remove obs2 support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Removed backwards compat with onion messages from v0.10.1.
2022-03-25 13:55:44 +10:30
Rusty Russell 726b6878d1 offers: import latest variant from draft.
In particular, this changes the name of a field in invoice_request:
`payer_signature` becomes simply `signature`.  So we allow both for
now, and send the old one unless deprecated_apis is disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Sjors Provoost 560b090bd8 Add more warnings to fundchannel_start
[ Slight modifications to fix schema -- RR ]
2022-03-24 14:11:40 +10:30
Rusty Russell f7aba31448 lightningd: allow 'style' 'legacy' for clboss compat (with deprecated_apis).
We keep the parameter around for future use (e.g. PTLC support?), but clboss
master still sets 'style' to "legacy".  Accept, but ignore it.

Reported-by: grubman on #c-lightning (IRC)
Changelog-Deprecated: JSON-RPC: `sendpay` `route` argument `style` "legacy" (don't use it at all, we ignore it now and always use "tlv" anyway).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-24 13:19:10 +10:30
Vincenzo Palazzo 53806d1abd cli: make the command line more user friendly.
Also has to fix up tests.

Changelog-Fixed: cli doesn't required anymore to confirm the password if the `hsm_secret` is already encrypted.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-24 09:53:38 +10:30
Vincenzo Palazzo 58ce4cf71c cli: check if the pass and the confirmation pass match
Changelog-Fixed: check if the pass and the confirmation pass match from the command line

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-24 09:53:38 +10:30
Rusty Russell 9c978f2c63 lightningd: free peer->uc when openingd fails.
This avoids reuse which can cause confusion; the long-term fix is to
rewrite this to use real channels like dualfunding does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell bcf3cef96c pytest: fix flake in test_connection.py::test_opener_feerate_reconnect
Make sure it sees disconnect before reconnect, otherwise the next command
fails since we're now disconnected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 07c4d39b75 memleak: fix double-free if we timeout.
... and then dualopend returns, and we access the fread leak_detect struct.

```
lightningd: FATAL SIGNAL 6 (version 065ca1e)
0x55ecd4be8145 send_backtrace
	common/daemon.c:33
0x55ecd4be81f1 crashdump
	common/daemon.c:46
0x7f200acab51f ???
	./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x7f200acff828 __pthread_kill_implementation
	./nptl/pthread_kill.c:44
0x7f200acff828 __pthread_kill_internal
	./nptl/pthread_kill.c:80
0x7f200acff828 __GI___pthread_kill
	./nptl/pthread_kill.c:91
0x7f200acab475 __GI_raise
	../sysdeps/posix/raise.c:26
0x7f200ac917b6 __GI_abort
	./stdlib/abort.c:79
0x55ecd4c6827f call_error
	ccan/ccan/tal/tal.c:93
0x55ecd4c68470 check_bounds
	ccan/ccan/tal/tal.c:165
0x55ecd4c684c2 to_tal_hdr
	ccan/ccan/tal/tal.c:175
0x55ecd4c68eb8 tal_free
	ccan/ccan/tal/tal.c:479
0x55ecd4b8bdd0 finish_report
	lightningd/memdump.c:138
0x55ecd4b8c115 leak_detect_req_done
	lightningd/memdump.c:201
0x55ecd4c68664 notify
	ccan/ccan/tal/tal.c:237
0x55ecd4c68b9e del_tree
	ccan/ccan/tal/tal.c:402
0x55ecd4c68bf3 del_tree
	ccan/ccan/tal/tal.c:412
0x55ecd4c68bf3 del_tree
	ccan/ccan/tal/tal.c:412
0x55ecd4c68f43 tal_free
	ccan/ccan/tal/tal.c:486
0x55ecd4c5751f io_close
	ccan/ccan/io/io.c:450
0x55ecd4bbce68 subd_shutdown_remaining
	lightningd/subd.c:911
0x55ecd4b8724a shutdown_subdaemons
	lightningd/lightningd.c:541
0x55ecd4b883cc main
	lightningd/lightningd.c:1207
0x7f200ac92fcf __libc_start_call_main
	../sysdeps/nptl/libc_start_call_main.h:58
0x7f200ac9307c __libc_start_main_impl
	../csu/libc-start.c:409
0x55ecd4b5cc54 ???
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 293cf3c2b2 connect: delay return until all subds ready.
We had some flakes because we returned from `connect`, but we hadn't started
subds yet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 1e4149f18a lightningd: clean up peer connection handling a little.
Update the address and direction as soon as it connects not just when
we're about to make it active: we want this even if we don't have
an active channel, or if the connect hook rejects it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 4e8239fcfe lightningd: don't tell connectd to discard peer unless no subds left.
Otherwise it waits for subds to exit, but they don't.  Plus, the others
may still be talking!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 75596b3e0f lightningd: use a better channel if available to next hop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell debc1b90d3 lightningd: remove checks which prevent us from opening multiple channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: we now support opening multiple channels with the same peer.
2022-03-23 13:20:12 +10:30
Rusty Russell 21e1d68e3b lightningd: remove (most) functions to search channels by status.
This is generally verboten now, since there can be multiple.  There are a
few exceptions:

1. We sometimes want to know if there are *any* active channels.
2. Some dev commands still take peer id when they mean channel_id.
3. We still allow peer id when it's fully determined.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `close` by peer id will fail if there is more than one live channel (use `channel_id` or `short_channel_id` as id arg).
2022-03-23 13:20:12 +10:30
Rusty Russell f85425d106 lightningd: don't assume a single channel per peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell ba1242af3e lightningd: add find_channel_by_scid
More efficient to search a known peer than the whole set.

Also, move find_channel_by_id() from channel_control.c into channel.c
where we'd expect it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell cb5dc48cab lightningd: make setchannelfee handle multiple channels per peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell b3438e9bba lightningd: associate connect commands with peer, not channel.
Sure, we want to connect (usually) because of an active channel, but
it's not specific to the channel itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 33abf93ec1 lightningd: rename activate_peers() to setup_peers().
Activate means a specific thing now (connectd said something), so avoid
confusing it with this function.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 90be2cc104 lightningd: remove some "single active channel" assumptions.
Generally this means converting a lazy "peer_active_channel(peer)" call
into an explicit iteration.

1. notify_feerate_change: call all channels (ignores non-active ones anyway).
2. peer_get_owning_subd remove unused function.
3. peer_connected hook: don't save channel, do lookup and iterate channels.
4. In json_setchannelfee "all" remove useless call to peer_active_channel
   since we check state anyway, and iterate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 7de7b7be61 lightningd: use channel_id when a peer is activated.
Rather than intuiting whether this is a new channel / active channel,
use the channel_id.  This simplifies things and makes them explicit,
and prepares for multiple live channels per peer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell fe9f391a93 connectd: tell lightningd the channel_id when we give it the active peer.
Now we always have it (either extracted from an unsolicited message,
or told to us by lightningd when it tells us it wants to talk), we can
always send it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 2bc58e2327 lightningd: always tell connectd the channel id.
This means lightningd needs to create the temporary one and tell it to
openingd/dualopend, rather than the other way around.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 57263a3eb2 lightningd: handle reestablish directly from connectd.
We don't need to hand it to channeld: it will read it!  We simply
need to tell it to expect it.

Similarly, openingd/dualopend will never see it, so remove that logic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 2424b7dea8 connectd: hold peer until we're interested.
Either because lightningd tells us it wants to talk, or because the peer
says something about a channel.

We also introduce a behavior change: we disconnect after a failed open.
We might want to modify this later, but we it's a side-effect of openingd
not holding onto idle connections.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 77b1087cdf lightningd: move notification of disconnect into when we hear from connectd.
Simpler, and closes a potential race.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell deecedb033 connectd: tell lightningd when disconnect is complete.
This avoids races in our tests where we assume it's sync (and is kind
of nicer).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 6cc9f37cab connectd: handle connect vs closing race better.
We would return success from connect even though the peer was closing;
this is technically correct but fairly undesirable.  Better is to pass
every connect attempt to connectd, and have it block if the peer is
exiting (and retry), otherwise tell us it's already connected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 10e36e073c openingd: disconnect from peer when an error occurs.
openingd currently holds the connection to idle peers, but we're about
to change that: it will only look after peers which are actively
opening a connection.  We can start this process by disconnecting
whenever we have a negotiation failure.

We could stay connected if we wanted to, but that would be up to
connectd to decide.  Right now it's easier if we disconnect from any
idle peer once it's been active.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 16e9ba0361 connectd: fix confusing names.
The message from lightningd simply acknowleges that we are allowed to
discard the peer (because no subdaemons are talking to it anymore).
This difference becomes more stark once connectd holds on to idle
peers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell eb203bf71e lightningd: clean up connect code.
1. The notification should be called every time.
2. channel can never be NULL, since it's tested above.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `connect` notification now called even if we already have a live channel.
2022-03-23 13:20:12 +10:30
Rusty Russell b99c04e605 lightningd: add explicit "connected" flag.
We currently intuit this by whether there's a subdaemon owning it.
But we're about to change the rules and allow connectd to hold idle
connections, so we need an explicit flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Rusty Russell 5704653d4c setchannel: don't let them advertize htlc_maximum_msat larger than capacity.
And check for the obvious setting min > max.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell 999c734bb5 setchannel: add minhtlc
Suggested by @m-schmook, I realized that if we append it later I'll
never get it right: I expect parameters min and max, not max and min!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: you can now alter the `htlc_minimum_msat` and `htlc_maximum_msat` your node advertizes.
2022-03-22 18:45:41 +10:30