Commit Graph

972 Commits

Author SHA1 Message Date
Rusty Russell c2c9f45dac offers: create a real blinded path, if necessary.
It's just to a direct peer, and we only create one, but this is
enough to test, and make payments to non-public nodes work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 744605997e offers: monitor blockheight.
We need this to create the payment_constraints for an invoice blinded path.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 4bc10579e6 listincoming: add htlc_min_msat, public and peer_features fields.
This is needed for offers to generate blinded paths.

No documentation changes since listincoming is an undocumented
internal hack interface which topology presents for production
of routehints.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 5becfa6ee1 onion_message: don't use general secret, use per-message secret.
We had a scheme where lightningd itself would put a per-node secret in
the blinded path, then we'd tell the caller when it was used.  Then it
simply checks the alias to determine if the correct path was used.

But this doesn't work when we start to offer multiple blinded paths.
So go for a far simpler scheme, where the secret is generated (and
stored) by the caller, and hand it back to them.

We keep the split "with secret" or "without secret" API, since I'm
sure callers who don't care about the secret won't check that it
doesn't exist!  And without that, someone can use a blinded path for a
different message and get a response which may reveal the node.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 4cfd972407 common/blindedpath: expose API at a lower level.
We actually want lightningd to create these, since it wants to put the
path_id secret in the last element.  So best API is actually a generic
one, rather than separate APIs to create first and last ones.

And really, the more explicit initialization makes the users clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 01a47720c3 plugins/libplugin-pay: hack in blinded path support.
We simply take the first one, and route to the start of that.  Then we
append the blinded path to the onion construction.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell cb1156cd32 libplugin: support filters.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell f0731d2ca1 common/json_stream: support filtering don't print fields not allowed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell 508a170598 common/json_filter: routine to turn "filter" JSON into a filter.
Since the "struct command" is different from plugins and lightningd, we
need an accessor for this to work (the plugin one is a dummy for now!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell 3c75770586 common/json_filter: routines for json filtering.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Christian Decker 2605e117c9 pytest: Add test for optional options in cln-plugin
Changelog-Added: cln-plugin: Options are no longer required to have a default value
2022-11-03 11:28:22 +01:00
Justin Moon f111d6772d Plugin config options with no defaults 2022-11-03 11:28:22 +01:00
Rusty Russell 5cf86a1a2e common: update to latest onion message spec.
Mainly, field name changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Protocol: Support for forwarding blinded payments (as per latest draft)
2022-10-26 11:29:06 +10:30
Rusty Russell 426886ff9b lightningd: return invalid_onon_blinding for any blinded payment error.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell c0ae2394d8 common/blindedpath: generalize construction routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell 662c6931f3 Remove point32.
The x-only dream is dead.  Remove all trace.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell 1cdf21678e offers: print out more details, fix up schema for decode of blinded paths.
We need to print out first_node_id, and "node_id" is now called
"blinded_node_id" in the spec.

And the schema didn't include the payment fields in the blinded path
for invoices (which broke as soon as we actually tested one!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell e30ea91908 BOLTs: update to more recent bolt12 spec.
It's 2b7ad577d7a790b302bd1aa044b22c809c76e49d, which reverts the
point32 changes.

It also restores send_invoice in `invoice`, which we had removed
from spec and put into the recurrence patch.

I originally had implemented compatibility, but other changes
which followed this are far too widespread.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: complete rework of spec from other teams (yay!) breaks previous compatibility (boo!)
2022-10-26 11:29:06 +10:30
Rusty Russell eac8401f84 Makefile: separate bolt12 wireobjects
Most things don't want them, so don't link it into everything by default.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell 82d98e4b96 gossmap: move gossmap_guess_node_id to pay plugin.
This removes a point32 dependency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell bed905a394 lightningd: use 33 byte pubkeys internally.
We still use 32 bytes on the wire, but internally don't use x-only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
niftynei e00857827f funder: cleanup datastore on state-change/channel failure
Let's not leave old state hanging around! Note that this fires
for pretty much every/any channel (even if we're not the opener).
2022-10-20 13:42:41 +02:00
niftynei efd096dc96 funder: filter prev-outs such that we only use still unspent ones
If for some reason a utxo we used previously is no longer 'unspent',
we shouldn't use it for the next transaction.
2022-10-20 13:42:41 +02:00
niftynei 38e2428f12 funder: use utxopsbt to build psbt for RBFs
We use the saved previous outputs (plus maybe some new ones?) to build a
psbt for an RBF request.

RBFs utxo reuse is now working so we can unfail the test (and update
it to reflect that the lease sticks around through an RBF cycle).

Changelog-Fixed: Plugins: `funder` now honors lease requests across RBFs
2022-10-20 13:42:41 +02:00
niftynei 45acc20a8d funder: use previous outputs in count towards available funding
Still need to use them to build the PSBT for the rbf however.
2022-10-20 13:42:41 +02:00
niftynei 7733c2b0f4 funder: pull out previous input list from datastore on RBF
It'd be nice to know which utxos we used previously, so we can rebuild a
transaction using them!
2022-10-20 13:42:41 +02:00
niftynei a5e9035e2e funder: save utxos of signed txs to datastore
We're going to need to know what utxos we used if we RBF this channel;
so we serialize our inputs and save them to the datastore under the channel_id.
2022-10-20 13:42:41 +02:00
niftynei dc9e79c445 funder: rm quote that makes nifty cringe every time she sees it 2022-10-20 13:42:41 +02:00
niftynei 85039d4f4e df: pass lease data back to funder for rbfs
let's let RBFs know about our lease info!
2022-10-20 13:42:41 +02:00
niftynei 8d864d2299 funder: we always pass in channel_max, no need to special case it
Parse the channel_max along with everything else.
2022-10-20 13:42:41 +02:00
Rusty Russell e855ac2f9e keysend: just strip even unknown fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `keysend` now removes unknown even (technically illegal!) fields, to try to accept more payments.
2022-10-04 17:48:08 +02:00
Rusty Russell 2fbe0f59f1 plugins/fetchinvoice: remove obsolete string-based API.
Generate the payload in the callers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-29 16:10:57 +09:30
Rusty Russell 41ef85318d onionmessages: remove obsolete onion message parsing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-29 16:10:57 +09:30
Rusty Russell 8771c86379 common/onion: expunge all trace of different onion styles.
In particular, remove special routines to pull length: it's there,
take it and check it yourself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-28 13:40:57 +02:00
Rusty Russell 68f15f17bb delforward: allow deletion of "unknown in_htlc_id" and fix autoclean to use it.
Note the caveats: we will delete *all* of them at once!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-27 14:42:03 +09:30
Christian Decker e1fc88ff70 cln-plugin: Prep the logging payload in a let
I wanted to debug why we weren't getting some log messages, and this
makes that a bit easier in future.
2022-09-25 15:13:12 +02:00
Christian Decker 064a5a6940 cln-plugin: Add log filtering support
We filter based on the environment variable `CLN_PLUGIN_LOG`,
defaulting to `info` as that is not as noisy as `debug` or `trace`, at
least libraries will not spam us too heavily.

Changelog-Added cln-plugin: The logs level from cln-plugins can be configured by the `CLN_PLUGIN_LOG` environment variable.
2022-09-25 15:13:12 +02:00
Christian Decker 8898511cf6 cln-plugin: Defer binding the plugin state until after configuring
We had a bit of a chicken-and-egg problem, where we instantiated the
`state` to be managed by the `Plugin` during the very first step when
creating the `Builder`, but then the state might depend on the
configuration we only get later. This would force developers to add
placeholders in the form of `Option` into the state, when really
they'd never be none after configuring.

This defers the binding until after we get the configuration and
cleans up the semantics:

 - `Builder`: declare options, hooks, etc
 - `ConfiguredPlugin`: we have exchanged the handshake with
   `lightningd`, now we can construct the `state` accordingly
 - `Plugin`: Running instance of the plugin

Changelog-Changed: cln-plugin: Moved the state binding to the plugin until after the configuration step
2022-09-25 15:13:12 +02:00
Rusty Russell bb49e1bea5 common: assume htlc_maximum_msat, don't check bit any more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell 253b25522b BOLT: update to version which requires option_channel_htlc_max.
We will now simply reject old-style ones as invalid.  Turns out the
only trace we could find is a channel between two nodes unconnected to
the rest of the network.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now require all channel_update messages include htlc_maximum_msat (as per latest BOLTs)
2022-09-24 15:22:27 +09:30
Rusty Russell 6338758018 gossmap: make API more robust against future changes.
Many changes to gossmap (including the pending ones!) don't actually
concern readers, as long as they obey certain rules:

1. Ignore unknown messages.
2. Treat all 16 upper bits of length as flags, ignore unknown ones.

So now we split the version byte into MAJOR and MINOR, and you can
ignore MINOR changes.

We don't expose the internal version (for creating the map)
programmatically: you should really hardcode what major version you
understand!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell f52ff07558 lightningd: allow delpay to delete a specific payment.
This is actually what the autoclean plugin wants, especially since
you can't otherwise delete a payment which has failed then succeeded.

But insist on neither or both being specified, at least for now.

Changelog-Added: JSON-RPC: `delpay` takes optional `groupid` and `partid` parameters to specify exactly what payment to delete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell fa7d732ba6 lightningd: allow a connection to specify db batching.
Previous commit was a hack which *always* batched where possible, this
is a more sophisticated opt-in varaint, with a timeout sanity check.

Final performance for cleaning up 1M pays/forwards/invoices:

```
$ time l1-cli autoclean-once succeededpays 1
{
   "autoclean": {
      "succeededpays": {
         "cleaned": 1000000,
         "uncleaned": 26895
      }
   }
}

real	6m9.828s
user	0m0.003s
sys	0m0.001s
$ time l2-cli autoclean-once succeededforwards 1
{
   "autoclean": {
      "succeededforwards": {
         "cleaned": 1000000,
         "uncleaned": 40
      }
   }
}

real	3m20.789s
user	0m0.004s
sys	0m0.001s
$ time l3-cli autoclean-once paidinvoices 1
{
   "autoclean": {
      "paidinvoices": {
         "cleaned": 1000000,
         "uncleaned": 0
      }
   }
}

real	6m47.941s
user	0m0.001s
sys	0m0.000s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `batching` command to allow database transactions to cross multiple back-to-back JSON commands.
2022-09-22 15:19:46 +02:00
Rusty Russell 8b7a8265e7 libplugin: avoid memmove if we have many outputs to lightningd.
Use linked list, not an array.

```
+   97.89%     0.01%  autoclean  autoclean             [.] next_plan
-   97.08%     0.01%  autoclean  autoclean             [.] json_stream_output_write
   - 97.06% json_stream_output_write
      - 84.29% ld_stream_complete
         - 83.87% tal_arr_remove_
            + 83.71% __memcpy_avx_unaligned_erms (inlined)
      + 12.76% rpc_stream_complete
+   96.59%     0.03%  autoclean  autoclean             [.] tal_arr_remove_
+   96.48%     0.00%  autoclean  libc.so.6             [.] __memcpy_avx_unaligned_erms (inlined)
+   94.98%    94.98%  autoclean  libc.so.6             [.] __memmove_avx_unaligned_erms
+   84.29%     0.01%  autoclean  autoclean             [.] ld_stream_complete
+   12.76%     0.00%  autoclean  autoclean             [.] rpc_stream_complete
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 4d8c321517 libplugin: optimize parsing lightningd rpc responses.
autoclean was using 98% of its time in memmove; we should simply keep
an offset, and memmove when it's empty.  And also, only memmove the
used region, not the entire buffer!

Running on product of giantnodes.py:

	$ time l1-cli autoclean-once failedpays 1
	{
	   "autoclean": {
	      "failedpays": {
	         "cleaned": 26895,
	         "uncleaned": 1000000
	      }
	   }
	}

Before:
	real	20m46.579s
	user	0m0.000s
	sys	0m0.001s
	
After:
	real	2m10.568s
	user	0m0.001s
	sys	0m0.000s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 540a6e4b99 autoclean: remove per-delete debugging messages.
They slow down benchmarking, which is kind of unfair!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 13e10877de autoclean: add autoclean-once command.
Changelog-Added: Plugins: `autoclean-once` command for a single cleanup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 399288db3f autoclean: use config variables, not commands.
It's more natural: we will eventually support dynamic config variables,
so this will be quite nice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `autoclean` can now delete old forwards, payments, and invoices automatically.
2022-09-22 15:19:46 +02:00
Rusty Russell a15f1be5f8 autoclean: clean up listforwards as well.
And take the opportunity to rename l0 and l1 in the tests to the
more natural l1 l2 (since we add l3).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 4cab396cc8 autoclean: handle cleaning of old payments (not just invoices).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00