Commit Graph

965 Commits

Author SHA1 Message Date
Rusty Russell f0731d2ca1 common/json_stream: support filtering don't print fields not allowed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell 508a170598 common/json_filter: routine to turn "filter" JSON into a filter.
Since the "struct command" is different from plugins and lightningd, we
need an accessor for this to work (the plugin one is a dummy for now!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell 3c75770586 common/json_filter: routines for json filtering.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Christian Decker 2605e117c9 pytest: Add test for optional options in cln-plugin
Changelog-Added: cln-plugin: Options are no longer required to have a default value
2022-11-03 11:28:22 +01:00
Justin Moon f111d6772d Plugin config options with no defaults 2022-11-03 11:28:22 +01:00
Rusty Russell 5cf86a1a2e common: update to latest onion message spec.
Mainly, field name changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Protocol: Support for forwarding blinded payments (as per latest draft)
2022-10-26 11:29:06 +10:30
Rusty Russell 426886ff9b lightningd: return invalid_onon_blinding for any blinded payment error.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell c0ae2394d8 common/blindedpath: generalize construction routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell 662c6931f3 Remove point32.
The x-only dream is dead.  Remove all trace.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell 1cdf21678e offers: print out more details, fix up schema for decode of blinded paths.
We need to print out first_node_id, and "node_id" is now called
"blinded_node_id" in the spec.

And the schema didn't include the payment fields in the blinded path
for invoices (which broke as soon as we actually tested one!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell e30ea91908 BOLTs: update to more recent bolt12 spec.
It's 2b7ad577d7a790b302bd1aa044b22c809c76e49d, which reverts the
point32 changes.

It also restores send_invoice in `invoice`, which we had removed
from spec and put into the recurrence patch.

I originally had implemented compatibility, but other changes
which followed this are far too widespread.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: complete rework of spec from other teams (yay!) breaks previous compatibility (boo!)
2022-10-26 11:29:06 +10:30
Rusty Russell eac8401f84 Makefile: separate bolt12 wireobjects
Most things don't want them, so don't link it into everything by default.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell 82d98e4b96 gossmap: move gossmap_guess_node_id to pay plugin.
This removes a point32 dependency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell bed905a394 lightningd: use 33 byte pubkeys internally.
We still use 32 bytes on the wire, but internally don't use x-only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
niftynei e00857827f funder: cleanup datastore on state-change/channel failure
Let's not leave old state hanging around! Note that this fires
for pretty much every/any channel (even if we're not the opener).
2022-10-20 13:42:41 +02:00
niftynei efd096dc96 funder: filter prev-outs such that we only use still unspent ones
If for some reason a utxo we used previously is no longer 'unspent',
we shouldn't use it for the next transaction.
2022-10-20 13:42:41 +02:00
niftynei 38e2428f12 funder: use utxopsbt to build psbt for RBFs
We use the saved previous outputs (plus maybe some new ones?) to build a
psbt for an RBF request.

RBFs utxo reuse is now working so we can unfail the test (and update
it to reflect that the lease sticks around through an RBF cycle).

Changelog-Fixed: Plugins: `funder` now honors lease requests across RBFs
2022-10-20 13:42:41 +02:00
niftynei 45acc20a8d funder: use previous outputs in count towards available funding
Still need to use them to build the PSBT for the rbf however.
2022-10-20 13:42:41 +02:00
niftynei 7733c2b0f4 funder: pull out previous input list from datastore on RBF
It'd be nice to know which utxos we used previously, so we can rebuild a
transaction using them!
2022-10-20 13:42:41 +02:00
niftynei a5e9035e2e funder: save utxos of signed txs to datastore
We're going to need to know what utxos we used if we RBF this channel;
so we serialize our inputs and save them to the datastore under the channel_id.
2022-10-20 13:42:41 +02:00
niftynei dc9e79c445 funder: rm quote that makes nifty cringe every time she sees it 2022-10-20 13:42:41 +02:00
niftynei 85039d4f4e df: pass lease data back to funder for rbfs
let's let RBFs know about our lease info!
2022-10-20 13:42:41 +02:00
niftynei 8d864d2299 funder: we always pass in channel_max, no need to special case it
Parse the channel_max along with everything else.
2022-10-20 13:42:41 +02:00
Rusty Russell e855ac2f9e keysend: just strip even unknown fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `keysend` now removes unknown even (technically illegal!) fields, to try to accept more payments.
2022-10-04 17:48:08 +02:00
Rusty Russell 2fbe0f59f1 plugins/fetchinvoice: remove obsolete string-based API.
Generate the payload in the callers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-29 16:10:57 +09:30
Rusty Russell 41ef85318d onionmessages: remove obsolete onion message parsing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-29 16:10:57 +09:30
Rusty Russell 8771c86379 common/onion: expunge all trace of different onion styles.
In particular, remove special routines to pull length: it's there,
take it and check it yourself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-28 13:40:57 +02:00
Rusty Russell 68f15f17bb delforward: allow deletion of "unknown in_htlc_id" and fix autoclean to use it.
Note the caveats: we will delete *all* of them at once!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-27 14:42:03 +09:30
Christian Decker e1fc88ff70 cln-plugin: Prep the logging payload in a let
I wanted to debug why we weren't getting some log messages, and this
makes that a bit easier in future.
2022-09-25 15:13:12 +02:00
Christian Decker 064a5a6940 cln-plugin: Add log filtering support
We filter based on the environment variable `CLN_PLUGIN_LOG`,
defaulting to `info` as that is not as noisy as `debug` or `trace`, at
least libraries will not spam us too heavily.

Changelog-Added cln-plugin: The logs level from cln-plugins can be configured by the `CLN_PLUGIN_LOG` environment variable.
2022-09-25 15:13:12 +02:00
Christian Decker 8898511cf6 cln-plugin: Defer binding the plugin state until after configuring
We had a bit of a chicken-and-egg problem, where we instantiated the
`state` to be managed by the `Plugin` during the very first step when
creating the `Builder`, but then the state might depend on the
configuration we only get later. This would force developers to add
placeholders in the form of `Option` into the state, when really
they'd never be none after configuring.

This defers the binding until after we get the configuration and
cleans up the semantics:

 - `Builder`: declare options, hooks, etc
 - `ConfiguredPlugin`: we have exchanged the handshake with
   `lightningd`, now we can construct the `state` accordingly
 - `Plugin`: Running instance of the plugin

Changelog-Changed: cln-plugin: Moved the state binding to the plugin until after the configuration step
2022-09-25 15:13:12 +02:00
Rusty Russell bb49e1bea5 common: assume htlc_maximum_msat, don't check bit any more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell 253b25522b BOLT: update to version which requires option_channel_htlc_max.
We will now simply reject old-style ones as invalid.  Turns out the
only trace we could find is a channel between two nodes unconnected to
the rest of the network.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now require all channel_update messages include htlc_maximum_msat (as per latest BOLTs)
2022-09-24 15:22:27 +09:30
Rusty Russell 6338758018 gossmap: make API more robust against future changes.
Many changes to gossmap (including the pending ones!) don't actually
concern readers, as long as they obey certain rules:

1. Ignore unknown messages.
2. Treat all 16 upper bits of length as flags, ignore unknown ones.

So now we split the version byte into MAJOR and MINOR, and you can
ignore MINOR changes.

We don't expose the internal version (for creating the map)
programmatically: you should really hardcode what major version you
understand!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell f52ff07558 lightningd: allow delpay to delete a specific payment.
This is actually what the autoclean plugin wants, especially since
you can't otherwise delete a payment which has failed then succeeded.

But insist on neither or both being specified, at least for now.

Changelog-Added: JSON-RPC: `delpay` takes optional `groupid` and `partid` parameters to specify exactly what payment to delete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell fa7d732ba6 lightningd: allow a connection to specify db batching.
Previous commit was a hack which *always* batched where possible, this
is a more sophisticated opt-in varaint, with a timeout sanity check.

Final performance for cleaning up 1M pays/forwards/invoices:

```
$ time l1-cli autoclean-once succeededpays 1
{
   "autoclean": {
      "succeededpays": {
         "cleaned": 1000000,
         "uncleaned": 26895
      }
   }
}

real	6m9.828s
user	0m0.003s
sys	0m0.001s
$ time l2-cli autoclean-once succeededforwards 1
{
   "autoclean": {
      "succeededforwards": {
         "cleaned": 1000000,
         "uncleaned": 40
      }
   }
}

real	3m20.789s
user	0m0.004s
sys	0m0.001s
$ time l3-cli autoclean-once paidinvoices 1
{
   "autoclean": {
      "paidinvoices": {
         "cleaned": 1000000,
         "uncleaned": 0
      }
   }
}

real	6m47.941s
user	0m0.001s
sys	0m0.000s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `batching` command to allow database transactions to cross multiple back-to-back JSON commands.
2022-09-22 15:19:46 +02:00
Rusty Russell 8b7a8265e7 libplugin: avoid memmove if we have many outputs to lightningd.
Use linked list, not an array.

```
+   97.89%     0.01%  autoclean  autoclean             [.] next_plan
-   97.08%     0.01%  autoclean  autoclean             [.] json_stream_output_write
   - 97.06% json_stream_output_write
      - 84.29% ld_stream_complete
         - 83.87% tal_arr_remove_
            + 83.71% __memcpy_avx_unaligned_erms (inlined)
      + 12.76% rpc_stream_complete
+   96.59%     0.03%  autoclean  autoclean             [.] tal_arr_remove_
+   96.48%     0.00%  autoclean  libc.so.6             [.] __memcpy_avx_unaligned_erms (inlined)
+   94.98%    94.98%  autoclean  libc.so.6             [.] __memmove_avx_unaligned_erms
+   84.29%     0.01%  autoclean  autoclean             [.] ld_stream_complete
+   12.76%     0.00%  autoclean  autoclean             [.] rpc_stream_complete
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 4d8c321517 libplugin: optimize parsing lightningd rpc responses.
autoclean was using 98% of its time in memmove; we should simply keep
an offset, and memmove when it's empty.  And also, only memmove the
used region, not the entire buffer!

Running on product of giantnodes.py:

	$ time l1-cli autoclean-once failedpays 1
	{
	   "autoclean": {
	      "failedpays": {
	         "cleaned": 26895,
	         "uncleaned": 1000000
	      }
	   }
	}

Before:
	real	20m46.579s
	user	0m0.000s
	sys	0m0.001s
	
After:
	real	2m10.568s
	user	0m0.001s
	sys	0m0.000s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 540a6e4b99 autoclean: remove per-delete debugging messages.
They slow down benchmarking, which is kind of unfair!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 13e10877de autoclean: add autoclean-once command.
Changelog-Added: Plugins: `autoclean-once` command for a single cleanup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 399288db3f autoclean: use config variables, not commands.
It's more natural: we will eventually support dynamic config variables,
so this will be quite nice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `autoclean` can now delete old forwards, payments, and invoices automatically.
2022-09-22 15:19:46 +02:00
Rusty Russell a15f1be5f8 autoclean: clean up listforwards as well.
And take the opportunity to rename l0 and l1 in the tests to the
more natural l1 l2 (since we add l3).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 4cab396cc8 autoclean: handle cleaning of old payments (not just invoices).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 660c9af1d9 autoclean: allow cleaning of paid invoices too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 7da51892e8 autoclean: save stats on how much we cleaned.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 17858c9490 lightningd: deprecated "delexpiredinvoice", put functionality in autoclean plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `delexpiredinvoice`: use `autoclean-once`.
2022-09-22 15:19:46 +02:00
Rusty Russell bd76a196f5 autoclean: new interface
In preparation for more things being autocleaned.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell 41a52929f7 libplugin: handle JSON reply after command freed.
This can happen, and in fact does below in our test_autoclean_once
test where we update the datastore, and return from the cmd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell df4b477e88 keysend: allow extratlvs parameter, even in non-experimental mode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `keysend` now has `extratlvs` option in non-EXPERIMENTAL builds.
2022-09-22 17:19:11 +09:30
Rusty Russell ce0f544073 keysend: try to find description in TLV.
"Who needs specs?"  FFS...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `keysend` will now attach the longest valid text field in the onion to the invoice (so you can have Sphinx.chat users spam you!)
2022-09-22 17:19:11 +09:30