Commit Graph

2421 Commits

Author SHA1 Message Date
Rusty Russell 85fc46f76c pyln-testing: don't default openchannel and fundwallet to p2sh-segwit, use bech32.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-14 21:17:45 +10:30
Rusty Russell 9ffaab7d22 pytest: fix race in test_bookkeeping_closing_subsat_htlcs
With the next change (which, as a side-effect, speeds up listpeers),
we seem to hit a race in this test.  The bookkeeper doesn't get to
process the final payment before the node is shutdown.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-13 10:42:42 +10:30
Vincenzo Palazzo a2347c7452 tests: use listpeerchannels. 2023-01-13 10:42:42 +10:30
Vincenzo Palazzo 1fa32333b9 tests/utils.py: use listpeerchannels. 2023-01-13 10:42:42 +10:30
niftynei 42c6d49082 tests: add account_id's and match by acct id, not test ordering
There's no guarantee as to iteration order for accounts/channels, but
this test was relying on them.

Adding account attribution and comparing by account_ids fixes

Fixes: #5869
Reported-By: @rustyrussell
2023-01-12 11:40:17 +10:30
niftynei 66a4d50098 tests: mark test as dev (times out otherwise) 2023-01-12 11:40:17 +10:30
Rusty Russell 404e961bad cli: add -c/--commando support.
It's easier to type:

```
lightning-cli --commando=03ce2d830369fc903ffec52ca1d7aba095c3cf5d17175b6c9a3ff058f6aece37bc:V08OylkJ2ZZPClAXbTaxrXJ9YpKnmucJxcQI-wvIGiE9MA== invoice any "Invoice Label" "Invoice Description"
lightning-cli --commando=03ce2d830369fc903ffec52ca1d7aba095c3cf5d17175b6c9a3ff058f6aece37bc:V08OylkJ2ZZPClAXbTaxrXJ9YpKnmucJxcQI-wvIGiE9MA== commando amount_msat=100000 label="invoice label" description="invoice description"
```

Than:

```
commando 03ce2d830369fc903ffec52ca1d7aba095c3cf5d17175b6c9a3ff058f6aece37bc invoice '["any", "Invoice Label", "Invoice Description"]' V08OylkJ2ZZPClAXbTaxrXJ9YpKnmucJxcQI-wvIGiE9MA==
commando 03ce2d830369fc903ffec52ca1d7aba095c3cf5d17175b6c9a3ff058f6aece37bc invoice '{"amount_msat": "100000", "label": "invoice label", "description": "invoice description"}' V08OylkJ2ZZPClAXbTaxrXJ9YpKnmucJxcQI-wvIGiE9MA==
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: cli: `--commando=peerid:rune` (or `-c peerid:rune`) as convenient shortcut for running commando commands.
2023-01-11 11:13:27 +10:30
Rusty Russell 3f0c5b985b commando: add filtering support.
1. When we receive a commando command from a remote using the `filter`
   field, use it.
2. Add a `filter` parameter to `commando` to send it: this is usually
   more efficient than using filtering locally.

Of course, older remote nodes will ignore the filter, but that's
harmless.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `commando` now supports `filter` as a parameter (for send and receive).
2023-01-11 11:13:27 +10:30
Rusty Russell 1250806060 commando: correctly replace the `id` field in responses.
This was reported a while ago: now do it properly.

Fixes: #5637
Changelog-Fixed: Plugins: `commando` now responds to remote JSON calls with the correct JSON `id` field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Rusty Russell b75ada7017 commando: track incoming and outgoing JSON IDs. Get upset if they don't match!
They currently don't, so we get some BROKEN messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Rusty Russell 0201e6977f commando: build ID of command based on the id they give us.
We change the libplugin API so commando can provide its own ID base.

This id chaining enables much nicer diagnostics!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Rusty Russell b3fa4b932e commando: send `id` inside JSON request.
All JSON-RPC calls should have one!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Rusty Russell 19db6a25e4 commando: require that we have an `id` field in JSON request.
We don't do this yet, so we add deprecated to those test (until next
patch!).

Changelog-Deprecated: plugins: `commando` JSON commands without an `id` (see doc/lightningd-rpc.7.md for how to construct a good id field).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Rusty Russell 435f8d84dc lightning-cli: fix error code on invalid options, document them.
The top of the file indicates the following errors:

    #define NO_ERROR 0
    #define ERROR_FROM_LIGHTNINGD 1
    #define ERROR_TALKING_TO_LIGHTNINGD 2
    #define ERROR_USAGE 3

But we didn't use the right one for opt_parse failure, and didn't use the
correct constants everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Rusty Russell a1e894a445 lightningd: treat JSON ids as direct tokens.
This avoids any confusion between primitive and string ids, and in
particular stops an issue with commando once it starts chaining ids,
that weird ids can be double-escaped and commando will not recognize
the response, leaving the client hanging.  It's the client's fault for
using a weird id, but it's still rude (and triggered by our tests!).

It also makes substituting the id in passthrough simpler, FTW.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-11 11:13:27 +10:30
Gregg Zigler 9ed138f5e5 pytest: add tests for devtools/mkfunding
Issue #5363 documented an earlier bug in mkfunding. These tests validate that fix as well as checking that basic usage produces the proper response and exit status.

Changelog-None
2023-01-09 14:50:30 -06:00
Rusty Russell d1b5943e90 tests: make test_libplugin use the datastore.
It's a bit convoluted, but it exercises all our datastore fetch APIs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-12-22 12:04:30 +01:00
Rusty Russell ec025344cc lightningd: don't announce names as DNS by default.
This broke BTCPayServer, so revert.  I originally (accidentally!)
implemented this such that it broadcast both DNS and IP entries, but
Michael reported earlier that they still don't propagage well, so
simply suppress them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5795
Changelog-Changeed: Config: `announce-addr-dns` needs to be set to *true* to put DNS names into node announcements, otherwise they are suppressed.
Changelog-Deprecated: Config: `announce-addr-dns` (currently defaults to `false`).  This will default to `true` once enough of the network has upgraded to understand DNS entries.
2022-12-08 11:04:53 +01:00
Rusty Russell 3ca1c70c44 lightningd: don't cap spendable_msat/receivable_msat for wumbo channels.
If we both support large channels, we can actually send giant HTLCs.

This, in turn, fixes pay (which relies on this field!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` now knows it can use locally-connected wumbo channels for large payments.
Fixes: #5250
Fixes: #5417
2022-11-30 19:51:22 +01:00
Rusty Russell b6555dccaf pytest: test for wumbo direct payments.
We should be able to make larger payments if we're directly connected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-30 19:51:22 +01:00
Christian Decker dfb963e249 db: Backfill missing HTLC IDs in the forwards table
We have a primary key that is spanning the `in_channel_id` and the
`in_htcl_id`. The latter gets set to NULL when the HTLC and channel
gets deleted, so we coalesce with a random large number that is
unlikely to collide for the primary key.
2022-11-28 14:47:43 +01:00
Rusty Russell 626998efce lightningd: don't timeout plugins if init is slow!
This is a minimal fix: we wait until all plugins reply from init before
continuing.  Really large or busy nodes can have other things monopolize
lightningd, then the timer goes off and we blame the plugin (which has
responded, we just haven't read it yet!).

The real answer is to have some timeouts only advance when we're idle,
or have them low-priority so we only activate them when we're idle (this
doesn't apply to all timers: some are probably important!).  But
this is a minimal fix for -rc3.

Fixes: https://github.com/ElementsProject/lightning/issues/5736
Changelog-Fixed: plugins: on large/slow nodes we could blame plugins for failing to answer init in time, when we were just slow.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-27 14:58:42 +01:00
Rusty Russell ece77840f9 pyln-client, libplugin, rust cln-plugin: explicitly flag that we allow non-numeric JSON ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-21 11:23:54 +01:00
Rusty Russell d5ce5cbab3 lightningd: only use non-numeric JSON ids if plugin says we can.
We also remember whether the id is a string or not, for replacement in
JSON passthrough.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-21 11:23:54 +01:00
Rusty Russell f2291c44d6 onchaind: cap RBF penalty fee for testnet/regtest
On testnet I noticed if we can't reach bitcoind for some reason, we'll
keep RBFing our penalty tx ("it didn't go in, RBF harder!!").  Makes
no sense to grossly exceed the amount needed for next block, so simply
cap penalty at 2x "estimatesmartfee 2 CONSERVATIVE".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-18 13:38:42 +01:00
Christian Decker 832b2e5e2e onchaind: Adjust witness weight estimate to be more conservative
We were missing the OP_PUSH for the pubkeys, and the spec mentions we
should be using 73 bytes to estimate the witness weight. Effectively
this adds 4 bytes which really just matters in case fees hit the
floor, and computing the weight becomes important.

Changelog-Fixed: onchaind: Witness weight estimations could be slightly lower than the VLS signer
2022-11-10 16:30:27 +01:00
Rusty Russell 37bc4603b8 lightningd: re-add 'offerout' functionality, as 'invoicerequest'.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 7906770489 lightningd: add "savetodb" argument to createinvoicerequest, add listinvoicerequests/disableinvoicerequest
This is how we put new invoice_requests into the db; this will be used
by a new "invoicerequest" command which replaces "offerout".

The API	is now the same as the offers api.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 02d7454226 db: add invoicerequests table.
We no longer use offers for "I want to send you money", but we'll use
invoice_requests directly.  Create a new table for them, and
associated functions.

The "localofferid" for "pay" and "sendpay" is now "localinvreqid".
This is an experimental-only option, so document the change under
experimental only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `pay` and `sendpay` `localofferid` is now `localinvreqid`.
2022-11-09 15:08:03 +01:00
Rusty Russell ef2f4a0648 bolt12: use spec field names, update decode API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 1e3cb01546 bolt12: import the latest spec, update to fit.
I know this is an unforgivably large diff, but the spec has changed so
much that most of this amounts to a rewrite.

Some points:
* We no longer have "offer_id" fields, we generate that locally, as all
  offer fields are mirrored into invoice_request and then invoice.
* Because of that mirroring, field names all have explicit offer/invreq/invoice
  prefixes.
* The `refund_for` fields have been removed from spec: will re-add locally later.
* quantity_min was removed, max == 0 now mean "must specify a quantity".
* I have put recurrence fields back in locally.

This brings us to 655df03d8729c0918bdacac99eb13fdb0ee93345 ("BOLT 12:
add explicit invoice_node_id.")

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 846a520bc2 offers: remove 'send-invoice' offers support.
This has radically changed in the spec, so remove it now, and we'll
reintroduce / rewrite it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 3afa5077fe offers: make them always unsigned.
This is in preparation for the spec update where the signature field
does not even exist.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 9a0d2040d3 common/features: add explicit bolt12 feature sets.
The spec only specifies the mpp bit for invoices, but in
general they are separate spaces.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
Rusty Russell 595fbd2a19 createinvoice: make a minimal blinded "path" in bolt12 invoice if none presented.
The "path" is just a message to ourselves.  This meets the minimal
requirement for bolt12 invoices: that there be a blinded path (at
least so we can use the path_id inside in place of "payment_secret").

We expose the method to make this path_id to a common routine: offers
will need this for generating more sophisticated paths.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 15:08:03 +01:00
niftynei 26f5dcd2a5 wallet: mark coinbase outputs as 'immature' until spendable
Changelog-Changed: JSON-RPC: `listfunds` now lists coinbase outputs as 'immature' until they're spendable
Changelog-Changed: JSON-RPC: UTXOs aren't spendable while immature
2022-11-09 11:55:25 +01:00
niftynei d60dbba43b tests: test for coinbase wallet spend.
Attempt to spend a coinbase transaction, expected to fail.
2022-11-09 11:55:25 +01:00
Rusty Russell ae3550cb00 lightning-cli: support --filter parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: cli: new `--filter` parameter to reduce JSON output.
2022-11-09 20:25:58 +10:30
Rusty Russell cb1156cd32 libplugin: support filters.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell b6134303d4 pyln: add context manager to simpify filter use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: LightningRpc has new `reply_filter` context manager for reducing output of RPC commands.
2022-11-09 20:25:58 +10:30
Rusty Russell 1436ad334d pytest: add filter tests.
We suppress schema reply checking when filter is set: we could just
remove all the `required` fields in the JSON schema.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell 22c42de6f1 tests/fuzz: don't pull in JSON common at all.
No tests currently use it, and if they do we'll want to do some
per-test objects.  Otherwise, we are about it introduce a dependency
on common/json_filter.o, which is a can of worms.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-09 20:25:58 +10:30
Rusty Russell a4c482dc07 common/sphinx: don't use fixed lengths anywhere.
1. Remove the very concept of ONION_REPLY_SIZE, instead make it a
   local variable in create_onionreply().

2. Use the proper fromwire_ primitives in unwrap_onionreply() so we
   don't have to do explicit length checks.

3. Make fromwire_tal_arrn() return NULL if it fails to pull, instead of
   a zero-length allocation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: we now correctly decrypt non-256-length onion errors (we always forwarded them fine, now we actually can parse them).
2022-11-08 17:40:57 +01:00
Rusty Russell fe1b285bba pytest: add test for generating non-standard length onion errors.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-08 17:40:57 +01:00
Alex Myers 8f48406985 pytest: test for gossip store corruption by private channel updates
Adds a simple test covering the gossip_store corruption issue #5656
stemming from failure to delete prior private channel updates in all
cases.
2022-11-03 11:29:11 +01:00
Christian Decker 2605e117c9 pytest: Add test for optional options in cln-plugin
Changelog-Added: cln-plugin: Options are no longer required to have a default value
2022-11-03 11:28:22 +01:00
Christian Decker 8a4f44a58d keysend: Allow quoted numbers in `extratlvs`
This is because JSON technically does not allow numeric keys in maps.

Changelog-Added: JSON-RPC: The `extratlvs` argument for `keysend` now allows quoting the type numbers in string
2022-11-01 17:05:30 +01:00
Rusty Russell e30ea91908 BOLTs: update to more recent bolt12 spec.
It's 2b7ad577d7a790b302bd1aa044b22c809c76e49d, which reverts the
point32 changes.

It also restores send_invoice in `invoice`, which we had removed
from spec and put into the recurrence patch.

I originally had implemented compatibility, but other changes
which followed this are far too widespread.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: complete rework of spec from other teams (yay!) breaks previous compatibility (boo!)
2022-10-26 11:29:06 +10:30
Rusty Russell 7745513c51 bolt12: change our payer_key calculation.
It was very tied to x-only keys; we could support it in a backwards
compatibility mode for a while, but getting refunds or proving old
pre-finalization invoices is not worth spending time on.

Changelog-EXPERIMENTAL: offers: old `payer_key` proofs won't work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
niftynei e00857827f funder: cleanup datastore on state-change/channel failure
Let's not leave old state hanging around! Note that this fires
for pretty much every/any channel (even if we're not the opener).
2022-10-20 13:42:41 +02:00