Commit Graph

363 Commits

Author SHA1 Message Date
Rusty Russell d8e68893f5 bcli: become less aggressive with onchain fee levels.
Users are more upset recently with the cost of unilateral closes
than they are the risk of being cheated.  While we complete our
anchor implementation so we can use low fees there, let's
get less aggressive (we already have 34 or 18 blocks to close
in the worst case).

The changes are:

- Commit transactions were "2 CONSERVATIVE" now "6 ECONOMICAL".
- HTLC resolution txs were "3 CONSERVATIVE" now "6 ECONOMICAL".
- Penalty txs were "3 CONSERVATIVE" now "12 ECONOMICAL".
- Normal txs were "4 ECONOMICAL" now "12 ECONOMICAL".

There can be no perfect levels, but we have had understandable
complaints recently about how high our default fee levels are.

Changelog-Changed: Protocol: channel feerates reduced to bitcoind's "6 block ECONOMICAL" rate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-11 11:25:16 +09:30
Christian Decker 9470ea3032 pytest: Fix racy `test_unreachable_routehint`
It was failing likely because we were racing on the log
processing. Using `wait_for_log` instead should address that.
2021-04-24 13:58:57 +09:30
Rusty Russell c0a40b37a7 pytest: fix flake in test_pay_disconnect
Channel can be inactive before it disconnects, apparently.  Check
explicitly for the disconnected state so we get the expected error.

Here's what happened:
```
        # Can't pay while its offline.
        with pytest.raises(RpcError, match=r'failed: WIRE_TEMPORARY_CHANNEL_FAILURE \(First peer not ready\)'):
>           l1.rpc.sendpay(route, rhash)
E           Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```

And the logs show that the outgoing HTLC was sent to channeld before it
realized the connection was closed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-12 23:03:47 +02:00
Christian Decker 6cfb72ea1b keysend: Add error when trying to send to self
There is little point in faking a self-payment, but we should also not
crash :-)

Fixes #4438

Changelog-Fixed: keysend: Keysend returns an error when a self-payment is requested
2021-03-19 10:18:42 +10:30
Christian Decker ea92466025 pay: Abort the payment if destination is unreachable
We were not aborting if we had routehints, even though all routehints
may have been filtered out.

Changelog-Fixed: pay: `pay` will now abort early if the destination is not reachable directly nor via routehints.
2021-03-11 14:09:51 +10:30
Christian Decker cb21a5384b pay: Remove routehints with unreachable entrypoints
Changelog-Added: pay: `pay` will now remove routehints that are unusable due to the entrypoint being unknown or unreachable.
2021-03-11 14:09:51 +10:30
Christian Decker 1e1d7b387c pay: Filter out routehints whose entrypoint is unknown
We would happily spin on attempts that are doomed to fail because we
don't know the entrypoint. Next up: remove routehints whose
entrypoints are known but unreachable.
2021-03-11 14:09:51 +10:30
Christian Decker ad7f59f7a1 keysend: Set an error message if we abort due to missing TLV 2021-03-11 14:09:51 +10:30
Rusty Russell f0fa5d1401 offers: handle re-fetching the same invoice twice.
We get a label clash: easy, just re-serve:

```
2021-02-18T04:29:37.474Z **BROKEN** plugin-offers: Failed invoice_request lnr1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcyyqwtp0rmsgquvuacqcl5cdfzwzmu3v8tqgvpqs8e80dlmxm7ey4xwrqdsqqqqqqqqqqqqqqqq2pqqfqpqynzqzx9rylzy40ernj4jzc3p2dwy3n8x6lqeaywwk725ghx4kx63pcfxgg2z3nsn80jzge06nt3ks8pr6rvnujq48376lpmrr3cq04nurpy783eyr0awh5773lrlmjek07rjf0nx4g9235ulkcs7jp2h5gumjyquhadh846da3jptxm9g0qz5lne4hjhag for offer 1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67: Got JSON error: {\"code\":900,\"message\":\"Duplicate label\",\"data\":{\"label\":\"1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67-08c5193e2255f91ce5590b110a9ae2466736be0cf48e75bcaa22e6ad8da88709-1\",\"bolt12\":\"lni1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcyyqwtp0rmsgquvuacqcl5cdfzwzmu3v8tqgvpqs8e80dlmxm7ey4xwzqrw4lauzsc2ajk26mv0ysxxmmxvejk2grxdaezqun4wd68jggvpkqqqqqqqqqqqqqqqqpgyqq7ypymf9efe2jj5r2mzunlqz67d75ht3ukxk0x9ftkcuknrgepsgupwfqpqynzqzx9rylzy40ernj4jzc3p2dwy3n8x6lqeaywwk725ghx4kx63pcf9qzxqt0dxq4zqwtz2qu44gzx7nzczc494cce2tgph5xgu5sn7vh8frky9z5n08xj9sp3yaxe9cqs5vss59r8pxwlyy3jl4xhrdqwz85xe9qqgcpda590qs9khxdx5qpetlx0j6ap0wsxagssmy2qjvhjp2kc3na54pht3pp76c405upne360lh8rzye32xxq6l0phpkk9pu9lwxnqkxuwt2nqqr9u\",\"payment_hash\":\"396250395aa046f4c58162a5ae31952d01bd0c8e5213f32e748ec428a9379cd2\",\"msatoshi\":7700446,\"amount_msat\":\"7700446msat\",\"status\":\"unpaid\",\"description\":\"Weekly coffee for rusty!\",\"expires_at\":1614832137,\"local_offer_id\":\"1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67\"}}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
Rusty Russell 874ca99c32 offers: make 'used' flag more useful.
We used to only set it for single-use offers (where it's required),
but it's still interesting for multi-use offers, so let's keep it
there.

We also put this field in the documentation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
Rusty Russell c397efee1c pytest: fix pay flake.
We weren't waiting for the channel to get to normal state:

E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt20m1pszfpezpp549um3vuyt52rgea32g7u55a5fv29yv7t94cmr2f0vjgcc33m3dvqdqzdgxqyjw5qcqp9sp59vu4tat2n53ylzrgxa95s5wu2s885a9llud64c0f6gjqts7h6tks9qy9qsq7u0j3w2h3sxxp9axpjxkz525znjn0t92gnrgk7y6plyq39zw9994g88xxjx52egk4965dp5qt2w08hk009eq9hm8nykwmxe7r705k8gpkqu9mw'}, error: {'code': 210, 'message': 'Ran out of routes to try after 1 attempt: see `paystatus`', 'attempts': [{'status': 'failed', 'failreason': 'Cannot attempt payment, we have no channel to which we can add an HTLC', 'partid': 1, 'amount': '2000000000msat'}]}

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-11 14:33:51 +01:00
Christian Decker 9869083a79 plugin: Check that we have a non-empty route in libplugin
Changelog-Fixed: pay: Payments with an empty route (self-payment) are now aborted.
2021-02-10 15:17:38 +01:00
Christian Decker d2281f53df pytest: Reproduce #4345 2021-02-10 15:17:38 +01:00
Rusty Russell f3159ec4ac pytest: detect warnings, too.
Since we turned many errors into warnings, we want our tests to fail
when they happen unexpectedly.  We make WARNING clear in the strings
we print, too, to help out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:52 +10:30
Rusty Russell f4ee41a989 common: remove peer_failed in favor of peer_failed_warn/peer_failed_err
And make all the callers choose which one.  In general, I prefer warn,
which lets them reconnect and try again, however some places are either
stated that they must be errors in the spec itself, or in openingd
where we abandon the channel when we close the connection anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we now send warning messages and close the connection, except on unrecoverable errors.
2021-02-04 12:02:52 +10:30
Rusty Russell f0659d0ab0 pytest: fix false bad gossip issue in test_forward
We construct the route manually so we may not have the
channel_announcement yet.  But we can get an update from the error
packet, which can lead to:

```
2021-01-29T01:38:23.4767334Z            ValueError: 
2021-01-29T01:38:23.4767987Z            Node errors:
2021-01-29T01:38:23.4768767Z             - lightningd-1: had bad gossip messages
2021-01-29T01:38:23.4769512Z            Global errors:
2021-01-29T01:38:23.4770300Z 
2021-01-29T01:38:23.4771109Z contrib/pyln-testing/pyln/testing/fixtures.py:197: ValueError
...
2021-01-29T01:38:23.7820197Z lightningd-1: 2021-01-29T01:26:57.460Z DEBUG   gossipd: Extracted channel_update 01027217b3086ad9f3dee1fa55b94c5fd2a4b0637bec70ba727ba4151a8de5173ddc749db3502d41ab0ae164addc8fd013d2088b6a12a2f478ae0affa94d76d8845c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000067000001000160136459010000060000000000000000000000010000000a000000003b023380 from onionreply 100d0000007500887217b3086ad9f3dee1fa55b94c5fd2a4b0637bec70ba727ba4151a8de5173ddc749db3502d41ab0ae164addc8fd013d2088b6a12a2f478ae0affa94d76d8845c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000067000001000160136459010000060000000000000000000000010000000a000000003b023380
2021-01-29T01:38:23.7837450Z lightningd-1: 2021-01-29T01:26:57.461Z DEBUG   gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x1/0
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:52 +10:30
Rusty Russell 1a85edd207 lightningd: no longer forward failures to gossipd, let caller do it.
We fix up the test by using pay, instead of sendpay (and making pay log
the expected message).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: sendpay no longer extracts updates from errors, the caller should do it from the `raw_message`.
2021-02-02 13:44:01 +01:00
Christian Decker 62cb1c3fbc pytest: Stabilize test_forward_stats 2021-01-29 10:29:09 +10:30
Christian Decker 8cc62d76e4 pytest: Stabilize test_channel_{spendable,receivable}
They were using TIMEOUT / 2 which may be way too long (hit against
test timeout), so we use a still ludicrous 30 seconds instead.
2021-01-29 10:29:09 +10:30
Christian Decker 07f5054700 pytest: Stabilize test_setchannelfee_state
Synching with the blockchain was slower than our timeout...
2021-01-29 10:29:09 +10:30
Rusty Russell 001b5d6416 offers: make it a runtime option.
The fetchinvoice and offers plugins disable themselves if the option
isn't enabled (it's enabled by default on EXPERIMENTAL_FEATURES).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `experimental-offers` enables fetch, payment and creation of (early draft) offers.
2021-01-13 14:45:36 +01:00
niftynei 3e330d7186 df-tests: test_forward_local_failed_stats 2021-01-10 13:44:04 +01:00
niftynei 973224dd14 df-tests: if we're using fundbalancechannel, set up plugin
nodes need to be using the accepter plugin, so they'll match funding.
we used to use push_msat for this, but v2 gets rid of it.

*sad trombone*
2021-01-10 13:44:04 +01:00
niftynei e964c31369 df-tests: test_payment_success|failed_persistence 2021-01-10 13:44:04 +01:00
niftynei f4b7904ce5 tests: don't enforce daemon name on opening logline 2021-01-10 13:44:04 +01:00
Rusty Russell af46a4f57d fetchinvoice: allow amounts to be specified.
As per lastest revision of the spec, we can specify amounts in invoice
requests even if the offer already specifies it, as long as we exceed
the amount given.  This allows for tipping, and amount obfuscation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 4bb05e46e9 offers: convert currency when they request an invoice.
Means a reshuffle of our logic: we want to multiply by quantity before
conversion for maximum accuracy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 9681d491df offer: allow offers in other currencies if we can convert.
This avoids a footgun where they create an offer then we can't create
the invoice because they don't have a converter plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell d867fab3a7 sendinvoice: call waitinvoice directly to on invoice payment.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 1aa7e8e011 offers: split offer send_invoice generation into new JSON command
We split `send_invoice` offers inoo offerout (for want of a better name).
This simplifies the API.

Also took the opportunity to move the `vendor` tag to immediately
follow `description` (our tests use arguments by keywords, so no
change there).

Suggested-by: shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 55983827b6 fetchinvoice: try direct peers if we can't route.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell bdfcec3eb6 fetchinvoice: check we're in the period before attempting to fetch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell dae477175c fetchinvoice: implement timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 075c25fc08 plugins/fetchinvoice: handle sendinvoice timeout, error or payment.
If they pay the invoice, they don't bother replying; that's just for
errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 43f5304539 plugins/offer: allow `offer` to create `send_invoice`, `refund` and `single_use` offers.
refund offers are implied send_invoice offers.  And send_invoice offers
are implied single-use offers, though it can also make sense to have
a non-send_invoice offer be single-use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Christian Decker 6b6e0fe34c pytest: Mark `test_lockup_drain` as disabled
It's been causing me headaches for some time, and it's just getting
annoying. Will file an issue to track progress.
2021-01-08 19:28:30 +01:00
Christian Decker 4b2efd69d7 pay: Fix a crash when waitblockheight times out
Fixes #4309
Changelog-Fixed: pay: Fixed an issue where waiting for the blockchain height to sync could time out.
2021-01-08 09:19:24 +01:00
Christian Decker ca204eacfa pytest: Reproduce waitblockheight timeout issue #4309 2021-01-08 09:19:24 +01:00
Rusty Russell 52af729641 plugins/offer and plugins/fetchinvoice: send and recv errors.
This also lets us extend our testing to cover error cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 96caf9f4ab fetchinvoice: return the next period for recurring offers.
This is useful for the caller to know when to call again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 19af1d10e6 invoice: hack in merkle of invoice as "payment_secret" (EXPERIMENTAL_FEATURES)
This lets actually pay the invoice that fetchinvoice returns.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 2baa410e2b pytest: finally test fetchinvoice send, and offer plugin receive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 08e110b568 JSON: offers plugin `offer` command.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 723c16072a cleanups: feedback from Christian Decker review.
1. Hoist 7200 constant into the bolt12 heade2.
2. Make preimage the last createinvoice arg, so we could make it optional.
3. Check the validity of the preimage in createinvoice.
4. Always output used flag in listoffers.
5. Rename wallet offer iterators to offer_id iterators.
6. Fix paramter typos.
7. Rename `local_offer_id` parameter to `localofferid`.
8. Add reference constraints on local_offer_id db fields.
9. Remove cut/paste comment.
10. Clarify source of fatal() messages in wallet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-15 12:05:02 +01:00
Christian Decker f7cdf1dd98 pytest: Debug test_lockup_drain 2020-12-11 16:55:55 -06:00
Christian Decker 5a69b94f9a pytest: Add test for keysend feature support 2020-12-11 15:50:09 +01:00
Christian Decker 3b599b846e keysend: Check that the destination supports keysend upon init
We were blindly initiating the keysend payment, which could lead to
confusing outcomes. This adds a very specific error message to the
error returned.

Changelog-Fixed: keysend: Keysend now checks whether the destination supports keysend before attempting a payment. If not a more informative error is returned.
2020-12-11 15:50:09 +01:00
Michael Schmoock ce1a2f134a pytest: tests fee_base and fee_ppm on listpeers 2020-12-07 14:26:36 +01:00
Rusty Russell ae4139ba10 lightningd: note whether payment secret is used or not.
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets.  We need to know this so we know when
we can make them compulsory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-28 12:03:14 +01:00
niftynei de34f08b82 tests,fundchannel: return the result from the fundchannel rpc call
We need this so we can verify the 'close_to' result
2020-10-16 13:51:57 +10:30
Christian Decker 81569de709 pytest: Stabilize test_setchannelfee_{restart,zero}
In both cases the flakyness arises from the destination not knowing about the
modified fees of the forwarding node, thus including the outdated details in
the routehint, and the sender being unlucky and always trying with the
routehint anyway.

The long-term solutions to this is going to be #4111, this commit just reduces
the flakyness to get back to business.
2020-10-07 09:36:30 +10:30
Jan Sarenik a777d21fb7 pytest: Rename all fund_channel to fundchannel 2020-09-25 21:04:56 +02:00
Vincenzo Palazzo f62d7bbe45 Added additional fix to code formatting and English spelling.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2020-09-10 17:27:51 +09:30
Vincenzo Palazzo 9fbeb9bcdc delpay: code style changes and fixed docs
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2020-09-10 17:27:51 +09:30
ZmnSCPxj jxPCSnmZ 0eb1e7e0ca plugins/libplugin-pay.c: Add new payee_incoming_limit to limit number of HTLCs based on payee connectivity.
Fixes: #3926

(probably)

Changelog-Fixed: pay: Also limit the number of splits if the payee seems to have a low number of channels that can enter it, given the max-concurrent-htlcs limit.
2020-09-10 16:50:52 +09:30
ZmnSCPxj jxPCSnmZ 2d86014e53 tests/test_pay.py: Test for overload of number of HTLCs at the payee. 2020-09-10 16:50:52 +09:30
Rusty Russell 62f80f74ec pytest: fix flake in test_mpp_interference_2
I screwed up the rotation logic in an earlier varient of this PR, and
it lead me to discover why test_mpp_interference_2 was flaky.

Really, we should keep a fuzzy estimator of how much payment is
outstanding, but in practice rotation is probably good enough.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 19:47:11 +09:30
Rusty Russell 39e4796ae3 json_command: command_fail_badparam helper.
It's common to want to complain a token is not what we expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 74e960c20b pytest: increase channel sizes so they reach min capacity with option_anchor_outputs
With a feerate of 7500perkw and subtracting 660 sats for anchors, a
20,000 sat channel has capacity about 9800 sat, below our default:

	You gave bad parameters: channel capacity with funding 20000sat, reserves 546sat/546sat, max_htlc_value_in_flight_msat is 18446744073709551615msat, channel capacity is 9818sat, which is below 10000000msat

So bump channel amounts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Christian Decker 94d4d20d5f pytest: Stabilize test_mpp_interference_2 2020-08-30 20:03:42 +02:00
Christian Decker e420bfd6a8 pytest: Skip test_lockup_drain if VALGRIND and not DEVELOPER
The test was failing very reliably, by not raising the expected exception. My
best guess is that it needs more synchronization to cause the issue.
2020-08-30 20:03:42 +02:00
Rusty Russell 4a9c162211 pytest: disable test_mpp_interference_2.
We're failing this too often: we'd fail it more but it's disabled
with VALGRIND (it shouldn't be: @slow_test removes VALGRIND if SLOW_MACHINE
is set).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-27 13:51:22 +02:00
Christian Decker 21d87f7075 pay: Implement simple presplit fix for ludicrous amounts
This is the simplest possible fix: increase the target amount until we get
the desired number of parts, while still bucketizing payments together that
are in approximately the same size.

The current logic puts all payments that are in the range x < amount <= 16*x
in the same bucket, making them harder to distinguish.

Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-27 10:19:21 +09:30
Christian Decker 3f6809c934 pytest: Reproduce ludicrous presplit issue
Huge payment + presplit modifier = too many HTLCs...
2020-08-27 10:19:21 +09:30
Christian Decker 8f028272f4 pytest: Stabilize test_pay_get_error_with_update 2020-08-27 09:45:35 +09:30
ZmnSCPxj jxPCSnmZ ba3f38077e lightningd/invoice.c: Use round-robin channel selection.
Changelog-Changed: We now make MPP-aware routehints in invoices.
2020-08-26 09:29:46 +09:30
ZmnSCPxj jxPCSnmZ f50951a0d6 tests/test_pay.py: Motivating scenario for round-robin routehinting. 2020-08-26 09:29:46 +09:30
Rusty Russell cbd0be5c0e EXPERIMENTAL_FEATURES: enable option_anchor_outputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 8cf2fff91b pytest: generalize tests.
Anchor outputs break many assumptions in our tests:

1. Remove some hardcoded numbers in favor of a fee calc, so we only have to
   change in one place.
   FIXME: This should also be done for elements!
2. Do binary search to get feerate for a given closing fee.
3. Don't assume output #0: anchor outputs perturb them.
4. Don't assume we can make 1ksat channels (anchors cost 660 sats!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Christian Decker 8769f9ed93 pay: Fix final TLV payload if not going through MPP modifiers
Reported-by: ZmnSCPxj
Signed-off-by: Christian Decker <@cdecker>

Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
2020-08-13 12:50:16 +02:00
ZmnSCPxj jxPCSnmZ 29df517966 tests/test_pay.py: Provide test showing that blockheight disagreement causes us to advance routehints too aggressively.
The worst effect is that unpublished nodes are harder to pay, but
even published ones make us do unnecessary work, since we are
losing routehints from the published ones that could help us
actually route better to them.
2020-08-13 12:50:16 +02:00
Christian Decker 0dcd974d97 pytest: Reproduce #3915 2020-08-12 19:10:48 +02:00
Vincenzo Palazzo 1521c29fcf listpays mod 1: add destination inside the response when bolt11 is null
Changelog-Added: JSON-RPC: `listpays` now lists the `destination` if it was provided (e.g., via the `pay` plugin or `keysend` plugin)
2020-08-09 16:03:03 +02:00
Vincent 700897f06e listpays: fixed bolt11 null with keysend and update doc command
listpays: make doc-all missed
Changelog-Added: JSON-RPC: `listpays` can be used to query payments using the `payment_hash`
Changelog-Added: JSON-RPC: `listpays` now includes the `payment_hash`
2020-08-09 16:03:03 +02:00
Christian Decker 723b7223b7 pay: Add timestamp of first part to `listpays`
Changelog-Added: JSON-RPC: The result returned by `listpays` now includes the timestamp of the first part of the payment
2020-08-08 12:53:12 +02:00
Rusty Russell 01a82d38f7 pytest: add slow_test marker.
And when it's set, and we're SLOW_MACHINE, simply disable valgrind.

Since Travis (SLOW_MACHINE=1) only does VALGRIND=1 DEVELOPER=1 tests,
and VALGRIND=0 DEVELOPER=0 tests, it was missing tests which needed
DEVELOPER and !VALGRIND.

Instead, this demotes them to non-valgrind tests for SLOW_MACHINEs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell fde353ab00 pytest: use get_nodes more widely.
I started replacing all get_node() calls, but got bored, so then just did the
tests which call get_node() 3 times or more.

Ends up not making a measurable speed difference, but it does make some
things neater and more standard.

Times with SLOW_MACHINE=1 (given that's how Travis tests):

Time before (non-valgrind):
	393 sec (had 3 failures?)
Time after (non-valgrind):
	410 sec

Time before (valgrind):
	890 seconds (had 2 failures)
Time after (valgrind):
	892 sec

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
ZmnSCPxj jxPCSnmZ 4fde45669c plugins/pay.c: Fix the routehints/presplit conflict.
Changelog-Fixed: pay: Fixed a bug where routehints would be ignored if the payment exceeded 10,000 satoshi. This is particularly bad if the payee is only reachable via routehints in an invoice.
2020-08-06 17:36:58 +02:00
ZmnSCPxj jxPCSnmZ a9b992a94a tests/test_pay.py: Add test for conflict between presplit and routehints paymods. 2020-08-06 17:36:58 +02:00
Rusty Russell 4517435810 pytest: fix flake in tests/test_pay.py::test_pay_exclude_node
```
        # Excludes channel, then ignores routehint which includes that, then
        # it excludes other channel.
>       assert len(status) == 2
E       assert 1 == 2
E         -1
E         +2
```

The invoice we use at the end has a routehint: 50% of the time it's
to l2 (which fails), 50% to l5 (which succeeds).

Change it to create invoice before channel with l5 so it does the
retry like we expect here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-29 11:42:09 +02:00
Christian Decker 311594b2cd jsonrpc: Add amount received by recipient to `listpays` result
We sum up the amounts just like we do with `amount_sent`, however we may have
parts that don't have that annotation, so make it optional.

Suggested-by: Rusty Russell <@rustyrussell>
2020-07-28 16:17:39 +09:30
Christian Decker d12ea51e63 pytest: Replicate the missing amount_msat in `listpays`
Since we started using `sendonion` in the `pay` plugin we no longer
automatically have the `amount` annotation on (partial) payments. This
replicates the issue so we can fix it.

Reported-by: Rusty Russell <@rustyrussell>
2020-07-28 16:17:39 +09:30
Vincent 81fd552e84 plugins/pay: hand bolt11 arg to sendonion if we have one (i.e. for `pay`)
[ Extracted into standalone patch and comment added by RR ]
2020-07-27 13:11:14 +02:00
Vincent f4f2853ded pytest: test to show that listpays doesn't remember bolt11 string.
[Separated into separate commit by RR]
2020-07-27 13:11:14 +02:00
Christian Decker 67e846fdbb pytest: Adjust test now that routehint order changed 2020-07-24 11:35:49 +02:00
Christian Decker 899a2e64b0 paymod: Randomly select a routehint, or none at random
The adaptive MPP test was showing an issue with always using a routehint, even
when it wasn't necessary: we would insist on routhing to the entrypoint of the
routehint, even through the actual destination. If a channel on that loop
would result being over capacity we'd slam below 0, and then increase again by
unapplying the route. The solution really is not to insist on routing through
a routehint, so we implement random skipping of routehints, and we rotate them
if we have multiples.
2020-07-23 10:14:21 +09:30
Vincent 54888d454b Fixed assertion in pay plugin
This PR includes the fix discussed on PR #3855. This fix was tested with the use case described inside the issue and worked.

Fixes: #3855

Changelog-None
2020-07-23 10:14:21 +09:30
ZmnSCPxj jxPCSnmZ 9b4fd11e98 tests/test_pay.py: Replicate #3855. 2020-07-23 10:14:21 +09:30
Rusty Russell 6951a9ea42 pytest: fix erroneous test_pay_retry result.
Seems like we get *4* failures, since failing to find a route counts now?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-23 10:14:21 +09:30
Rusty Russell 09eb7110e0 sendpay: insist that partid be an exact duplicate if in progress.
The test had part 1 and 2 backward, but still worked.  When I copied that to
*after* the test had succeeded, it complained.  It should always complain,
to catch bugs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-21 13:30:15 +02:00
Rusty Russell 73d5d96d2a sendpay: don't allow a new part payment if any part has succeeded.
This wasn't important before, but now we have MPP it's good to enforce.

Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-21 13:30:15 +02:00
Christian Decker de096eeed9 pytest: Disable test_pay_routeboost for valgrind due to timeout 2020-07-15 11:32:58 +02:00
Christian Decker de90606490 pytest: Add an adaptive MPP test
This exercises something that is simply not possible without MPP, i.e., the
bundling of multiple paths to get sufficient capacity to perform the payment.
2020-07-15 11:32:58 +02:00
Christian Decker dad23066dc pytest: Disable MPP sending for 4 tests
These mostly deal with exact HTLC counts, and fixed number of attempts to
conclusion, so the randomization that MPP adds is not desirable.
2020-07-15 11:32:58 +02:00
Christian Decker b813974e13 mpp: Add the presplit MPP modifier
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
2020-07-15 11:32:58 +02:00
Christian Decker 02e44e9903 paymod: Set the STOPPED_RETRYING status code if we stopped retrying 2020-07-14 06:25:29 +09:30
Christian Decker 5776a33116 paymod: Activate paymod and move legacy pay to `legacypay` command
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.

Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.

Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
2020-07-14 06:25:29 +09:30
Christian Decker 01a475161d paymod: Implement keysend sending support as a native RPC command
This makes use of the payment modifier structure to just add the preimage to
the TLV payload for the last hop.

Changelog-Added: JSON-RPC: The `keysend` command allows sending to a node without requiring an invoice first.
2020-07-13 14:19:16 +02:00
Christian Decker 0aa5c197ec pytest: Fix tests broken by the pay and paystatus changes
This commit collects the changes required to the tests caused by the changes
to the `pay` and `paystatus` commands. They are also rather good hints as to
what these changes entail.
2020-07-07 23:25:45 +02:00
Christian Decker 2ac87c1490 paymod: Reorder paymod arguments to match pay
We've been adding modifiers and arguments out of order, and we need the
arguments order to match up if we want `paymod` to be a drop-in replacement.
2020-07-07 23:25:45 +02:00
Christian Decker da8eb6fb4c paymod: Expose riskfactor and wire through to getroute 2020-07-07 23:25:45 +02:00
Christian Decker 5601a1ab21 paymod: Add a deadline to the pay command and retry modifier 2020-07-07 23:25:45 +02:00
Christian Decker f557955515 paymod: Consolidate parsing RPC results in libplugin
We handle these in a number of different ways, and regularly get the parsing
and logic for optional fields wrong, so let's consolidate them here.
2020-07-07 23:25:45 +02:00
Christian Decker cb00cbac7c paymod: Add support for the msatoshi override argument 2020-07-07 23:25:45 +02:00
Christian Decker 2331cd62e1 paymod: Add user-provided label back into the paystatus result 2020-07-07 23:25:45 +02:00
Christian Decker 935578e567 paymod: Exclude most expensive/slowest chan if limits are exceeded 2020-07-07 23:25:45 +02:00
Christian Decker 4648d2867f paymod: Compute fee and CLTV delta limits for a payment
So far we got away with not caring about these but since we're implementing
modifiers that impact these limits, we better keep track of them.
2020-07-07 23:25:45 +02:00
Christian Decker 2f0e535b81 paymod: Add reason why a payment was attempted
This is a slight change in interface in that the string no longer spells out
that a specific node was excluded.
2020-07-07 23:25:45 +02:00
Rusty Russell 3fec96a7b1 tests: fix flake in test_partial_payment_htlc_loss
Make sure we've actually confirmed the HTLC; if it's not confirmed yet
then we won't fast-fail it, and we'll timeout instead:

```
>           l1.rpc.waitsendpay(payment_hash=inv['payment_hash'], timeout=TIMEOUT, partid=1)
E           AssertionError: Pattern 'WIRE_PERMANENT_CHANNEL_FAILURE \\(reply from remote\\)' not found in "RPC call failed: method: waitsendpay, payload: {'payment_hash': 'c186643391469aa8190415496c85b1eb789cb2b756a76d4c9ce21dd34c698d92', 'timeout': 30, 'partid': 1}, error: {'code': 200, 'message': 'Timed out while waiting'}"
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-07 17:02:21 +09:30
Christian Decker ddb0424a4d paymod: Remove the dummy payment modifier
It was there only for demonstration purposes, and is no longer useful.
2020-07-02 11:54:12 +02:00
Christian Decker 81dce4096e pytest: Fix up test_pay_no_secret to wait for node_announcements
This was racy since we didn't know whether the peer supports TLV payloads yet
so we defaulted to legacy, which doesn't support secrets.
2020-07-02 11:54:12 +02:00
Christian Decker 04c9c5e0d1 paymod: Collect and return results of a tree of partial payments
The status of what started as a simple JSON-RPC call is now spread across an
entire tree of partial payments and payment attempts. So we collect the status
in a single struct in order to report back success of failure.
2020-07-01 12:19:02 +02:00
Christian Decker 23b4dca3c7 paymod: Add a simple test-command to test the paymod state-machine
This commit can be reverted/skipped once we have implemented all the logic and
have feature parity with the normal `pay`. It's main purpose is to expose the
unfinished functionality to test it, without completely breaking the existing
`pay` command.
2020-07-01 12:19:02 +02:00
ZmnSCPxj jxPCSnmZ c100de6d93 common/utils.c: Correctly handle NULL `take`n pointer in `tal_dup_talarr`.
Fixes: #3757

Reported-by: @sumBTC

Changelog-None
2020-06-23 12:48:20 +09:30
ZmnSCPxj jxPCSnmZ a25af262e9 tests/test_pay.py: Add test to replicate #3757. 2020-06-23 12:48:20 +09:30
Rusty Russell 1fb505d118 pytest: extend timeout for test_payment_duplicate_uncommitted
We've been seeing some Travis timeouts under VALGRIND, with the
10 second timeout here: use TIMEOUT as per standard.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-12 21:57:43 +02:00
Antoine Poinsot 4bb7b4621f pay: respect maxfeepercent when choosing a shadow route
And the percentage of the initial amount, not the constently increasing
one !

Changelog-Fixed: pay: we now respect maxfeepercent, even for tiny amounts.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-05 14:10:42 +09:30
Antoine Poinsot f83a1e0f2d Reproduce issue #3684
Fixed-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-05 14:10:42 +09:30
lisa neigut 0e20e3c5e7 df: rename 'funder' to 'opener'
Previously we've used the term 'funder' to refer to the peer
paying the fees for a transaction; v2 of openchannel will make
this no longer true. Instead we rename this to 'opener', or the
peer sending the 'open_channel' message, since this will be universally
true in a dual-funding world.
2020-05-04 10:22:26 +09:30
Christian Decker 9f298330b2 libplugin: It's featurebits, not features
The documentation was wrong, and I copied my mistake to `libplugin` where it
was then ignored instead of ORed into the node's featurebits. This fixes both.
2020-04-28 10:39:07 -05:00
Christian Decker 8c23f22591 pytest: Reproduce issue #3666
It seems we aren't registering the featurebits correctly.
2020-04-28 10:39:07 -05:00
Christian Decker 568773daad pytest: Add a test for keysend
It currently uses a borrowed sending implementation from the noise plugin, but
we'll implement that functionality in the native keysend plugin next.
2020-04-16 18:03:35 +09:30
Rusty Russell 284bd7de0c pytest: fix flaky test_excluded_adjacent_routehint
We weren't actually waiting until l3 got the channel_update from l2,
so it might not be able to create the routehint.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-15 20:52:13 -05:00
Rusty Russell 1dc281c111 pytest: test for pay adjacent routehint crash.
When route returns a result which is too expensive, we try to figure out which
hop is most expensive to exclude it for next time.

If it's a single-hop route, we don't count it, since the first hop is free.
That's not usually a problem, since single-hop routes can't exceed our limits
(they're always "free"!).

But if we are using a routehint, the total cost could exceed our limits,
even if the start of the routehint is a single hop away.

This reproduces that test case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 14:13:26 -05:00
Rusty Russell 1d29228136 lightningd: make sendpay support `blinding` and `enctlv` fields (EXPERIMENTAL)
This is what actually lets us pay blinded invoices.

Unfortunately, our internal logic assumes every hop in a path has a
next `short_channel_id`, so we have to use a dummy.  This is
sufficient for testing, however.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 12:51:18 +09:30
Rusty Russell bf2a42ada5 channeld: defer first update_fee until we have an HTLC to send.
Sending update_fee immediately after channel establishment seems to
upset LND, so work around it by deferring it.  The reason we increase
the fee after establishment is because now we might need to close the
channel in a hurry due to htlcs, but until there are htlcs that's
unnecessary.

Fixes: #3596
Changelog-Changed: Added workaround for lnd rejecting our commitment_signed when we send an update_fee after channel confirmed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-13 20:58:06 -05:00
Christian Decker 8370a627bb invoice: Parse amounts without a multiplier as BTC not msats
The spec states that invoices with an amount, but lacking a multiplier, should
be interpreted as integer Bitcoin amounts:

   `amount`: optional number in that currency, followed by an optional
   `multiplier` letter. The unit encoded here is the 'social' convention of a
   payment unit -- in the case of Bitcoin the unit is 'bitcoin' NOT satoshis.

Suggested-by: Stefano Pellegrini <@St333p>
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: invoice: The invoice parser assumed that an amount without a multiplier was denominated in msatoshi instead of bitcoins.
2020-04-11 08:43:06 +09:30
darosior 7c0af81c21 bcli: use a more urgent feerate for HTLCs and penalty transactions
A CONSERVATIVE/3 target for them.

Some noisy changes to the tests as we had to update the estimatesmartfee
mock.

Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
2020-04-01 23:02:47 -05:00
Michael Schmoock 0998164214 test: receivable_msat similar to spendable_msat 2020-03-27 12:33:23 +01:00
Michael Schmoock 80ff9c5b63 test: refactor lockup_drain add helpers drain and force_feerates 2020-03-27 12:33:23 +01:00
Christian Decker 9736c7bfd0 pytest: Make test_pay_retry less flaky for liquid-regtest 2020-03-24 09:52:33 +10:30
Michael Schmoock af7e879308 fix: rfc #740 requires 100% feespike margin
Changelog-Fixed: Use lightning-rfc #740 feespike margin factor of 2
2020-03-19 18:56:12 +01:00
Christian Decker ac5cba2243 htlc: Return an invalid payload error if payload is invalid
We somehow were always returning `badonion` when really it is an invalid
payload inside the onion.
2020-03-04 22:57:24 +01:00
Christian Decker 9529529815 pytest: Repro wrong error being returned on invalid onion payloads
We are returning a `BADONION` error despite the cause being an invalid onion
payload containing an unknown even TLV type. It really should return
`INVALID_ONION_PAYLOAD` errors instead.
2020-03-04 22:57:24 +01:00
Rusty Russell 86c28b2272 channeld: channel drain mitigation.
Add new check if we're funder trying to add HTLC, keeping us
with enough extra funds to pay for another HTLC the peer might add.

We also need to adjust the spendable_msat calculation, and update
various tests which try to unbalance channels.  We eliminate
the now-redundant test_channel_drainage entirely.

Changelog-Fixed: Corner case where channel could become unusable (https://github.com/lightningnetwork/lightning-rfc/issues/728)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 15:34:05 +01:00
Rusty Russell 6efde31ade pytest: simplified drain test (xfail)
This is inspired by @m-schmook's https://github.com/ElementsProject/lightning/pull/3498
except this is simply a two-channel version which probes for the amount.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 15:34:05 +01:00
Michael Schmoock e92ab89087 test: uses wait_for_htlcs helper in test_pay 2020-02-10 12:14:19 +10:30
Rusty Russell f3600d22a0 lightningd: disallow `msatoshi` arg to sendpay unless exact when non-MPP.
Using it with a different value to the amount sent causes a crash in 0.8.0,
which is effectively deprecating it, so let's disallow it now.

Changelog-Changed: If the optional `msatoshi` param to sendpay for non-MPP is set, it must be the exact amount sent to the final recipient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-01-31 14:23:18 +01:00
Christian Decker 9038364c63 sphinx: Actually use the pad stream to generate the packet
We flipped two buffers and were not actually using the chacha20 stream.
2020-01-27 22:48:42 +01:00
darosior dc3aa33927 pytest: use pyln.client for functional tests 2020-01-21 22:38:14 +01:00
ZmnSCPxj a9f0f05eea pay: Implement retry in case of final CLTV being too soon for receiver.
Changelog-Fixed: Detect a previously non-permanent error (`final_cltv_too_soon`) that has been merged into a permanent error (`incorrect_or_unknown_payment_details`), and retry that failure case in `pay`.
2020-01-21 22:23:21 +01:00
ZmnSCPxj 72a24a2bdd test_pay.py: Add test for blockheight disagreement. 2020-01-21 22:23:21 +01:00
Christian Decker 5e44895264 sphinx: Check the payload size at construction and in createonion
Fixes #3377

Changelog-Fixed: JSON-RPC: The arguments for `createonion` are now checked to ensure they fit in the onion packet.
2020-01-10 21:10:42 +01:00
Christian Decker 0eaba5600a pytest: Reproduce issue #3377 2020-01-10 21:10:42 +01:00
Rusty Russell 1d0c433dc4 channeld: treat all incoming errors as "soft", so we retry.
We still close the channel if we *send* an error, but we seem to have hit
another case where LND sends an error which seems transient, so this will
make a best-effort attempt to preserve our channel in that case.

Some test have to be modified, since they don't terminate as they did
previously :(

Changelog-Changed: quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:18 +01:00
Rusty Russell 839909d2cf Protocol: make var_onion, payment_secret and basic_mpp non-EXPERIMENTAL.
Thanks to @t-bast, who made this possible by interop testing with Eclair!

Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:16:03 +01:00
Rusty Russell edab0df611 lightningd: fix spurious "more than twice final" error.
Bastien TEINTURIER <bastien@acinq.fr> writes:
> It looks like the split on c-lightning side is quite limited at the moment:
> the only option is to split a payment in exactly its two halves,
> otherwise I get rejected because of the rule of overpaying more than
> twice the amount?

We only tested exactly two equal-size payments; indeed, our finalhop
test was backwards.  We only complain if the final hop pays more than
twice msat (technically, this test is still too loose for mpp: the
spec says we should sum to the exact amount).

Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 326ceab8ad pytest: add more multi-part-payment tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 207689c274 plugins: listpays will now consolidate multi-part payments.
This won't usually be visible to the end-user, since the pay plugin doesn't
do multi-part yet (and mpp requires EXPERIMENTAL_FEATURES), but we're ready
once it does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 8e3234e67a lightningd: sew in htlc set.
The invoice_try_pay code now takes a set, rather than a single htlc, but
it's basically the same thing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00