Commit Graph

420 Commits

Author SHA1 Message Date
Christian Decker dad23066dc pytest: Disable MPP sending for 4 tests
These mostly deal with exact HTLC counts, and fixed number of attempts to
conclusion, so the randomization that MPP adds is not desirable.
2020-07-15 11:32:58 +02:00
Christian Decker b813974e13 mpp: Add the presplit MPP modifier
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
2020-07-15 11:32:58 +02:00
Christian Decker 02e44e9903 paymod: Set the STOPPED_RETRYING status code if we stopped retrying 2020-07-14 06:25:29 +09:30
Christian Decker 5776a33116 paymod: Activate paymod and move legacy pay to `legacypay` command
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.

Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.

Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
2020-07-14 06:25:29 +09:30
Christian Decker 01a475161d paymod: Implement keysend sending support as a native RPC command
This makes use of the payment modifier structure to just add the preimage to
the TLV payload for the last hop.

Changelog-Added: JSON-RPC: The `keysend` command allows sending to a node without requiring an invoice first.
2020-07-13 14:19:16 +02:00
Christian Decker 0aa5c197ec pytest: Fix tests broken by the pay and paystatus changes
This commit collects the changes required to the tests caused by the changes
to the `pay` and `paystatus` commands. They are also rather good hints as to
what these changes entail.
2020-07-07 23:25:45 +02:00
Christian Decker 2ac87c1490 paymod: Reorder paymod arguments to match pay
We've been adding modifiers and arguments out of order, and we need the
arguments order to match up if we want `paymod` to be a drop-in replacement.
2020-07-07 23:25:45 +02:00
Christian Decker da8eb6fb4c paymod: Expose riskfactor and wire through to getroute 2020-07-07 23:25:45 +02:00
Christian Decker 5601a1ab21 paymod: Add a deadline to the pay command and retry modifier 2020-07-07 23:25:45 +02:00
Christian Decker f557955515 paymod: Consolidate parsing RPC results in libplugin
We handle these in a number of different ways, and regularly get the parsing
and logic for optional fields wrong, so let's consolidate them here.
2020-07-07 23:25:45 +02:00
Christian Decker cb00cbac7c paymod: Add support for the msatoshi override argument 2020-07-07 23:25:45 +02:00
Christian Decker 2331cd62e1 paymod: Add user-provided label back into the paystatus result 2020-07-07 23:25:45 +02:00
Christian Decker 935578e567 paymod: Exclude most expensive/slowest chan if limits are exceeded 2020-07-07 23:25:45 +02:00
Christian Decker 4648d2867f paymod: Compute fee and CLTV delta limits for a payment
So far we got away with not caring about these but since we're implementing
modifiers that impact these limits, we better keep track of them.
2020-07-07 23:25:45 +02:00
Christian Decker 2f0e535b81 paymod: Add reason why a payment was attempted
This is a slight change in interface in that the string no longer spells out
that a specific node was excluded.
2020-07-07 23:25:45 +02:00
Rusty Russell 3fec96a7b1 tests: fix flake in test_partial_payment_htlc_loss
Make sure we've actually confirmed the HTLC; if it's not confirmed yet
then we won't fast-fail it, and we'll timeout instead:

```
>           l1.rpc.waitsendpay(payment_hash=inv['payment_hash'], timeout=TIMEOUT, partid=1)
E           AssertionError: Pattern 'WIRE_PERMANENT_CHANNEL_FAILURE \\(reply from remote\\)' not found in "RPC call failed: method: waitsendpay, payload: {'payment_hash': 'c186643391469aa8190415496c85b1eb789cb2b756a76d4c9ce21dd34c698d92', 'timeout': 30, 'partid': 1}, error: {'code': 200, 'message': 'Timed out while waiting'}"
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-07 17:02:21 +09:30
Christian Decker ddb0424a4d paymod: Remove the dummy payment modifier
It was there only for demonstration purposes, and is no longer useful.
2020-07-02 11:54:12 +02:00
Christian Decker 81dce4096e pytest: Fix up test_pay_no_secret to wait for node_announcements
This was racy since we didn't know whether the peer supports TLV payloads yet
so we defaulted to legacy, which doesn't support secrets.
2020-07-02 11:54:12 +02:00
Christian Decker 04c9c5e0d1 paymod: Collect and return results of a tree of partial payments
The status of what started as a simple JSON-RPC call is now spread across an
entire tree of partial payments and payment attempts. So we collect the status
in a single struct in order to report back success of failure.
2020-07-01 12:19:02 +02:00
Christian Decker 23b4dca3c7 paymod: Add a simple test-command to test the paymod state-machine
This commit can be reverted/skipped once we have implemented all the logic and
have feature parity with the normal `pay`. It's main purpose is to expose the
unfinished functionality to test it, without completely breaking the existing
`pay` command.
2020-07-01 12:19:02 +02:00
ZmnSCPxj jxPCSnmZ c100de6d93 common/utils.c: Correctly handle NULL `take`n pointer in `tal_dup_talarr`.
Fixes: #3757

Reported-by: @sumBTC

Changelog-None
2020-06-23 12:48:20 +09:30
ZmnSCPxj jxPCSnmZ a25af262e9 tests/test_pay.py: Add test to replicate #3757. 2020-06-23 12:48:20 +09:30
Rusty Russell 1fb505d118 pytest: extend timeout for test_payment_duplicate_uncommitted
We've been seeing some Travis timeouts under VALGRIND, with the
10 second timeout here: use TIMEOUT as per standard.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-12 21:57:43 +02:00
Antoine Poinsot 4bb7b4621f pay: respect maxfeepercent when choosing a shadow route
And the percentage of the initial amount, not the constently increasing
one !

Changelog-Fixed: pay: we now respect maxfeepercent, even for tiny amounts.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-05 14:10:42 +09:30
Antoine Poinsot f83a1e0f2d Reproduce issue #3684
Fixed-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-05 14:10:42 +09:30
lisa neigut 0e20e3c5e7 df: rename 'funder' to 'opener'
Previously we've used the term 'funder' to refer to the peer
paying the fees for a transaction; v2 of openchannel will make
this no longer true. Instead we rename this to 'opener', or the
peer sending the 'open_channel' message, since this will be universally
true in a dual-funding world.
2020-05-04 10:22:26 +09:30
Christian Decker 9f298330b2 libplugin: It's featurebits, not features
The documentation was wrong, and I copied my mistake to `libplugin` where it
was then ignored instead of ORed into the node's featurebits. This fixes both.
2020-04-28 10:39:07 -05:00
Christian Decker 8c23f22591 pytest: Reproduce issue #3666
It seems we aren't registering the featurebits correctly.
2020-04-28 10:39:07 -05:00
Christian Decker 568773daad pytest: Add a test for keysend
It currently uses a borrowed sending implementation from the noise plugin, but
we'll implement that functionality in the native keysend plugin next.
2020-04-16 18:03:35 +09:30
Rusty Russell 284bd7de0c pytest: fix flaky test_excluded_adjacent_routehint
We weren't actually waiting until l3 got the channel_update from l2,
so it might not be able to create the routehint.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-15 20:52:13 -05:00
Rusty Russell 1dc281c111 pytest: test for pay adjacent routehint crash.
When route returns a result which is too expensive, we try to figure out which
hop is most expensive to exclude it for next time.

If it's a single-hop route, we don't count it, since the first hop is free.
That's not usually a problem, since single-hop routes can't exceed our limits
(they're always "free"!).

But if we are using a routehint, the total cost could exceed our limits,
even if the start of the routehint is a single hop away.

This reproduces that test case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 14:13:26 -05:00
Rusty Russell 1d29228136 lightningd: make sendpay support `blinding` and `enctlv` fields (EXPERIMENTAL)
This is what actually lets us pay blinded invoices.

Unfortunately, our internal logic assumes every hop in a path has a
next `short_channel_id`, so we have to use a dummy.  This is
sufficient for testing, however.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 12:51:18 +09:30
Rusty Russell bf2a42ada5 channeld: defer first update_fee until we have an HTLC to send.
Sending update_fee immediately after channel establishment seems to
upset LND, so work around it by deferring it.  The reason we increase
the fee after establishment is because now we might need to close the
channel in a hurry due to htlcs, but until there are htlcs that's
unnecessary.

Fixes: #3596
Changelog-Changed: Added workaround for lnd rejecting our commitment_signed when we send an update_fee after channel confirmed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-13 20:58:06 -05:00
Christian Decker 8370a627bb invoice: Parse amounts without a multiplier as BTC not msats
The spec states that invoices with an amount, but lacking a multiplier, should
be interpreted as integer Bitcoin amounts:

   `amount`: optional number in that currency, followed by an optional
   `multiplier` letter. The unit encoded here is the 'social' convention of a
   payment unit -- in the case of Bitcoin the unit is 'bitcoin' NOT satoshis.

Suggested-by: Stefano Pellegrini <@St333p>
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: invoice: The invoice parser assumed that an amount without a multiplier was denominated in msatoshi instead of bitcoins.
2020-04-11 08:43:06 +09:30
darosior 7c0af81c21 bcli: use a more urgent feerate for HTLCs and penalty transactions
A CONSERVATIVE/3 target for them.

Some noisy changes to the tests as we had to update the estimatesmartfee
mock.

Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
2020-04-01 23:02:47 -05:00
Michael Schmoock 0998164214 test: receivable_msat similar to spendable_msat 2020-03-27 12:33:23 +01:00
Michael Schmoock 80ff9c5b63 test: refactor lockup_drain add helpers drain and force_feerates 2020-03-27 12:33:23 +01:00
Christian Decker 9736c7bfd0 pytest: Make test_pay_retry less flaky for liquid-regtest 2020-03-24 09:52:33 +10:30
Michael Schmoock af7e879308 fix: rfc #740 requires 100% feespike margin
Changelog-Fixed: Use lightning-rfc #740 feespike margin factor of 2
2020-03-19 18:56:12 +01:00
Christian Decker ac5cba2243 htlc: Return an invalid payload error if payload is invalid
We somehow were always returning `badonion` when really it is an invalid
payload inside the onion.
2020-03-04 22:57:24 +01:00
Christian Decker 9529529815 pytest: Repro wrong error being returned on invalid onion payloads
We are returning a `BADONION` error despite the cause being an invalid onion
payload containing an unknown even TLV type. It really should return
`INVALID_ONION_PAYLOAD` errors instead.
2020-03-04 22:57:24 +01:00
Rusty Russell 86c28b2272 channeld: channel drain mitigation.
Add new check if we're funder trying to add HTLC, keeping us
with enough extra funds to pay for another HTLC the peer might add.

We also need to adjust the spendable_msat calculation, and update
various tests which try to unbalance channels.  We eliminate
the now-redundant test_channel_drainage entirely.

Changelog-Fixed: Corner case where channel could become unusable (https://github.com/lightningnetwork/lightning-rfc/issues/728)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 15:34:05 +01:00
Rusty Russell 6efde31ade pytest: simplified drain test (xfail)
This is inspired by @m-schmook's https://github.com/ElementsProject/lightning/pull/3498
except this is simply a two-channel version which probes for the amount.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 15:34:05 +01:00
Michael Schmoock e92ab89087 test: uses wait_for_htlcs helper in test_pay 2020-02-10 12:14:19 +10:30
Rusty Russell f3600d22a0 lightningd: disallow `msatoshi` arg to sendpay unless exact when non-MPP.
Using it with a different value to the amount sent causes a crash in 0.8.0,
which is effectively deprecating it, so let's disallow it now.

Changelog-Changed: If the optional `msatoshi` param to sendpay for non-MPP is set, it must be the exact amount sent to the final recipient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-01-31 14:23:18 +01:00
Christian Decker 9038364c63 sphinx: Actually use the pad stream to generate the packet
We flipped two buffers and were not actually using the chacha20 stream.
2020-01-27 22:48:42 +01:00
darosior dc3aa33927 pytest: use pyln.client for functional tests 2020-01-21 22:38:14 +01:00
ZmnSCPxj a9f0f05eea pay: Implement retry in case of final CLTV being too soon for receiver.
Changelog-Fixed: Detect a previously non-permanent error (`final_cltv_too_soon`) that has been merged into a permanent error (`incorrect_or_unknown_payment_details`), and retry that failure case in `pay`.
2020-01-21 22:23:21 +01:00
ZmnSCPxj 72a24a2bdd test_pay.py: Add test for blockheight disagreement. 2020-01-21 22:23:21 +01:00
Christian Decker 5e44895264 sphinx: Check the payload size at construction and in createonion
Fixes #3377

Changelog-Fixed: JSON-RPC: The arguments for `createonion` are now checked to ensure they fit in the onion packet.
2020-01-10 21:10:42 +01:00
Christian Decker 0eaba5600a pytest: Reproduce issue #3377 2020-01-10 21:10:42 +01:00
Rusty Russell 1d0c433dc4 channeld: treat all incoming errors as "soft", so we retry.
We still close the channel if we *send* an error, but we seem to have hit
another case where LND sends an error which seems transient, so this will
make a best-effort attempt to preserve our channel in that case.

Some test have to be modified, since they don't terminate as they did
previously :(

Changelog-Changed: quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:18 +01:00
Rusty Russell 839909d2cf Protocol: make var_onion, payment_secret and basic_mpp non-EXPERIMENTAL.
Thanks to @t-bast, who made this possible by interop testing with Eclair!

Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:16:03 +01:00
Rusty Russell edab0df611 lightningd: fix spurious "more than twice final" error.
Bastien TEINTURIER <bastien@acinq.fr> writes:
> It looks like the split on c-lightning side is quite limited at the moment:
> the only option is to split a payment in exactly its two halves,
> otherwise I get rejected because of the rule of overpaying more than
> twice the amount?

We only tested exactly two equal-size payments; indeed, our finalhop
test was backwards.  We only complain if the final hop pays more than
twice msat (technically, this test is still too loose for mpp: the
spec says we should sum to the exact amount).

Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 326ceab8ad pytest: add more multi-part-payment tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 207689c274 plugins: listpays will now consolidate multi-part payments.
This won't usually be visible to the end-user, since the pay plugin doesn't
do multi-part yet (and mpp requires EXPERIMENTAL_FEATURES), but we're ready
once it does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 8e3234e67a lightningd: sew in htlc set.
The invoice_try_pay code now takes a set, rather than a single htlc, but
it's basically the same thing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 280523e72a pytest: Add tests to make sure received onion is as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell b670b51fa2 waitsendpay: add partid arg.
We need to be able to wait for a unique payment, now payment_hash is not
always unique.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 63fffd41c8 configure: make partid payments only available with EXPERIMENTAL_FEATURES and payment_secret
Explicit #if EXPERIMENTAL_FEATURES check in case we enable them at different
times, but it requires a payment_secret since we put them in the same field.

This incidently stops it working on legacy nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell cd35835c5a sendpay/sendonion: add optional partid arg, finesse msatoshi argument.
msatoshi was used to indicate the amount the invoice asked for, but
for parallel sendpay it's required, as it allows our sanity check of
limiting the total payments in flight, ie. it becomes
'total_msat'.

There's a special case for sendonion, which always tells us the value is 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell ce4403d638 lightningd: change amount-in-flight check to be more nuanced.
We currently refuse a payment if one is already in flight.  For parallel
payments, it's a bit more subtle: we want to refuse if it we already have
the total-amount-of-invoice in flight.

So we get all the current payments, and sum the pending ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Christian Decker 301b634699 pytest: Skip some tests in elements if paratrization is convoluted
These mainly use hard-coded addresses
2019-12-11 16:18:34 +01:00
Rusty Russell f7ebbb2ec5 common: make sphinx code ignorant of payload format.
Now "raw_payload" is always the complete string (including realm or length
bytes at the front).

This has several effects:
1. We can receive an decrypt an onion which is grossly malformed.
2. We can still hand this to the htlc_accepted hook.
3. We then fail it unless the htlc_accepted accepts it manually.
4. The createonion API now takes the raw payload, and does not know
   anything about "style".

The only caveat is that the sphinx code needs to know the payload
length: we have a call for that, which simply tells it to copy the
entire onion (and treat us as the final node) if it's invalid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-09 14:33:31 +01:00
fiatjaf c0368a717a also return preimage on listinvoices. 2019-12-03 01:16:07 +00:00
darosior d9aeeb4d14 pytest: skip test_sendonion if not DEVELOPER 2019-12-02 14:41:13 +01:00
Christian Decker 21b5b59c8c json-rpc: Rename onion payload type to style
Suggested-by: Rusty Russell <@rustyrussell>
2019-12-01 15:40:47 +01:00
Christian Decker 16b6c31010 json-rpc: Add the error onion if we stored it in the DB
If we can't decode the onion, because the onion got corrupted or we used
`sendonion` without specifying the `shared_secrets` used, the best we can do
is tell the caller instead.
2019-12-01 15:40:47 +01:00
Christian Decker 27547ce8d4 pay: Allow `sendonion` callers to provide `shared_secrets`
This means that c-lightning can now internally decrypt an eventual error
message, and not force the caller to implement the decryption. The main
difficulty was that we now have a new state (channels and nodes not specified,
while shared_secrets are specified) which needed to be handled.
2019-12-01 15:40:47 +01:00
Christian Decker b8ce175fd2 pay: Handle payment failures resulting from sendonion correctly
We are breaking with a couple of assumptions, namely that we have the
`path_secrets` to decode the error onion. If this happens we just want it to
error out.
2019-12-01 15:40:47 +01:00
Christian Decker cb912c9ae8 pytest: Add a test for the `sendonion` method 2019-12-01 15:40:47 +01:00
Christian Decker 8daf9b2577 pytest: Add a test for `createonion` 2019-12-01 15:40:47 +01:00
Rusty Russell 8393d21503 common/features: add `payment_secret` feature if EXPERIMENTAL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell e5247a68b6 lightningd: check payment secret on htlc receipt.
We don't set the secret to compulsory (yet!) but put code in for the
future.  Meanwhile, if there is a secret, check it is correct.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell ef7a820ab1 log: make formatting more consistent.
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
2019-11-18 04:50:22 +00:00
Rusty Russell 21555b277d tests: test that we only use tlv onion when advertized.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-14 10:15:33 +01:00
darosior 208123771e pytest: test shadow routing in pay plugin 2019-11-08 03:27:58 +00:00
darosior 7dac7a8c52 pytest: deactivate shadow routing for some tests that use 'pay'
So that we can assert payment values
2019-11-08 03:27:58 +00:00
Rusty Russell a70f2dc650 pytest: two more timeouts on Travis.
Reduce test_feerate_stress iterations, and simply don't run
test_pay_retry under VALGRIND with SLOW_MACHINE at all.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 2801d98e34 pytest: allow bad gossip in test_pay_direct.
Whenever we have multi-connected nodes, out-of-order gossip is possible.
In particular, if a node_announcement is 1 second fresher than the
channel_announcement, a timestamp_filter might get one and not the
other.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-21 14:08:05 +02:00
Rusty Russell 79ca9bf998 gossipd: use per-peer information to make messages clearer.
We can (usually) indicate what peer caused the bad gossip error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Christian Decker e23d8a4aff pytest: Mark some tests as skippable that are too bitcoin-specific 2019-10-03 04:32:57 +00:00
Christian Decker 136d4b2568 pytest: Consolidate fee-fetching in test_setchannelfee_usage
The short_channel_id is already in text format, no need to hexlify it :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-22 02:03:43 +00:00
Rusty Russell 147eaced2e developer: consolidiate gossip timing options into one --dev-fast-gossip.
It's generally clearer to have simple hardcoded numbers with an
#if DEVELOPER around it, than apparent variables which aren't, really.

Interestingly, our pruning test was always kinda broken: we have to pass
two cycles, since l2 will refresh the channel once to avoid pruning.

Do the more obvious thing, and cut the network in half and check that
l1 and l3 time out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell 0bab2580fc gossipd: clean up local channel updates.
Make update_local_channel use a timer if it's too soon to make another
update.

1. Implement cupdate_different() which compares two updates.
2. make update_local_channel() take a single arg for timer usage.
3. Set timestamp of non-disable update back 5 minutes, so we can
   always generate a disable update if we need to.
4. Make update_local_channel() itself do the "unchanged update" suppression.
   gossipd: clean up local channel updates.
5. Keep pointer to the current timer so we override any old updates with
   a new one, to avoid a race.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
trueptolemy c737fa6b91 pay: Fix logic of the intereface `find_worst_channel` 2019-09-17 21:06:12 +02:00
trueptolemy d4104c15aa pytest: Test `pay` can exclude error node 2019-09-16 12:22:06 +08:00
Rusty Russell a134062f98 bolt11: handle `9` fields for new features.
This implements https://github.com/lightningnetwork/lightning-rfc/pull/656

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-05 23:39:05 -05:00
Rusty Russell d943d8abbc lightningd: expose full onion error when we have it.
Mainly useful for testing.  In particular, we don't save it to the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell a525427272 pytest: don't run test_forward_local_failed_stats under Travis w/ VALGRIND
It's timing out: I suspect it's simply too much memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell c43d09dfd3 pytest: two more slow tests I missed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 02:22:01 +00:00
Rusty Russell 0edc0ae5e9 pytest: don't use deprecated options for close() in tests.
Only downside is you have to wait 1 second at least before
unilaterally closing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell b35dc4689b pytest: enable deprecated APIs for multi-arg closes.
We're about to change the API, so this makes the tests still work
across the transition (and, as a bonus, tests our backwards compat
shim).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell 4fcd94a091 pytest: use absolute paths for plugin arguments.
We're going to change this in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-03 09:10:11 +00:00
Rusty Russell 26cdf9d3dc plugins/pay: don't retry routehint if it contains already-eliminated channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-12 02:00:15 +00:00
Rusty Russell 260febd88b plugins/pay: fix attempt counter on failure message.
An "attempt" is when we actually try to send, not every route lookup
we do.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-12 02:00:15 +00:00
Rusty Russell db22d2ba9a pytest: simplify and annotate test_channel_drainage.
Remove gratuitous prints, add explanations of what's going on,
and demonstrate that we can add a final trimmed HTLC but not
a non-trimmed one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 431401ad35 channeld: don't subtract both reserves from advertized htlc_max.
Subtracting both arbitrarily reduces our capacity, even for ourselves
since the routing logic uses this maximum.

I also changed 'advertise' to 'advertize', since we use american
spelling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 044860881e lightningd: don't let spendable_msat go wumbo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell bb6c34fb13 lightningd: take into account outstanding HTLCs for 'spendable_msat'
The current calculation ignores them, which is unrealistic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 7b7d46c166 pytest: extract separate tests that spendable_msat is accurate.
Turns out we needed more comprehensive testing; we ended up with three
separate tests.  To avoid changing test_channel_drainage as we fix
spendable_msat, I substituted raw numbers there.

The first is a variation of the existing tests, testing we can't
exceed spendable_msat, and we can pay it, both ways.

The second is with a larger amount, which triggers a different problem.

The final is with a giant channel, which tests our 2^32-1 msat cap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 409368e058 pytest: move test_channel_drainage to test_pay.py
This is where payment tests should go.  Also mark it xfail for the moment,
and remove developer-only tag (propagating gossip is only 60 seconds, which
is OK).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell a3b646be31 pytest: fix test_hlcs_cltv_only_difference
Broken by 909913c265, but since Travis
skips this test ("temporarily", according to the commit msg in January!)
it wasn't caught.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-07 10:33:24 +02:00
Christian Decker ced2ff8aa7 pytest: Mark repeat offenders as flaky
It's always the same few tests that fail, so I'm marking them as flaky and
will do a cleanup round later to individually address them.

I've been tracking the failures on the master branch for a few months now and
these are the worst offenders:

|-----------------------------------------|--------|--------|-------|
| test_name                               | state  | branch | count |
|-----------------------------------------|--------|--------|-------|
| test_pay_direct                         | FAILED | master | 31    |
| test_reconnect_gossiping                | FAILED | master | 20    |
| test_htlc_send_timeout                  | FAILED | master | 15    |
| test_pay_limits                         | FAILED | master | 13    |
| test_permfail                           | FAILED | master | 13    |
| test_funding_reorg_private              | FAILED | master | 12    |
| test_invoice                            | FAILED | master | 12    |
| test_invoice_preimage                   | FAILED | master | 12    |
| test_shutdown                           | FAILED | master | 12    |
| test_withdraw                           | FAILED | master | 12    |
| test_gossip_store_load_v3               | FAILED | master | 11    |
| test_onchain_multihtlc_their_unilateral | FAILED | master | 10    |
| test_opening_tiny_channel               | FAILED | master | 10    |
| test_channel_reenable                   | FAILED | master | 9     |
| test_crashlog                           | FAILED | master | 9     |
| test_gossip_weirdalias                  | FAILED | master | 9     |
| test_invoice_expiry                     | FAILED | master | 9     |
| test_onchain_multihtlc_our_unilateral   | FAILED | master | 9     |
| test_peerinfo                           | FAILED | master | 9     |
| test_private_channel                    | FAILED | master | 9     |
| test_shutdown_reconnect                 | FAILED | master | 9     |
| test_closing                            | FAILED | master | 8     |
| test_closing_different_fees             | FAILED | master | 8     |
| test_closing_while_disconnected         | FAILED | master | 8     |
| test_gossip_timestamp_filter            | FAILED | master | 8     |
| test_gossipwith                         | FAILED | master | 8     |
| test_invoice_routeboost                 | FAILED | master | 8     |
| test_onchain_middleman                  | FAILED | master | 8     |
| test_plugin_notifications               | FAILED | master | 8     |
| test_reconnect_channel_peers            | FAILED | master | 8     |
| test_bad_onion                          | FAILED | master | 7     |
| test_closing_torture                    | FAILED | master | 7     |
| test_fulfill_incoming_first             | FAILED | master | 7     |
| test_funding_reorg_disagree_scid_route  | FAILED | master | 7     |
| test_pay_disconnect                     | FAILED | master | 7     |
| test_balance                            | FAILED | master | 6     |
| test_check_command                      | FAILED | master | 6     |
| test_closing_id                         | FAILED | master | 6     |
| test_gossip_addresses                   | FAILED | master | 6     |
| test_gossip_notices_close               | FAILED | master | 6     |
| test_setchannelfee_usage                | FAILED | master | 6     |
| test_announce_address                   | FAILED | master | 5     |
| test_connect_by_gossip                  | FAILED | master | 5     |
| test_gossip_badsig                      | FAILED | master | 5     |
| test_gossip_disable_channels            | FAILED | master | 5     |
| test_gossip_jsonrpc                     | FAILED | master | 5     |
| test_gossip_persistence                 | FAILED | master | 5     |
| test_gossip_pruning                     | FAILED | master | 5     |
| test_onchain_different_fees             | FAILED | master | 5     |
| test_onchain_dust_out                   | FAILED | master | 5     |
|-----------------------------------------|--------|--------|-------|
2019-05-26 16:19:10 +02:00
Rusty Russell 15dc0a5c18 pay: fix transient status 'failed' in listpays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 18d2506958 pytest: test for transient "failed" status during ongoing payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 71460ac073 pay: don't say "Could not find a route" unless we never tried a payment.
It's deeply confusing: we say this after exhausting all other routes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
trueptolemy 77f98f8a5c test_pay: Add test_forward_local_failed_stats() in 5 cases
Here I add the test for this 5 local_failed case in this commit.
There 5 cases for FORWARD_LOCAL_FAILED status:
    1. When Msater resolves the reply about the next peer infor(sent by Gossipd), and need handle unknown next peer failure in channel_resolve_reply();
    2. When Master handle the forward process with the htlc_in and the id of next hop, it tries to drive a new htlc_out but fails in forward_htlc();
    3. When we send htlc_out, Master asks Channeld to add a new htlc into the outgoing channel but Channeld fails. Master need handle and store this failure in rcvd_htlc_reply();
    4. When Channeld receives a new revoke message, if the state of corresponding htlc is RCVD_ADD_ACK_REVOCATION, Master will tries to resolve onionpacket and handle the failure before resolving the next hop in peer_got_revoke();
    5. When Onchaind finds the htlc time out or missing htlc, Master need handle these failure as FORWARD_LOCAL_FAILED in if it's forward payment case.
2019-05-03 11:45:57 +02:00
Rusty Russell bd655678a6 pytest: fix old code in test, which can cause spurious failures.
We generated blocks to announce the channel, but it can also expire
the HTLC if the timing is wrong.  We don't need to anyway, since we
fixed the FIXME; we store local unannounced channels for restoration

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-17 09:25:55 +02:00
Rusty Russell 6630b99cf7 lightningd: move local invoice resolution into invoice.c function.
We're going to make it async, so start by moving the core code into
invoice.c and having that directly call fail/success functions for the
htlc.

We add an extra check in fulfill_htlc() that the HTLC state is correct:
that can't happen now, but may once we're async.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-12 03:32:48 +00:00
Rusty Russell da884751e8 gossipd: make routing_add_channel_update discard old timestamps.
This is currently done higher up, in handle_channel_update(), but
that's one reason why handle_channel_update() has to do a channel
lookup.  Moving the check down means handle_channel_update() can do a
minimal "get node id for this channel" so it can check the signature.

This helps, because the chan lookup semantics are changing in the next
few patches.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Christian Decker 27afc804d5 json-rpc: Include received and resolved time to listforward result
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-10 23:48:39 +00:00
Rusty Russell a498b1fdbf pytest: fix flaky test_setchannel_fee_restart test.
gossipd in l1 might not have registered l2 reconnecting, thus considering
the channel local-disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-03-19 14:22:27 +01:00
Michael Schmoock 162a97894d test: adds test_setchannelfee_all
- will also update old setchannelfee tests to new command output
2019-03-19 00:14:09 +00:00
Michael Schmoock 7e1cc2897c test: add tests for setchannelfee command
Goissp related tests are disabled for non-developers.
New setchannelfee testscases

* py.test -v tests/test_pay.py -k test_setchannelfee_usage
* py.test -v tests/test_pay.py -k test_setchannelfee_routing
* py.test -v tests/test_pay.py -k test_setchannelfee_state
* py.test -v tests/test_pay.py -k test_setchannelfee_zero
* py.test -v tests/test_pay.py -k test_setchannelfee_restart
2019-03-15 02:48:18 +00:00
Rusty Russell 203ef2ed0b listsendpays: updated version of listpayments.
New name is less confusing, and most people should be transitioning to
listpays rather than this anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-23 05:45:25 +00:00
Rusty Russell 1ec959752b plugin/pay: rename 'description' to 'label', deprecate 'description'.
This is the same deprecation, but one level up.  For the moment, we
still support invoices with a `h` field (where description will be
necessary) but that will be removed once this option is removed.

Note that I just changed pylightning without backwards compatibility,
since the field was unlikely to be used, but we could do something
more complex here?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-23 05:45:25 +00:00
Rusty Russell 0d30b89043 channeld: use amount_msat for struct htlc amount.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell cd341b34d6 plugins/pay: use struct amount_msat.
This is particularly interesting because we handle overflow during route
calculation now; this could happen in theory once we wumbo.

It fixes a thinko when we print out routehints, too: we want to print
them out literally, not print out the effect they have on fees (which
is in the route, which we also print).

This ABI change doesn't need a CHANGELOG, since paystatus is new since
release.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell cc95a56544 pylightning: handle msat fields in JSON more appropriately.
Little point having users handle the postfixes manually, this
translates them, and also allows Millisatoshi to be used wherever an
'int' would be previously.

There are also helpers to create the formatting in a way c-lightning's
JSONRPC will accept.

All standard arithmetic operations with integers work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 03:44:44 +00:00
Rusty Russell 83adb94583 lightningd and routing: use struct amount_msat.
We use it in route_hop, and paper over it in the JSON APIs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 03:44:44 +00:00
Rusty Russell 3ba544bfde common/bolt11: use struct amount_msat
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 03:44:44 +00:00
Rusty Russell 7a1d13a77e pytest: make test_pay_direct more robust.
We had occasional failures, because the fuzz could overwhelm the difference
in routes.  Increasing the amount to 2,000,000 millisatoshis makes the
riskfactor 53msat (2000000 * 14 * 10 / 5259600) which is always greater
than the worst-case fuzz of 5% on the fee of 1002msat.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-18 15:32:22 +01:00
Rusty Russell cb6a97152e pytest: fix race in test_pay_direct.
I got a spurious failure because the final node gave a CLTV error and
so it decided to use a different channel.  It should probably handle
this corner case better, but meanwhile make the test robust.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-18 15:32:22 +01:00
Christian Decker 59fa47bf64 pytest: Mark the worst gossip offenders as developer-only tests
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-02-18 00:06:02 +00:00
Christian Decker b7222531fe pytest: Stabilize the test_pay_direct test
It was waiting for a remote channel, but not for all the interesting
channels we want to check. It can sometimes happen that further away
channels are added before closer ones are added, depending on
propagation path, flush timers and bitcoind poll timers. This now just
checks for all channels, which also reduces the ambiguity of whether
we selected a path solely because we were lacking alternatives.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-02-08 01:08:44 +00:00
Rusty Russell fbf2168902 pytest: don't time out on test_pay_direct !DEVELOPER
Travis timed out.

Waiting for three fundchannel commands depends on the bitcoind polling
interval (30 seconds), and then waiting for gossip propagation
requires two propagation intervals (120 seconds).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
Rusty Russell 3ae0c20026 getroute: change definition (and pay default) for riskfactor.
Up until now, riskfactor was useless due to implementation bugs, and
also the default setting is wrong (too low to have an effect on
reasonable payment scenarios).

Let's simplify the definition (by assuming that P(failure) of a node
is 1), to make it a simple percentage.  I examined the current network
fees to see what would work, and under this definition, a default of
10 seems reasonable (equivalent to 1000 under the old definition).

It is *this* change which finally fixes our test case!  The riskfactor
is now 40msat (1500000 * 14 * 10 / 5259600 = 39.9), comparable with
worst-case fuzz is 50msat (1001 * 0.05 = 50).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
Rusty Russell 706debf3d4 pytest: make test_pay_direct more effective.
The test sometimes passes: our routing logic always chooses between
the shorter of two equal-cost routes (because we compare best with <
not <=).

By adding another hop, we add more noise, and by making the alternate
route fee 0 we provide the worst case.

But to be fair, we make the amount of the payment ~50c (15,000,000
msat), and increase our cltv-delay to 14 and fee-base 1000 to match
mainnet.  The final patch shows the effect of this choice.

Otherwise our risk penalty is completely in the noise on
mainnet which has the vast majority of fees set at 1000msat + 1ppm.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
Rusty Russell 5d658012d6 plugins/pay: try without routehints first.
This is the direct cause of the failure of the original
test_pay_direct test and it makes sense: invoice routehints may not be
necessary, so try without them *first* rather than last.

We didn't mention the use of routehints in CHANGELOG at all yet, so
do that now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
ZmnSCPxj 38a2f6c616 test_pay.py: Add test that we prefer direct route. 2019-02-06 18:39:52 +01:00
Christian Decker 27b66997da pytest: Temporarily disable test_htlcs_cltv_only_difference with VG
It is suddenly timing out a lot and is breaking master, so we
temporarily disable it until it is fixed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-01-29 00:50:09 +00:00
Rusty Russell 29b106720e plugins/pay: paystatus should explicitly describe why it is making each attempt.
So add a new 'strategy' field.  This makes it clearer what is going
on, currently one of:

* "Initial attempt"
* "Excluded channel <scid>"
* "Removed route hint"
* "Excluded expensive channel <scid>"
* "Excluded delaying channel <scid>"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 9d33a3d752 plugins/pay: eliminate worst channel if we go over fee / delay threshold.
But keep the error in this case, so we don't always report "no route".

Reported-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 0a8b4f8935 pay: remove inbuilt command in favor of plugin.
This doesn't actually remove some of the now-unnecessary infrastructure
though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 3f8dd7a95f plugins/pay: add paystatus command to get gory details of payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell b17c344b71 plugins/pay: retry when routehint fails.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell e3f893444d pytest: test more-than-one-hop route hints.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 2b72ebb3b6 plugins/pay: implement routeboost hints, naive version.
We sanitize the routes: firstly, we assume appending so eliminate the
first hop if the route points straight to us.  Secondly, eliminate empty
hints.  Thirdly, trim overlong hints.

Then we just use the first route hint.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell b787200752 pytest: test that we correctly use routeboost information from bolt11 invoice.
This is implemented in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell a8f2f28c72 plugins/pay: implement maxfeepercent, maxdelay and exemptfee.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 139604a618 pytest: test that pay command retries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 9f1f79587e short_channel_id_dir: new primitive for one direction of short_channel_id
Currently only used by gossipd for channel elimination.

Also print them in canonical form (/[01]), so tests need to be
changed.

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Christian Decker 659a26ea5a misc: Update short_channel_id representation to use 'x' separators
Reported-by: Alex Bosworth <@alexbosworth>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-01-15 03:50:27 +00:00
Rusty Russell a8e0e1709a channeld: fix fee calculation.
Funder can't spend the fee it needs to pay for the commitment transaction:
we were not converting to millisatoshis, however!

This breaks our routeboost test, which no longer has sufficient funds
to make payment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-15 13:50:15 +01:00
lisa neigut eab992cecd py-tests: rename 'announce' to 'wait_for_announce'
Better description of what the option actually does -- if true
waits for the announcement messages to be generated and exchanged.
2018-12-08 15:15:55 -08:00
Rusty Russell debbdc0781 bolt11: accept lightning: prefix.
The Blockstream store produces these, for example, so let's ignore them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-20 21:02:31 +01:00
Rusty Russell 7d614aaf25 pytest: really remove all bitcoin generate RPC calls.
generate was deprecated some time ago, so we added the generate_block()
helper.  But many calls crept back in, and git master refuses it.

(test_blockchaintrack relied on the return value, so make generate_block
return the list of blocks).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 38e6aa66ff python: quieten modern flake8.
After Ubuntu 18.10 upgrade, lots of new flake8 warnings.

$ flake8 --version:
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 1.6.0) CPython 3.6.7rc1 on Linux

Note it seems that W503 warned about line breaks before binary
operators, and W504 complains about them after.  I prefer W504, so
disable W503.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 3746ea36e2 channeld: tiebreak identical HTLC outputs by CLTV.
This was suggested by Pierre-Marie as the solution to the 'same HTLC,
different CLTV' signature mismatch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell 0006d5771d pytest: test for HTLCs with identical payment_hash and different CLTVs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell c403415caa lightningd: format JSON directly into json connection membuf.
My test case is a mainnet gossip store with 22107 channels, and
time to do `lightning-cli listchannels`:

Before: `lightning-cli listchannels` DEVELOPER=0
	real	0m1.303000-1.324000(1.3114+/-0.0091)s

After:
	real	0m0.629000-0.695000(0.64985+/-0.019)s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 22:02:11 +00:00
Christian Decker f583e6bc58 pytest: Silence some more warnings about raw_strings in regexps 2018-10-19 22:01:22 +00:00
Rusty Russell 31d8a5a9a4 pytest: cleanup test_forward_stats.
Use dev_ignore_htlcs to make sure HTLC gets stuck.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 21:58:27 +00:00
Rusty Russell 604638712b jsonrpc: Only show total fees collected in getinfo.
And use wallet_forward_status_in_db() everywhere in db code.
And clean up extra CHANGELOG.md entry (looks like rebase error?)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 21:58:27 +00:00
Christian Decker 6b7546b94d json-rpc: Rename `getroutestats` and move stats to getinfo
Signed-off-by: Christian Decker <@cdecker>
2018-10-19 21:58:27 +00:00
Christian Decker facd7d16aa json-rpc: Add `listforwardings` command 2018-10-19 21:58:27 +00:00
Christian Decker 6efdd5275c pytest: Test that we record forwardings correctly
Adapts the `test_forward_stats` test to include checks for the
`forwarded_payments` table. Will add checks for the `listforwardings`
RPC call next.

Signed-off-by: Christian Decker <@cdecker>
2018-10-19 21:58:27 +00:00
lisa neigut 0ae1d03513 BOLT7: broadcast `htlc_maximum_msat` in `channel_update s
Have c-lightning nodes send out the largest value for
`htlc_maximum_msat` that makes sense, ie the lesser of
the peer's max_inflight_htlc value or the total channel
capacity minus the total channel reserve.
2018-10-16 03:32:27 +00:00
Christian Decker 81f5b89fe3 wallet: Actually load wallet stats when asked to
The call to `sqlite3_step` is actually needed, otherwise we'll always
get the default values for all types.
2018-10-13 10:58:53 +00:00
Christian Decker 2354b3e706 json-rpc: Create test showcasing the null-accounting issue
There is an issue with the way we retrieve the channel accounting info
that will result in always showing 0 for all stats. This tests for it
and the next commit will fix it.
2018-10-13 10:58:53 +00:00
Rusty Russell b3fb23f921 pytest: make line_graph wait for gossip propagation if announce param is True
This is what the callers want; generalize it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell e450c6bbdb gossipd: remove time-delayed local channel_update, produce DISABLE on-demand.
We have a lot of infrastructure to delay local channel_updates to
avoid spamming on each peer reconnect; we had to keep tracking of
pending ones though, in case we needed the very latest for sending an
error when failing an HTLC.

Instead, it's far simpler to set the local_disabled flag on a channel
when we disconnect, but only send a disabling channel_update if we
actually fail an HTLC.

Note: handle_channel_update() TAKES update (due to tal_arr_dup), but we
didn't use that before.  Now we do, add annotation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-26 03:21:35 +00:00
Rusty Russell d93be58bd0 pytest: remove use dev-override-feerates.
Manipulate fees via fake-bitcoin-cli.  It's not quite the same, as
these are pre-smoothing, so we need a restart to override that where
we really need an exact change.  Or we can wait until it reaches a
certain value in cases we don't care about exact amounts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-24 02:17:51 +00:00
Rusty Russell 607d4bf9d2 channel: update fees after lockin.
We don't respond to fee changes until we're locked in: make sure we catch
up at that point.

Note that we use NORMAL fees during opening, but IMMEDIATE after, so
this often sends a fee update.  The tests which break, we set those
feerates to be equal.

This (sometimes) changes the behavior of test_permfail, as we now
get an immediate commit, so that is fixed too so we always wait for
that to complete.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-24 02:17:51 +00:00
Rusty Russell 36b1cac6e6 lightningd: new state AWAITING_UNILATERAL.
When in this state, we send a canned error "Awaiting unilateral close".
We enter this both when we drop to chain, and when we're trying to get
them to drop to chain due to option_data_loss_protect.

As this state (unlike channel errors) is saved to the database, it means
we will *never* talk to a peer again in this state, so they can't
confuse us.

Since we set this state in channel_fail_permanent() (which is the only
place we call drop_to_chain for a unilateral close), we don't need to
save to the db: channel_set_state() does that for us.

This state change has a subtle effect: we return WIRE_UNKNOWN_NEXT_PEER
instead of WIRE_TEMPORARY_CHANNEL_FAILURE as soon as we get a failure
with a peer.  To provoke a temporary failure in test_pay_disconnect we
take the node offline.

Reported-by: Christian Decker @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-23 14:46:22 +02:00
Rusty Russell 50f5eb34b4 openingd: take peer before we're opening, wait for explicit funding msg.
Prior to this, lightningd would hand uninteresting peers back to connectd,
which would then return it to lightningd if it sent a non-gossip msg,
or if lightningd asked it to release the peer.

Now connectd hands the peer to lightningd once we've done the init
handshake, which hands it off to openingd.

This is a deep structural change, so we do the minimum here and cleanup
in the following patches.

Lightningd:
1. Remove peer_nongossip handling from connect_control and peer_control.
2. Remove list of outstanding fundchannel command; it was only needed to
   find the race between us asking connectd to release the peer and it
   reconnecting.
3. We can no longer tell if the remote end has started trying to fund a
   channel (until it has succeeded): it's very transitory anyway so not
   worth fixing.
4. We now always have a struct peer, and allocate an uncommitted_channel
   for it, though it may never be used if neither end funds a channel.
5. We start funding on messages for openingd: we can get a funder_reply
   or a fundee, or an error in response to our request to fund a channel.
   so we handle all of them.
6. A new peer_start_openingd() is called after connectd hands us a peer.
7. json_fund_channel just looks through local peers; there are none
   hidden in connectd any more.
8. We sometimes start a new openingd just to send an error message.

Openingd:
1. We always have information we need to accept them funding a channel (in
   the init message).
2. We have to listen for three fds: peer, gossip and master, so we opencode
   the poll.
3. We have an explicit message to start trying to fund a channel.
4. We can be told to send a message in our init message.

Testing:
1. We don't handle some things gracefully yet, so two tests are disabled.
2. 'hand_back_peer .*: now local again' from connectd is no longer a message,
   openingd says 'Handed peer, entering loop' once its managing it.
3. peer['state'] used to be set to 'GOSSIPING' (otherwise this field doesn't
   exist; 'state' is now per-channel.  It doesn't exist at all now.
4. Some tests now need to turn on IO logging in openingd, not connectd.
5. There's a gap between connecting on one node and having connectd on
   the peer hand over the connection to openingd.  Our tests sometimes
   checked getpeers() on the peer, and didn't see anything, so line_graph
   needed updating.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Christian Decker 13d9f30a6a pytest: Move forwarded payments to test_pay.py 2018-08-07 00:54:19 +00:00
Christian Decker 605bf8c89d pytest: Migrate the gossip related tests to the new fixture model 2018-08-07 00:54:19 +00:00
Christian Decker 2e26d4fc42 pytest: Migrate payment related tests to use modern fixtures 2018-08-07 00:54:19 +00:00