Commit Graph

83 Commits

Author SHA1 Message Date
Rusty Russell a134062f98 bolt11: handle `9` fields for new features.
This implements https://github.com/lightningnetwork/lightning-rfc/pull/656

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-05 23:39:05 -05:00
Rusty Russell d943d8abbc lightningd: expose full onion error when we have it.
Mainly useful for testing.  In particular, we don't save it to the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell a525427272 pytest: don't run test_forward_local_failed_stats under Travis w/ VALGRIND
It's timing out: I suspect it's simply too much memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell c43d09dfd3 pytest: two more slow tests I missed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 02:22:01 +00:00
Rusty Russell 0edc0ae5e9 pytest: don't use deprecated options for close() in tests.
Only downside is you have to wait 1 second at least before
unilaterally closing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell b35dc4689b pytest: enable deprecated APIs for multi-arg closes.
We're about to change the API, so this makes the tests still work
across the transition (and, as a bonus, tests our backwards compat
shim).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell 4fcd94a091 pytest: use absolute paths for plugin arguments.
We're going to change this in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-03 09:10:11 +00:00
Rusty Russell 26cdf9d3dc plugins/pay: don't retry routehint if it contains already-eliminated channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-12 02:00:15 +00:00
Rusty Russell 260febd88b plugins/pay: fix attempt counter on failure message.
An "attempt" is when we actually try to send, not every route lookup
we do.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-12 02:00:15 +00:00
Rusty Russell db22d2ba9a pytest: simplify and annotate test_channel_drainage.
Remove gratuitous prints, add explanations of what's going on,
and demonstrate that we can add a final trimmed HTLC but not
a non-trimmed one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 431401ad35 channeld: don't subtract both reserves from advertized htlc_max.
Subtracting both arbitrarily reduces our capacity, even for ourselves
since the routing logic uses this maximum.

I also changed 'advertise' to 'advertize', since we use american
spelling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 044860881e lightningd: don't let spendable_msat go wumbo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell bb6c34fb13 lightningd: take into account outstanding HTLCs for 'spendable_msat'
The current calculation ignores them, which is unrealistic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 7b7d46c166 pytest: extract separate tests that spendable_msat is accurate.
Turns out we needed more comprehensive testing; we ended up with three
separate tests.  To avoid changing test_channel_drainage as we fix
spendable_msat, I substituted raw numbers there.

The first is a variation of the existing tests, testing we can't
exceed spendable_msat, and we can pay it, both ways.

The second is with a larger amount, which triggers a different problem.

The final is with a giant channel, which tests our 2^32-1 msat cap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell 409368e058 pytest: move test_channel_drainage to test_pay.py
This is where payment tests should go.  Also mark it xfail for the moment,
and remove developer-only tag (propagating gossip is only 60 seconds, which
is OK).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-11 23:19:11 +00:00
Rusty Russell a3b646be31 pytest: fix test_hlcs_cltv_only_difference
Broken by 909913c265, but since Travis
skips this test ("temporarily", according to the commit msg in January!)
it wasn't caught.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-07 10:33:24 +02:00
Christian Decker ced2ff8aa7 pytest: Mark repeat offenders as flaky
It's always the same few tests that fail, so I'm marking them as flaky and
will do a cleanup round later to individually address them.

I've been tracking the failures on the master branch for a few months now and
these are the worst offenders:

|-----------------------------------------|--------|--------|-------|
| test_name                               | state  | branch | count |
|-----------------------------------------|--------|--------|-------|
| test_pay_direct                         | FAILED | master | 31    |
| test_reconnect_gossiping                | FAILED | master | 20    |
| test_htlc_send_timeout                  | FAILED | master | 15    |
| test_pay_limits                         | FAILED | master | 13    |
| test_permfail                           | FAILED | master | 13    |
| test_funding_reorg_private              | FAILED | master | 12    |
| test_invoice                            | FAILED | master | 12    |
| test_invoice_preimage                   | FAILED | master | 12    |
| test_shutdown                           | FAILED | master | 12    |
| test_withdraw                           | FAILED | master | 12    |
| test_gossip_store_load_v3               | FAILED | master | 11    |
| test_onchain_multihtlc_their_unilateral | FAILED | master | 10    |
| test_opening_tiny_channel               | FAILED | master | 10    |
| test_channel_reenable                   | FAILED | master | 9     |
| test_crashlog                           | FAILED | master | 9     |
| test_gossip_weirdalias                  | FAILED | master | 9     |
| test_invoice_expiry                     | FAILED | master | 9     |
| test_onchain_multihtlc_our_unilateral   | FAILED | master | 9     |
| test_peerinfo                           | FAILED | master | 9     |
| test_private_channel                    | FAILED | master | 9     |
| test_shutdown_reconnect                 | FAILED | master | 9     |
| test_closing                            | FAILED | master | 8     |
| test_closing_different_fees             | FAILED | master | 8     |
| test_closing_while_disconnected         | FAILED | master | 8     |
| test_gossip_timestamp_filter            | FAILED | master | 8     |
| test_gossipwith                         | FAILED | master | 8     |
| test_invoice_routeboost                 | FAILED | master | 8     |
| test_onchain_middleman                  | FAILED | master | 8     |
| test_plugin_notifications               | FAILED | master | 8     |
| test_reconnect_channel_peers            | FAILED | master | 8     |
| test_bad_onion                          | FAILED | master | 7     |
| test_closing_torture                    | FAILED | master | 7     |
| test_fulfill_incoming_first             | FAILED | master | 7     |
| test_funding_reorg_disagree_scid_route  | FAILED | master | 7     |
| test_pay_disconnect                     | FAILED | master | 7     |
| test_balance                            | FAILED | master | 6     |
| test_check_command                      | FAILED | master | 6     |
| test_closing_id                         | FAILED | master | 6     |
| test_gossip_addresses                   | FAILED | master | 6     |
| test_gossip_notices_close               | FAILED | master | 6     |
| test_setchannelfee_usage                | FAILED | master | 6     |
| test_announce_address                   | FAILED | master | 5     |
| test_connect_by_gossip                  | FAILED | master | 5     |
| test_gossip_badsig                      | FAILED | master | 5     |
| test_gossip_disable_channels            | FAILED | master | 5     |
| test_gossip_jsonrpc                     | FAILED | master | 5     |
| test_gossip_persistence                 | FAILED | master | 5     |
| test_gossip_pruning                     | FAILED | master | 5     |
| test_onchain_different_fees             | FAILED | master | 5     |
| test_onchain_dust_out                   | FAILED | master | 5     |
|-----------------------------------------|--------|--------|-------|
2019-05-26 16:19:10 +02:00
Rusty Russell 15dc0a5c18 pay: fix transient status 'failed' in listpays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 18d2506958 pytest: test for transient "failed" status during ongoing payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 71460ac073 pay: don't say "Could not find a route" unless we never tried a payment.
It's deeply confusing: we say this after exhausting all other routes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
trueptolemy 77f98f8a5c test_pay: Add test_forward_local_failed_stats() in 5 cases
Here I add the test for this 5 local_failed case in this commit.
There 5 cases for FORWARD_LOCAL_FAILED status:
    1. When Msater resolves the reply about the next peer infor(sent by Gossipd), and need handle unknown next peer failure in channel_resolve_reply();
    2. When Master handle the forward process with the htlc_in and the id of next hop, it tries to drive a new htlc_out but fails in forward_htlc();
    3. When we send htlc_out, Master asks Channeld to add a new htlc into the outgoing channel but Channeld fails. Master need handle and store this failure in rcvd_htlc_reply();
    4. When Channeld receives a new revoke message, if the state of corresponding htlc is RCVD_ADD_ACK_REVOCATION, Master will tries to resolve onionpacket and handle the failure before resolving the next hop in peer_got_revoke();
    5. When Onchaind finds the htlc time out or missing htlc, Master need handle these failure as FORWARD_LOCAL_FAILED in if it's forward payment case.
2019-05-03 11:45:57 +02:00
Rusty Russell bd655678a6 pytest: fix old code in test, which can cause spurious failures.
We generated blocks to announce the channel, but it can also expire
the HTLC if the timing is wrong.  We don't need to anyway, since we
fixed the FIXME; we store local unannounced channels for restoration

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-17 09:25:55 +02:00
Rusty Russell 6630b99cf7 lightningd: move local invoice resolution into invoice.c function.
We're going to make it async, so start by moving the core code into
invoice.c and having that directly call fail/success functions for the
htlc.

We add an extra check in fulfill_htlc() that the HTLC state is correct:
that can't happen now, but may once we're async.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-12 03:32:48 +00:00
Rusty Russell da884751e8 gossipd: make routing_add_channel_update discard old timestamps.
This is currently done higher up, in handle_channel_update(), but
that's one reason why handle_channel_update() has to do a channel
lookup.  Moving the check down means handle_channel_update() can do a
minimal "get node id for this channel" so it can check the signature.

This helps, because the chan lookup semantics are changing in the next
few patches.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Christian Decker 27afc804d5 json-rpc: Include received and resolved time to listforward result
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-10 23:48:39 +00:00
Rusty Russell a498b1fdbf pytest: fix flaky test_setchannel_fee_restart test.
gossipd in l1 might not have registered l2 reconnecting, thus considering
the channel local-disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-03-19 14:22:27 +01:00
Michael Schmoock 162a97894d test: adds test_setchannelfee_all
- will also update old setchannelfee tests to new command output
2019-03-19 00:14:09 +00:00
Michael Schmoock 7e1cc2897c test: add tests for setchannelfee command
Goissp related tests are disabled for non-developers.
New setchannelfee testscases

* py.test -v tests/test_pay.py -k test_setchannelfee_usage
* py.test -v tests/test_pay.py -k test_setchannelfee_routing
* py.test -v tests/test_pay.py -k test_setchannelfee_state
* py.test -v tests/test_pay.py -k test_setchannelfee_zero
* py.test -v tests/test_pay.py -k test_setchannelfee_restart
2019-03-15 02:48:18 +00:00
Rusty Russell 203ef2ed0b listsendpays: updated version of listpayments.
New name is less confusing, and most people should be transitioning to
listpays rather than this anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-23 05:45:25 +00:00
Rusty Russell 1ec959752b plugin/pay: rename 'description' to 'label', deprecate 'description'.
This is the same deprecation, but one level up.  For the moment, we
still support invoices with a `h` field (where description will be
necessary) but that will be removed once this option is removed.

Note that I just changed pylightning without backwards compatibility,
since the field was unlikely to be used, but we could do something
more complex here?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-23 05:45:25 +00:00
Rusty Russell 0d30b89043 channeld: use amount_msat for struct htlc amount.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell cd341b34d6 plugins/pay: use struct amount_msat.
This is particularly interesting because we handle overflow during route
calculation now; this could happen in theory once we wumbo.

It fixes a thinko when we print out routehints, too: we want to print
them out literally, not print out the effect they have on fees (which
is in the route, which we also print).

This ABI change doesn't need a CHANGELOG, since paystatus is new since
release.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell cc95a56544 pylightning: handle msat fields in JSON more appropriately.
Little point having users handle the postfixes manually, this
translates them, and also allows Millisatoshi to be used wherever an
'int' would be previously.

There are also helpers to create the formatting in a way c-lightning's
JSONRPC will accept.

All standard arithmetic operations with integers work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 03:44:44 +00:00
Rusty Russell 83adb94583 lightningd and routing: use struct amount_msat.
We use it in route_hop, and paper over it in the JSON APIs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 03:44:44 +00:00
Rusty Russell 3ba544bfde common/bolt11: use struct amount_msat
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 03:44:44 +00:00
Rusty Russell 7a1d13a77e pytest: make test_pay_direct more robust.
We had occasional failures, because the fuzz could overwhelm the difference
in routes.  Increasing the amount to 2,000,000 millisatoshis makes the
riskfactor 53msat (2000000 * 14 * 10 / 5259600) which is always greater
than the worst-case fuzz of 5% on the fee of 1002msat.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-18 15:32:22 +01:00
Rusty Russell cb6a97152e pytest: fix race in test_pay_direct.
I got a spurious failure because the final node gave a CLTV error and
so it decided to use a different channel.  It should probably handle
this corner case better, but meanwhile make the test robust.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-18 15:32:22 +01:00
Christian Decker 59fa47bf64 pytest: Mark the worst gossip offenders as developer-only tests
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-02-18 00:06:02 +00:00
Christian Decker b7222531fe pytest: Stabilize the test_pay_direct test
It was waiting for a remote channel, but not for all the interesting
channels we want to check. It can sometimes happen that further away
channels are added before closer ones are added, depending on
propagation path, flush timers and bitcoind poll timers. This now just
checks for all channels, which also reduces the ambiguity of whether
we selected a path solely because we were lacking alternatives.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-02-08 01:08:44 +00:00
Rusty Russell fbf2168902 pytest: don't time out on test_pay_direct !DEVELOPER
Travis timed out.

Waiting for three fundchannel commands depends on the bitcoind polling
interval (30 seconds), and then waiting for gossip propagation
requires two propagation intervals (120 seconds).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
Rusty Russell 3ae0c20026 getroute: change definition (and pay default) for riskfactor.
Up until now, riskfactor was useless due to implementation bugs, and
also the default setting is wrong (too low to have an effect on
reasonable payment scenarios).

Let's simplify the definition (by assuming that P(failure) of a node
is 1), to make it a simple percentage.  I examined the current network
fees to see what would work, and under this definition, a default of
10 seems reasonable (equivalent to 1000 under the old definition).

It is *this* change which finally fixes our test case!  The riskfactor
is now 40msat (1500000 * 14 * 10 / 5259600 = 39.9), comparable with
worst-case fuzz is 50msat (1001 * 0.05 = 50).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
Rusty Russell 706debf3d4 pytest: make test_pay_direct more effective.
The test sometimes passes: our routing logic always chooses between
the shorter of two equal-cost routes (because we compare best with <
not <=).

By adding another hop, we add more noise, and by making the alternate
route fee 0 we provide the worst case.

But to be fair, we make the amount of the payment ~50c (15,000,000
msat), and increase our cltv-delay to 14 and fee-base 1000 to match
mainnet.  The final patch shows the effect of this choice.

Otherwise our risk penalty is completely in the noise on
mainnet which has the vast majority of fees set at 1000msat + 1ppm.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
Rusty Russell 5d658012d6 plugins/pay: try without routehints first.
This is the direct cause of the failure of the original
test_pay_direct test and it makes sense: invoice routehints may not be
necessary, so try without them *first* rather than last.

We didn't mention the use of routehints in CHANGELOG at all yet, so
do that now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-06 18:39:52 +01:00
ZmnSCPxj 38a2f6c616 test_pay.py: Add test that we prefer direct route. 2019-02-06 18:39:52 +01:00
Christian Decker 27b66997da pytest: Temporarily disable test_htlcs_cltv_only_difference with VG
It is suddenly timing out a lot and is breaking master, so we
temporarily disable it until it is fixed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-01-29 00:50:09 +00:00
Rusty Russell 29b106720e plugins/pay: paystatus should explicitly describe why it is making each attempt.
So add a new 'strategy' field.  This makes it clearer what is going
on, currently one of:

* "Initial attempt"
* "Excluded channel <scid>"
* "Removed route hint"
* "Excluded expensive channel <scid>"
* "Excluded delaying channel <scid>"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 9d33a3d752 plugins/pay: eliminate worst channel if we go over fee / delay threshold.
But keep the error in this case, so we don't always report "no route".

Reported-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 0a8b4f8935 pay: remove inbuilt command in favor of plugin.
This doesn't actually remove some of the now-unnecessary infrastructure
though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 3f8dd7a95f plugins/pay: add paystatus command to get gory details of payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell b17c344b71 plugins/pay: retry when routehint fails.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00