Commit Graph

8486 Commits

Author SHA1 Message Date
Rusty Russell 7ca00ca7d7 ccan: update so we can compile with -O2 on Ubuntu.
Otherwise we get a configurator failure:

    In file included from /usr/include/string.h:495,
                     from configuratortest.c:2:
    In function ‘strncpy’,
        inlined from ‘main’ at configuratortest.c:6:2:
    /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound 8 equals destination size [-Wstringop-truncation]
      106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
          |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-17 13:54:06 +02:00
Rusty Russell 73a5f5b313 fundpsbt: make parameters more usable.
fundpsbt forces the caller to manually add their weight * feerate
to the satoshis they ask for.  That means no named feerates.

Instead, create a startweight parameter and do the calc for them
internally, and return the feerate we used (and, while we're at it,
the estimated final weight).

This API change is best done now, as it would otherwise have to
be appended as a parameter.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-17 13:53:46 +02:00
ZmnSCPxj jxPCSnmZ fe119fc8fd CHANGELOG.md: Update security changes for 0.7.1.
The CVE was fully disclosed, so we can safely add it to the Security
field for the 0.7.1 changelog.

Also removed the "No security changes were necessary" text.
If we do this for releases, then either we lie about a CVE-level problem,
or we leak that a release fixes a CVE-level problem.
2020-07-17 01:03:22 +00:00
Christian Decker cc2f9b4541 release: Add changelog entries for v0.9.0 release 2020-07-15 16:11:25 +02:00
Christian Decker de096eeed9 pytest: Disable test_pay_routeboost for valgrind due to timeout 2020-07-15 11:32:58 +02:00
Christian Decker 86ad15d040 travis: Spread the valgrind load on more configurations 2020-07-15 11:32:58 +02:00
Christian Decker 214f418c3b plugin: Fix a memory leak and a missing dereference in listconfigs
`listconfigs` calls were setting the description twice and was using the
pointer to the boolean value as the boolean value, resulting in always
returning `true`.
2020-07-15 11:32:58 +02:00
Christian Decker de90606490 pytest: Add an adaptive MPP test
This exercises something that is simply not possible without MPP, i.e., the
bundling of multiple paths to get sufficient capacity to perform the payment.
2020-07-15 11:32:58 +02:00
Christian Decker dad23066dc pytest: Disable MPP sending for 4 tests
These mostly deal with exact HTLC counts, and fixed number of attempts to
conclusion, so the randomization that MPP adds is not desirable.
2020-07-15 11:32:58 +02:00
Christian Decker 212a3c5ec5 ld: We might not have a failing channel if localfail and sendonion
This happens to be an edge case with the way we use `sendonion` in
MPP. `sendonion` does not attempt to recover the route even if we supply the
shared secrets (it'd require us to map forwarding channels to the nodes etc),
so `failnode` will always be unset, unless it is the first hop, which gets
stored. This is not a problem if it weren't for the fact that we don't store
the partial route, consisting solely of the channel leading to the first hop,
therefore the assertion that either both are NULL or both aren't fails on the
first hop.

This went unnoticed since with MPP we have more concurrent payments in flight,
increasing the chances of a exhausted first hop considerably.
2020-07-15 11:32:58 +02:00
Christian Decker a3610d66ac retrymod: Make retry modifier slightly more verbose
I found it rather useful to trace how a payment is getting retried in the logs.
2020-07-15 11:32:58 +02:00
Christian Decker 041ee930a4 mpp: Consider an abort as the payment being finished
If one part sets the root to be aborted, there is little point in continuing
to wait for the remainder, return to the caller immediately.
2020-07-15 11:32:58 +02:00
Christian Decker 5bef4fc196 mpp: Lower amounts below the presplit amount
Some tests were failing because they were running into the presplit modifier
and then surprised that the payment got split.
2020-07-15 11:32:58 +02:00
Christian Decker 718b6e3398 mpp: Detect if destination supports MPP from invoice and abort early
We abort on the root since that is the coordination point for all parts of the payment.
2020-07-15 11:32:58 +02:00
Christian Decker de75d3ac0c mpp: Add CLI option to opt-out of multi-part payments
Several tests are not well-suited for mpp, so I added a CLI option to opt-out
of the MPP support at startup time.
2020-07-15 11:32:58 +02:00
Christian Decker a287bbe55d mpp: Enable adaptive splitter
Changelog-Added: The adaptive multi-part payment modifier will split payments that are failing due to their size into smaller parts, and re-attempted.
2020-07-15 11:32:58 +02:00
Christian Decker 535aaca109 paymod: Implement adaptive splitter
This modifier splits a payment that has been attempted a number of times (by a
modifier earlier in the mod chain) and has failed consistently. It splits the
amount roughly in half, with a but if random fuzz, and then starts a new round
of attempts for the two smaller amounts.
2020-07-15 11:32:58 +02:00
Christian Decker 443643e0b0 retrymod: Reset retry counter if parent is a split
If the parent is a split we have new payment parameters, and want to perform a
number of attempts with those.
2020-07-15 11:32:58 +02:00
Christian Decker d0eb3a79eb paymod: Not having a result doesn't mean we failed at getroute
Specifically if we split, there is no result, but we shouldn't add a failure
message.
2020-07-15 11:32:58 +02:00
Christian Decker b813974e13 mpp: Add the presplit MPP modifier
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
2020-07-15 11:32:58 +02:00
Christian Decker 3f399d15cb paymod: Don't assume that the first payment was executed at all
With the `presplit`-modifier we actually skip execution of the root altogether
which results in the root not having a result at all. Instead we should use
the result returned by `payment_collect_result`.
2020-07-15 11:32:58 +02:00
Christian Decker f6745682c3 tlvstream: Allow overwriting an already set value
This is necessary in the next commit to override the total_msat that is being
delivered to the destination.
2020-07-15 11:32:58 +02:00
Christian Decker c97ff05ffb mpp: Add the presplit-modifier that splits a root payment first 2020-07-15 11:32:58 +02:00
Christian Decker 7f53ade64b paymod: Allow callers to opt out of shadow routing amount fuzzing
With MPP we require that the sum of parts is equal to the `total_msat` amount
declared in the onion. Since that can't be changed once the first part arrives
we need a way to disable amount fuzzing for MPP.
2020-07-15 11:32:58 +02:00
Rusty Russell 406d0d09cf doc/STYLE.md: be clear we're talking about JSON output.
Suggested-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:50:49 +09:30
Rusty Russell 371cabf976 txprepare: revert 1fb9a078b6 (`psbt` field)
We're actually going to deprecate this, so don't add new features!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: ***REMOVE*** JSON-API: `txprepare` returns a psbt version of the created transaction
2020-07-15 18:49:02 +09:30
Rusty Russell e85c895c5b pytest: restore test_sign_and_send_psbt.
It uses reservations heavily, and assumed we generated change, etc.
It's now a simpler test, in many ways.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell 31d7e013bd pytest: test fundpsbt.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell 1091be3d0d doc: document fundpsbt.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell 10e62af7c7 fundpsbt: add reserve arg.
It's easier for us to call it atomically than have the user loop and
retry!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell bd19ec2292 fundpsbt: new JSON API to gather UTXOs.
Technically, they could do this themselves, but it's much nicer to have one
place to do it (and it makes sure we get the required information into the
PSBT, which is actually not entirely accessible through listfunds, as that
doesn't want to consult with the HSM for close outputs).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON RPC: new low-level coin selection `fundpsbt` routine.
2020-07-15 18:49:02 +09:30
Rusty Russell 4ee527a59c pytest: test reserve and unreserve.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell be17a9392f reserveinputs: add exclusive flag.
This is the normal case: you only want to reserve inputs which
are not already reserved.  This saves you iterating through the
results and unreserving some if you weren't exclusive.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell 56ea215ba0 wallet: new JSON commands reserveinputs and unreserveinputs.
reserveinputs marks UTXOs reserved for 12 hours, so we won't select them
for spending: unreserveinputs marks them available again.

Exposes param_psbt() for wider use.

Disabled the test_sign_and_send_psbt since we're altering the API;
the final patch restores it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell 27b48959d7 wallet: add wallet_find_utxo().
This is a new fundamental routine to obtain UTXOs from the database.

It's not the most efficient approach, as it returns a single UTXO at a
time, but it can consolidate all our UTXO handling (becoming more
complex by the reservation timeout logic).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell 50ff0b26fd wallet: explicit routines to reserve/unreserve a UTXO.
These keep the struct utxo in sync with the database, explicitly:
these will be the only places where utxo->status is set.

The old routines will be removed at the end.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
niftynei 7ebdc14397 utxos: add a 'reserved_til' marker for utxos
Allow a utxo to be reserved until explicitly unreserved or until a timer
runs out. Currently unused.

We explicitly do not unreserve these at startup.
2020-07-15 18:49:02 +09:30
Rusty Russell 899ec2b3d4 JSON API: fix up two existing warnings to be conformant.
Technically an API break, but nobody relies on these I hope!

Note that the feerates warning was buried inside the style object:
it should be top-level.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-14 21:28:26 +02:00
Rusty Russell f53624517b doc: add JSON API advice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-14 21:28:26 +02:00
Christian Decker 65963bc003 exemptfee: Only apply to the root payment and fix logs
We were applying the fee exemption to all payments individually, which is ok
until we switch to MPP, where amounts change. Also the log entry was referring
to the total amount, and not the fee of the payment.
2020-07-14 06:25:29 +09:30
Christian Decker 9be4d552fa paymod: Add a bit more information to the `pay` response
Amount, parent_part_id  and own partid can be helpful when debugging.
2020-07-14 06:25:29 +09:30
Christian Decker 6f40cb0202 paymod: Inherit exemptfee modifier data on retry 2020-07-14 06:25:29 +09:30
Christian Decker b88a55b2a4 paymod: Only wait on blockheight if we need to
We were wrongfully identifying all payment failures as blockheight mismatches.
2020-07-14 06:25:29 +09:30
Christian Decker 817f7c703b travis: Parallelize non-valgrind tests even if under developer
Tests were timing out, and there is no point in artificially limiting
ourselves if we can be quicker.
2020-07-14 06:25:29 +09:30
Christian Decker f20091fc72 pay: Do not duplicate "message" field in response to `pay` 2020-07-14 06:25:29 +09:30
Christian Decker 411b760985 paymod: Reset the step if we're waiting for a block
This was causing the state flapping test to fail, since we were yielding
control of the io_loop, waiting for the blockheight to be reached, and not
setting the status beforehand. An interim `paystatus` call would then find a
failed leaf and deduce the entire payment failed. Setting it back to the
previous state keeps the overall payment pending while we wait.
2020-07-14 06:25:29 +09:30
Christian Decker 02e44e9903 paymod: Set the STOPPED_RETRYING status code if we stopped retrying 2020-07-14 06:25:29 +09:30
Christian Decker 5776a33116 paymod: Activate paymod and move legacy pay to `legacypay` command
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.

Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.

Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
2020-07-14 06:25:29 +09:30
Christian Decker 514c4044c8 tlv: Allow passing a raw pointer and a length to tlvstream_set_raw
Allows us to do fewer allocations, since the argument doesn't have to be tal
allocated itself.

Suggested-by: Rusty Russell <@rustyrussell>
2020-07-13 14:19:16 +02:00
Christian Decker 7978a13cae doc: Add a man-page for the new keysend command 2020-07-13 14:19:16 +02:00