We were blindly initiating the keysend payment, which could lead to
confusing outcomes. This adds a very specific error message to the
error returned.
Changelog-Fixed: keysend: Keysend now checks whether the destination supports keysend before attempting a payment. If not a more informative error is returned.
The latter is available only starting with Python 3.7, and it's the
only test that fails with <3.7. This allows us to keep on testing with
older versions.
If we're already attempting to connect to a peer, we would ignore
new connection requests. This is problematic if your node has bad
connection details for the node -- you can't update it while inflight.
This patch appends new connection suggestions to the list of connections
to try.
Fixes#4154
We've been getting a bunch of 'warnings' about an unknown mark. This
silences the warning, by registering the mark as expected.
=========================================================== warnings summary ===========================================================
tests/test_closing.py:152
/home/niftynei/dev/lightning/tests/test_closing.py:152: PytestUnknownMarkWarning: Unknown pytest.mark.slow_test - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
@pytest.mark.slow_test
tests/test_closing.py:214
/home/niftynei/dev/lightning/tests/test_closing.py:214: PytestUnknownMarkWarning: Unknown pytest.mark.slow_test - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
@pytest.mark.slow_test
tests/test_closing.py:704
We used to create the entire reply, the if it was too big, split in
half and retry.
Now that the main network is larger, this always happens with a full
request, which is inefficient.
Instead, produce a reply assuming no compression, then compress as a
bonus. This is simpler and more efficient, at cost of sending more
packets.
I also renamed an internal dev var to make it clearer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We previously registered hooks up in who-replies-to-getmanifest-first
order, but then if any had dependencies it would scatter that order.
This allows users to manually set dependencies developers have
forgotten by specifying the plugins manually in their configuration or
cmdline. This was an excellent consideration by @mschmook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets. We need to know this so we know when
we can make them compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We let the plugin decide what feerate to accept/whether or not to add
funds to the open. To aid this decision, we also send the plugin what we
(c-lightning) currently have as our max and min acceptable feerates.
We also now use these as our default for max/min acceptable feerate
range when sending an openchannel offer to a peer.
In the future, it might be a good idea to make these more easily
changeable, either via a config setting (?) or a command param.
For compatibility, we only do this if `allow-deprecated-apis` is false
for now. Otherwise scripts parsing should use `grep -v '^# '` or
start using `-N none`.
Changelog-Added: JSON-RPC: `close` now sends notifications for slow closes (if `allow-deprecated-apis`=false)
Changelog-Deprecated: cli: scripts should filter out '^# ' or use `-N none`, as commands will start returning notifications soon
Fixes: #3925
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a new configuration, --enable-fuzzing (which is more than
welcome to be coupled with --enable-address-sanitizer), to pass the
fuzzer sanitizer argument when compiling objects. This allows libfuzzer
to actually be able "to fuzz" by detecting coverage and be smart when
mutating inputs.
As libfuzzer brings its own ~~fees~~ main(), we compile objects with
fsanitize=fuzzer-no-link, and special-case the linkage of the fuzz
targets.
A "lib" is added to abstract out the interface to the fuzzing tool used.
This allow us to use the same targets to fuzz using AFL, hongfuzz or w/e
by adding their entrypoints into libfuzz. (h/t to practicalswift who
introduced this for bitcoin-core, which i mimiced)
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Instead of a boutique message, use a "real" channel_announcement for
private channels (with fake sigs and pubkeys). This makes it far
easier for gossmap to handle local channels.
Backwards compatible update, since we update old stores.
We also fix devtools/dump-gossipstore to know about the tombstone markers.
Since we increment our channel_announce count for local channels now,
the stats in the tests changed too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since `fundchannel` now supports the 'close_to' argument, we can remove
all the logic needed to call fundchannel_start here.
Underneath, we're still calling `fundchannel_start`, we're just one (or
two, if you count multifundchannel) call levels away from it now.
We had one report of this, and then Eugene and Roasbeef of Lightning
Labs confirmed it; they saw misordered HTLCs on reconnection too.
Since we didn't enforce this when we receive HTLCs, we never noticed :(
Fixes: #3920
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: fixed retransmission order of multiple new HTLCs (causing channel close with LND)
We didn't care, but other implementations (particularly lnd) do. And it
does violate the spec.
(We need to use skip not xfail on the test which catches this, since
xfail doesn't seem to stop errors reported by cleanup)
(Includes Christian's typo fix!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're rarely in a hurry here, and bitcoind is aggressive with fees.
You can always spend this output if you really have to, using CPFP.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: mutual closing feerate reduced to "slow" to avoid overpaying.
lightning-5 can sometimes see itself sweeping the unilateral output resulting
in this weird line:
```log
HTLC already resolved by SELF when we found preimage
```
In both cases the flakyness arises from the destination not knowing about the
modified fees of the forwarding node, thus including the outdated details in
the routehint, and the sender being unlucky and always trying with the
routehint anyway.
The long-term solutions to this is going to be #4111, this commit just reduces
the flakyness to get back to business.
We really are just interested in their on-chain footprint, so actually
starting the nodes is pointless overhead, and caused a lot of flakyness due to
the output ordering sometimes not matching up. We now just use the `bitcoind`
API to fund, sign and send a raw transaction that matches the stashed gossip
messages.
We added a conversion of failcodes that do not have sufficient information in
faac4b28ad. That means that a failcode that'd require additional information
in order to be a correct error to return in an onion is mapped to a generic
one since we can't backfill the information.
This tests that the mapping is performed correctly and replicates the
situation in #4070
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Fixes: #3926
(probably)
Changelog-Fixed: pay: Also limit the number of splits if the payee seems to have a low number of channels that can enter it, given the max-concurrent-htlcs limit.
Hooks do not tolerate failures at all. If we return a JSON-RPC error to a hook
call the only thing the main daemon can really do is to crash. This commit
adds a mapping of error to a safe fallback result, including a warning to the
node operator that this should be addressed in the plugin. The warning is
reported as a `**BROKEN**` message, and should therefore fail any testing done
on the plugin.
Changelog-Fixed: pyln: Fixed HTLCs hanging indefinitely if the hook function raises an exception. A safe fallback result is now returned instead.
This test is flaky because the generated PSBT seems to not have the change
output adjusted, or it is missing.
Tracking-Issue: ElementsProject/lightning#3998
v2 channel open uses a different method to derive the channel_id, so now
we save it to the database so that we dont have to remember how to
derive it for each.
includes a migration for existing channels
fundpsbt / utxopsbt create a (typically) output-less PSBT,
however for elements we require the fees to be encapsulated in an
output.
this patch updates fundpsbt / utxopsbt to add a fee output for elements
transactions. includes test updates.
Fixes#3998
Too trivial a fix to really list in Changelog, but I noticed that we
specified "wumbo" twice. We should really just use the proper name
in listconfigs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: plugin: `bcli` replacements should note that `sendrawtransaction` now has a second required Boolean argument, `allowhighfees`, which if `true`, means ignore any fee limits and just broadcast the transaction. Use `--deprecated-apis` to use older `bcli` replacement plugins that only support a single argument.
I screwed up the rotation logic in an earlier varient of this PR, and
it lead me to discover why test_mpp_interference_2 was flaky.
Really, we should keep a fuzzy estimator of how much payment is
outstanding, but in practice rotation is probably good enough.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This removes the reservation cleanup at startup, too, now they're all
using 'reserved_til'.
This changes test_withdraw, since it asserted that outputs were marked
spent as soon as we broadcast a transaction: now they're reserved until
it's mined. Similarly, test_addfunds_from_block assumed we'd see funds
as soon as we broadcast the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` now randomizes input and output order, not BIP69.
These tests appear in both test_misc.py and test_wallet.py.
I checked, and they're the only duplcates; they were moved in
0543149b89 then re-added (merge error?)
in 67fc8ffbcf.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
First, simplify: amount is set to 1000000, but then we deposit 1000000 + 0.01btc
(i.e. 2000000), and we always use 2 * amount. Just use a single constant to
make it clear.
Secondly, we assume that the wallet considers outputs spent as soon as
we created the tx: this will not be true once withdraw uses sendpsbt.
So, we generate blocks, but now sometimes withdraw will pick up change
txs, so we need to reserve them to avoid that messing our coinmovements.
Finally, we assumed the withdrawl order was BIP69, which becomes
variable.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Marking spent means if the transaction doesn't confirm for some
reason, the user will need to force a rescan to find the funds. Now
we have timed reservations, reserving for (an additional) 12 hours
should be sufficient.
We also take this opportunity (now we have our own callback path)
to record the tx in the wallet only on success.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some minor phrasing differences cause test changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: txprepare reservations stay across restarts: use fundpsbt/reservepsbt/unreservepsbt
Changelog-Removed: txprepare `destination` `satoshi` argument form removed (deprecated v0.7.3)
With a feerate of 7500perkw and subtracting 660 sats for anchors, a
20,000 sat channel has capacity about 9800 sat, below our default:
You gave bad parameters: channel capacity with funding 20000sat, reserves 546sat/546sat, max_htlc_value_in_flight_msat is 18446744073709551615msat, channel capacity is 9818sat, which is below 10000000msat
So bump channel amounts.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And document exactly what it does: insist that an HTLC can pass of
this value (module assumptions of feerate).
Note that we remove the "is_opener" test from the capacity calculation
for anchor fees: it doesn't matter which side it is, someone has to pay
for anchor fees to it deducts from capacity.
This change breaks the test, which we rewrite.
Changelog-Changed: config: `min-capacity-sat` is now stricter about checking usable capacity of channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS. Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's actually not possible to currently tell if you're using anchor_outputs
with a peer (since it depends on whether you both supported it at *channel open*).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-added: JSON-RPC: `listpeers` shows `features` list for each channel.
We're failing this too often: we'd fail it more but it's disabled
with VALGRIND (it shouldn't be: @slow_test removes VALGRIND if SLOW_MACHINE
is set).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the simplest possible fix: increase the target amount until we get
the desired number of parts, while still bucketizing payments together that
are in approximately the same size.
The current logic puts all payments that are in the range x < amount <= 16*x
in the same bucket, making them harder to distinguish.
Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
You can't pay them anyway, and at least one person used 0 instead of "any".
Closes: #3808
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `invoice` no longer accepts zero amounts (did you mean "any"?)
This is an extra safety check for dual funding, where we only want to sign
the inputs we provided!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `signpsbt` takes an optional `signonly` array to limit what inputs to sign.
Required for dual funding where the opener sets it.
Changelog-Added: JSON-RPC: `fundpsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Anchor outputs break many assumptions in our tests:
1. Remove some hardcoded numbers in favor of a fee calc, so we only have to
change in one place.
FIXME: This should also be done for elements!
2. Do binary search to get feerate for a given closing fee.
3. Don't assume output #0: anchor outputs perturb them.
4. Don't assume we can make 1ksat channels (anchors cost 660 sats!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's currently always 0, but it won't be once we replace txprepare.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel` has new `outnum` field indicating which output of the transaction funds the channel.
Reported-by: ZmnSCPxj
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
The worst effect is that unpublished nodes are harder to pay, but
even published ones make us do unnecessary work, since we are
losing routehints from the published ones that could help us
actually route better to them.
On my test machine, we queried bitcoind before the close tx was sent:
```
# When output is spent, it should give us null !
txo = l1.rpc.call("getutxout", {"txid": txid, "vout": 0})
> assert txo["amount"] is txo["script"] is None
E AssertionError: assert '20000000msat' is '00205b8cd3b914cf67cdd8fa6273c930353dd36476734fbd962102c2df53b90880cd'
tests/test_plugin.py:1221: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It might have already happened, and anyway, we wait for the actual state
below.
```
# make database snapshot of l2
l2.stop()
l2_db_path = os.path.join(l2.daemon.lightning_dir, chainparams['name'], 'lightningd.sqlite3')
l2_db_path_bak = os.path.join(l2.daemon.lightning_dir, chainparams['name'], 'lightningd.sqlite3.bak')
copyfile(l2_db_path, l2_db_path_bak)
l2.start()
sync_blockheight(bitcoind, [l2])
# push some money from l3->l2, so that the commit counter advances
l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
> l2.daemon.wait_for_log('now ACTIVE')
tests/test_closing.py:908:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:288: in wait_for_log
return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.testing.utils.LightningD object at 0x7f0c145a32d0>
regexs = ['now ACTIVE'], timeout = 60
def wait_for_logs(self, regexs, timeout=TIMEOUT):
"""Look for `regexs` in the logs.
We tail the stdout of the process and look for each regex in `regexs`,
starting from last of the previous waited-for log entries (if any). We
fail if the timeout is exceeded or if the underlying process
exits before all the `regexs` were found.
If timeout is None, no time-out is applied.
"""
logging.debug("Waiting for {} in the logs".format(regexs))
exs = [re.compile(r) for r in regexs]
start_time = time.time()
pos = self.logsearch_start
while True:
if timeout is not None and time.time() > start_time + timeout:
print("Time-out: can't find {} in logs".format(exs))
for r in exs:
if self.is_in_log(r):
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('now ACTIVE')]" in logs.
contrib/pyln-testing/pyln/testing/utils.py:264: TimeoutError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The modded_psbt is not necessarily invalid:
# Try a modified (invalid) PSBT string
modded_psbt = psbt[:-3] + 'A' + psbt[-3:]
with pytest.raises(RpcError, match=r"should be a PSBT, not"):
> l1.rpc.signpsbt(modded_psbt)
E AssertionError: Pattern 'should be a PSBT, not' not found in 'RPC call failed: method: signpsbt, payload: {'psbt': 'cHNidP8BAM0CAAAABJ9446mTRp/ml8OxSLC1hEvrcxG1L02AG7YZ4syHon2sAQAAAAD9////JFJH/NjKwjwrP9myuU68G7t8Q4VIChH0KUkZ5hSAyqcAAAAAAP3///8Uhrj0XDRhGRno8V7qEe4hHvZcmEjt3LQSIXWc+QU2tAEAAAAA/f///wstLikuBrgZJI83VPaY8aM7aPe5U6TMb06+jvGYzQLEAQAAAAD9////AcDGLQAAAAAAFgAUyQltQ/QI6lJgICYsza18hRa5KoEAAAAAAAEBH0BCDwAAAAAAFgAUqc1Qh7Q5kY1noDksmj7cJmHaIbQAAQEfQEIPAAAAAAAWABS3bdYeQbXvBSryHNoyYIiMBwu5rwABASBAQg8AAAAAABepFD1r0NuqAA+R7zDiXrlP7J+/PcNZhwEEFgAUKvGgVL/ThjWE/P1oORVXh/ObucYAAQEgQEIPAAAAAAAXqRRsrE5ugA1VJnAith5+msRMUTwl8ocBBBYAFMrfGCiLi0ZnOCY83ERKJ1sLYMY8AAAA='}, error: {'code': -1, 'message': 'No wallet inputs to sign'}'
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
listpays: make doc-all missed
Changelog-Added: JSON-RPC: `listpays` can be used to query payments using the `payment_hash`
Changelog-Added: JSON-RPC: `listpays` now includes the `payment_hash`
And when it's set, and we're SLOW_MACHINE, simply disable valgrind.
Since Travis (SLOW_MACHINE=1) only does VALGRIND=1 DEVELOPER=1 tests,
and VALGRIND=0 DEVELOPER=0 tests, it was missing tests which needed
DEVELOPER and !VALGRIND.
Instead, this demotes them to non-valgrind tests for SLOW_MACHINEs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I started replacing all get_node() calls, but got bored, so then just did the
tests which call get_node() 3 times or more.
Ends up not making a measurable speed difference, but it does make some
things neater and more standard.
Times with SLOW_MACHINE=1 (given that's how Travis tests):
Time before (non-valgrind):
393 sec (had 3 failures?)
Time after (non-valgrind):
410 sec
Time before (valgrind):
890 seconds (had 2 failures)
Time after (valgrind):
892 sec
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I thought this was timing out because I made it slow with the
change to txprepare as a plugin. In fact, it was timing out
because sometimes gossip comes so fast it gets suppressed
and we never get the log messags.
Still, before this it took 98 seconds under valgrind and
24 under non-valgrind, so it's an improvement to time as
well as robustness.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: pay: Fixed a bug where routehints would be ignored if the payment exceeded 10,000 satoshi. This is particularly bad if the payee is only reachable via routehints in an invoice.
Changelog-Added: New option `--important-plugin` loads a plugin is so important that if it dies, `lightningd` will exit rather than continue. You can still `--disable-plugin` it, however, which trumps `--important-plugin` and it will not be started at all.
the way we use PSBTs to sign things requires that we have the
scriptpubkey available on the utxo so we can populate the witness-utxo
field with it.
this causes problems if we don't already have the scriptpubkey cached in
the database, as in *some* cases we require a round trip to the HSM to
populate them
to get over this hump, we backfill any and all missing scriptpubkey
information for the utxo's that we hold in our wallet.
this will allow us to clean up the NULL handling of missing
scriptpubkeys.
```
# Excludes channel, then ignores routehint which includes that, then
# it excludes other channel.
> assert len(status) == 2
E assert 1 == 2
E -1
E +2
```
The invoice we use at the end has a routehint: 50% of the time it's
to l2 (which fails), 50% to l5 (which succeeds).
Change it to create invoice before channel with l5 so it does the
retry like we expect here.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We sum up the amounts just like we do with `amount_sent`, however we may have
parts that don't have that annotation, so make it optional.
Suggested-by: Rusty Russell <@rustyrussell>
Since we started using `sendonion` in the `pay` plugin we no longer
automatically have the `amount` annotation on (partial) payments. This
replicates the issue so we can fix it.
Reported-by: Rusty Russell <@rustyrussell>
The adaptive MPP test was showing an issue with always using a routehint, even
when it wasn't necessary: we would insist on routhing to the entrypoint of the
routehint, even through the actual destination. If a channel on that loop
would result being over capacity we'd slam below 0, and then increase again by
unapplying the route. The solution really is not to insist on routing through
a routehint, so we implement random skipping of routehints, and we rotate them
if we have multiples.
Changelog-Fixed: plugin: `bcli` no longer logs a harmless warning about being unable to connect to the JSON-RPC interface.
Changelog-Added: plugin: Plugins can opt out of having an RPC connection automatically initialized on startup.
This PR includes the fix discussed on PR #3855. This fix was tested with the use case described inside the issue and worked.
Fixes: #3855
Changelog-None
The test had part 1 and 2 backward, but still worked. When I copied that to
*after* the test had succeeded, it complained. It should always complain,
to catch bugs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This wasn't important before, but now we have MPP it's good to enforce.
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
While we were unsetting the `payment->cmd` in case of a success to signal that
we should not return to the JSON-RPC command twice, we were not doing that in
the case of failures. This was causing multiple responses to a single incoming
command, and `lightningd` was correctly killing the plugin. This issue was
introduced through early returns (anything setting `payment->abort=true`) and
was caused in Rusty's case through an MPP timeout.
Fixes#3847
Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
fundpsbt forces the caller to manually add their weight * feerate
to the satoshis they ask for. That means no named feerates.
Instead, create a startweight parameter and do the calc for them
internally, and return the feerate we used (and, while we're at it,
the estimated final weight).
This API change is best done now, as it would otherwise have to
be appended as a parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This exercises something that is simply not possible without MPP, i.e., the
bundling of multiple paths to get sufficient capacity to perform the payment.
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
We're actually going to deprecate this, so don't add new features!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: ***REMOVE*** JSON-API: `txprepare` returns a psbt version of the created transaction
It uses reservations heavily, and assumed we generated change, etc.
It's now a simpler test, in many ways.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
reserveinputs marks UTXOs reserved for 12 hours, so we won't select them
for spending: unreserveinputs marks them available again.
Exposes param_psbt() for wider use.
Disabled the test_sign_and_send_psbt since we're altering the API;
the final patch restores it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Technically an API break, but nobody relies on these I hope!
Note that the feerates warning was buried inside the style object:
it should be top-level.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.
Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.
Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
This makes use of the payment modifier structure to just add the preimage to
the TLV payload for the last hop.
Changelog-Added: JSON-RPC: The `keysend` command allows sending to a node without requiring an invoice first.
This commit collects the changes required to the tests caused by the changes
to the `pay` and `paystatus` commands. They are also rather good hints as to
what these changes entail.
Since we change the interface for `pay` and `paystatus` we need a way to
determine which one to expect. This just parses the COMPAT_Vxyz flags and
allows us to check if, based on the configuration, we are in compat mode for
that set of changes or not.
Make sure we've actually confirmed the HTLC; if it's not confirmed yet
then we won't fast-fail it, and we'll timeout instead:
```
> l1.rpc.waitsendpay(payment_hash=inv['payment_hash'], timeout=TIMEOUT, partid=1)
E AssertionError: Pattern 'WIRE_PERMANENT_CHANNEL_FAILURE \\(reply from remote\\)' not found in "RPC call failed: method: waitsendpay, payload: {'payment_hash': 'c186643391469aa8190415496c85b1eb789cb2b756a76d4c9ce21dd34c698d92', 'timeout': 30, 'partid': 1}, error: {'code': 200, 'message': 'Timed out while waiting'}"
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It looked like we weren't printing the address on closing outputs.
But we are, because the 'scriptPubkey' field is in the 'outputs' db
table since 0.7.3 (66a47d2761).
So make the logic clearer, and remove a completely bogus comment (UTXOs
with closing_info are definitely spendable!).
We export the json_add_utxos() for future use, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are pulled from wallet/wallet.c, with the fix now that we grind sigs.
This reduces the fees we pay slightly, as you can see in the coinmoves changes.
I now print out all the coin moves in suitable format before we match:
you only see this if the test fails, but it's really helpful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By setting nLocktime to the current block, the reorg test
"test_funding_reorg_remote_lags" actually drops the funding transaction
entirely when a reorg happens.
Except the 1 in 10 cases where nLocktime is randomly set to 1-10
blocks earlier.
This implies, strongly, that we hit "restart" too often on Travis.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There is a race between `getroute` learning that our peer accepts TLVs and us
initiating the payment. Waiting for announcements ensures we always use TLVs,
matching our expectation in the test / plugin.
Changelog-Changed: `txprepare` now prepares transactions whose `nLockTime` is set to the tip blockheight, instead of using 0. `fundchannel` will use `nLockTime` set to the tip blockheight as well.
The status of what started as a simple JSON-RPC call is now spread across an
entire tree of partial payments and payment attempts. So we collect the status
in a single struct in order to report back success of failure.
This commit can be reverted/skipped once we have implemented all the logic and
have feature parity with the normal `pay`. It's main purpose is to expose the
unfinished functionality to test it, without completely breaking the existing
`pay` command.
If the daemon already knows about the channel before it was stopped,
it won't get this message from gossipd. That's OK, since we explicitly
test for the channel being active two lines down.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our existing coin_moves tracking logic assumed that any tx we had an
input in belonged to *all* of our wallet (not a bad assumption as long
as there was no way to update a tx that spends our wallets)
Now that we've got `signpsbt` implemented, however, we need to be
careful about how we account for withdrawals. For now we do a best guess
at what the feerate is, and lump all of our spent outputs as a
'withdrawal' when it's impossible to disambiguate
Changelog-Added: JSON-RPC: new call `signpsbt` which will add the wallet's signatures to a provided psbt
Changelog-Added: JSON-RPC: new call `sendpsbt` which will finalize and send a signed PSBT
Reserve and unreserve wallet UTXOs using a PSBT which includes those
inputs.
Note that currently we unreserve inputs everytime the node restarts.
This will be addressed in a future commit.
Changelog-Added: JSON-RPC: Adds two new rpc methods, `reserveinputs` and `unreserveinputs`, which allow for reserving or unreserving wallet UTXOs
Changelog-Changed: `fundchannel_cancel` will now succeed even when executed while a `fundchannel_complete` is ongoing; in that case, it will be considered as cancelling the funding *after* the `fundchannel_complete` succeeds.
Let me introduce the concept of "Sequential Consistency":
All operations on parallel processes form a single total order agreed upon by all processes.
So for example, suppose we have parallel invocations of `fundchannel_complete` and `fundchannel_cancel`:
+--[fundchannel_complete]-->
|
--[fundchannel_start]-+
|
+--[fundchannel_cancel]---->
What "Sequential Consistency" means is that the above parallel operations can be serialized as a single total order as:
--[fundchannel_start]--[fundchannel_complete]--[fundchannel_cancel]-->
Or:
--[fundchannel_start]--[fundchannel_cancel]--[fundchannel_complete]-->
In the first case, `fundchannel_complete` succeeds, and the `fundchannel_cancel` invocation also succeeds, sending an `error` to the peer to make them forget the chanel.
In the second case, `fundchannel_cancel` succeeds, and the succeeding `fundchannel_complete` invocation fails, since the funding is already cancelled and there is nothing to complete.
Note that in both cases, `fundchannel_cancel` **always** succeeds.
Unfortunately, prior to this commit, `fundchannel_cancel` could fail with a `Try fundchannel_cancel again` error if the `fundchannel_complete` is ongoing when the `fundchannel_cancel` is initiated.
This violates Sequential Consistency, as there is no single total order that would have caused `fundchannel_cancel` to fail.
This commit is a minimal patch which just reschedules `fundchannel_cancel` to occur after any `fundchannel_complete` that is ongoing.
We passed below the floor when the user specified `1000perkb`.
Matt Whitlock says :
I was withdrawing with feerate=1000perkb, which should be the minimum-allowed fee rate. Indeed, bitcoin-cli getmempoolinfo reports:
{
"loaded": true,
"size": 15097,
"bytes": 9207924,
"usage": 32831760,
"maxmempool": 64000000,
"mempoolminfee": 0.00001000,
"minrelaytxfee": 0.00001000
}
Changelog-fixed: rpc: The `feerate` parameters now correctly handle the standardness minimum when passed as `perkb`.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Reported-by: Matt Whitlock
We erase peer data after the last channel close transaction for that
peer is 100 blocks deep. We were failing to finish the migration because
the peer_id lookup on these was failing.
Now we ignore any channel with a null peer_id.
Fixes#3768
We use a database snapshot with 3 channels -- two of which have HTLCs
dangling and one is an initial open channel tx in the 'old' tx hex
format in last_tx and confirm that they are successfully updated to PSBT
format on start.
Spec is wrong (it says it should be compulsory), and Eclair doesn't set it
at all, leading to an error when they send their announcement_signatures.
Fixes: #3703
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: large-channels: negotiate successfully with Eclair nodes.
There are various places where our tests failed with
--enable-expimental-features. And our plugin test overlapped an
existing feature.
We make our expected_feature functions more generic, and use them
everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Much nicer for grepping, since `{ "foo": { "bar": [7] } }` is turned into
`foo.bar[0]=7`.
Changelog-Added: cli: New `--flat` mode for easy grepping.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
-H removes the top-level if there's only one, and 'format-hint'
breaks this heuristic, so we end up with:
```
help=command=autocleaninvoice [cycle_seconds] [expired_by]
category=plugin
description=Set up autoclean of expired invoices.
verbose=Perform cleanup every {cycle_seconds} (default 3600), or disable autoclean if 0. Clean up expired invoices that have expired for {expired_by} seconds (default 86400).
command=check command_to_check
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We've been seeing some Travis timeouts under VALGRIND, with the
10 second timeout here: use TIMEOUT as per standard.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's possible for our peer to publish a commitment tx that has already
updated our balance for an htlc before we've completed removing it from
our commitment tx (aka before we've updated our balance). This used to
crash, now we just update our balance (and the channel balance logs!)
and keep going.
If they've removed anything from our balance, we'll end up counting it
as chain_fees below. Not ideal but fine... probably.
This moves the notification for our coin spends from when it's
successfully submited to the mempool to when they're confirmed in a
block.
We also add an 'informational' notice tagged as `spend_track` which
can be used to track which transaction a wallet output was spent in.
Previously we were annotating every movement with the blockheight of
lightningd at notification time. Which is lossy in terms of info, and
won't be helpful for reorg reconciliation. Here we switch over to
logging chain moves iff they've been confirmed.
Next PR will fix this up for withdrawals, which are currently tagged
with a blockheight of zero, since we log on successful send.
On node start we replay onchaind's transactions from the database/from
our loaded htlc table. To keep things tidy, we shouldn't notify the
ledger about these, so we wrap pretty much everything in a flag that
tells us whether or not this is a replay.
There's a very small corner case where dust transactions will get missed
if the node crashes after the htlc has been added to the database but
before we've successfully notified onchaind about it.
Notably, most of the obtrusive updates to onchaind wrappings are due to
the fact that we record dust (ignored outputs) before we receive
confirmation of its confirmation.
Mostly we update existing tests to account for channel balances. In a
few places, new tests were needed as there wasn't an existing pathway
that tested the chain-fees for a few penalty cases
For cheats, we do a little bit of weird accounting. First we 'update'
our on-ledger balance to be the entirety of the channel's balance. Then,
as outputs get resolved, we record the fees and outputs as withdrawals
from this amount.
It's possible that they might successfully 'cheat', in which case we
record those as 'penalty' but debits (not credits).
The previous patch changed the gossip_store, but in a trivial way.
The next patch will implement upgrading, so this is the test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And the percentage of the initial amount, not the constently increasing
one !
Changelog-Fixed: pay: we now respect maxfeepercent, even for tiny amounts.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
The previous implementation was a bit lazy: in particular, since we didn't
remember the disabled plugins, we would load them on rescan.
Changelog-Changed: config: the `plugin-disable` option works even if specified before the plugin is found.
That's more convenient for most callers, which don't need a fmt.
Fixed-by: Darosior <darosior@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is what I expected from plugin_kill, and now all the callers do the
equivalent anywat, it's easy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we now clean up options in startup plugins (that was only
done by dynamic code!), and now they both share the 60 second timeout
instead of 20 seconds for dynamic.
For the dynamic case though, it's 60 seconds to both complete
getmanifest and init, which seems fair.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We modify the slow_init() so it doesn't go too slowly for this test.
This demonstrates a crash, where we currently try to fail a command
multiple times.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This actually passes fine, but it's an interesting case to test.
Fixed-by: Darosior <darosior@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Telling `lightningd` to pass a `-datadir` to `bitcoin-cli` so it doesn't go
snooping where it doesn't belong (i.e., the user's home directory and config).
Changelog-None
Suggested-by: Simon Vrouwe <@SimonVrouwe>
Signed-off-by: Christian Decker <@cdecker>
I noticed the following in logs for tests/test_connection.py::test_feerate_stress:
```
DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Failing HTLC 18446744073709551615 due to peer death
DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: local_routing_failure: 8194 (WIRE_TEMPORARY_NODE_FAILURE)
```
This is because it reports the (transient) node_failure error, because
our channel_failure message is incomplete. Fix this wart up.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't wait for close tx to reach mempool, it might not get to
depth 100, and we don't get 'onchaind complete, forgetting peer'.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't wait until l2 processed the final state of HTLC #2, so
it might not include it when it drops onchain, leading to us only
getting 3 (not 4) sendrawtx calls.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously we've used the term 'funder' to refer to the peer
paying the fees for a transaction; v2 of openchannel will make
this no longer true. Instead we rename this to 'opener', or the
peer sending the 'open_channel' message, since this will be universally
true in a dual-funding world.
They were looking for specific amounts which proved to be rather flaky. Now
they look for specific outputs being available in the `listfunds` result after
everything was settled.
The new `keysend` plugin modifies the node features that we send to
peers. This commit breaks out the 'expected_features' we use for tests
to encompass this differentiation.
The documentation was wrong, and I copied my mistake to `libplugin` where it
was then ignored instead of ORed into the node's featurebits. This fixes both.
As discussed with Christian, prepending the length to the payload returned
is awkward, but it's the only way to set a legacy payload. As this will
be soon deprecated, simplify the external API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We weren't actually waiting until l3 got the channel_update from l2,
so it might not be able to create the routehint.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When route returns a result which is too expensive, we try to figure out which
hop is most expensive to exclude it for next time.
If it's a single-hop route, we don't count it, since the first hop is free.
That's not usually a problem, since single-hop routes can't exceed our limits
(they're always "free"!).
But if we are using a routehint, the total cost could exceed our limits,
even if the start of the routehint is a single hop away.
This reproduces that test case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes testing easier, and makes sense: lightningd might not
*know* about other connected channels, depending on gossip, but if the
user specifies it we should obey it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON: `invoice` `exposeprivatechannels` now includes explicitly named channels even if they seem like dead-ends.
This is what actually lets us pay blinded invoices.
Unfortunately, our internal logic assumes every hop in a path has a
next `short_channel_id`, so we have to use a dummy. This is
sufficient for testing, however.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sending update_fee immediately after channel establishment seems to
upset LND, so work around it by deferring it. The reason we increase
the fee after establishment is because now we might need to close the
channel in a hurry due to htlcs, but until there are htlcs that's
unnecessary.
Fixes: #3596
Changelog-Changed: Added workaround for lnd rejecting our commitment_signed when we send an update_fee after channel confirmed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now track all pending RPC passthrough calls, and terminate them with an
error if the plugin dies.
Changelog-Fixed: JSON-RPC: Pending RPC method calls are now terminated if the handling plugin exits prematurely.
The spec states that invoices with an amount, but lacking a multiplier, should
be interpreted as integer Bitcoin amounts:
`amount`: optional number in that currency, followed by an optional
`multiplier` letter. The unit encoded here is the 'social' convention of a
payment unit -- in the case of Bitcoin the unit is 'bitcoin' NOT satoshis.
Suggested-by: Stefano Pellegrini <@St333p>
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: invoice: The invoice parser assumed that an amount without a multiplier was denominated in msatoshi instead of bitcoins.
We had them split according the separate use-cases:
- testing
- doc-gen
- wire-gen
But that was causing new contributors to miss some dependencies when they
first got hacking. So this consolidates all of our own dependencies in a root
requirements.txt, with the notable exception of `pyln-client`, `pyln-testing`
and `pyln-proto` which are distributed as PyPI modules and therefore have
their own dependencies that need to be tracked in the module root.
Closes#3518
Note that now we check capacity once we've figured out which peer, which
broke a test (we returned "unknown peer" instead of "capacity exceeded"),
so we rework that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Shows what features we use in various contexts, including those added
by plugins in getmanifest.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugin: `feature_set` object added to `init`
A CONSERVATIVE/3 target for them.
Some noisy changes to the tests as we had to update the estimatesmartfee
mock.
Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
Changelog-Fixed: Passing 0 as minconf to withdraw allows you to use unconfirmed transaction outputs, even if explicitly passed as the `utxos` parameter
This is to prepare for dynamic features, including making plugins first
class citizens at setting them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This restriction was removed from the spec as of
86c2ebcc5973a4133d3ce4d80ae1c203061a1646.
We also fix up some strange formatting in that part of the documentation.
Changelog-changed: We now announce multiple addresses of the same type, if given.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON: `fundchannel` and `fundchannel_start` `satoshi` parameter removed (renamed to `amount` in 0.7.3).
This adapts our fee estimations requests to the Bitcoin backend to the
new semantic, and batch the requests.
This makes our request for fees much simpler, and leaves some more
flexibility for a plugin to do something smart (it could still lie before
but now it's explicit, at least.) as we don't explicitly request
estimation for a specific mode and a target.
Changelog-Changed: We now batch the requests for fee estimation to our Bitcoin backend.
Changelog-Changed: We now get more fine-grained fee estimation from our Bitcoin backend.
We kept track of an URGENT, a NORMAL, and a SLOW feerate. They were used
for opening (NORMAL), mutual (NORMAL), UNILATERAL (URGENT) transactions
as well as minimum and maximum estimations, and onchain resolution.
We now keep track of more fine-grained feerates:
- `opening` used for funding and also misc transactions
- `mutual_close` used for the mutual close transaction
- `unilateral_close` used for unilateral close (commitment transactions)
- `delayed_to_us` used for resolving our output from our unilateral close
- `htlc_resolution` used for resolving onchain HTLCs
- `penalty` used for resolving revoked transactions
We don't modify our requests to our Bitcoin backend, as the next commit
will batch them !
Changelog-deprecated: The "urgent", "slow", and "normal" field of the `feerates` command are now deprecated.
Changelog-added: The fields "opening", "mutual_close", "unilateral_close", "delayed_to_us", "htlc_resolution" and "penalty" have been added to the `feerates` command.
pytest captures the output by monkey patching out `sys.stdout`. This may
conflict with our use of `sys.stdout` when configuring logging, resulting in
the "Write to closed file" issue that is spamming the logs. By making the
logging configuration a fixture hopefully we always use the correct
stdout (after pytest has monkey-patched it).
Some tests may not spawn a node at all, so make sure that our assumption that
the directory exists in the fixture cleanup is correct by creating the
directory.
a few things. one is that `is_in_log` returns a result rather than
enforcing a condition. so these lines all need asserts
two is that with the 'allow_deprecated_apis' option on, the python json
parser overwrites the now typed input with the later-added string
version, so the only option value present in the option key-value set is
the last, string one. the check for this has been updated to only verify
that the string version is included (i manually verified that both are
printed to the JSON message)
We were nesting like the following:
```json
{"params": {
"rpc_command": {
"rpc_command": {
}
}
}
```
This is really excessive, so we unwrap once, and now have the following:
```json
{"params": {
"rpc_command": {
}
}
```
Still more wrapping than necessary (the method is repeated in the `params`
object), but it's getting closer.
Changelog-Deprecated: JSON-RPC: Removed double wrapping of `rpc_command` payload in `rpc_command` JSON field.
Suggested-by: @fiatjaf
Signed-off-by: Christian Decker <@cdecker>
Before this patch we would only update `channel->last_tx` with the newly
proposed closure tx from the peer if the fee of the new one was lower.
In negotiations where we are at the higher end and the peer starts
lower, all peer's subsequent proposals will be higher than his initial
proposal and in this case we would never update `channel->last_tx`
and would wrongly broadcast his initial proposal at the end of the
negotiation.
Fixes https://github.com/ElementsProject/lightning/issues/3549
Changelog-Fixed: Always broadcast the latest close transaction at the end of the close fee negotiation, instead of sometimes broadcasting the peer's initial closing proposal.
We are returning a `BADONION` error despite the cause being an invalid onion
payload containing an unknown even TLV type. It really should return
`INVALID_ONION_PAYLOAD` errors instead.
ChangeLog-Added: New `getsharedsecret` command, which lets you compute a shared secret with this node knowing only a public point. This implements the BOLT standard of hashing the ECDH point, and is incompatible with ECIES.
Instead of saving a stripped_update, we use the new
local_fail_in_htlc_needs_update.
One minor change: we return the more correct
towire_temporary_channel_failure when the node is still syncing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For bitcoind_fail_first:
We only ever send `getblock` if we got a successful block hash from
`getblockhash`, and if we can't get the block in that case it means
our Bitcoin backend is faulty and we shouldnt continue.
So, mock `getblockhash` instead, which is authorized to spuriously fail.
For both bitcoind_fail_first and bitcoind_failure:
Adapt the logs.
Add new check if we're funder trying to add HTLC, keeping us
with enough extra funds to pay for another HTLC the peer might add.
We also need to adjust the spendable_msat calculation, and update
various tests which try to unbalance channels. We eliminate
the now-redundant test_channel_drainage entirely.
Changelog-Fixed: Corner case where channel could become unusable (https://github.com/lightningnetwork/lightning-rfc/issues/728)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make the `htlc_accepted` hook the first chained hook in our repertoire. The
plugins are called one after the other in order until we have no more plugins
or the HTLC was handled by one of the plugins. If no plugins handles the HTLC
we continue to handle it internally like always.
Handling in this case means the plugin returns either `{"result": "resolve",
...}` or `{"result": "fail", ...}`.
Changelog-Changed: plugin: Multiple plugins can now register for the htlc_accepted hook.
This is the last venue we need to add custom featurebits to, so we also unmark
the test as xfail.
Changelog-Added: plugin: Plugins can now signal support for experimental protocol extensions by registering featurebits for `node_announcement`s, the connection handshake, and for invoices. For now this is limited to non-dynamic plugins only
restrict fundchannel_cancel usage to only the opener side
Changelog-Changed: Only the opener of a fundchannel can cancel the channel open with fundchannel_cancel
As a separated commit because it was pre-existent (changelog + xfail test).
This also fix a logical problem in lightningd/plugin_control: we were
assuming a plugin started with 'plugin start' but which did not comport
a 'dynamic' entry in its manifest to be dynamic, though it should have
been treated as static.
Changelog-fixed: plugins: Dynamic C plugins can now be managed when lightningd is up
Using it with a different value to the amount sent causes a crash in 0.8.0,
which is effectively deprecating it, so let's disallow it now.
Changelog-Changed: If the optional `msatoshi` param to sendpay for non-MPP is set, it must be the exact amount sent to the final recipient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Relative plugin paths are not relative to startup (deprecated v0.7.2.1)
Changelog-Removed: Dummy fields in listforwards (deprecated v0.7.2.1)
1. We asserted that there wouldn't be a raw failcode.
2. We didn't pass the failure information via JSON in this case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We could use sendonion to do this, but it actually takes a different path through
pay, and I wanted to test all of it, so I made a new dev flag.
We currently get upset with the response:
lightningd/pay.c:556: payment_failed: Assertion `!hout->failcode' failed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This completes the custommsg epic, finally we are back where we began all that
time ago (about 4 hours really...): in a plugin that implements some custom
logic.
This solves a couple of issues with the need to synchronously drop the
connection in case we were required to understand what the peer was talking
about while still allowing users to experiment, just not kill connections.
The number of outputs got updated, but the map used to calculate the
change output's location did not (still assumes only one output). This
patch fixes this to make the output map a variable size.
Changelog-Fixed: JSON API: `txprepare` no longer crashes when more than two outputs are specified
Changelog-Fixed: Detect a previously non-permanent error (`final_cltv_too_soon`) that has been merged into a permanent error (`incorrect_or_unknown_payment_details`), and retry that failure case in `pay`.
This is needed to fully implement handling of blockheight disagreements
between us and payee.
If payee believes the blockheight is higher than ours, then `pay`
should wait for our node to achieve that blockheight.
Changelog-Add: Implement `waitblockheight` to wait for a specific blockheight.
Changelog-Changed: plugin: `notify_sendpay_success` and `notify_sendpay_failure` are now always called, even if there is no command waiting on the result.
We clone the test above, but this time we don't attach waiters (they'd be racy
anyway), and we wait for the notification to be called. This fails, but is
fixed in the next two commits.
Valgrind doesn't really like crashes if compiled without DEVELOPER since that
seems to compile out the debug symbols, resulting in the following error:
```
Optimistic lock on the database failed. There may be a concurrent access to the database. Aborting since concurrent access is unsafe.
lightningd: FATAL SIGNAL 6 (version 0.0.99)
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd: FATAL SIGNAL 11 (version 0.0.99)
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
lightningd/lightningd: libbacktrace: no debug info in ELF executable
2020-01-07 15:26:03.539 EST [11583] LOG: unexpected EOF on client connection with an open transaction
--------------------------- Captured stdout teardown ---------------------------
DEBUG:root:Calling stop with payload None
------------------------------- Valgrind errors --------------------------------
Valgrind error file: valgrind-errors.11409
==11409== Jump to the invalid address stated on the next line
==11409== at 0x0: ???
==11409== by 0x1C00A8: backtrace_full (backtrace.c:127)
==11409== by 0x147B0A: send_backtrace (daemon.c:46)
==11409== by 0x147B55: crashdump (daemon.c:54)
==11409== by 0x6071F1F: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==11409== by 0x6071E96: __libc_signal_restore_set (nptl-signals.h:80)
==11409== by 0x6071E96: raise (raise.c:48)
==11409== by 0x6073800: abort (abort.c:79)
==11409== by 0x12B2FF: fatal (log.c:819)
==11409== by 0x16FA3B: db_data_version_incr (db.c:826)
==11409== by 0x16FA9E: db_commit_transaction (db.c:841)
==11409== by 0x124D20: io_loop_with_timers (io_loop_with_timers.c:34)
==11409== by 0x129260: main (lightningd.c:860)
==11409== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==11409==
==11409==
==11409== Process terminating with default action of signal 11 (SIGSEGV)
==11409== Bad permissions for mapped region at address 0x0
==11409== at 0x0: ???
==11409== by 0x1C00A8: backtrace_full (backtrace.c:127)
==11409== by 0x147B0A: send_backtrace (daemon.c:46)
==11409== by 0x147B55: crashdump (daemon.c:54)
==11409== by 0x6071F1F: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
--------------------------------------------------------------------------------
```
This lets us do more flexible filtering in the next patch. But it also
keeps some weird logic out of gossipd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The optimistic lock prevents multiple instances of c-lightning making
concurrent modifications to the database. That would be unsafe as it messes up
the state in the DB. The optimistic lock is implemented by checking whether a
gated update on the previous value of the `data_version` actually results in
an update. If that's not the case the DB has been changed under our feet.
The lock provides linearizability of DB modifications: if a database is
changed under the feet of a running process that process will `abort()`, which
from a global point of view is as if it had crashed right after the last
successful commit. Any process that also changed the DB must've started
between the last successful commit and the unsuccessful one since otherwise
its counters would not have matched (which would also have aborted that
transaction). So this reduces all the possible timelines to an equivalent
where the first process died, and the second process recovered from the DB.
This is not that interesting for `sqlite3` where we are also protected via the
PID file, but when running on multiple hosts against the same DB, e.g., with
`postgres`, this protection becomes important.
Changelog-Added: DB: Optimistic logging prevents instances from running concurrently against the same database, providing linear consistency to changes.
Some Linux OSs impose a length limit on the path a Unix socket may have. This
is not an issue in `lightningd` since we `chdir()` into that directory before
opening the socket, however in pyln this became a problem for some tests,
since we use absolute paths in the testing framework. It's also a rather
strange quirk to expose to users.
This patch introduces a `UnixSocket` abstraction that attempts to work around
these limitations by aliasing the directory containing the socket into
`/proc/self/fd` and then connecting using that alias.
It was inspired by Open vSwitch code here https://github.com/openvswitch/ovs/blob/master/python/ovs/socket_util.py
Signed-off-by: Christian Decker <@cdecker>
it's that time of year (merry xmas!)
enables the ability to push_msat on fundchannel
Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
Trying to `pip install psycopg2-binary==2.8.3` raised an error for a
long time. Since version `2.8.4` is recent and also seem to work I
suggest updating the requirements so other users dont run into the
following error, where pip tries to use `pg_config` for the BINARY
package:
Collecting psycopg2-binary==2.8.3 (from -r contrib/pyln-testing/requirements.txt (line 6))
Using cached 91911be018dd876c1e0f12e1b5fb90/psycopg2-binary-2.8.3.tar.gz
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-zapr0fhs/psycopg2-binary/
Complete output (23 lines):
running egg_info
creating pip-egg-info/psycopg2_binary.egg-info
writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Changelog-None
We still close the channel if we *send* an error, but we seem to have hit
another case where LND sends an error which seems transient, so this will
make a best-effort attempt to preserve our channel in that case.
Some test have to be modified, since they don't terminate as they did
previously :(
Changelog-Changed: quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Thanks to @t-bast, who made this possible by interop testing with Eclair!
Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an intermediary step: we still don't save it to the database,
but we do use the fee_states struct to track it internally.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Bastien TEINTURIER <bastien@acinq.fr> writes:
> It looks like the split on c-lightning side is quite limited at the moment:
> the only option is to split a payment in exactly its two halves,
> otherwise I get rejected because of the rule of overpaying more than
> twice the amount?
We only tested exactly two equal-size payments; indeed, our finalhop
test was backwards. We only complain if the final hop pays more than
twice msat (technically, this test is still too loose for mpp: the
spec says we should sum to the exact amount).
Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This won't usually be visible to the end-user, since the pay plugin doesn't
do multi-part yet (and mpp requires EXPERIMENTAL_FEATURES), but we're ready
once it does.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The invoice_try_pay code now takes a set, rather than a single htlc, but
it's basically the same thing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Explicit #if EXPERIMENTAL_FEATURES check in case we enable them at different
times, but it requires a payment_secret since we put them in the same field.
This incidently stops it working on legacy nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
msatoshi was used to indicate the amount the invoice asked for, but
for parallel sendpay it's required, as it allows our sanity check of
limiting the total payments in flight, ie. it becomes
'total_msat'.
There's a special case for sendonion, which always tells us the value is 0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently refuse a payment if one is already in flight. For parallel
payments, it's a bit more subtle: we want to refuse if it we already have
the total-amount-of-invoice in flight.
So we get all the current payments, and sum the pending ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were hardcoding the chainparams->chain_hash which caused the query to
return an empty result. By parametrizing the test we can make it work on
elements.
Now "raw_payload" is always the complete string (including realm or length
bytes at the front).
This has several effects:
1. We can receive an decrypt an onion which is grossly malformed.
2. We can still hand this to the htlc_accepted hook.
3. We then fail it unless the htlc_accepted accepts it manually.
4. The createonion API now takes the raw payload, and does not know
anything about "style".
The only caveat is that the sphinx code needs to know the payload
length: we have a call for that, which simply tells it to copy the
entire onion (and treat us as the final node) if it's invalid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular:
1. It must redirect to an existing command.
2. It must contain method, params and id.
And update the docs to show the id, which is vital.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
cheroot release(/"changes"?) notes:
#218 via PR #219: Fix HTTP parser to return 400 on invalid major-only HTTP version in Request-Line.
#198 via 9f7affe: Fix race condition when toggling stats counting in the middle of request processing.
Improve post Python 3.9 compatibility checks.
Fix support of abstract namespace sockets.
#222 via 621f4ee: Fix socket.SO_PEERCRED constant fallback value under PowerPC
Revisit PR #85 under PR #221. Now backports.functools_lru_cache is only required on Python 3.2 and earlier.
CherryPy #1206 via PR #204: Fix race condition in threadpool shrink code.
PR #224: Refactored “open URL” behavior in webtest to rely on retry_call. Callers can no longer pass raise_subcls or ssl_context positionally, but must pass them as keyword arguments.
#231 via PR #232: Remove custom setup.cfg parser handling, allowing the project (including sdist) to build/run on setuptools 41.4. Now building cheroot requires setuptools 30.3 or later (for declarative config support) and preferably 34.4 or later (as indicated in pyproject.toml).
Workers are now request-based, addressing the long-standing issue with keep-alive connections (#91 via PR #199).
Deprecated use of negative timeouts as alias for infinite timeouts in ThreadPool.stop.
CherryPy #1662 via PR #74: For OPTION requests, bypass URI as path if it does not appear absolute.
CherryPy #1818: Restore support for None default argument to WebCase.getPage().
https://github.com/cherrypy/cheroot/blob/master/CHANGES.rst
flaky changes:
Bugfixes - Reraise KeyboardInterrupt when running tests under pytest.
https://github.com/box/flaky/blob/v3.6.1/HISTORY.rst#361-2019-08-06
python-bitcoinlib:
New RPC `generatetoaddress(self,numblocks,addr)`.
Fixed Python 2.7 incompatibility.
Various OpenSSL fixes, including a memory leak.
https://github.com/petertodd/python-bitcoinlib/blob/python-bitcoinlib-v0.10.2/release-notes.md#v0102
pytest release notes:
A lot of misc fixes, see https://docs.pytest.org/en/latest/changelog.html.
If we can't decode the onion, because the onion got corrupted or we used
`sendonion` without specifying the `shared_secrets` used, the best we can do
is tell the caller instead.
This means that c-lightning can now internally decrypt an eventual error
message, and not force the caller to implement the decryption. The main
difficulty was that we now have a new state (channels and nodes not specified,
while shared_secrets are specified) which needed to be handled.
We are breaking with a couple of assumptions, namely that we have the
`path_secrets` to decode the error onion. If this happens we just want it to
error out.
addresses issue #2753.
Formatting the JSON with the default parameters will escape the unicode
symbols in a way that c-lightning won't allow, leading to an exception.
Changelog-Fixed: `pylightning` now handles unicode characters in JSON-RPC requests and responses correctly.
We were using sleeps to hope we catch the password prompt. This makes the test
flaky. So I added a help text followed by a `fflush` to make sure we catcht he
right moment, instead of guessing. The `fflush` is also useful for debugging
if a user ever pipes the output to a file it'd get buffered and the user would
wait forever. The same applies for automated systems such as `expect` or
`pexpect` based scripts that enter the password on prompt.
This will change the command `listconfigs` output in several ways:
- Deprecated the duplicated "plugin" JSON output by replacing it with
- a "plugins" array with substructures for each plugin with:
- path, name and their options
Changelog-Changed: JSON-RPC: `listconfigs` now structures plugins and include their options
Changelog-Deprecated: JSON-RPC: `listconfigs` duplicated "plugin" paths
The spec is (RSN!) going to explicitly denote where each feature should
be presented, so create that infrastructure.
Incorporate the new proposed bolt11 features, which need this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't set the secret to compulsory (yet!) but put code in for the
future. Meanwhile, if there is a secret, check it is correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Do the same thing '--help' does with them; append `...`.
Valgrind noticed that we weren't NUL-terminarting if answer was over
78 characters.
Changelog-Fixed: JSONRPC: listconfigs appends '...' to truncated config options.
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets you have a default, but also a network-specific config.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: `config` and <network>/`config` read by default.
Postgresql does not guarantee insertion order when querying, so we need to
actually go and look for the correct transaction.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Rounds out the application of `upfront_shutdown_script`, allowing
an accepting node to specify a close_to address.
Prior to this, only the opening node could specify one.
Changelog-Added: Plugins: Allow the 'accepter' to specify an upfront_shutdown_script for a channel via a `close_to` field in the openchannel hook result
This leads to all sorts of problems; in particular it's incredibly
slow (days, weeks!) if bitcoind is a long way back. This also changes
the behaviour of a rescan argument referring to a future block: we will
also refuse to start in that case, which I think is the correct behavior.
We already ignore bitcoind if it goes backwards while we're running.
Also cover a false positive memleak.
Changelog-Fixed: If bitcoind goes backwards (e.g. reindex) refuse to start (unless forced with --rescan).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So far we've only handled legacy payloads, which meant we could drop the realm
byte since it was always 0x00. Once we start handling TLV payloads the first
byte, i.e., the former realm byte, is important since it gives us the length
of the payload. This is a breaking change, however I don't think there's
anyone using the `raw_payload` as of yet.
Changelog-Changed: JSON-RPC: the `raw_payload` now includes the first byte, i.e., the realm byte, of the payload as well. This allows correct decoding of a TLV payload in the plugins.
Spaces just make life a little harder for everyone.
(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can definitely get a pong from l1 (should test be slow enough):
it's l3 we are concerned about.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.
Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
A log can have a default node_id, which can be overridden on a per-entry
basis. This changes the format of logging, so some tests need rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: JSON API: `htlc_accepted` hook has `type` (currently `legacy` or `tlv`) and other fields directly inside `onion`.
Changelog-deprecated: JSON API: `htlc_accepted` hook `per_hop_v0` object deprecated, as is `short_channel_id` for the final hop.
If a 'upfront_shutdown_script' was specified, show the address +
scriptpubky in `listpeers`
Changelog-added: JSON API: `listpeers` channels now include `close_to` and `close_to_addr` iff a `close_to` address was specified at channel open
Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.
Reduce test_feerate_stress iterations, and simply don't run
test_pay_retry under VALGRIND with SLOW_MACHINE at all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It currently works because we inject it so fast that it's still doing the
txout lookup, but that's about to change.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A long time ago (93dcd5fed7), I
simplified the htlc reload code so it adjusted the amounts for HTLCs
in id order. As we presumably allowed them to be added in that order,
this avoided special-casing overflow (which was about to deliberately
be made harder by the new amount_msat code).
Unfortunately, htlc id order is not canonical, since htlc ids are
assigned consecutively in both directions! Concretely, we can have two HTLCs:
HTLC #0 LOCAL->REMOTE: 500,000,000 msat, state RCVD_REMOVE_REVOCATION
HTLC #0 REMOTE->LOCAL: 10,000 msat, state SENT_ADD_COMMIT
On a new remote-funded channel, in which we have 0 balance, these
commits *only* work in this order. Sorting by HTLC ID is not enough!
In fact, we'd have to worry about redemption order as well, as that
matters.
So, regretfully, we offset the balances halfway to UINT64_MAX, then check
they didn't underflow at the end. This loses us this one sanity check,
but that's probably OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to change it so we always send our local messages, which
breaks this test. Add a new node which doesn't have any local
messages, so the test works correctly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Christian Decker
Michael Schmoock
ZmnSCPxj jxPCSnmZ
niftynei
fiatjaf
Antoine Poinsot
Rusty Russell
Jan Sarenik
Vincenzo Palazzo
Vincent
rbndg
Vasil Dimov
Saibato
Jarret Dyrbye