Commit Graph

1853 Commits

Author SHA1 Message Date
Christian Decker 8cc62d76e4 pytest: Stabilize test_channel_{spendable,receivable}
They were using TIMEOUT / 2 which may be way too long (hit against
test timeout), so we use a still ludicrous 30 seconds instead.
2021-01-29 10:29:09 +10:30
Christian Decker 6384cadd69 pytest: Stabilize the negotiation tests
We also make the logic a bit nicer to read. The failure was due to
more than one status message being present if we look at the wrong
time:

```
arr = ['CLOSINGD_SIGEXCHANGE:We agreed on a closing fee of 20334 satoshi for tx:17f1e9d377840edf79d8b6f1ed0faba59bb307463461...9b98', 'CLOSINGD_SIGEXCHANGE:Waiting for another closing fee offer: ours was 20334 satoshi, theirs was 20332 satoshi,']                                                                                                   │

     def only_one(arr):
         """Many JSON RPC calls return an array; often we only expect a single entry
         """
 >       assert len(arr) == 1
 E       AssertionError
```
2021-01-29 10:29:09 +10:30
Christian Decker 04ed93f5f8 pytest: Stabilize test_funding_external_wallet_corners 2021-01-29 10:29:09 +10:30
Christian Decker 18483ca582 pytest: Disable test_v2_open if not developer
It requires `--dev-force-features` which isn't available without
`DEVELOPER=1`
2021-01-29 10:29:09 +10:30
Christian Decker 7962db821c pytest: Stabilize test_channel_state_changed_bilateral 2021-01-29 10:29:09 +10:30
Christian Decker 07f5054700 pytest: Stabilize test_setchannelfee_state
Synching with the blockchain was slower than our timeout...
2021-01-29 10:29:09 +10:30
Christian Decker 03449e3cf0 pytest: Stabilize test_gossip_persistence
We weren't waiting for the `dev_fail` transaction to hit the mempool,
throwing the results off.
2021-01-29 10:29:09 +10:30
Christian Decker ae40c10bcb pytest: Stabilize test_onchain_timeout
The timeout on the pay future was too short under valgrind.
2021-01-29 10:29:09 +10:30
Christian Decker c564f165fa pytest: Stabilize test_penalty_htlc_tx_timeout
We weren't waiting for the transactions to enter the mempool which
could cause all of our fine-tuned block counts to be off. Now just
waiting for the expected number of txs.
2021-01-29 10:29:09 +10:30
Christian Decker c0f06f2779 pytest: Simplify and stabilize test_reconnect_no_update 2021-01-29 10:29:09 +10:30
Christian Decker 2b12cac31e pytest: Skip hsm encryption test if we don't have a TTY 2021-01-29 10:29:09 +10:30
Rusty Russell 001b5d6416 offers: make it a runtime option.
The fetchinvoice and offers plugins disable themselves if the option
isn't enabled (it's enabled by default on EXPERIMENTAL_FEATURES).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `experimental-offers` enables fetch, payment and creation of (early draft) offers.
2021-01-13 14:45:36 +01:00
Rusty Russell 27c006f7aa libplugin: make init return a string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: libplugin: init can return a non-NULL string to disable the plugin.
2021-01-13 14:45:36 +01:00
Rusty Russell 529ae0d766 plugins: allow plugins to disable themselves at startup.
By returning 'disable: <reason>' inside getmanifest or init result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: plugins: plugins can now disable themselves by returning `disable`, even if marked important.
2021-01-13 14:45:36 +01:00
Rusty Russell fc3e679c97 lightningd: control onion messages by `experimental-onion-messages` option.
Note that this also changes so the feature is not represented in channels,
reflecting the recent drafts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `experimental-onion-messages` enables send, receive and relay of onion messages.
2021-01-13 14:45:36 +01:00
niftynei 7e324177de df-tests: simultaneous openchannel_init (while in progress)
Reject a peer's request to open a channel while we're already in progress
2021-01-10 13:44:04 +01:00
niftynei 2c9ce2566c df-tests: test_lightningd_still_loading 2021-01-10 13:44:04 +01:00
niftynei a357e75c5d df-tests: test_disconnect_half_signed + test_reconnect_signed 2021-01-10 13:44:04 +01:00
niftynei fec1d9b707 df-tests: test_reconnect_openingd 2021-01-10 13:44:04 +01:00
niftynei 7eb6213fca df-tests: test_reconnect_sender_add1, others
We're still ignoring multifundchannel failure tests, but now we use the
'dual-fund' flag instead of a blanket true.
2021-01-10 13:44:04 +01:00
niftynei d9c68a708a df-tests: test_reconnect_receiver_add 2021-01-10 13:44:04 +01:00
niftynei 96c42262ab df-tests: test_peerinfo 2021-01-10 13:44:04 +01:00
niftynei 135cd178f7 df-tests: test_channel_persistence 2021-01-10 13:44:04 +01:00
niftynei a82bfa83ff df-tests: have the df_accepter plugin keep track of attempts
Test connection/reconnection handling for v2 opens. We needed to fixup
the accepter plugin so that we were freeing up inputs on
disconnect/failure.
2021-01-10 13:44:04 +01:00
niftynei 88b4b5b2bb df-tests: test_disconnect_opener 2021-01-10 13:44:04 +01:00
niftynei 3e330d7186 df-tests: test_forward_local_failed_stats 2021-01-10 13:44:04 +01:00
niftynei 973224dd14 df-tests: if we're using fundbalancechannel, set up plugin
nodes need to be using the accepter plugin, so they'll match funding.
we used to use push_msat for this, but v2 gets rid of it.

*sad trombone*
2021-01-10 13:44:04 +01:00
niftynei f822a93cf4 df-tests: add `setacceptmaxfunding` to df_accepter plugin
Method for setting a ceiling on the fund matching capabilities of
df_accepter. Setting it to zero means we don't fund the channel anymore.
2021-01-10 13:44:04 +01:00
niftynei e964c31369 df-tests: test_payment_success|failed_persistence 2021-01-10 13:44:04 +01:00
niftynei 09da171381 df-tests: test_io_logging
Move pid collection down to when dualopend is definitely dead
2021-01-10 13:44:04 +01:00
niftynei 609d509ca5 df-tests: test_channel_state_changed_unilateral 2021-01-10 13:44:04 +01:00
niftynei ac88c72f1e df-tests: test_channel_state_changed_bilateral 2021-01-10 13:44:04 +01:00
niftynei 450a094d26 df-tests: test_coin_movement_notices 2021-01-10 13:44:04 +01:00
niftynei 923e1f5c80 df-tests: test_feature_set 2021-01-10 13:44:04 +01:00
niftynei db65771f38 df-tests: test_plugin_feature_announce 2021-01-10 13:44:04 +01:00
niftynei 21b9bec93b df-tests: test_channel_state_change_history 2021-01-10 13:44:04 +01:00
niftynei 959ca793b0 df-tests: test_openchannel_hook_error_handling 2021-01-10 13:44:04 +01:00
niftynei 803d9701d3 df-tests: test_openchannel_hook_chaining 2021-01-10 13:44:04 +01:00
niftynei b0a8b10b41 df-tests: test_openchannel_hook
Have it check both openchannel + openchannel2
2021-01-10 13:44:04 +01:00
niftynei 0d45823b82 df-tests: test_funding_reorg_private 2021-01-10 13:44:04 +01:00
niftynei 09834bbc8d df-tests: test_node_reannounce correct features 2021-01-10 13:44:04 +01:00
niftynei c055198869 df-tests: test_fail_unconfirmed, duplicate with correct wires 2021-01-10 13:44:04 +01:00
niftynei dfa0eb0f2f df-tests: test_shutdown_awaiting_lockin check correct state 2021-01-10 13:44:04 +01:00
niftynei 5c7cc15a75 df-tests: adjust logsearch/features for test_dataloss_protection 2021-01-10 13:44:04 +01:00
niftynei e661d87440 df-tests: fixup expected features for dual-funding channels 2021-01-10 13:44:04 +01:00
niftynei 08200f03a7 df-tests: test_multifunding_v2_v1_mixed update to use flag
Now that we've got a fancy flag to enable dual funding, let's use it
2021-01-10 13:44:04 +01:00
niftynei 8fb97fd7e1 df-tests: updates for plugin hook_accepter helper
Fixes so that test_funding_close_upfront works with v2 opens
2021-01-10 13:44:04 +01:00
niftynei f4b7904ce5 tests: don't enforce daemon name on opening logline 2021-01-10 13:44:04 +01:00
niftynei 9951638d5f df: add flag to signal to run v2 channel open tests 2021-01-10 13:44:04 +01:00
niftynei 99a621dd99 df-reconnects: allow tx-sigs in channeld iff we're reconnecting
There's a case where a dropped funding_locked will result in the peer
moving onto channeld, while we stay in dualopend. As we haven't
received their funding_locked, we retransmit tx_sigs, which channeld
will need to handle.

With the patch the peer drops it on the floor; the peer will resend
funding_locked on reconnect, which will correctly advance us to
channeld and CHANNELD_NORMAL
2021-01-10 13:44:04 +01:00
Rusty Russell af46a4f57d fetchinvoice: allow amounts to be specified.
As per lastest revision of the spec, we can specify amounts in invoice
requests even if the offer already specifies it, as long as we exceed
the amount given.  This allows for tipping, and amount obfuscation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 4bb05e46e9 offers: convert currency when they request an invoice.
Means a reshuffle of our logic: we want to multiply by quantity before
conversion for maximum accuracy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 9681d491df offer: allow offers in other currencies if we can convert.
This avoids a footgun where they create an offer then we can't create
the invoice because they don't have a converter plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 23af9d4972 onion_message: support variable-length onion messages.
Updated to the BOLT, and a few tweaks, and we can send giant onion_messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell d867fab3a7 sendinvoice: call waitinvoice directly to on invoice payment.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 1aa7e8e011 offers: split offer send_invoice generation into new JSON command
We split `send_invoice` offers inoo offerout (for want of a better name).
This simplifies the API.

Also took the opportunity to move the `vendor` tag to immediately
follow `description` (our tests use arguments by keywords, so no
change there).

Suggested-by: shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 55983827b6 fetchinvoice: try direct peers if we can't route.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell bdfcec3eb6 fetchinvoice: check we're in the period before attempting to fetch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell dae477175c fetchinvoice: implement timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 075c25fc08 plugins/fetchinvoice: handle sendinvoice timeout, error or payment.
If they pay the invoice, they don't bother replying; that's just for
errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 43f5304539 plugins/offer: allow `offer` to create `send_invoice`, `refund` and `single_use` offers.
refund offers are implied send_invoice offers.  And send_invoice offers
are implied single-use offers, though it can also make sense to have
a non-send_invoice offer be single-use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Christian Decker 6b6e0fe34c pytest: Mark `test_lockup_drain` as disabled
It's been causing me headaches for some time, and it's just getting
annoying. Will file an issue to track progress.
2021-01-08 19:28:30 +01:00
niftynei d0dcb7cb42 tests: wait for drop to chain test_onchain_middleman_their_unilateral_in
this was catching the wrong 'sendrawtx exit 0' and consequently
not mining the close tx into a block
2021-01-08 19:28:30 +01:00
Rusty Russell 1a8978100e invoice: add ctlv option.
This is required if we want to create a "bouncer" plugin (in my copious free time!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now takes an optional `cltv` parameter.
2021-01-08 10:45:56 +01:00
Christian Decker 4b2efd69d7 pay: Fix a crash when waitblockheight times out
Fixes #4309
Changelog-Fixed: pay: Fixed an issue where waiting for the blockchain height to sync could time out.
2021-01-08 09:19:24 +01:00
Christian Decker ca204eacfa pytest: Reproduce waitblockheight timeout issue #4309 2021-01-08 09:19:24 +01:00
Christian Decker 70410b8ee8 pytest: Test LightningRpc and plugin command notification support 2021-01-07 08:32:40 +01:00
Christian Decker 84b3653606 pyln: Add command notification support for plugins
Changelog-Added: pyln-client: Plugin methods can now report progress or status via the `Request.notify` function
2021-01-07 08:32:40 +01:00
Christian Decker a1d33c3781 pytest: Add a test for `listinvoices`
We now have some more logic in the query, so let's test it
exhaustively.
2021-01-06 18:48:06 +01:00
Antoine Poinsot a4f07a3e78 tests/fuzz: add a fuzz target for hsm_encryption
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot 506d524508 libfuzz: add a NULL-termination in to_string
It's more useful if we actually want to use the output as, well, a
string..

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot 56c223886c lightning: confirm password on hsm_secret encryption
Changelog-changed: lightningd: the `--encrypted-hsm` now asks you to confirm your password when first set
Changelog-changed: hsmtool: the `encrypt` now asks you to confirm your password
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
niftynei 0c9072272f df-plugin: add `signonly` to signing in df_accepter
We're the accepter, so we look up which inputs we need to sign and only
sign those.
2021-01-05 19:12:00 +01:00
niftynei e3a1d1a7f3 pyln-client: to_whole_satoshi returns the rounded up satoshi value
A fractional satoshi value isn't really useful; rounding up loses
precision but that's why you called "whole satoshi", wasn't it?

Changelog-Changed: pyln-client: Millisatoshi has new method, `to_whole_satoshi`; *rounds value up* to the nearest whole satoshi
2021-01-05 19:12:00 +01:00
niftynei cce0020568 pyln-client: add missing `min_witness_weight` to fundpsbt/utxopsbt
We need to use it for the 'df_accepter' plugin, so we get the feerate
correct.

Changelog-Added: pyln-client: `fundpsbt`/`utxopsbt` now support `min_witness_weight` param
2021-01-05 19:12:00 +01:00
Antoine Poinsot 19afe15968 hsmtool: only write 32 bytes to hsm_secret on generatehsm
Fixes #4302
Changelog-fixed: hsmtool: the `generatehsm` command now generates an appropriately-sized hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot aa2586a45a pytest: reproduce issue #4302
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot 1513a2d07e hsmtool: input encryption password from stdin
This slightly breaks the API, but still accept the input: we just don't
take it into account anymore.

For `dumponchaindescriptors`, we have to still take the old place of the
`network` parameter into account to not entirely break the API.

Changelog-Added: hsmtool: password must now be entered on stdin. Password passed on the command line are discarded.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 16:15:40 +01:00
Christian Decker 5b3af00e28 pytest: Really only check DB statements if the env-var is "1" 2021-01-04 11:25:50 +01:00
Christian Decker 94820ad559 pytest: Disable test_funding_cancel_race on slow machines
The CI regularly trips up on this very taxing test (100 nodes) so only
run it if we have the required horsepower.
2021-01-04 11:25:50 +01:00
Christian Decker b3fd47e5f6 pytest: Stabilize test_routing_gossip
We were getting bad gossip because some nodes discarded the channel
announcement for being in the future. This is because the node was, at
that time, below the confirmation height. It'd then discard the
followup messages because not preceded by an announcement, and getting
upset about that.
2021-01-04 11:25:50 +01:00
Christian Decker 7e867e5ee6 pytest: Add throttler to limit load on the test system
Both my machine and apparently the CI tester machines regularly run
into issues with load on the system, causing timeouts (and
unresponsiveness). The throttler throttles the speed with which new
instances of c-lightning get started to avoid overloading. Since the
plugin used for parallelism when testing spawns multiple processes we
need to lock on the fs. Since we have that file open already, we'll
also write a couple of performance metics to it.
2021-01-04 11:25:50 +01:00
Rusty Russell 52af729641 plugins/offer and plugins/fetchinvoice: send and recv errors.
This also lets us extend our testing to cover error cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 96caf9f4ab fetchinvoice: return the next period for recurring offers.
This is useful for the caller to know when to call again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 19af1d10e6 invoice: hack in merkle of invoice as "payment_secret" (EXPERIMENTAL_FEATURES)
This lets actually pay the invoice that fetchinvoice returns.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 2baa410e2b pytest: finally test fetchinvoice send, and offer plugin receive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 08e110b568 JSON: offers plugin `offer` command.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Karol Hosiawa e582c0ac0a Added listfunds test 2020-12-22 13:00:02 +01:00
Rusty Russell 723c16072a cleanups: feedback from Christian Decker review.
1. Hoist 7200 constant into the bolt12 heade2.
2. Make preimage the last createinvoice arg, so we could make it optional.
3. Check the validity of the preimage in createinvoice.
4. Always output used flag in listoffers.
5. Rename wallet offer iterators to offer_id iterators.
6. Fix paramter typos.
7. Rename `local_offer_id` parameter to `localofferid`.
8. Add reference constraints on local_offer_id db fields.
9. Remove cut/paste comment.
10. Clarify source of fatal() messages in wallet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-15 12:05:02 +01:00
Rusty Russell a33e39b7e8 pay, wallet: rename internal `bolt11` vars to `invstring`.
And handle bolt12 strings if EXPERIMENTAL_FEATURES.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-15 12:05:02 +01:00
Rusty Russell d971e3de98 Plugin: support extra args to "start".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `start` command can now take plugin-specific parameters.
2020-12-15 09:28:56 +10:30
Rusty Russell 8a9976c4c1 plugins: support concatenation of multiple args.
"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
2020-12-15 09:28:56 +10:30
Rusty Russell 646c564ec5 plugins: remove deprecated string plugin options.
This was fixed in 0.8.2.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: plugins: options to init are no longer given as strings if they are bool or int types (deprecated in 0.8.2).
2020-12-15 09:28:56 +10:30
Christian Decker f7cdf1dd98 pytest: Debug test_lockup_drain 2020-12-11 16:55:55 -06:00
Christian Decker 5a69b94f9a pytest: Add test for keysend feature support 2020-12-11 15:50:09 +01:00
Christian Decker 3b599b846e keysend: Check that the destination supports keysend upon init
We were blindly initiating the keysend payment, which could lead to
confusing outcomes. This adds a very specific error message to the
error returned.

Changelog-Fixed: keysend: Keysend now checks whether the destination supports keysend before attempting a payment. If not a more informative error is returned.
2020-12-11 15:50:09 +01:00
Christian Decker 71fafd23fd pytest: Add a test to reproduce #4258 2020-12-09 06:56:21 +10:30
Christian Decker eae49727b1 pytest: Disable flaky test_fundchannel_start_alternate
This test has been causing most CI runs to fail. See tracking issue #4265
for discussion on fixing and re-enabling it.

Changelog-None
2020-12-08 09:14:25 +10:30
Michael Schmoock ce1a2f134a pytest: tests fee_base and fee_ppm on listpeers 2020-12-07 14:26:36 +01:00
Christian Decker bb0910999e pytest: Use datetime.strptime instead of datetime.fromisoformat
The latter is available only starting with Python 3.7, and it's the
only test that fails with <3.7. This allows us to keep on testing with
older versions.
2020-12-04 18:34:12 -06:00
ZmnSCPxj jxPCSnmZ a437936c78 lightningd/log.c: Fix up handling of SIGHUP.
Fixes: #4240

ChangeLog-Fixed: log: Do not terminate on the second received SIGHUP.
2020-12-02 16:48:48 +10:30
ZmnSCPxj jxPCSnmZ 67d8fdcc75 tests/test_misc.py: Add a test for issue #4240. 2020-12-02 16:48:48 +10:30
niftynei b23c6c50a0 df-tests: use multifundchannel pathways instead of custom plugin
Since fundchannel/multifundchannel now handles v2 opens, we use that
instead of this hacked together plugin.
2020-12-02 14:19:08 +10:30
Michael Schmoock 899022e452 pytest: use default fundamount on two tests 2020-12-01 11:15:30 +10:30
ZmnSCPxj jxPCSnmZ 32de621886 lightningd/plugin_hook.c: Make `db_write` a chained hook.
Fixes: #4219

Changelog-Changed: Plugins: Multiple plugins can now register `db_write` hooks.
2020-11-30 10:40:11 +10:30
niftynei 405453859c mfc-df: add 'happy path' tests for the v1+v2 things
We can't test disconnects et.al. quite yet because the 'cancel' flow for
openchannelv2 still needs to be resolved
2020-11-23 12:41:05 -06:00
niftynei da98a9d0af df-accepter plugin: temporarily dont pass in signpsbt 2020-11-23 12:41:05 -06:00
niftynei 78d32b12d0 nit,df test plugin: change up how feerate is formatted
Suggested-by: @cdecker
2020-11-23 12:41:05 -06:00
niftynei 7e60ef6974 tests: make connection details correct 2020-11-16 20:00:51 -06:00
niftynei 4a3ee19a22 connectd: Update connection list with new address
If we're already attempting to connect to a peer, we would ignore
new connection requests. This is problematic if your node has bad
connection details for the node -- you can't update it while inflight.

This patch appends new connection suggestions to the list of connections
to try.

Fixes #4154
2020-11-16 20:00:51 -06:00
niftynei cd7d5cdff9 pytest: use the 'contrib' plugin set, not the 'test' set
They're clobbering each other, and the test hangs
2020-11-13 14:28:29 -06:00
Michael Schmoock 183f7698c3 chore: reduce CI load merging a testcase 2020-11-13 14:25:02 -06:00
Michael Schmoock 4134c24cc9 pytest: fix a probable flake on test_channel_opener 2020-11-13 14:25:02 -06:00
Michael Schmoock 90a4476f16 pytest: tests closer on unilateral close 2020-11-13 14:25:02 -06:00
fiatjaf c5aa2ea911 failure_onion: a test that only ensures lightningd doesn't crash. 2020-11-10 19:03:23 -06:00
niftynei 72dc97c0c1 pytest: silence pytest warnings about unregistered 'slow_test' marker
We've been getting a bunch of 'warnings' about an unknown mark. This
silences the warning, by registering the mark as expected.

=========================================================== warnings summary ===========================================================
tests/test_closing.py:152
  /home/niftynei/dev/lightning/tests/test_closing.py:152: PytestUnknownMarkWarning: Unknown pytest.mark.slow_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    @pytest.mark.slow_test

tests/test_closing.py:214
  /home/niftynei/dev/lightning/tests/test_closing.py:214: PytestUnknownMarkWarning: Unknown pytest.mark.slow_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    @pytest.mark.slow_test

tests/test_closing.py:704
2020-11-10 16:17:04 -06:00
Antoine Poinsot fc964edc06 pytest: test onchain wallet descriptor import on bitcoind
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Antoine Poinsot 4f2ae48c54 fuzz: add a fuzz target for common/descriptor_checksum
A small one just to check that we don't crash nor go out of bounds!

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Antoine Poinsot bf0a2f2b85 fuzz: add a 'to_string' function to libfuzz
We want to use it outside of fuzz-amount

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Rusty Russell 9a575a98a0 gossipd: simplify large reply code.
We used to create the entire reply, the if it was too big, split in
half and retry.

Now that the main network is larger, this always happens with a full
request, which is inefficient.

Instead, produce a reply assuming no compression, then compress as a
bonus.  This is simpler and more efficient, at cost of sending more
packets.

I also renamed an internal dev var to make it clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 17:37:10 -06:00
Rusty Russell 4745e7e3d0 pytest: make test_hook_dep tests more stable.
I think this is what Travis is having an issue with, but it work
fine locally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Michael Schmoock f3804daabb pytest: simplify test_openchannel_hook_chaining now order is deterministic. 2020-11-09 15:22:33 -06:00
Rusty Russell fb295ffb51 plugin: sort topological candidates by specified order.
We previously registered hooks up in who-replies-to-getmanifest-first
order, but then if any had dependencies it would scatter that order.

This allows users to manually set dependencies developers have
forgotten by specifying the plugins manually in their configuration or
cmdline.  This was an excellent consideration by @mschmook.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell d429e21db3 pytest: test that we maintain load order unless hook deps require a change.
Suggested-by: @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell 852e14c947 plugins: check order once all plugins have returned from getmanifest.
This means we need to stop at this stage even in the runtime-loaded
case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell a4f79e8e7e pytest: test that we don't complain about missing dependencies at startup.
We fail this, because we check dependencies as they come in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell 62c52fe868 libplugin: add support for before and after deps on hooks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell e16ed0e207 pyln: add support for dependent hooks.
And use that to add simple tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
ZmnSCPxj jxPCSnmZ 0fd87b85da openingd/: Fail `fundchannel_start` if we already are, or will become, the fundee.
Fixes: #4108

Changelog-Fixed: Network: Fixed a race condition when us and a peer attempt to make channels to each other at nearly the same time.
2020-11-07 17:18:49 +00:00
ZmnSCPxj jxPCSnmZ 712e0796fc tests/test_connection.py: Test for #4108. 2020-11-07 17:18:49 +00:00
Michael Schmoock 4a8722bf1d pytest: state_change history 2020-11-06 14:47:04 -06:00
Michael Schmoock 68ce25c92d feat: adds timestamp to state_change notification 2020-11-06 14:47:04 -06:00
Michael Schmoock 67f4970eb1 pytest: test opener and closer of listpeers 2020-11-06 14:47:04 -06:00
Michael Schmoock c52efe0f37 pytest: channel state change cause and message 2020-11-06 14:47:04 -06:00
Rusty Russell ae4139ba10 lightningd: note whether payment secret is used or not.
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets.  We need to know this so we know when
we can make them compulsory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-28 12:03:14 +01:00
niftynei 4ea9d9e928 df: Pass new feerate options through to plugin, set reasonable bounds
We let the plugin decide what feerate to accept/whether or not to add
funds to the open. To aid this decision, we also send the plugin what we
(c-lightning) currently have as our max and min acceptable feerates.

We also now use these as our default for max/min acceptable feerate
range when sending an openchannel offer to a peer.

In the future, it might be a good idea to make these more easily
changeable, either via a config setting (?) or a command param.
2020-10-27 19:52:05 -05:00
Rusty Russell 1e5789d421 close: add notification for slow closes.
For compatibility, we only do this if `allow-deprecated-apis` is false
for now.  Otherwise scripts parsing should use `grep -v '^# '` or
start using `-N none`.

Changelog-Added: JSON-RPC: `close` now sends notifications for slow closes (if `allow-deprecated-apis`=false)
Changelog-Deprecated: cli: scripts should filter out '^# ' or use `-N none`, as commands will start returning notifications soon
Fixes: #3925
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-23 13:53:16 +10:30
Rusty Russell 44f6b8a746 pytest: add notifications to tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-23 13:53:16 +10:30
Antoine Poinsot 85aa53c778 fuzz: add a runner Python script for fuzz targets
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot c149a88bbe fuzz: add a fuzz target for initial_channel
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 1aa94ccd9e fuzz: add a fuzz target for the close tx
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 33c7eccb3b fuzz: add a fuzz target for channel_id derivation
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot f0061d0606 fuzz: add a fuzz target for common/bip32
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot a89a018475 fuzz: add a fuzz target for common/bigsize
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 5d987f2dec fuzz: add a target for base32/64 encoding
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 63b1ae54a1 fuzz: add a target for common/bech32
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 2f90c45454 fuzz: add a fuzz target for amounts parsing
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 62b54d0125 build: introduce a fuzzing mode
This adds a new configuration, --enable-fuzzing (which is more than
welcome to be coupled with --enable-address-sanitizer), to pass the
fuzzer sanitizer argument when compiling objects. This allows libfuzzer
to actually be able "to fuzz" by detecting coverage and be smart when
mutating inputs.

As libfuzzer brings its own ~~fees~~ main(), we compile objects with
fsanitize=fuzzer-no-link, and special-case the linkage of the fuzz
targets.

A "lib" is added to abstract out the interface to the fuzzing tool used.
This allow us to use the same targets to fuzz using AFL, hongfuzz or w/e
by adding their entrypoints into libfuzz. (h/t to practicalswift who
introduced this for bitcoin-core, which i mimiced)

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
niftynei ede5f5be3c mfc: blackbox test for commitment vs funding tx feerates
Liquid is excluded because the mempool entry doesn't contain a 'weight'
field.
2020-10-21 11:24:16 +10:30
niftynei 26bc4f5239 tx,bugfix: correct signature length estimate
71-bytes for a signature already includes the sighash byte.

 2-bytes	30 44 (DER- prefix thing)
34-bytes	02 20 6e29c8df67fffdda1613cef1413eb1a9ef3627f1fc5e4d910837274eafcc7b2a (r)
34-bytes	02 20 4b8563d79b92fdd830a546862439f80b24132d09318af2c7220c791067067e29 (s)
 1-byte		01 (sighash)
==
71-bytes
2020-10-21 11:24:16 +10:30
Rusty Russell 639eddf840 common/gossmap: digest private channel information too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-21 08:58:34 +10:30
Rusty Russell 83aea6b2bb gossip_store: make private channels more similar to channel_announcement
Instead of a boutique message, use a "real" channel_announcement for
private channels (with fake sigs and pubkeys).  This makes it far
easier for gossmap to handle local channels.

Backwards compatible update, since we update old stores.

We also fix devtools/dump-gossipstore to know about the tombstone markers.

Since we increment our channel_announce count for local channels now,
the stats in the tests changed too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-21 08:58:34 +10:30
Rusty Russell c6625943b5 pytest: test that route can see private channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-21 08:58:34 +10:30
niftynei a7d4b8780e elementsd: disable dual-funding test; don't add non-witness utxo
Elementsd needs to be updated to v0.20.0 in order for this to work as
expected
2020-10-20 12:50:31 +10:30
niftynei b696ec89a5 df-open: use channel_id for openchannel_update and openchannel_signed
Be as specific as possible is a good rule for things, I think
2020-10-20 12:50:31 +10:30
niftynei 5fbd25f938 dual-fund: test (turned off) and plugin for executing an openchannel
Proof that v2 openchannel works, plus a plugin for sewing all the
requisite bits together
2020-10-20 12:50:31 +10:30
niftynei e2a6fd7112 common: pull up `param_psbt`
Usable other places, not just in wallet
2020-10-20 12:50:31 +10:30
niftynei 8f253b2adc test: simplify test for close_to
Since `fundchannel` now supports the 'close_to' argument, we can remove
all the logic needed to call fundchannel_start here.

Underneath, we're still calling `fundchannel_start`, we're just one (or
two, if you count multifundchannel) call levels away from it now.
2020-10-16 13:51:57 +10:30
niftynei de34f08b82 tests,fundchannel: return the result from the fundchannel rpc call
We need this so we can verify the 'close_to' result
2020-10-16 13:51:57 +10:30
Rusty Russell a3c30441d3 channeld: order htlcs by id before retransmission.
We had one report of this, and then Eugene and Roasbeef of Lightning
Labs confirmed it; they saw misordered HTLCs on reconnection too.

Since we didn't enforce this when we receive HTLCs, we never noticed :(

Fixes: #3920
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: fixed retransmission order of multiple new HTLCs (causing channel close with LND)
2020-10-14 19:58:57 +10:30
Rusty Russell d04597cbb6 pytest: add test for HTLC ordering.
[ Includes tidyups by Christian Decker ]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-14 19:58:57 +10:30
Rusty Russell d5c91a347a channeld: log broken message if we receive HTLCs out of order.
We didn't care, but other implementations (particularly lnd) do.  And it
does violate the spec.

(We need to use skip not xfail on the test which catches this, since
xfail doesn't seem to stop errors reported by cleanup)

(Includes Christian's typo fix!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-14 19:58:57 +10:30
Rusty Russell 6195d953cc plugins: use "slow" feerate for mutual close negotiation.
We're rarely in a hurry here, and bitcoind is aggressive with fees.
You can always spend this output if you really have to, using CPFP.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: mutual closing feerate reduced to "slow" to avoid overpaying.
2020-10-13 20:53:34 +02:00
Christian Decker 9f8c9d6406 pytest: Check for connection close and reset errors in timeout test
I get both errors from time to time, seems like a timing issue.
2020-10-13 14:30:45 +02:00
Rusty Russell 79a21755c4 pytest: fix flaky test if one side hasn't processed close yet.
```
    @unittest.skipIf(TEST_NETWORK != 'regtest', "External wallet support doesn't work with elements yet.")
    def test_funding_close_upfront(node_factory, bitcoind):
...

        # check that you can provide a closing address upfront
        addr = l1.rpc.newaddr()['bech32']
>       _fundchannel(l1, l2, amt_normal, addr)
...
pyln.client.lightning.RpcError: RPC call failed: method: fundchannel_start, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'amount': 100000, 'announce': True, 'close_to': 'bcrt1qctx2k9cu9fd7nk449mzphqjcvvpyc4rxh6826x'}, error: {'code': -1, 'message': 'They sent error channel 2a1ca624cd1127761cb7a4395df2c3fd6d0abb3732c1f85a5345b0da716540d0: Multiple channels unsupported'}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-12 13:14:39 +02:00
Christian Decker 7d6892abba pytest: Verification mode of expanded stmts for the db_write hook
This can be used to find instances of unparseable expanded SQL statements like
the one fixed in #4090
2020-10-10 11:34:43 +10:30
Rusty Russell 711133e3c7 pytest: fix flaky test.
l2 only gives up 100 blocks after it ignores the HTLC, so sometimes
this test fails.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-09 15:20:03 +10:30
Rusty Russell 2d897be10a pytest: fix experimental test.
Broken back in 3b8c0a7397 which
changed the behavior of funding not to mark as immediately spent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-08 11:00:23 +10:30
Christian Decker ee0ee046ff pytest: Stabilize test_penalty_htlc_tx_timeout
lightning-5 can sometimes see itself sweeping the unilateral output resulting
in this weird line:

```log
HTLC already resolved by SELF when we found preimage
```
2020-10-07 09:36:30 +10:30
Christian Decker 9021bb26d1 pyln: Decode process output once before storing it 2020-10-07 09:36:30 +10:30
Christian Decker 81569de709 pytest: Stabilize test_setchannelfee_{restart,zero}
In both cases the flakyness arises from the destination not knowing about the
modified fees of the forwarding node, thus including the outdated details in
the routehint, and the sender being unlucky and always trying with the
routehint anyway.

The long-term solutions to this is going to be #4111, this commit just reduces
the flakyness to get back to business.
2020-10-07 09:36:30 +10:30
Christian Decker 1563bbc2fa pytest: Cleanup test_gossip_ratelimit
Drive-by code cleanups I stumbled over while investigating the issue fixed in
the previous commit.
2020-10-07 09:36:30 +10:30
Christian Decker 05934724dd pytest: Don't start 3 nodes in test_gossip_ratelimit
We really are just interested in their on-chain footprint, so actually
starting the nodes is pointless overhead, and caused a lot of flakyness due to
the output ordering sometimes not matching up. We now just use the `bitcoind`
API to fund, sign and send a raw transaction that matches the stashed gossip
messages.
2020-10-07 09:36:30 +10:30
Jan Sarenik a777d21fb7 pytest: Rename all fund_channel to fundchannel 2020-09-25 21:04:56 +02:00
Christian Decker 5a87e88ee5 pytest: Add test for failcode conversion
We added a conversion of failcodes that do not have sufficient information in
faac4b28ad. That means that a failcode that'd require additional information
in order to be a correct error to return in an onion is mapped to a generic
one since we can't backfill the information.

This tests that the mapping is performed correctly and replicates the
situation in #4070
2020-09-24 11:23:33 +09:30
Michael Schmoock 980a9517c6 fix: openchannel_hook log BROKEN on dup close_to
Changelog-None
2020-09-24 11:22:47 +09:30
Christian Decker d9ba39d2cb pytest: Add test for key-value style postgresql DSNs 2020-09-23 20:20:40 +09:30
niftynei 53ba34fe3a doc: correct weight numbers for fundpsbt 2020-09-17 10:14:22 +09:30
Rusty Russell f658dd0d78 pytest: test connection timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-11 21:27:45 +09:30
Vincenzo Palazzo f62d7bbe45 Added additional fix to code formatting and English spelling.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2020-09-10 17:27:51 +09:30
Vincenzo Palazzo 9fbeb9bcdc delpay: code style changes and fixed docs
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2020-09-10 17:27:51 +09:30
ZmnSCPxj jxPCSnmZ 0eb1e7e0ca plugins/libplugin-pay.c: Add new payee_incoming_limit to limit number of HTLCs based on payee connectivity.
Fixes: #3926

(probably)

Changelog-Fixed: pay: Also limit the number of splits if the payee seems to have a low number of channels that can enter it, given the max-concurrent-htlcs limit.
2020-09-10 16:50:52 +09:30
ZmnSCPxj jxPCSnmZ 2d86014e53 tests/test_pay.py: Test for overload of number of HTLCs at the payee. 2020-09-10 16:50:52 +09:30
ZmnSCPxj jxPCSnmZ 9587425f2d plugins/spender/fundchannel.c: Make fundchannel a thin layer around multifundchannel.
Changelog-Changed: protocol: `fundchannel` now shuffles inputs and outputs, and no longer follows BIP69.
2020-09-10 15:28:50 +09:30
niftynei 9771273c4d elements: these can coin moves can arrive in either order 2020-09-10 12:31:36 +09:30
niftynei eab7fcf8e5 test_utxopsbt: account for fee output diff when elements 2020-09-10 12:31:36 +09:30
niftynei 4311c658d0 test_txprepare: skip over fee outputs (elements) 2020-09-10 12:31:36 +09:30
niftynei 250af67908 elements: run sign + send psbt check for elements tests
Now that we're using an 'elements' witness_utxo and issue 3998 has been
resolved, this should work as expected.
2020-09-10 12:31:36 +09:30
Christian Decker bd811fbd1a pyln: Add safe fallback results for hooks
Hooks do not tolerate failures at all. If we return a JSON-RPC error to a hook
call the only thing the main daemon can really do is to crash. This commit
adds a mapping of error to a safe fallback result, including a warning to the
node operator that this should be addressed in the plugin. The warning is
reported as a `**BROKEN**` message, and should therefore fail any testing done
on the plugin.

Changelog-Fixed: pyln: Fixed HTLCs hanging indefinitely if the hook function raises an exception. A safe fallback result is now returned instead.
2020-09-10 10:27:24 +09:30
Christian Decker 3d6c4e93b3 pytest: Add a test to reproduce #3748 2020-09-10 10:27:24 +09:30
Michael Schmoock 160c564e5f fix: suppress duplicated channel_state_changed events 2020-09-10 10:24:06 +09:30
Michael Schmoock ddc759c49e test: plugin channel_state_changed notification 2020-09-10 10:24:06 +09:30
ZmnSCPxj jxPCSnmZ c6186829df tests/test_wallet.py: Simple multiwithdraw tests. 2020-09-09 20:36:08 +09:30
Christian Decker c7f5889a32 pytest: Skip test_sign_and_send_psbt for elements
This test is flaky because the generated PSBT seems to not have the change
output adjusted, or it is missing.

Tracking-Issue: ElementsProject/lightning#3998
2020-09-09 20:17:26 +09:30
Christian Decker 0260a4d212 pytest: Skip test_feerates for elements 2020-09-09 20:17:26 +09:30
niftynei b3cbb0b653 tests: plugin for dual-funding tests in ln-prototest
FIXME: requires wallycore
2020-09-09 19:54:20 +09:30
niftynei 6607f1d629 openingd: pull out common code, rename some things
We're going to reuse all this code for dualopend, which is coming soon.
2020-09-09 19:54:20 +09:30
niftynei 864f2f3e21 channel_id: save to database, dont derive from funding_txid
v2 channel open uses a different method to derive the channel_id, so now
we save it to the database so that we dont have to remember how to
derive it for each.

includes a migration for existing channels
2020-09-09 19:54:20 +09:30
niftynei 437ca83fa5 elementsd: add a fee output to a fund/utxopsbt for elements transactions
fundpsbt / utxopsbt create a (typically) output-less PSBT,
however for elements we require the fees to be encapsulated in an
output.

this patch updates fundpsbt / utxopsbt to add a fee output for elements
transactions. includes test updates.

Fixes #3998
2020-09-09 16:50:14 +09:30
ZmnSCPxj jxPCSnmZ 45e1b3828a plugins/multifundchannel.c: Add minchannels flag to multifundchannel.
[ changed from best_effort binary to minchannels counter -- RR]
2020-09-09 16:45:56 +09:30
ZmnSCPxj jxPCSnmZ c9f78d4e0a tests/test_connection.py: Add more complex multifunding tests. 2020-09-09 16:45:56 +09:30
ZmnSCPxj jxPCSnmZ 1162655b0e tests/test_connection.py: Add simple multifunding tests. 2020-09-09 16:45:56 +09:30
Rusty Russell 91caf4cc30 options: handle wumbo and large-channels aliases properly.
Too trivial a fix to really list in Changelog, but I noticed that we
specified "wumbo" twice.  We should really just use the proper name
in listconfigs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 22:09:37 -05:00
ZmnSCPxj jxPCSnmZ 5bac63c6a3 tests/test_closing.py: Test new onchaind earth-scorching behavior. 2020-09-09 12:38:19 +09:30
ZmnSCPxj jxPCSnmZ ee276bcb86 plugins/bcli.c: `sendrawtransaction` now has a required `allowhighfees` argument.
Changelog-Deprecated: plugin: `bcli` replacements should note that `sendrawtransaction` now has a second required Boolean argument, `allowhighfees`, which if `true`, means ignore any fee limits and just broadcast the transaction. Use `--deprecated-apis` to use older `bcli` replacement plugins that only support a single argument.
2020-09-09 12:38:19 +09:30
Rusty Russell dbfaa9c4d4 pytest: work around flaky test.
I believe Christian has a more complex rewrite, but this fixes the main
flake I was seeing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 12:15:33 +09:30
Rusty Russell 62f80f74ec pytest: fix flake in test_mpp_interference_2
I screwed up the rotation logic in an earlier varient of this PR, and
it lead me to discover why test_mpp_interference_2 was flaky.

Really, we should keep a fuzzy estimator of how much payment is
outstanding, but in practice rotation is probably good enough.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 19:47:11 +09:30
Rusty Russell 83298c030a wallet: switch over to withdraw in module, remove lots of unused code.
This removes the reservation cleanup at startup, too, now they're all
using 'reserved_til'.

This changes test_withdraw, since it asserted that outputs were marked
spent as soon as we broadcast a transaction: now they're reserved until
it's mined.  Similarly, test_addfunds_from_block assumed we'd see funds
as soon as we broadcast the tx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` now randomizes input and output order, not BIP69.
2020-09-08 10:14:42 +09:30
Rusty Russell 7a93363b0b pytest: remove duplicate tests.
These tests appear in both test_misc.py and test_wallet.py.

I checked, and they're the only duplcates; they were moved in
0543149b89 then re-added (merge error?)
in 67fc8ffbcf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell 2696ec6ccb pytest: fix assumptions in test_withdraw_misc
First, simplify: amount is set to 1000000, but then we deposit 1000000 + 0.01btc
(i.e. 2000000), and we always use 2 * amount.  Just use a single constant to
make it clear.

Secondly, we assume that the wallet considers outputs spent as soon as
we created the tx: this will not be true once withdraw uses sendpsbt.
So, we generate blocks, but now sometimes withdraw will pick up change
txs, so we need to reserve them to avoid that messing our coinmovements.

Finally, we assumed the withdrawl order was BIP69, which becomes
variable.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell e5f7e5b577 pytest: allow variable-order coin_moves, and give more information on failure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell 3b8c0a7397 sendpsbt: just reserve (maybe bump) inputs on send, don't mark spent.
Marking spent means if the transaction doesn't confirm for some
reason, the user will need to force a rescan to find the funds.  Now
we have timed reservations, reserving for (an additional) 12 hours
should be sufficient.

We also take this opportunity (now we have our own callback path)
to record the tx in the wallet only on success.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell 6b2a3f4dfb txprepare: remove old code, switch to plugin.
Some minor phrasing differences cause test changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: txprepare reservations stay across restarts: use fundpsbt/reservepsbt/unreservepsbt
Changelog-Removed: txprepare `destination` `satoshi` argument form removed (deprecated v0.7.3)
2020-09-08 10:14:42 +09:30
Rusty Russell 843d3d6745 pytest: remove test_txprepare_restart
This tested that we unreserved everything on restart, which we no
longer do.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Michael Schmoock 2816c08036 openchannel: test new hook chainable mechanics 2020-09-08 06:14:29 +09:30
Rusty Russell e4b5a1e0e8 dev: option to set what features we offer.
And use it to smoke-test !option_static_remotekey.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 10:52:49 +09:30
Rusty Russell 39e4796ae3 json_command: command_fail_badparam helper.
It's common to want to complain a token is not what we expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 74e960c20b pytest: increase channel sizes so they reach min capacity with option_anchor_outputs
With a feerate of 7500perkw and subtracting 660 sats for anchors, a
20,000 sat channel has capacity about 9800 sat, below our default:

	You gave bad parameters: channel capacity with funding 20000sat, reserves 546sat/546sat, max_htlc_value_in_flight_msat is 18446744073709551615msat, channel capacity is 9818sat, which is below 10000000msat

So bump channel amounts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell 9fabf614f4 pytest: run test_opening_tiny_channel with option_anchor_outputs.
It's now easy to fix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell 9b8f08a8a0 lightningd: return min-capacity-sat to its intended purpose.
And document exactly what it does: insist that an HTLC can pass of
this value (module assumptions of feerate).

Note that we remove the "is_opener" test from the capacity calculation
for anchor fees: it doesn't matter which side it is, someone has to pay
for anchor fees to it deducts from capacity.

This change breaks the test, which we rewrite.

Changelog-Changed: config: `min-capacity-sat` is now stricter about checking usable capacity of channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell 1746406e41 Makefile: normalize all the Makefiles
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS.  Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Christian Decker 94d4d20d5f pytest: Stabilize test_mpp_interference_2 2020-08-30 20:03:42 +02:00
Christian Decker 8b080dff90 pytest: Test that amountless invoices show the received amount
This was asked in a chat somewhere, and I just want to make sure it works as
expected.
2020-08-30 20:03:42 +02:00
Christian Decker 1884500bf8 pytest: Fix a typo in a test plugin
Reported-by: Henri Beqi
2020-08-30 20:03:42 +02:00
Christian Decker e420bfd6a8 pytest: Skip test_lockup_drain if VALGRIND and not DEVELOPER
The test was failing very reliably, by not raising the expected exception. My
best guess is that it needs more synchronization to cause the issue.
2020-08-30 20:03:42 +02:00
Rusty Russell dd8cd81e91 listpeers: add `features` array using BOLT9 names.
It's actually not possible to currently tell if you're using anchor_outputs
with a peer (since it depends on whether you both supported it at *channel open*).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-added: JSON-RPC: `listpeers` shows `features` list for each channel.
2020-08-28 14:46:09 -05:00
Rusty Russell 4a9c162211 pytest: disable test_mpp_interference_2.
We're failing this too often: we'd fail it more but it's disabled
with VALGRIND (it shouldn't be: @slow_test removes VALGRIND if SLOW_MACHINE
is set).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-27 13:51:22 +02:00
Christian Decker 21d87f7075 pay: Implement simple presplit fix for ludicrous amounts
This is the simplest possible fix: increase the target amount until we get
the desired number of parts, while still bucketizing payments together that
are in approximately the same size.

The current logic puts all payments that are in the range x < amount <= 16*x
in the same bucket, making them harder to distinguish.

Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-27 10:19:21 +09:30
Christian Decker 3f6809c934 pytest: Reproduce ludicrous presplit issue
Huge payment + presplit modifier = too many HTLCs...
2020-08-27 10:19:21 +09:30
Christian Decker bdad5a6f94 pytest: Stabilize test_onchain_middleman
It wasn't waiting for the tx to be in the mempool
2020-08-27 09:45:35 +09:30
Christian Decker 8f028272f4 pytest: Stabilize test_pay_get_error_with_update 2020-08-27 09:45:35 +09:30
Christian Decker 8f05d28049 pytest: Stabilize test_onchaind_replay 2020-08-27 09:45:35 +09:30
ZmnSCPxj jxPCSnmZ ba3f38077e lightningd/invoice.c: Use round-robin channel selection.
Changelog-Changed: We now make MPP-aware routehints in invoices.
2020-08-26 09:29:46 +09:30
ZmnSCPxj jxPCSnmZ f50951a0d6 tests/test_pay.py: Motivating scenario for round-robin routehinting. 2020-08-26 09:29:46 +09:30
Rusty Russell ef3fbab551 invoice: don't allow zero-value invoices.
You can't pay them anyway, and at least one person used 0 instead of "any".

Closes: #3808
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `invoice` no longer accepts zero amounts (did you mean "any"?)
2020-08-25 12:22:45 +09:30
Rusty Russell 7ad8fde060 bolt11: update ctlv expiry, always write it.
As per https://github.com/lightningnetwork/lightning-rfc/pull/785

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: protocol: bolt11 invoices always include CLTV fields (see lightning-rfc#785)
2020-08-24 20:55:47 +09:30
Rusty Russell 7435d50970 signpsbt: add signonly parameter to restrict/enforce what inputs to sign.
This is an extra safety check for dual funding, where we only want to sign
the inputs we provided!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `signpsbt` takes an optional `signonly` array to limit what inputs to sign.
2020-08-18 11:38:25 -05:00
Rusty Russell aab3808668 utxopsbt: let caller specify locktime, add tests and python binding.
Changelog-Added: JSON-RPC: `utxopsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-18 11:38:25 -05:00
Rusty Russell 14baaaa8ba fundpsbt: let caller specify locktime.
Required for dual funding where the opener sets it.

Changelog-Added: JSON-RPC: `fundpsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-18 11:38:25 -05:00
Rusty Russell cbd0be5c0e EXPERIMENTAL_FEATURES: enable option_anchor_outputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 8cf2fff91b pytest: generalize tests.
Anchor outputs break many assumptions in our tests:

1. Remove some hardcoded numbers in favor of a fee calc, so we only have to
   change in one place.
   FIXME: This should also be done for elements!
2. Do binary search to get feerate for a given closing fee.
3. Don't assume output #0: anchor outputs perturb them.
4. Don't assume we can make 1ksat channels (anchors cost 660 sats!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 6b39731c6a fundchannel: provide outnum
It's currently always 0, but it won't be once we replace txprepare.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel` has new `outnum` field indicating which output of the transaction funds the channel.
2020-08-13 12:37:02 -05:00
Christian Decker 8769f9ed93 pay: Fix final TLV payload if not going through MPP modifiers
Reported-by: ZmnSCPxj
Signed-off-by: Christian Decker <@cdecker>

Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
2020-08-13 12:50:16 +02:00
ZmnSCPxj jxPCSnmZ 29df517966 tests/test_pay.py: Provide test showing that blockheight disagreement causes us to advance routehints too aggressively.
The worst effect is that unpublished nodes are harder to pay, but
even published ones make us do unnecessary work, since we are
losing routehints from the published ones that could help us
actually route better to them.
2020-08-13 12:50:16 +02:00
Rusty Russell 2f4c16c388 pytest: fix race in test_bcli.
On my test machine, we queried bitcoind before the close tx was sent:

```
        # When output is spent, it should give us null !
        txo = l1.rpc.call("getutxout", {"txid": txid, "vout": 0})
>       assert txo["amount"] is txo["script"] is None
E       AssertionError: assert '20000000msat' is '00205b8cd3b914cf67cdd8fa6273c930353dd36476734fbd962102c2df53b90880cd'

tests/test_plugin.py:1221: AssertionError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 12:49:49 +02:00
Rusty Russell 185b377551 pytest: don't wait for channel active message in test_closing.py::test_penalty_htlc_tx_timeout
It might have already happened, and anyway, we wait for the actual state
below.

```
        # make database snapshot of l2
        l2.stop()
        l2_db_path = os.path.join(l2.daemon.lightning_dir, chainparams['name'], 'lightningd.sqlite3')
        l2_db_path_bak = os.path.join(l2.daemon.lightning_dir, chainparams['name'], 'lightningd.sqlite3.bak')
        copyfile(l2_db_path, l2_db_path_bak)
        l2.start()
        sync_blockheight(bitcoind, [l2])

        # push some money from l3->l2, so that the commit counter advances
        l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
>       l2.daemon.wait_for_log('now ACTIVE')

tests/test_closing.py:908:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:288: in wait_for_log
    return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyln.testing.utils.LightningD object at 0x7f0c145a32d0>
regexs = ['now ACTIVE'], timeout = 60

    def wait_for_logs(self, regexs, timeout=TIMEOUT):
        """Look for `regexs` in the logs.

        We tail the stdout of the process and look for each regex in `regexs`,
        starting from last of the previous waited-for log entries (if any).  We
        fail if the timeout is exceeded or if the underlying process
        exits before all the `regexs` were found.

        If timeout is None, no time-out is applied.
        """
        logging.debug("Waiting for {} in the logs".format(regexs))
        exs = [re.compile(r) for r in regexs]
        start_time = time.time()
        pos = self.logsearch_start
        while True:
            if timeout is not None and time.time() > start_time + timeout:
                print("Time-out: can't find {} in logs".format(exs))
                for r in exs:
                    if self.is_in_log(r):
                        print("({} was previously in logs!)".format(r))
>               raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E               TimeoutError: Unable to find "[re.compile('now ACTIVE')]" in logs.

contrib/pyln-testing/pyln/testing/utils.py:264: TimeoutError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 10:39:56 +09:30
Rusty Russell 0fa5cde782 pytest: fix occasional error in test_sign_and_send_psbt:
The modded_psbt is not necessarily invalid:

       # Try a modified (invalid) PSBT string
        modded_psbt = psbt[:-3] + 'A' + psbt[-3:]
        with pytest.raises(RpcError, match=r"should be a PSBT, not"):
>           l1.rpc.signpsbt(modded_psbt)
E           AssertionError: Pattern 'should be a PSBT, not' not found in 'RPC call failed: method: signpsbt, payload: {'psbt': 'cHNidP8BAM0CAAAABJ9446mTRp/ml8OxSLC1hEvrcxG1L02AG7YZ4syHon2sAQAAAAD9////JFJH/NjKwjwrP9myuU68G7t8Q4VIChH0KUkZ5hSAyqcAAAAAAP3///8Uhrj0XDRhGRno8V7qEe4hHvZcmEjt3LQSIXWc+QU2tAEAAAAA/f///wstLikuBrgZJI83VPaY8aM7aPe5U6TMb06+jvGYzQLEAQAAAAD9////AcDGLQAAAAAAFgAUyQltQ/QI6lJgICYsza18hRa5KoEAAAAAAAEBH0BCDwAAAAAAFgAUqc1Qh7Q5kY1noDksmj7cJmHaIbQAAQEfQEIPAAAAAAAWABS3bdYeQbXvBSryHNoyYIiMBwu5rwABASBAQg8AAAAAABepFD1r0NuqAA+R7zDiXrlP7J+/PcNZhwEEFgAUKvGgVL/ThjWE/P1oORVXh/ObucYAAQEgQEIPAAAAAAAXqRRsrE5ugA1VJnAith5+msRMUTwl8ocBBBYAFMrfGCiLi0ZnOCY83ERKJ1sLYMY8AAAA='}, error: {'code': -1, 'message': 'No wallet inputs to sign'}'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 10:39:56 +09:30
Christian Decker 0dcd974d97 pytest: Reproduce #3915 2020-08-12 19:10:48 +02:00
Rusty Russell fc097b8b48 libplugin: allow commands and options to mark themselves deprecated.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:43:18 +09:30
Vincenzo Palazzo 1521c29fcf listpays mod 1: add destination inside the response when bolt11 is null
Changelog-Added: JSON-RPC: `listpays` now lists the `destination` if it was provided (e.g., via the `pay` plugin or `keysend` plugin)
2020-08-09 16:03:03 +02:00