Commit Graph

2230 Commits

Author SHA1 Message Date
Rusty Russell 94e3e2f704 pytest: fix bad gossip flake.
If l3 is too slow, it can get channel_announcement after channel
is closed, so it gets upset at the node_announcement which follows:

   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Updated pending announce with update 103x1x1/1
   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: channel_announcement: no unspent txout 103x1x1
   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_NODE_ANNOUNCEMENT before announcement 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-14 12:52:48 +09:30
Rusty Russell f5e08c3dae pyln-client: document and test that init can self-disable.
mypy says it returns None, but it actually doesn't have to!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-14 12:52:19 +09:30
Rusty Russell 66af5f8a28 chaintopology: tell gossipd that channels no longer exist on reorg.
This actually caused the flake in test_funding_reorg_private, where
l1 and l2 might not mark the original channel disabled.  In fact, they
should *remove* it as it gets reorged out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-14 12:07:38 +09:30
niftynei f719343b4e tests: check that we don't re-up the leased funds when we RBF
Check that the peer won't put funds into a RBF'd channel lease.

FIXME: allow leases to pass through to RBFs

Changelog-None.
2021-08-09 07:11:05 +09:30
niftynei a9de23f993 tests: promote method to utility file
We'll reuse it later
2021-08-09 07:11:05 +09:30
niftynei 9df4234e8f funder: default to only funding leases
Make the default to only lease out funds.

Changelog-Changed: funder plugin defaults to leases-only
2021-08-09 07:11:05 +09:30
niftynei 5c7d2f09c1 df-tests: wait for disconnect before asserting unknown channel
fixes:

FAILED tests/test_connection.py::test_funding_v2_corners - Failed: DID NOT RAISE
2021-08-05 12:53:38 -05:00
niftynei 44dd353aa2 df-tests: bump feerates to get over the 'insufficient fee' limit
Naively calling bump with the next feerate doesnt add enough extra to
get over the min-relay requirement for RBF.
2021-08-05 12:53:38 -05:00
Rusty Russell a7e92b84d1 pytest: update test_queryrates for API change.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-05 12:53:38 -05:00
Rusty Russell a8b3a1c29c offers: make `decode` command available even without experimental-offers.
It subsumes `decodepay`, and it's nicer if people can just assume it's
available at all times.

Changelog-Added: JSON-RPC: `decode` now available without `experimental-offers`
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-02 13:23:31 -04:00
Rusty Russell 7bde0ead4d connectd: allow out-of-bounds fees unless they're actually getting *worse*.
Pointed out by @fiatjaf, and indeed it happened to me as well; a peer with
a high feerate reconnects and sends a similar (but now ludicrous) feerate,
and we get upset:

```
$ lightning-cli listpeers 039c73f53daad1050a6a72afb5353a2152f3152ee17168cd0ab28c2cb3e0050e36
{
   "peers": [
      {
         "id": "039c73f53daad1050a6a72afb5353a2152f3152ee17168cd0ab28c2cb3e0050e36",
         "connected": false,
         "channels": [
            {
               "state": "CHANNELD_NORMAL",
               "scratch_txid": "d796aa9c44920cc7169cdb61e36437bf180cedaec44103a69591ce2baac9b1d9",
               "last_tx_fee": "14329000msat",
               "last_tx_fee_msat": "14329000msat",
               "feerate": {
                  "perkw": 19791,
                  "perkb": 79164
               },
```

Then in the logs:
```
2021-07-23T19:34:56.227Z DEBUG   039c73f53daad1050a6a72afb5353a2152f3152ee17168cd0ab28c2cb3e0050e36-channeld-chan#39381: billboard perm: update_fee 17055 outside range 253-7210
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-25 10:18:20 +09:30
Rusty Russell ceb40dea38 lightningd: don't turn zero-length tlv fields into NULL.
Fixes: #4667
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
Rusty Russell 1eb7edb0bb pytest: test for offer failing decode when empty description.
In particular, Shesek tried an empty offer description, and the
resulting signature didn't match since it was omitted entirely from
the bolt12 string!

Reported-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
Rusty Russell ce338dc88f offerout: don't insist that invoice vendor match our vendor.
This was a bug in the spec, actually.

Reported-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #4666
2021-07-21 13:27:27 -04:00
Rusty Russell cb9e0268a7 offer / offerout: return existing if its still active.
As requested by @shesek: it's weird to fail if they ask for the exact
same thing (which is quite possible, since offers don't expire by
default).

And add a new "created" field so they can tell if they have an old
one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
Rusty Russell 57cc4e3c34 fetchinvoice: don't consider "msat" a change if fetchinvoice specified it.
We don't automatically *reject* an invoice which asks for a different
msat than we specified (caller may!), but we don't bother noting it
unless it is different.

Reported-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
Rusty Russell 0a0b2ce940 offers: make sure quantity limits are sane.
You could have quantity_min of 0, which makes no sense; spec has been
updated, so quote and enforce that.

Reported-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
niftynei 90c5f9a051 funderupdate: add documentation, update some params
Documentation for the funderupdate command on the funder plugin.
Realized we were missing the "leases_only" in the output; also adds
'_msat' to msat denominated outputs so they're parsed correctly by our
python bindings (also matches our naming conventions)

Changelog-EXPERIMENTAL: funder: `funderupdate` command to view and update params for contributing our wallet funds to v2 channel openings. Provides params for enabling `option_will_fund`.
2021-07-21 11:38:30 -04:00
niftynei 35bec51a97 printlogs for failing tests 2021-07-20 13:28:38 -04:00
niftynei 77d2c538b3 queryrates: make it dev-only
Since we now use 'compact_lease' to gate an open (if the rates have
changed, we fail), we no longer need to rely on query rates for figuring
things out, so we make it dev-only.

Changelog-Changed: JSON-API: queryrates is now developer only
2021-07-20 13:28:38 -04:00
niftynei b067820dd8 lease-channels: tests for the test gods
Check that channel leases work as expected for unilateral closes
and cheats!
2021-07-20 13:28:38 -04:00
niftynei 70bf57d54b channel-lease: reject if we're not currently advertising liquidity
If there's no plugin currently in place, we simply won't return any
funding at all, in which case we'd expect them to handle however
they want. (our implementation would fail the open, as we only accept
opens that have at least as much as we've requested provided)
2021-07-20 13:28:38 -04:00
niftynei 43ae30df21 chaintopology: new command 'parsefeerate'
Useful for parsing a passed in feerate before calling lightningd with
it, e.g. when you need to know what the feerate is for a fundpsbt before
calling fundpsbt

Changelog-Added: JSON-RPC: new command `parsefeerate` which takes a feerate string and returns the calculated perkw/perkb
2021-07-20 13:28:38 -04:00
niftynei 67b8a22aa7 channel: wire blockheight updates from channeld in to database 2021-07-20 13:28:38 -04:00
niftynei 265f960cfe liquidity-ads: persist channel blockheight states to disk
Adds new tables to database, backfills, basically copies the fee_rates
state machine for channeld.
2021-07-20 13:28:38 -04:00
niftynei 89f210fcd6 rpc: add queryrates
Undocumented RPC call for asking a peer what their rates are
2021-07-20 13:28:38 -04:00
niftynei 3ae64efbe4 tests: add test for setting/updating node announce
Make sure everything updates/flows through as expected.
2021-07-20 13:28:38 -04:00
niftynei 9431984cef funder_policy: set default to 0
As we move to advertising liquidity, set the default to not have any
fuzz. This means we won't randomly reject an offer to buy liquidity.
2021-07-20 13:28:38 -04:00
niftynei 876ff6821e setleaserates: new RPC to pass in lease rates
Changelog-Experimental: JSON-RPC: new RPC `setleaserates`, for passing in the rates to advertise for a channel lease (option_will_fund)
2021-07-20 13:28:38 -04:00
niftynei 376e6f8bd1 dual-funding: update fee_step to be a feerate
Using a 'feestep' is more restrictive than you'd want, instead we
enforce that the next feerate must be at least 1/64th more than the
last, but put no upper limit on it

Includes update to lnprototest changes

Contributed-By: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-EXPERIMENTAL: Protocol: Replaces init_rbf's `fee_step` for RBF of v2 opens with `funding_feerate_perkw`, breaking change
2021-07-19 16:13:24 -04:00
Michael Schmoock 896da4fc65 pytest: rescan restarts plugin on update 2021-07-15 13:26:05 -04:00
Christian Decker 0f18f203e2 pytest: Add a `keysend` test that includes `routehints` 2021-07-15 12:13:06 -04:00
niftynei 8654c817da sendcustommsg: promote to non-dev
Enable non-dev builds to send custom messages.

Preserves 'dev-' for compat-enabled builds.

Changelog-Changed: JSON-RPC: moved dev-sendcustommsg to sendcustommsg
2021-07-14 14:39:44 -05:00
Rusty Russell 28553e9bea route: assume modern TLV support if we don't see a node_announcement.
This is in preparation for removing support (next release?).

Changelog-Changed: Protocol: We now assume nodes support TLV onions (non-legacy) unless we have a node_announcement which says they don't.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-14 14:38:00 -05:00
Rusty Russell e7716c86fb lightningd: insist on payment_secret if bit compulsory on invoices.
This grandfathers in old invoices for the moment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-14 14:38:00 -05:00
Rusty Russell 9eb531868f lightningd: make invoices insist on payment_secret.
In fact, we make it compulsory, which means if you don't understand it
you'll hang up on us!

Add some logging for that in future.

Changelog-Changed: Protocol: All new invoices require a payment_secret (i.e. modern TLV format onion)
Changelog-Changed: Protocol: We can no longer connect to peers which don't support `payment_secret`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-14 14:38:00 -05:00
Rusty Russell 8f86b45f26 pytest: make test_sendonion_rpc use a TLV onion for final hop.
It's going to be compulsory soon!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-14 14:38:00 -05:00
Rusty Russell 4e881e56ce pytest: always provide payment_secret when making payments.
They're about to become compulsory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-14 14:38:00 -05:00
Michael Schmoock 246d230486 pytest: listchannels filter by destination 2021-07-14 14:34:00 -05:00
niftynei e4e1396447 tests: update coin movements for dual-funded mutual closes
This bug originated in 8f33f46960,
which updates the closing fee calculations.
2021-07-11 10:41:08 +09:30
niftynei bc7bbaa50d tests: update deprecated 'funding_msat' for new output format 2021-07-11 10:41:08 +09:30
niftynei e733e3210e schema: add 'funding_outnum' to openchannel_update schema 2021-07-11 10:41:08 +09:30
Rusty Russell adab9eb301 lightningd: add force-feerates option.
Useful for regtest and testnet.  Sure, you shouldn't use this on mainnet,
but I haven't restricted it because our users are usually pretty clever.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #1806
Changelog-Added: config: `force_feerates` option to allow overriding feerate estimates (mainly for regtest).
2021-07-09 07:26:09 +09:30
Rusty Russell 063366ed7e listinvoices: support listing by local offer_id.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Rusty Russell 2cb16a65c4 invoices: display the payer note if it's for local offer, allow in fetchinvoice.
We don't do it for sendinvoice (yet?).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `fetchinvoice` can take a payer note, and `listinvoice` will show the payer_notes received.
2021-07-03 12:13:45 +09:30
Rusty Russell f9fe814ea3 offers: signatures are now optional.
As per latest spec revision.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: BOLT12 offers can now be unsigned, for really short QR codes.
2021-07-03 12:13:45 +09:30
Rusty Russell 6336d39a95 features: renumber from 102/103 to 38/39.
100+ is for experimentation, modern spec practice is to assign feature bits
sequentially as PRs get added, to avoid later renumbering.

Still respect the old bit for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Rusty Russell 3657f26163 pytest: test for autoconnect in different circumstances.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Rusty Russell 8e3783a995 pytest: test that we try direct connection even for public nodes.
And even if we can't find *ourselves* because we have no channels
(we won't be able to pay, sure, but we can request an invoice!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Rusty Russell 33a40ca73b listinvoice: populate local_offer_id when iterating whole db.
And add the local_offer_id to the schemas too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSONRPC: `listoffers` now shows `local_offer_id` when listing all offers.
2021-07-03 12:13:45 +09:30
Rusty Russell 8f33f46960 closingd: use a more accurate fee for closing fee negotiation.
We were actually using the last commit tx's size, since we were
setting it in lightningd.  Instead, hand the min and desired feerates
to closingd, and (as it knows the weight of the closing tx), and have
it start negotiation from there.

This can be significantly less when anchor outputs are enabled: for
example in test_closing.py, the commit tx weight is 1124 Sipa, the
close is 672 Sipa!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Use a more accurate fee for mutual close negotiation.
2021-06-30 14:12:24 +09:30
Rusty Russell c1ad7ac5cb pytest: fix weird case in test_closing_different_fees
We set amounts to a list, but then don't use the values except
as a boolean.

Make it a boolean, which should also speed the test up!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-30 14:12:24 +09:30
Christian Decker 5f9e5d598e pytest: Add test for keysend extra-tlvs 2021-06-26 10:55:13 +09:30
Rusty Russell 84190af72c pytest: Add test that we can redo closing negotiation even after onchain.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 13:23:28 +09:30
Rusty Russell ca0358f978 channeld: send shutdown_complete even if reestablish_only.
This lets us transition (with a few supporting changes) to closingd,
which will happily let them mutual close with us.

We already handle the case where this mutual close is redundant (for
packet loss), so this is easy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: We will now reestablish and negotiate mutual close on channels we've already closed (great if peer has lost their database).
2021-06-25 13:23:28 +09:30
Rusty Russell 44829d1361 openingd: tell lightningd if we get a reestablish.
It simply uses connectd to send an error if it doesn't know anything
about the channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 13:23:28 +09:30
Rusty Russell b223a6acbb common/read_peer_msg: don't try to handle reestablish/reopen.
Let the callers do that (only channeld needs to do this).

We temporarily send an error on unknown reestablish in openingd, as
this mimic previous behavior and avoids breaking tests (it does leave
a BROKEN message in the logs though, so
test_funding_external_wallet_corners needs to ignore that for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 13:23:28 +09:30
Rusty Russell ef138575e1 pytest: add test (failing) which allows talking about closed channels.
In particular, if one side sees the final CLOSING_SIGNED and the other
doesn't, we won't talk when it reconnects.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 13:23:28 +09:30
Rusty Russell 9f8a6e2314 lightningd: always pass closing connections through channeld.
It handles all the cases of retransmission, and in the normal case
retransmits shutdown and immediately returns for us to run closingd.

This is actually far simpler and reduces code duplication.

[ Includes fixup to stop warn_unused_result from Christian ]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: We could get stuck on signature exchange if we needed to retransmit the final revoke_and_ack.
2021-06-25 13:23:28 +09:30
Rusty Russell a997e798ee pytest: test for closing needing retranmission bug.
This was turned by at random by CI:

1. Alice has sent shutdown, but it still waiting for revoke_and_ack.
2. Bob has sent and received shutdown, and sent revoke_and_ack,
   so it considers it time for signature exchange.
3. Disconnect before Alice received revoke_and_ack.
4. Reconnect, Bob is in closingd, which doesn't rexmit revoke_and_ack.
5. Timeout.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 13:23:28 +09:30
Rusty Russell 1a181a3240 doc/schemas: listfunds, listinvoices, listnodes, listoffers
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 09:49:33 +09:30
Rusty Russell 0a99b8c64c listtransactions: don't create a msat field called "satoshis".
That's a terrible, terrible idea.  (Documentation comes in later patch
which has the schema).

Also, blockheight is a u32, so simplify.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listtransactions` `outputs` `satoshis` field (use `msat` instead).
2021-06-25 09:49:33 +09:30
Rusty Russell d8136d856b listpeers: clean up JSON output.
In general, it's better to omit a field than put in a 'null', and
putting variable-named fields in an object is also a bad idea.

This is reflected in how hard this is to express in JSON schema, too.

Others:
1. Remove the obsolete "funding": "LOCAL" from unopened channels, but add
   "opener": "local" as used in normal channels.
2. htlc cltv_expiry is a u16.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listfunds` `channels` `funding_allocation_msat` and `funding_msat`: use `funding`.
Changelog-Deprecated: JSON-RPC: `listfunds` `channels` `last_tx_fee`: use `last_tx_fee_msat`.
Changelog-Deprecated: JSON-RPC: `listfunds` `channels` `closer` is now omitted if it does not apply, not JSON `null`.
2021-06-25 09:49:33 +09:30
Rusty Russell 50ebdfb5e8 invoice: allow creation of giant invoices.
https://github.com/lightningnetwork/lightning-rfc/pull/877 talks about
removing this restriction (only Electrum actually enforced it on
receive), so start by allowing creation of giant invoices, though
we mark them as requiring mpp.

Changelog-Changed: JSON-RPC: `invoice` now allows creation of giant invoices (>= 2^32 msat)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-20 14:25:40 +09:30
Antoine Poinsot e19b67f6da qa: test hsm encryption password provided from stdin
Test that it roundtrips with the non-stdin way, in order to make sure we
don't introduce a discrepancy between the two.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-06-18 10:40:07 +09:30
Rusty Russell 50cb810bd0 pyln-client: revert old/new comparisons for getroute, listchannels, listnodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell cda8f8190b invoice: overhaul routehints to use topology.listincoming, cleanup.
This turned into a more extensive cleanup than intended.  The previous
warnings were overlapping and confusing, especially now MPP is the norm.

*warning_capacity* is now the "even under best circumstances, we don't
have enough incoming capacity", which is really what
warning_mpp_capacity was trying to say (no longer printed).

*warning_offline* and *warning_deadends* are only given if adding such
peers would have helped give capacity (i.e. not if *warning_capacity*
is set).  The new *warning_private_unused* tells you that we would
have sufficient capacity, but we refused to expose private channels.

The test cases have been enhanced to cover the new warnings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now gives `warning_private_unused` if unused unannounced channels could have provided sufficient capacity.
Changelog-Changed: JSON-RPC: `invoice` warnings are now better defined, and `warning_mpp_capacity` is no longer included (since `warning_capacity` covers that).
2021-06-16 10:29:17 +09:30
Rusty Russell e81e640e76 topology: plugin to implement getroute command.
Temporarily rename old getroute to getrouteold (we will remove this).

Changelog-Changed: JSON-RPC: `getroute` is now implemented in a plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell 2910bb0235 pytest: fix flake in test_misc.py::test_funding_reorg_* xxremote_lags
If l2 doesn't have the funding locked in, the rest of the test fails
(we got a timeout on `wait_for(lambda: [c['active'] for c in l2.rpc.listchannels('103x1x0')['channels']] == [False, False])`)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-15 06:36:36 +09:30
Rusty Russell 064ad486e3 close: check that destination is going to be accepted.
Prior to this, sending a v1 address (or, in fact, any random crap!)
would cause the unsupporting node to unilaterally close.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-11 13:54:20 +09:30
Rusty Russell 3cf98085d4 tests: test future segwit versions for withdraw / close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-11 13:54:20 +09:30
Rusty Russell 4cc3eaf79c common/bech32: update to bech32m reference.
I did this by copying the updated bech32 code, and then re-patching in
our minor changes:

1. Headers modded (we need size_t)
2. Explicit length for bech32_encode/decode (not 90).
3. Exposing and bech32_ prefix for convert_bits, charset, charset_rev.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-11 13:54:20 +09:30
Christian Decker 610d8a0a74 plugin: Add a status field to htlcs in listpeers
Annotating the htlc in `listpeers` with their current status, and
which plugin is currently holding on to them with their
`htlc_accepted` hook can help us debug where plugins may go wrong.

Changelog-Added: jsonrpc: HTLCs in `listpeers` are now annotated with a status if they are waiting on an `htlc_accepted` hook of a plugin.
2021-06-05 17:47:32 +09:30
Rusty Russell 67c03c02a0 pytest: speed up test_opening.py::test_funder_contribution_limits when !DEVELOPER
This test takes 695 seconds, because fundwallet waits for the wallet to
notice the tx, which takes 60 seconds if not DEVELOPER.  Do all the waiting
at once, and this speeds the test up to 153 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-05 09:47:50 +09:30
Rusty Russell 2f21435195 pytest: test that both sides refuse upgrade if not quiescent.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-04 16:13:08 +09:30
Rusty Russell ebcadc50c3 pytest: test onchaind handles unilateral and penalty txs on either side of upgrade.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-04 16:13:08 +09:30
Rusty Russell 5526c6797b channeld: set desired_type to upgrade `option_static_remotekey` if not already.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-04 16:13:08 +09:30
Rusty Russell d305b8ada6 channeld: send upgradable types, add logging (EXPERIMENTAL_FEATURES)
For now the only upgrade possible is to enable option_static_remotekey.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-04 16:13:08 +09:30
Christian Decker 5e1fadf799 pay: Skip the presplitter mod if it'd exhaust our HTLC budget
Changelog-Fixed: pay: The presplitter mod will no longer exhaust the HTLC budget.
2021-06-03 16:59:53 +09:30
Rusty Russell e619bf00fb pytest: show problem with pay when not enough HTLCs available.
As you can see, I did a lot of debugging before realizing that the
actual problem is in the pay plugin :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-03 16:59:53 +09:30
niftynei e95fc74884 tests: cleanup any unfinished/hanging channel opens
Otherwise it hangs on cleanup, as l2 is waiting for l1 to respond (and
it's not going to), so the memleak call hangs.
2021-06-03 11:30:05 +09:30
Antoine Poinsot b5aaee6679 Revert "pytest: Skip hsm encryption test if we don't have a TTY"
This reverts commit 2b12cac31e. There is
no need to skip the test in this case, and it seems to simply be an artifact
of CI-debugging hell :)

Changelog-None
2021-06-03 11:27:16 +09:30
Rusty Russell 03cfe0b468 EXPERIMENTAL: dev-quiesce to initiate (and test) quiescence.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-01 12:08:51 -05:00
Rusty Russell a794e87edd pytest: test pay ordering.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-01 11:14:38 -05:00
Rusty Russell ec83d7a8a5 doc/schemas: disableoffer, disconnect, feerates, fetchinvoice, fundchannel, fundchannel_cancel, fundchannel_complete, fundchannel_start, fundpsbt, getinfo, getlog, getroute.
We also add a test for getlog, since it was never called by the
testsuite.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-27 20:28:49 +09:30
Rusty Russell ea99a05249 pytest: add schema support for JSON responses.
This adds our first (basic) schema, and sews support into pyln-testing
so it will load schemas for any method for doc/schemas/{method}.schema.json.

All JSON responses in a test run are checked against the schema (if any).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-27 20:28:49 +09:30
Rusty Russell 129d3f65e7 pytest: don't use command_success_str in test_libplugin.c
result should *always* be an object.  This allows it to add fields
without breaking the API.  A command which returns "result" as a
string is living in sin.

This changes one of the two callers of "command_success_str".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-27 20:28:49 +09:30
Rusty Russell d3f370944e Makefile: update to latest spec.
This includes anysegwit and the updated HTLC tiebreak test vector.  It
also adds explicit wording for invalid per_commitment_secret (which
nicely matches our code already!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-26 20:01:03 +09:30
Rusty Russell b6223eb117 lightningd: option_shutdown_anysegwit is no longer experimental.
https://github.com/lightningnetwork/lightning-rfc/pull/672 was merged.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `option_shutdown_anysegwit` allows future segwit versions on shutdown transactions.
2021-05-26 20:01:03 +09:30
Rusty Russell 14eddb95ba fundchannel, multifundchannel: reserve inputs for two weeks, not 12 hours.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `fundchannel` and `multifundchannel` will now reserve funding they use for 2 weeks instead of 12 hours.
2021-05-26 15:08:01 +09:30
Rusty Russell 95f77f119b pytest: test double-spending an opening tx input.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-26 15:08:01 +09:30
Rusty Russell f24dc9173d wallet: add "reserved_to_block" field to listfunds.
We already have this field in reserveinputs and unreserveinputs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listfunds` has a new `reserved_to_block` field.
2021-05-26 15:08:01 +09:30
Nalin Bhardwaj 94419c7d73 common: Check chain hash in gossip_timestamp_filter
Changelog-Fixed: Validate chain hash for gossip_timestamp_filter messages
2021-05-26 10:43:50 +09:30
Rusty Russell 96bd0961a4 pay: fix corner case where we eliminate our own hint.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `pay`: Fix occasional crash paying an invoice with a routehint to us.
2021-05-26 08:07:38 +09:30
Rusty Russell df594be80a pytest: add message check to test_blockheight_disagreement
Make sure there was an actual disagreement!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-25 16:25:17 +02:00
Rusty Russell 61d146a2e7 pytest: fix gossip flake in test_restart_many_payments
```
2021-05-23T09:18:56.9768962Z lightningd-4: 2021-05-23T08:33:31.918Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Ignoring future channel_announcment for 103x4x0 (current block 102)
2021-05-23T09:18:56.9771608Z lightningd-4: 2021-05-23T08:33:31.918Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x4x0/0
2021-05-23T09:18:56.9774035Z lightningd-4: 2021-05-23T08:33:31.919Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x4x0/1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-25 16:25:17 +02:00
Antoine Poinsot fe8074c8c3 Refuse to parse v2 onion addresses without deprecated_apis
Tor v2 hidden services have been deprecated for a while:
https://blog.torproject.org/v2-deprecation-timeline .

This prevents user from being able to set them in the configuration
and to connect to them while still letting us be able to parse them
for gossip.

Changelog-Deprecated: lightningd: v2 Tor addresses.  Use v3.  See https://blog.torproject.org/v2-deprecation-timeline.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-05-24 20:22:45 +09:30
Rusty Russell d70661d32e pytest: check nonstandard onion generation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-24 12:52:19 +02:00
niftynei 9d9fdfdfa6 rbf-test: confirm that rbf'd inflight is opened and closed as expected
We weren't testing that a non-tip inflight would get opened correctly.
Until now, that is.
2021-05-24 12:17:48 +09:30
niftynei d6bd6cc5cf open-rbf: we broadcast all the commitments for a channel when closed
If you drop an rbf'd channel to chain (before any updates have been
made) we should drop *all* of the inflights to chain
2021-05-24 12:17:48 +09:30
niftynei dff9516cad rbf-tests: check every inflight has signed commitment produced
Little check to make sure that we can produce a signed commitment tx for
every inflight we've got saved.
2021-05-24 12:17:48 +09:30
niftynei 4247ec3a05 inflights: save the whole psbt to the database
Otherwise we're missing info when we go to broadcast these and can't
properly sign the transaction to close it.

Found-by: @jasan
2021-05-24 12:17:48 +09:30
niftynei 727d7e9493 df-rbf: test that, with enough bumps, we can eventually add to mempool
We don't pass the minimum fee requirement the first few times we attempt
an RBF, so it fails. Keep going until actually replaces
2021-05-24 12:17:48 +09:30
niftynei 82fa3fa2ef df-rbf: order inflights by funding_feerate
When we re-populate from disk, we need to know what order to recreate the
inflights list in.

Fixes #4511
2021-05-24 12:17:48 +09:30
niftynei 8ae9f6ac9e df-tests: add better explainer for test logic 2021-05-24 12:17:48 +09:30
niftynei bab5ef4aff df-test: test for a failed rbf attempt, currently crashes
An attempted + failed RBF results in a crashed/dead node
2021-05-24 12:17:48 +09:30
niftynei e2867be609 df-tests: re-write test_funding_cancel_race for v2 commands
test_funding_cancel_race uses fundchannel_open etc; this new test does a
similar (but not exact thing, as 'aborts' dont work after an update
is confirmed) thing, using openchannel_update and openchannel_abort.
2021-05-23 17:42:09 +09:30
niftynei 2d4939fa6d df-tests: re-write the funding wallet corners test for v2 2021-05-23 17:42:09 +09:30
niftynei c93bd5bd51 funder: test for adding utxos, calculating total available
Make sure that 100% available actually puts in 100% of what we can add
(note that this isn't 'all' -- we don't include uneconomic utxos)
2021-05-23 08:19:50 +09:30
Rusty Russell 14febd4ecc pytest: disable mpp test which is blocking merges.
I'm still diagnosing exactly why this fails, but it will need Christian's
help at this stage.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-23 08:13:27 +09:30
Rusty Russell 33736b860a lightningd: attach HTLC timeout to htlc itself, fix gratuitous disconnect bug.
We set the timeout on first HTLC, but didn't clear it if that HTLC failed.

It's saner to have a per-HTLC timeout (since that's what it is!) and
also our timer infra is specially coded to scale approximately infinitely so
trying to optimize this is vastly premature.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: We would sometimes gratuitously disconnect 30 seconds after an HTLC failed.
2021-05-21 14:45:05 +09:30
Rusty Russell 11180e7aa1 pytest: add test for HTLC timeout on failed htlcs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-21 14:45:05 +09:30
Rusty Russell e141b71675 pytest: fix flake in test_reconnect_no_update
If l2 didn't get FUNDING_LOCKED from l1 before it disconnected, it
won't be in state CHANNELD_NORMAL: it will be in DUALOPEND_AWAITING_LOCKIN.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-17 18:58:10 -05:00
niftynei ce1e5bd388 df-test: make flakey test not as flakey
This would flake fairly regularly, what we really care about is
asserting that the l2 node is in CHANNELD_NORMAL state, while the l1
node hasn't progressed that far yet.
2021-05-12 11:25:41 +09:30
niftynei e375932108 df-tests: test_coin_movement_notices, make work?
No idea how this slipped past the first time
2021-05-12 11:25:41 +09:30
niftynei 71a4a2e31c df: rework closing logic
Trying to put all the disconnect logic into the same path was a dumb
idea. If you asked to reconnect but passed in an 'unsaved' channel, we
would not call the 'reconnect' code.

Instead, we make a differentiation between "unsaved" channels
(ones that we haven't received commitment tx for) and handle the
disconnect for these separate from where we want to do a reconnect.
2021-05-12 11:25:41 +09:30
niftynei efdc36c8e9 tests: mark test as requiring developer
hangs with EXP_DF when developer=0
2021-05-12 11:25:41 +09:30
niftynei 5cac36724e df-test: v2 has reconnects 2021-05-12 11:25:41 +09:30
niftynei 6d3fb11bc6 df-tests: patch for state == AWAITING_UNILATERAL problem
Found on CI where DEVELOPER=0 EXPERIMENTAL_DUAL_FUND=1,
as we turn off automatic reconnects when DEVELOPER=1

This test has been modified to make the error happen every time, and
then fixed.

lightningd-2: 2021-05-07T20:12:03.790Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Peer has reconnected, state AWAITING_UNILATERAL
lightningd-2: 2021-05-07T20:12:03.812Z **BROKEN** lightningd: FATAL SIGNAL 6 (version e8b3f78)
lightningd-2: 2021-05-07T20:12:03.812Z **BROKEN** lightningd: backtrace: common/daemon.c:44 (send_backtrace) 0x56384ee072e9
lightningd-2: 2021-05-07T20:12:03.813Z **BROKEN** lightningd: backtrace: common/daemon.c:52 (crashdump) 0x56384ee0733b

----------------------------- Captured stderr call -----------------------------
lightningd: lightningd/peer_control.c:1100: peer_connected_hook_final: Assertion `channel->state == DUALOPEND_OPEN_INIT || channel->state == DUALOPEND_AWAITING_LOCKIN' failed.
lightningd: FATAL SIGNAL 6 (version e8b3f78)
0x56384ee072a1 send_backtrace
	common/daemon.c:39
0x56384ee0733b crashdump
	common/daemon.c:52
0x7f88486a020f ???
	???:0
0x7f88486a018b ???
	???:0
0x7f884867f858 ???
	???:0
0x7f884867f728 ???
	???:0
0x7f8848690f35 ???
	???:0
0x56384eddc94e peer_connected_hook_final
	lightningd/peer_control.c:1100
0x56384edea2ed plugin_hook_call_
	lightningd/plugin_hook.c:275
0x56384eddfeb8 plugin_hook_call_peer_connected
	lightningd/peer_control.c:1156
0x56384eddfeb8 peer_connected
	lightningd/peer_control.c:1209
0x56384edc30cd connectd_msg
	lightningd/connect_control.c:332
0x56384edebe6f sd_msg_read
	lightningd/subd.c:509
0x56384edebfb1 read_fds
	lightningd/subd.c:310
0x56384ee483b0 next_plan
	ccan/ccan/io/io.c:59
0x56384ee4885b do_plan
	ccan/ccan/io/io.c:407
0x56384ee488f8 io_ready
	ccan/ccan/io/io.c:417
0x56384ee4a23c io_loop
	ccan/ccan/io/poll.c:445
0x56384edcabda io_loop_with_timers
	lightningd/io_loop_with_timers.c:24
0x56384edce826 main
	lightningd/lightningd.c:1111
0x7f88486810b2 ???
	???:0
0x56384edb52ad ???
	???:0
0xffffffffffffffff ???
	???:0
2021-05-12 11:25:41 +09:30
niftynei 558abe288a tests: explicitly mark tests to run both as v1+v2
By default, tests only run as v1 unless marked as v2.

These tests we want to run as both v1+v2

Includes fixes to have tests pass
2021-05-12 11:25:41 +09:30
niftynei dc758f616b tests: mark as v1 or v2 openchannel required 2021-05-12 11:25:41 +09:30
niftynei 3a2d602922 tests: add marker for v1/v2 channel opens
Tests that will only run when !EXPERIMENTAL_DUAL_FUND:

	@pytest.marker.openchannel('v1')
	def test_...()

Tests that will only run when EXPERIMENTAL_DUAL_FUND:

	@pytest.marker.openchannel('v2')
	def test_...()
2021-05-12 11:25:41 +09:30
niftynei d0bbf07655 tests: not DEVELOPER -> mark.developer
Nicer syntaxtic sugar for marking pytests as 'developer required'
2021-05-12 11:25:41 +09:30
niftynei b4e24ac8ba df: anchor outputs are on if EXP_DF
So we should treat it the same as EXPERIMENTAL_FEATURES
2021-05-12 11:25:41 +09:30
niftynei 484d6bde83 tests: move EXP_DF into the testing utils 2021-05-12 11:25:41 +09:30
niftynei 7a488cc11e df-tests: remove df_accepter plugin
The `funder` plugin, does the same stuff as this temporary test plugin,
so we move over to using that instead.
2021-05-12 11:25:41 +09:30
Rusty Russell d8e68893f5 bcli: become less aggressive with onchain fee levels.
Users are more upset recently with the cost of unilateral closes
than they are the risk of being cheated.  While we complete our
anchor implementation so we can use low fees there, let's
get less aggressive (we already have 34 or 18 blocks to close
in the worst case).

The changes are:

- Commit transactions were "2 CONSERVATIVE" now "6 ECONOMICAL".
- HTLC resolution txs were "3 CONSERVATIVE" now "6 ECONOMICAL".
- Penalty txs were "3 CONSERVATIVE" now "12 ECONOMICAL".
- Normal txs were "4 ECONOMICAL" now "12 ECONOMICAL".

There can be no perfect levels, but we have had understandable
complaints recently about how high our default fee levels are.

Changelog-Changed: Protocol: channel feerates reduced to bitcoind's "6 block ECONOMICAL" rate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-11 11:25:16 +09:30
Rusty Russell 9825f32874 lightningd: implement --log-timestamps=false.
Fixes: #4494
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: config: New option `log-timestamps` allow disabling of timestamp prefix in logs.
2021-05-05 17:19:19 -05:00
Christian Decker c6fd849aa3 pay: Add notification for pay_success 2021-05-03 11:20:15 +09:30
Christian Decker 98aa3c3da7 plugin: Make unannounced notification topics no longer fatal
Since plugins will start sending them soon, and they are likely to get
it wrong sometimes, be a bit more lenient, warn them in the logs
instead and then make sure it doesn't accidentally work anyway.
2021-05-03 11:20:15 +09:30
Christian Decker f963a6a551 libplugin: Add notification topics to plugin_main 2021-05-03 11:20:15 +09:30
Christian Decker 62e3358a5b plugin: Wrap custom notifications in a dict with additional origin
This should allow us to differentiate the origin of the notification,
and further prevent plugins from spoofing native notifications.
2021-05-03 11:20:15 +09:30
Christian Decker 083b41f090 plugin: Add a list of notification topics registered by plugin
We will eventually start emitting and dispatching custom notifications
from plugins just like we dispatch internal notifications. In order to
get reasonable error messages we need to make sure that the topics
plugins are asking for were correctly registered. When doing this we
don't really care about whether the plugin that registered the
notification is still alive or not (it might have died, but
subscribers should stay up and running), so we keep a list of all
topics attached to the `struct plugins` which gathers global plugin
information.
2021-05-03 11:20:15 +09:30
niftynei 29155c2fe8 tests: add test for funder options 2021-05-03 11:06:10 +09:30
niftynei cd5970243a funder: add a plugin, `funder`. policies for dual-funding
Behold! An immaculately concepted plugin for configuring your node to do
amazing things*

*fund channel open requests

Changelog-Added:  Plugins: Add `funder` plugin, which allows you to setup a policy for funding v2 channel open requests. Requres --experimental-dual-fund option
2021-05-03 11:06:10 +09:30
niftynei 7c76363e20 openchannel2: add channel_max_msat to openchannel2 hook payload
Changelog-Added: Plugins: add a `channel_max_msat` value to the `openchannel2` hook. Tells you the total max funding this channel is allowed to have.
2021-05-03 11:06:10 +09:30
niftynei 69cc9201de plugin-tests: cleanup outdated openchannel2 hook fields
We removed/changed the fields on the openchannel2 hook but never
updated this test (which doesn't run with the CI for #reasons)
2021-05-03 11:06:10 +09:30
niftynei 0e753224ae tests: remove EXPERIMENTAL_FEATURE flag from openchannel2 hooks
These are now included in all builds
2021-05-03 11:06:10 +09:30
Christian Decker 9470ea3032 pytest: Fix racy `test_unreachable_routehint`
It was failing likely because we were racing on the log
processing. Using `wait_for_log` instead should address that.
2021-04-24 13:58:57 +09:30
Rusty Russell e3d1115598 pytest: fix race in test_penalty_htlc_tx_timeout
We don't always get two transactions on line 1019; the comment is
confused (only one penalty tx successfully comes out of l3).  So make
sure we get the other transactions when we expect them, and then
make this test more specific.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-24 13:57:33 +09:30
Rusty Russell b352df4179 pytest: make it work with latest bitcoind master branch.
They seem to have changed the JSON output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-24 13:57:16 +09:30
Rusty Russell 32d650f9df lightningd: don't abort on incorrect versions, but try to re-exec.
You still shouldn't do this (you could get some transient failures),
but at least you have a decent chance if you reinstall over a running
daemon, instead of getting confusing internal errors if message
formats have changed.

Changelog-Added: lightningd: we now try to restart if subdaemons are upgraded underneath us.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #4346
2021-04-24 13:56:58 +09:30
Rusty Russell 4e2410742a tests/fuzz: update for fmt_amount changes, create pseudorand for fuzzing.
Reported-by: Christian Reitter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-21 06:51:03 +09:30
Rusty Russell 8cef2c17ad pytest: fix flake in test_v2_open_sigs_restart_while_dead
We didn't wait for the sendtx to complete before shutting down,
and hence timed out:

```
        l2.daemon.wait_for_log('Broadcasting funding tx')

        l1.stop()
        l2.stop()
        bitcoind.generate_block(6)
        l1.restart()
        l2.restart()

        # Make sure we're ok.
>       l2.daemon.wait_for_log(r'to CHANNELD_NORMAL')
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-12 23:03:47 +02:00
Rusty Russell c0a40b37a7 pytest: fix flake in test_pay_disconnect
Channel can be inactive before it disconnects, apparently.  Check
explicitly for the disconnected state so we get the expected error.

Here's what happened:
```
        # Can't pay while its offline.
        with pytest.raises(RpcError, match=r'failed: WIRE_TEMPORARY_CHANNEL_FAILURE \(First peer not ready\)'):
>           l1.rpc.sendpay(route, rhash)
E           Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```

And the logs show that the outgoing HTLC was sent to channeld before it
realized the connection was closed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-12 23:03:47 +02:00
Rusty Russell e684f87ef7 pytest: fix spurious failure in test_local_basepoints_cache.
Change the address in the canned db: it seems we won the lottery and
l1 connected, and got an error!

```
 E           ValueError:
 E           Node errors:
 E            - lightningd-1: had warning messages
 E           Global errors:
...
 lightningd-1: 2021-04-07T02:44:53.579Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_out WIRE_CHANNEL_REESTABLISH
 lightningd-1: 2021-04-07T02:44:53.579Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_in WIRE_GOSSIP_TIMESTAMP_FILTER
 lightningd-1: 2021-04-07T02:44:53.580Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_in WIRE_ERROR
 lightningd-1: 2021-04-07T02:44:53.580Z INFO    022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: error channel fdeb1ea12e02aa043f66ba581e969a1882d21142b19429995c6733bb71070bb6: Multiple channels unsupported
 lightningd-1: 2021-04-07T02:44:53.580Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Will try reconnect in 60 seconds
```

So I changed the port in the db to "1" which will never succeed:

```
sqlite> .dump peers
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE peers (  id INTEGER, node_id BLOB UNIQUE, address TEXT, PRIMARY KEY (id));
INSERT INTO peers VALUES(1,X'022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59','127.0.0.1:38723');
COMMIT;
sqlite> UPDATE peers SET address="127.0.0.1:1"
   ...> ;
sqlite> .dump peers
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE peers (  id INTEGER, node_id BLOB UNIQUE, address TEXT, PRIMARY KEY (id));
INSERT INTO peers VALUES(1,X'022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59','127.0.0.1:1');
COMMIT;
sqlite>
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-12 23:03:47 +02:00
Rusty Russell 9c3cf5aff9 newaddr: don't include "address" field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: `newaddr` no longer includes `address` field (deprecated in 0.7.1)
2021-04-07 14:34:39 +09:30
niftynei 1275af6fbb df-tests: update to take 'no channel_id' into account; use exp-df flag
The logs we're looking for don't exist anymore, we got rid of them.
2021-03-25 20:05:11 +10:30
Rusty Russell a52517bd81 pytest: fix up rbf tests which use dev-disconnect.
They must be marked DEVELOPER-only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-25 20:05:11 +10:30
niftynei e549746578 features: match up feature names to feature bit
Aligns the feature bit number to the name-label. Is this correct?
2021-03-25 20:05:11 +10:30
niftynei 3e8f575f9e dual-funding: convert to runtime flag, --experimental-dual-fund
You can now activate dual-funded channels using the
`--experimental-dual-fund` flag

Changelog-Changed: Config: `--experimental-dual-fund` runtime flag will enable dual-funded protocol on this node
2021-03-25 20:05:11 +10:30
niftynei 3c87e1f338 test_v2_open_sigs_restart: dont mine blocks until funding tx in mempool
This was flaky because sometimes we'd generate blocks before the funding
transaction reached the mempool.

If we wait until it's in, it works as expected.

[ Neatened to use wait_for_mempool arg --RR ]
2021-03-24 06:56:26 +10:30
niftynei 66a2826976 df-tests: make bad tests a bit better, maybe
We updated the "UNKNOWN TYPE channel_id" -> the actual channel id;
the reason for why the error fails shouldn't be restrictive (we just
know that it fails)
2021-03-24 06:56:26 +10:30
Christian Decker 6cfb72ea1b keysend: Add error when trying to send to self
There is little point in faking a self-payment, but we should also not
crash :-)

Fixes #4438

Changelog-Fixed: keysend: Keysend returns an error when a self-payment is requested
2021-03-19 10:18:42 +10:30
niftynei 2f2981c4e0 df: more reconnection tests 2021-03-17 10:25:18 +10:30
Rusty Russell 6c9d9ee9a2 connect: return address we actually connected to.
Otherwise, we might find an address other than the one given and
the user might think that address worked.

Fixes: #4185
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `connect` returns `address` it actually connected to
2021-03-17 08:38:08 +10:30
Rusty Russell 72ad260e9d pytest: fix now-obsolete use of fundchannel_complete via txid.
This was merged almost at the same time as the API change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-16 14:21:42 +10:30
Rusty Russell b563cafd83 lightningd: don't complain about bad funding PSBT for elements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-16 13:10:07 +10:30
Rusty Russell 22e1107581 lightningd/opening_control: deprecate old fundchannel_complete args.
And update all the in-tree callers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `fundchannel_complete` `txid` and `txout` parameters (use `psbt`)
2021-03-16 13:10:07 +10:30
Rusty Russell a1b43a3653 onchaind: see closes when wrong_funding shutdowns are used.
Fairly easy to do, though we also have to add the watch when we load
from the database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-16 13:08:40 +10:30
Rusty Russell b62706aa01 close: accept wrong_funding outpoint arg if we negotiated the feature.
Changelog-Added: lightningd: experimental-shutdown-wrong-funding to allow remote nodes to close incorrectly opened channels.
Changelog-Added: JSON-RPC: close has a new `wrong_funding` option to try to close out unused channels where we messed up the funding tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-16 13:08:40 +10:30
niftynei ef1e36efe5 mfc: check that we can retry when things go haywire
There's a version of this that keeps the PSBT in memory and does some
fancy addition/subtraction of unuseable parts for the v2's, however
it's much easier and simpler to simply error on the peer and re-start
from the very beginning.

This only works if we haven't gotten commitments from the peer yet (in
fact either method would only work if we haven't got commitments from
the peer yet), so if we've got commitments from them we simply mark them
as failed an go again.

In a perfect world, we'd remember what inputs we used last time, and
reuse those again on the re-attempt, which would pefectly guarantee both
that the failed opens (ones w/ commitments exchanged) would be canceled
after this completes (and we could re-try the failed again).

As it is, this is not perfect. It is, however, servicable.
2021-03-15 14:08:44 +10:30
niftynei bec96a6c5b df: add openchannel_abort command
Allows us to clean up an in-progress open that we won't be completing

Changelog-Added: EXPERIMENTAL JSON-RPC: Permit user-initiated aborting of in-progress opens. Only valid for not-yet-committed opens and RBF-attempts
2021-03-15 14:08:44 +10:30
niftynei 8182e9cea4 df: update the openchannel2 parameter 'accepter' -> 'our'
The `rbf_channel` hook uses `our_funding_msat`, which is a nicer
and more easily understood than the `openchannel2`
`accepter_funding_msat`.

This updates the `openchannel2` hook to use the same nomenclature as
`rbf_channel`.
2021-03-12 14:00:19 +10:30
Christian Decker ea92466025 pay: Abort the payment if destination is unreachable
We were not aborting if we had routehints, even though all routehints
may have been filtered out.

Changelog-Fixed: pay: `pay` will now abort early if the destination is not reachable directly nor via routehints.
2021-03-11 14:09:51 +10:30
Christian Decker cb21a5384b pay: Remove routehints with unreachable entrypoints
Changelog-Added: pay: `pay` will now remove routehints that are unusable due to the entrypoint being unknown or unreachable.
2021-03-11 14:09:51 +10:30
Christian Decker 1e1d7b387c pay: Filter out routehints whose entrypoint is unknown
We would happily spin on attempts that are doomed to fail because we
don't know the entrypoint. Next up: remove routehints whose
entrypoints are known but unreachable.
2021-03-11 14:09:51 +10:30
Christian Decker ad7f59f7a1 keysend: Set an error message if we abort due to missing TLV 2021-03-11 14:09:51 +10:30
Christian Decker 039751b875 wallet: Load local basepoints from the database alongside the hsmd 2021-03-09 14:58:29 +10:30
Christian Decker ae7d722ec5 wallet: Store local channel basepoints when a new channel is created 2021-03-09 14:58:29 +10:30
Christian Decker 4887a5a18f db: Add migration for the local basepoints and the funding pubkey 2021-03-09 14:58:29 +10:30
Christian Decker 2f6e33a7cd wallet: Add migration to cache local basepoints from the HSMd 2021-03-09 14:58:29 +10:30
niftynei 31e3bdb42d df-spec: consolidate dual-funding patches, update feerate protocol
We consolidate to the latest/singular RFC patch for dual-funding, so
there's just a single patchfile for the change. Plus we move back to the
opener setting the desired feerate, the accepter merely declines to
participate if they disagree with the set rate.
2021-03-09 14:55:05 +10:30
Christian Decker 21355edc43 plugin: Do not send the internal framed message over the wire
Looks like #4394 treated a symptom but not the root cause. We were
actually sending the message framed with the WIRE_CUSTOMMSG_OUT and
the length prefix over the encrypted connection to the peer. It just
happened to be a valid custommsg...

This fixes the issue, and this time I made sure we actually send the
raw message over the wire. However for backward compatibility we
needed to imitate the faulty behavior which is 90% of this patch :-)

Changelog-Fixed: plugin: `dev-sendcustommsg` included the type and length prefix when sending a message.
2021-03-09 14:39:22 +10:30
niftynei 0e8c49e077 Revert "df: turn off integration tests that depend on v2 funding"
This reverts commit 2e40a13508444cb48e63030593400ef34de0bd33.
2021-03-06 15:03:56 +10:30
niftynei 6a89a94fb5 df-tests: check that rbf includes overlapping inputs 2021-03-06 15:03:56 +10:30
niftynei 7ce7577254 df-tests: check that multiple RBFs and tx-sigs errors during RBF work 2021-03-06 15:03:56 +10:30
niftynei c1fcc1c819 df-rbf: test for interrupts during tx construction 2021-03-06 15:03:56 +10:30
niftynei e0a2d47903 df-rbf: reconnection tests (init_rbf + ack_rbf) 2021-03-06 15:03:56 +10:30
niftynei 73ac0fb6ce df-tests: happy path rbf test
I can has RBF?
2021-03-06 15:03:56 +10:30
Rusty Russell 4dcd4ca155 listpeers: add latest feerate and actual last fee amount.
Users have no idea what they would pay for unilateral closes.
At least this gives them a clue!

Reported-by: @az0re on IRC.

Changelog-Added: JSON-RPC: `listpeers` now shows latest feerate and unilaral close fee.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-05 13:12:41 +10:30
niftynei b8183f2eeb df: turn off integration tests that depend on v2 funding
It'll be broken til we're done re-writing this
2021-03-03 12:28:22 +10:30
Karol Hosiawa 7b72ea7061 Added listforwards test 2021-03-03 09:19:20 +10:30
Michael Schmoock d753ee27a2 pytest: test rpc_command hook chain 2021-03-03 09:18:53 +10:30
Rusty Russell 48e91da829 pytest: reduce test to edgecases for CI.
It's timing out, even though we disable valgrind.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-02 14:42:34 +10:30
Rusty Russell d0946b75bc common: support opt_shutdown_anysegwit checks (EXPERIMENTAL_FEATURES).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-02 14:42:34 +10:30
Rusty Russell 26627bdf7d openingd: check upfront shutdown script.
The spec doesn't say to do this, but it makes sense, otherwise
they'll never be able to mutually close the channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-02 14:42:34 +10:30
Christian Decker ebb1b19c65 plugin: Fix the custommsg hook not to include the internal prefix
We were always prefixing the `message` field with the internal type
prefix 0x0407, followed by the length prefix. Neither is needed since
the type being constant is of no interest to the plugin and the length
being implicit due to the JSON-encoding.

Reported-by: Ilya Evdokimov
Changelog-Fixed: plugin: The `custommsg` hook no longer includes the internal type prefix and length prefix in its `payload`
Changelog-Deprecated: plugin: The `message` field on the `custommsg` hook is deprecated in favor of the `payload` field, which skips the internal prefix.
2021-03-02 14:41:16 +10:30
Rusty Russell 804c2c2c20 pytest: test fee estimates which start working after channel establishment.
They need to specify fees to get a channel before this, but it's possible.
The test revealed no surprises (other than the last change).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-02 13:34:55 +10:30
Rusty Russell 4848c0022f plugins/bcli: fake minimum fee if we're in regtest mode.
Saves a great deal of confusion for regtest users.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: If bitcoind won't give a fee estimate in regtest, use minimum.
2021-03-02 13:34:55 +10:30
Rusty Russell 171af7742a plugin: log self-disable at debug level, other manifest fails at unusual.
This avoids spamming the logs.  We also remove the duplicate debug
logs on self-disable (plugin_kill logs it for us).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-01 12:16:42 +01:00
Rusty Russell ee5da52677 lightningd: differentiate cases of plugin death correctly.
If a plugin died due to connection close, we'd always say
"Plugin exited before completing handshake.", which was often
wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-01 12:16:42 +01:00
Rusty Russell c6bd87ccab pytest: fix "bad gossip" flake in test_forward_event_notification
If not all nodes are up-to-date with the new blocks, they can reject
announcements:

```
lightningd-4: 2021-02-23T02:02:47.832Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Ignoring future channel_announcment for 124x1x0 (current block 123)
lightningd-4: 2021-02-23T02:02:47.848Z DEBUG   lightningd: Adding block 133: 2d950451211398de9c10bf9df7eb53b385390eca31e306bc8fc1387b53d9f9a2
lightningd-4: 2021-02-23T02:02:47.865Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 124x1x0/0
lightningd-4: 2021-02-23T02:02:47.866Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 124x1x0/1
```

Technically, this change is not sufficient either, since *gossipd* might
not know about new block yet.  But it makes this case less likely.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
Rusty Russell f0fa5d1401 offers: handle re-fetching the same invoice twice.
We get a label clash: easy, just re-serve:

```
2021-02-18T04:29:37.474Z **BROKEN** plugin-offers: Failed invoice_request lnr1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcyyqwtp0rmsgquvuacqcl5cdfzwzmu3v8tqgvpqs8e80dlmxm7ey4xwrqdsqqqqqqqqqqqqqqqq2pqqfqpqynzqzx9rylzy40ernj4jzc3p2dwy3n8x6lqeaywwk725ghx4kx63pcfxgg2z3nsn80jzge06nt3ks8pr6rvnujq48376lpmrr3cq04nurpy783eyr0awh5773lrlmjek07rjf0nx4g9235ulkcs7jp2h5gumjyquhadh846da3jptxm9g0qz5lne4hjhag for offer 1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67: Got JSON error: {\"code\":900,\"message\":\"Duplicate label\",\"data\":{\"label\":\"1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67-08c5193e2255f91ce5590b110a9ae2466736be0cf48e75bcaa22e6ad8da88709-1\",\"bolt12\":\"lni1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcyyqwtp0rmsgquvuacqcl5cdfzwzmu3v8tqgvpqs8e80dlmxm7ey4xwzqrw4lauzsc2ajk26mv0ysxxmmxvejk2grxdaezqun4wd68jggvpkqqqqqqqqqqqqqqqqpgyqq7ypymf9efe2jj5r2mzunlqz67d75ht3ukxk0x9ftkcuknrgepsgupwfqpqynzqzx9rylzy40ernj4jzc3p2dwy3n8x6lqeaywwk725ghx4kx63pcf9qzxqt0dxq4zqwtz2qu44gzx7nzczc494cce2tgph5xgu5sn7vh8frky9z5n08xj9sp3yaxe9cqs5vss59r8pxwlyy3jl4xhrdqwz85xe9qqgcpda590qs9khxdx5qpetlx0j6ap0wsxagssmy2qjvhjp2kc3na54pht3pp76c405upne360lh8rzye32xxq6l0phpkk9pu9lwxnqkxuwt2nqqr9u\",\"payment_hash\":\"396250395aa046f4c58162a5ae31952d01bd0c8e5213f32e748ec428a9379cd2\",\"msatoshi\":7700446,\"amount_msat\":\"7700446msat\",\"status\":\"unpaid\",\"description\":\"Weekly coffee for rusty!\",\"expires_at\":1614832137,\"local_offer_id\":\"1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67\"}}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
Rusty Russell 874ca99c32 offers: make 'used' flag more useful.
We used to only set it for single-use offers (where it's required),
but it's still interesting for multi-use offers, so let's keep it
there.

We also put this field in the documentation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
niftynei db8fdb848c df-tests: send payment over opened channel
For good measure
2021-02-24 10:06:42 -06:00
niftynei 7ba6097d3c tests: test_no_fee_estimate robustify
Nested `with` exception checks don't work; fix flakes i'm seeing with
valgrind causing failures because blockchain not up to date when
`fundchannel` called.
2021-02-24 10:06:42 -06:00
Rusty Russell 9bb2b4a909 pytest: fix flake in test_no_fee_estimate()
We reserve utxos by blockheight, and producing 100 blocks can unreserve them!

```
>           raise RpcError(method, payload, resp['error'])
E           pyln.client.lightning.RpcError: RPC call failed: method: fundchannel, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'amount': 1000000, 'feerate': 'slow', 'announce': True}, error: {'code': -1, 'message': 'Aborting PSBT signing. UTXO d32e772e81db79c5eb3a8b58ab69174c540eefb6fa8b444fae300a7e35d5c987:0 is not reserved'}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-11 14:33:51 +01:00
Rusty Russell c397efee1c pytest: fix pay flake.
We weren't waiting for the channel to get to normal state:

E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt20m1pszfpezpp549um3vuyt52rgea32g7u55a5fv29yv7t94cmr2f0vjgcc33m3dvqdqzdgxqyjw5qcqp9sp59vu4tat2n53ylzrgxa95s5wu2s885a9llud64c0f6gjqts7h6tks9qy9qsq7u0j3w2h3sxxp9axpjxkz525znjn0t92gnrgk7y6plyq39zw9994g88xxjx52egk4965dp5qt2w08hk009eq9hm8nykwmxe7r705k8gpkqu9mw'}, error: {'code': 210, 'message': 'Ran out of routes to try after 1 attempt: see `paystatus`', 'attempts': [{'status': 'failed', 'failreason': 'Cannot attempt payment, we have no channel to which we can add an HTLC', 'partid': 1, 'amount': '2000000000msat'}]}

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-11 14:33:51 +01:00
Christian Decker 1341ca2ae0 pytest: Stabilize test_onchain_feechange
dev-disconnect seems to be causing warning that we need to allow.
2021-02-11 09:48:55 +01:00
Christian Decker 5a066d8536 pytest: Stabilize test_forward_event_notification
LightningNode.join_nodes and LightningNode.openchannel internally
generate blocks, which causes nodes to be out of sync, and ignore
"future" announcements, because they haven't seen that block yet.
2021-02-11 09:48:55 +01:00
Christian Decker ec04e1bee5 pytest: Stabilize funding cancel race by killing in parallel
It was using a trick to only shut down the first node, and forgetting
about the others. This could lead to processes not being stopped
correctly and to test failures because the directory isn't cleaned up
correctly.

Now we use the executor to shut as many nodes as possible in parallel.
2021-02-11 09:48:55 +01:00
Christian Decker 9869083a79 plugin: Check that we have a non-empty route in libplugin
Changelog-Fixed: pay: Payments with an empty route (self-payment) are now aborted.
2021-02-10 15:17:38 +01:00
Christian Decker d2281f53df pytest: Reproduce #4345 2021-02-10 15:17:38 +01:00
niftynei 5b2be130a5 df-tests: use `excess_as_change`, test two-sided channels better
Now that we've got an 'excess_as_change' flag, we can use it.

Further, make sure we test both directions on dual-funded channels
2021-02-04 13:25:34 -06:00
niftynei ea95ad9c12 fund/utxopsbt: flag 'excess_as_change' to add a change output for excess
In the case where you want a PSBT and also want the output to be added
as a change address, use `excess_as_change` = true.

Generates a change address to use. If you want to pay the excess
elsewhere, you will have to add separately.

Changelog-Added: JSON-RPC: Add new parameter `excess_as_change` to fundpsbt+utxopsbt
2021-02-04 13:25:34 -06:00
Rusty Russell f3159ec4ac pytest: detect warnings, too.
Since we turned many errors into warnings, we want our tests to fail
when they happen unexpectedly.  We make WARNING clear in the strings
we print, too, to help out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:52 +10:30
Rusty Russell 6b11cc8b8c common: disallow NULL channel_id to peer_failed_err.
No more sending "all-channel" errors; in particular, gossipd now only
sends warnings (which make us hang up), not errors, and peer_connected
rejections are warnings (and disconnect), not errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `peer_connected` rejections now send a warning, not an error, to the peer.
2021-02-04 12:02:52 +10:30
Rusty Russell f4ee41a989 common: remove peer_failed in favor of peer_failed_warn/peer_failed_err
And make all the callers choose which one.  In general, I prefer warn,
which lets them reconnect and try again, however some places are either
stated that they must be errors in the spec itself, or in openingd
where we abandon the channel when we close the connection anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we now send warning messages and close the connection, except on unrecoverable errors.
2021-02-04 12:02:52 +10:30
Rusty Russell f0659d0ab0 pytest: fix false bad gossip issue in test_forward
We construct the route manually so we may not have the
channel_announcement yet.  But we can get an update from the error
packet, which can lead to:

```
2021-01-29T01:38:23.4767334Z            ValueError: 
2021-01-29T01:38:23.4767987Z            Node errors:
2021-01-29T01:38:23.4768767Z             - lightningd-1: had bad gossip messages
2021-01-29T01:38:23.4769512Z            Global errors:
2021-01-29T01:38:23.4770300Z 
2021-01-29T01:38:23.4771109Z contrib/pyln-testing/pyln/testing/fixtures.py:197: ValueError
...
2021-01-29T01:38:23.7820197Z lightningd-1: 2021-01-29T01:26:57.460Z DEBUG   gossipd: Extracted channel_update 01027217b3086ad9f3dee1fa55b94c5fd2a4b0637bec70ba727ba4151a8de5173ddc749db3502d41ab0ae164addc8fd013d2088b6a12a2f478ae0affa94d76d8845c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000067000001000160136459010000060000000000000000000000010000000a000000003b023380 from onionreply 100d0000007500887217b3086ad9f3dee1fa55b94c5fd2a4b0637bec70ba727ba4151a8de5173ddc749db3502d41ab0ae164addc8fd013d2088b6a12a2f478ae0affa94d76d8845c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000067000001000160136459010000060000000000000000000000010000000a000000003b023380
2021-01-29T01:38:23.7837450Z lightningd-1: 2021-01-29T01:26:57.461Z DEBUG   gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x1/0
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:52 +10:30
Rusty Russell 06a54606a3 check-includes: allow redundant "config.h"
We should actually be including this (as it may define _GNU_SOURCE
etc) before any system headers.  But where we include <assert.h> we
often didn't, because check-includes would complain that the headers
included it too.

Weaken that check, and include config.h in C files before assert.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:36 +10:30
Rusty Russell 1a85edd207 lightningd: no longer forward failures to gossipd, let caller do it.
We fix up the test by using pay, instead of sendpay (and making pay log
the expected message).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: sendpay no longer extracts updates from errors, the caller should do it from the `raw_message`.
2021-02-02 13:44:01 +01:00
Rusty Russell 3c5502426b lightningd: addgossip API to inject gossip messages.
Importantly, this is synchronous, so pay will be able to use it
reliably.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-02 13:44:01 +01:00
Michael Schmoock bc40287ade pytest: peer_connected chainable tests 2021-02-01 09:57:15 +10:30
Michael Schmoock bdf0d60fd6 chore: fix typo in openchannel hook log
Nit: The underscore in "openchannel_hook" is wrong, bcause the name of
the hook is just "openchannel". The "_hook" implied this to be part of
the name.

Changelog-None
2021-02-01 09:57:15 +10:30
Michael Schmoock 3a0b1c5b1d pytest: improve test_openchannel_hook_chaining
The current test was not checking for the output of the first plugin in
the chain.
2021-01-29 13:37:42 +10:30
Michael Schmoock 67f2939540 pytest: custommsg chainable tests 2021-01-29 13:37:42 +10:30
Michael Schmoock 8e71c7a1f1 feat: make custommsg hook chainable
Changelog-Changed: custommsg hook is now chainable
2021-01-29 13:37:42 +10:30
Christian Decker 2834aaced0 gci: Stabilize test_forward_event_notification 2021-01-29 10:29:09 +10:30
Christian Decker 62cb1c3fbc pytest: Stabilize test_forward_stats 2021-01-29 10:29:09 +10:30
Christian Decker da2e956538 pytest: Stabilize test_routing_gossip
openchannel internally generates blocks, which may cause nodes to be
out of sync and ignore "future" channel announcements, resulting in
bad gossip.
2021-01-29 10:29:09 +10:30
Christian Decker 5ecaff65ee pytest: Give each run of the hsmtool its own pty 2021-01-29 10:29:09 +10:30
Christian Decker 1463797c61 pytest: Stabilize test_funding_close_upfront
Reconnections and unsynchronized states where causing us some issues.
2021-01-29 10:29:09 +10:30
Christian Decker bbdf35c6fe pytest: Stabilize test_closing_negotiation_reconnect
The test was not considering that concurrent sendrawtx of the same tx
is not stable, and either endpoint will submit it first. Now just
checking state transitions and the mempool.
2021-01-29 10:29:09 +10:30
Christian Decker 52e82b76b6 pytest: Stabilize test_bad_onion 2021-01-29 10:29:09 +10:30
Christian Decker 3d4c111721 pytest: Stabilize test_multiple_channels
If we're quick (or the node is slow) we end up reconnecting before our
counterparty has realized the state transition, resulting in an
unexpected re-establish.
2021-01-29 10:29:09 +10:30
Christian Decker 542f3225e3 pytest: Parameterize process waits for hsmtool calls
We were sometimes waiting only 5 seconds, which is way too short on a
heavily loaded machine such as CI. Making it 30 seconds and collecting
it in a single place so we can adjust more easily.
2021-01-29 10:29:09 +10:30
Christian Decker 8cc62d76e4 pytest: Stabilize test_channel_{spendable,receivable}
They were using TIMEOUT / 2 which may be way too long (hit against
test timeout), so we use a still ludicrous 30 seconds instead.
2021-01-29 10:29:09 +10:30
Christian Decker 6384cadd69 pytest: Stabilize the negotiation tests
We also make the logic a bit nicer to read. The failure was due to
more than one status message being present if we look at the wrong
time:

```
arr = ['CLOSINGD_SIGEXCHANGE:We agreed on a closing fee of 20334 satoshi for tx:17f1e9d377840edf79d8b6f1ed0faba59bb307463461...9b98', 'CLOSINGD_SIGEXCHANGE:Waiting for another closing fee offer: ours was 20334 satoshi, theirs was 20332 satoshi,']                                                                                                   │

     def only_one(arr):
         """Many JSON RPC calls return an array; often we only expect a single entry
         """
 >       assert len(arr) == 1
 E       AssertionError
```
2021-01-29 10:29:09 +10:30
Christian Decker 04ed93f5f8 pytest: Stabilize test_funding_external_wallet_corners 2021-01-29 10:29:09 +10:30
Christian Decker 18483ca582 pytest: Disable test_v2_open if not developer
It requires `--dev-force-features` which isn't available without
`DEVELOPER=1`
2021-01-29 10:29:09 +10:30
Christian Decker 7962db821c pytest: Stabilize test_channel_state_changed_bilateral 2021-01-29 10:29:09 +10:30
Christian Decker 07f5054700 pytest: Stabilize test_setchannelfee_state
Synching with the blockchain was slower than our timeout...
2021-01-29 10:29:09 +10:30
Christian Decker 03449e3cf0 pytest: Stabilize test_gossip_persistence
We weren't waiting for the `dev_fail` transaction to hit the mempool,
throwing the results off.
2021-01-29 10:29:09 +10:30
Christian Decker ae40c10bcb pytest: Stabilize test_onchain_timeout
The timeout on the pay future was too short under valgrind.
2021-01-29 10:29:09 +10:30
Christian Decker c564f165fa pytest: Stabilize test_penalty_htlc_tx_timeout
We weren't waiting for the transactions to enter the mempool which
could cause all of our fine-tuned block counts to be off. Now just
waiting for the expected number of txs.
2021-01-29 10:29:09 +10:30
Christian Decker c0f06f2779 pytest: Simplify and stabilize test_reconnect_no_update 2021-01-29 10:29:09 +10:30
Christian Decker 2b12cac31e pytest: Skip hsm encryption test if we don't have a TTY 2021-01-29 10:29:09 +10:30
Rusty Russell 001b5d6416 offers: make it a runtime option.
The fetchinvoice and offers plugins disable themselves if the option
isn't enabled (it's enabled by default on EXPERIMENTAL_FEATURES).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `experimental-offers` enables fetch, payment and creation of (early draft) offers.
2021-01-13 14:45:36 +01:00
Rusty Russell 27c006f7aa libplugin: make init return a string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: libplugin: init can return a non-NULL string to disable the plugin.
2021-01-13 14:45:36 +01:00
Rusty Russell 529ae0d766 plugins: allow plugins to disable themselves at startup.
By returning 'disable: <reason>' inside getmanifest or init result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: plugins: plugins can now disable themselves by returning `disable`, even if marked important.
2021-01-13 14:45:36 +01:00
Rusty Russell fc3e679c97 lightningd: control onion messages by `experimental-onion-messages` option.
Note that this also changes so the feature is not represented in channels,
reflecting the recent drafts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `experimental-onion-messages` enables send, receive and relay of onion messages.
2021-01-13 14:45:36 +01:00
niftynei 7e324177de df-tests: simultaneous openchannel_init (while in progress)
Reject a peer's request to open a channel while we're already in progress
2021-01-10 13:44:04 +01:00
niftynei 2c9ce2566c df-tests: test_lightningd_still_loading 2021-01-10 13:44:04 +01:00
niftynei a357e75c5d df-tests: test_disconnect_half_signed + test_reconnect_signed 2021-01-10 13:44:04 +01:00
niftynei fec1d9b707 df-tests: test_reconnect_openingd 2021-01-10 13:44:04 +01:00
niftynei 7eb6213fca df-tests: test_reconnect_sender_add1, others
We're still ignoring multifundchannel failure tests, but now we use the
'dual-fund' flag instead of a blanket true.
2021-01-10 13:44:04 +01:00
niftynei d9c68a708a df-tests: test_reconnect_receiver_add 2021-01-10 13:44:04 +01:00
niftynei 96c42262ab df-tests: test_peerinfo 2021-01-10 13:44:04 +01:00
niftynei 135cd178f7 df-tests: test_channel_persistence 2021-01-10 13:44:04 +01:00
niftynei a82bfa83ff df-tests: have the df_accepter plugin keep track of attempts
Test connection/reconnection handling for v2 opens. We needed to fixup
the accepter plugin so that we were freeing up inputs on
disconnect/failure.
2021-01-10 13:44:04 +01:00
niftynei 88b4b5b2bb df-tests: test_disconnect_opener 2021-01-10 13:44:04 +01:00
niftynei 3e330d7186 df-tests: test_forward_local_failed_stats 2021-01-10 13:44:04 +01:00
niftynei 973224dd14 df-tests: if we're using fundbalancechannel, set up plugin
nodes need to be using the accepter plugin, so they'll match funding.
we used to use push_msat for this, but v2 gets rid of it.

*sad trombone*
2021-01-10 13:44:04 +01:00
niftynei f822a93cf4 df-tests: add `setacceptmaxfunding` to df_accepter plugin
Method for setting a ceiling on the fund matching capabilities of
df_accepter. Setting it to zero means we don't fund the channel anymore.
2021-01-10 13:44:04 +01:00
niftynei e964c31369 df-tests: test_payment_success|failed_persistence 2021-01-10 13:44:04 +01:00
niftynei 09da171381 df-tests: test_io_logging
Move pid collection down to when dualopend is definitely dead
2021-01-10 13:44:04 +01:00
niftynei 609d509ca5 df-tests: test_channel_state_changed_unilateral 2021-01-10 13:44:04 +01:00
niftynei ac88c72f1e df-tests: test_channel_state_changed_bilateral 2021-01-10 13:44:04 +01:00
niftynei 450a094d26 df-tests: test_coin_movement_notices 2021-01-10 13:44:04 +01:00
niftynei 923e1f5c80 df-tests: test_feature_set 2021-01-10 13:44:04 +01:00
niftynei db65771f38 df-tests: test_plugin_feature_announce 2021-01-10 13:44:04 +01:00
niftynei 21b9bec93b df-tests: test_channel_state_change_history 2021-01-10 13:44:04 +01:00
niftynei 959ca793b0 df-tests: test_openchannel_hook_error_handling 2021-01-10 13:44:04 +01:00
niftynei 803d9701d3 df-tests: test_openchannel_hook_chaining 2021-01-10 13:44:04 +01:00
niftynei b0a8b10b41 df-tests: test_openchannel_hook
Have it check both openchannel + openchannel2
2021-01-10 13:44:04 +01:00
niftynei 0d45823b82 df-tests: test_funding_reorg_private 2021-01-10 13:44:04 +01:00
niftynei 09834bbc8d df-tests: test_node_reannounce correct features 2021-01-10 13:44:04 +01:00
niftynei c055198869 df-tests: test_fail_unconfirmed, duplicate with correct wires 2021-01-10 13:44:04 +01:00
niftynei dfa0eb0f2f df-tests: test_shutdown_awaiting_lockin check correct state 2021-01-10 13:44:04 +01:00
niftynei 5c7cc15a75 df-tests: adjust logsearch/features for test_dataloss_protection 2021-01-10 13:44:04 +01:00
niftynei e661d87440 df-tests: fixup expected features for dual-funding channels 2021-01-10 13:44:04 +01:00
niftynei 08200f03a7 df-tests: test_multifunding_v2_v1_mixed update to use flag
Now that we've got a fancy flag to enable dual funding, let's use it
2021-01-10 13:44:04 +01:00
niftynei 8fb97fd7e1 df-tests: updates for plugin hook_accepter helper
Fixes so that test_funding_close_upfront works with v2 opens
2021-01-10 13:44:04 +01:00
niftynei f4b7904ce5 tests: don't enforce daemon name on opening logline 2021-01-10 13:44:04 +01:00
niftynei 9951638d5f df: add flag to signal to run v2 channel open tests 2021-01-10 13:44:04 +01:00
niftynei 99a621dd99 df-reconnects: allow tx-sigs in channeld iff we're reconnecting
There's a case where a dropped funding_locked will result in the peer
moving onto channeld, while we stay in dualopend. As we haven't
received their funding_locked, we retransmit tx_sigs, which channeld
will need to handle.

With the patch the peer drops it on the floor; the peer will resend
funding_locked on reconnect, which will correctly advance us to
channeld and CHANNELD_NORMAL
2021-01-10 13:44:04 +01:00
Rusty Russell af46a4f57d fetchinvoice: allow amounts to be specified.
As per lastest revision of the spec, we can specify amounts in invoice
requests even if the offer already specifies it, as long as we exceed
the amount given.  This allows for tipping, and amount obfuscation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 4bb05e46e9 offers: convert currency when they request an invoice.
Means a reshuffle of our logic: we want to multiply by quantity before
conversion for maximum accuracy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 9681d491df offer: allow offers in other currencies if we can convert.
This avoids a footgun where they create an offer then we can't create
the invoice because they don't have a converter plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell 23af9d4972 onion_message: support variable-length onion messages.
Updated to the BOLT, and a few tweaks, and we can send giant onion_messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 12:45:31 +01:00
Rusty Russell d867fab3a7 sendinvoice: call waitinvoice directly to on invoice payment.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 1aa7e8e011 offers: split offer send_invoice generation into new JSON command
We split `send_invoice` offers inoo offerout (for want of a better name).
This simplifies the API.

Also took the opportunity to move the `vendor` tag to immediately
follow `description` (our tests use arguments by keywords, so no
change there).

Suggested-by: shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 55983827b6 fetchinvoice: try direct peers if we can't route.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell bdfcec3eb6 fetchinvoice: check we're in the period before attempting to fetch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell dae477175c fetchinvoice: implement timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 075c25fc08 plugins/fetchinvoice: handle sendinvoice timeout, error or payment.
If they pay the invoice, they don't bother replying; that's just for
errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Rusty Russell 43f5304539 plugins/offer: allow `offer` to create `send_invoice`, `refund` and `single_use` offers.
refund offers are implied send_invoice offers.  And send_invoice offers
are implied single-use offers, though it can also make sense to have
a non-send_invoice offer be single-use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-09 14:34:35 +10:30
Christian Decker 6b6e0fe34c pytest: Mark `test_lockup_drain` as disabled
It's been causing me headaches for some time, and it's just getting
annoying. Will file an issue to track progress.
2021-01-08 19:28:30 +01:00
niftynei d0dcb7cb42 tests: wait for drop to chain test_onchain_middleman_their_unilateral_in
this was catching the wrong 'sendrawtx exit 0' and consequently
not mining the close tx into a block
2021-01-08 19:28:30 +01:00
Rusty Russell 1a8978100e invoice: add ctlv option.
This is required if we want to create a "bouncer" plugin (in my copious free time!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now takes an optional `cltv` parameter.
2021-01-08 10:45:56 +01:00
Christian Decker 4b2efd69d7 pay: Fix a crash when waitblockheight times out
Fixes #4309
Changelog-Fixed: pay: Fixed an issue where waiting for the blockchain height to sync could time out.
2021-01-08 09:19:24 +01:00
Christian Decker ca204eacfa pytest: Reproduce waitblockheight timeout issue #4309 2021-01-08 09:19:24 +01:00
Christian Decker 70410b8ee8 pytest: Test LightningRpc and plugin command notification support 2021-01-07 08:32:40 +01:00
Christian Decker 84b3653606 pyln: Add command notification support for plugins
Changelog-Added: pyln-client: Plugin methods can now report progress or status via the `Request.notify` function
2021-01-07 08:32:40 +01:00
Christian Decker a1d33c3781 pytest: Add a test for `listinvoices`
We now have some more logic in the query, so let's test it
exhaustively.
2021-01-06 18:48:06 +01:00
Antoine Poinsot a4f07a3e78 tests/fuzz: add a fuzz target for hsm_encryption
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot 506d524508 libfuzz: add a NULL-termination in to_string
It's more useful if we actually want to use the output as, well, a
string..

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot 56c223886c lightning: confirm password on hsm_secret encryption
Changelog-changed: lightningd: the `--encrypted-hsm` now asks you to confirm your password when first set
Changelog-changed: hsmtool: the `encrypt` now asks you to confirm your password
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
niftynei 0c9072272f df-plugin: add `signonly` to signing in df_accepter
We're the accepter, so we look up which inputs we need to sign and only
sign those.
2021-01-05 19:12:00 +01:00
niftynei e3a1d1a7f3 pyln-client: to_whole_satoshi returns the rounded up satoshi value
A fractional satoshi value isn't really useful; rounding up loses
precision but that's why you called "whole satoshi", wasn't it?

Changelog-Changed: pyln-client: Millisatoshi has new method, `to_whole_satoshi`; *rounds value up* to the nearest whole satoshi
2021-01-05 19:12:00 +01:00
niftynei cce0020568 pyln-client: add missing `min_witness_weight` to fundpsbt/utxopsbt
We need to use it for the 'df_accepter' plugin, so we get the feerate
correct.

Changelog-Added: pyln-client: `fundpsbt`/`utxopsbt` now support `min_witness_weight` param
2021-01-05 19:12:00 +01:00
Antoine Poinsot 19afe15968 hsmtool: only write 32 bytes to hsm_secret on generatehsm
Fixes #4302
Changelog-fixed: hsmtool: the `generatehsm` command now generates an appropriately-sized hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot aa2586a45a pytest: reproduce issue #4302
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot 1513a2d07e hsmtool: input encryption password from stdin
This slightly breaks the API, but still accept the input: we just don't
take it into account anymore.

For `dumponchaindescriptors`, we have to still take the old place of the
`network` parameter into account to not entirely break the API.

Changelog-Added: hsmtool: password must now be entered on stdin. Password passed on the command line are discarded.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 16:15:40 +01:00
Christian Decker 5b3af00e28 pytest: Really only check DB statements if the env-var is "1" 2021-01-04 11:25:50 +01:00
Christian Decker 94820ad559 pytest: Disable test_funding_cancel_race on slow machines
The CI regularly trips up on this very taxing test (100 nodes) so only
run it if we have the required horsepower.
2021-01-04 11:25:50 +01:00
Christian Decker b3fd47e5f6 pytest: Stabilize test_routing_gossip
We were getting bad gossip because some nodes discarded the channel
announcement for being in the future. This is because the node was, at
that time, below the confirmation height. It'd then discard the
followup messages because not preceded by an announcement, and getting
upset about that.
2021-01-04 11:25:50 +01:00
Christian Decker 7e867e5ee6 pytest: Add throttler to limit load on the test system
Both my machine and apparently the CI tester machines regularly run
into issues with load on the system, causing timeouts (and
unresponsiveness). The throttler throttles the speed with which new
instances of c-lightning get started to avoid overloading. Since the
plugin used for parallelism when testing spawns multiple processes we
need to lock on the fs. Since we have that file open already, we'll
also write a couple of performance metics to it.
2021-01-04 11:25:50 +01:00
Rusty Russell 52af729641 plugins/offer and plugins/fetchinvoice: send and recv errors.
This also lets us extend our testing to cover error cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 96caf9f4ab fetchinvoice: return the next period for recurring offers.
This is useful for the caller to know when to call again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 19af1d10e6 invoice: hack in merkle of invoice as "payment_secret" (EXPERIMENTAL_FEATURES)
This lets actually pay the invoice that fetchinvoice returns.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 2baa410e2b pytest: finally test fetchinvoice send, and offer plugin receive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Rusty Russell 08e110b568 JSON: offers plugin `offer` command.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-23 14:24:39 +01:00
Karol Hosiawa e582c0ac0a Added listfunds test 2020-12-22 13:00:02 +01:00
Rusty Russell 723c16072a cleanups: feedback from Christian Decker review.
1. Hoist 7200 constant into the bolt12 heade2.
2. Make preimage the last createinvoice arg, so we could make it optional.
3. Check the validity of the preimage in createinvoice.
4. Always output used flag in listoffers.
5. Rename wallet offer iterators to offer_id iterators.
6. Fix paramter typos.
7. Rename `local_offer_id` parameter to `localofferid`.
8. Add reference constraints on local_offer_id db fields.
9. Remove cut/paste comment.
10. Clarify source of fatal() messages in wallet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-15 12:05:02 +01:00
Rusty Russell a33e39b7e8 pay, wallet: rename internal `bolt11` vars to `invstring`.
And handle bolt12 strings if EXPERIMENTAL_FEATURES.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-15 12:05:02 +01:00
Rusty Russell d971e3de98 Plugin: support extra args to "start".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `start` command can now take plugin-specific parameters.
2020-12-15 09:28:56 +10:30
Rusty Russell 8a9976c4c1 plugins: support concatenation of multiple args.
"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
2020-12-15 09:28:56 +10:30
Rusty Russell 646c564ec5 plugins: remove deprecated string plugin options.
This was fixed in 0.8.2.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: plugins: options to init are no longer given as strings if they are bool or int types (deprecated in 0.8.2).
2020-12-15 09:28:56 +10:30
Christian Decker f7cdf1dd98 pytest: Debug test_lockup_drain 2020-12-11 16:55:55 -06:00
Christian Decker 5a69b94f9a pytest: Add test for keysend feature support 2020-12-11 15:50:09 +01:00
Christian Decker 3b599b846e keysend: Check that the destination supports keysend upon init
We were blindly initiating the keysend payment, which could lead to
confusing outcomes. This adds a very specific error message to the
error returned.

Changelog-Fixed: keysend: Keysend now checks whether the destination supports keysend before attempting a payment. If not a more informative error is returned.
2020-12-11 15:50:09 +01:00
Christian Decker 71fafd23fd pytest: Add a test to reproduce #4258 2020-12-09 06:56:21 +10:30
Christian Decker eae49727b1 pytest: Disable flaky test_fundchannel_start_alternate
This test has been causing most CI runs to fail. See tracking issue #4265
for discussion on fixing and re-enabling it.

Changelog-None
2020-12-08 09:14:25 +10:30
Michael Schmoock ce1a2f134a pytest: tests fee_base and fee_ppm on listpeers 2020-12-07 14:26:36 +01:00
Christian Decker bb0910999e pytest: Use datetime.strptime instead of datetime.fromisoformat
The latter is available only starting with Python 3.7, and it's the
only test that fails with <3.7. This allows us to keep on testing with
older versions.
2020-12-04 18:34:12 -06:00
ZmnSCPxj jxPCSnmZ a437936c78 lightningd/log.c: Fix up handling of SIGHUP.
Fixes: #4240

ChangeLog-Fixed: log: Do not terminate on the second received SIGHUP.
2020-12-02 16:48:48 +10:30
ZmnSCPxj jxPCSnmZ 67d8fdcc75 tests/test_misc.py: Add a test for issue #4240. 2020-12-02 16:48:48 +10:30
niftynei b23c6c50a0 df-tests: use multifundchannel pathways instead of custom plugin
Since fundchannel/multifundchannel now handles v2 opens, we use that
instead of this hacked together plugin.
2020-12-02 14:19:08 +10:30
Michael Schmoock 899022e452 pytest: use default fundamount on two tests 2020-12-01 11:15:30 +10:30
ZmnSCPxj jxPCSnmZ 32de621886 lightningd/plugin_hook.c: Make `db_write` a chained hook.
Fixes: #4219

Changelog-Changed: Plugins: Multiple plugins can now register `db_write` hooks.
2020-11-30 10:40:11 +10:30
niftynei 405453859c mfc-df: add 'happy path' tests for the v1+v2 things
We can't test disconnects et.al. quite yet because the 'cancel' flow for
openchannelv2 still needs to be resolved
2020-11-23 12:41:05 -06:00
niftynei da98a9d0af df-accepter plugin: temporarily dont pass in signpsbt 2020-11-23 12:41:05 -06:00
niftynei 78d32b12d0 nit,df test plugin: change up how feerate is formatted
Suggested-by: @cdecker
2020-11-23 12:41:05 -06:00
niftynei 7e60ef6974 tests: make connection details correct 2020-11-16 20:00:51 -06:00
niftynei 4a3ee19a22 connectd: Update connection list with new address
If we're already attempting to connect to a peer, we would ignore
new connection requests. This is problematic if your node has bad
connection details for the node -- you can't update it while inflight.

This patch appends new connection suggestions to the list of connections
to try.

Fixes #4154
2020-11-16 20:00:51 -06:00
niftynei cd7d5cdff9 pytest: use the 'contrib' plugin set, not the 'test' set
They're clobbering each other, and the test hangs
2020-11-13 14:28:29 -06:00
Michael Schmoock 183f7698c3 chore: reduce CI load merging a testcase 2020-11-13 14:25:02 -06:00
Michael Schmoock 4134c24cc9 pytest: fix a probable flake on test_channel_opener 2020-11-13 14:25:02 -06:00
Michael Schmoock 90a4476f16 pytest: tests closer on unilateral close 2020-11-13 14:25:02 -06:00
fiatjaf c5aa2ea911 failure_onion: a test that only ensures lightningd doesn't crash. 2020-11-10 19:03:23 -06:00
niftynei 72dc97c0c1 pytest: silence pytest warnings about unregistered 'slow_test' marker
We've been getting a bunch of 'warnings' about an unknown mark. This
silences the warning, by registering the mark as expected.

=========================================================== warnings summary ===========================================================
tests/test_closing.py:152
  /home/niftynei/dev/lightning/tests/test_closing.py:152: PytestUnknownMarkWarning: Unknown pytest.mark.slow_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    @pytest.mark.slow_test

tests/test_closing.py:214
  /home/niftynei/dev/lightning/tests/test_closing.py:214: PytestUnknownMarkWarning: Unknown pytest.mark.slow_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    @pytest.mark.slow_test

tests/test_closing.py:704
2020-11-10 16:17:04 -06:00
Antoine Poinsot fc964edc06 pytest: test onchain wallet descriptor import on bitcoind
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Antoine Poinsot 4f2ae48c54 fuzz: add a fuzz target for common/descriptor_checksum
A small one just to check that we don't crash nor go out of bounds!

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Antoine Poinsot bf0a2f2b85 fuzz: add a 'to_string' function to libfuzz
We want to use it outside of fuzz-amount

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Rusty Russell 9a575a98a0 gossipd: simplify large reply code.
We used to create the entire reply, the if it was too big, split in
half and retry.

Now that the main network is larger, this always happens with a full
request, which is inefficient.

Instead, produce a reply assuming no compression, then compress as a
bonus.  This is simpler and more efficient, at cost of sending more
packets.

I also renamed an internal dev var to make it clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 17:37:10 -06:00
Rusty Russell 4745e7e3d0 pytest: make test_hook_dep tests more stable.
I think this is what Travis is having an issue with, but it work
fine locally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Michael Schmoock f3804daabb pytest: simplify test_openchannel_hook_chaining now order is deterministic. 2020-11-09 15:22:33 -06:00
Rusty Russell fb295ffb51 plugin: sort topological candidates by specified order.
We previously registered hooks up in who-replies-to-getmanifest-first
order, but then if any had dependencies it would scatter that order.

This allows users to manually set dependencies developers have
forgotten by specifying the plugins manually in their configuration or
cmdline.  This was an excellent consideration by @mschmook.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell d429e21db3 pytest: test that we maintain load order unless hook deps require a change.
Suggested-by: @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell 852e14c947 plugins: check order once all plugins have returned from getmanifest.
This means we need to stop at this stage even in the runtime-loaded
case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell a4f79e8e7e pytest: test that we don't complain about missing dependencies at startup.
We fail this, because we check dependencies as they come in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell 62c52fe868 libplugin: add support for before and after deps on hooks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell e16ed0e207 pyln: add support for dependent hooks.
And use that to add simple tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
ZmnSCPxj jxPCSnmZ 0fd87b85da openingd/: Fail `fundchannel_start` if we already are, or will become, the fundee.
Fixes: #4108

Changelog-Fixed: Network: Fixed a race condition when us and a peer attempt to make channels to each other at nearly the same time.
2020-11-07 17:18:49 +00:00
ZmnSCPxj jxPCSnmZ 712e0796fc tests/test_connection.py: Test for #4108. 2020-11-07 17:18:49 +00:00
Michael Schmoock 4a8722bf1d pytest: state_change history 2020-11-06 14:47:04 -06:00
Michael Schmoock 68ce25c92d feat: adds timestamp to state_change notification 2020-11-06 14:47:04 -06:00
Michael Schmoock 67f4970eb1 pytest: test opener and closer of listpeers 2020-11-06 14:47:04 -06:00
Michael Schmoock c52efe0f37 pytest: channel state change cause and message 2020-11-06 14:47:04 -06:00
Rusty Russell ae4139ba10 lightningd: note whether payment secret is used or not.
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets.  We need to know this so we know when
we can make them compulsory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-28 12:03:14 +01:00
niftynei 4ea9d9e928 df: Pass new feerate options through to plugin, set reasonable bounds
We let the plugin decide what feerate to accept/whether or not to add
funds to the open. To aid this decision, we also send the plugin what we
(c-lightning) currently have as our max and min acceptable feerates.

We also now use these as our default for max/min acceptable feerate
range when sending an openchannel offer to a peer.

In the future, it might be a good idea to make these more easily
changeable, either via a config setting (?) or a command param.
2020-10-27 19:52:05 -05:00
Rusty Russell 1e5789d421 close: add notification for slow closes.
For compatibility, we only do this if `allow-deprecated-apis` is false
for now.  Otherwise scripts parsing should use `grep -v '^# '` or
start using `-N none`.

Changelog-Added: JSON-RPC: `close` now sends notifications for slow closes (if `allow-deprecated-apis`=false)
Changelog-Deprecated: cli: scripts should filter out '^# ' or use `-N none`, as commands will start returning notifications soon
Fixes: #3925
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-23 13:53:16 +10:30
Rusty Russell 44f6b8a746 pytest: add notifications to tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-23 13:53:16 +10:30
Antoine Poinsot 85aa53c778 fuzz: add a runner Python script for fuzz targets
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot c149a88bbe fuzz: add a fuzz target for initial_channel
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 1aa94ccd9e fuzz: add a fuzz target for the close tx
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 33c7eccb3b fuzz: add a fuzz target for channel_id derivation
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot f0061d0606 fuzz: add a fuzz target for common/bip32
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot a89a018475 fuzz: add a fuzz target for common/bigsize
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 5d987f2dec fuzz: add a target for base32/64 encoding
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 63b1ae54a1 fuzz: add a target for common/bech32
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 2f90c45454 fuzz: add a fuzz target for amounts parsing
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
Antoine Poinsot 62b54d0125 build: introduce a fuzzing mode
This adds a new configuration, --enable-fuzzing (which is more than
welcome to be coupled with --enable-address-sanitizer), to pass the
fuzzer sanitizer argument when compiling objects. This allows libfuzzer
to actually be able "to fuzz" by detecting coverage and be smart when
mutating inputs.

As libfuzzer brings its own ~~fees~~ main(), we compile objects with
fsanitize=fuzzer-no-link, and special-case the linkage of the fuzz
targets.

A "lib" is added to abstract out the interface to the fuzzing tool used.
This allow us to use the same targets to fuzz using AFL, hongfuzz or w/e
by adding their entrypoints into libfuzz. (h/t to practicalswift who
introduced this for bitcoin-core, which i mimiced)

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
niftynei ede5f5be3c mfc: blackbox test for commitment vs funding tx feerates
Liquid is excluded because the mempool entry doesn't contain a 'weight'
field.
2020-10-21 11:24:16 +10:30
niftynei 26bc4f5239 tx,bugfix: correct signature length estimate
71-bytes for a signature already includes the sighash byte.

 2-bytes	30 44 (DER- prefix thing)
34-bytes	02 20 6e29c8df67fffdda1613cef1413eb1a9ef3627f1fc5e4d910837274eafcc7b2a (r)
34-bytes	02 20 4b8563d79b92fdd830a546862439f80b24132d09318af2c7220c791067067e29 (s)
 1-byte		01 (sighash)
==
71-bytes
2020-10-21 11:24:16 +10:30
Rusty Russell 639eddf840 common/gossmap: digest private channel information too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-21 08:58:34 +10:30
Rusty Russell 83aea6b2bb gossip_store: make private channels more similar to channel_announcement
Instead of a boutique message, use a "real" channel_announcement for
private channels (with fake sigs and pubkeys).  This makes it far
easier for gossmap to handle local channels.

Backwards compatible update, since we update old stores.

We also fix devtools/dump-gossipstore to know about the tombstone markers.

Since we increment our channel_announce count for local channels now,
the stats in the tests changed too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-21 08:58:34 +10:30
Rusty Russell c6625943b5 pytest: test that route can see private channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-21 08:58:34 +10:30
niftynei a7d4b8780e elementsd: disable dual-funding test; don't add non-witness utxo
Elementsd needs to be updated to v0.20.0 in order for this to work as
expected
2020-10-20 12:50:31 +10:30
niftynei b696ec89a5 df-open: use channel_id for openchannel_update and openchannel_signed
Be as specific as possible is a good rule for things, I think
2020-10-20 12:50:31 +10:30
niftynei 5fbd25f938 dual-fund: test (turned off) and plugin for executing an openchannel
Proof that v2 openchannel works, plus a plugin for sewing all the
requisite bits together
2020-10-20 12:50:31 +10:30
niftynei e2a6fd7112 common: pull up `param_psbt`
Usable other places, not just in wallet
2020-10-20 12:50:31 +10:30
niftynei 8f253b2adc test: simplify test for close_to
Since `fundchannel` now supports the 'close_to' argument, we can remove
all the logic needed to call fundchannel_start here.

Underneath, we're still calling `fundchannel_start`, we're just one (or
two, if you count multifundchannel) call levels away from it now.
2020-10-16 13:51:57 +10:30
niftynei de34f08b82 tests,fundchannel: return the result from the fundchannel rpc call
We need this so we can verify the 'close_to' result
2020-10-16 13:51:57 +10:30
Rusty Russell a3c30441d3 channeld: order htlcs by id before retransmission.
We had one report of this, and then Eugene and Roasbeef of Lightning
Labs confirmed it; they saw misordered HTLCs on reconnection too.

Since we didn't enforce this when we receive HTLCs, we never noticed :(

Fixes: #3920
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: fixed retransmission order of multiple new HTLCs (causing channel close with LND)
2020-10-14 19:58:57 +10:30
Rusty Russell d04597cbb6 pytest: add test for HTLC ordering.
[ Includes tidyups by Christian Decker ]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-14 19:58:57 +10:30
Rusty Russell d5c91a347a channeld: log broken message if we receive HTLCs out of order.
We didn't care, but other implementations (particularly lnd) do.  And it
does violate the spec.

(We need to use skip not xfail on the test which catches this, since
xfail doesn't seem to stop errors reported by cleanup)

(Includes Christian's typo fix!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-14 19:58:57 +10:30
Rusty Russell 6195d953cc plugins: use "slow" feerate for mutual close negotiation.
We're rarely in a hurry here, and bitcoind is aggressive with fees.
You can always spend this output if you really have to, using CPFP.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: mutual closing feerate reduced to "slow" to avoid overpaying.
2020-10-13 20:53:34 +02:00
Christian Decker 9f8c9d6406 pytest: Check for connection close and reset errors in timeout test
I get both errors from time to time, seems like a timing issue.
2020-10-13 14:30:45 +02:00
Rusty Russell 79a21755c4 pytest: fix flaky test if one side hasn't processed close yet.
```
    @unittest.skipIf(TEST_NETWORK != 'regtest', "External wallet support doesn't work with elements yet.")
    def test_funding_close_upfront(node_factory, bitcoind):
...

        # check that you can provide a closing address upfront
        addr = l1.rpc.newaddr()['bech32']
>       _fundchannel(l1, l2, amt_normal, addr)
...
pyln.client.lightning.RpcError: RPC call failed: method: fundchannel_start, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'amount': 100000, 'announce': True, 'close_to': 'bcrt1qctx2k9cu9fd7nk449mzphqjcvvpyc4rxh6826x'}, error: {'code': -1, 'message': 'They sent error channel 2a1ca624cd1127761cb7a4395df2c3fd6d0abb3732c1f85a5345b0da716540d0: Multiple channels unsupported'}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-12 13:14:39 +02:00
Christian Decker 7d6892abba pytest: Verification mode of expanded stmts for the db_write hook
This can be used to find instances of unparseable expanded SQL statements like
the one fixed in #4090
2020-10-10 11:34:43 +10:30
Rusty Russell 711133e3c7 pytest: fix flaky test.
l2 only gives up 100 blocks after it ignores the HTLC, so sometimes
this test fails.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-09 15:20:03 +10:30
Rusty Russell 2d897be10a pytest: fix experimental test.
Broken back in 3b8c0a7397 which
changed the behavior of funding not to mark as immediately spent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-08 11:00:23 +10:30
Christian Decker ee0ee046ff pytest: Stabilize test_penalty_htlc_tx_timeout
lightning-5 can sometimes see itself sweeping the unilateral output resulting
in this weird line:

```log
HTLC already resolved by SELF when we found preimage
```
2020-10-07 09:36:30 +10:30
Christian Decker 9021bb26d1 pyln: Decode process output once before storing it 2020-10-07 09:36:30 +10:30
Christian Decker 81569de709 pytest: Stabilize test_setchannelfee_{restart,zero}
In both cases the flakyness arises from the destination not knowing about the
modified fees of the forwarding node, thus including the outdated details in
the routehint, and the sender being unlucky and always trying with the
routehint anyway.

The long-term solutions to this is going to be #4111, this commit just reduces
the flakyness to get back to business.
2020-10-07 09:36:30 +10:30
Christian Decker 1563bbc2fa pytest: Cleanup test_gossip_ratelimit
Drive-by code cleanups I stumbled over while investigating the issue fixed in
the previous commit.
2020-10-07 09:36:30 +10:30
Christian Decker 05934724dd pytest: Don't start 3 nodes in test_gossip_ratelimit
We really are just interested in their on-chain footprint, so actually
starting the nodes is pointless overhead, and caused a lot of flakyness due to
the output ordering sometimes not matching up. We now just use the `bitcoind`
API to fund, sign and send a raw transaction that matches the stashed gossip
messages.
2020-10-07 09:36:30 +10:30