Commit Graph

2074 Commits

Author SHA1 Message Date
Simon Vrouwe 32fb8a42fd testing: cleanup in test_plugin_shutdown 2022-03-01 09:38:40 +10:30
Christian Decker 86b83e473b pytest: Remove 3 stress-tests
These tests have proven to be:

 a) very expensive, as they spin up many nodes, and perform long setup
 b) are not testing anything specific, they just fuzz functionality
    that is already tested otherwise
 c) have not helped pinpoint any issues in living memory
 d) are very flaky, making for really bad signal-to-noise, so much
    that devs usually just restart without even looking at the logs
 e) even if we were to look at the logs, we'd be unable to reproduce
    due to the inherent randomness involved in these tests
 f) are really noisy neighbors, causing other tests to flake as well,
    further muddying the water

All in all, these tests are a waste of time, and source of
frustration.

[ Cleaned up python unused imports --RR ]
Changelog-None
2022-02-26 14:36:50 +10:30
Rusty Russell 1da9b30b9a gossipd: don't send updates in error messages for unannounced channels.
This restores the behaviour prior to `lightningd: use our cached
channel_update for errors instead of asking gossipd.`, where gossipd
would refuse to give us channel_updates for unannounced channels.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-23 14:47:39 +10:30
Rusty Russell 01f2ca4fe7 pytest: remove test_htlc_rexmit_while_closing as too flaky.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-23 14:46:10 +10:30
Michael Schmoock 6db97b4235 pytest: check for remote_addr 2022-02-22 05:45:47 +10:30
Rusty Russell d0c7e18995 bitcoind: importmulti fails (bitcoin master), use importdescriptors
But this requires a watch-only wallet, and python-bitcoinlib doesn't support
multiple wallets, so we need to unload the original one, but then we need
to generate a block, so that can't generate a new address, so we need
an address arg to generate_block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-20 19:29:44 +10:30
Rusty Russell c4ec1d576e pytest: fix test_statictor_onions
It wasn't binding to .onion:<l2.port>, but .onion:9735.  Test both cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-20 19:29:44 +10:30
Rusty Russell 00cbe6959b pytest: YA bitcoind API break.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-20 19:29:44 +10:30
Christian Decker 787350eaa9 pytest: Test the rust bindings from cln-rpc 2022-02-11 16:02:25 +01:00
Rusty Russell bba9525cc3 pytest: note unfixable test_htlc_rexmit_while_closing test.
We really need our own lnprototest tests for packet-based stuff;
these message-based tests are inherently delicate and awkward.

In particular, connectd now does dev-disconnect, so the socket is not
immediately closed after a dev-disconnect command.  In this case, the
WIRE_SHUTDOWN has often already been written from connectd to channeld.

But it sometimes works, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell fc2401c1ac pytest: make test_mpp_adaptive more reliable
If the HTLCs are completely negotiated, we can get a channel break when
we mine a pile of blocks.  This is mainly seen with Postgres, due to the db
speed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell d9b1e69243 dual_funding: don't steal inflight field in update_channel_from_inflight.
If we call update_channel_from_inflight *twice* with the same inflight, we
will get bad results.  Using tal_steal() here was a premature optimization:

```
Valgrind error file: valgrind-errors.496395
==496395== Invalid read of size 8
==496395==    at 0x22A9D3: to_tal_hdr (tal.c:174)
==496395==    by 0x22B4B5: tal_steal_ (tal.c:498)
==496395==    by 0x16A13D: update_channel_from_inflight (peer_control.c:1225)
==496395==    by 0x16A4C7: funding_depth_cb (peer_control.c:1299)
==496395==    by 0x182807: txw_fire (watch.c:232)
==496395==    by 0x182AA9: watch_topology_changed (watch.c:300)
==496395==    by 0x1290ED: updates_complete (chaintopology.c:624)
==496395==    by 0x129BF4: get_new_block (chaintopology.c:835)
==496395==    by 0x125EEF: getrawblockbyheight_callback (bitcoind.c:362)
==496395==    by 0x176ECC: plugin_response_handle (plugin.c:584)
==496395==    by 0x1770F5: plugin_read_json_one (plugin.c:690)
==496395==    by 0x1772D9: plugin_read_json (plugin.c:735)
==496395==  Address 0x89fbb08 is 24 bytes inside a block of size 104 free'd
==496395==    at 0x483CA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==496395==    by 0x22B193: del_tree (tal.c:421)
==496395==    by 0x22B461: tal_free (tal.c:486)
==496395==    by 0x16A123: update_channel_from_inflight (peer_control.c:1223)
==496395==    by 0x16A4C7: funding_depth_cb (peer_control.c:1299)
==496395==    by 0x182807: txw_fire (watch.c:232)
==496395==    by 0x182AA9: watch_topology_changed (watch.c:300)
==496395==    by 0x1290ED: updates_complete (chaintopology.c:624)
==496395==    by 0x129BF4: get_new_block (chaintopology.c:835)
==496395==    by 0x125EEF: getrawblockbyheight_callback (bitcoind.c:362)
==496395==    by 0x176ECC: plugin_response_handle (plugin.c:584)
==496395==    by 0x1770F5: plugin_read_json_one (plugin.c:690)
==496395==  Block was alloc'd at
==496395==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==496395==    by 0x22AC1C: allocate (tal.c:250)
==496395==    by 0x22B1DD: tal_alloc_ (tal.c:428)
==496395==    by 0x22B3A6: tal_alloc_arr_ (tal.c:471)
==496395==    by 0x22C094: tal_dup_ (tal.c:805)
==496395==    by 0x12B274: new_inflight (channel.c:187)
==496395==    by 0x136D4C: wallet_commit_channel (dual_open_control.c:1260)
==496395==    by 0x13B084: handle_commit_received (dual_open_control.c:2839)
==496395==    by 0x13B6AF: dual_opend_msg (dual_open_control.c:2976)
==496395==    by 0x1809FF: sd_msg_read (subd.c:553)
==496395==    by 0x218F5D: next_plan (io.c:59)
==496395==    by 0x219B65: do_plan (io.c:407)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell e8d2176e6b pytest: protect against bad gossip messages from mining confirms too fast.
If we fund a channel between two nodes, then mine all the blocks to
announce it, any other nodes may see the announcement before the
blocks, causing CI to complain about "bad gossip":

```
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Ignoring future channel_announcment for 113x1x1 (current block 112)
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 113x1x1/0
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 113x1x1/1
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_NODE_ANNOUNCEMENT before announcement 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e
```

Add a new helper for this case, and use it where there are more than 2 nodes.

Cleans up test_routing_gossip and a few other places which did this manually.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 6d9f6ffd67 pytest: make test_gossip_no_empty_announcements more robust.
Don't assume gossip send order: explicitly disconnect and reconnect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell f7f9f35f2a pytest: remove flake in test_upgrade_statickey_onchaind
We were relying on the fee update to create an additional tx.  That's
ugly; do an actual payment and make sure we definitely complete a new
tx by waiting for that *then* both revoke_and_ack.

(Without this, we could get a unilateral close instead of a penalty).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 1c71c9849b connectd: handle custom messages.
This is neater than what we had before, and slightly more general.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON_RPC: `sendcustommsg` now works with any connected peer, even when shutting down a channel.
2022-02-08 11:15:52 +10:30
Rusty Russell 0ac5c4f8df pytest: ignore pings when doing query_gossip.
Next patch starts a timeout ping, which can interfere with results.

In theory, we should reply, but in practice (so far!) we seem to get enough
time that it doesn't hang up on us.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 5065bd6fc2 lightningd: use our cached channel_update for errors instead of asking gossipd.
We also no longer strip the type off: everyone handles both forms, and
Eclair doesn't strip (and it's easier!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 430a380e35 gossipd: feed lightningd the channel_updates as soon as we make them.
Even if we're deferring putting them in the store and broadcasting them,
we tell lightningd so it will use it in any error messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell c0e3155bb6 pytest: update elements for new, more accurate feerate calc.
And skip some tests: I'd simply be pasting in the results, which is
not very useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-27 12:22:36 +01:00
Rusty Russell de28bbd792 closingd, lightningd: use bitcoin_tx_2of2_input_witness_weight
This fixes lightningd's chronic weight underestimate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: closingd: more accurate weight estimation helps mutual closing near min/max feerates.
2022-01-27 12:22:36 +01:00
Rusty Russell 169c113d15 pytest: test that our closingd tx weight estimate is correct.
Compare against lightningd's, and the actual tx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-27 12:22:36 +01:00
niftynei 4dafeede5c coin moves: notify when we make deposits to external accounts
The blockheight is zero though, since these aren't included in a block
yet.

We also don't issue an 'external' deposit event if we can tell that the
address you're sending to actually belongs to our wallet (we'll issue a
deposit event when it gets included in a block)
2022-01-26 13:34:45 +10:30
Rusty Russell 16dd091d8b pytest: fix flake in test_upgrade_statickey_fail
```
        l1.rpc.disconnect(l2.info['id'], force=True)
        l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
    
>       l1.daemon.wait_for_log('option_static_remotekey enabled at 2/2')

tests/test_connection.py:3653: 
```

If l2's channeld gets killed (due to reconnect) before it tells
lightningd it got the revoke_and_ack it will need a retransmission
*again*.

This makes the test more robust, and does more checks too. 

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell 3281d72169 pytest: clean up test_channel_state_changed_unilateral
OK, now this test makes more sense!  Now we don't ignore errors, we
*will* drop to chain if we reconnect after one side has dropped to
chain.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-25 06:26:52 +10:30
Rusty Russell fc44846e10 pytest: disable test_closing_different_fees for elements temporarily.
There's actually a bug in our closing tx size estimation; I'll do
a separate patch for this, though.

Seems this used to be flaky, now we always flush queues, so it's
more reliably caught.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 3ccb3da2c5 pytest: disable automatic reconnection.
We seem to hit a race between manual reconnect (with address hint) and an automatic
reconnection attempt which fails:

```
 >       l4.rpc.connect(l3.info['id'], 'localhost', l3.port)
...
 E           pyln.client.lightning.RpcError: RPC call failed: method: connect, payload: {'id': '035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d', 'host': 'localhost', 'port': 41285}, error: {'code': 401, 'message': 'All addresses failed: 127.0.0.1:36678: Connection establishment: Connection refused. '}
```

See how it didn't even try the given address?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 109b3e62e9 pytest: disable tests/test_closing.py::test_onchain_all_dust
Here's the "Normal Tet Config clang-fuzzing" setup where it fails:

```
CC=clang
CONFIGURATOR_CC=clang
CWARNFLAGS=-Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror
CDEBUGFLAGS=-std=gnu11 -g -fstack-protector-strong
COPTFLAGS=
SQLITE3_CFLAGS=
SQLITE3_LDLIBS=-lsqlite3
POSTGRES_INCLUDE=-I/usr/include/postgresql
POSTGRES_LDLIBS=-L/usr/lib/x86_64-linux-gnu -lpq
VALGRIND=0
DEVELOPER=1
EXPERIMENTAL_FEATURES=0
COMPAT=1
```

Here's the truncated test output:

```
        if anchor_expected():
            expected_1['B'].append(('external', ['anchor'], None, None))
            expected_2['B'].append(('external', ['anchor'], None, None))
            expected_1['B'].append(('wallet', ['anchor'], None, None))
            expected_2['B'].append(('wallet', ['anchor'], None, None))
    
        tags = check_utxos_channel(l1, [channel_id], expected_1)
>       check_utxos_channel(l2, [channel_id], expected_2, tags)

tests/test_closing.py:2662: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils.py:321: in check_utxos_channel
    txid = matchup_events(u_set, evs, chans, tag_list)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

u_set = [[{'account_id': 'external', 'blockheight': 104, 'coin_type': 'bcrt', 'credit': '0msat', ...}, {'account_id': 'external', 'blockheight': 110, 'coin_type': 'bcrt', 'credit': '0msat', ...}]]
evs = [('external', ['to_them'], None, None), ('external', ['htlc_timeout'], None, None)]
chans = ['8ede62cea34c5196467c68175f70b8915f0edda421c5087a99584d4197cfb6c4']
tag_list = {'A': 'c5b6cf97414d58997a08c521a4dd0e5f91b8705f17687c4696514ca3ce62de8e', 'B': 'bb09b25d6653aeeec188961347ff80e90dca6f4a29cc017856f6585adb8cb468'}

    def matchup_events(u_set, evs, chans, tag_list):
        assert len(u_set) == len(evs) and len(u_set) > 0
    
        txid = u_set[0][0]['utxo_txid']
        for ev in evs:
            found = False
            for u in u_set:
                # We use 'cid' as a placeholder for the channel id, since it's
                # dyanmic, but we need to sub it in. 'chans' is a list of cids,
                # which are mapped to `cid` tags' suffixes. eg. 'cid1' is the
                # first cid in the chans list
                if ev[0][:3] == 'cid':
                    idx = int(ev[0][3:])
                    acct = chans[idx - 1]
                else:
                    acct = ev[0]
    
                if u[0]['account_id'] != acct or u[0]['tags'] != ev[1]:
                    continue
    
                if ev[2] is None:
>                   assert u[1] is None
E                   AssertionError
```
2022-01-20 15:24:06 +10:30
Rusty Russell e366cb17f6 pytest: fix flake in test_reconnect_sender_add1
l1 might split in a commitment_signed before it notices the disconnect, and this test fails:

```
        for i in range(0, len(disconnects)):
            with pytest.raises(RpcError):
                l1.rpc.sendpay(route, rhash, payment_secret=inv['payment_secret'])
>               l1.rpc.waitsendpay(rhash)
E               Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6115ed02e8 subdaemons: don't stream gossip_store at all.
We now let gossipd do it.

This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 029d65cf2e connectd: serve gossip_store file for the peer.
We actually intercept the gossip_timestamp_filter, so the gossip_store
mechanism inside the per-peer daemon never kicks off for normal connections.

The gossipwith tool doesn't set OPT_GOSSIP_QUERIES, so it gets both, but
that only effects one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell e37a638c0c connectd: do nagle by packet type.
channeld can't do it any more: it's using local sockets.  Connectd
can do it, and simply does it by type.

Amazingly, on my machine the timing change *always* caused
test_channel_receivable() to fail, due to a latent race.

Includes feedback from @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 7a514112ec connectd: do dev_disconnect logic.
As connectd handles more packets itself, or diverts them to/from gossipd,
it's the only place we can implement the dev_disconnect logic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 8f3b390356 pytest: make test_channel_state_changed_unilateral more robust.
This test started mostly failing (in non-DEVELOPER mode) after the
next patch, due to timing issues.

Handle both cases for now, and we'll add more enhancements later to
things we should be handling more consistently.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Michael Schmoock 019b31c522 options: only allow one DNS announcement 2022-01-02 16:01:38 +01:00
Rusty Russell b659fbbdf7 pytest: further deflake test_funding_push.
I also got an error under CI; it seems the sleep() was insufficient.
So try adding a sleep inside the check_coin_moves, which should cover
everyone.

```
                acct_moves = acct_moves[number_moves:]
            else:
>               if not move_matches(m, acct_moves[0]):
E               IndexError: list index out of range
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell 70ed47d77a channeld: add dev-disable-commit-after instead of dev-disconnect -nocommit
It was always a hack, but an impossible one once connectd will be
interpreting dev-disconnect!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 888745be16 dev_disconnect: remove @ marker.
Once connectd is doing this, we can't close as soon as we send,
and in fact we can't do 'fail write' either.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
Rusty Russell 560fa06f42 pytest: disable dev-disconnect blackhole tests.
These would have to be done by connectd, not the local daemon, once it's
intermediating.  Otherwise the remote peer won't see any change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
niftynei 70a73928cb balance-snaps: add a `balance_snapshot` event; fires after first catchup
Fire off a snapshot of current account balances (node wallet + every
'active' channel) after we've caught up to the chain tip for the *first*
time (in other words, on start).
2021-12-28 04:42:42 +10:30
niftynei f169597a02 test plugins/coin_moves: clean up/tidy coin moves plugin logic
removes unused in-memory stash of coin-moves; don't write the entire set
of coinmoves out on every update.
2021-12-28 04:42:42 +10:30
niftynei 8225a9decf coin_mvt: log events for pushes/lease_fees for leased channels
We need to stash/save the amount of the lease fees on a leased channel,
we do this by re-using the 'push' amount field on channel (which is
technically correct, since we're essentially pushing the fee amount to
the peer).

Also updates a bit of how the pushes are accounted for (pushed to now
has an event; their channel will open at zero but then they'll
immediately register a push event).

Leases fees are treated exactly the same as pushes, except labeled
differently.

Required adding a 'lease_fee' field to the inflights so we keep track of
the fee for the lease until the open happens.
2021-12-28 04:42:42 +10:30
niftynei bddd3694fa coin_mvt: record fees for an outbound htlc
If we initialized the payment, the fees are the entire fee-chain
(final hop amount - starting hop amount)

If it's a payment we routed, the fees are the diff between the
inbound htlc and the outbound (net gain by this routing)

Added to database so data persists nicely.
2021-12-28 04:42:42 +10:30
niftynei 29c6718297 coin_mvt: record new 'fees' field on htlc channel moves
We record the amount of fees collected for a routed payment. For
simplicity's sake on the data agg side, we record the fee payment on
*BOTH* the incoming htlc and the outgoing htlc. Note that this results
in double counting if you add up the fees from both an in-routed and
out-routed payment.
2021-12-28 04:42:42 +10:30
niftynei b6463174d6 coin moves: turn 'tag' into 'tags' array, add OPENER tag
Channels that the node has hopened will now be tagged with 'opener' in a
list of tags.
2021-12-28 04:42:42 +10:30
niftynei 4506f639fa coin_moves: remove 'index' for moves; bump version
Get rid of the 'movement_idx', since we replay events now.

Since we're removing a field from the 'coin_movement' event emission, we
bump the version type.

Changelog-Updated: `coin_movements` events have been revamped and are now on version 2.
2021-12-28 04:42:42 +10:30
niftynei 27ff87c045 coin_moves: de-dupe moves as they come in
test_onchain_dust_out restarts a node, which produces duplicate events.
this is expected, but we need to de-duplicate the events stream to get
accurate results
2021-12-28 04:42:42 +10:30
niftynei d2c4d4aec2 coin_mvts: rewrite how onchain events are recorded, update tests
The old model of coin movements attempted to compute fees etc and log
amounts, not utxos. This is not as robust, as multi-party opens and dual
funded channels make it hard to account for fees etc correctly.

Instead, we move towards a 'utxo' view of the onchain events. Every
event is either the creation or 'destruction' of a utxo. For cases where
the value of the utxo is not (fully) debited/credited to our account, we
also record the output_value. E.g. channel closings spend a utxo who's
entire value we may not own.

Since we're now tracking UTXOs onchain, we can now do more complex
assertions about the onchain footprint of them. The integration tests
have been updated to now use more 'chain aware' assertions about the
ending state.
2021-12-28 04:42:42 +10:30
Vincenzo Palazzo 43ff949ea7 lightningd: support hsm error code
Suggested-by: Rusty Russell
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Changed: Support hsm specific error error code in lightning-cli
2021-12-15 12:24:54 +10:30
Rusty Russell 1ac5a1b7e2 pytest: restore gossip advertizement tests for default port.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-14 13:41:31 +10:30
Simon Vrouwe 28816c387c testing: remove test_htlc_accepted_hook_shutdown, move relevant parts
into test_plugin_shutdown

The fact that plugins are kept alive untill the very end also ensures
their hooks are not silently unregistered.
2021-12-14 09:33:10 +10:30
Simon Vrouwe 9b70c9d63b testing: fix flake in test_fetchinvoice_3hop 2021-12-14 09:33:10 +10:30
Michael Schmoock c2d2cc1274 connectd: fix empty error message
1. Adds the missing DNS error massages so they can be handled by
   connect_control.
2. Prepends a 'All addresses failed' to code 401 message, so we
   always have at least some error message to the user.

Changelog-None
2021-12-08 13:52:24 +01:00
Michael Schmoock bad09887e0 pytest: fix test_pluginconnected_hook_chaining
The last line of the testcase was checking on the wrong node l3
instead of l1. l3 didn't had the plugins configured that would
produce the log entry we were looking for not to be present.

Changelog-None
2021-12-06 14:53:37 +01:00
Jan Sarenik 96f28323bd Set default port according to network
The idea is to have different default ports for different networks.

Current default port is `9735` for everything. Let's use it for
the mainnet and reuse the difference added to the default port
from `rpc_port` values in `bitcoin/chainstate.c`.

Testnet would be `19735` (adding rpc_port - 8332 = `10000`).

Signet would be `39735` (adding rpc_port - 8332 = `30000`).

Regtest would be `19846` (adding rpc_port - 8332 = `10111`).

With Vincenzo's kind pair-programming help over tmate.

Two other commits were squashed into this one so that bisecting
never ends up in half-baked state:

1. chainparams: Fix regtest default rpc_port

   bitcoind -help says this:

    -rpcport=<port>
         Listen for JSON-RPC connections on <port> (default: 8332, testnet:
         18332, signet: 38332, regtest: 18443)

2. test_gossip: Default port for regtest

   hex: 2607 is now .... (could be 4d86 but Elements uses another port)
   dec: 9735 is now any port (could be 19846 ^^ but now is for any port)

   The lines which were binding to default port were removed as the
   default port is different on each network.

NOTE: Remember not to modify gossip_store tests which loads everything raw
      including the checksums.

Changelog-Changed: If the port is unspecified, the default port is chosen according to used network similarly to Bitcoin Core.
2021-12-06 17:10:08 +10:30
Rusty Russell 4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell bdabef9a9b pytest: fix occasional flake in test_announce_address
We make sure the gossip msg was sent, but the other node might not
have digested it yet:

```
        # Check other node can parse these
>       addresses = l2.rpc.listnodes(l1.info['id'])['nodes'][0]['addresses']
E       KeyError: 'addresses'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:18 +10:30
Rusty Russell b45544c659 options: fix handling of wildcard (allproto) address.
We treated ':' as an empty DNS name in EXPERIMENTAL, which is wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-05 16:43:09 +01:00
Rusty Russell cf40e585c3 pytest: test to show conflict between websocket and wildcard addresses.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-05 16:43:09 +01:00
Andrew Toth 69bc1191cb tests: add pay test for exclude arg 2021-12-04 21:37:42 +10:30
Rusty Russell 5a5cf8c696 pytest: fix flake in testing.
As noted in 0a406230d0 (diff-5871d4c569454db5e625383975462132da0bd03d32df145d8d72d8fafd86d952R3544-R3546)

Turns out we sometimes hang up before l2 sees the previous tx revoked,
so we get a normal unilateral close, not a cheat.

Reported-by: Simon Vrouwe
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 09:48:00 +10:30
Rusty Russell 605b3bf985 pytest: re-enable modern/obsolete fetchonion tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell b74848f6f6 common: remove support for pre v0.10.2 onionmessages.
Temporarily disable sendpay_blinding test which uses obsolete onionmsg;
there's still some debate on the PR about how blinded HTLCs will work.

Changelog-EXPERIMENTAL: onionmessage: removed support for v0.10.1 onion messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Simon Vrouwe aae5e7822f testing: fix test_upgrade_statickey_onchaind
Seems a timing issue that should be figured out, his makes the test pass.
2021-11-30 13:34:44 +10:30
Simon Vrouwe 2240f09f8b testing: fix test_closing_higherfee
shutdown_subdaemons frees the channel and calls destroy_close_command_on_channel_destroy, see gdb:

    0  destroy_close_command_on_channel_destroy (_=0x55db6ca38e18, cc=0x55db6ca43338) at lightningd/closing_control.c:94
    1  0x000055db6a8181b5 in notify (ctx=0x55db6ca38df0, type=TAL_NOTIFY_FREE, info=0x55db6ca38e18, saved_errno=0) at ccan/ccan/tal/tal.c:237
    2  0x000055db6a8186bb in del_tree (t=0x55db6ca38df0, orig=0x55db6ca38e18, saved_errno=0) at ccan/ccan/tal/tal.c:402
    3  0x000055db6a818a47 in tal_free (ctx=0x55db6ca38e18) at ccan/ccan/tal/tal.c:486
    4  0x000055db6a73fffa in shutdown_subdaemons (ld=0x55db6c8b4ca8) at lightningd/lightningd.c:543
    5  0x000055db6a741098 in main (argc=21, argv=0x7ffffa3e8048) at lightningd/lightningd.c:1192

Before this PR, there was no io_loop after shutdown_subdaemons and client side raised a
general `Connection to RPC server lost.`
Now we test the more specific `Channel forgotten before proper close.`, which is good!

BTW, this test was added recently in PR #4599.
2021-11-30 13:34:44 +10:30
Simon Vrouwe 89cbdf4dce testing: test that RPC calls made in shutdown fail and receive error code -5
and the two conditions in which plugins can receive shutdown notification
2021-11-30 13:34:44 +10:30
Simon Vrouwe ef503f2fea testing: remove test_stop_pending_fundchannel
Not needed anymore, see previous commit
2021-11-30 13:34:44 +10:30
Simon Vrouwe 5fb3674233 testing: test hook semantics is preserved in shutdown
plugins expect their hooks to work also in shutdown, see issue #4883
2021-11-30 13:34:44 +10:30
Michael Schmoock a3ea9fdc87 chore: use EXPERIMENTAL for BOLT7 DNS #911
Changelog-EXPERIMENTAL: Ability to announce DNS addresses
2021-11-30 09:38:17 +10:30
Michael Schmoock 2b8896a0b5 pytest: test connecting to a DNS only announced node 2021-11-30 09:38:17 +10:30
Michael Schmoock 01e8a523e9 bolt7: allow announcement of ADDR_TYPE_DNS 2021-11-30 09:38:17 +10:30
Rusty Russell 7a6fd70078 pytest: Reduce memory consumption by test_plugin_disable
It runs 6 nodes: under valgrind this ends up consuming 5.3 GB RSS.  By
stopping nodes between, we peak about 1G RSS.

Measured using:
	(while true; do echo $(for i in 4 5 6; do ps uh | tr -s ' ' | cut -d\  -f$i | tally; done); sleep 5; done)&

(Which measures my other processes as well, but that's only about 100M).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-27 12:41:37 +00:00
ZmnSCPxj jxPCSnmZ a294683675 wallet/db_sqlite3.c: Support direct replication of SQLITE3 backends.
ChangeLog-Added: With the `sqlite3://` scheme for `--wallet` option, you can now specify a second file path for real-time database backup by separating it from the main file path with a `:` character.
2021-11-17 12:10:07 +10:30
Rusty Russell 68043c2e8c common: clean up autodata in common_shutdown().
valgrind locally complains about the allocations in autodata leaking:

```
==138200== 16 bytes in 1 blocks are still reachable in loss record 1 of 2
==138200==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==138200==    by 0x10D41A: autodata_register_ (autodata.c:20)
==138200==    by 0x10E7B8: register_autotype_type_to_string (type_to_string.h:79)
==138200==    by 0x10F5CA: register_one_type_to_string0 (block.c:259)
==138200==    by 0x19734C: __libc_csu_init (in /home/rusty/devel/cvs/lightning/common/test/run-route-specific)
==138200==    by 0x4A3D03F: (below main) (libc-start.c:264)
==138200== 
==138200== 176 bytes in 1 blocks are still reachable in loss record 2 of 2
==138200==    at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==138200==    by 0x10D472: autodata_register_ (autodata.c:26)
==138200==    by 0x122D37: register_autotype_type_to_string (type_to_string.h:79)
==138200==    by 0x122F1F: register_one_type_to_string0 (node_id.c:50)
==138200==    by 0x19734C: __libc_csu_init (in /home/rusty/devel/cvs/lightning/common/test/run-route-specific)
==138200==    by 0x4A3D03F: (below main) (libc-start.c:264)
==138200== 
make: *** [Makefile:638: unittest/common/test/run-route-specific] Error 7
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 82c66b10df pytest: allow bad gossip in pruning test.
If we forget a channel, we can get upset when we get an update about it:

```
2021-11-04T00:35:43.8242370Z lightningd-3: 2021-11-04T00:29:22.073Z DEBUG   gossipd: Pruning channel 103x1x1 from network view (ages 61 and 22s)
...
2021-11-04T00:35:43.8263502Z lightningd-3: 2021-11-04T00:29:22.509Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x1/0
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 65bb989cf1 pytest: don't checksum plugins on startup in VALGRIND developer mode.
This loads up 20MB of plugins temporarily; we seem to be getting OOM
killed under CI and I wonder if this is contributing.

Doesn't significantly reduce runtime here, but I have lots of memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 09bbda4bca pytest: don't create 5 nodes in test_fetchinvoice.
CI seems to be OOM killing us; 5 may be too many under valgrind.

VALGRIND=1 pytest tests/test_pay.py::test_fetchinvoice 
Before:
	1 passed in 199.33s (0:03:19)

After:
	1 passed in 177.91s (0:02:57)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 9d18180172 lightningd: really do allow two Torv3 addresses.
This surprised me, since the CHANGELOG for [0.8.2] said:

	We now announce multiple addresses of the same type, if given. ([3609](https://github.com/ElementsProject/lightning/pull/3609))

But it lied!

Changelog-Fixed: We really do allow providing multiple addresses of the same type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 2f247c7bfb torv2: remove support for advertizing and connecting.
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.

I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell c00202a0da pytest: remove test_shutdown_alternate_txid.
We're about to require that fundchannel_complete() take a PSBT, where it
does sanity checks to avoid this error, making this a difficult mistake
to make.

Is it time to remove this functionality anyway?  @cdecker?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Christian Decker 012c2ecf17 pytest: Create less dust in `test_delpay_payment_split`
We were triggering the dust panic.
2021-10-23 12:59:13 +02:00
niftynei 303fbce479 tests: raise dust limit on mpp test
Fails liquid-regtest otherwise; liquid tends to hit the dust limit
earlier than non-liquid tx, and MPP exacerbates this by divvying up
payments into dusty bits then attempting to shove them through the same
channel, hitting the dust max. The MPP then fails as not all the parts
were able to arrive at their destination.
2021-10-23 12:59:13 +02:00
niftynei b57fed047a dusty htlcs: don't fail the channel, make it error a whole bunch
Let's make this a softer launch by just warning on the channel til the
feerates go back down.

You can also 'fix' this by upping your dust limit with
the `max-dust-htlc-exposure-msat` config.
2021-10-23 12:59:13 +02:00
niftynei 150d065088 tests: check for incoming + outgoing dust limits 2021-10-23 12:59:13 +02:00
niftynei 208b161226 tests: bump the ceiling for dust, fixes failures for liquid
Liquid's threshold for dust is a bit higher, so we bump up the max limit
here so we can actually get the whole MPP'd payment sent over the wire
2021-10-23 12:59:13 +02:00
Rusty Russell ed6eaf9171 experimental-websocket-port: option to create a WebSocket port.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-22 11:56:30 +02:00
Rusty Russell c503232cde common: use bitcoin_outpoint.
I started pulling this thread, and the entire codebase got unravelled.

Oh well, it's done now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell 012dedc3d8 fuzz: make it build again.
How did this pass CI?  I saw this break in my PR, but it's (long)
broken in master.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Vincenzo Palazzo 67b790e3dc tests: Change currency from `msat` to `sat` in the regex string
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-10-14 10:09:20 +10:30
Christian Decker cd7d87f98e pay: listpays groups by payment_hash and groupid
Fixes #4482
Fixes #4481

Changelog-Added: pay: Payment attempts are now grouped by the pay command that initiated them
Changelog-Fixed: pay: `listpays` returns payments orderd by their creation date
Changelog-Fixed: pay: `listpays` no longer groups attempts from multiple attempts to pay an invoice
2021-10-13 13:41:18 +10:30
Christian Decker 98f74762a5 pytest: Adjust `test_sendpay` to the new semantics 2021-10-13 13:41:18 +10:30
Christian Decker ce3d3d8e54 pytest: Fix `test_onchain_timeout` to use `groupid` 2021-10-13 13:41:18 +10:30
Christian Decker 428982a9e7 pytest: Add `groupid` to `test_partial_payment_{timeout,restart}` 2021-10-13 13:41:18 +10:30
Christian Decker bc089d3350 pytest: Fix up `test_partial_payment` to use a single `groupid` 2021-10-13 13:41:18 +10:30
Christian Decker 324257b2bf pytest: Reproduce #4482 2021-10-13 13:41:18 +10:30
Rusty Russell c936540e0b pytest: fix flake in test_upgrade_statickey_onchaind
Reconnect manually, don't wait for automatic reconnect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-10 15:32:57 +02:00
Rusty Russell b2206744c0 pytest: fix flake when test reconnects by itself.
If it reconnects by itself, it will get a warning message:

```
lightningd-2: 2021-10-08T01:40:42.446Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#3: billboard: Sent reestablish, waiting for theirs
lightningd-2: 2021-10-08T01:40:42.446Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#3: peer_in WIRE_ERROR
lightningd-2: 2021-10-08T01:40:42.447Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#3: billboard perm: Received error channel 0a6220a3e904d17e72b5c3499928dc8a65720063c6395c999a129a0ff0b06afb: Forcibly closed by `close` command timeout
lightningd-2: 2021-10-08T01:40:42.448Z INFO    0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#3: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: error channel 0a6220a3e904d17e72b5c3499928dc8a65720063c6395c999a129a0ff0b06afb: Forcibly closed by `close` command timeout
```

And this will make CI complain.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-10 15:32:57 +02:00
Rusty Russell c9b82bf1d2 channeld: restore ping command, but only for channeld.
It's probably not worth fixing for the other daemons.

Changelog-Changed: JSON-RPC: `ping` now only works if we have a channel with the peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-10 15:32:57 +02:00
Rusty Russell be2622a4ff channeld: perform regular keepalive pings.
Send a ping every 15-45 seconds.  If we try to send another one and we
haven't got a reply, hang up.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Send regular pings to detect dead connections (particularly for Tor).
2021-10-10 15:32:57 +02:00
Rusty Russell 9138ebf807 channeld: remove liveness logic pre-commitment.
We're going to continuously ping, so this is redundant.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-10 15:32:57 +02:00