Commit Graph

11126 Commits

Author SHA1 Message Date
Christian Decker 4b9bf22193 pyln: Migrate pyln-testing to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker dd8d2c138c pyln: Migrate pyln-client to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker b0f8a99310 pyln: Migrate pyln-bolt7 to PEP517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker 74fd685219 pyln: Fix the pyln-proto version and migrate to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Rusty Russell a3696f0469 gossipd: allow faster updating, 2 per day, before ratelimiting.
This limit applies to both node_announcements (which we now send 1 per
day), and channel_updates; I've had reports of LND nodes going down
daily for database compation, so they end up ratelimited.

Changelog-Protocol: We now allow two channel_updates or node_announcements per day, up from 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-08 04:54:45 +10:30
Rusty Russell 93c00cc07d gossipd: force a fresh node_announcement every 24 hours.
Even if nothing has changed.  Note that this is different from simply
re-xmitting the old one, in that it has a different timestamp, so others
will re-xmit it too.

This is a side-effect of reports that node_announcement propagation
through the network is poor:

	https://github.com/ElementsProject/lightning/issues/5037

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: We now refresh our ndoe_announcement every 24 hours, due to propagation issues.
2022-03-08 04:54:45 +10:30
Rusty Russell bf8cb640b7 gossipd: always re-sent our own node_announcement on startup.
Even if it hasn't changed, re-send it to everyone.  That should help
propagation a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-08 04:54:45 +10:30
Rusty Russell d991f13516 gossipd: make sure we set the `urgent` bit if we move our own node_announcement.
node_announcement has to follow at least one channel_announcement.
When channels close, if this isn't the case, we remove the old
node_announcement and put it at the end of the gossip_store.

But we lose the "send even if they don't want it" bit in the case it's
our own node_announceent, so keep it.

This only happens if you don't change your node configuration at all
since you opened your first channel, but still worth fixing.

We expose the force_node_announce_rexmit() for later use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-08 04:54:45 +10:30
Rusty Russell b5a1715c2b connectd: also fail without a scary backtrace when listen fails.
For example, if you do:

```
./lightningd/lightningd --network=regtest --experimental-websocket-port=19846
```

Then you're trying to reuse the normal port as the websocket port, but this
only fails at *listen* time, when we activate connectd.  Catch this too.

Fixes incorrect fatal() message, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell 885a6f50ae connectd: make sure we announce websocket addr which succeeded.
By accessing `addr` after the loop, it's possible that it's one which
failed, in complex scenarios.

Also gives us a chance to warn if they specify a websocket but don't
actually end up advertizing it (you *must* advertize a normal addr as
well).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell c075d78431 connectd: use listen_fd array directly, rather than returning binding arr.
We always added to both arrays, might as well just keep one.  

We make mayfail an explicit flag, rather than relying on the presence
of errstr, which is never NULL now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell a62f5e5d82 connectd: hoist find_local_address so we can give more graceful Tor erros.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell 200a8a985b connectd: add is_websocket and wireaddr to struct listen_fd.
This lets us give a better error message if listen fails, and also
moved the callback closer to where it's needed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
Rusty Russell f1ed373c97 connectd: be more graceful when we an address is in use.
Aditya had this issue due to a config line, and the result was hard
to diagnose even for me.

It's now:

```
$ ./lightningd/lightningd --network=regtest --addr=:18444
2022-02-26T05:01:28.705Z **BROKEN** connectd: Failed to bind socket for 0.0.0.0:18444: Address already in use
```

Whereas before it doesn't even give the address it's trying to bind:

```
rusty@rusty-XPS-13-9370:~/devel/cvs/lightning (master)$ ./lightningd/lightningd --network=regtest --addr=:18444
lightning_connectd: Failed to bind on 2 socket: Address already in use (version v0.10.2-331-g86b83e4)
0x558a8b8d9a12 send_backtrace
	common/daemon.c:33
0x558a8b8e91e1 status_failed
	common/status.c:221
0x558a8b8c8e4f make_listen_fd
	connectd/connectd.c:1090
0x558a8b8c8f55 handle_wireaddr_listen
	connectd/connectd.c:1129
0x558a8b8c993d setup_listeners
	connectd/connectd.c:1312
0x558a8b8ca344 connect_init
	connectd/connectd.c:1517
0x558a8b8cbb57 recv_req
	connectd/connectd.c:1896
0x558a8b8d9f9f handle_read
	common/daemon_conn.c:31
0x558a8b9247c1 next_plan
	ccan/ccan/io/io.c:59
0x558a8b9253c9 do_plan
	ccan/ccan/io/io.c:407
0x558a8b92540b io_ready
	ccan/ccan/io/io.c:417
0x558a8b9276fe io_loop
	ccan/ccan/io/poll.c:453
0x558a8b8cbf36 main
	connectd/connectd.c:2033
0x7fe4d02940b2 ???
	???:0
0x558a8b8c285d ???
	???:0
0xffffffffffffffff ???
	???:0
2022-02-26T05:02:27.547Z **BROKEN** connectd: Failed to bind on 2 socket: Address already in use (version v0.10.2-331-g86b83e4)
2022-02-26T05:02:27.547Z **BROKEN** connectd: backtrace: common/daemon.c:38 (send_backtrace) 0x558a8b8d9a68
2022-02-26T05:02:27.547Z **BROKEN** connectd: backtrace: common/status.c:221 (status_failed) 0x558a8b8e91e1
2022-02-26T05:02:27.547Z **BROKEN** connectd: backtrace: connectd/connectd.c:1090 (make_listen_fd) 0x558a8b8c8e4f
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1129 (handle_wireaddr_listen) 0x558a8b8c8f55
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1312 (setup_listeners) 0x558a8b8c993d
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1517 (connect_init) 0x558a8b8ca344
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:1896 (recv_req) 0x558a8b8cbb57
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: common/daemon_conn.c:31 (handle_read) 0x558a8b8d9f9f
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x558a8b9247c1
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x558a8b9253c9
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x558a8b92540b
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x558a8b9276fe
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: connectd/connectd.c:2033 (main) 0x558a8b8cbf36
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fe4d02940b2
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x558a8b8c285d
2022-02-26T05:02:27.548Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2022-02-26T05:02:27.548Z **BROKEN** connectd: STATUS_FAIL_INTERNAL_ERROR: Failed to bind on 2 socket: Address already in use
lightningd: connectd failed (exit status 242), exiting.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-05 15:48:03 +10:30
niftynei 9075c74e3d balance_snapshot: don't count unconfirmed utxos
This was causing journal_entries to show up in the accountant plugin,
since we don't emit events for unconfirmed events until they're actually
confirmed onchain.
2022-03-05 15:12:12 +10:30
niftynei b33868f7c2 coin_mvt: only chain moves have a blockheight
channel moves don't have blockheights
2022-03-05 15:12:12 +10:30
niftynei 7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei 590f12145b listpeers: include the `pushed_msat` amount
Useful for accounting for missed/historical channel opens, to figure out
what the actual sat contribution from each peer is at a utxo level

Changelog-Added: JSONRPC: `listpeers` now includes a `pushed_msat` value. For leased channels, is the total lease_fee.
2022-03-05 15:12:12 +10:30
niftynei ecb19ba6f2 coin_mvt: report mutual close outputs also
It's better to report every single utxo on close so we know when
to mark a channel account as definitively closed.
2022-03-05 15:12:12 +10:30
niftynei d246e9c1a2 tx_parts: as a convenience, copy over the value as the 'satoshi'
Reporting coin movements was crashing for liquid-regtest tests because
we were using an un-initialized field (the tx_part output's satoshi
field).

We fill this in 'as a convenience' for other wally_tx_outputs that are
on liquid elsewhere, here we do the same for tx_parts sent over the
wire to onchaind.
2022-03-05 15:12:12 +10:30
niftynei 7a00277b43 coin_mvt: mark coins destined for wallet w/ originating acct
Only shows up on delayed to us outputs, but nice to have anyway.

It's missing for channel index destined deposits, maybe nice to add at
some point in the future; right now you can figure out which close a
wallet deposit comes from via the channel close txid
2022-03-05 15:12:12 +10:30
niftynei 24a1c91045 coin_mvt: report the number of outputs on a channel close tx
The bookkeeper needs to know how many outputs to expect before we can
consider a channel resolved onchain.
2022-03-05 15:12:12 +10:30
niftynei 12cbf614cd coin_mvt: mark every event that already has a destination
Every event that's coming out of here that specifies a different place
to be deposited should be marked as originating from this channel's
 account.
2022-03-05 15:12:12 +10:30
niftynei b30328a91f coin_mvt: rm unncessary if statement
We don't need to switch on this; if it's zero the below ops will be
effectiely no-ops.
2022-03-05 15:12:12 +10:30
niftynei ea36c3a938 db: collapse the db_queries files into the generated ones
There's no reason to have these be independent of the generated files,
all the data contained within them is 'formulaic'
2022-03-05 15:03:34 +10:30
niftynei 453ef7f0ce db: mark that column is unused/ignored
2022-01-25T23:41:45.2994844Z ----------------------------- Captured stderr call -----------------------------
2022-01-25T23:41:45.2995230Z lightningd: Never accessed column 1 in query SELECT 1  FROM offers WHERE offer_id = ?;
2022-03-05 15:03:34 +10:30
niftynei ce12d2b8a9 database: pull out database code into a new module
We're going to reuse the database controllers for the accounting plugin
2022-03-05 15:03:34 +10:30
niftynei 03c950bae8 db: decouple `fatal` reliance, have as impl defined function
`fatal` is defined in lightningd and has logfile dependencies etc.

Make it more generic by allowing declaration in the use file (wallet.c)
2022-03-05 15:03:34 +10:30
niftynei b0829fc52a lightningd/Make: cleanup lightningd+wallet headers
There is no "wallet_lib_headers" variable in wallet/Makefile

Likewise, there were two "lightningd_headers", a couple of unused
variables and some other nonsene in lightningd/Makefile
2022-03-05 15:03:34 +10:30
Rusty Russell cd37753849 plugins/bcli: neaten to fix spurious leak report.
```
plugin-bcli: MEMLEAK: 0x1d2a118
plugin-bcli:   label=plugins/bcli.c:636:const char *[]
plugin-bcli:   backtrace:
plugin-bcli:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
plugin-bcli:     ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)
plugin-bcli:     plugins/bcli.c:636 (estimatefees_next)
plugin-bcli:     plugins/bcli.c:684 (estimatefees)
plugin-bcli:     plugins/libplugin.c:1307 (ld_command_handle)
plugin-bcli:     plugins/libplugin.c:1348 (ld_read_json_one)
plugin-bcli:     plugins/libplugin.c:1368 (ld_read_json)
plugin-bcli:     ccan/ccan/io/io.c:59 (next_plan)
plugin-bcli:     ccan/ccan/io/io.c:407 (do_plan)
plugin-bcli:     ccan/ccan/io/io.c:417 (io_ready)
plugin-bcli:     ccan/ccan/io/poll.c:453 (io_loop)
plugin-bcli:     plugins/libplugin.c:1565 (plugin_main)
plugin-bcli:     plugins/bcli.c:965 (main)
plugin-bcli:   parents:
plugin-bcli:     plugins/libplugin.c:1235:struct command
```

It's neater to convert start_bitcoin_cli() to take varargs, and handle take:
the above "leak" is because we don't keep a pointer to the params[] array
which we use to pass to start_bitcoin_cli().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-03 10:23:11 +00:00
Dustin Dettmer 4386ccd842 bitcoin: Comment typo fix 2022-03-03 11:22:19 +10:30
niftynei 7bfbe1492a test_close: sync l1 to the blockchain, so that the events are there
Flake on CI coming from `penalty_in/outhtlc` due to the fact that all
the events haven't arrived for the check.

If we wait to sync `l1` as well as `l2`, this should resolve the flake.

        # We use a subset of tags in expected_2 that are used in expected_1
>       tags = check_utxos_channel(l1, [channel_id], expected_1)

tests/test_closing.py:726:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/utils.py:321: in check_utxos_channel
    txid = matchup_events(u_set, evs, chans, tag_list)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

u_set = [[{'account_id': 'external', 'blockheight': 104, 'coin_type': 'bcrt', 'credit': '100000000msat', ...}, None]]
evs = [('external', ['penalty'], None, None), ('external', ['penalty'], None, None), ('external', ['penalty'], None, None)]
chans = ['2722a5fe49a8b5fa4004c19828f3e903632ac02712c6fe78ebea418daad2691f']
tag_list = {'0': '892c64c7d8c8f15d7bdcdcde34b615817d273d2e33d9d775cc9ff38e8e3deeb2', 'A': '1e69d2aa8d41eaeb78fec61227c02a6303e9f32898c10440fab5a849fea52227', 'B': '0e11e2ca01bf5f30b4c54522af172af055dcff8e3810f80a5069a2394cad74b5'}

    def matchup_events(u_set, evs, chans, tag_list):
>       assert len(u_set) == len(evs) and len(u_set) > 0
E       AssertionError
2022-03-01 16:19:34 -06:00
Christian Decker 6332578070 rust: Use $CARGO_OPTS when building the example plugin 2022-02-28 23:25:37 +00:00
Christian Decker 75f0b8e916 cln-grpc: Add `listpeers` RPC method
This is pretty much the hardest to map, but we map it correctly, with
the exception of the state_changes[] array we truncated out in the
last commit.
2022-02-28 23:25:37 +00:00
Christian Decker b73405a4c2 cln-grpc: Allow fields to be mapped to None to ignore them
`listpeers` is a rather deeply nested structure which has a couple of
caveats, namely that we use the same enum multiple times, which causes
naming clashes. So we truncate the state_changes[]. We can later map
them if needed, but it'll get much easier once we have an abstract
model description that isn't JSON schema, which unrolls all types,
causing us to generate those enums multiple times.
2022-02-28 23:25:37 +00:00
Christian Decker fd2d126ec4 cln-grpc: Add conversion from pb Amount to json Amount 2022-02-28 23:25:37 +00:00
Christian Decker d95c9d2990 cln-grpc: Add conversion from pb Amount to json Amount 2022-02-28 23:25:37 +00:00
Christian Decker bb4946a6e6 msggen: Support renaming methods in GRPC
There is at least one clash with a built-in for the grpc server trait,
namely `connect` so we add support for renaming a method when
generating the scaffolding
2022-02-28 23:25:37 +00:00
Christian Decker efed7d8617 msggen: Support enums in requests too
They are sent as i32 over protobuf, so we need to convert them into
their enum representation.
2022-02-28 23:25:37 +00:00
Christian Decker 494243d41c msggen: Handle some more types in request conversions 2022-02-28 23:25:37 +00:00
Christian Decker 24e44ecbb6 cln-grpc: Add glue to get all pieces to work together 2022-02-28 23:25:37 +00:00
Christian Decker 62dc078271 cln-grpc: Generate server dispatcher
The server doesn't do much more than unwrapping the request from its
grpc envelope, convert it into the matching JSON-RPC binding struct,
initiate the RPC connection (until we have connection pooling), and
then forwards the converted request. The inverse then happens for the
result.
2022-02-28 23:25:37 +00:00
Christian Decker 5d6e9d6dae cln-grpc: Add generation of request conversion
This is taking protobuf requests on one side and converting them into
the JSON-RPC requests.
2022-02-28 23:25:37 +00:00
Christian Decker 8d3871d791 cln-grpc: Add result conversion generator to `msggen`
This takes the Rust bindings and converts them into the generated
protobuf bindings:

> JSON-RPC -> Rust bindings -> grpc bindings -> protobuf
2022-02-28 23:25:37 +00:00
Christian Decker 4c105d2424 cln-grpc: Generate grpc bindings from proto
This is the easy way, since there's already tooling for this.
2022-02-28 23:25:37 +00:00
Christian Decker d01b2c21a7 cln-grpc: Add generation of grpc protobuf file from schema 2022-02-28 23:25:37 +00:00
Christian Decker edf6832f8f rust: Actually check if we have cargo
Seems a temporary edit to check the detection slipped in during the
previous PR. This fixes that :-)
2022-02-28 23:25:37 +00:00
Simon Vrouwe 175e0b54b3 testing: fix flake in test_ping_timeout
[ Folded 2 fixups --RR ]
2022-03-01 09:38:40 +10:30
Simon Vrouwe c286eb053f pyln-client: Plugin, improve background comment 2022-03-01 09:38:40 +10:30
Simon Vrouwe ad22535d34 pyln-testing: improve description of wait_for_logs 2022-03-01 09:38:40 +10:30