This contains the zeroconf stuff, with funding_locked renamed to
channel_ready. I change that everywhere, and try to fix up the
comments.
Also the `alias` field is called `short_channel_id`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `funding_locked` is now called `channel_ready` as per latest BOLTs.
We have them split over common/param.c, common/json.c,
common/json_helpers.c, common/json_tok.c and common/json_stream.c.
Change that to:
* common/json_parse (all the json_to_xxx routines)
* common/json_parse_simple (simplest the json parsing routines, for cli too)
* common/json_stream (all the json_add_xxx routines)
* common/json_param (all the param and param_xxx routines)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will only add the discovered `remote_addr` IPs if no other
addresses would be announced. Meaning whenever a public address was
found by autobind or an address was specified via commandline or config,
IP discovery will be disabled.
Addresses: #5305
Note from the author: We could/should also enable IP discovery when we only
have a TOR address (but without --always-use-proxy ofc). This will give
nodes an option to have a bootstrap way to be reached until IP discovery
can do the job in a more stable way.
Changelog-Changed: Only use IP discovery as fallback when no addresses would be announced
This commit got reduced to just changing a comment because
the stuff it initially did was already merged in before by
commit 7ff62b4a
So I just kept the changed comment as its more precise.
Changelog-None
routing.c fixed to properly remove rate-limited gossip_store entries
when channels are closed. This caused gossipd to crash on a subsequent
gossip_store_load. Also corrects an overzealous limit of one gossip_store
entry per message (should now allow one broadcastable and one
rate-limited). Addresses issues 5387, 5395.
Changelog-None
This grows the routing state in order to index both okay-to-broadcast
and rate-limited gossip. The gossip_store also logs the rate-limited
gossip if useful. This allows the broadcast of the last non-rate-limited
gossip.
routing.c now flags rate-limited gossip as it enters the gossip_store but
makes use of it in updating the routing graph. Flagged gossip is not
rebroadcast to gossip peers.
Changelog-Changed: gossipd: now accepts spam gossip, but squelches it for
peers.
This will be used to decouple internal use of gossip from what is
passed to gossip peers. Updates GOSSIP_STORE_VERION to 10.
Changelog-Changed: gossip_store updated to version 10.
@whitslack complained of large CPU usage by connectd at startup;
I ran perf record on connectd on my machine (which sees a little spike, only)
and I see the cost of reading and discarding the entries:
```
- 95.52% 5.24% lightning_conne lightning_connectd [.] gossip_store_next
- 90.28% gossip_store_next
+ 40.27% tal_alloc_arr_
+ 22.78% tal_free
+ 11.74% crc32c
+ 9.32% fromwire_peektype
+ 4.10% __libc_pread64 (inlined)
1.70% be32_to_cpu
```
Much of this is caused by the search for our own gossip: keeping this separately
would be even better, but this fix is minimal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: connectd: reduce initial CPU load when connecting to peers.
We had json_add_amount_msat_only(), which was designed to be used to
print out msat fields, if we had sats.
However, we misused it, so split it into the three different cases:
1. json_add_amount_sat_msat: We are using it correctly, with a field called
xxx_msat.
2. json_add_amount_sats_deprecated: We were using it wrong, so deprecate
the old field and create a new one which does end in _msat.
3. json_add_sats: we were using it to hand sats as a JSON parameter to an
interface, where "XXXsat".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `rbf_channel` and `openchannel2` hooks `their_funding` (use `their_funding_msat`)
Changelog-Deprecated: Plugins: `openchannel2` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `funding_satoshis` (use `funding_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `channel_reserve_satoshis` (use `channel_reserve_msat`)
Changelog-Deprecated: Plugins: `channel_opened` notification `amount` (use `funding_msat`)
Changelog-Deprecated: JSON-RPC: `listtransactions` `msat` (use `amount_msat`)
Changelog-Deprecated: Plugins: `htlc_accepted` `forward_amount` (use `forward_msat`)
This was eliminated this morning in the latest spec. We still accept them,
we just don't produce them any more.
Changelog-Removed: Protocol: We no longer create gossip messages which use zlib encoding (we still understand them, for now!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have an explicit filter against redundant node_announcement
updates; we only allow 1 a week. This means that our change to force
a reannouncement every 24 hours did not work!
Allow once a day, instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This attempted to make us re-xmit our own node_announcement at restart,
by moving the node_announcement to the end of the gossip store. But,
as nothing is connected, yet, this had no effect!
We will rexmit it anyway, since it's marked PUSH.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We do this (send warnings) in almost all cases anyway, so mainly this
is a textual update, but there are some changes:
1. Send ERROR not WARNING if they send a malformed commitment secret.
2. Send WARNING not ERROR if they get the shutdown_scriptpubkey wrong (vs upfront)
3. Send WARNING not ERROR if they send a bad shutdown_scriptpubkey (e.g. p2pkh in future)
4. Rename some vars 'err' to 'warn' to make it clear we send a warning.
This means test_option_upfront_shutdown_script can be made reliable, too,
and it now warns and doesn't automatically close channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Gossipd didn't actually suppress all gossip, resulting in a flake!
Doing it in connectd now makes much more sense.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Requiring the caller to allocate them is ugly, and differs from
other types.
This means we need a context arg if we don't have one already.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Simply exit, like we do when master daemon_conn exits.
```
Valgrind error file: valgrind-errors.2211908
==2211908== Invalid read of size 8
==2211908== at 0x12AC13: daemon_conn_send (daemon_conn.c:137)
==2211908== by 0x113CD9: queue_peer_msg (gossipd.c:118)
==2211908== by 0x11B806: query_channel_range (queries.c:1169)
==2211908== by 0x1250DD: peer_gossip_probe_scids (seeker.c:706)
==2211908== by 0x1253B1: check_firstpeer (seeker.c:788)
==2211908== by 0x1256CA: seeker_check (seeker.c:884)
==2211908== by 0x1366AC: timer_expired (timeout.c:62)
==2211908== by 0x1163D1: main (gossipd.c:1146)
==2211908== Address 0x4cafdf0 is 48 bytes inside a block of size 88 free'd
==2211908== at 0x48460C4: free (vg_replace_malloc.c:872)
==2211908== by 0x1805EA: del_tree (tal.c:421)
==2211908== by 0x1808BE: tal_free (tal.c:486)
==2211908== by 0x12AB25: destroy_dc_from_conn (daemon_conn.c:112)
==2211908== by 0x17FFDF: notify (tal.c:237)
==2211908== by 0x180519: del_tree (tal.c:402)
==2211908== by 0x1808BE: tal_free (tal.c:486)
==2211908== by 0x16EE9A: io_close (io.c:450)
==2211908== by 0x16ECA9: do_plan (io.c:401)
==2211908== by 0x16ED16: io_ready (io.c:417)
==2211908== by 0x1710B2: io_loop (poll.c:453)
==2211908== by 0x1163C5: main (gossipd.c:1144)
==2211908== Block was alloc'd at
==2211908== at 0x484384F: malloc (vg_replace_malloc.c:381)
==2211908== by 0x180064: allocate (tal.c:250)
==2211908== by 0x180634: tal_alloc_ (tal.c:428)
==2211908== by 0x12AB65: daemon_conn_new_ (daemon_conn.c:122)
==2211908== by 0x1155F4: gossip_init (gossipd.c:763)
==2211908== by 0x116014: recv_req (gossipd.c:999)
==2211908== by 0x12A828: handle_read (daemon_conn.c:31)
==2211908== by 0x16E09F: next_plan (io.c:59)
==2211908== by 0x16ECD4: do_plan (io.c:407)
==2211908== by 0x16ED16: io_ready (io.c:417)
==2211908== by 0x1710B2: io_loop (poll.c:453)
==2211908== by 0x1163C5: main (gossipd.c:1144)
==2211908==
```
This is the cheapest algo I came up with that simply checks that the
same `remote_addr` has been report by two different peers. Can be
improved in many ways:
- Check by connecting to a radonm peers in the network
- Check for more than two confirmations or a certain fraction
- ...
Changelog-Added: Send updated node_annoucement when two peers report the same remote_addr.
This limit applies to both node_announcements (which we now send 1 per
day), and channel_updates; I've had reports of LND nodes going down
daily for database compation, so they end up ratelimited.
Changelog-Protocol: We now allow two channel_updates or node_announcements per day, up from 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Even if nothing has changed. Note that this is different from simply
re-xmitting the old one, in that it has a different timestamp, so others
will re-xmit it too.
This is a side-effect of reports that node_announcement propagation
through the network is poor:
https://github.com/ElementsProject/lightning/issues/5037
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: We now refresh our ndoe_announcement every 24 hours, due to propagation issues.
node_announcement has to follow at least one channel_announcement.
When channels close, if this isn't the case, we remove the old
node_announcement and put it at the end of the gossip_store.
But we lose the "send even if they don't want it" bit in the case it's
our own node_announceent, so keep it.
This only happens if you don't change your node configuration at all
since you opened your first channel, but still worth fixing.
We expose the force_node_announce_rexmit() for later use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This restores the behaviour prior to `lightningd: use our cached
channel_update for errors instead of asking gossipd.`, where gossipd
would refuse to give us channel_updates for unannounced channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`hc` is never NULL, since it's `hc = &chan->half[direction];`;
we really meant "is it initialized", and valgrind under CI finally
caught it:
```
==69243== Conditional jump or move depends on uninitialised value(s)
==69243== at 0x11C595: handle_local_channel_update (gossip_generation.c:758)
==69243== by 0x115254: recv_req (gossipd.c:986)
==69243== by 0x128F8D: handle_read (daemon_conn.c:31)
==69243== by 0x16BEE1: next_plan (io.c:59)
==69243== by 0x16CAE9: do_plan (io.c:407)
==69243== by 0x16CB2B: io_ready (io.c:417)
==69243== by 0x16EE1E: io_loop (poll.c:453)
==69243== by 0x1154DA: main (gossipd.c:1089)
==69243==
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Gossipd now simply gets told by channeld when peers arrive or leave.
(it only needs to know for the seeker).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We also no longer strip the type off: everyone handles both forms, and
Eclair doesn't strip (and it's easier!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Even if we're deferring putting them in the store and broadcasting them,
we tell lightningd so it will use it in any error messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want it to keep the latest, so it can make its own error msgs without
asking us. This installs (but does not use!) the message handler.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
An "err" is only returned if the channel_update is malformed: more common
is that it's fine, but we don't know the scid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's weird to have connectd ask gossipd, when lightningd can just do it
and hand all the addresses together.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now let gossipd do it.
This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
local_chan was mainly around so we could "soft" disable channels (and
really disable them once we used the channel_update in an error
message).
Instead we introduce the idea of a "deferred_update": it's either
deferred indefinitely (a peer goes offline, if we need to send it in
an error we'll apply it immediatly), or simply delayed to avoid
spamming everyone.
The resulting rewrite is much clearer, IMHO.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Increasingly we want to know is it local, and get the direction: it's
more efficient to do both at once.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).
config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's very similar to the previous, but there are a few changes:
1. The enctlv fields are numbered differently.
2. The message itself is a different number.
The onionmsg_path type is the same, however, so we keep that constant
at least.
The result is a lot of cut & paste, but we will delete the old one
next release.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is from 6e99c5feaf60cb797507d181fe583224309318e9
We renamed the enctlv field to encrypted_recipient_data in the spec, and the
new onion_message is message 513. We don't handle it until the next patch.
Two renames:
1. blinding_seed -> blinding_point.
2. enctlv -> encrypted_recipient_data.
We don't do a compat cycle for our JSON APIs for these experimental
features only used by our own plugins, we just rename.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Temporarily disable sendpay_blinding test which uses obsolete onionmsg;
there's still some debate on the PR about how blinded HTLCs will work.
Changelog-EXPERIMENTAL: onionmessage: removed support for v0.10.1 onion messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can have an update pending because it's too fast, but
refresh_local_channel is supposed to make sure we're up-to-date, so
force immediate application in that case.
Otherwise, we call update_local_channel at the bottom which frees the
pending update. This can mean that we miss a change in fees, for example.
Changelog-Fixed: errors: Errors returning a `channel_update` no longer return an outdated one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To minimize the diffs, we #if 0 the code. We'll reenable it once
channeld is ready.
We also temporarily disable the ping tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a new hook: onion_message_ourpath for when we know a message
came in via a blinded path we created. The onion_message_blinded hook
is now called for all other messages, since all messages are now
blinded.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By popular merge-hell demand.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Build: Python is now required to build, as generated files are no longer checked into the repository.
One change from the obsolete version handling, gossipd will no longer send
forwarding onion msgs to lightningd, but will forward it directly.
That was the effect before, anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We never tested that we can correctly unwrap on the next step after
unblinding: it failed because we mangled the onion in place! Fix that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before:
Ten builds, laptop -j5, no ccache:
```
real 0m36.686000-38.956000(38.608+/-0.65)s
user 2m32.864000-42.253000(40.7545+/-2.7)s
sys 0m16.618000-18.316000(17.8531+/-0.48)s
```
Ten builds, laptop -j5, ccache (warm):
```
real 0m8.212000-8.577000(8.39989+/-0.13)s
user 0m12.731000-13.212000(12.9751+/-0.17)s
sys 0m3.697000-3.902000(3.83722+/-0.064)s
```
After:
Ten builds, laptop -j5, no ccache: 8% faster
```
real 0m33.802000-35.773000(35.468+/-0.54)s
user 2m19.073000-27.754000(26.2542+/-2.3)s
sys 0m15.784000-17.173000(16.7165+/-0.37)s
```
Ten builds, laptop -j5, ccache (warm): 1% faster
```
real 0m8.200000-8.485000(8.30138+/-0.097)s
user 0m12.485000-13.100000(12.7344+/-0.19)s
sys 0m3.702000-3.889000(3.78787+/-0.056)s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is best-practice (to ensure prototypes match up), but there were a
few places we didn't (at least, directly). Make it a requirement,
either of form "foo.h" or <dir/foo.h>.
The noise is the change to our print templates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We make it a first-class citizen internally, even though we won't use
it over the wire (at least, non-experimental builds). This scheme
follows the latest draft, in which features are flagged compulsory.
We also add several helper functions.
Since uses the *even* bits (as per latest spec), not the *odd* bits,
we have some other fixups.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to use this to handle the simple description for channel_type.
It also needs to handle variable-size types (just like subtypes).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If there's a rate-card for liquidity, we don't know about it until
after startup (the plugin *should* call us at init to tell us what their
current rates are)
Rusty Russell
Alex Myers
Michael Schmoock
Vincenzo Palazzo
Ken Sedgwick
niftynei
Simon Vrouwe