Commit Graph

205 Commits

Author SHA1 Message Date
Rusty Russell 270368e50b gossipd: use gossip_query_ex to query only nodes when node probing.
If the peer supports `gossip_query_ex` we can use query_flags to simply
request the node_announcements when probing for nodes, rather than
getting everything.  If a peer doesn't support `gossip_query_ex` then
it's harmless to add it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell d33ebd3629 seeker: probe for node announcements.
We pick some nodes which don't seem to have node_announcements and we
ask a channel associated with them.  Again, if this reveals more
node_announcements, we probe for twice as many next time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell ba3c79e560 seeker: seek unknown scids.
If we have any unknown short_channel_ids, we ask a random peer for
those channels.  Once it responds, we probe again for a small random
range in case more are missing, again enlarging if we find some.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell 4b13c92802 seeker: use hash table for unknown short_channel_ids.
Instead of a linear array which is fairly inefficient if it turns out
we know nothing at all.

We remove the gossip_missing() call by changing the api to
remove_unknown_scid() to include a flag as to whether the scid turned
out to be real or not.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell 83575f27a1 seeker: add code to check range of scids.
Once we've finished streaming gossip from the first peer, we ask a
random peer (maybe the same one) for all short_channel_ids in the last
6 blocks from the latest channel we know about.

If this reveals new channels we didn't know about, we expand the probe
by a factor of 2 each time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell 521c7f7121 seeker: take over gossip control.
The seeker starts by asking a peer (the first peer!) for all gossip
since a minute before the modified time of the gossip store.

This algorithm is enhanced in successive patches.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell 79ca9bf998 gossipd: use per-peer information to make messages clearer.
We can (usually) indicate what peer caused the bad gossip error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell 0091300ee3 gossipd: track what peer gave us gossip msgs so we can credit it.
Since we have to validate, there can be a delay (and peer might
vanish) between receiving the gossip and actually confirming it, hence
the use of softref.

We will use this information to check that the peers are making progress
as we start asking them for specific information.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-10 21:48:52 -05:00
Rusty Russell aab9e9f010 gossipd: remove internal dev helpers for queries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-30 07:08:07 +00:00
Rusty Russell 4bf0bc1f28 gossipd: age txout_failures map.
We do this by keeping a current and an old map, and moving the current to old
every hour or 10,000 entries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-27 02:32:53 +00:00
Rusty Russell 722b4942ed common: rename decode_short_channel_ids.{c,h} to decode_array.{c.h}
This encoding scheme is no longer just used for short_channel_ids, so make
the names more generic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-27 02:32:53 +00:00
Rusty Russell d24c850899 gossipd: restore a flag for fast pruning
I was seeing some accidental pruning under load / Travis, and in
particular we stopped accepting channel_updates because they were 103
seconds old.  But making it too long makes the prune test untenable,
so restore a separate flag that this test can use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-27 00:01:34 +00:00
Rusty Russell 2b47922ea5 gossipd: move query functions into their own file.
The only real change is dump_gossip() used to call
maybe_create_next_scid_reply(), but now I've simply renamed
that to maybe_send_query_responses() and we call it directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-25 04:01:56 +00:00
Rusty Russell fd2d74aa9b gossipd: defer asking about txouts until we're synced or they're 6 deep.
The first one means we don't discard channels just because we're not
synced, and the second is implied by the spec: don't accept
channel_announcement if the channel isn't 6 deep.  Since LND defers in
such cases, we do too (unless it's newer than the current block, in
which case we simply discard).  Otherwise there's a risk that a slow
node might discard valid gossip.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-25 04:01:56 +00:00
Rusty Russell 6f9c5f2936 gossipd: get fed the blockheight from lightningd when we know it.
This will let gossipd be more intelligent about gossiping before we're
synced, and also it might know how far behind we are.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-25 04:01:56 +00:00
trueptolemy d8dce6e61f cleanup: Use `u32` as the type of `max_hops` in `gossipd` 2019-09-24 16:01:24 +02:00
Rusty Russell 27790832a5 gossipd: gossip_queries_ex is not longer experimental.
The master spec has some typos which make it not parse, so I created
a PR and generated the CSV from that:

https://github.com/lightningnetwork/lightning-rfc/pull/673

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-22 01:17:11 +00:00
Rusty Russell 6a8d18c7e3 gossipd: naming cleanups.
Suggested-by: @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell 147eaced2e developer: consolidiate gossip timing options into one --dev-fast-gossip.
It's generally clearer to have simple hardcoded numbers with an
#if DEVELOPER around it, than apparent variables which aren't, really.

Interestingly, our pruning test was always kinda broken: we have to pass
two cycles, since l2 will refresh the channel once to avoid pruning.

Do the more obvious thing, and cut the network in half and check that
l1 and l3 time out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell a92ead48bf gossipd: ignore redundant channel_update and node_announcement.
If you send a message which simply changes timestamp and signature, we
drop it.  You shouldn't be doing that, and the door to ignoring them
was opened by by option_gossip_query_ex, which would allow clients to
ignore updates with the same checksum.

This is more aggressive at reducing spam messages, but we allow refreshes
(to be conservative, we allow them even when 1/2 of the way through the
refresh period).

I dropped the now-unnecessary sleep from test_gossip_pruning, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell 21a6d502db gossipd: move gossip message generation into its own file.
gossipd.c is doing too many things: this is a start.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell 178baeba6c gossipd: get gossip_min_interval from lightningd.
Default is 5 x gossip interval == 5 minutes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
trueptolemy 5361a5d059 JSON-API: `getroute` now also support `exclude` nodes 2019-09-16 12:22:06 +08:00
Rusty Russell 91072f56b0 developer: add 'dev-gossip-set-time' call to manipulate gossipd's time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-12 05:11:56 +00:00
Rusty Russell 2577ad87d5 gossipd: use gossip_time_now() everywhere.
We've been slack, but it's going to be important for testing
ratelimiting.  And it currently has a minor memory leak.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-12 05:11:56 +00:00
lisa neigut fe6c7f8f80 gossip queries: patch up valgrind errors in tests
These were giving me valgrind errors locally; fixed now.
2019-09-11 23:56:27 +00:00
darosior 0b0ad4c22d transition from status_trace() to status_debug 2019-09-10 02:02:51 +00:00
Rusty Russell aca2e4f722 common/memleak: add dynamic hooks for assisting memleak.
Rather than reaching into data structures, let them register their own
callbacks.  This avoids us having to expose "memleak_remove_xxx"
functions, and call them manually.

Under the hood, this is done by having a specially-named tal child of
the thing we want to assist, containing the callback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:35:01 +02:00
Rusty Russell 837e6232c3 common: reduce header differences for DEVELOPER vs non-DEVELOPER.
`make update-mocks` is usually run in DEVELOPER mode, but then it includes
definitions for functions which aren't declared in non-DEVELOPER mode.

We hacked this in a few places, but it's fragile, and worst, now we
have EXPERIMENTAL_FEATURES as well, it's complex.

Instead, declare developer-only functions (but don't define them).
This is a bit more awkward if you accidentally use one in
non-DEVELOPER code (link error rather than compile error), but makes
autogenerating test mocks much easier.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:35:01 +02:00
Rusty Russell acf3952acc JSON: remove handling of pre-Adelaide (B:T:N) short_channel_ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:19:14 +02:00
Rusty Russell 02b9b7f6e6 tests: update mocks for --enable-experimental-features builds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Rusty Russell 855dff704c gossipd: test crc32 routines using test vectors from PR.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Rusty Russell 51541f53d8 gossipd: test vectors for https://github.com/lightningnetwork/lightning-rfc/pull/557
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
lisa neigut 7046d0220c makefiles: move all unit tests under `make check-units`
Isolate unit tests under their own make directive.
2019-06-30 16:41:30 +09:30
Rusty Russell f1b57063f7 bitcoin/tx: use fromwire_fail in pull_bitcoin_tx.
This is the correct way to mark failure: it also sets *max to 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-21 03:56:59 +00:00
Rusty Russell 1a3886c116 wallet: keep a list of unreleased transactions.
We're going to use this in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-06 04:47:44 +00:00
Rusty Russell 3e733afb2b gossipd: remove broadcast map altogether.
This clarifies things a fair bit: we simply add and remove from the
gossip_store directly.

Before this series: (--disable-developer, -Og)
    store_load_msec:20669-20902(20822.2+/-82)
    vsz_kb:439704-439712(439706+/-3.2)
    listnodes_sec:0.890000-1.000000(0.92+/-0.04)
    listchannels_sec:11.960000-13.380000(12.576+/-0.49)
    routing_sec:3.070000-5.970000(4.814+/-1.2)
    peer_write_all_sec:28.490000-30.580000(29.532+/-0.78)

After: (--disable-developer, -Og)
    store_load_msec:19722-20124(19921.6+/-1.4e+02)
    vsz_kb:288320
    listnodes_sec:0.860000-0.980000(0.912+/-0.056)
    listchannels_sec:10.790000-12.260000(11.65+/-0.5)
    routing_sec:2.540000-4.950000(4.262+/-0.88)
    peer_write_all_sec:17.570000-19.500000(18.048+/-0.73)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-04 01:29:39 +00:00
Rusty Russell 5591c0b5d8 gossipd: don't send gossip stream, let per-peer daemons read it themselves.
Keeping the uintmap ordering all the broadcastable messages is expensive:
130MB for the million-channels project.  But now we delete obsolete entries
from the store, we can have the per-peer daemons simply read that sequentially
and stream the gossip itself.

This is the most primitive version, where all gossip is streamed;
successive patches will bring back proper handling of timestamp filtering
and initial_routing_sync.

We add a gossip_state field to track what's happening with our gossip
streaming: it's initialized in gossipd, and currently always set, but
once we handle timestamps the per-peer daemon may do it when the first
filter is sent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-04 01:29:39 +00:00
Rusty Russell df00f20e4a gossipd: erase old entries from the store, don't just append.
We use the high bit of the length field: this way we can still check
that the checksums are valid on deleted fields.

Once this is done, serially reading the gossip_store file will result
in a complete, ordered, minimal gossip broadcast.  Also, the horrible
corner case where we might try to delete things from the store during
load time is completely gone: we only load non-deleted things.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-04 01:29:39 +00:00
Rusty Russell 696dc6b597 gossipd: disable gossip_store upgrade.
We're about to bump version again, and the code to upgrade it was
quite hairy (and buggy!).  It's not worthwhile for such a
poorly-tested path: I will just add code to limit how much incoming
gossip we get to avoid flooding when we upgrade, however.

I also use a modern gossip_store version in our test_gossip_store_load
test, instead of relying on the upgrade path.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-04 01:29:39 +00:00
Rusty Russell 180a552fba gossip_store: mark private updates separately from normal ones.
They're really gossipd-internal, and we don't want per-peer daemons
to confuse them with normal updates.

I don't bump the gossip_store version; that's coming with another update
anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-04 01:29:39 +00:00
Rusty Russell 654e89b5fc gossipd: free channels in routing_state destructor.
Cleans up the tests.

Suggested-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell d1f43d993a gossipd: use explicit destructor for struct chan.
Each destructor2 costs 40 bytes, and struct chan is only 120 bytes.  So
this drops our memory usage quite a bit:

MCP bench results change:
   -vsz_kb:580004-580016(580006+/-4.8)
   +vsz_kb:533148

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell cb9c44ef27 gossipd: remove unnecessary dev_unknown_channel_satoshis arg.
We now have a test blockchain for MCP which has the correct channels,
so this is not needed.

Also fix a benchmark script bug where 'mv "$DIR"/log
"$DIR"/log.old.$$' would fail if you log didn't exist from a previous run.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell 0e37ac2433 common: move gossip_store read routine where subdaemons can access it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-13 05:16:18 +00:00
Rusty Russell 7ede5aac31 gossip_store: change format so we store raw messages.
Save some overhead, plus gets us ready for giving subdaemons direct
store access.  This is the first time we *upgrade* the gossip_store,
rather than just discarding.

The downside is that we need to add an extra message after each
channel_announcement, containing the channel capacity.

After:
  store_load_msec:28337-30288(28975+/-7.4e+02)
  vsz_kb:582304-582316(582306+/-4.8)
  store_rewrite_sec:11.240000-11.800000(11.55+/-0.21)
  listnodes_sec:1.800000-1.880000(1.84+/-0.028)
  listchannels_sec:22.690000-26.260000(23.878+/-1.3)
  routing_sec:2.280000-9.570000(6.842+/-2.8)
  peer_write_all_sec:48.160000-51.480000(49.608+/-1.1)

Differences:
  -vsz_kb:582320
  +vsz_kb:582316
  -listnodes_sec:2.100000-2.170000(2.118+/-0.026)
  +listnodes_sec:1.800000-1.880000(1.84+/-0.028)
  -peer_write_all_sec:51.600000-52.550000(52.188+/-0.34)
  +peer_write_all_sec:48.160000-51.480000(49.608+/-1.1)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-13 05:16:18 +00:00
Rusty Russell 0fc42415c2 gossipd/routing: remove BFG implementation.
Now we can benchmark, and remove 500 bytes per node.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:35093-37907(36146+/-1.1e+03)
	vsz_kb:555168
	store_rewrite_sec:12.120000-13.750000(12.7+/-0.6)
	listnodes_sec:1.270000-1.370000(1.322+/-0.039)
	listchannels_sec:29.770000-31.600000(30.82+/-0.64)
	routing_sec:0.00
	peer_write_all_sec:63.630000-67.850000(65.432+/-1.7)

MCP notable changes from pre-Dijkstra (>1 stddev):
	-vsz_kb:577456
	+vsz_kb:555168
	-routing_sec:60.70
	+routing_sec:12.04

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell e197956032 gossipd/routing: Iterate on Dijkstra when route is too long.
If a route is too long, we try to bias Dijkstra towards choosing a
shorter route by adding a per-hop cost.  We do a naive "shortest path"
pass, then using that cost as a ceiling on per-hop cost, we do a
binary search.

There are some subtleties: we use risk rather than total as our
counter field (we normally bias this by 1 anyway, so it's easy to make
that a variable), and we set riskfactor to a mimimal value once we're
iterating.  It's good enough to get a solution, we don't need to do a
2-dimensional search on riskfactor and riskbias.

Of course, this is extremely slow if we hit it on our benchmark,
though it doesn't happen in a more realistic network:

$ gossipd/test/run-bench-find_route 100000 100:

Before:
	100 (79 succeeded) routes in 100000 nodes in 25341 msec (253412314 nanoseconds per route)

After:
	100 (100 succeeded) routes in 100000 nodes in 97346 msec (973461784 nanoseconds per route)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell 7caa37f0f1 gossipd: implement Dijkstra.
Use a uintmap as our minheap.

Note that Dijkstra can give overlength routes, so some checks are disabled.

Comparison using gossipd/test/run-bench-find_route 100000 10:

Before:
	10 (10 succeeded) routes in 100000 nodes in 120087 msec (12008708402 nanoseconds per route)
After:
	10 (10 succeeded) routes in 100000 nodes in 2269 msec (226925462 nanoseconds per route)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell 6dfa46d65a gossipd/test: add test for handling overlong routes.
This is a weakness with Dijkstra, so write an explicit unit test that
we can find a short enough (but more expensive) route.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell 261921dee2 gossipd: adjust peers' broadcast_offset when compacting store.
When we compact the store, we need to adjust the broadast index for
peers so they know where they're up to.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Rusty Russell fdb42c3170 gossipd: don't keep channel_updates in memory.
This requires some trickiness when we want to re-add unannounced channels
to the store after compaction, so we extract a common "copy_message" to
transfer from old store to new.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:36034-37853(37109.8+/-5.9e+02)
	vsz_kb:577456
	store_rewrite_sec:12.490000-13.250000(12.862+/-0.27)
	listnodes_sec:1.250000-1.480000(1.364+/-0.09)
	listchannels_sec:30.820000-31.480000(31.068+/-0.24)
	routing_sec:26.940000-27.990000(27.616+/-0.39)
	peer_write_all_sec:65.690000-68.600000(66.698+/-0.99)

MCP notable changes from previous patch (>1 stddev):
	-vsz_kb:1202316
	+vsz_kb:577456

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Rusty Russell eb4564c3cd gossipd: embed broadcast information into each structure.
This is more compact, but also required once we replace the arbitrary
"index" with an actual offset into the gossip store.  That will let us
remove the in-memory variants entirely.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:35685-38538(37090.4+/-9.1e+02)
	vsz_kb:2288768
	store_rewrite_sec:35.530000-41.230000(37.904+/-2.3)
	listnodes_sec:0.720000-0.810000(0.762+/-0.041)
	listchannels_sec:30.750000-35.990000(32.704+/-2)
	routing_sec:29.570000-34.010000(31.374+/-1.8)
	peer_write_all_sec:51.140000-58.350000(55.69+/-2.4)

MCP notable changes from previous patch (>1 stddev):
	-vsz_kb:2621808
	+vsz_kb:2288768

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Rusty Russell 91849dddc4 wire: use struct node_id for node ids.
Don't turn them to/from pubkeys implicitly.  This means nodeids in the store
don't get converted, but bitcoin keys still do.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:33934-35251(34531.4+/-5e+02)
	vsz_kb:2637488
	store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
	listnodes_sec:1.020000-1.290000(1.146+/-0.086)
	listchannels_sec:51.110000-58.240000(54.826+/-2.5)
	routing_sec:30.000000-33.320000(30.726+/-1.3)
	peer_write_all_sec:50.370000-52.970000(51.646+/-1.1)

MCP notable changes from previous patch (>1 stddev):
	-store_load_msec:46184-47474(46673.4+/-4.5e+02)
	+store_load_msec:33934-35251(34531.4+/-5e+02)
	-vsz_kb:2638880
	+vsz_kb:2637488
	-store_rewrite_sec:46.750000-48.280000(47.512+/-0.51)
	+store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell a2fa699e0e Use node_id everywhere for nodes.
I tried to just do gossipd, but it was uncontainable, so this ended up being
a complete sweep.

We didn't get much space saving in gossipd, even though we should save
24 bytes per node.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell 417e1bab7d gossipd: use iterator helpers for iterating node channels.
Makes the next step easier.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:45791-46917(46330.4+/-3.6e+02)
	vsz_kb:2641316
	store_rewrite_sec:47.040000-48.720000(47.684+/-0.57)
	listnodes_sec:1.140000-1.340000(1.2+/-0.072)
	listchannels_sec:50.970000-54.250000(52.698+/-1.3)
	routing_sec:29.950000-31.010000(30.332+/-0.37)
	peer_write_all_sec:51.570000-52.970000(52.1+/-0.54)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell 2bd7df93c6 gossipd: preserve unannounced channels across store compaction.
Otherwise we'd forget them on restart, again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00
Rusty Russell 5b12007a4f gossipd: dev option to allow unknown channels.
This lets us benchmark without a valid blockchain.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>



Header from folded patch 'fixup!_gossipd__dev_option_to_allow_unknown_channels.patch':

fixup! gossipd: dev option to allow unknown channels.

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00
Rusty Russell f8f6533dba dev: --dev-gossip-time so gossipd doesn't prune old data.
This is useful for canned data, such as the million channels project.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00
Rusty Russell b2c93beaed gossipd: use htable instead of simple array for node's channels.
For giant nodes, it seems we spend a lot of time memmoving this array.
Normally we'd go for a linked list, but that's actually hard: each
channel has two nodes, so needs two embedded list pointers, and when
iterating there's no good way to figure out which embedded pointer
we'd be using.

So we (ab)use htable; we don't really need an index, but it's good for
cache-friendly iteration (our main operation).  We can actually change
to a hybrid later to avoid the extra allocation for small nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00
Rusty Russell 00f3a84af2 test: fix thinko in gossipd/test/run-bench-find_route.c
Reported-by: @cdecker
Fixes: #2440
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-03-05 11:42:43 +01:00
Rusty Russell 28f5da7b2f tools/generate-wire: use amount_msat / amount_sat for peer protocol.
Basically we tell it that every field ending in '_msat' is a struct
amount_msat, and 'satoshis' is an amount_sat.  The exceptions are
channel_update's fee_base_msat which is a u32, and
final_incorrect_htlc_amount's incoming_htlc_amt which is also a
'struct amount_msat'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell 3ac0e814d0 daemons: use amount_msat/amount_sat in all internal wire transfers.
As a side-effect of using amount_msat in gossipd/routing.c, we explicitly
handle overflows and don't need to pre-prune ridiculous-fee channels.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell b99293fbb6 short_channel_id: don't accept :-separated in JSON if --allow-deprecated-apis=false
We need to still accept it when parsing the database, but this flag
should allow upgrade testing for devs building on top

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-08 16:52:30 -08:00
Rusty Russell 358b7fda91 getroute: allow caller to specify maximum hops.
This is required for routeboost.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell c0cfddfa95 test/run-bench-find_route: fix so it runs properly.
We didn't populate the channels properly so it always failed.

Additionally, somewhere along the line we kept using the single scid
so we only created one channel.

Also, the next patch will start comparing the pubkeys, so make valid
ones: use an array so we don't affect the benchmark too much.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Christian Decker 659a26ea5a misc: Update short_channel_id representation to use 'x' separators
Reported-by: Alex Bosworth <@alexbosworth>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-01-15 03:50:27 +00:00
Rusty Russell ab735dcbe6 gossipd: wire up memleak detection.
For simplicity we dump leaks to logs, and just return a bool to master.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-22 05:15:42 +00:00
Rusty Russell 9620393109 gossipd: store chainparams internally.
We keep a chain_hash in struct daemon, becayse otherwise we end up with
`&peer->daemon->rstate->chainparams->genesis_blockhash` which is a bit
ridiculous.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-21 21:43:37 +00:00
Rusty Russell 5c60d7ffb2 gossipd: split wire types into msgs from lightningd and msgs from per-peer daemons
This avoids some very ugly switch() statements which mixed the two,
but we also take the chance to rename 'towire_gossip_' to
'towire_gossipd_' for those inter-daemon messages; they're messages to
gossipd, not gossip messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-21 00:36:31 +00:00
Christian Decker 8e83d43c39 opts: Split early from non-early args so plugins can register theirs
The idea is that `plugin` is an early arg that is parsed (from command
line or the config file). We can then start the plugins and have them
tell us about the options they'd like to add to the mix, before we
actually parse them.

Signed-off-by: Christian Decker <@cdecker>
2018-11-13 00:44:50 +01:00
lisa neigut b9331e5ac8 gossipd: parse and respect optional `htlc_maximum_msat`
If another channel has set the optional `htlc_maximum_msat` field,
we should correctly parse that field and respect it when drawing up
routes for payments.
2018-10-09 23:22:52 +00:00
lisa neigut b1ceaf9910 gossipd: Update BOLT-split flags in channel_update
BOLT 7's been updated to split the flags field in `channel_update`
into two: `channel_flags` and `message_flags`. This changeset does the
minimal necessary to get to building with the new flags.
2018-09-21 00:24:12 +00:00
Christian Decker 84905eac2b routing: Make the capacity a parameter to new_chan
As pointed out by @rustyrussell the capacity is now always defined, so we can
fold that into the construction of the channel itself.

Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2018-08-06 22:46:02 +02:00
Christian Decker 8201764117 routing: Skip channels that require larger HTLCs than we are routing
The `htlc_minimum_msat` parameter was ignored so far, and we'd be attempting to
pay and hitting a brick wall by doing so. This patch just skips channels that
are not eligible anyway.
2018-08-06 22:46:02 +02:00
Christian Decker 14000a22bc routing: Skip channels that don't have sufficient capacity
We know the total channel capacity after checking for its existence on-chain, so
we can actually make use of that information to discard channels that don't have
a sufficient capacity anyway, reducing the number of failed attempts.
2018-08-06 22:46:02 +02:00
Christian Decker 8a34933c1a gossip: Annotate locally added channels with their capacity
We were adding channels without their capacity, and eventually annotated them
when we exchanged `channel_update`s. This worked as long as we weren't
considering the channel capacity, but would result in local-only channels to be
unusable once we start checking.
2018-08-06 22:46:02 +02:00
Rusty Russell 7b2641ed0d gossipd: remove peer-related fields and wire messages.
This completes the removal of peer-related messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell 0d442b5ff2 gossipd: move files into connectd.
These source files are only used for peer-related things, so move them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell fed5a117e7 Update ccan/structeq.
structeq() is too dangerous: if a structure has padding, it can fail
silently.

The new ccan/structeq instead provides a macro to define foo_eq(),
which does the right thing in case of padding (which none of our
structures currently have anyway).

Upgrade ccan, and use it everywhere.  Except run-peer-wire.c, which
is only testing code and can use raw memcmp(): valgrind will tell us
if padding exists.

Interestingly, we still declared short_channel_id_eq, even though
we didn't define it any more!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-04 23:57:00 +02:00
arowser 25f60f9456 remove unused return value 2018-06-30 04:27:34 +00:00
Christian Decker 74a1cbd877 gossip: Implement gossip_store compaction
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-09 13:38:46 +02:00
Rusty Russell def18a7bc1 gossipd: implement broadcast_del to delete a specific index.
Required if we want to reorder node_announcement broadcasts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:53:34 +02:00
Rusty Russell a38c619486 gossipd: keep index of node and channel announcements.
This lets detect if a node announce preceeds a channel announce once we
delete the node announcement.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:53:34 +02:00
Rusty Russell 7a32637b5f gossipd: add timestamp to each broadcast message.
This lets us filter by timestamp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 8ee60e2d8e testing: make sure we don't see gossip in bad order.
This is something which generally shouldn't happen, but we didn't
notice it previously.

We ignore this warning in the case where a channel was deleted: this
happens because one side can send an update while the other notices
that the channel is closed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-19 15:52:56 -04:00
Rusty Russell 177a1fc88e gossipd: handle local channel creation separately from update.
Note: this will break the gossip_store if they have current channels,
but it will fail to parse and be discarded.

Have local_add_channel do just that: the update is logically separate
and can be sent separately.

This removes the ugly 'bool add_to_store' flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-19 15:52:56 -04:00
Rusty Russell cca791d1cb routing: clean up channel public/active states.
1. If we have a channel_announcement, the channel is public, otherwise
   it's not.  Not all channels are public, as they can be local: those
   have a NULL channel_announcement.

2. If we don't have a channel_update, we know nothing about that half
   of the channel, and no other fields are valid.

3. We can tell if a half channel is disabled by the flags field directly.

Note that we never send halfchannels without an update over
gossip_getchannels_reply so that marshalling/unmarshalling can be
vastly simplified.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-10 21:35:53 +02:00
Rusty Russell 9d1e496b11 gossipd: use a real update in local_add_channel.
We generate one now, so let's use it.  That lets us simplify the
code, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-10 21:35:53 +02:00
Rusty Russell c71e16f784 broadcast: invert ownership of messages.
Make the update/announce messages own the element in the broadcast map
not the other way around.

Then we keep a pointer to the message, and when we free it
(eg. channel closed, update replaces it), it gets freed from the
broadcast map automatically.

The result is much nicer!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-10 21:35:53 +02:00
Rusty Russell d40d22b68e gossipd: don't try to connect to non-routable addresses.
Someone could try to announce an internal address, and we might probe
it.

This breaks tests, so we add '--dev-allow-localhost' for our tests, so
we don't eliminate that one.  Of course, now we need to skip some more
tests in non-developer mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell 73cd009a4c gossipd/lightningd: use wireaddr_internal.
This replacement is a little menial, but it explicitly catches all
the places where we allow a local socket.  The actual implementation of
opening a AF_UNIX socket is almost hidden in the patch.

The detection of "valid address" is now more complex:

	p->addr.itype != ADDR_INTERNAL_WIREADDR || p->addr.u.wireaddr.type != ADDR_TYPE_PADDING

But most places we do this, we should audit: I'm pretty sure we can't
get an invalid address any more from gossipd (they may be in db, but
we should fix that too).

Closes: #1323
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
practicalswift 5db73c6e27 Avoid static analyzer warnings about potentially uninitialized values 2018-05-01 17:14:33 +02:00
practicalswift abf510740d Force the use of the POSIX C locale for all commands and their subprocesses 2018-04-27 14:02:59 +02:00
Christian Decker 64fbea1528 gossip_store: Save local_add_channel messages and replay them
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-22 12:50:34 +02:00
Christian Decker 7497f972f1 moveonly: Move handle_local_add_channel to routing.h
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-22 12:50:34 +02:00
Rusty Russell b0c2e3cd5c gossipd: use a separate CSV file for the gossip_store types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-11 15:58:18 +02:00
Rusty Russell 2b8293c9f6 gossipd: don't use pwrite, better error messaging on init.
Since we open with O_APPEND, any write() will append as we want it to.

But we want to distinguish a new store creation from a truncation due
to bad version.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-11 15:58:18 +02:00
Christian Decker 63f22d70b5 gossip: Store channel deletions so we don't re-add them on restart
If we only remember the actions that added channels then we'd restore them when
re-reading the gossip_store, so put a tombstone in there to remember to delete
it. These will be cleared upon re-writing the store since the announcements wont
be written anymore.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-30 16:35:00 +02:00
Christian Decker 3826518bd6 gossip: Wrap node_announcements in gossip_store messages
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-25 23:56:59 +00:00