Commit Graph

4129 Commits

Author SHA1 Message Date
Christian Decker 37dc458b4d gossip: Have the broadcast_state track its message count
This is far more precise than bolting on the stale tracking in the
`gossip_store`.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-09 13:38:46 +02:00
Jan Sarenik eab2a62894 external/libsodium: Update to 1.0.16
Fixes lots of extra warnings (#1540)
2018-06-09 06:42:02 +00:00
Nadav Ivgi d9bd25cff8 README: Add ./configure to installation instructions
Now required, after changes in c49765553f
2018-06-08 21:04:55 +02:00
Christian Decker 4e7fc99ae1 gossip: Duplicate removes can result in null pointers in broadcast
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-08 20:00:27 +02:00
arowser 4429c6e7cd udpate broken link 2018-06-08 17:56:48 +02:00
Rusty Russell 109b429311 libbacktrace: pull latest version, which has config.guess fix applied.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 5aa1f37f07 devtools/Makefile: add devtools/onion.c to DEVTOOLS_TOOL_SRC
This way the object file correctly depends on external headers.  Currently
a parallel build on a clean tree can give:

```
In file included from ./common/sphinx.h:6:0,
                 from devtools/onion.c:5:
./bitcoin/pubkey.h:8:10: fatal error: secp256k1.h: No such file or directory
 #include <secp256k1.h>
          ^~~~~~~~~~~~~
compilation terminated.
<builtin>: recipe for target 'devtools/onion.o' failed
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 5d0a54b7f0 travis: use VALGRIND rather than NO_VALGRIND.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell c49765553f Makefile: not running configure first is so yesterday:
Make should only run configure if config.var already exists:

    $ make
    ./configure --reconfigure
    ./configure: 65: .: config.vars: not found
    ./configure --reconfigure
    ./configure: 65: .: config.vars: not found
    Makefile:179: recipe for target 'ccan/config.h' failed
    make: *** [ccan/config.h] Error 2
2018-06-08 17:56:01 +02:00
Rusty Russell 4bd27ec722 Makefile: clean should not delete configure results, distclean only.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 72d103d6bb Makefile: import config.vars.
We leave VALGRIND env var as an override for testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell ca8671de0a configure: add --enable-valgrind/disable-valgrind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell f6d22ed0cf configure: new script.
You can use environment variables or the commandline to set defaults.

It looks very autoconf, but you don't need to learn m4.

Doesn't cover all the obscure flags, but it's easy to extend.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell d395b58ffa ccan: update to have new configurator.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 76dd1577bd external: fix submodule handling for parallel builds, submodule URL changes.
If we change an upstream URL, all submodules break.  Users would need
to run 'git submodule sync'.  Note that the libbacktrace fix was merged
upstream so this is no longer necessary, but it's good for future changes.

Also, stress-testing reveals that git submodule fails locking
'.git/config' when run in paralell.  It also segfaults and other
problems.

This is my final attempt to fix submodules; I've wasted far too many
days on obscure problems it creates: I've already lost one copy of my
repo to apparently unfixable submodule preoblems.  The next "fix" will
be to simply import the source code so it works properly.

Reported-by: @jsarenik
Fixes: #1543
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 5d6a9f3fb0 gossipd: check consistency.
This is a hack to check that our gossip state is consistent on every
insert and delete.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:53:34 +02:00
Rusty Russell da55d3c0ff gossipd: handle node_announcement when channel_announcement removed.
Two cases:
1. Node no longer has any public channels: remove node_announcement.
2. Node's node_announcement now preceeds all the channel_announcements:
   move node_announcement to the end.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:53:34 +02:00
Rusty Russell def18a7bc1 gossipd: implement broadcast_del to delete a specific index.
Required if we want to reorder node_announcement broadcasts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:53:34 +02:00
Rusty Russell a38c619486 gossipd: keep index of node and channel announcements.
This lets detect if a node announce preceeds a channel announce once we
delete the node announcement.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:53:34 +02:00
Christian Decker 294c76e749 gitignore: Ignore .pytest_cache
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:32:27 +00:00
Christian Decker de3b359782 pytest: Use pytest fixtures for the test directory and clean it up
The modern, pytest based, tests now clean up after themselves by removing
directories of successful tests and the base directory if there was no failure.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:32:27 +00:00
Christian Decker 38b7a0e2d2 pytest: Remove directories of successful tests
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:32:27 +00:00
Christian Decker d43403257d wallet: Cleanup db files if the tests succeed
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:32:27 +00:00
Christian Decker 6298ce3b03 gossip: Don't ask bitcoind for outpoints we should know
Compares the `blocknum` in the `short_channel_id` with the range of blocks we
store in the database and abort if we should have known about it. Avoids
bombarding `bitcoind` with requests for channels that have already been spent or
were invalid in the first place.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:30:02 +00:00
Christian Decker 2415f48723 topo: Tell chain_topology about the min and max block height
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:30:02 +00:00
Christian Decker 024dca0fff wallet: Return both min and max block heights
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:30:02 +00:00
Christian Decker 0d4b7eaa2c topo: Have chain_topology track both min and max block heights
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:30:02 +00:00
Rusty Russell 1bb7713274 gossipd: minor cleanups.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell e4457e59ed pytest: test_gossip_jsonrpc can always test aliases.
The whole test is under DEVELOPER anyway, but even if it weren't it
would work fine.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 035d6067e4 Rename consider_own_node_announce to maybe_send_own_node_announce.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 383b309a00 utils: make subd_pid return the *last* pid, in case we restarted daemon.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 5ec454c7b2 gossipd: don't queue node_announce unless we've queued channel_announce.
We *accept* a node_announce if we have a channel_announce, but we
can't queue it until we queue the channel_announce, which we only do
once we have recieved a channel_update.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell e9483f12b6 test_lightningd.py: actually make sure we do zlib encoding.
Previous replies weren't large enough; add another channel and then
use IO tracing to make sure the reply is zlib encoded.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell f52245d442 gossipd: support and use zlib encoding in short_channel_id encoding.
We still use uncompressed if zlib turns out to be larger.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 083a2cee07 zlib: add as a requirement.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 9e51e196c1 gossipd: dev-set-max-scids-encode-size to artificially force "full" replies.
We cap each reply at a single one, which forces the code into our
recursion logic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 118f099dd8 gossip: dev-query-channel-range to test query_channel_range.
We keep a crappy bitmap, and finish when their replies cover
everything we asked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 0dda5d4e1c gossipd: handle query_channel_range
We send them all the short_channel_ids we have in a given range.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell c34b49c356 gossipd: add dev-send-timestamp-filter command for testing timestamp filtering.
Since we currently only (ab)use it to send everything, we need a way to
generate boutique queries for testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell db6a6442cb gossipd: single-thread the gossip timer.
We have a function called 'wake_pkt_out' which is really 'start
gossiping', so rename it to 'wake_gossip_out'.

In addition, it's fired both on a timer, and in response to our first
gossip_timestamp_filter, which leads to very confusing (though,
technically, not incorrect) behavior.

Keep a single timer at all times, which now doubles as the flag to
indicating we're syncing right now.  Set it once we're done syncing
gossip.

Technically this means we got from once-every-60-seconds to
quiet-for-60-seconds-between-gossip, but that's OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 531c82b6ad gossipd: handle gossip_timestamp_filter message.
And initialize filter (to "never") when we negotiated LOCAL_GOSSIP_QUERIES,
and send initial filter message.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 97bb6c5a28 gossipd: ensure incoming timestamps are reasonable.
This is kind of orthogonal to the other changes, but makes sense: if we
would instantly or never prune the message, don't accept it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 7a32637b5f gossipd: add timestamp to each broadcast message.
This lets us filter by timestamp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell c633cbe2ee tests: add dev-query-scids
And write the test for it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 4d8b29089b gossipd: wire up infrastructure to generate query_short_channel_ids msg.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 7ee5da858c gossipd: handle query_short_channel_ids message.
This doesn't handle zlib yet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 32c39c2979 gossipd: send node announcements after short_channel_id replies.
We use the same system as for gossip: we trickle out replies when we're
otherwise idle.

As we trickle out replies to query_short_channel_ids, we remember the
pubkeys of nodes we mention.  At the end, we sort and uniquify, and
then send any node_announcements we have for those.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 5864415d31 gossipd: infrastructure to handle short_channel_id replies.
We use the same system as for gossip: we trickle out replies when we're
otherwise idle.

This is minimal infrastructure: we don't actually process the
query_short_channel_ids message yet, nor do we append node
announcements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 6c6da45f53 wire: Update to lastest BOLT draft.
This includes the gossip query messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 3e07971582 features: define LOCAL_GOSSIP_QUERIES feature.
From BOLT #9 proposed update.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00