Commit Graph

6345 Commits

Author SHA1 Message Date
Christian Decker 245dff26e0 pytest: Updated requirements.txt to latest versions
This is an issue that was raised in #2665: some of the dependencies where
causing warnings to be added to the logs about deprecated dependencies. Since
I did not get these warnings I just blanket updated all the dependencies in
the hopes of getting the warnings to resolve.

Signed-off-by: Christian Decker <@cdecker>
2019-09-02 16:31:36 +02:00
Christian Decker c3254e6639 docs: Update manpages to be recognized correctly by mrkd
mrkd started enforcing the `name -- short description` style of top-level
headings somewhere, and was thus failing to build the man-pages. I swapped
the title and with the existing short description to make it work
again. `mrkd` will automatically infer the section from the filename so no
need to put it in the title as well.

In addition I removed the "last updated" lines at the bottom since they are
out of date at best, and misleading at the worst. If we want to keep them, I'd
suggest generating them from the commit that last touched them.
2019-09-02 16:31:36 +02:00
Rusty Russell 049529542a lightningd: delay reprocessing of incoming htlcs at startup until plugins ready.
Fixes: #2923
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-01 16:55:32 +02:00
Rusty Russell f0ab4d72da pytest: add test that htlc_accepted hook only gets called after init.
We fail this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-01 16:55:32 +02:00
nicolas.dorier 49496ab3ba [Docker] Use stretch-slim instead of Alpine 2019-08-30 16:17:51 +02:00
darosior 8f8e955421 Update installation instructions and Dockerfile
Get rid of the 'asciidoc' dependency, make developers also install 'doc/requirements.txt'
2019-08-30 00:34:11 +00:00
darosior 092f20aeeb README: Replace broken links to manpages 2019-08-30 00:34:11 +00:00
Rusty Russell 3ef5160fa7 travis: add --enable-experimental-features (build-only) test.
Also, add a 32-bit build-only test, and move all the build-only tests
to the top.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Rusty Russell 02b9b7f6e6 tests: update mocks for --enable-experimental-features builds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Christian Decker b77270fa46 docs: Add the title of the manpage to TOC and sort alphabetically
Fixes #2987
2019-08-29 14:56:24 +02:00
Rusty Russell 5e3e2f4e17 run-secret_eq_consttime: try harder.
This will never be reliable under high load, without making it unable
to detect real errors.  But the test is useful because if we don't
have this test we'll never notice if we break the const-timedness of
our implementation.

So, move the calloc out of the test loop (which seems to make it more
reliable), and then after we've run it, check the 1-minute load
average.  Too high, we don't complain about results.  It's not
perfect, but it's better.

Running 100 times (-O3) serially gave 100 successes with the following results:
	Constant: Within 5% 562-926(832.89+/-73)/1000 times
        Non-constant: More than 5% slower 860-990(956.35+/-26)/1000 times

More importantly, if we swap the const and non-const tests, we get
the expected 100 failures:
	Non-constant: Within 5% 14-79(41.17+/-14)/1000 times
	Constant: More than 5% slower 44-231(111.89+/-33)/1000 times

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell 6901732ee0 lightningd: create --list-features-only which lists what features we support.
This allows the lightning-rfc protocol tests to automatically query what
features we support.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell 17edcfa24d features: don't assume we'll always only advertize optional features.
Generalize things a bit so OPTIONAL_FEATURE() and COMPULSORY_FEATURE()
work with either odd or even features, then explicitly use OPTIONAL_FEATURE
in our internal feature array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell a38131f349 EXPERIMENTAL: advertize the extended_queries feature.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell d943d8abbc lightningd: expose full onion error when we have it.
Mainly useful for testing.  In particular, we don't save it to the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell 189b2f1313 BOLT: update CSV to latest bolt version.
This removes the WIRE_FINAL_EXPIRY_TOO_SOON which leaked too much info,
and adds the blockheight to WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell 2600a6ed2e channeld: get current block height when an HTLC fails.
We need it to put in the error code for
WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell 6349222ea2 Spec: Update to latest BOLT, include our first global feature definition.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell 6310a183af pytest: ignore occasional bad gossip in reorg tests.
It can happen, as expected when we reorg and update for a now-invalid
channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell 20a2bf9547 pytest: make test_forward_event_notification more explicit.
We seem to be getting intermittant failures, but it's hard
to disgnose.  Simplify it by moving all the test logic into
the test itself, and making the plugin dumber.  This means we'll
see exactly what the differences are if it fails again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell dbc1ebe06a Makefile: fix race when checking manpages.
By not depending on plugins, we can call lightningd before plugins are
built, and not get all options:

    diff of command names vs manpage names:
    --- /dev/fd/63  2019-08-22 05:06:55.265659216 +0000
    +++ /dev/fd/62  2019-08-22 05:06:55.265659216 +0000
    @@ -3,6 +3,8 @@
     allow-deprecated-apis=
     always-use-proxy=
     announce-addr=
    +autocleaninvoice-cycle=
    +autocleaninvoice-expired-by=
     autolisten=
     bind-addr=
     bitcoin-cli=

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell 394aef2391 Makefile: fix race with 'make check'
installcheck calls $(MAKE) under the covers, which can race with the
current builds, and we can try installing something which is still
being built:

    collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
    compilation terminated.
    /usr/bin/ld: can not read symbols: File truncated
    /usr/bin/ld: .eh_frame/.stab edit: File truncated
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.interp' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.note.ABI-tag' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.note.gnu.build-id' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.hash' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynsym' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynstr' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.version' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.version_r' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.rela.dyn' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.rela.plt' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.init' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.plt' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.plt.got' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.text' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.fini' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.rodata' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.eh_frame' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.init_array' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.fini_array' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.data.rel.ro' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynamic' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.got' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.data' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_json_command' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_hooks' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_type_to_string' not in segment
    Makefile:390: recipe for target 'lightningd/lightningd' failed
    make[1]: *** [lightningd/lightningd] Error 1
    make[1]: Leaving directory '/home/rusty/lightning'
    Makefile:553: recipe for target 'installcheck' failed
    make: *** [installcheck] Error 2

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell 3f1ca18f2f pytest: optimize test_closing_torture
VALGRIND=1, SLOW_MACHINE=0:
Before: 197.74 seconds
After: 135.43 seconds

Note that we now spend about 13 seconds in teardown, could probably
be optimized.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell 686df235a2 pytest: fix race in test_block_backfill.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell 11833192a1 tools: fix Makefile warning
Makefile:424: warning: overriding recipe for target 'clean'
    tools/Makefile:12: warning: ignoring old recipe for target 'clean'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell 28185c397c gossipd: fix gossip send in case query_flags cause no output.
Fortunately, again, only happens with EXPERIMENTAL_FEATURES.

If the query causes us not to actually send anything, we won't
get called again.  This can validly happen if they only asked for
the node_announcements, for example.

(Found by protocol tests).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Rusty Russell 855dff704c gossipd: test crc32 routines using test vectors from PR.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Rusty Russell d1a1592cc8 gossipd: fix calculation of crc32 of update.
Currently EXPERIMENTAL_FEATURES only, fortunately.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Simon Vrouwe 5b5d70d640 lightningd/plugin: extend plugins->startup window to include plugins_config
Fixes incorrect configuration[`startup`] in plugin `init`, modified tests
to test this.
2019-08-27 00:02:20 +00:00
Rusty Russell c2f58aec9d contrib/plugins: cowsay
___________________________________________________________________________
/ The most important plugin. Ever. Much thanks to @practicalswift and @jb55 \
\ for their improvements!                                                   /
 ---------------------------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 23:10:36 +00:00
Rusty Russell 0ec8304901 gossipd: fix premature towire_reply_short_channel_ids_end if no node_announcement.
Our "are we finished?" logic was wrong: it tested if there are no more
node_announcements, but it's possible that there were no node_announcements
for either end of the channel whose information we sent.

This is actually quite unusual on the real network: looking at mainnet
statis from last May, 4301 of 4337 nodes have node_announcements.

However, with query flags it's much more likely, since they might not
ask for node announcements at all.

(Found by gossip protocol tests)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 23:09:00 +00:00
Rusty Russell af645b38c8 common: fix missing HOME case, when we override --lightning-dir.
It fails because option.c tal_free's the old value, which must be
a tal ptr.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 23:09:00 +00:00
Simon Vrouwe 2f5f364e17 pylightning/plugin: add `lightning_version` member to plugin 2019-08-26 15:39:03 -07:00
Simon Vrouwe da74f0118d lightningd/plugin: at plugin start, set env LIGHTNINGD_VERSION
So plugins can know what version they are dealing with.
2019-08-26 15:39:03 -07:00
Rusty Russell 9dd314226d devtools/mkclose: make a mutual close transaction.
Guess who's been writing gossip protocol tests?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell c9817b225b bitcoin/tx: pass struct amount_sat by copy.
This is the normal convention for this type; it makes using converters
a little easier.  See next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell 2f1e116510 gossipd: use htable_count() rather than reaching into htable struct.
Now ccan/htable provides the helper, let's use it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell 39b34a35c8 bitcoin/tx.c: don't free witness implicitly.
This causes a crash in mkfunding, which didn't expect it:

    $ devtools/mkfunding 16835ac8c154b616baac524163f41fb0c4f82c7b972ad35d4d6f18d854f6856b 1 0.01btc 253 76edf0c303b9e692da9cb491abedef46ca5b81d32f102eb4648461b239cb0f99 0000000000000000000000000000000000000000000000000000000000000010 0000000000000000000000000000000000000000000000000000000000000020
    # funding sig: 798d96d5a057b5b7797988a855217f41af05ece3ba8278366e2f69763c72e78565d5dd7eeddc0766ddf65557c92b9c52c301f23f94d2cf681860d32153e6ae1e
    # funding witnesses: [
    Aborted (core dumped)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Damian Mee e4d0ef6e11 doc: fix wording for fundchannel_complete 2019-08-23 08:23:42 -07:00
Rusty Russell 997af6f3f4 contrib: make stricter shellcheck from Ubuntu 18.04 happy
In contrib/bootstrap-node.sh line 7:
if type lightning-cli >/dev/null 2>&1; then
   ^-- SC2039: In POSIX sh, 'type' is undefined.

In contrib/startup_regtest.sh line 41:
	type lightning-cli || return
        ^-- SC2039: In POSIX sh, 'type' is undefined.

In contrib/startup_regtest.sh line 42:
	type lightningd || return
        ^-- SC2039: In POSIX sh, 'type' is undefined.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-23 08:22:28 -07:00
darosior 3d851541f4 Travis: install doc/ requirements 2019-08-22 01:35:01 +00:00
darosior 529630a453 doc/Makefile: update readthedocs pages on doc-all
Stolen from @rustyrussell's #2928
2019-08-22 01:35:01 +00:00
darosior 77f34fad2b doc/Makefile: adapt 'check-manpages' to markdown 2019-08-22 01:35:01 +00:00
darosior 49be9b5545 doc/manpages: Add manpages to readthedocs 2019-08-22 01:35:01 +00:00
darosior d37a2d7c3e doc/manpages: regenerate manpages using markdown templates 2019-08-22 01:35:01 +00:00
darosior 76a4c5f287 doc/Makefile: Use mrkd to generate manpages 2019-08-22 01:35:01 +00:00
darosior 80927039e0 doc/manpages: replace asciidoc by markdown 2019-08-22 01:35:01 +00:00
darosior 3f29216f7b doc/Makefile: Add lightning-plugin manpage 2019-08-22 01:35:01 +00:00
Rusty Russell 0954feddc7 json: speed up shutdown.
We currently end up sleeping for 1 second for channeld and gossipd:
better to use a normal blocking waitpid and an alarm to wake us in
case they don't exit.

This speeds up `lightning-cli stop` on my machine from 2.008s to 0.008s:
a 286 times speedup!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-22 01:34:03 +00:00
Rusty Russell 9dfc854d94 tools/Makefile: put all tools/ stuff here.
Generally a Makefile should control make within that directory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00