Commit Graph

6309 Commits

Author SHA1 Message Date
Rusty Russell 2f1e116510 gossipd: use htable_count() rather than reaching into htable struct.
Now ccan/htable provides the helper, let's use it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell 39b34a35c8 bitcoin/tx.c: don't free witness implicitly.
This causes a crash in mkfunding, which didn't expect it:

    $ devtools/mkfunding 16835ac8c154b616baac524163f41fb0c4f82c7b972ad35d4d6f18d854f6856b 1 0.01btc 253 76edf0c303b9e692da9cb491abedef46ca5b81d32f102eb4648461b239cb0f99 0000000000000000000000000000000000000000000000000000000000000010 0000000000000000000000000000000000000000000000000000000000000020
    # funding sig: 798d96d5a057b5b7797988a855217f41af05ece3ba8278366e2f69763c72e78565d5dd7eeddc0766ddf65557c92b9c52c301f23f94d2cf681860d32153e6ae1e
    # funding witnesses: [
    Aborted (core dumped)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Damian Mee e4d0ef6e11 doc: fix wording for fundchannel_complete 2019-08-23 08:23:42 -07:00
Rusty Russell 997af6f3f4 contrib: make stricter shellcheck from Ubuntu 18.04 happy
In contrib/bootstrap-node.sh line 7:
if type lightning-cli >/dev/null 2>&1; then
   ^-- SC2039: In POSIX sh, 'type' is undefined.

In contrib/startup_regtest.sh line 41:
	type lightning-cli || return
        ^-- SC2039: In POSIX sh, 'type' is undefined.

In contrib/startup_regtest.sh line 42:
	type lightningd || return
        ^-- SC2039: In POSIX sh, 'type' is undefined.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-23 08:22:28 -07:00
darosior 3d851541f4 Travis: install doc/ requirements 2019-08-22 01:35:01 +00:00
darosior 529630a453 doc/Makefile: update readthedocs pages on doc-all
Stolen from @rustyrussell's #2928
2019-08-22 01:35:01 +00:00
darosior 77f34fad2b doc/Makefile: adapt 'check-manpages' to markdown 2019-08-22 01:35:01 +00:00
darosior 49be9b5545 doc/manpages: Add manpages to readthedocs 2019-08-22 01:35:01 +00:00
darosior d37a2d7c3e doc/manpages: regenerate manpages using markdown templates 2019-08-22 01:35:01 +00:00
darosior 76a4c5f287 doc/Makefile: Use mrkd to generate manpages 2019-08-22 01:35:01 +00:00
darosior 80927039e0 doc/manpages: replace asciidoc by markdown 2019-08-22 01:35:01 +00:00
darosior 3f29216f7b doc/Makefile: Add lightning-plugin manpage 2019-08-22 01:35:01 +00:00
Rusty Russell 0954feddc7 json: speed up shutdown.
We currently end up sleeping for 1 second for channeld and gossipd:
better to use a normal blocking waitpid and an alarm to wake us in
case they don't exit.

This speeds up `lightning-cli stop` on my machine from 2.008s to 0.008s:
a 286 times speedup!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-22 01:34:03 +00:00
Rusty Russell 9dfc854d94 tools/Makefile: put all tools/ stuff here.
Generally a Makefile should control make within that directory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell 278b12982d tools/test/Makefile: disable unless we're in DEVELOPER mode.
Because it required update-mocks, which is a hack which relies on the
format of linker errors (!) I'd prefer to make this --enable-developer
only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell c74c1e56cc tools/test/Makefile: fix always-on generation of tools/test/gen_test.c
We need to tell it that the tmp file is an intermediate, so doesn't need
remaking if it doesn't exist.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell 0fdf22988d configure: absorb python3-mako check.
This is a better place for such checks than the Makefile, and we'll likely
have more in future.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
lisa neigut e99720344e wire-gen: ensure that tlv messages are correctly ordered
Our TLV serializer relies on TLV outputs to be ordered by type
number. Prior to this commit we relied on 1) the ordering in the
RFC to be correct and 2) users to be using a version of Python that
respects stable ordering of dicts (i.e. Python 3.7+)

Instead of relying on these implicitly, we now explicitly sort messages
by type number when the TLV sets.

Resolves #2956.

Thanks-To: @ScottTre for the sort function
Reported-By: @ZmnSCPxj
2019-08-21 03:53:54 +00:00
trueptolemy 4929034a40 json: Make payment_hash use `json_add_sha256` 2019-08-21 09:32:21 +08:00
trueptolemy a9e346a1f4 json: Add the json interface for `struct sha256` 2019-08-21 09:32:21 +08:00
trueptolemy 23bfdc307f wallet: Use `struct sha256` for payment_hash in `struct forwarding` 2019-08-21 09:32:21 +08:00
trueptolemy 5f6196a42d cleanup: Use the most common abbreviation of 'ctx' in `json_tok_address_scriptpubkey` 2019-08-21 09:30:50 +08:00
ZmnSCPxj 8eddbb4561 CHANGELOG.md: Reset for next release. 2019-08-21 01:05:20 +00:00
Sjors Provoost 55776c1605 systemd: After=network-online.target 2019-08-20 20:14:36 +02:00
Rusty Russell 804aee766e v0.7.2.1: Nakamoto's Pre-approval by US Congress
(Last-minute fixes, I hate last-minute fixes!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-20 05:36:16 +00:00
Christian Decker 8b8538024d bitcoind: Defer initialization of filteredblock_call->result
During sync it is highly likely that we can coalesce multiple calls and share
results among them. We also report back failures for non-existing blocks early
on, so we don't run into issues with blocks that our bitcoind doesn't have
yet.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-20 00:07:38 +00:00
Christian Decker 187e493ab8 gossip: Stop backfilling the future
This was caused by us not checking against the max_blockheight, but rather the
min_blockheight which can be negative with a newly created node. This is still
safe since we check for duplicates anyway in `wallet_filteredblock_add`.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-20 00:07:38 +00:00
Rusty Russell 944439853a pytest: two tests for gossip of channels in as-yet-unknown blocks.
Two tests which crash lightningd in different ways.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-20 00:07:38 +00:00
Rusty Russell e40f07803c devtools/gossipwith: allow --max-messages=0 to avoid reading msgs at all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-20 00:07:38 +00:00
Nicolas Dorier a358b9ab57 [Docker] Fixing the environment variable for arm32 2019-08-19 07:09:47 +00:00
Karl-Johan Alm 46c184a50e signet: update genesis hash to use new POW-valid version 2019-08-19 07:09:06 +00:00
Rusty Russell 04a57ae4af doc/MAKING-RELEASES.md: update.
1. These days we delete the [Unreleased] tag during rcs.
2. Make sure we test the release build process during rc1, since I
   screwed that up last release.
3. Add a section on rc2, etc.
4. Do final release via a github PR, since I screwed that up on the
   prior release.
5. Update `tools/build-release.sh` and instructions to show that we now
   make a reproducible build for Ubuntu 18.04 x86-64.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell 303c221641 tools/test/Makefile: we depend on tools/test/gen_test.h
This is because update-mocks actually tries to build it to get link
errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell de361456d5 Makefile: detect version properly from dirname.
Our reproducibile builds use the dirname to get version, but they have
a v in them (the tools/repro-build.sh script gets this right, so I
copied that).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell a525427272 pytest: don't run test_forward_local_failed_stats under Travis w/ VALGRIND
It's timing out: I suspect it's simply too much memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell f73ae3c198 test: make run-secret_eq_consttime more robust.
This is more reliable under load now: shorten the times so it is
likely to run in a single timeslice, and add a nanosleep so it's
likely to be at the start of the timeslice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell bc23baf895 Makefile: fix make check so we can run in parallel.
And make travis use -j for `make check`

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell 43edd91440 Travis: Try sharding 12 ways, not 7.
I tried 8, then 9... then jumped to 12.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell c0ba5879df tools/test: can't generate mocks until submodules checked out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 14:08:43 +09:30
Rusty Russell dd617f9cbe v0.7.2
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 04:27:26 +00:00
Rusty Russell dbb15b0045 Makefile: fix occasional race where we fail to compile tools/tests.
This simplifies the dependencies:
1. Objs depend on headers, not other objs.
2. Programs depend on objs.
3. A .o file will generally implicitly depend on the .c file it's built from.
4. If a file has a build line, it's often better to list all deps there.
5. I spotted some missing 'make clean' files.

The particular problem in this case seems to be that make would use
tools/test/gen_test.c before it was ready.  It's probably confused by
the use of recursive make via update-mocks, so explicitly split that
into two stages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 04:20:05 +00:00
Rusty Russell 403c054c4e contrib: add mako requirement to docker files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-16 00:26:09 +00:00
Rusty Russell c7b5216be8 v0.7.2-rc2
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 04:26:10 +00:00
Rusty Russell 1bd1829a8e doc: update lightning-listforwards(7)
It didn't have the new payment_hash field in it, and needed updating to
list optional fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 03:12:56 +00:00
Rusty Russell f18b911032 lightningd: listforwards shouldn't put in zero fields for fields we don't know.
Technically, this is an API change :(  So I made it conditional.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 03:12:56 +00:00
lisa neigut 802ebe768c rpc: fix crash 'listforwards' when payment_hash is empty
```
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): FATAL SIGNAL 11 (version v0.7.2rc1)
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: common/daemon.c:45 (send_backtrace) 0x563349d07879
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: common/daemon.c:53 (crashdump) 0x563349d078c9
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0x7efd7b996f1f
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/str/hex/hex.c:59 (hex_encode) 0x563349d57fec
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/json.c:380 (json_add_hex) 0x563349cd9dd3
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/peer_htlcs.c:2151 (json_format_forwarding_object) 0x563349cfa7ac
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/peer_htlcs.c:2198 (listforwardings_add_forwardings) 0x563349cfa99d
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/peer_htlcs.c:2216 (json_listforwards) 0x563349cfaa55
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/jsonrpc.c:650 (parse_request) 0x563349cdc184
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/jsonrpc.c:748 (read_json) 0x563349cdc5ae
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x563349d4bbe5
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x563349d4c762
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x563349d4c7a0
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x563349d4e7f5
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0x563349cd8afe
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: lightningd/lightningd.c:834 (main) 0x563349cded3a
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0x7efd7b979b96
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0x563349cc5909
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0xffffffffffffffff
```

[ Modified to simply omit field --RR ]
2019-08-15 03:12:56 +00:00
lisa neigut 58fb1528dd add_htlc hook: fix crash when failing UPDATE failcode
Passing in an UPDATE failcode crashes, since the next hop's channel id
was passed in as NULL. Fixed by passing in id.

```
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): FATAL SIGNAL 11 (version v0.7.2rc1-8-gbf3b77a-modded)
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: common/daemon.c:45 (send_backtrace) 0x55fef4ef036f
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: common/daemon.c:53 (crashdump) 0x55fef4ef03bf
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0x7f7762401f1f
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: lightningd/peer_htlcs.c:104 (fail_in_htlc) 0x55fef4edd9d7
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: lightningd/peer_htlcs.c:785 (htlc_accepted_hook_callback) 0x55fef4edf2c7
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin_hook.c:86 (plugin_hook_callback) 0x55fef4ee765f
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin.c:251 (plugin_response_handle) 0x55fef4ee44b2
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin.c:341 (plugin_read_json_one) 0x55fef4ee4637
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin.c:366 (plugin_read_json) 0x55fef4ee4764
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55fef4f38c7a
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55fef4f397f7
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55fef4f39835
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55fef4f3b88a
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0x55fef4ec0afe
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/lightningd.c:834 (main) 0x55fef4ec6f5a
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0x7f77623e4b96
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0x55fef4ead909
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0xffffffffffffffff
```
2019-08-15 02:24:18 +00:00
Rusty Russell c43d09dfd3 pytest: two more slow tests I missed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 02:22:01 +00:00
Rusty Russell 25ddb80823 ccan: update to fix shutdown slowness after plugin load.
Dynamic plugins were keeping fds open; they should not have these
at all anyway, but worse, they interfere with operation because
we don't notice they're closed.

The symptom was that shutdown of the test_plugin_slowinit and
test_plugin_command was 30 seconds (10 seconds grace to kill each daemon).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 02:22:01 +00:00
Rusty Russell bf3b77a947 Travis: skip testing VALGRIND=1 DEVELOPER=0, remove the slowest non-developer tests.
I don't remember ever seeing a bug which only showed up in VALGRIND=1 with developer
mode disabled, so don't test that, and spread out the other test more evenly.

In addition, disable the worst-performing tests in DEVELOPER=0 mode.

Here timings from my build machine: the worst 6 (- DEVELOPER=0 VALGRIND=0)
with the same tests (+ DEVELOPER=1 VALGRIND=1)

-452.42s call     tests/test_pay.py::test_channel_spendable
+87.69s call     tests/test_pay.py::test_channel_spendable
-335.66s call     tests/test_gossip.py::test_gossip_store_compact_on_load
+47.41s call     tests/test_gossip.py::test_gossip_store_compact_on_load
-332.07s call     tests/test_connection.py::test_opening_tiny_channel
+89.71s call     tests/test_connection.py::test_opening_tiny_channel
-331.97s call     tests/test_pay.py::test_channel_spendable_large
+56.23s call     tests/test_pay.py::test_channel_spendable_large
-305.28s call     tests/test_invoices.py::test_invoice_routeboost
+37.57s call     tests/test_invoices.py::test_invoice_routeboost
-284.28s call     tests/test_plugin.py::test_htlc_accepted_hook_forward_restart
+49.12s call     tests/test_plugin.py::test_htlc_accepted_hook_forward_restart

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 11:14:38 +00:00