Commit Graph

6341 Commits

Author SHA1 Message Date
trueptolemy 4929034a40 json: Make payment_hash use `json_add_sha256` 2019-08-21 09:32:21 +08:00
trueptolemy a9e346a1f4 json: Add the json interface for `struct sha256` 2019-08-21 09:32:21 +08:00
trueptolemy 23bfdc307f wallet: Use `struct sha256` for payment_hash in `struct forwarding` 2019-08-21 09:32:21 +08:00
trueptolemy 5f6196a42d cleanup: Use the most common abbreviation of 'ctx' in `json_tok_address_scriptpubkey` 2019-08-21 09:30:50 +08:00
ZmnSCPxj 8eddbb4561 CHANGELOG.md: Reset for next release. 2019-08-21 01:05:20 +00:00
Sjors Provoost 55776c1605 systemd: After=network-online.target 2019-08-20 20:14:36 +02:00
Rusty Russell 804aee766e v0.7.2.1: Nakamoto's Pre-approval by US Congress
(Last-minute fixes, I hate last-minute fixes!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-20 05:36:16 +00:00
Christian Decker 8b8538024d bitcoind: Defer initialization of filteredblock_call->result
During sync it is highly likely that we can coalesce multiple calls and share
results among them. We also report back failures for non-existing blocks early
on, so we don't run into issues with blocks that our bitcoind doesn't have
yet.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-20 00:07:38 +00:00
Christian Decker 187e493ab8 gossip: Stop backfilling the future
This was caused by us not checking against the max_blockheight, but rather the
min_blockheight which can be negative with a newly created node. This is still
safe since we check for duplicates anyway in `wallet_filteredblock_add`.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-20 00:07:38 +00:00
Rusty Russell 944439853a pytest: two tests for gossip of channels in as-yet-unknown blocks.
Two tests which crash lightningd in different ways.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-20 00:07:38 +00:00
Rusty Russell e40f07803c devtools/gossipwith: allow --max-messages=0 to avoid reading msgs at all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-20 00:07:38 +00:00
Nicolas Dorier a358b9ab57 [Docker] Fixing the environment variable for arm32 2019-08-19 07:09:47 +00:00
Karl-Johan Alm 46c184a50e signet: update genesis hash to use new POW-valid version 2019-08-19 07:09:06 +00:00
Rusty Russell 04a57ae4af doc/MAKING-RELEASES.md: update.
1. These days we delete the [Unreleased] tag during rcs.
2. Make sure we test the release build process during rc1, since I
   screwed that up last release.
3. Add a section on rc2, etc.
4. Do final release via a github PR, since I screwed that up on the
   prior release.
5. Update `tools/build-release.sh` and instructions to show that we now
   make a reproducible build for Ubuntu 18.04 x86-64.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell 303c221641 tools/test/Makefile: we depend on tools/test/gen_test.h
This is because update-mocks actually tries to build it to get link
errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell de361456d5 Makefile: detect version properly from dirname.
Our reproducibile builds use the dirname to get version, but they have
a v in them (the tools/repro-build.sh script gets this right, so I
copied that).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell a525427272 pytest: don't run test_forward_local_failed_stats under Travis w/ VALGRIND
It's timing out: I suspect it's simply too much memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell f73ae3c198 test: make run-secret_eq_consttime more robust.
This is more reliable under load now: shorten the times so it is
likely to run in a single timeslice, and add a nanosleep so it's
likely to be at the start of the timeslice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell bc23baf895 Makefile: fix make check so we can run in parallel.
And make travis use -j for `make check`

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell 43edd91440 Travis: Try sharding 12 ways, not 7.
I tried 8, then 9... then jumped to 12.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 01:38:12 +00:00
Rusty Russell c0ba5879df tools/test: can't generate mocks until submodules checked out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 14:08:43 +09:30
Rusty Russell dd617f9cbe v0.7.2
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 04:27:26 +00:00
Rusty Russell dbb15b0045 Makefile: fix occasional race where we fail to compile tools/tests.
This simplifies the dependencies:
1. Objs depend on headers, not other objs.
2. Programs depend on objs.
3. A .o file will generally implicitly depend on the .c file it's built from.
4. If a file has a build line, it's often better to list all deps there.
5. I spotted some missing 'make clean' files.

The particular problem in this case seems to be that make would use
tools/test/gen_test.c before it was ready.  It's probably confused by
the use of recursive make via update-mocks, so explicitly split that
into two stages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 04:20:05 +00:00
Rusty Russell 403c054c4e contrib: add mako requirement to docker files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-16 00:26:09 +00:00
Rusty Russell c7b5216be8 v0.7.2-rc2
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 04:26:10 +00:00
Rusty Russell 1bd1829a8e doc: update lightning-listforwards(7)
It didn't have the new payment_hash field in it, and needed updating to
list optional fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 03:12:56 +00:00
Rusty Russell f18b911032 lightningd: listforwards shouldn't put in zero fields for fields we don't know.
Technically, this is an API change :(  So I made it conditional.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 03:12:56 +00:00
lisa neigut 802ebe768c rpc: fix crash 'listforwards' when payment_hash is empty
```
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): FATAL SIGNAL 11 (version v0.7.2rc1)
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: common/daemon.c:45 (send_backtrace) 0x563349d07879
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: common/daemon.c:53 (crashdump) 0x563349d078c9
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0x7efd7b996f1f
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/str/hex/hex.c:59 (hex_encode) 0x563349d57fec
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/json.c:380 (json_add_hex) 0x563349cd9dd3
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/peer_htlcs.c:2151 (json_format_forwarding_object) 0x563349cfa7ac
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/peer_htlcs.c:2198 (listforwardings_add_forwardings) 0x563349cfa99d
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/peer_htlcs.c:2216 (json_listforwards) 0x563349cfaa55
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/jsonrpc.c:650 (parse_request) 0x563349cdc184
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: lightningd/jsonrpc.c:748 (read_json) 0x563349cdc5ae
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x563349d4bbe5
2019-08-14T17:50:39.100Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x563349d4c762
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x563349d4c7a0
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x563349d4e7f5
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0x563349cd8afe
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: lightningd/lightningd.c:834 (main) 0x563349cded3a
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0x7efd7b979b96
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0x563349cc5909
2019-08-14T17:50:39.101Z **BROKEN** lightningd(11355): backtrace: (null):0 ((null)) 0xffffffffffffffff
```

[ Modified to simply omit field --RR ]
2019-08-15 03:12:56 +00:00
lisa neigut 58fb1528dd add_htlc hook: fix crash when failing UPDATE failcode
Passing in an UPDATE failcode crashes, since the next hop's channel id
was passed in as NULL. Fixed by passing in id.

```
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): FATAL SIGNAL 11 (version v0.7.2rc1-8-gbf3b77a-modded)
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: common/daemon.c:45 (send_backtrace) 0x55fef4ef036f
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: common/daemon.c:53 (crashdump) 0x55fef4ef03bf
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0x7f7762401f1f
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: lightningd/peer_htlcs.c:104 (fail_in_htlc) 0x55fef4edd9d7
2019-08-15T00:19:49.639Z **BROKEN** lightningd(17070): backtrace: lightningd/peer_htlcs.c:785 (htlc_accepted_hook_callback) 0x55fef4edf2c7
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin_hook.c:86 (plugin_hook_callback) 0x55fef4ee765f
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin.c:251 (plugin_response_handle) 0x55fef4ee44b2
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin.c:341 (plugin_read_json_one) 0x55fef4ee4637
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/plugin.c:366 (plugin_read_json) 0x55fef4ee4764
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55fef4f38c7a
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55fef4f397f7
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55fef4f39835
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55fef4f3b88a
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0x55fef4ec0afe
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: lightningd/lightningd.c:834 (main) 0x55fef4ec6f5a
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0x7f77623e4b96
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0x55fef4ead909
2019-08-15T00:19:49.640Z **BROKEN** lightningd(17070): backtrace: (null):0 ((null)) 0xffffffffffffffff
```
2019-08-15 02:24:18 +00:00
Rusty Russell c43d09dfd3 pytest: two more slow tests I missed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 02:22:01 +00:00
Rusty Russell 25ddb80823 ccan: update to fix shutdown slowness after plugin load.
Dynamic plugins were keeping fds open; they should not have these
at all anyway, but worse, they interfere with operation because
we don't notice they're closed.

The symptom was that shutdown of the test_plugin_slowinit and
test_plugin_command was 30 seconds (10 seconds grace to kill each daemon).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-15 02:22:01 +00:00
Rusty Russell bf3b77a947 Travis: skip testing VALGRIND=1 DEVELOPER=0, remove the slowest non-developer tests.
I don't remember ever seeing a bug which only showed up in VALGRIND=1 with developer
mode disabled, so don't test that, and spread out the other test more evenly.

In addition, disable the worst-performing tests in DEVELOPER=0 mode.

Here timings from my build machine: the worst 6 (- DEVELOPER=0 VALGRIND=0)
with the same tests (+ DEVELOPER=1 VALGRIND=1)

-452.42s call     tests/test_pay.py::test_channel_spendable
+87.69s call     tests/test_pay.py::test_channel_spendable
-335.66s call     tests/test_gossip.py::test_gossip_store_compact_on_load
+47.41s call     tests/test_gossip.py::test_gossip_store_compact_on_load
-332.07s call     tests/test_connection.py::test_opening_tiny_channel
+89.71s call     tests/test_connection.py::test_opening_tiny_channel
-331.97s call     tests/test_pay.py::test_channel_spendable_large
+56.23s call     tests/test_pay.py::test_channel_spendable_large
-305.28s call     tests/test_invoices.py::test_invoice_routeboost
+37.57s call     tests/test_invoices.py::test_invoice_routeboost
-284.28s call     tests/test_plugin.py::test_htlc_accepted_hook_forward_restart
+49.12s call     tests/test_plugin.py::test_htlc_accepted_hook_forward_restart

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 11:14:38 +00:00
Rusty Russell 63103db3f3 pytest: disable some v. slow tests when !DEVELOPER.
We've gone over 50 minutes with valgrind, and my measurements show
these are the slowest.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 05:53:36 +00:00
Rusty Russell a7a9f1c9d3 onchaind: fix bug when grinding feerates with multiple possibilities.
Fixes: #2820
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 05:53:36 +00:00
Rusty Russell 9424477cb5 onchaind/test: new test, for grinding feerate when multiple possibilities.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 05:53:36 +00:00
Rusty Russell 0f5036d866 cli: remove formatting hint correclty when it's not the last element.
I noticed that the 'lightning-cli summary' output was wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-13 20:45:45 +00:00
Rusty Russell 69b02e287a cli: avoid use-after-realloc when we delete format hint.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-13 20:45:45 +00:00
Rusty Russell f1e84b3d99 devtools: make clean should remove devtools/gen_print_onion_wire.[c,h,o]
Reported-by: @JavierRSobrino
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-13 05:12:10 +00:00
nicolas.dorier 6cf42fe6c1 Add python-mako to the arm32 build 2019-08-12 05:44:11 +00:00
Rusty Russell 3fec154413
CHANGELOG.md: shuffle and format into order, label rc1
(Squashed in fixup by Darosior <darosior@protonmail.com>)
2019-08-11 22:54:25 +00:00
Cryptcoin Junkey bdf1ffcb4f Enable to build the docker image (ElementsProject#2941).
Signed-off-by: Cryptcoin Junkey <cryptcoin.junkey@gmail.com>
2019-08-11 18:21:03 +02:00
nicolas.dorier 19cf199f0e Add docker ARM64 support 2019-08-11 18:19:22 +02:00
Rusty Russell 854e3c5303 pytest: wait for sync by default when starting lightningd.
Otherwise we get some spurious test failures.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell ca28c30eff funding: don't allow funding new channels until we're synced.
This is probably worth preventing.

1. Our depth estimate would be inaccurate possibly leading to us
   timing out too early.
2. If we're not up-to-date our onchain funds are unknown.
3. We wouldn't be able to send or receive HTLCs until we're synced anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell c3a35416da lightningd: don't allow channeld to accept HTLCs if we're not synced.
We want to still allow incoming connections, and reestablishment of
channels, but if one tries to give us an HTLC, stall until we're
synced.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell 6195a878f7 lightningd: don't allow sending of HTLCs while still syncing.
If we don't know block height, we shouldn't be sending HTLCs.  This
stops us forwarding HTLCs as well as new payments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell 3eebd0cc20 lightningd: add flag for whether we're synced, and callback infrastructure.
We consider ourselves synced when bitcoind is synced and we're synced
with that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell faded9a9cf bitcoind: detect when it's still syncing, add field to getinfo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell be8ebf2667 pytest: fix btcproxy mock logic.
You're supposed to be able to hand mock_rpc either a function to call,
or a dict canned response.  We never did the latter, and the logic
was broken.

It was testing the key, not the value for whether it was a dict.  And
it could never have given a valid response anyway, since it wouldn't
know the id to use.  So assume it's a successful result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell 7a592a2b5c contrib/startup_regtest.sh: avoid getting stuck in initialblockdownload.
And give a hint as to what cmds are available, since I forget!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00