Commit Graph

6350 Commits

Author SHA1 Message Date
nicolas.dorier 19cf199f0e Add docker ARM64 support 2019-08-11 18:19:22 +02:00
Rusty Russell 854e3c5303 pytest: wait for sync by default when starting lightningd.
Otherwise we get some spurious test failures.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell ca28c30eff funding: don't allow funding new channels until we're synced.
This is probably worth preventing.

1. Our depth estimate would be inaccurate possibly leading to us
   timing out too early.
2. If we're not up-to-date our onchain funds are unknown.
3. We wouldn't be able to send or receive HTLCs until we're synced anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell c3a35416da lightningd: don't allow channeld to accept HTLCs if we're not synced.
We want to still allow incoming connections, and reestablishment of
channels, but if one tries to give us an HTLC, stall until we're
synced.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell 6195a878f7 lightningd: don't allow sending of HTLCs while still syncing.
If we don't know block height, we shouldn't be sending HTLCs.  This
stops us forwarding HTLCs as well as new payments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell 3eebd0cc20 lightningd: add flag for whether we're synced, and callback infrastructure.
We consider ourselves synced when bitcoind is synced and we're synced
with that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell faded9a9cf bitcoind: detect when it's still syncing, add field to getinfo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell be8ebf2667 pytest: fix btcproxy mock logic.
You're supposed to be able to hand mock_rpc either a function to call,
or a dict canned response.  We never did the latter, and the logic
was broken.

It was testing the key, not the value for whether it was a dict.  And
it could never have given a valid response anyway, since it wouldn't
know the id to use.  So assume it's a successful result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell 7a592a2b5c contrib/startup_regtest.sh: avoid getting stuck in initialblockdownload.
And give a hint as to what cmds are available, since I forget!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00
Rusty Russell 2a09124574 contrib/startup_regtest.sh: set sane defaults for lightning & bitcoin dirs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00
Rusty Russell 97b39d84d5 README.md: more rewords.
0. Add an index
1. Mention regtest early on.
2. Link to doc/INSTALL.md for build-it-yourself.
   Since they're reading this doc, they're probably already on GH so don't discourage it.
3. Talk about contrib/startup_regtest.sh.
4. Use mainnet in the examples, since that's what people probably want.
5. Refer to default .lightning dir, and manpage.
6. Trim the JSON-API section down to simple commands.
7. Add section on getting started: bootstrap script and plugins.
8. Less details on config files, just refer to the man page.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:00:49 +02:00
Rusty Russell 125f14a8d5 pytest: fix flaky "peer reconnected" message in test_option_upfront_shutdown_script
If l2 doesn't think we're onchain yet, it treats the new connection from l1
as a reconnection, triggering 'ValueError: 1 nodes had unexpected reconnections'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 18:56:35 +02:00
Rusty Russell 4274b9f0af lightingd: increase listen queue on rpc socket.
I suspect multiple plugins trying to connect at the same
time are overrunning the 1-deep listen queue:

From man listen(2):

       The backlog argument defines the maximum length to which the  queue  of
       pending  connections  for sockfd may grow.  If a connection request ar‐
       rives when the queue is full, the client may receive an error  with  an
       indication  of ECONNREFUSED

Fixes: #2922
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 10:03:07 +00:00
lisa neigut 0c96c89d67 db-fix: resolve crash on fundchannel
Fixes error introduced by 1dbdc74bc where a new fundchannel
can cause a crash after start if the max dbid is for a closed
channel.
2019-08-10 02:52:13 +00:00
Rusty Russell 5e78960be0 CHANGELOG: note the support for #557.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell 51541f53d8 gossipd: test vectors for https://github.com/lightningnetwork/lightning-rfc/pull/557
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell 4de47f6db5 gossipd: use default zlib compression, hack for zlib expansion.
These both allow us to reproduce the test vectors in the next patch.  But
using Z_DEFAULT_COMPRESSION is a reasonable idea anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell f6cf4bf62a spec: remove encoding byte from checksums.
Make the TLV element a simple array.  This is a bit neater, in fact, and
makes the test vectors in that 557 PR work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell 8abd850d3c gossipd: append timestamps & checksums to reply_channel_range if asked (EXPERIMENTAL)
In fact, we always generate them, we only send them if asked.  And we set
the flags to 0 if not --enable-experimental-features, so we never send in
that case.

Generating checksums involves pulling the channel_update from the
gossip_store, which is suboptimal: there's a FIXME to store the
checksum in memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell c7853197ae gossipd: generalize encoding functions
We're about to use the for gossip extended info too, which *don't* put
the encoding byte at the beginning of the data stream.  So this removes
some "scids" from function names and separates out the "prepend a byte"
case from the "external encoding_type" case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell 0de11da5e4 gossipd: decode and obey query_short_channel_ids's TLV query_flags (EXPERIMENTAL)
These indicate what fields we are to return.  If there's now TLV, or we
haven't got --enable-experimental-features, it's set to all 1s so behaviour
is unchanged.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell e7728c0eea common: implement decoding of query-flags for query_short_channel_ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell d2030539e1 EXPERIMENTAL: pull in PR 557 (with minor fixes): range query support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
darosior b966e5e783 Add a test for 'fundchannel_start' crash on deconnection 2019-08-10 00:04:01 +08:00
Christian Decker cc5f2dcae3 readme: First pass at homogenizing the readme a bit
We haven't touched the readme for quite some time, just randomly added to it,
and it's starting to show. This is my attempt at cleaning it up a bit (more to
come):

 - No longer discourage users from running on mainnet, we're way beyond that
   point.
 - No longer instruct users to build from source, when we have real binary
   releases, on the PPA, the releases page and the docker images.
 - Cut down on the docker specific instructions, they are taking a lot of room
   when only a minority will likely run them that way
 - Generally make the README more of a dispatch for more in-depth
   documentation rather than trying to address everything right on the
   front-page.
 - Add a bit of context about running on top of a pruned node

Signed-off-by: Christian Decker <decker.christian@gmail.com>


Header from folded patch 'fixup!_readme__first_pass_at_homogenizing_the_readme_a_bit.patch':

fixup! readme: First pass at homogenizing the readme a bit
2019-08-09 10:29:37 +00:00
Rusty Russell 710d015e5b lightningd: fix crash when peer disconnects after fundchannel_start, before cancel/complete
Fixes: #2831
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 10:57:42 +02:00
Rusty Russell f9ecc76d99 gossipd: check that we don't try to access a deleted gossip entry.
We ignored this before, which meant that the DEVELOPER-mode check that we
delete the correct record didn't check that it wasn't already deleted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 08:58:05 +02:00
Rusty Russell 0edc0ae5e9 pytest: don't use deprecated options for close() in tests.
Only downside is you have to wait 1 second at least before
unilaterally closing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell 83e654a106 close: change to a unilateraltimeout argument.
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call.  `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.

Timing out JSON calls is generally deprecated: that's the job of the
client.  And the semantics of this are confusing, even to me!  A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.

The transition code is awkward, but we'll manage for the three
releases until we can remove it.

The new defaults are to unilaterally close after 48 hours.

Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell b35dc4689b pytest: enable deprecated APIs for multi-arg closes.
We're about to change the API, so this makes the tests still work
across the transition (and, as a bonus, tests our backwards compat
shim).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rene Pickhardt dfac1d15a2 included feedback by Rusty to check the max_concurrent_htlc value for both peers of a channel 2019-08-09 05:45:06 +00:00
Rene Pickhardt 1dd890825a doc: Document max-concurrent-htlcs option. 2019-08-09 05:45:06 +00:00
Rene Pickhardt 8e7428da53 Added possibility to configure max_concurrent_htlcs value for our channels. Eclaire has a default of 30 and I thought why not going with their value and while doing so make it configureable. 2019-08-09 05:45:06 +00:00
lisa neigut 0bd95fc068 contrib: add txindex to bitcoind for regtest 2019-08-09 05:07:18 +00:00
lisa neigut 0acdeeec06 fixup mkcommit's reverse ordered chainparams 2019-08-09 05:07:18 +00:00
darosior f98ae2d1d3 Correct version regex in Makefile, add git as dependency 2019-08-09 02:49:45 +00:00
ZmnSCPxj 5dbff077c2 Makefile: Also install manpage for lightningd. 2019-08-09 02:36:59 +00:00
ZmnSCPxj 821832032d doc/lightningd.8.txt: Initial manpage for lightningd itself.
Closes: #1360
2019-08-09 02:36:59 +00:00
Rusty Russell 69b7ef1508 lightningd: fix up typesafe-cb bitcoind_getfilteredblock
`const struct filteredblock *` everywhere, as the typesafe_cb_preargs
macro required.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 02:31:51 +00:00
Christian Decker 5e36257375 pytest: Test that we delete channels opened below our birth height
This was the initial issue that was addressed by #2756 and now we just test
that all is working as expected.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 674017fa1e changelog: Add note on the block backfilling and cleanup fix
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Rusty Russell 99236f86f9 lightningd: rename 'satoshis' to 'amount' to avoid confusing check-source.
The type is enough (it's a struct amount_sat) to avoid confusion with
btc or msats.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 02:31:51 +00:00
Christian Decker 379079c5f3 gossip: Only backfill blocks that are below our birth height
If we were to just insert filtered blocks in the range that we will scan later
we'd be hitting the uniqueness constraints later.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 95d891ebf1 bitcoind: Queue up calls to getfilteredblock and dispatch results
Instead of allowing all calls to `getfilteredblock` to be scheduled on the
`bitcoind` queue right away we instead add them in a separate queue, and
process a single call at a time. This limits the concurrency and avoids
thrashing `bitcoind`. At the  same time we dispatch incoming results back to
all calls that were queued for that particular blockheight, reducing the
overall number of calls and an increase in overall speed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 040cda2875 bitcoind: Initialize filteredblock->outpoints with filteredblock
We will be calling the callback out of order once we fan out the results of a
single lookip to multiple calls, so being sure that everything is allocated
ahead of time is necessary.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 43b3eea783 bitcoind: Don't log when a transaction output is detected as spent
Since we now check all P2WSH outputs in a block, this is getting quite a
common occurence, so logging just produces lots of noise.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 98447e454e gossip: Use the getfilteredblock method to look up scid outputs
Just a tiny shim to reconcile the `get_output` with `getfileteredblock`.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 3dbaae38e3 wallet: Add function to add filteredblocks from backfilling
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker f4e434d8e1 bitcoind: Add a multi-step getfilteredblock method
This will eventually replace the multi-step `getblockhash` + `getblock` +
`gettxout` mechanism, and return entire filtered blocks which can be added to
the DB, and represent the full set of P2WSH UTXOs.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker 5bb411b572 pytest: Add test for the block backfilling trigger by gossip
This is just the test that we use to verify block backfilling below the wallet
birth height is working correctly.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00