Commit Graph

4664 Commits

Author SHA1 Message Date
Rusty Russell ea4308ce52 channeld: allow WIRE_ANNOUNCEMENT_SIGNATURES before funding_locked.
LND does this, and we get upset with it.  I had assumed we would only
do this after funding_locked (since we don't consider the channel
shortid stable until that point), but TBH 6 confirms is probably
enough.

Fixes: #1985
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-13 04:13:35 +00:00
trueserve dbb9b0c077 options: Parse config file before opening working directory
Right now, the `config` file is read *after* the configuration working directory is moved to in the software. However one configuration option `lightning-dir` settable in the `config` file sets this working directory. As the directory is already opened (which defaults to `$HOME/.lightning`) before the configuration is read, the configured directory will not be used.

This patch parses the configuration file before opening the working directory, fixing this bug.

[ Update CHANGELOG.md and man pages -- RR ]
2018-10-13 04:09:49 +00:00
Rusty Russell f6b0f794b2 jsonrpc: fix crash on bad JSON input.
It went something like:
   niftynei: Hey, cppcheck complains this might be NULL, so I put in a check.
   rusty: cppcheck is dumb.  Make it an assert("Rusty always right!").
   niftynei: You seem certain of this so I shall do that.
             https://github.com/ElementsProject/lightning/pull/1994
   ...
   renepickhardt: I asked fiatjaf to run
                  `lightning-cli sendpay "[{'id':'02db8f487fcc0a'}]" 4efe0ba89b`
                  and his node crashed!
   rusty: grep Assertion logs/*
          lightningd/jsonrpc.c:326: connection_complete_error: Assertion `Rusty is always right!' failed.

It turns out that in the 'can't parse' error case, we hand NULL cmd to
connection_compete_error.

Next time, less asserting, more grepping!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-12 21:19:52 +00:00
Simon Vrouwe c22d7012a3 lightningd: fail fundchannel command when feerate is below feerate_floor 2018-10-12 02:14:19 +00:00
Christian Decker 0ee0da0eb8 common: Add a test for io_lock
Totally forgot to add this test. It just shows how a writer can take
exclusive access of a socket over multiple `io_write` calls, and
queuing all others behind it.
2018-10-11 01:43:55 +00:00
Christian Decker b86edf3cf1 jsonrpc: Simple demonstration on how jcon can be locked for streams
This is a bit of overkill now that we simply accumulate the entire
JSON response in the buffer before flushing, but when we move to
streamed responses it allows us to have a single command that has
exclusive access to the out direction of the JSON-RPC connection.
2018-10-11 01:43:55 +00:00
Christian Decker bb38541d9e common: Added a small locking mechanism to simplify how we lock IO
We've done this a number of times already where we're getting
exclusive access to either the out direction of a connection, or we
try to lock out the read side while we are responding to a previous
request. They usually are really cumbersome because we reach around to
the other direction to stop it from proceeding, or we flag our
exclusive access somewhere, and we always need to know whom to notify.

PR ElementsProject/lightning#1970 adds two new instances of this:

 - Streaming a JSON response requires that nothing else should write
   while the stream is active.
 - We also want to stop reading new requests while we are responding
   to one.

To remove the complexity of having to know whom to stop and notify
when we're done, this adds a simple `io_lock` primitive that can be
used to get exclusive access to a connection. This inverts the
requirement for notifications, since everybody registers interest in
the lock and they get notified if the lock holder releases it.
2018-10-11 01:43:55 +00:00
practicalswift a46d712154 Avoid applying the unary minus operator to an unsigned value 2018-10-11 01:39:54 +00:00
Rusty Russell 1e467bb986 lightningd: fail HTLCs which are in-transit as we shut down.
This is the source of failure in the test_restart_many_payments stress
test: we don't commit the outgoing HTLC immediately, instead waiting for
gossip to tell us the peer for the outgoing channel, then waiting for
that channeld to tell is it's committed.  The result was incoming HTLCs
with no outgoing.

I initially pushed the HTLCs through that same path, but of course
(since peers are not connected yet!) the only result was that we failed
these HTLCs immediately.  So I chose the far simpler course of just
failing them directly.

To reproduce this, I had to increase the test_restart_many_payments
num to 10, and run it with nice -20 taskset -c 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-10 18:14:13 +02:00
William Casarin b3f7236201 python: add cheroot to requirements.txt
It looks like this is a new dependency.

Sort the list while we're at it.

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-10 18:00:22 +02:00
Rusty Russell fcb5310873 pytest: make wait_for do exponential backoff, start at 0.25 seconds.
This doesn't alter runtime very much, but does reduce log spam.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-10 06:10:42 +00:00
Rusty Russell 84b9e3e72b lightningd: reduce log spam from bitcoin-cli invocations.
During tests, this is half our log!  And Travis truncates it if we get
a failure in test_restart_many_payments.

Interestingly, test_logging had a bug which relied on this spam :)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-10 06:10:42 +00:00
William Casarin 8f405ca9a7 cli: fix human help output for the new jsonrpc help response
Now that we're returning all the help data, we need to update the human_help
formatter to handle the extra data.

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-10 06:09:29 +00:00
William Casarin d23a0e8adc cli: handle missing manpages gracefully
Instead of exiting when we can't find a manpage, set the command and continue so
that we can try the json rpc for help.

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-10 06:09:29 +00:00
William Casarin cc4357f0a6 rpc: add json_add_help_command for help command objects
Instead of two code paths that return different help objects, simplify things by
always returning the full help object. This not only includes description and
the command name, but the verbose description as well.

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-10 06:09:29 +00:00
lisa neigut 762c795c9b gossip: reject channel_update with invalid `htlc_max_msat`
If the channel update signals an invalid `htlc_maximum_msat` value,
we ignore the update.
2018-10-09 23:22:52 +00:00
lisa neigut 1b6bd3fded wire: add test for parsing optional version of channel_update 2018-10-09 23:22:52 +00:00
lisa neigut a289282bad gossipd: use u64 for `htlc_minimum_msat` field
It's u64 in the spec, so we should use u64 too.
2018-10-09 23:22:52 +00:00
lisa neigut b9331e5ac8 gossipd: parse and respect optional `htlc_maximum_msat`
If another channel has set the optional `htlc_maximum_msat` field,
we should correctly parse that field and respect it when drawing up
routes for payments.
2018-10-09 23:22:52 +00:00
Rusty Russell a3e87af19e CHANGELOG.md: update for flurry of HTLC fixes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell a4730089e8 pytest: make test_restart_many_payments more Pythony!
IIUC, namedtuple is like tuple for grown-ups: Pythonify!

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 0226ef0572 htlc: rename local flag to am_origin, add FIXME.
Noted by @cdecker, the term 'local' is grossly overused, and the hout
preimage is basically only used as a sanity check (though I've just put
a FIXME there for now).

Also eliminated spurious blank line which crept into wallet.c.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 6c96bcacd7 lightningd: fix inconsistency without COMPAT enabled.
We don't expect payment or payment->route_channels to be NULL without an
old db, but putting an assert there reveals that we try to fail an HTLC
which has already succeeded in 'test_onchain_unwatch'.

Obviously we only want to fail an HTLC which goes onchain if we don't
already have the preimage!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 4146950496 lightningd: don't access htlc_in's failoutchannel on db restore.
failoutchannel tells us which channel to send an update for (specifically
for temporary_channel_failure); but we don't save it into the db.  It's
not even clear we should, since it's a corner case and the channel might
not even exist when we come back.

So on db restore, change such errors to WIRE_TEMPORARY_NODE_FAILURE
which doesn't need an update.

We also don't memset it to 0 in the normal case (we only access if it
failcode has the UPDATE bit set) so valgrind will trigger if we're
wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 79ebb8a92e db: save the failcode / failuremsg into db.
Now we can finally move the fixup code under COMPAT_V061, so it's only
for old nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 2293f4ce9e db: restore failuremsg/failcode from db.
We don't write it in there yet, so this change currently has no effect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell d85251ac6c db: fix up HTLCs which are missing failure information.
We don't save them to the database, so fix things up as we load them.

Next patch will actually save them into the db, and this will become
COMPAT code.

Also: call htlc_in_check() with NULL on db load, as otherwise it aborts
internally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 77be009354 pytest: add restart-during-n-way payment test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 1526f024fb channeld: disable check for my_current_per_commitment_point.
Under stress, it fails (test_restart_many_payments, the next test).

I suspect a deep misunderstanding in the comparison code, will chase
separately.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell ec74aadce2 lightningd: save outgoing HTLC's preimage to db.
We can now wrap the 'missing preimage' hack in COMPAT_V061.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 65f6813706 lightningd: handle the case where the db contains a resolved HTLC without a preimage.
We need to handle this case (old db) before the next commit, which actually
fixes it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell c94ab7370c pytest: extend the test_fulfill_incoming_first case to cover reconnect.
Which we don't handle, due to a separate bug, so it's xfail.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 9ef67e50ff lightningd: don't leave htlc_out's in pointer dangling when htlc_in freed.
Now we know this can happen (see previous patch), we need to handle it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 4040c53258 lightningd: handle case where incoming HTLC vanished before fulfilled outgoing.
We now need an explicit 'local' flag, rather than relying on the existence
of the 'in' pointer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 3643e1bd90 pytest: add test for when an incoming fulfilled HTLC expires before outgoing.
Usually, we only close an incoming HTLC once the outgoing HTLC is completely
resolved.  However, we short-cut this in the FULFILL case: we have the
preimage, so might as well use it immediately (in fact, we wait for it to
be committed, but we don't need to in theory).

As a side-effect of this, our assumption that every outgoing HTLC has
a corresponding incoming HTLC is incorrect, and this test (xfail) tickles
that corner case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell b77906634e lightningd: even more HTLC consistency checking: check states.
This means we need to check when we've altered the state, so the checks
are moved to the callers of htlc_in_update_state and htlc_out_update_state.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell a516e26331 lightningd: beef up HTLC consistency checking.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 123713fbf5 lightningd: fix outstanding taken pointer.
lightningd: Outstanding taken pointers: lightningd/pay.c:243:channel_update

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 0b1870908a Makefile: our first db-from-0.6.1 bugfix is coming.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell b3fb23f921 pytest: make line_graph wait for gossip propagation if announce param is True
This is what the callers want; generalize it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell aa9e15edb0 options: don't end default name with '-' in non-DEVELOPER mode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 08:40:52 +00:00
Rusty Russell de37586a97 gossipd: use riskfactor in getroute, not "1".
AFAICT, this was there in the original commit by @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 08:40:52 +00:00
Rusty Russell d946e965a6 gossipd: test that fromwire from lightningd messages succeeds.
Also tiny drive-by cleanup for gossip_disable_local_channels to modern form.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 08:40:52 +00:00
Rusty Russell 864812019f gossipd: use tal_arr_expand instead of open-coding it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 08:40:52 +00:00
Rusty Russell 915ffe35ed gossipd: clean up getnodes handling.
globalfeatures should not be accessed if we haven't received a
channel_update.  Treat it like the other fields which are only
initialized and marshalled/unmarshalled if the timestamp is positive.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 08:40:52 +00:00
Rusty Russell df27fc55af More renaming of gfeatures to globalfeatures.
Use the BOLT #1 naming.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 08:40:52 +00:00
William Casarin f2488fbe14 make: fix broken error message in bolt-check
It looks like a printf was missing for the bolt-check error message

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-09 08:39:53 +00:00
William Casarin 7c4b9c8a0d build: fix compile error on gcc ~7.3.0
It seems to be having a bit of trouble understanding the control flow to realize
it's not actually uninitialized.

Add an error handler after the switch in case we miss a real uninitialized error
in the future.

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-09 05:56:03 +00:00
lisa neigut 45b07e7e89 jsonrpc: fix cppcheck warning for potential null pointer deref
cppcheck was failing with a warning for a null pointer deref at this
line, this makes it stop complaining (and lets make check-source
succeed)
2018-10-09 00:08:41 +02:00
lisa neigut 0ecbdf3421 makefile: don't print shellcheck to stdout 2018-10-09 00:08:41 +02:00