Commit Graph

5934 Commits

Author SHA1 Message Date
Rusty Russell 0c189fe3a7 devtools/decode-iolog: tool to decode hexstrings from io logging.
Slow, but useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-31 18:36:38 +02:00
Rusty Russell 695bec531c plugin: fix printing of bad plugin responses.
Before:
	Plugin for invoice_payment returned non-result response 

	"subscriptions": [], "hooks": ["invoice_payment"]}}

	�V

After:
	Plugin for invoice_payment returned non-result response {"jsonrpc": "2.0", "id": 6, "error": "Error while processing invoice_payment: ValueError(\"invalid literal for int() with base 10: '5.0'\")"}

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-31 18:36:38 +02:00
Rusty Russell 8f9c48254b plugins: do I/O logging.
I was trying to trace a problem with a plugin, and needed this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-31 18:36:38 +02:00
ZmnSCPxj 48df6c8566 lightningd/io_loop_with_timers.c: Move mainloop to its own source file, have chaintopology use it.
Fixes: #2687
2019-05-31 17:57:10 +02:00
Christian Decker 883d4b4e6e pytest: Add a test that reproduces #2687
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-05-31 17:57:10 +02:00
ZmnSCPxj 3466261b20 lightningd/channel.c: Transfer peer to connectd only if connectd alive.
Fixes: #2677
2019-05-31 15:01:58 +02:00
ZmnSCPxj 37440e9447 lightningd/subd.c: Return NULL from subd_shutdown.
And set pointers to shut down daemons as NULL in lightningd.
2019-05-31 15:01:58 +02:00
trueptolemy 11412435ce Channeld: init channel with remote announcement info when restart
1. Add remote_ann_node_sigs and remote_bitcoin_sigs fields in channel_init message;
2. Master add announcement signatures into channel_init message, and send this message to Channeld.
Channeld will initial the channel with this signatures when it reenables the channel.
2019-05-29 11:46:33 +02:00
trueptolemy d93f61407a Channeld: Add new wire type:channel_got_announcement
Channeld sends announcement signatures to Master by this message.
When Channeld receive a new channel announcement msg, (After channel locking)it will sends announcement signatures to Master by this message.
2019-05-29 11:46:33 +02:00
trueptolemy a645fbdecd run-wallet: Add the check of case that channel received ann_sigs 2019-05-29 11:46:33 +02:00
trueptolemy b41d529b28 DB: Store the remote channel announcement signatures into DB
1. Add the fields of remote channel announcement signatures into TABLE channels
2. Add the related function
2019-05-29 11:46:33 +02:00
Rene Pickhardt 22600faed7 manually added a list-forward manpage because I have no idea how a2x works 2019-05-27 17:35:06 +02:00
Simon Vrouwe 52c84b2387 pytest: add test_funding_reorg_remote_lags
Nodes may disagree about short_channel_id before channel
announcement.
2019-05-27 13:29:32 +02:00
Simon Vrouwe 5b0c174251 pytest: add test_funding_reorg_private, reorg changes funding height
Reorg changes short_channel_id after lockin of private channel, while
one node restarts.

test that:
- peer->depth_togo in billboard decrements
- reorg and scid change is detected by running node and restarting node
- both `old` and `new` scids are in rtable

Also added a comment to test_blockchaintrack to clarify.
2019-05-27 13:29:32 +02:00
Simon Vrouwe 54a67c2444 channeld: rephrase depth_togo line in billboard_update 2019-05-27 13:29:32 +02:00
Simon Vrouwe 8c1bbf33e5 lightningd: funding_lockin_cb, handle reorgs that change short_channel_id
Keep watching and updating scid until ANNOUNCE_MIN_DEPTH, even when channel is private.
When scid changes, we fail channeld so it will restart and initialize with updated
scid and add it to rtable. Reorgs can change funding tx's height/index after lockin,
which could happen with small minimum_depth=1.
2019-05-27 13:29:32 +02:00
Simon Vrouwe 7726681aa6 pytest: added BitcoinD.simple_reorg() method, which can change tx height and/or txindex
Now without bitcoind restart.
bitcoin-cli `prioritisetransaction` came to the rescue!
Its argument `fee_delta` (apparently) lowers the txs _effective_ feerate
soo low that bitcoind wont mine it ... untill we raise it when we want
it to be mined.
2019-05-27 13:29:32 +02:00
Simon Vrouwe 24f43e5910 pytest: fundwallet tighten up, make sure we see `CONFIRMED` in log line 2019-05-27 13:29:32 +02:00
Simon Vrouwe 245807a085 pytest: tighten a log test `Owning output...` with address type and confirmation
Because the call (wallet_extract_owned outputs) that prints that line can happen
_before_ or _after_ confirmation in block, adding `CONFIRMED` in the later.
2019-05-27 13:29:32 +02:00
Simon Vrouwe eb3495c23d chaintopology: add log line when we remove stale block from topo->tip
Added comments
2019-05-27 13:29:32 +02:00
Simon Vrouwe b261c5ba0b chaintopology: add REORG to logline when tx watch is fired after reorg
This makes clear that txwatch_fire was called with depth=0 when
chain tip got removed after a reorg. And now we can test for that.
2019-05-27 13:29:32 +02:00
Simon Vrouwe db57d9c5d2 lightningd: suppress IO_OUT logging for getlog command
Before this, the response of `getlog io` blew up quickly
when called multiple times.
2019-05-26 23:53:20 +00:00
Christian Decker 8feb05aef4 subd: Remove overly verbose log when receiving a message
These tend to spam the logs, so I removed them :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-05-26 16:19:10 +02:00
Christian Decker ced2ff8aa7 pytest: Mark repeat offenders as flaky
It's always the same few tests that fail, so I'm marking them as flaky and
will do a cleanup round later to individually address them.

I've been tracking the failures on the master branch for a few months now and
these are the worst offenders:

|-----------------------------------------|--------|--------|-------|
| test_name                               | state  | branch | count |
|-----------------------------------------|--------|--------|-------|
| test_pay_direct                         | FAILED | master | 31    |
| test_reconnect_gossiping                | FAILED | master | 20    |
| test_htlc_send_timeout                  | FAILED | master | 15    |
| test_pay_limits                         | FAILED | master | 13    |
| test_permfail                           | FAILED | master | 13    |
| test_funding_reorg_private              | FAILED | master | 12    |
| test_invoice                            | FAILED | master | 12    |
| test_invoice_preimage                   | FAILED | master | 12    |
| test_shutdown                           | FAILED | master | 12    |
| test_withdraw                           | FAILED | master | 12    |
| test_gossip_store_load_v3               | FAILED | master | 11    |
| test_onchain_multihtlc_their_unilateral | FAILED | master | 10    |
| test_opening_tiny_channel               | FAILED | master | 10    |
| test_channel_reenable                   | FAILED | master | 9     |
| test_crashlog                           | FAILED | master | 9     |
| test_gossip_weirdalias                  | FAILED | master | 9     |
| test_invoice_expiry                     | FAILED | master | 9     |
| test_onchain_multihtlc_our_unilateral   | FAILED | master | 9     |
| test_peerinfo                           | FAILED | master | 9     |
| test_private_channel                    | FAILED | master | 9     |
| test_shutdown_reconnect                 | FAILED | master | 9     |
| test_closing                            | FAILED | master | 8     |
| test_closing_different_fees             | FAILED | master | 8     |
| test_closing_while_disconnected         | FAILED | master | 8     |
| test_gossip_timestamp_filter            | FAILED | master | 8     |
| test_gossipwith                         | FAILED | master | 8     |
| test_invoice_routeboost                 | FAILED | master | 8     |
| test_onchain_middleman                  | FAILED | master | 8     |
| test_plugin_notifications               | FAILED | master | 8     |
| test_reconnect_channel_peers            | FAILED | master | 8     |
| test_bad_onion                          | FAILED | master | 7     |
| test_closing_torture                    | FAILED | master | 7     |
| test_fulfill_incoming_first             | FAILED | master | 7     |
| test_funding_reorg_disagree_scid_route  | FAILED | master | 7     |
| test_pay_disconnect                     | FAILED | master | 7     |
| test_balance                            | FAILED | master | 6     |
| test_check_command                      | FAILED | master | 6     |
| test_closing_id                         | FAILED | master | 6     |
| test_gossip_addresses                   | FAILED | master | 6     |
| test_gossip_notices_close               | FAILED | master | 6     |
| test_setchannelfee_usage                | FAILED | master | 6     |
| test_announce_address                   | FAILED | master | 5     |
| test_connect_by_gossip                  | FAILED | master | 5     |
| test_gossip_badsig                      | FAILED | master | 5     |
| test_gossip_disable_channels            | FAILED | master | 5     |
| test_gossip_jsonrpc                     | FAILED | master | 5     |
| test_gossip_persistence                 | FAILED | master | 5     |
| test_gossip_pruning                     | FAILED | master | 5     |
| test_onchain_different_fees             | FAILED | master | 5     |
| test_onchain_dust_out                   | FAILED | master | 5     |
|-----------------------------------------|--------|--------|-------|
2019-05-26 16:19:10 +02:00
Christian Decker 56f7efad46 pytest: Stabilize test_reconnect_gossiping
We weren't waiting for l2 to register the peer before asking it to ping it.
2019-05-26 16:19:10 +02:00
Christian Decker a0fc915674 pytest: Start the Bitcoin RPC proxy as soon as it is created
Avoids attempting to rebind on restart.
2019-05-26 16:19:10 +02:00
Jan Sarenik 180ea5bb38 test_closing.py: isAlive -> is_alive
isAlive() is deprecated, use is_alive() instead
2019-05-26 15:07:33 +02:00
Kristaps Kaupe 441c571650 Better describe "urgent", "normal" and "slow" feerates 2019-05-25 23:48:37 +00:00
ZmnSCPxj bbec8f62e0 doc/lightning-pay.7.txt: Remove references to value randomization, since it is no longer implemented, and describe shadow routes.
Fixes: #2659

Value randomization is not implemented since `pay` was made into a plugin.
Nobody has been asking for it, so do not implement it and just remove
from documentation.

Shadow routes is implemented, and not describe, so describe it now.
2019-05-24 19:08:19 +02:00
ZmnSCPxj edfb1ace4a wallet/db.c: Be resilient against 32-bit time_t.
Fixes: #2621
2019-05-24 16:07:33 +02:00
lisa neigut ccc54dd64d jsontok: fixup spelling error in code comment 2019-05-24 16:06:35 +02:00
lisa neigut be502a095d devscript: always clean up pid files for lightning daemons 2019-05-24 16:06:35 +02:00
ZmnSCPxj 8ecc6c7743 lightningd/plugin.c: Do not call `closedir` with NULL DIR*.
Fixes: #2667

Really-by: @darwin
2019-05-23 17:45:57 -07:00
ZmnSCPxj 09761cd4c9 plugins/.gitignore: Also ignore autoclean. 2019-05-23 04:51:06 +00:00
ZmnSCPxj a80e5c7212 plugins/Makefile: Add PLUGIN_AUTOCLEAN_OBJS to ALL_OBJS.
This adds externals as dependencies of autoclean compilation.
2019-05-23 04:51:06 +00:00
ZmnSCPxj 30ad98a3ea plugins/autoclean.c: Remove unnecessary inclusions. 2019-05-23 03:02:05 +00:00
Michael Schmoock 0efd72d882 test: pylightning to_approx_str 2019-05-22 22:13:06 +02:00
Michael Schmoock d8599e5f67 pylightning: adds msat.to_approx_str() method
Tries to return the approxmost posible string of a Millisatoshi amount using
various unit representations. The function will round to an effective
number of digits. Default: 3.

```
>>> Millisatoshi("100000sat").to_approx_str()
'0.001btc'
>>> Millisatoshi("100msat").to_approx_str()
'0.1sat'
>>> Millisatoshi("10000000sat").to_approx_str()
'0.1btc'
```
2019-05-22 22:13:06 +02:00
Rusty Russell 654e89b5fc gossipd: free channels in routing_state destructor.
Cleans up the tests.

Suggested-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell d1f43d993a gossipd: use explicit destructor for struct chan.
Each destructor2 costs 40 bytes, and struct chan is only 120 bytes.  So
this drops our memory usage quite a bit:

MCP bench results change:
   -vsz_kb:580004-580016(580006+/-4.8)
   +vsz_kb:533148

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell 59e75f1b2c gossipd: reply to large listchannels in parts.
This has two effects: most importantly, it avoids the problem where
lightningd creates a 800MB JSON blob in response to listchannels,
which causes OOM on the Raspberry Pi (our previous max allocation was
832MB).  This is because lightning-cli can start draining the JSON
while we're filling the buffer, so we end up with a max allocation of
68MB.

But despite being less efficient (multiple queries to gossipd), it
actually speeds things up due to the parallelism:

MCP with -O3 -flto before vs after:
-listchannels_sec:8.980000-9.330000(9.206+/-0.14)
+listchannels_sec:7.500000-7.830000(7.656+/-0.11)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell 4ea1d13077 cli: handle OOM by directly streaming output.
We use raw malloc here, again, to handle the failure cases more easily.

I tested it with this hack, then ran the result through `jq --stream '.'`
before and after to make sure it was the same.

    diff --git a/cli/lightning-cli.c b/cli/lightning-cli.c
    index f840c0786..d83555a51 100644
    --- a/cli/lightning-cli.c
    +++ b/cli/lightning-cli.c
    @@ -295,6 +295,14 @@ static void oom_dump(int fd, char *resp, size_t resp_len, size_t off)
     	exit(0);
     }
     
    +static void *xrealloc(void *p, size_t len)
    +{
    +	if (len > 1000000)
    +		return NULL;
    +	return realloc(p, len);
    +}
    +#define realloc xrealloc
    +
     int main(int argc, char *argv[])
     {
     	setup_locale();

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell 0b79538b18 lightningd: hang up on clients if they make us run out of memory.
This happened with the 800M JSON for the MCP listchannels on the raspberry
pi, and tal calls abort() by default.

We switch to raw malloc here; we could override the error hook for
tal, but this is neater since we're doing low-level things anyway,

I tested it manually with this patch:

   diff --git a/lightningd/json_stream.c b/lightningd/json_stream.c
   index cec9f5771..206ba37c0 100644
   --- a/lightningd/json_stream.c
   +++ b/lightningd/json_stream.c
   @@ -43,6 +43,14 @@ static void free_json_stream_membuf(struct json_stream *js)
    	free(membuf_cleanup(&js->outbuf));
    }
    
   +static void *membuf_realloc_hack(struct membuf *mb, void *rawelems,
   +				 size_t newsize)
   +{
   +	if (newsize > 1000000000)
   +		return NULL;
   +	return realloc(rawelems, newsize);
   +}
   +
    struct json_stream *new_json_stream(const tal_t *ctx,
    				    struct command *writer,
    				    struct log *log)
   @@ -53,7 +61,7 @@ struct json_stream *new_json_stream(const tal_t *ctx,
    	js->reader = NULL;
    	/* We don't use tal here, because we handle failure externally (tal
    	 * helpfully aborts with a msg, which is usually right) */
   -	membuf_init(&js->outbuf, malloc(64), 64, membuf_realloc);
   +	membuf_init(&js->outbuf, malloc(64), 64, membuf_realloc_hack);
    	tal_add_destructor(js, free_json_stream_membuf);
    #if DEVELOPER
    	js->wrapping = tal_arr(js, jsmntype_t, 0);

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell cb9c44ef27 gossipd: remove unnecessary dev_unknown_channel_satoshis arg.
We now have a test blockchain for MCP which has the correct channels,
so this is not needed.

Also fix a benchmark script bug where 'mv "$DIR"/log
"$DIR"/log.old.$$' would fail if you log didn't exist from a previous run.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell 85d8848ede gossipd: neaten insert_broadcast a little.
Suggested-by: @cdecker.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell 3d2af2237c pytest: fix test_htlc_send_timeout now pay doesn't return ROUTE_NOT_FOUND.
This brings up an interesting quirk though, in that we report "3
attempts", where we really should have done one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 5506924f2f pay: update documentation.
Reported-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 15dc0a5c18 pay: fix transient status 'failed' in listpays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 18d2506958 pytest: test for transient "failed" status during ongoing payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell 71460ac073 pay: don't say "Could not find a route" unless we never tried a payment.
It's deeply confusing: we say this after exhausting all other routes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00