Commit Graph

822 Commits

Author SHA1 Message Date
Rusty Russell b17c344b71 plugins/pay: retry when routehint fails.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell e3f893444d pytest: test more-than-one-hop route hints.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 2b72ebb3b6 plugins/pay: implement routeboost hints, naive version.
We sanitize the routes: firstly, we assume appending so eliminate the
first hop if the route points straight to us.  Secondly, eliminate empty
hints.  Thirdly, trim overlong hints.

Then we just use the first route hint.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell b787200752 pytest: test that we correctly use routeboost information from bolt11 invoice.
This is implemented in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 802b772cad plugins/pay: use final_cltc from bolt11 invoice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell a8f2f28c72 plugins/pay: implement maxfeepercent, maxdelay and exemptfee.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 139604a618 pytest: test that pay command retries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell d9fa8a3536 plugins/pay: retry on failure in a loop.
We use the 'exclude' option to getroute for successive attempts.  This
is more robust than having gossipd disable for some limited time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 40637d0017 contrib/pylightning: temporarily convert to use plugin/pay for tests.
That this simply pay plugin passes the tests is a poor reflection on our
test cases, really.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell ad1e1bd528 plugins: minimal 'pay' plugin.
I wrote this sync first, then rewrote async, then developed libplugin.
But committing all that just wastes reviewer time, so I present it as
if it was always asnc and using the library helper.

Currently the command it registers is 'pay2', but when it's complete
we'll remove the internal 'pay' and rename it. This does a single
'getroute/sendpay' call.  No retries, no options.

Shockingly, this by itself is almost sufficient to pass our current test
suite with `pay`->`pay2`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-17 13:02:24 +01:00
Rusty Russell 0ba547ee10 gossipd: handle overflowing query properly (avoid slow 100% CPU reports)
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  #5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  #6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  #7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  #8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  #9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  #10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  #11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  #12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  #13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  #14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  #15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  #16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  #17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  #18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  #19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  #20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  #21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  #22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  #23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  #24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  #25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  #26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  #27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  #28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  #29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  #30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  #31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  #32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  #33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  #34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  #35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  #36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 11:34:45 -08:00
Rusty Russell ba41d6e3df pytest: failing test for overflow in query_channel_range
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 11:34:45 -08:00
Rusty Russell 52750f2dcc pytest: tighten the query_channel_range test.
Make the two channels adjacent, and specify exactly the number of
divide-and-conquer steps there are.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 11:34:45 -08:00
Rusty Russell 4c57bf002e pytest: neaten fund_channel scid extraction.
We used to have a bug where decoderawtransaction would fail, fixed in
fedcfd661 (pytest: hand 'True' to decoderawtransaction so it doesn't
get confused.).

So we can remove the fallback decode, and might as well extract the
ugliness into a helper function.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 11:34:45 -08:00
Rusty Russell 9f1f79587e short_channel_id_dir: new primitive for one direction of short_channel_id
Currently only used by gossipd for channel elimination.

Also print them in canonical form (/[01]), so tests need to be
changed.

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 80753bfbd5 Feedback from @niftynei.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell dc2ee9639b listchannels: allow source arg to list channels by their source node.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 5b3abd80b1 pytest: don't skip valgrinding plugins.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 3d016e7249 getroute: allow array of channels to exclude.
The pay plugin will use this, rather than the current "suppress for 90 second" hacks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 1567238dd9 invoice: option to expose/not-expose private channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell fe4a600bc7 routeboost: don't use channels to dead-end nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 547d6ab878 routeboost: expose private channel in invoice iff we have no public ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Christian Decker 659a26ea5a misc: Update short_channel_id representation to use 'x' separators
Reported-by: Alex Bosworth <@alexbosworth>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-01-15 03:50:27 +00:00
Rusty Russell 61420bf79b configure: (mostly) revert b7a56f0531
Plugins are a first-class citizen again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 02:52:13 +00:00
Rusty Russell c3e96e058e bolt: Updated the BOLT specification to unify UNKNOWN_PAYMENT_HASH & INCORRECT_PAYMENT_AMOUNT
This is based on Christian's change, but removes all trace of the old codes.

I've proposed another spec change which removes this code altogether:
	https://github.com/lightningnetwork/lightning-rfc/pull/544

Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Rusty Russell <@rustyrussell>
2019-01-15 02:19:56 +00:00
Christian Decker b7a56f0531 plugin: Gate the plugin subsystem with the --enable-plugins configure flag
Since we are planning to release a bug fix release, and the plugin
subsystem is not yet complete, it is better to make plugin support
opt-in while we continue testing.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-10 17:15:10 -08:00
Rusty Russell 109c6eb3a3 channeld: include proper sha value in BADONION errors.
Fortunately, we can calculate the sha256 ourselves, so the
outgoing channeld doesn't need to tell us.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-08 19:20:28 +01:00
Rusty Russell 8f8783c0e3 pay: correctly blame the *next* node on BADONION error.
The node which sent the error is doing so because the following
one sent WIRE_UPDATE_FAIL_MALFORMED_HTLC.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-08 19:20:28 +01:00
Rusty Russell 5b6c82384a pytest: test that we get a correct result when peer says onion is bad.
Currently fails: we get a garbage error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-08 19:20:28 +01:00
lisa neigut efa38875b2 listpeers: include `private` field in `channels` output
Reveal channel's 'privacy' in `listpeers` output

Suggested-By: @shesek
2019-01-08 02:21:32 +00:00
Christian Decker 25ca49c444 pytest: Add a test for the event subscription and notification
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-30 14:36:02 +01:00
Christian Decker 643480cfd8 pylightning: Migrate the test plugin to use the new wrapper
Should be a lot easier to see what happens :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-15 15:04:32 +01:00
Rusty Russell a8e0e1709a channeld: fix fee calculation.
Funder can't spend the fee it needs to pay for the commitment transaction:
we were not converting to millisatoshis, however!

This breaks our routeboost test, which no longer has sufficient funds
to make payment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-15 13:50:15 +01:00
Christian Decker a304db9be2 plugin: Handle log notifications from plugins
Logs are parsed and injected into the main daemon's logs.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-13 02:36:43 +00:00
Rusty Russell 6aa511fa7a channeld: only enable option_data_loss_protect if EXPERIMENTAL_FEATURES.
We have an incompatibility with lnd it seems: I've lost channels on
reconnect with 'sync error'.  Since I never got this code to be reliable,
disable it for next release since I suspect it's our fault :(

And reenable the check which didn't work, for others to untangle.

I couldn't get option_data_loss_protect to be reliable, and I disabled
the check.  This was a mistake, I should have either spent even more
time trying to get to the bottom of this (especially, writing test
vectors for the spec and testing against other implementations).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-10 22:25:32 +00:00
Rusty Russell 981b000949 pytest: add trivial test to make sure reconnect works with no channel activity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-10 22:25:32 +00:00
Mark Beckwith 816840e9c4 rpc: check error now consistent with lightning-cli
We now return JSONRPC2_METHOD_NOT_FOUND if the command is not found,
just like lightning-cli does.

Signed-off-by: Mark Beckwith <wythe@intrig.com>
2018-12-10 09:27:49 +01:00
Rusty Russell 0c5f8a3a15 pytest: add test for check command.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Mark Beckwith <wythe@intrig.com>
2018-12-10 09:27:49 +01:00
Franck Royer dc7b76e5e6 Update testing requirements and instructions 2018-12-09 16:26:30 +01:00
lisa neigut a39c97c960 channeld: support private channel creation, fixes #2125
Adds a new 'announce' field for `fundchannel`, which if false
won't broadcast a `channel_announcement`.
2018-12-08 15:15:55 -08:00
lisa neigut eab992cecd py-tests: rename 'announce' to 'wait_for_announce'
Better description of what the option actually does -- if true
waits for the announcement messages to be generated and exchanged.
2018-12-08 15:15:55 -08:00
Rusty Russell fab5027d64 configure: add --experimental-features flag (default: == DEVELOPER).
This will be used for option_simplified_commitment and other
still-being-specified options.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-06 23:11:51 +01:00
Rusty Russell aee2197a66 jsonrpc: make sure even errors are valid json.
We often quote their msg in our reply; sanitize it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-06 23:11:51 +01:00
Christian Decker f5a3f1f0a2 plugin: Add a test for timeout and broken manifest
Both of these plugins will fail in interesting ways, and we should
still handle them correctly.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-05 23:15:59 +00:00
Rusty Russell eb03b33655 plugins: add and install built-in plugin dir, add clear and disable options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-05 01:22:55 +01:00
Rusty Russell ac5002a79e pytest: add hack to force options into a given order.
Needed for testing plugin options which are order-senditive.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-05 01:22:55 +01:00
Rusty Russell a4287f99fd lightningd: add --plugin-dir option to load directory full of plugins.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-05 01:22:55 +01:00
Christian Decker 1d75a70995 pytest: Add a test for the JSON-RPC passthrough
Tests JSON-RPC registration, as well as success and failures.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-02 22:55:47 +00:00
Christian Decker 4d95bb0539 pytest: Mark test_reserve_enforcement as flaky
It has been causing a lot of Travis failures due to a presumed memory
leak:

```
Exception: Node /tmp/ltests-fhjg26a2/test_reserve_enforcement_1/lightning-2/ has memory leaks: [{'label': 'lightningd/json_stream.c:48:struct json_stream', 'backtrace': ['ccan/ccan/tal/tal.c:435 (tal_alloc_)', 'lightningd/json_stream.c:48 (new_json_stream)', 'lightningd/jsonrpc.c:90 (jcon_new_json_stream)', 'lightningd/jsonrpc.c:444 (attach_json_stream)', 'lightningd/jsonrpc.c:455 (json_start)', 'lightningd/jsonrpc.c:464 (json_stream_success)', 'lightningd/pay.c:932 (json_sendpay_on_resolve)', 'lightningd/pay.c:444 (payment_store)', 'lightningd/pay.c:540 (payment_failed)', 'lightningd/peer_htlcs.c:143 (fail_out_htlc)', 'lightningd/peer_htlcs.c:360 (destroy_hout_subd_died)', 'ccan/ccan/tal/tal.c:235 (notify)', 'ccan/ccan/tal/tal.c:395 (del_tree)', 'ccan/ccan/tal/tal.c:405 (del_tree)', 'ccan/ccan/tal/tal.c:405 (del_tree)', 'ccan/ccan/tal/tal.c:479 (tal_free)', 'ccan/ccan/io/io.c:451 (io_close)', 'lightningd/subd.c:500 (sd_msg_read)', 'lightningd/subd.c:302 (read_fds)', 'ccan/ccan/io/io.c:59 (next_plan)', 'ccan/ccan/io/io.c:395 (do_plan)', 'ccan/ccan/io/io.c:405 (io_ready)', 'ccan/ccan/io/poll.c:310 (io_loop)', 'lightningd/lightningd.c:769 (main)'], 'parents': ['lightningd/jsonrpc.c:681:struct json_connection', 'common/configdir.c:29:char[]'], 'value': '0x125be08'}]
```

Signed-off-by: Christian Decker <@cdecker>
2018-11-29 23:01:11 +00:00
Christian Decker 522c3039ec pytest: Pretty print the memleak output in tests
It wasn't JSON formatted either so there was no nice pretty-printing
way. This jsonifies and pretty prints it.

Signed-off-by: Christian Decker <@cdecker>
2018-11-29 23:01:11 +00:00
Christian Decker 685a76e714 pytest: test_closing_different_fees no longer requires DEVELOPER=1
We no longer use `dev-override-fees` to set the fees, rather we
instrument the `bitcoind` proxy to return the desired feerates.

Signed-off-by: Christian Decker <@cdecker>
2018-11-29 23:01:11 +00:00
Christian Decker 879f9b7986 plugin: Make a common directory of plugins in contrib
No need to create a directory each.

Signed-off-by: Christian Decker <@cdecker>
2018-11-26 22:53:37 +00:00
Christian Decker e27b2ea69b pytest: Add a test for the plugin option passthrough
Signed-off-by: Christian Decker <@cdecker>
2018-11-26 22:53:37 +00:00
Christian Decker f520052b95 pytest: Add two more dependencies
pytest was an indirect dependency so far, making that one
explicit, and the timeout plugin should allow us to kill a stuck test
before travis kills it, and thus allow us to see where it got stuck.

Signed-off-by: Christian Decker <@cdecker>
2018-11-26 22:53:37 +00:00
Rusty Russell 1d7b287439 pytest: speed up test_restart_many_payments when !DEVELOPER.
Because gossip in this case takes up to a minute, this test took 10
minutes.  The workaround is to do the waiting-for-gossip all at once.

Now it takes 362 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-22 05:15:42 +00:00
Rusty Russell 8fb1b609ce closingd: handle our own memleak detection.
Unlike other daemons, closingd doesn't listen to the master, but runs
simply to its own beat.  So instead of responding to the JSON dev_memleak
command, we always check for memory leaks, and make sure that the
python tests fail if they see MEMLEAK in the logs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-22 05:15:42 +00:00
Rusty Russell 2c582cbcfb pytest: test that we don't revert config handling again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-21 01:08:42 +00:00
Rusty Russell debbdc0781 bolt11: accept lightning: prefix.
The Blockstream store produces these, for example, so let's ignore them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-20 21:02:31 +01:00
Rusty Russell 6f7650e471 pytest: test using malformed JSONRPC commands.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-20 16:51:19 +01:00
Rusty Russell 7799368144 pytest: add test for multiplexed RPC output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-20 16:51:19 +01:00
Rusty Russell e0d14bddb9 jsonrpc: allow multiple commands at once.
We now keep multiple commands for a json_connection, and an array of
json_streams.

When a command wants to write something, we allocate a new json_stream
at the end of the array.

We always output from the first available json_stream; once that
command has finished, we free that and move to the next.  Once all are
done, we wake the reader.

This means we won't read a new command if output is still pending, but
as most commands don't start writing until they're ready to write
everything, we still get command parallelism.

In particular, you can now 'waitinvoice' and 'delinvoice' and it will
work even though the 'waitinvoice' blocks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-20 16:51:19 +01:00
Rusty Russell 0c3f85d931 lightning.py: parse multiple JSON RPC commands accurately.
We need to keep the remaining buffer, and we need to try to parse it
before we read the next.  I first tried keeping it in the object, but
its lifetime is that of the *socket*, which we actually reopen for
every command.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-19 21:36:40 +01:00
Rusty Russell fe11ee5406 pytest: make test_multirpc check that RPC commands complete.
This was hanging sometimes in travis, but actually checking the result
of the commands makes it *always* hang.  We remove the waitinvoice
which will not return.

ZmnSCPxj points out that this behavior, introduced in
ce0bd7abd3, is a regression: it would be
nice to be able to cancel a waitinvoice.  But that fix is more complex,
and will have to be another PR.

This test will now hang, but it's OK: we're about to fix it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-19 21:36:40 +01:00
Christian Decker 46b2e7502c bitcoin: If we fail to estimate the fee in testnet use the minfee
When developing in regtest or testnet it is really inconvenient to
have to fake traffic and generate blocks just to get estimatesmartfee
to return a valid estimate. This just sets the minfee if bitcoind
doesn't return a valid estimate.

Reported-by: Rene Pickhardt <@renepickhardt>
Signed-off-by: Christian Decker <@cdecker>
2018-10-29 03:20:08 +00:00
Rusty Russell aab91557f3 Travis: eliminate 4 slowest tests, with new SLOW_MACHINE flag.
In one case we can reduce, in the others we eliminated if VALGRIND.

Here are the ten slowest tests on my laptop:

469.75s call     tests/test_closing.py::test_closing_torture
243.61s call     tests/test_closing.py::test_onchain_multihtlc_our_unilateral
222.73s call     tests/test_closing.py::test_onchain_multihtlc_their_unilateral
217.80s call     tests/test_closing.py::test_closing_different_fees
146.14s call     tests/test_connection.py::test_dataloss_protection
138.93s call     tests/test_connection.py::test_restart_many_payments
129.66s call     tests/test_gossip.py::test_gossip_persistence
128.73s call     tests/test_connection.py::test_no_fee_estimate
122.46s call     tests/test_misc.py::test_htlc_send_timeout
118.79s call     tests/test_closing.py::test_onchain_dust_out

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 7d614aaf25 pytest: really remove all bitcoin generate RPC calls.
generate was deprecated some time ago, so we added the generate_block()
helper.  But many calls crept back in, and git master refuses it.

(test_blockchaintrack relied on the return value, so make generate_block
return the list of blocks).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 38e6aa66ff python: quieten modern flake8.
After Ubuntu 18.10 upgrade, lots of new flake8 warnings.

$ flake8 --version:
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 1.6.0) CPython 3.6.7rc1 on Linux

Note it seems that W503 warned about line breaks before binary
operators, and W504 complains about them after.  I prefer W504, so
disable W503.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 0bc80c4687 pytest: make sure we wait for node announcements in line_graph too.
Occasional failure in test_fulfill_incoming_first where the channel
closed before the final message from dev_disonnect was read.  Cause
was the peer writing a gossip msg and failing due to ECONNRESET, before
it read the final message.

(Managed to reproduce under strace -f, FTW).

This is really a symptom of the fact that line_graph's announce=True
didn't wait for node announcements.  Let's do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-24 16:18:32 +02:00
Rusty Russell c5cd4791be onchaind: allow multiple candidate HTLCs for output match
When we have multiple HTLCs with the same preimage and the same CLTV,
it doesn't matter what order we treat them (they're literally
identical).  But when we offer HTLCs with the same preimage but
different CLTVs, the commitment tx outputs look identical, but the
HTLC txs are different: if we simply take the first HTLC which matches
(and that's not the right one), the HTLC signature we got from them
won't match.  As we rely on the signature matching to detect the fee
paid, we get:

	onchaind: STATUS_FAIL_INTERNAL_ERROR: grind_fee failed

So we alter match_htlc_output() to return an array of all matching
HTLC indices, which can have more than one entry for offered HTLCs.
If it's our commitment, we loop through until one of the HTLC
signatures matches.  If it's their commitment, we choose the HTLC with
the largest CLTV: we're going to ignore it once that hits anyway, so
this is the most conservative approach.  If it's a penalty, it doesn't
matter since we steal all HTLC outputs the same independent of CLTV.

For accepted HTLCs, the CLTV value is encoded in the witness script,
so this confusion isn't possible.  We nonetheless assert that the
CLTVs all match in that case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell a0511a4496 pytest: more thorough tests for HTLC confusion.
We set up HTLCs with the same preimage and both different and same
CLTVs in both directions, then make sure that onchaind is OK and that
the HTLCs are failed without causing downstream failure.

We do this for both our-unilateral and their-unilateral cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell a4dbd6e0f0 pytest: simple addition to test_onchain_timeout to trigger grind failure.
Create a second HTLC with a different CTLV but same preimage; onchaind
uses the wrong signature and fails to grind it.

Reported-by: molz (#c-lightning)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell 3746ea36e2 channeld: tiebreak identical HTLC outputs by CLTV.
This was suggested by Pierre-Marie as the solution to the 'same HTLC,
different CLTV' signature mismatch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell 0006d5771d pytest: test for HTLCs with identical payment_hash and different CLTVs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell ad2519a6f4 spelling: Check LockTime Verify.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell c403415caa lightningd: format JSON directly into json connection membuf.
My test case is a mainnet gossip store with 22107 channels, and
time to do `lightning-cli listchannels`:

Before: `lightning-cli listchannels` DEVELOPER=0
	real	0m1.303000-1.324000(1.3114+/-0.0091)s

After:
	real	0m0.629000-0.695000(0.64985+/-0.019)s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 22:02:11 +00:00
Rusty Russell 71a40faae7 withdraw: fix incorrect error when we have an empty wallet.
This also highlights the danger of searching the logs: that error
appeared previously in the logs, so we didn't notice that the actual
withdraw call gave a different error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 22:02:11 +00:00
Christian Decker f583e6bc58 pytest: Silence some more warnings about raw_strings in regexps 2018-10-19 22:01:22 +00:00
Rusty Russell 31d8a5a9a4 pytest: cleanup test_forward_stats.
Use dev_ignore_htlcs to make sure HTLC gets stuck.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 21:58:27 +00:00
Rusty Russell 604638712b jsonrpc: Only show total fees collected in getinfo.
And use wallet_forward_status_in_db() everywhere in db code.
And clean up extra CHANGELOG.md entry (looks like rebase error?)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 21:58:27 +00:00
Christian Decker 6b7546b94d json-rpc: Rename `getroutestats` and move stats to getinfo
Signed-off-by: Christian Decker <@cdecker>
2018-10-19 21:58:27 +00:00
Christian Decker facd7d16aa json-rpc: Add `listforwardings` command 2018-10-19 21:58:27 +00:00
Christian Decker 6efdd5275c pytest: Test that we record forwardings correctly
Adapts the `test_forward_stats` test to include checks for the
`forwarded_payments` table. Will add checks for the `listforwardings`
RPC call next.

Signed-off-by: Christian Decker <@cdecker>
2018-10-19 21:58:27 +00:00
Rusty Russell 09b33015c4 connectd: give user a hint when wrong key is used.
When the wrong key is used, the remote end simply hangs up.

We used to get a random errno, which tends to be "Operation now in progress."
Now it's defined to be 0, detect and provide a better error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-19 00:53:00 +00:00
lisa neigut 0ae1d03513 BOLT7: broadcast `htlc_maximum_msat` in `channel_update s
Have c-lightning nodes send out the largest value for
`htlc_maximum_msat` that makes sense, ie the lesser of
the peer's max_inflight_htlc value or the total channel
capacity minus the total channel reserve.
2018-10-16 03:32:27 +00:00
Rusty Russell 030fe1ce53 gossipd: don't expose private channels for routeboost.
We don't create unannouncable channels, but other implementations can.
Not only is it rude to expose these via invoices, it's probably not
useable anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-15 23:04:17 +00:00
Christian Decker a44491fff0 benchmark: Add two benchmarks for forwarded payments
We weren't benchmarking them, so do that now.
2018-10-13 10:58:53 +00:00
Christian Decker 81f5b89fe3 wallet: Actually load wallet stats when asked to
The call to `sqlite3_step` is actually needed, otherwise we'll always
get the default values for all types.
2018-10-13 10:58:53 +00:00
Christian Decker 2354b3e706 json-rpc: Create test showcasing the null-accounting issue
There is an issue with the way we retrieve the channel accounting info
that will result in always showing 0 for all stats. This tests for it
and the next commit will fix it.
2018-10-13 10:58:53 +00:00
William Casarin b3f7236201 python: add cheroot to requirements.txt
It looks like this is a new dependency.

Sort the list while we're at it.

Signed-off-by: William Casarin <jb55@jb55.com>
2018-10-10 18:00:22 +02:00
Rusty Russell fcb5310873 pytest: make wait_for do exponential backoff, start at 0.25 seconds.
This doesn't alter runtime very much, but does reduce log spam.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-10 06:10:42 +00:00
Rusty Russell 84b9e3e72b lightningd: reduce log spam from bitcoin-cli invocations.
During tests, this is half our log!  And Travis truncates it if we get
a failure in test_restart_many_payments.

Interestingly, test_logging had a bug which relied on this spam :)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-10 06:10:42 +00:00
Rusty Russell a4730089e8 pytest: make test_restart_many_payments more Pythony!
IIUC, namedtuple is like tuple for grown-ups: Pythonify!

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell d85251ac6c db: fix up HTLCs which are missing failure information.
We don't save them to the database, so fix things up as we load them.

Next patch will actually save them into the db, and this will become
COMPAT code.

Also: call htlc_in_check() with NULL on db load, as otherwise it aborts
internally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 77be009354 pytest: add restart-during-n-way payment test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 65f6813706 lightningd: handle the case where the db contains a resolved HTLC without a preimage.
We need to handle this case (old db) before the next commit, which actually
fixes it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell c94ab7370c pytest: extend the test_fulfill_incoming_first case to cover reconnect.
Which we don't handle, due to a separate bug, so it's xfail.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 4040c53258 lightningd: handle case where incoming HTLC vanished before fulfilled outgoing.
We now need an explicit 'local' flag, rather than relying on the existence
of the 'in' pointer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell 3643e1bd90 pytest: add test for when an incoming fulfilled HTLC expires before outgoing.
Usually, we only close an incoming HTLC once the outgoing HTLC is completely
resolved.  However, we short-cut this in the FULFILL case: we have the
preimage, so might as well use it immediately (in fact, we wait for it to
be committed, but we don't need to in theory).

As a side-effect of this, our assumption that every outgoing HTLC has
a corresponding incoming HTLC is incorrect, and this test (xfail) tickles
that corner case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
Rusty Russell b3fb23f921 pytest: make line_graph wait for gossip propagation if announce param is True
This is what the callers want; generalize it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
lisa neigut a9bf1f5573 tests: quiet down DeprecationWarnings for escape sequences
Nuke all the `DeprecationWarning: invalid escape sequence
\[` messages that show up when you run python tests.
2018-10-08 13:18:31 +02:00
Christian Decker d6fcfe00c7 pytest: Stabilize test_feerates 2018-09-28 22:45:30 +00:00
Rusty Russell bb5e2ffafb gossipd: don't create redundant node_announcements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 18:20:17 +02:00
Rusty Russell 6c54a22d63 pytest: make test_node_reannounce check for redundant announce (xfail).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 18:20:17 +02:00
Rusty Russell b0d9182fc3 pytest: allow log_all_io through get_nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 18:20:17 +02:00
Rusty Russell 0baa5f7071 gossipd: send node announcement on startup.
I suspect this fixes #1660 too, but checking would be good.

Fixes: #1781
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 18:20:17 +02:00
Rusty Russell cbd1d1d0f2 pytest: test that we reannounce node after restart.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 18:20:17 +02:00
Rusty Russell c8c3294a23 json-invoice: add routeboost, warnings.
We split json_invoice(), as it now needs to round-trip to the gossipd,
and uniqueness checks need to happen *after* gossipd replies to avoid
a race.

For every candidate channel gossipd gives us, we check that it's in
state NORMAL (not shutting down, not still waiting for lockin), that
it's connected, and that it has capacity.  We then choose one with
probability weighted by excess capacity, so larger channels are more
likely.

As a side effect of this, we can tell if an invoice is unpayble (no
channels have sufficient incoming capacity) or difficuly (no *online*
channels have sufficient capacity), so we add those warnings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 15:03:42 +02:00
Rusty Russell 9455331575 json: use bolt naming for features arrays in listnodes, listpeers.
Deprecate the old names.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 04:14:28 +00:00
Rusty Russell 3358437062 connectd: don't log every time a peer disconnects.
Great for a few of our tests, but generally spammy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 04:14:28 +00:00
Rusty Russell a90c200fcb pytest: wait_channel_quiescent helper to wait for resolved htlcs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-27 22:57:19 +02:00
Rusty Russell 704f8b5b0e pytest: make test_autocleaninvoice more reliable.
We're about to slow down the invoice rpc (esp. under valgrind), which
breaks the delicate timing of the autocleaninvoice test.

Change that so the autocleaner (and timestamp) starts after the invoices
are added.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-27 22:57:19 +02:00
Christian Decker d590302523 pytest: Fix benchmarks after the fixture migration 2018-09-27 00:18:24 +00:00
Rusty Russell 59bdba6bac pytest: don't test crashing under valgrind at all.
Travis failures:

valgrind: m_scheduler/sema.c:104 (vgModuleLocal_sema_down): Assertion 'sema->owner_lwpid != lwpid' failed.
host stacktrace:
==1296==    at 0x38083F48: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x38084064: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x380841F1: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x38135DAE: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x380D328D: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x3809A4AC: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x3809AE43: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==1296==    by 0x380988CF: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
sched status:
  running_tid=0
Thread 1: status = VgTs_WaitSys (lwpid 1296)
==1296==    at 0x5729730: __poll_nocancel (syscall-template.S:84)
==1296==    by 0x4348DF: daemon_poll (daemon.c:78)
==1296==    by 0x4169E7: io_poll_lightningd (lightningd.c:543)
==1296==    by 0x471ECD: io_loop (poll.c:282)
==1296==    by 0x416E06: main (lightningd.c:744)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-26 16:32:08 -07:00
Rusty Russell e450c6bbdb gossipd: remove time-delayed local channel_update, produce DISABLE on-demand.
We have a lot of infrastructure to delay local channel_updates to
avoid spamming on each peer reconnect; we had to keep tracking of
pending ones though, in case we needed the very latest for sending an
error when failing an HTLC.

Instead, it's far simpler to set the local_disabled flag on a channel
when we disconnect, but only send a disabling channel_update if we
actually fail an HTLC.

Note: handle_channel_update() TAKES update (due to tal_arr_dup), but we
didn't use that before.  Now we do, add annotation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-26 03:21:35 +00:00
Rusty Russell 16e16a725e gossipd: apply private updates to announce channel.
We trade channel_update before channel_announce makes the channel
public, and currently forget them when we finally get the
channel_announce.  We should instead apply them, and not rely on
retransmission (which we remove in the next patch!).

This earlier channel_update means test_gossip_jsonrpc triggers too
early, so have that wait for node_announcement.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-26 03:21:35 +00:00
Mark Beckwith cbde3e20f7 cli: help command now also prints usage
The help command now adds command usage to its output by calling each
command handler in CMD_USAGE mode.

Instead of seeing, for example:

	decodepay
	    Decode {bolt11}, using {description} if necessary

we see:

	decodepay bolt11 [description]
	    Decode {bolt11}, using {description} if necessary

Signed-off-by: Mark Beckwith <wythe@intrig.com>
2018-09-25 15:11:45 +02:00
Rusty Russell 8455b12781 Revert "gossipd: handle premature node_announcements in the store."
This reverts commit e2f426903d.

With the new store version, this can't happen.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 17:56:15 +02:00
Rusty Russell 48de77d56e gossipd: invalidate old gossip_stores.
Incrementing version number means stores which were prior to the previous
commit will be removed, and refreshed.  The simplest fix, if not the most
efficient.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 17:56:15 +02:00
Rusty Russell 0855422110 gossip_control: when searching for a txout, make sure it's not spent!
There's no reason for the db to ever return non-NULL if it's spent.  And there's
only one caller, for which that is definitely true.

Suggested-by: @cdecker
Fixes: #1934
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 17:56:15 +02:00
Rusty Russell 24c386c086 test_gossip: gossip retransmit on spent UTXO test.
Aka #1934.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 17:56:15 +02:00
Rusty Russell 6228c25643 test_gossip: basic 'node notices close' test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 17:56:15 +02:00
Rusty Russell 0925daa087 gossipwith: simple tool to snarf gossip from a node.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 17:56:15 +02:00
Rusty Russell edb74260f4 pytest: more reliable onchain tests.
We extract the tx from the logs, and then we wait until that hits
the mempool.  This is more reliable than 'sendrawtx' in the logs,
which might catch a previous sendrawtx; it's also more explicit
that we expect that tx exactly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 00:28:44 +00:00
lisa neigut b1ceaf9910 gossipd: Update BOLT-split flags in channel_update
BOLT 7's been updated to split the flags field in `channel_update`
into two: `channel_flags` and `message_flags`. This changeset does the
minimal necessary to get to building with the new flags.
2018-09-21 00:24:12 +00:00
Rusty Russell da9d92960d lightningd: accept hsmstatus_client_bad_request messages (and log!)
We currently just ignore them.  This is one reason the hsm (in some places)
explicitly calls log_broken so we get some idea.

This was the only subdaemon which had a NULL msgcb and msgname, so eliminate
those checks in subd.c.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-20 09:49:39 +02:00
Rusty Russell e41e1a177e pytest: wait until mock is called for set_feerates.
Got a spurious failure in test_no_fee_estimate; we fired too soon from the logs (presumably
we raced in on the first response, but estimatesmartfee gets called 3 times).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-20 09:12:28 +02:00
Rusty Russell 252bbe1d2d pytest: don't wait for sendrawtx, wait for expected tx.
In particular, test_no_fee_estimate was flaky due to seeing the funding
tx being sent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-19 13:04:01 +02:00
Christian Decker f29f92a5fe pytest: Clean up bitcoind_cmd_override, it's no longer used 2018-09-16 00:05:34 +02:00
Christian Decker 9e5d7dacb0 pytest: Use the mock bitcoind everywhere 2018-09-16 00:05:34 +02:00
Christian Decker 16869e3fe6 pytest: Use the bitcoind proxy to mock feerates 2018-09-16 00:05:34 +02:00
Christian Decker aa80a330f1 pytest: Remove auto-proxying in favor of a per-node btc proxy 2018-09-16 00:05:34 +02:00
Christian Decker 2dabc5af93 pytest: Set correct header in mock bitcoind 2018-09-16 00:05:34 +02:00
Christian Decker 74f228deb8 btcproxy: Unpack batched JSON-RPC calls and issue them separately 2018-09-16 00:05:34 +02:00
Christian Decker 88186020e0 pytest: Implement method mocking for ProxiedBitcoinRpc 2018-09-16 00:05:34 +02:00
Christian Decker e132dffa0b pytest: Add an RPC proxy inbetween bitcoind and bitcoin-cli
This is a simple reverse proxy that `bitcoin-cli` can talk to when invoked by
`lightningd`. It allows us to trace `bitcoin-cli` calls, and intercept calls to
mock the replies, better than the current bash-script based method.
2018-09-16 00:05:34 +02:00
Christian Decker f505a9418b pytest: Fix lint error 2018-09-16 00:05:34 +02:00
Christian Decker f1e931f7bb pytest: Fix flaky test_logging
File was rotated away but didn't wait for the first line to be actually written.
2018-09-14 21:19:50 +02:00
Christian Decker 79da1b9aa2 pytest: Keep the test directory even if the failure is in the fixtures 2018-09-14 21:19:50 +02:00
Rusty Russell cefb6925b2 db: save and restore last_sent_commit correctly.
It's an array: we were only saving the single element; if there was more than
one changed HTLC we'd get a bad signature!

The report in #1907 is probably caused by the other side re-requesting
something we considered already finalized; to avoid this particular error,
we should set the field to NULL if there's no last_sent_commit.

I'm increasingly of the opinion we want to just save all the update
packets to the db and blast them out, instead of doing this
second-guessing dance.

Fixes: #1907
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-04 14:43:27 +02:00
Rusty Russell db12a1452f pytest: reproduce problem with restarting and retransmitting multiple outgoing htlcs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-04 14:43:27 +02:00
Rusty Russell e2f426903d gossipd: handle premature node_announcements in the store.
These happen after we compact the store; every log I've seen of a
restart on a real node has a message about truncating the store,
because node_announcements predate channel_announcements.

I extracted one such case from testnet, and reduced it to test here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-04 14:36:05 +02:00
Rusty Russell 312209ad60 pytest: support simple subdaemon debugging.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-04 14:36:05 +02:00
Rusty Russell 7e1fd9c38a pytest: make test_no_fee_estimate more reliable.
1. Wait for a 'sendrawtransaction' *after* the dev-fail message; don't be
   fooled by a previous one.
2. Turning on estimate fee sets fees exactly; just wait for it to be processed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-30 16:33:35 +02:00
Rusty Russell 33c6285787 feerates: turn it into a simple query API, remove setting.
It's probably unnecessary to have this weird way of injecting results
now we have explicit feerate args.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-30 16:33:35 +02:00
Rusty Russell db3c387264 feerate: allow names 'urgent' 'normal' and 'slow'.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-30 16:33:35 +02:00
Rusty Russell e0952ceff2 feerate: use suffix, not separate argument.
And, reluctantly, default to bitcoind style.
"It's wrong to be right too soon."

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-30 16:33:35 +02:00
Rusty Russell 14dc1c37ab fundchannel / withdraw: allow explicit feerate setting.
These are the two cases where we'll refuse without a fee estimate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-30 16:33:35 +02:00
Rusty Russell e2d4b7cc8d cleanup: extract and formalize feerate conversion.
I didn't want to create a new file for this now, as that would totally
break #1880.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-30 16:33:35 +02:00
Rusty Russell af4fa9a359 feerates: rename sipa/bitcoind to perkw/perkb.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-25 00:33:12 +00:00
Rusty Russell a4b952ebc7 feerate: include rough estimates of actual tx costs.
We could refine this later (based on existing wallet, for example), but
this gives some estimate.

[ Rename onchain_estimates -> onchain_fee_estimates Suggested-by: @SimonVrouwe ]
[ Factor of 1000 fix Reported-by: @SimonVrouwe ]
Suggested-by: @molxyz
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-25 00:33:12 +00:00