Commit Graph

705 Commits

Author SHA1 Message Date
Christian Decker 88217369c2 pytest: Move NodeFactory to utils.py
We are starting to move things out of test_lightningd.py so this is a logical
first step.
2018-08-07 00:54:19 +00:00
Christian Decker 6dae525c07 pytest: Check description of newly created invoices 2018-08-07 00:54:19 +00:00
Christian Decker b55d03cb30 pytest: Pass result to fixtures and keep directories of failed tests
@Reported-by: Rusty Russell <@rustyrussell>
@Signed-off-by: Christian Decker <@cdecker>
2018-08-07 00:54:19 +00:00
Rusty Russell 8f38a46584 lightningd: correctly store our own channel_reserve_satoshis
openingd calculates our reserve based on the channel amount (even if
we're funding, to keep the calculation in one place), but it wasn't
reporting it back to the master daemon.  We initialized it to 0 so that
valgrind wouldn't get upset, as it's part of a structure we send over
the wire.

Have openingd report back, and also initialize it to an impossible value
as extra assurance.  And remove a stray (harmless but weird) semicolon.

Reported-by: Gálli Zoltán
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell 6d79f7679c pytest: test reserve enforcement.
A bit tricky, since we don't normally let ourselves make bad HTLCs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell 6e10237d10 pytest: allow db_query to manipulate db.
Needed for the next test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell c5b5f74965 lightningd: display all addresses in listnodes.
We kept overwriting the first one on marshalling, so any following
addresses were junk.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:33:46 +02:00
Christian Decker d3edfc8028 onion: Print the extracted channel_update and onionreply to debug
This adds one line with the onion and the channel_update we extract from
it. This in turn allows us to check that the channel_update in the onion is not
type prefixed, and that we patch it correctly before passing it to gossipd.
2018-07-30 21:19:04 +00:00
Rusty Russell 52303029aa fundchannel: cap 'all' at 2^24-1 satoshi.
The easiest way to do this is to play with the 'wallet_tx' semantics
and have 'amount' have meaning even when 'all_funds' is set.

Note that we change the string 'Cannot afford funding transaction' to
'Cannot afford transaction' as this code is also used for withdrawls.

Inspired-by: molz on #c-lightning
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-30 11:46:41 +02:00
Christian Decker 687f171e17 json-rpc: Add description to `listpayments` and similar 2018-07-30 03:04:45 +00:00
Rusty Russell 65103ac426 pytest: give more time for test_closing_torture.
The logs in various Travis failures show that it takes 20 seconds just for
closingd to read the init message.  As a result, the close times out (default
is 30 seconds).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-29 16:23:27 +02:00
Rusty Russell be3b782cb4 pytest: add test for update being decoded from onion error.
This actually tests that we correctly process an error packet, and
disable the channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-27 14:12:00 +02:00
Rusty Russell 2f78c90782 pytest: fix flakiness in test_penalty_outhtlc
We were *supposed* to be waiting for the next commitment tx so we
made sure the one we broadcast was old, *but* the 'revoke_and_ack'
we were waiting for could be matched by the completion of the previous
'revoke_and_ack'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-27 11:36:22 +02:00
Rusty Russell f73ea5d0ba pytest: test for correct wallet balances, re-enable penalty test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-27 11:36:22 +02:00
William Casarin 8869204730 test: add pytest-xdist to the python test requirements
Signed-off-by: William Casarin <jb55@jb55.com>
2018-07-26 23:18:37 +02:00
Christian Decker d6048de100 json-rpc: Shutdown the JSON-RPC in the context of a DB transaction
This needs to be done separately from the rest of the daemon since we can
otherwise not make sure that it happens before the DB is freed and we might
still need the DN, and be running in a DB transaction, for some destructors to
run.
2018-07-26 19:14:56 +02:00
Christian Decker ead0c8e17a pytest: Add test for issue 1737 that is expected to fail
This reproduces the issue described in issue #1737 and will be fixed by the next
commit.
2018-07-26 19:14:56 +02:00
Rusty Russell b5fcd54ef0 channeld: don't read from gossipfd while we're reconnecting.
That was the cause of the bad gossip order failures: gossipd thought our
channel was live, but the other end didn't receive message last time.

Now gossipd doesn't use fd to kill us (connectd tells master to do so), we
can implement read_peer_msg_nogossip().

Fixes: #1706
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell dba7f9002f gossipd: provide connectd with address resolution.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell 3d3d2ef9af gossipd: remove connectd functionality, enable connectd.
This patch guts gossipd of all peer-related functionality, and hands
all the peer-related requests to channeld instead.

gossipd now gets the final announcable addresses in its init msg, since
it doesn't handle socket binding any more.

lightningd now actually starts connectd, and activates it.  The init
messages for both gossipd and connectd still contain redundant fields
which need cleaning up.

There are shims to handle the fact that connectd's wire messages are
still (mostly) gossipd messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell f4e2a9c921 pytest: don't expect alias and color in listpeers.
Gossipd combines the information if it knows it, but that's really the
job of 'listnodes'.  More importantly, channeld won't have access to
this information.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell 7986af1b1e pytest: only_one() helper to catch if RPC returns more elements than we expect
I saw an error in test_gossip_weirdalias in Travis, where listnodes(nodeid)
returned *BOTH* nodes; it happened to fail because [0] was the wrong one, but
it would have passed if the order had been different.

This helper asserts that we really do only have one element, and should
catch such bugs faster.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-18 15:11:05 +02:00
Rusty Russell edf043c89b pytest: make utils.py read config.vars.
I could not figure out why test_announce_address suddenly stopped working:
I had previously been using DEVELOPER=1 on the cmdline for historical
reasons when testing locally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-16 23:16:34 +02:00
Rusty Russell a37e692af6 pytest: mark test_reconnect_normal flaky for now.
The bad gossip order bug is going to take a while to fix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-16 22:48:41 +02:00
Rusty Russell e0c21debc2 pytest: fix sleep(1) in test_channel_persistence.
It's obviously not enough under load, as Travis demonstrates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-15 17:37:29 +02:00
Rusty Russell b1182702ae pytest: increase timeout for test_payment_failed_persistence
This happened under travis.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-15 17:37:29 +02:00
Rusty Russell dd7afc33ee pytest: fix flaky test in test_htlc_sig_persistence
We can shutdown before the tx hits bitcoind.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-07 16:26:23 +02:00
Rusty Russell ed83bbe623 pytest: fix flaky race in test_gossip_query_channel_range.
We weren't waiting for gossipd to actually process the
dev_set_max_scids_encode_size message, so under Travis it sometimes
split the reply before processing that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-07 16:26:23 +02:00
Rusty Russell 55a09d79b9 pytest: make test_gossip_no_empty_announcements robust when update delayed.
In this test we tell l3 to disconnect on sending WIRE_CHANNEL_ANNOUNCEMENT.
This is hit by gossipd (to disconnect from l2) but *also* channeld to
disconnect from l4.  That's OK, because normally by this point l4 has
sent its real channel_update.

However, the next patch introduces a delay in sending channel_updates,
meaning l4 hasn't sent it yet.  If l3 doesn't reconnect to l4, we
never get the channel_update and the test which expects l1 to eventually
see both sides of the channel fails.

So we manually reconnect then.  Note that we remove the redundant
'dev-no-reconnect' option from l2: it's added automatically as it
doesn't set 'may_reconnect'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-07 16:07:53 +02:00
Rusty Russell 9fa738a741 listpeers: expose peer features as 'local_features' and 'global_features'
For now, just the connected peers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-07 16:07:53 +02:00
Rusty Russell c46f373205 options: refuse two --announce-addr of the same type.
Gossipd will ignore the second one, but doing it in the front end
gives an explicit error message.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-01 15:03:21 +02:00
Rusty Russell f67182ff20 gossipd: order node_announcement addresses correctly, remove duplicate types.
Fixes: #1596
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-01 15:03:21 +02:00
Rusty Russell e6f31b3c66 pytest: allow duplicated options.
Python dict can't have duplicate entries, but some options can be specified
multiple times.  The easiest way is to put a list in the dict.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-01 15:03:21 +02:00
Christian Decker 6c82d1b8b0 pylightning: Subclass ValueError to access to the returned error 2018-07-01 04:13:04 +00:00
Rusty Russell ce5b4bddf2 tests: fix to work with bitcoind master branch.
E               ConnectionRefusedError: [Errno 111] Connection refused

And in debug.log:

2018-05-17T04:06:35Z Warning: Config setting for -rpcport only applied on regtest network when in [regtest] section.

Unfortunately, current versions including 0.16.1 *ignore* the contents of
a '[regtest]' section, so we need it in *both* places.

Also remove the misleading 'rpcport' initialization which we always
override.

Note that we don't fix this message though:

	2018-05-17T04:06:35Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-21 13:43:32 +02:00
Rusty Russell 91d62ad697 pytest: create only a single config file.
For some reason, we created a second bitcoin.conf in the regtest/ directory,
which AFAICT nothing uses.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-21 13:43:32 +02:00
Rusty Russell 2375302534 pytest: move regtest argument into config file.
It's not optional for our test setup, and this makes it easier to invoke
bitcoin-cli manually, for example.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-21 13:43:32 +02:00
Rusty Russell c78afa9201 onchaind: use lowball fee instead of donating to miners.
As of bitcoind 0.16.1, you can't send a single-input OP_RETURN output,
as you get 'tx-too-small'.

	sendrawtx exit 26, gave error code: -26?error message:?tx-size-small (code 64)?'

So instead we use the minimum fee we can, but otherwise ignore it and
don't wait for it to be mined.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-21 13:43:32 +02:00
Christian Decker e556bf4f36 pytest: Disable DNS lookups during test runs
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-21 11:21:16 +02:00
Mark Beckwith 8f0ef1636f Added wallet related error codes
New codes: FUND_MAX_EXCEEDED, FUND_CANNOT_AFFORD, FUND_DUST_LIMIT_UNMET.

The error message "Cannot afford fee" was not exactly correct because
it would also occur if the amount requested could not be afforded.  So
I changed it to the more generic "Cannot afford transaction".

Other things:

* Fixed off-by-one satoshi in fundchannel manpage.
* Changed 'arror' to 'error' because we are not pirates.
2018-06-18 12:33:25 +02:00
Christian Decker 583c1c50f8 pytest: Explicitly state the max fee multiplier we assumed in tests
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-14 00:59:42 +00:00
Christian Decker 4dca6daf34 opts: Bump locktime_max so we don't disagree as much with lnd
I still believe that 2 weeks is way too much, but we were promised that these
defaults would be slowly reduced to saner values as the stability increases.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-14 00:59:42 +00:00
Rusty Russell 8183590cdb pytest: test_onchaind_replay needs to wait for tx broadcast.
Waiting for 'Disabling channel' is not enough, since it's async to the
actual tx broadcast: I caught a case where the unilateral close wasn't
in the block, and so failed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-13 16:22:23 +02:00
Rusty Russell 965c20caae tests: reenable developer tests.
72d103d6bb deprecated DEVELOPER env var
in favor of config.vars, but didn't update test_closing.py or test_gossip.py.

5d0a54b7f0 then removed the explicit
DEVELOPER= setting from Travis.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-13 16:22:23 +02:00
Christian Decker 135bd962f7 pytest: Mark test_closing_different_fees as flaky
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-09 13:39:27 +02:00
Rusty Russell 72d103d6bb Makefile: import config.vars.
We leave VALGRIND env var as an override for testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Christian Decker de3b359782 pytest: Use pytest fixtures for the test directory and clean it up
The modern, pytest based, tests now clean up after themselves by removing
directories of successful tests and the base directory if there was no failure.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:32:27 +00:00
Christian Decker 38b7a0e2d2 pytest: Remove directories of successful tests
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-06 03:32:27 +00:00
Rusty Russell e4457e59ed pytest: test_gossip_jsonrpc can always test aliases.
The whole test is under DEVELOPER anyway, but even if it weren't it
would work fine.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 383b309a00 utils: make subd_pid return the *last* pid, in case we restarted daemon.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell e9483f12b6 test_lightningd.py: actually make sure we do zlib encoding.
Previous replies weren't large enough; add another channel and then
use IO tracing to make sure the reply is zlib encoded.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 9e51e196c1 gossipd: dev-set-max-scids-encode-size to artificially force "full" replies.
We cap each reply at a single one, which forces the code into our
recursion logic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 118f099dd8 gossip: dev-query-channel-range to test query_channel_range.
We keep a crappy bitmap, and finish when their replies cover
everything we asked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell c34b49c356 gossipd: add dev-send-timestamp-filter command for testing timestamp filtering.
Since we currently only (ab)use it to send everything, we need a way to
generate boutique queries for testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell c633cbe2ee tests: add dev-query-scids
And write the test for it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell c2189229ca gossipd: only broadcast channel_announcement once we have a channel_update.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 601a855543 tests/test_lightningd.py: test new delayed gossip channel_announce behaviour.
The gossip-query spec enhancements say not to forward an channel_announcement until
you have receive a channel_update.  This test fails for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Rusty Russell 498ad91318 tests: new helper to get pid of a particular subdaemon.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-06 03:25:56 +00:00
Christian Decker 27a186be9c pytest: Stabilize LightningNode.openchannel for multiple channels
When opening a number channels from a single node we could end up not waiting
for the funding tx to make it into the mempool, instead triggering on a previous
`sendrawtransaction` or `CHANNEL_NORMAL` in the logs. This now checks that the
actual funding transaction makes it into the mempool and that we wait for the
depth change for that specific channel.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-31 02:30:27 +00:00
Christian Decker c17848a3f3 gossip: Disable local channels after loading the gossip_store
We don't have any connection yet, so how could they be active? Disable both
sides to avoid trying to route through them or telling others to use them as
`contact_points` in invoices.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-31 02:30:27 +00:00
Christian Decker 202ce5e4ea pytest: Replace wait_for_channels with an actual check
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-31 02:30:27 +00:00
Christian Decker 568cd9ff36 pytest: Making test_routing_gossip less flaky
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-31 02:30:27 +00:00
ZmnSCPxj 0087105d6c test_lightningd: Test forget-if-fundee behavior. 2018-05-23 14:37:32 -07:00
Christian Decker f2b89a9965 pytest: Simplify test_routing_gossip
Settling first and then asserting is longer than just waiting for it to succeed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker 1a45efd0a9 pytest: Use utils.TIMEOUT when settling in test_routing_gossip
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker 2e8b3066e4 pytest: Filter out some unimportant log lines
Mostly `lightningd` complaining about not being able to estimate fees. Safes us
a lot of log space when some tests time out, and safes us a few context switches
between log appender and log watchers.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker 1a9c7783b7 pytest: Yet another instance where we wait only on one side
Waiting on one end and then just asserting on the other is racy.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker f1c64db5e7 pytest: Wait on both sides in test_peerinfo
We were sort of assuming that one side telling it's ok would automagically make
the other side succeed as well.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker eb869cf144 pytest: Make test_sendpay less flaky
While 1 second is very generous, it resolving the HTLCs may take longer, so we
just loop over this instead of making it one-shot

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker a35cc9a79f pytest: Stabilize test_autocleaninvoice
The RPC calls aren't really free, so let's wait for absolute times, computed
from the `start_time` instead.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker dcbd27e7cf pytest: Actually wait for payments to confirm in benchmark
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker 126f024cda pytest: Added a few more benchmarks
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker 4d40ff37aa pytest: Fix the benchmark script
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Christian Decker c4ca2ab4b9 pytest: Fix the benchmark script
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-23 00:20:19 +00:00
Rusty Russell 90dc5173d5 test_lightningd.py: remove 5 slowest tests from non-developer mode.
We can restore them once we get parallel testing on Travis, but meanwhile
we time out because of the 30 seconds bitcoind poll.

Running on my laptop with --duration=5:

=========================== slowest 5 test durations ===========================
184.07s call     tests/test_lightningd.py::LightningDTests::test_multiple_channels
156.66s call     tests/test_lightningd.py::LightningDTests::test_forward
155.77s call     tests/test_lightningd.py::LightningDTests::test_closing
126.83s call     tests/test_lightningd.py::LightningDTests::test_waitinvoice
126.11s call     tests/test_lightningd.py::LightningDTests::test_waitanyinvoice

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell a2dc3d02a8 locktime-blocks: rename to watchtime-blocks.
And clarify the descriptions for end users.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell fd86fbf2fc lightningd: rename --anchor-confirms to --funding-confirms.
That nomenclature won out in Milan.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell 8739b4cbe8 lighningd: Remove --debug-subdaemon-io.
We can use SIGUSR1, even in non-developer builds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell edf1b3cec9 More option cleanups.
Because we have too many which are never used and I don't want to document
them.

1. Remove unused anchor_onchain_wait.  When implemented, it should be
   hardcoded to 100 or more.
2. Remove anchor_confirms_max.  10 always reasonable, and we can readd
   an override option should someone need it.
3. max_htlc_expiry should be the same as locktime_max (which increases
   from 3 to 5 days by default): they're both a limit on how long
   funds can be locked up.
4. channel_update_interval should always be a dev option.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell 0aa22741df option cleanup: --dev-override-fee-rates
Make --override-fee-rates a dev option.  We use default-fee-rate in
its place, which (since bitcoind won't give fee estimates in regtest
mode for short chains) gives an effective feerate of 15000/7500/3750.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell c8cc8fd83f option cleanup: --dev-bitcoind-poll
Make --bitcoind-poll a dev option.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
Rusty Russell 85a062164a test_lightningd: test_closing_torture timeout increase.
I hit this timeout on pytest and valgrind with -n5; use the timeout variable
from utils.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-19 15:52:56 -04:00
Rusty Russell c546b1bbb6 gossipd: specify origin of updates in errors.
@cdecker points out that in test_forward, where we manually create a route,
we get an error back which contains an update for an unknown channel.

We should still note this, but it's not an error for testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-19 15:52:56 -04:00
Rusty Russell 8ee60e2d8e testing: make sure we don't see gossip in bad order.
This is something which generally shouldn't happen, but we didn't
notice it previously.

We ignore this warning in the case where a channel was deleted: this
happens because one side can send an update while the other notices
that the channel is closed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-19 15:52:56 -04:00
Rusty Russell cdded4f53d tests/utils.py: make default timeout configurable via $TIMEOUT
This is useful when you're expecting a test to fail, you've just added
diagnostics.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-10 21:35:53 +02:00
Rusty Russell c33bbb2639 test_lightningd.py: wait longer test_permfail for l2 to notice new blocks.
Reproduced this failure locally: l2 hadn't seen the block yet.  Timeout
was too aggressive.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-09 06:13:49 +00:00
Rusty Russell 2caeab506b tests/test_lightningd.py: make test_blockchaintrack safe to re-run.
It gets confused if re-run due to flaky, since:
1. Using the same HSM, it generates the same spend and sees a previous one.
2. The block height numbers are off.

Fixes: #1479
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-09 06:13:49 +00:00
Rusty Russell ecebbe700b tests/test_lightningd.py: remove unneeded btc variable.
We have a 'bitcoind' global: getting it from inside one of the daemons
was a mistake I've copied widely.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-09 06:13:49 +00:00
ZmnSCPxj 757c059cd4 test_lightningd: Add closingd torture test. 2018-05-09 04:43:21 +00:00
Christian Decker 783327622f pytest: Mark test_blockchaintrack as flaky
The flakyness is probably due to timing issues when reorging.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-08 22:06:04 +02:00
Christian Decker fac1d24ce3 pytest: Use the port from the lightning node
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-08 22:06:04 +02:00
Christian Decker 5f7b0fdf92 pytest: Marking `test_permfail` as flaky
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-08 22:06:04 +02:00
Christian Decker 104645ba3a pytest: Fix flaky `test_closing_id` test
It simply wasn't waiting for us to register the peer before attempting to open a
connection. Moved into a separate test to be able rerun multiple times.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-08 05:43:35 +00:00
Rusty Russell 79902b48c7 test_lightningd.py: remove gratuitous 'addr' option from test_connect_by_gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell d40d22b68e gossipd: don't try to connect to non-routable addresses.
Someone could try to announce an internal address, and we might probe
it.

This breaks tests, so we add '--dev-allow-localhost' for our tests, so
we don't eliminate that one.  Of course, now we need to skip some more
tests in non-developer mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell af065417e1 gossipd: handle wildcard addresses correctly.
If we're given a wildcard address, we can't announce it like that: we need
to try to turn it into a real address (using guess_address).  Then we
use that address.  As a side-effect of this cleanup, we only announce
*any* '--addr' if it's routable.

This fix means that our tests have to force '--announce-addr' because
otherwise localhost isn't routable.

This means that gossipd really controls the addresses now, and breaks
them into two arrays: what we bind to, and what we announce.  That is
now what we return to the master for json_getinfo(), which prints them
as 'bindings' and 'addresses' respectively.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell 52917ff6c9 More flexible address wildcards, only add wildcard if nothing else.
1. Add special option where an empty host means 'wildcard for IPv4 and/or IPv6'
   which means ':1234' can be used to set only the portnum.
2. Only add this protocol wildcard if --autolisten=1 (default)
   and no other addresses specified.
3. Pass it down to gossipd, so it can handle errors correctly: in most cases,
   it's fatal not to be able to bind to a port, but for this case, it's OK
   if we can only bind to one of IPv4/v6 (fatal iff neither).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell fe96fe10c7 Clean up network options.
It's become clear that our network options are insufficient, with the coming
addition of Tor and unix domain support.

Currently:

1. We always bind to local IPv4 and IPv6 sockets, unless --port=0, --offline,
   or any address is specified explicitly.  If they're routable, we announce.
2. --addr is used to announce, but not to control binding.

After this change:

1. --port is deprecated.
2. --addr controls what we bind to and announce.
3. --bind-addr/--announce-addr can be used to control one and not the other.
4. Unless --autolisten=0, we add local IPv4 & IPv6 port 9735 (and announce if they are routable).
5. --offline still overrides listening (though announcing is still the same).

This means we can bind to as many ports/interfaces as we want, and for
special effects we can announce different things (eg. we're sitting
behind a port forward or a proxy).

What remains to implement is semi-automatic binding: we should be able
to say '--addr=0.0.0.0:9999' and have the address resolve at bind
time, or even '--addr=0.0.0.0:0' and have the port autoresolve too
(you could determine what it was from 'lightning-cli getinfo'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell 00537fde43 lightningd: deprecate --ipaddr in favor of --addr.
We're going to add sockets, and later onion addresses, so the current name
is bad.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell 6ba9a7cb57 tests/test_lightningd.py: move port into node.
We're about to change the JSONRPC, so let's put an explicit 'port' into
our node class.

We initialize it at startup time: in future I hope to use ephemeral ports
to make our tests more easily parallelizable.

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Rusty Russell 7d27501c94 tests/utils.py: remove unused connect routine.
It's broken anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-07 22:37:28 +02:00
Christian Decker 6231e9969f pytest: Give lightningd nodes a numeric ID to prefix logs
This used to be the port, but since we no longer have fixed ports, and we start
them in random order we can't easily distinguish them by the port anymore. Just
use a numeric ID that matches their lightning-dirs.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker 65cac1ba85 pytest: Add flaky dependency and mark as test_closing_different_fees
This is mainly just a stopgap solution until we get to stabilize the test.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker e00c7012cc pytest: Ensure unique test directories per test even if rerun
We add an attempt number to the test directory to improve the test-isolation and
allow for multiple reruns of the same test, without re-using any of the
lightning-dirs or bitcoin-datadirs.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker 071ef628db pytest: Use random ports for bitcoind and lightningd to allow parallel testing
Adds a new dependency, but totally worth it :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker bbeb44faac pytest: Make the directory of the NodeFactory configurable
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker 727d115296 pytest: Add py.test fixtures and migrate first example test
This is the first example of the py.test style fixtures which should allow us to
write much cleaner and nicer tests.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker b307df0002 pytest: Let `connect` start nodes in parallel
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker e3bac12683 pytest: Allow stdout to be dropped for bitcoind
We never really look at the output, and it's rather noisy, so we just stop
writing to the log.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker 4f63acc77c pytest: Add helper to start a number of nodes in parallel
Especially with valgrind this allows us to safe quite some time on multi-core
machines since startup is about the most expensive operation in the tests. In a
simple test spinning up 3 nodes this gave me about 25% - 30% test time
reduction. The effects will be smaller for single core machines, hoping Travis
handles these gracefully.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker 81b715aae9 pytest: Move wait_for to utils.py and make fund_channel a member
Slowly moving towards a more pythonic approach to the testing code.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 02:40:50 +00:00
Christian Decker 7aa13cc949 channel: Queue a channel_update to the peer upon funding_locked
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-05-07 01:10:48 +00:00
Rusty Russell ac51231166 lightningd: remove --dev-hsm-seed option.
We can create the hsm file from python directly; that works even if we
don't have DEVELOPER set, and is simpler.

We add a test that the aliases are correct.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-05 17:55:10 +02:00
ZmnSCPxj b1ffcc3b52 test_lightningd: Update close to use scid, add test for peer id and full cid. 2018-05-03 22:47:07 +02:00
Rusty Russell c6af2a8cb2 lightningd: loosen feerate minimum.
We're getting spurious closures, even on mainnet.  Using --ignore-fee-limits
is dangerous; it's slightly less so to lower the minimum (which is the
usual cause of problems).

So let's halve it, but beware the floor.

This is a workaround, until we get independent feerates in the spec.

Fixes: #613
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-01 18:45:04 +02:00
ZmnSCPxj 69cdfba3c8 gossip: Use gossiped node_announcement to locate nodes.
So we can get via address hint, DNS seed, or node_announcement
gossip.
2018-04-26 11:45:38 +00:00
Rusty Russell c6483a57d0 gossipd: give more distinct errors.
At least say whether we failed to connect at all, or failed cryptographic
handshake, or failed reading/writing init messages.

The errno can be "Operation now in progress" if the other end closes the
socket on us: this happens when we handshake with the wrong key and it
hangs up on us.  Fixing this would require work on ccan/io though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-26 05:47:57 +00:00
Rusty Russell a134ca9659 gossipd: use exponential backoff on reconnect for important peers.
We start at 1 second, back off to 5 minutes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-26 05:47:57 +00:00
Rusty Russell bc4809aa85 gossipd: make sure master only ever sees one active connection.
When we get a reconnection, kill the current remote peer, and wait for the
master to tell us it's dead.  Then we hand it the new peer.

Previously, we would end up with gossipd holding multiple peers, and
the logging was really hard to interpret; I'm not completely convinced
that we did the right thing when one terminated, either.

Note that this now means we can have peers with neither ->local nor ->remote
populated, so we check that more carefully.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-26 05:47:57 +00:00
Rusty Russell 72c459dd6c gossipd: keep reaching struct only when we're actively connecting, and don't retry
1. Lifetime of 'struct reaching' now only while we're actively doing connect.
2. Always free after a single attempt: if it's an important peer, retry
   on a timer.
3. Have a single response message to master, rather than relying on
   peer_connected on success and other msgs on failure.
4. If we are actively connecting and we get another command for the same
   id, just increment the counter

The result is much simpler in the master daemon, and much nicer for
reconnection: if they say to connect they get an immediate response,
rather than waiting for 10 retries.  Even if it's an important peer,
it fires off another reconnect attempt, unless it's actively
connecting now.

This removes exponential backoff: that's restored in next patch.  It
also doesn't handle multiple addresses for a single peer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-26 05:47:57 +00:00
Rusty Russell 5db8454755 test_lightningd.py: make tests more robust by suppressing reconnects.
Got some intermittant failures, mainly caused by the tests being slow
enough that the peer reconnected.  We should always suppress
reconnection if we can, and not stress too much in the !DEVELOPER case
where we can't.

We should turn off dev-no-reconnect *always* unless told we will
reconnect, and since we can't if !DEVELOPER, don't do the connection
check there.

Instead of adding an option to line_graph, we remove it in favor
of connect (since we only use it with n=2 anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-26 05:47:57 +00:00
ZmnSCPxj 079778e357 invoice: Check duplicate preimage when explicitly sprcified.
Reported-by: @mcudev
2018-04-26 05:47:09 +00:00
Christian Decker ac9e5581c8 pytest: Start nodes with --rescan=1
This shaves off about 15% of our integration testing suite on my machine. It
assumes we never reorg below the first block the node starts with, which is true
for all tests, so it's safe.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-25 14:33:38 +02:00
Christian Decker 74fa107578 pytest: Add a test for the new --rescan option
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-25 14:33:38 +02:00
Christian Decker 96352858d6 chaintopology: Simplify rescan offset computation
Simplification of the offset calculation to use the rescan parameter, and rename
of `wallet_first_blocknum`. We now use either relative rescan from our last
known location, or absolute if a negative rescan was given. It's all handled in
a single location (except the case in which the blockcount is below our
precomputed offset), so this should reduce surprises.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-25 14:33:38 +02:00
Christian Decker 8e7ac53f5a pytest: test onchaind restarts from the DB 2018-04-25 14:33:38 +02:00
ZmnSCPxj eb42804fcc invoice: Support providing preimage when making invoice. 2018-04-24 11:54:02 +02:00
Rusty Russell e2ba1d2290 test_lightningd.py: test_closing_different_fees must wait for txs to hit mempool
Careful log examination revealed that we were generating a block before one
of the mutual close txs had entered the mempool.  This is rare because it
means that both peers have to be too slow.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-23 20:18:15 +00:00
Rusty Russell d0bfd8407a test_lightningd.py: catch unexpected reconnections.
I had a weird failure which was caused by an unexpected disconnect and
reconnecct.  Since we are prersistend and recover from these, they can
slip through our tests; most tests don't involve reconnection, so we
need to catch this explicitly.

For the connect() helper, we always suppress reconnection; tests which
want it all want other options so don't use this helper anyway.  (Actually,
after I said that, test_closing_while_disconnected was added when I
rebased, which did require it, so I had to open-code that one).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-23 20:18:15 +00:00
ZmnSCPxj, ZmnSCPxj jxPCSmnZ d6bf7930b8 Loosen` close` timeout in `test_closing_different_fees` 2018-04-23 05:24:46 +00:00
ZmnSCPxj 0b331a2b60 test_lightningd.py: Clean up some uses of 'close' RPC. 2018-04-23 05:24:46 +00:00
Rusty Russell 68758a5d42 json_close: test that it works while disconnected.
It should, indeed, close once they reconnect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-23 05:24:46 +00:00
ZmnSCPxj 2cee1ab20f peer_control: Make close wait for complete closure, with timeout.
Also report tx and txid, and whether we closed unilaterally or
bilaterally, if we could close the channel.

Also make a manpage.

Fixes: #1207
Fixes: #714
Fixes: #622
2018-04-23 05:24:46 +00:00
ZmnSCPxj 78a074af5a test_lightningd: Also test receive of P2WPKH is correct.
For completeness.
Existing test will check P2SH-P2WPKH address only.
P2WPKH seems OK but test it anyway in case of regression.
2018-04-22 12:51:03 +02:00
ZmnSCPxj 3bf626a750 walletrpc: Correct generation of P2SH-P2WPKH addresses for listfunds. 2018-04-22 12:51:03 +02:00
ZmnSCPxj c2e85cd051 test_lightningd: Check listfunds address matches newaddr paid to. 2018-04-22 12:51:03 +02:00
Christian Decker 140123b4c8 pytest: Add local_add tests to test_gossip_persistence and cleanup
Mixing the test into the existing test allows us to reduce the execution, but
adds a bit of complexity, so I added two small helpers.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-22 12:50:34 +02:00
Christian Decker 2de7f622cb gossip: Add an explicit debug message when handing back a peer
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-09 00:21:20 +00:00
Christian Decker 1757732236 pytest: Fix flaky test_blockchaintrack test
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-09 00:21:20 +00:00
Rusty Russell 09c4203767 bolt11: allow multiple fallback addresses.
We can have more than one; eg we might offer both bech32 and a p2sh
address, and in future we might offer v1 segwit, etc.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-06 14:26:53 +02:00
Christian Decker 4da3d407b4 pytest: Use a dict to pass options to lightningd
This allows us to have some default options that can then be overridden easily
on a per-test basis.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-04-06 06:52:22 +00:00
Rusty Russell 5f1c77d249 test_lightning.py: add test for onchain with different feerates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-04 02:31:41 +00:00
Rusty Russell 1f9ad06056 lightningd: allow us to close channel while still awaiting lockin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-03 23:22:59 +02:00
Rusty Russell 71de234255 test_lightningd.py: test shutdown while lockin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-03 23:22:59 +02:00
Christian Decker e41d565103 pytest: Add a test for the gossip perstistence
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-30 16:35:00 +02:00
Christian Decker 428f5608fa pytest: Testing for a node whose channels were closed will not work
In the next commit we remove channels whose outpoint was spent from our network
view, so checking for it will not work anymore.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-30 16:35:00 +02:00
Christian Decker 1b604d5dd6 pytest: Wait for the funding transaction to be in the mempool
This can lead to failures in the valgrind configuration.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-30 16:34:31 +02:00
Rusty Russell a85ead7058 invoice: allow numeric labels again.
Fixes: #1291
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-30 00:01:10 +02:00
Christian Decker f341b508e1 pytest: Add checks that funding change and to_us get confirmed
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-27 23:17:17 +00:00
Rusty Russell ed9093fcbd json: allow strange characters through our JSON parser.
Fixes: #387
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-26 00:20:53 +00:00
Rusty Russell 54431d2b08 lightningd: escape our own alias when we print it in logs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-26 00:20:53 +00:00
Rusty Russell 4139f06780 json_getnodes: escape the alias when we print it out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-26 00:20:53 +00:00
Rusty Russell a077c3defb lightningd: test for weird aliases.
Test has inverted conditions to make it "pass", we fix as we go.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-26 00:20:53 +00:00
Rusty Russell 8c838ebd76 test_lightning: Write test for invoices with weird characters.
It's currently expected to fail.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-26 00:20:53 +00:00
Mark Beckwith 489d8d04a6 Fund all (#1249)
* Funding channel with "all"
* Added test for fund all
2018-03-21 00:06:04 +01:00
ZmnSCPxj 5557958c47 test_lightningd: Add test for autocleaninvoice. 2018-03-20 17:25:51 +01:00
ZmnSCPxj de853b4847 test_lightningd.py: Test delexpiredinvoice. 2018-03-20 17:25:51 +01:00
ZmnSCPxj 6391645b83 test_lightningd: Check payment fields of pay return value. 2018-03-19 00:27:55 +00:00
ZmnSCPxj 234d67da34 channel_fail_permanent: Use a channel-level error, not an all-channels error.
Fixes: #1229
2018-03-17 13:26:58 +01:00
Rusty Russell 1b9791f0f5 pay: delete HTLC when we delete payment.
This fixes the root cause of https://github.com/ElementsProject/lightning/issues/1212
where we deleted the payment because we wanted to retry, then retry failed
so we had an (old) HTLC without a matching payment.  We then fed that
HTLC to onchaind, which tells us it's missing, and we try to fail the
payment and deref a NULL pointer.

Fixes: #1212
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-16 04:01:06 +00:00
Rusty Russell d8cecf03dd test_lightningd.py: add test to trigger crash bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-16 04:01:06 +00:00
ZmnSCPxj f2aa15ffee test_lightningd.py: Check additional fields of sendpay. 2018-03-16 01:24:43 +00:00
ZmnSCPxj ed3984f444 test_lightningd.py: Modify test_payment_duplicate_uncommitted.
`pay` command will now wait for payment to succeed if the
payment has been initiated.
2018-03-14 05:33:09 +00:00
ZmnSCPxj 752b7f5277 test_lightningd.py: Update for nonblocking sendpay. 2018-03-14 05:33:09 +00:00
Christian Decker 3e9e19951b pytest: Re-enable formerly flaky test_reconnect_normal
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-08 05:53:58 +01:00
Rusty Russell 25cb8caae8 onchaind: fix confusing message about delayed txs.
We say "in N blocks" but we actually mean "N blocks after this tx" which is
actually N-1 or less.  Change wording and tighten tests which misunderstood
this.

Also, the 'assert not l1.daemon.is_in_log('onchaind complete, forgetting peer')'
are unlikely to work until the daemon has actually seen the block, so add
sync_blockheight before all of those.

These changes reveal some sloppy testing, which we fix.
  
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-07 18:55:51 +01:00
Rusty Russell a48c300df2 onchaind: fix too-eager OUR_HTLC_TIMEOUT_TX.
With the following patch applied, we could clearly see onchaind try to
broadcast the timeout tx one block too early:

	sendrawtx exit 26, gave error code: -26?error message:?non-final (code 64)?

This is because of an out-by-one error in calculating the relative
depth required, since the out->tx_blockheight is already 1 before the
current block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-07 18:55:51 +01:00
Rusty Russell eb52dde5b5 onchaind: add a new state for where we're not producing an output.
DONATING_TO_MINERS is pretty clear, I think.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-07 18:55:51 +01:00
Rusty Russell 0489aabfbe test_lightningd.py: test onchaind when all turn into fees.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-07 18:55:51 +01:00
Rusty Russell 27c4e926bd test_lightningd.py: test case where onchaind restarts with different fees.
This was revealed in #1114; onchaind isn't actually completely idempotent
due to fee changes (and the now-fixed change in keys used).

This triggers the bug by restarting with different fees, resulting in
onchaind not recognizing its own proposal:

	2018-03-05T09:38:15.550Z lightningd(23076): lightning_onchaind-022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59 chan #1: STATUS_FAIL_INTERNAL_ERROR: THEIR_UNILATERAL/OUR_HTLC spent with weird witness 3

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-07 18:55:51 +01:00
John Barboza 30b290cb8f Explicit error message when disconnect fails
Whether the peer is not connected or not gossiping.
2018-03-07 16:14:01 +01:00
John Barboza 9a2950628d test: disconnect command 2018-03-07 16:14:01 +01:00
Christian Decker c9985805cd pytest: Add blockchain tracking test
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-06 18:59:27 +01:00
ZmnSCPxj 8e8d7c2aba pay/sendpay: Use spec names for rhash and r. 2018-03-05 20:21:37 +00:00
ZmnSCPxj c03f584e19 test_lightningd.py: Adapt to new help output. 2018-03-03 19:40:22 +01:00
Rusty Russell d7b5882f61 routing: don't free a single nc on prune, only entire channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-03 19:29:35 +01:00
Rusty Russell 33726b0a08 gossip: instead of refresh interval, have routing know prune_timeout.
This is twice the 'update_channel_interval' we get handed.

We delete the non-existent channel_add_connection and delete_connection
declarations from the header too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-03 19:29:35 +01:00
Rusty Russell b7bf414ac4 gossipd: prune announced-but-not-updated channels eventually.
We currently give them a free pass.  The simplest fix is to give them
an old timestamp on initialization.

We still skip unannounced channels, on the assumption that they're
ours.  And we set the last_update_timestamp to -1 when we convert to
gossip_getchannels_entry to indicate no update.

This breaks the DEVELOPER=1 pruning test, since we hardcode the 1
week timeout.  That's fixed in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-03 19:29:35 +01:00
John Barboza 35ce13179c Add tests for peer information containing alias 2018-03-02 02:25:30 +00:00
John Barboza 6f14736803 revert 0e40c8 and renable test_withdraw
I initially disabled this until 0.16 because the withdraw command
was modified to require 'brct1' addresses for regtest.
But commit bd07a9 allows a regular testnet address to work
just as well. So renable this check.
2018-03-01 23:32:58 +01:00
Rusty Russell 742885c95e test_lightning.py: reduce 121 to 101 blocks, remove segwit comment.
FWIW, the tests without valgrind take 662 seconds before we reduced
the number of blocks, and only 648 seconds now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-03-01 03:52:21 +00:00
Jan Sarenik 0017b9b363 test_lightningd.py: Enable bech32 test for DEVELOPER
Since bitcoind 0.16 is already released it is safe to enable
this test already.
2018-03-01 03:52:21 +00:00
Christian Decker 565059a12f pytest: Require bitcoin-0.16.0 and reduce number of generated blocks
0.16.0 is required since we rely on it for some tests and the block
reduction allows us to waste less time during setup. 121 blocks were
chosen so that we have at least one mature output to spend.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-03-01 03:52:21 +00:00
ZmnSCPxj, ZmnSCPxj jxPCSmnZ 590a9a657d test_lightningd.py: Silence F841
Just do not assign to variable.
2018-02-28 11:17:08 +01:00
ZmnSCPxj 6792acf04c test_lightningd: Add some more testing for waitinvoice. 2018-02-28 11:17:08 +01:00
Rusty Russell c7718aa2e4 test_lightningd.py: repair flaky test_bitcoin_failure
If bitcoind is still fetching blocks, we might accidentally inject
the failure between getblockhash and getblock.  That's OK, but
it's not the failure we test for.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-28 01:51:40 +00:00
Rusty Russell 58fae47868 test_lightningd.py: more minor flake8 fixes.
Looks like rebasing the flake8 branch caused breakage, as new violations
had occurred since that check was written

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-26 22:36:05 +01:00
practicalswift d2109bfe0c Fix flake8 warnings 2018-02-26 12:38:00 +01:00
John Barboza 0e40c85691 Disable test until 0.16
This test generates a pre-bip173 testnet P2PKH address with bitcoin 0.15.1
which fails under the new verification checks for the bip173 network name.
This should be renabled with bitcoin 0.16 which can generate a bip173 address
for regtest with the bcrt prefix.
2018-02-26 03:09:15 +00:00
John Barboza b028a0a439 invoice: add fallback address to invoice command
* Modifies invoice command to have the following format
  invoice <msatoshi> <label> <desc> <?expiry> <?fallbackaddr>
* Adds support for Segwit bcrt1 addresses for withdraw
* Add test case for fallback address in invoice creation
* Create a common json_tok_address_scriptpubkey to be used
  by invoice and withdraw commands.
2018-02-26 03:09:15 +00:00
practicalswift bb12c234f2 Fix recently introduced PEP-8 violations 2018-02-24 11:44:32 +01:00
Christian Decker 330682abfe pytest: Remove any existing lightning-dirs before running
This was causing some issue when automatically rerunning tests on
travis.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-02-24 10:40:03 +01:00
Rusty Russell 737a7148b5 test_lightningd.py: make sure we recover from transient bitcoin failures.
We can do similar tricks to test other things, even to run without a
real bitcoind for faster testing, but for now we simply exit if a magic
file says so.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-24 10:37:28 +01:00
Rusty Russell 309dc96f97 fixup! More test fixups from last patch. 2018-02-23 18:02:00 +01:00
Rusty Russell cccdb53bec channel_states: fold all the ONCHAIND states into one.
The billboard is now far more useful to tell what's going on, and this
gets us closer to a state == owner mapping.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-23 18:02:00 +01:00
Rusty Russell 866ba00445 test_lightning.py: test in billboard in various closing cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-23 18:02:00 +01:00
practicalswift e18948a6a9 Remove unused variables. Fix flake8 warnings. 2018-02-22 22:21:31 +01:00
Rusty Russell b8c636514b pong: embed version string into ping replies if DEVELOPER=1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-22 12:24:26 +01:00
Rusty Russell 85f1a9b825 lightningd: append (as much as we can) version into default name if DEVELOPER=1
This helps for bug reporting, where we may need to diagnose the other
end.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-22 12:24:26 +01:00
practicalswift ae7d857c44 Make Python code conform to the Python PEP 8 style guide 2018-02-22 10:47:19 +01:00
Rusty Russell b536e97df2 dev_ping: don't crash with silly values.
It's a dev command, but still.

Fixes: #985
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-22 01:10:09 +00:00
Rusty Russell 4cf274b1c0 lightningd/opening_control: routines to control openingd (move from peer_control.c)
We also fold opening_got_hsm_funding_sig() into the caller; it was
previously a callback before we decided to always use the HSM
synchronously.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-20 22:36:21 +01:00
Rusty Russell d47d85fa30 wallet: add list of upgrades.
Useful for debugging a db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-20 01:38:32 +00:00
ZmnSCPxj ea3ee01215 peer_control: Support xxx@yyy:zzz form for connecting. 2018-02-19 11:59:11 +01:00
Rusty Russell 4653493507 test_lightningd: make test_payment_failed_persistence more reliable.
We don't want the invoice to fail locally, which can happen if we're
slow enough.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-19 02:56:51 +00:00
Rusty Russell e17b0ebcb4 channeld: map htlc add/remove errors to names.
I couldn't figure out what 'Bad peer_add_htlc: 7' meant!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-19 02:56:51 +00:00
Rusty Russell 8db8c51201 lightningd: struct uncommitted_channel for opening channels.
Each peer can have one 'uncommitted' channel, which is in the process
of opening.  This is used for openingd, and then on return we convert
it into a full-fledged struct channel and commit it into the database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-19 02:56:51 +00:00
Anton Astafiev 8787766d44 Pylightning (#1003)
* Fix dev_setfees to set slow and normal fees correctly.

Due to a bug def_setfees(100, slow=100) would instead set immediate and
normal fees to 100. This behavior has been updated to set fees to
correct values, make the values truly optional as per documentation and
unit test this behavior.

* Fix pay() to set msatoshi, description and risk factor properly

Due to a bug pay(invoice, description='1000') resulted in payment of
1000 msatoshi instead. This was fixed and covered with tests.

* Fix named args in listpayments, listpeers and connect

* Do not pass None to methods where it is default value

* Make description on invoice and pay match.

Suggested-by: @ZmnSCPxj

* Fix dev_setfees to set slow and normal fees correctly.

Due to a bug def_setfees(100, slow=100) would instead set immediate and
normal fees to 100. This behavior has been updated to set fees to
correct values, make the values truly optional as per documentation and
unit test this behavior.

* Fix pay() to set msatoshi, description and risk factor properly

Due to a bug pay(invoice, description='1000') resulted in payment of
1000 msatoshi instead. This was fixed and covered with tests.

* Fix named args in listpayments, listpeers and connect

* Do not pass None to methods where it is default value

* Make description on invoice and pay match.

Suggested-by: @ZmnSCPxj
2018-02-16 22:40:55 +01:00
Rusty Russell 43ec3f0761 jsonrpc: allow multiple commands in-flight from single JSON connection.
We now keep a list of commands for the jcon instead of a simple
'current' pointer: the assertions become a bit more complex, but
the rest is fairly mechanical.

Fixes: #1007
Reported-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-16 17:51:04 +01:00
ZmnSCPxj 0489c7eb93 test_lightningd: Change test_pay0 to use sendpay. 2018-02-16 13:08:29 +01:00
ZmnSCPxj d9163dbb4f test_lightningd: Modify test to remove repeated pay attempt. 2018-02-16 13:08:29 +01:00
Rusty Russell df4c669a60 test: test for funds sent while we were offline.
As described by @lvaccaro in #990.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-16 13:07:12 +01:00
Rusty Russell c8f9ea7bf7 fixup: test_multirpc does not terminate.
We need to set expiry, otherwise waitinvoice would take 1 hr, and we
can't read once for every cmd, since each read may consume more than
a single result, and we block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-16 12:56:48 +01:00
Christian Decker cab7e3e43a pytest: Add failing test with JSON-RPC requests crossing each other
Suggested-By: @ZmnSCPxj
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-02-16 12:56:48 +01:00
Rusty Russell a08bcfdbd3 jsonrpc: don't crash on multiple commands at once.
Once we read a command, we are supposed to io_wait until it finishes.
However, we are actually woken in two places: when it's complete
(which is correct), and when it's written out (which is wrong).

We don't care when it's written out, only when it's finished:
refactor to make json_done() free and NULL the old ->current,
rather than have the callers do it.  Now it's clear that it's
ready for both new output and new input.

Fixes: #934
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-16 12:56:48 +01:00
Christian Decker db5c0aa6c1 pytest: Make pep8 happy(er)
A number of fixes to make our code more pythonic
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Co-authored-by: Guido Dassori @gdassori
2018-02-15 23:56:38 +00:00
Christian Decker 0e40b9b08c pylightning: Switch to dict-based params instead of positional
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Co-authored-by: Guido Dassori @gdassori
2018-02-15 23:56:38 +00:00
Rusty Russell d822ba1eee lightningd: allow a new channel open from peer if no *active* channels.
And return the correct error message for the channel they give, if
they try to re-establish on an error channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-14 11:31:58 +01:00
Rusty Russell e20fff9340 lightningd: remove almost all other peer2channel / channel2peer shims.
This final sweep only keepl peer2channel within peer_control.c for
the reconnect case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-14 11:31:58 +01:00
Rusty Russell 32411de90e lightningd: split struct peer into struct peer and struct channel.
Much like the database; peer contains id, address, channel contains
per-channel information.  Where we create a channel, we always create
the peer too.

For the moment, peer->log and channel->log coexist side-by-side, to
reduce some of the churn.

Note that this changes the API to dev-forget-channel: if we have more
than one channel, we insist they specify the short-channel-id.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-14 11:31:58 +01:00
Christian Decker 96667a3028 fixup! pytest: Test storing last_htlc_sigs and onchaind fund recovery 2018-02-11 01:13:07 +01:00
Christian Decker ea9aa95625 pytest: Test storing last_htlc_sigs and onchaind fund recovery
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-02-11 01:13:07 +01:00
Rusty Russell d5effcb961 test_lightning: fix race on testing, esp. test_closing_different_fees.
We get intermittant failure: WIRE_UNKNOWN_NEXT_PEER (First peer not ready)
because CHANNELD_NORMAL and actually telling gossipd that the channel
is available are distinct things: we need both.

(For test_closing_different_fees, we were testing CHANNELD_NORMAL on
the peer, not on l1, too).

But we may also directly send the announcement sigs if the height is
sufficient, so the simplest is to unify the messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-09 13:01:19 +01:00
practicalswift 4f4756bd20 Fix a-vs-an typos 2018-02-08 22:49:34 +01:00
Rusty Russell cc9ca82821 status: separate types for peer failure vs "impossible" failures.
Ideally we'd rename status_failed() to status_fatal(), but that's
too much churn for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 6ca0c6e0ec test_lightningd: race fix for activation of channels in test_gossip_jsonrpc
```
>       assert [c['active'] for c in l2.rpc.listchannels()['channels']] == [True, True]
E       AssertionError: assert [True, False] == [True, True]
E         At index 1 diff: False != True
E         Full diff:
E         - [True, False]
E         + [True, True]
```

We don't actually wait that l2's gossipd has also processed the message.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Christian Decker b3d4e161b5 pytest: Add test for `dev-forget-channel`
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-02-07 00:47:13 +00:00
Rusty Russell d77972e827 test_lightningd.py: test io logging and toggling with SIGUSR1
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-07 00:46:49 +00:00
ZmnSCPxj 711d8822b0 test_lightning: Update to new error message format. 2018-02-06 17:05:12 +01:00
Rusty Russell b3534462e0 test_lightningd.py: fix race in test_fee_limits.
Sometimes the super-low-fee commitment tx succeeds, and we see
that 'sendrawtx exit 0' instead of the one we're expecting.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-06 02:50:30 +00:00
Rusty Russell 07c275e436 test_lightningd: fix race in test_closing_different_fees
fundrawtransaction returns before the actual sendrawtx, so we can
end up mining blocks before it's sent, thus not having enough confirms.

We handle this correctly in fund_channel, but this test open-codes it
for speed with multiple peers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-06 02:50:30 +00:00
ZmnSCPxj dcafce81ad test_lightningd: Increase times in expiration waiting test.
Fixes: #916

Makes it less flaky, as, the increased times reduce the
relative delay in processing by slower machines.
2018-02-06 02:50:30 +00:00
Rusty Russell 37670c04e0 test_lightning.py: disable test which needs bitcoind 0.16 (for now!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-06 01:37:58 +00:00
William Casarin 3d27bbb47d test: test_bech32_funding
1. Test wallet funding to a bech32 p2wpkh address
2. Test channel opening with this address

Signed-off-by: William Casarin <jb55@jb55.com>
2018-02-06 01:37:58 +00:00
William Casarin 0e59e091e7 test: switch invoice tests to use proper bip173 name
Signed-off-by: William Casarin <jb55@jb55.com>
2018-02-06 01:37:58 +00:00
Christian Decker 81551b2e60 pytest: Increate timeout when waiting for gossip
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-02-05 18:28:39 +00:00
Christian Decker 739e78a8c7 gossip: Reduce verbosity
Now we only report serious failures, or messages that actually changed
the local view.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-02-05 18:28:39 +00:00
ZmnSCPxj 0925e404d6 test_lightningd.py: Test new expiration system in more detail. 2018-02-05 08:52:42 +00:00
ZmnSCPxj 4ac5d123d6 test_lightningd: Test missing arguments. 2018-02-02 23:45:18 +00:00
Rusty Russell a497a9943d test_lightning.py: speed up test_closing_different_fees
It was taking over 10 minutes under valgrind, causing Travis to time it out.

This shrinks it to its essential tests, and also batches.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 20:33:49 +01:00
Rusty Russell f527f39613 test_lightning: add tests for closing with differetn feerates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 20:33:49 +01:00
alaniz b7856e3512 Onboarding: tweak grammar conventions for RPC responses 2018-02-02 06:16:23 +00:00
Rusty Russell d1ab2e05f4 test_lightningd.py: test lightning-cli.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 00:05:00 +01:00
Rusty Russell 327e2b8fd8 JSONRPC: listconfigs command.
With the new 'human-readable' mode of lightning-cli, this actually produces
a valid config file.  It's a bit hacky though...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 00:05:00 +01:00
Rusty Russell 0600aac68f lightningd: rename --no-reconnect to --dev-no-reconnect.
It's a dev option.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 00:05:00 +01:00
cornwarecjp 6112df3a90 Bring back the time-out on the start-up of bitcoind, but make it 60 seconds instead of the original 10 seconds. This is much larger than the normal start-up time of bitcoind (so we should never hit the time-out), but still lets a test terminate in a reasonable time in the unlikely case that bitcoind hangs. 2018-02-01 23:00:43 +01:00
cornwarecjp 896f7ef37d Make it possible to run the test suite in Python 3.4 (bytes.hex was introduced in 3.5) 2018-02-01 23:00:43 +01:00
cornwarecjp 0b0708d86a Bitcoind should always eventually finish starting, but there is no guarantee on how long it takes, so don't apply a time-out. In case bitcoind hangs indefinitely, the test will hang indefinitely too, but this should be solved in bitcoind, not in the test code. 2018-02-01 23:00:43 +01:00
cornwarecjp 859070a5ed Terminate LightningD if, for some reason, it doesn't start properly. This cleans up the environment for future test runs. 2018-02-01 23:00:43 +01:00
cornwarecjp 71581ff96e Terminate BitcoinD if, for some reason, it doesn't start properly. This cleans up the environment for future test runs. 2018-02-01 23:00:43 +01:00