Commit Graph

144 Commits

Author SHA1 Message Date
lisa neigut a39c97c960 channeld: support private channel creation, fixes #2125
Adds a new 'announce' field for `fundchannel`, which if false
won't broadcast a `channel_announcement`.
2018-12-08 15:15:55 -08:00
lisa neigut eab992cecd py-tests: rename 'announce' to 'wait_for_announce'
Better description of what the option actually does -- if true
waits for the announcement messages to be generated and exchanged.
2018-12-08 15:15:55 -08:00
Rusty Russell fab5027d64 configure: add --experimental-features flag (default: == DEVELOPER).
This will be used for option_simplified_commitment and other
still-being-specified options.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-06 23:11:51 +01:00
Rusty Russell ac5002a79e pytest: add hack to force options into a given order.
Needed for testing plugin options which are order-senditive.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-05 01:22:55 +01:00
Christian Decker 522c3039ec pytest: Pretty print the memleak output in tests
It wasn't JSON formatted either so there was no nice pretty-printing
way. This jsonifies and pretty prints it.

Signed-off-by: Christian Decker <@cdecker>
2018-11-29 23:01:11 +00:00
Christian Decker e27b2ea69b pytest: Add a test for the plugin option passthrough
Signed-off-by: Christian Decker <@cdecker>
2018-11-26 22:53:37 +00:00
Rusty Russell 1d7b287439 pytest: speed up test_restart_many_payments when !DEVELOPER.
Because gossip in this case takes up to a minute, this test took 10
minutes.  The workaround is to do the waiting-for-gossip all at once.

Now it takes 362 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-22 05:15:42 +00:00
Rusty Russell 2c582cbcfb pytest: test that we don't revert config handling again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-21 01:08:42 +00:00
Rusty Russell aab91557f3 Travis: eliminate 4 slowest tests, with new SLOW_MACHINE flag.
In one case we can reduce, in the others we eliminated if VALGRIND.

Here are the ten slowest tests on my laptop:

469.75s call     tests/test_closing.py::test_closing_torture
243.61s call     tests/test_closing.py::test_onchain_multihtlc_our_unilateral
222.73s call     tests/test_closing.py::test_onchain_multihtlc_their_unilateral
217.80s call     tests/test_closing.py::test_closing_different_fees
146.14s call     tests/test_connection.py::test_dataloss_protection
138.93s call     tests/test_connection.py::test_restart_many_payments
129.66s call     tests/test_gossip.py::test_gossip_persistence
128.73s call     tests/test_connection.py::test_no_fee_estimate
122.46s call     tests/test_misc.py::test_htlc_send_timeout
118.79s call     tests/test_closing.py::test_onchain_dust_out

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 7d614aaf25 pytest: really remove all bitcoin generate RPC calls.
generate was deprecated some time ago, so we added the generate_block()
helper.  But many calls crept back in, and git master refuses it.

(test_blockchaintrack relied on the return value, so make generate_block
return the list of blocks).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 38e6aa66ff python: quieten modern flake8.
After Ubuntu 18.10 upgrade, lots of new flake8 warnings.

$ flake8 --version:
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 1.6.0) CPython 3.6.7rc1 on Linux

Note it seems that W503 warned about line breaks before binary
operators, and W504 complains about them after.  I prefer W504, so
disable W503.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-28 16:03:12 +01:00
Rusty Russell 0bc80c4687 pytest: make sure we wait for node announcements in line_graph too.
Occasional failure in test_fulfill_incoming_first where the channel
closed before the final message from dev_disonnect was read.  Cause
was the peer writing a gossip msg and failing due to ECONNRESET, before
it read the final message.

(Managed to reproduce under strace -f, FTW).

This is really a symptom of the fact that line_graph's announce=True
didn't wait for node announcements.  Let's do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-24 16:18:32 +02:00
Christian Decker 6efdd5275c pytest: Test that we record forwardings correctly
Adapts the `test_forward_stats` test to include checks for the
`forwarded_payments` table. Will add checks for the `listforwardings`
RPC call next.

Signed-off-by: Christian Decker <@cdecker>
2018-10-19 21:58:27 +00:00
Rusty Russell fcb5310873 pytest: make wait_for do exponential backoff, start at 0.25 seconds.
This doesn't alter runtime very much, but does reduce log spam.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-10 06:10:42 +00:00
Rusty Russell b3fb23f921 pytest: make line_graph wait for gossip propagation if announce param is True
This is what the callers want; generalize it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-09 23:17:54 +00:00
lisa neigut a9bf1f5573 tests: quiet down DeprecationWarnings for escape sequences
Nuke all the `DeprecationWarning: invalid escape sequence
\[` messages that show up when you run python tests.
2018-10-08 13:18:31 +02:00
Rusty Russell b0d9182fc3 pytest: allow log_all_io through get_nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-28 18:20:17 +02:00
Rusty Russell a90c200fcb pytest: wait_channel_quiescent helper to wait for resolved htlcs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-27 22:57:19 +02:00
Rusty Russell edb74260f4 pytest: more reliable onchain tests.
We extract the tx from the logs, and then we wait until that hits
the mempool.  This is more reliable than 'sendrawtx' in the logs,
which might catch a previous sendrawtx; it's also more explicit
that we expect that tx exactly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-21 00:28:44 +00:00
lisa neigut b1ceaf9910 gossipd: Update BOLT-split flags in channel_update
BOLT 7's been updated to split the flags field in `channel_update`
into two: `channel_flags` and `message_flags`. This changeset does the
minimal necessary to get to building with the new flags.
2018-09-21 00:24:12 +00:00
Rusty Russell e41e1a177e pytest: wait until mock is called for set_feerates.
Got a spurious failure in test_no_fee_estimate; we fired too soon from the logs (presumably
we raced in on the first response, but estimatesmartfee gets called 3 times).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-20 09:12:28 +02:00
Rusty Russell 252bbe1d2d pytest: don't wait for sendrawtx, wait for expected tx.
In particular, test_no_fee_estimate was flaky due to seeing the funding
tx being sent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-19 13:04:01 +02:00
Christian Decker f29f92a5fe pytest: Clean up bitcoind_cmd_override, it's no longer used 2018-09-16 00:05:34 +02:00
Christian Decker 16869e3fe6 pytest: Use the bitcoind proxy to mock feerates 2018-09-16 00:05:34 +02:00
Christian Decker aa80a330f1 pytest: Remove auto-proxying in favor of a per-node btc proxy 2018-09-16 00:05:34 +02:00
Christian Decker e132dffa0b pytest: Add an RPC proxy inbetween bitcoind and bitcoin-cli
This is a simple reverse proxy that `bitcoin-cli` can talk to when invoked by
`lightningd`. It allows us to trace `bitcoin-cli` calls, and intercept calls to
mock the replies, better than the current bash-script based method.
2018-09-16 00:05:34 +02:00
Christian Decker f505a9418b pytest: Fix lint error 2018-09-16 00:05:34 +02:00
Rusty Russell 312209ad60 pytest: support simple subdaemon debugging.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-04 14:36:05 +02:00
Rusty Russell 9d517ddc1d options: remove default-fee-rate now we don't use it.
And no more filtering out messages, as we should no longer spam the
logs with them (the 'Connected json input' one was removed some time
ago).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-25 00:33:12 +00:00
Rusty Russell 4c891f4661 common: log when we toggle IO logging, don't edit env in tests!
Tests were failing when in the same thread after a test which set
log_all_io=True, because SIGUSR1 seemed to be turning logging *off*.

This is due to Python using references not copies for assignment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-23 14:46:22 +02:00
Rusty Russell 9f044305db pytest: dev env var LIGHTNINGD_DEV_LOG_IO turns io logging on immediately.
This is required for the next test, which has to log messages from channeld
as soon as it starts (so might be too late if it sends SIGUSR1).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-23 14:46:22 +02:00
Rusty Russell cda6c97a57 pytest: allow NodeFactory to create non-started peers.
Useful it we want to intercept bitcoin-cli first.

We move the getinfo() caching into start(), as that's when we can actually
use RPC.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-22 12:13:23 +02:00
Rusty Russell 7eed9cba90 pytest: always override estimatesmartfee.
It (almost?) always fails for regtest; best to override it directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-22 12:13:23 +02:00
Rusty Russell 7a77b81dff pytest: always use the fake-bitcoin-cli, name it more appropriately.
We're going to use it to override specific commands.  It's non-valgrinded
already since we use '--trace-children-skip=*bitcoin-cli*' so the overhead
should be minimal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-22 12:13:23 +02:00
Rusty Russell 5e20eedb41 pytest: allow more elaborate bitcoin-cli override.
In particular, this lets us intercept individual commands, such as
estimatesmartfee.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-22 12:13:23 +02:00
Rusty Russell e59cbb3e2c pytest: make sure receiving peer's openingd is ready.
There's now a potential race: the source peer connect returns, but in
destination peer the master hasn't read the connect message from
connectd, so the peer isn't in listpeers yet.

(Previously the connection stayed in connectd, so there was no such
window).

This is an occasional issue in a few places.

Note that we take the opportunity to speed up test_disconnectpeer too
while we're there.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 5135d3ad7d pytest: make sure we truncate disconnect file for blackhole processes.
In particular, I found lightning_openingd processes after running
tests.  When we use the dev_disconnect blackhole '0' option, they
stick around until the dev_disconnect file is truncated (there is only
so much you can do with only a file descriptor), so let's do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 8668b0028e pytest: make sure both sides of channel are ready before returning from fund_channel
The following changes revealed this race, where expecting listchannels()
to contain two channels immediately after fund_channel() was racy.

We also derive the short_channel_id first, so we can search logs for the
exact messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 329270525c pytest: only use dev-allow-localhost when needed.
The next patches get better at reconecting, so if we use dev-allow-localhost
nodes can often find each other and reconnect before shutting down; only
use that option where we actually need it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell fedcfd661f pytest: hand 'True' to decoderawtransaction so it doesn't get confused.
This explains the very-very occasional issue we had parsing (hence the
random-looking fixes!).  The decoderawtransaction heuristic sometimes
thinks it's a zero-input tx, not a segwit tx.  Setting 'iswitness' to
true makes it reliable.

Here's the example I finally caught:

```
rusty$ bitcoin-cli decoderawtransaction  0200000000010180f80017ceb208d84cd5be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d00000000
{
  "txid": "3b8b78c18d30036f93b10a67eb8731325927fb046be969d24075e5b2e1e66e07",
  "hash": "3b8b78c18d30036f93b10a67eb8731325927fb046be969d24075e5b2e1e66e07",
  "version": 2,
  "size": 235,
  "vsize": 235,
  "locktime": 0,
  "vin": [
  ],
  "vout": [
    {
      "value": 6267898963.53775617,
      "n": 0,
      "scriptPubKey": {
        "asm": "be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb0 OP_TUCK",
        "hex": "4cd5be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d",
        "type": "nonstandard"
      }
    }
  ]
}
rusty$ bitcoin-cli decoderawtransaction  0200000000010180f80017ceb208d84cd5be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d00000000 true
{
  "txid": "d1f0e478ada951d4ee2d952a526a90cda181da6226980d69b345f644ed57a05d",
  "hash": "3b8b78c18d30036f93b10a67eb8731325927fb046be969d24075e5b2e1e66e07",
  "version": 2,
  "size": 235,
  "vsize": 153,
  "locktime": 0,
  "vin": [
    {
      "txid": "4e3b4539273a63f74165da5ac59817b9acc1214e0dbed54cd808b2ce1700f880",
      "vout": 1,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "3045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e01",
        "02225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d"
      ],
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00995433,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 774b1c651a1b409213057783547e2bd37a717312",
        "hex": "0014774b1c651a1b409213057783547e2bd37a717312",
        "reqSigs": 1,
        "type": "witness_v0_keyhash",
        "addresses": [
          "bc1qwa93ceg6rdqfyyc9w7p4gl3t6da8zucjnugke0"
        ]
      }
    },
    {
      "value": 0.01000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 5f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e",
        "hex": "00205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e",
        "reqSigs": 1,
        "type": "witness_v0_scripthash",
        "addresses": [
          "bc1qta6rzgletp98vpvt4sg59mptcmrqkje278f3ghn5gxx5rtj3qz0qgkydgs"
        ]
      }
    }
  ]
}

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 13:10:39 +02:00
Christian Decker 90f74907f9 pytest: Use the file object and don't use print without line-endings 2018-08-07 00:54:19 +00:00
Christian Decker d3731b08b1 pytest: Migrate all remaining tests into test_misc.py 2018-08-07 00:54:19 +00:00
Christian Decker 58709cf190 pytest: Migrate connection tests to new fixture model 2018-08-07 00:54:19 +00:00
Christian Decker 19092a8f1b pytest: Migrate close and onchain tests to test_closing.py 2018-08-07 00:54:19 +00:00
Christian Decker 605bf8c89d pytest: Migrate the gossip related tests to the new fixture model 2018-08-07 00:54:19 +00:00
Christian Decker 2e26d4fc42 pytest: Migrate payment related tests to use modern fixtures 2018-08-07 00:54:19 +00:00
Christian Decker 2c77fc5bf2 pytest: Add a helper to determine a node's peer channel state 2018-08-07 00:54:19 +00:00
Christian Decker 88217369c2 pytest: Move NodeFactory to utils.py
We are starting to move things out of test_lightningd.py so this is a logical
first step.
2018-08-07 00:54:19 +00:00
Rusty Russell 6e10237d10 pytest: allow db_query to manipulate db.
Needed for the next test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell edf043c89b pytest: make utils.py read config.vars.
I could not figure out why test_announce_address suddenly stopped working:
I had previously been using DEVELOPER=1 on the cmdline for historical
reasons when testing locally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-16 23:16:34 +02:00