Commit Graph

4296 Commits

Author SHA1 Message Date
Rusty Russell 5624afc340 connectd: clean up unused structure fields.
They can be local variables.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 02966a4857 connectd: remove unused handback APIs and code.
We now simply maintain a pubkey set for connected peers (we only care
if there's a reconnect), not the entire peer structure.

lightningd no longer queries us for getpeers: it knows more than we do
already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell e59cbb3e2c pytest: make sure receiving peer's openingd is ready.
There's now a potential race: the source peer connect returns, but in
destination peer the master hasn't read the connect message from
connectd, so the peer isn't in listpeers yet.

(Previously the connection stayed in connectd, so there was no such
window).

This is an occasional issue in a few places.

Note that we take the opportunity to speed up test_disconnectpeer too
while we're there.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 50f5eb34b4 openingd: take peer before we're opening, wait for explicit funding msg.
Prior to this, lightningd would hand uninteresting peers back to connectd,
which would then return it to lightningd if it sent a non-gossip msg,
or if lightningd asked it to release the peer.

Now connectd hands the peer to lightningd once we've done the init
handshake, which hands it off to openingd.

This is a deep structural change, so we do the minimum here and cleanup
in the following patches.

Lightningd:
1. Remove peer_nongossip handling from connect_control and peer_control.
2. Remove list of outstanding fundchannel command; it was only needed to
   find the race between us asking connectd to release the peer and it
   reconnecting.
3. We can no longer tell if the remote end has started trying to fund a
   channel (until it has succeeded): it's very transitory anyway so not
   worth fixing.
4. We now always have a struct peer, and allocate an uncommitted_channel
   for it, though it may never be used if neither end funds a channel.
5. We start funding on messages for openingd: we can get a funder_reply
   or a fundee, or an error in response to our request to fund a channel.
   so we handle all of them.
6. A new peer_start_openingd() is called after connectd hands us a peer.
7. json_fund_channel just looks through local peers; there are none
   hidden in connectd any more.
8. We sometimes start a new openingd just to send an error message.

Openingd:
1. We always have information we need to accept them funding a channel (in
   the init message).
2. We have to listen for three fds: peer, gossip and master, so we opencode
   the poll.
3. We have an explicit message to start trying to fund a channel.
4. We can be told to send a message in our init message.

Testing:
1. We don't handle some things gracefully yet, so two tests are disabled.
2. 'hand_back_peer .*: now local again' from connectd is no longer a message,
   openingd says 'Handed peer, entering loop' once its managing it.
3. peer['state'] used to be set to 'GOSSIPING' (otherwise this field doesn't
   exist; 'state' is now per-channel.  It doesn't exist at all now.
4. Some tests now need to turn on IO logging in openingd, not connectd.
5. There's a gap between connecting on one node and having connectd on
   the peer hand over the connection to openingd.  Our tests sometimes
   checked getpeers() on the peer, and didn't see anything, so line_graph
   needed updating.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 5cd72c9620 connectd: explicitly log whether connection is IN or OUT.
Useful for debugging: it wasn't immediately obvious from the logs
which side was spuriously reconnecting.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 5135d3ad7d pytest: make sure we truncate disconnect file for blackhole processes.
In particular, I found lightning_openingd processes after running
tests.  When we use the dev_disconnect blackhole '0' option, they
stick around until the dev_disconnect file is truncated (there is only
so much you can do with only a file descriptor), so let's do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 8668b0028e pytest: make sure both sides of channel are ready before returning from fund_channel
The following changes revealed this race, where expecting listchannels()
to contain two channels immediately after fund_channel() was racy.

We also derive the short_channel_id first, so we can search logs for the
exact messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Rusty Russell 329270525c pytest: only use dev-allow-localhost when needed.
The next patches get better at reconecting, so if we use dev-allow-localhost
nodes can often find each other and reconnect before shutting down; only
use that option where we actually need it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 19:44:27 +02:00
Ronald Mannak e47ac33a37 Fix readme docker-compose port setting 2018-08-09 13:13:38 +02:00
Rusty Russell fedcfd661f pytest: hand 'True' to decoderawtransaction so it doesn't get confused.
This explains the very-very occasional issue we had parsing (hence the
random-looking fixes!).  The decoderawtransaction heuristic sometimes
thinks it's a zero-input tx, not a segwit tx.  Setting 'iswitness' to
true makes it reliable.

Here's the example I finally caught:

```
rusty$ bitcoin-cli decoderawtransaction  0200000000010180f80017ceb208d84cd5be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d00000000
{
  "txid": "3b8b78c18d30036f93b10a67eb8731325927fb046be969d24075e5b2e1e66e07",
  "hash": "3b8b78c18d30036f93b10a67eb8731325927fb046be969d24075e5b2e1e66e07",
  "version": 2,
  "size": 235,
  "vsize": 235,
  "locktime": 0,
  "vin": [
  ],
  "vout": [
    {
      "value": 6267898963.53775617,
      "n": 0,
      "scriptPubKey": {
        "asm": "be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb0 OP_TUCK",
        "hex": "4cd5be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d",
        "type": "nonstandard"
      }
    }
  ]
}
rusty$ bitcoin-cli decoderawtransaction  0200000000010180f80017ceb208d84cd5be0d4e21c1acb91798c55ada6541f7633a2739453b4e0100000000ffffffff0269300f0000000000160014774b1c651a1b409213057783547e2bd37a71731240420f00000000002200205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e02483045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e012102225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d00000000 true
{
  "txid": "d1f0e478ada951d4ee2d952a526a90cda181da6226980d69b345f644ed57a05d",
  "hash": "3b8b78c18d30036f93b10a67eb8731325927fb046be969d24075e5b2e1e66e07",
  "version": 2,
  "size": 235,
  "vsize": 153,
  "locktime": 0,
  "vin": [
    {
      "txid": "4e3b4539273a63f74165da5ac59817b9acc1214e0dbed54cd808b2ce1700f880",
      "vout": 1,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "3045022100ba65a905cf4ebbb9728dc682fcf17cb73ade0ca224729a1878f689a8afa9a49e02206f323f224c5171d170aafb8ff57e2761411a27dea304ac8f9a3663c456d21f3e01",
        "02225ce166e84b3833d9f620863b4e713099de616f559e8768f44ff674054bb07d"
      ],
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00995433,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 774b1c651a1b409213057783547e2bd37a717312",
        "hex": "0014774b1c651a1b409213057783547e2bd37a717312",
        "reqSigs": 1,
        "type": "witness_v0_keyhash",
        "addresses": [
          "bc1qwa93ceg6rdqfyyc9w7p4gl3t6da8zucjnugke0"
        ]
      }
    },
    {
      "value": 0.01000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 5f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e",
        "hex": "00205f743123f9584a76058bac1142ec2bc6c60b4b2af1d3145e74418d41ae51009e",
        "reqSigs": 1,
        "type": "witness_v0_scripthash",
        "addresses": [
          "bc1qta6rzgletp98vpvt4sg59mptcmrqkje278f3ghn5gxx5rtj3qz0qgkydgs"
        ]
      }
    }
  ]
}

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 13:10:39 +02:00
Rusty Russell 58d090c3c2 pytest: fix flaky test.
Saw this in Travis: technically we return from the dev_set_max_scids...
cmd after sending it to gossipd, but we should wait for it to log.
Adding an internal reply message for a dev command seems overkill.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-09 12:39:59 +02:00
Rusty Russell 60b95d4764 pylightning: add method and payload as proper fields.
They're much more useful being programatically-accessible, AFAICT.

The string stays the same so they're backwards compatible.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-08 16:05:58 +02:00
Rusty Russell d8a6028214 connectd: fix binding to same port on IPv4 and IPv6.
1. If the IPv6 address was public, that changed the wireaddr and thus the ipv4 bind
   would not be to a wildcard and would fail.
2. Binding two fds to the same port on both wildcard IPv4 and IPv6 succeeds; we only
   fail when we try to listen, so allow error at this point.

For some reason this triggered on my digital ocean machine.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-08 15:04:06 +02:00
Rusty Russell 17f7f50814 wireaddr: correctly parse ':portnum' (meaning IPv4 and IPv6)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-08 15:04:06 +02:00
Corné Plooy f1df9dfa4d lightning-cli: consider empty inputs to be strings 2018-08-07 13:47:07 +02:00
Christian Decker 2a14a98ead pytest: Migrate test_withdraw to use LightningNode.db_query
This was causing lock issues in some cases.
2018-08-07 00:54:19 +00:00
Christian Decker 90f74907f9 pytest: Use the file object and don't use print without line-endings 2018-08-07 00:54:19 +00:00
Christian Decker ae99e493b8 pytest: Remove test_lightningd and all the legacy testing framework 2018-08-07 00:54:19 +00:00
Christian Decker d3731b08b1 pytest: Migrate all remaining tests into test_misc.py 2018-08-07 00:54:19 +00:00
Christian Decker 13d9f30a6a pytest: Move forwarded payments to test_pay.py 2018-08-07 00:54:19 +00:00
Christian Decker 58709cf190 pytest: Migrate connection tests to new fixture model 2018-08-07 00:54:19 +00:00
Christian Decker 19092a8f1b pytest: Migrate close and onchain tests to test_closing.py 2018-08-07 00:54:19 +00:00
Christian Decker 605bf8c89d pytest: Migrate the gossip related tests to the new fixture model 2018-08-07 00:54:19 +00:00
Christian Decker 2e26d4fc42 pytest: Migrate payment related tests to use modern fixtures 2018-08-07 00:54:19 +00:00
Christian Decker de99ccca81 pytest: Move invoice-related tests into their own file
Mostly a move-only but simplified with the new fixtures as well.
2018-08-07 00:54:19 +00:00
Christian Decker 2c77fc5bf2 pytest: Add a helper to determine a node's peer channel state 2018-08-07 00:54:19 +00:00
Christian Decker 88217369c2 pytest: Move NodeFactory to utils.py
We are starting to move things out of test_lightningd.py so this is a logical
first step.
2018-08-07 00:54:19 +00:00
Christian Decker 6dae525c07 pytest: Check description of newly created invoices 2018-08-07 00:54:19 +00:00
Christian Decker b55d03cb30 pytest: Pass result to fixtures and keep directories of failed tests
@Reported-by: Rusty Russell <@rustyrussell>
@Signed-off-by: Christian Decker <@cdecker>
2018-08-07 00:54:19 +00:00
Christian Decker 6627da5eb5 routing: Do not consider risk when capping transfers
Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2018-08-06 22:46:02 +02:00
Christian Decker 84905eac2b routing: Make the capacity a parameter to new_chan
As pointed out by @rustyrussell the capacity is now always defined, so we can
fold that into the construction of the channel itself.

Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2018-08-06 22:46:02 +02:00
Christian Decker 8201764117 routing: Skip channels that require larger HTLCs than we are routing
The `htlc_minimum_msat` parameter was ignored so far, and we'd be attempting to
pay and hitting a brick wall by doing so. This patch just skips channels that
are not eligible anyway.
2018-08-06 22:46:02 +02:00
Christian Decker 14000a22bc routing: Skip channels that don't have sufficient capacity
We know the total channel capacity after checking for its existence on-chain, so
we can actually make use of that information to discard channels that don't have
a sufficient capacity anyway, reducing the number of failed attempts.
2018-08-06 22:46:02 +02:00
Christian Decker 8a34933c1a gossip: Annotate locally added channels with their capacity
We were adding channels without their capacity, and eventually annotated them
when we exchanged `channel_update`s. This worked as long as we weren't
considering the channel capacity, but would result in local-only channels to be
unusable once we start checking.
2018-08-06 22:46:02 +02:00
Rusty Russell 8f38a46584 lightningd: correctly store our own channel_reserve_satoshis
openingd calculates our reserve based on the channel amount (even if
we're funding, to keep the calculation in one place), but it wasn't
reporting it back to the master daemon.  We initialized it to 0 so that
valgrind wouldn't get upset, as it's part of a structure we send over
the wire.

Have openingd report back, and also initialize it to an impossible value
as extra assurance.  And remove a stray (harmless but weird) semicolon.

Reported-by: Gálli Zoltán
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell 6d79f7679c pytest: test reserve enforcement.
A bit tricky, since we don't normally let ourselves make bad HTLCs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell 6e10237d10 pytest: allow db_query to manipulate db.
Needed for the next test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:34:43 +02:00
Rusty Russell c5b5f74965 lightningd: display all addresses in listnodes.
We kept overwriting the first one on marshalling, so any following
addresses were junk.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:33:46 +02:00
Rusty Russell 584ee26200 gossipd: fix thinko in node_announcement address parsing which made us miss final address
'cursor < ser + max' isn't valid because we reduce 'max' as we go!  Effectively
we'll stop once we're past halfway, which can only happen with ipv6 + a torv2
address.

Ths fix is one-line, but we rename 'max' to 'len' which makes its purpose
clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-06 19:33:46 +02:00
Douglas Schilling Landgraf ea6c0d4506 contrib: Dockerfile.builder.fedora bump fedora release
Fedora 28 released
2018-08-05 13:09:58 +02:00
Rusty Russell 63a1e510d9 lightningd: count first block of channel on exit from openingd, not start.
It's only used so we can timeout being fundee after a few hundred
blocks, but when openingd is started for idle connections, the
difference can be huge.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 15542a01eb lightningd: note that we should send an error msg on channel open timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 45ad2498f3 connectd: check for supported features ourselves.
Checking in the master doesn't help anything, and it's weird.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

1diff --git a/connectd/connect.c b/connectd/connect.c
index 138b73fc..b01d1546 100644
2018-08-05 02:03:58 +00:00
Rusty Russell 136f10e4a3 common/read_peer_msg: remove.
Also means we simplify the handle_gossip_msg() since everyone wants it to
use sync_crypto_write().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 41c9431ae0 closingd: use deconstructed read_peer_msg helpers.
This is more verbose but I still think it's clearer.  Almost exactly
the same as the openingd one, but just different enough to be painful
to share.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 90b43a6f47 openingd: use deconstructed read_peer_msg helpers.
This is more verbose but I still think it's clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 6b5462b637 channeld: use deconstructed read_peer_msg helpers.
This is clearer and neater, and even slightly more efficient, since
read_peer_msg() was calling poll() again on gossipfd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 0b08601951 sync_crypto_write/sync_crypto_read: just fail, don't return NULL.
There's only one thing the caller ever does, just do that internally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 09cce4a9c7 common/read_peer_msg: deconstruct into individual helper routines.
The One Big API is confusing, and has enough corner cases that we should
ditch it rather than add more.

See: https://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction

In particular, when openingd is changed to chat to peers even when
it's not actively opening a channel, it wants to handle (most) errors
by continuing, not calling peer_failed().

This exposes the constituent parts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00
Rusty Russell 74d428109a channeld: it's OK to block on writing to peer.
In fact, it's good.  We don't want to queue up infinite gossip.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-05 02:03:58 +00:00