Commit Graph

8298 Commits

Author SHA1 Message Date
Rusty Russell 754765c139 utxo: keep flag to recognize to-remote option_anchor_outputs closes.
We need to remember this in the db (it's a P2WSH for option_anchor_outputs),
and we need to set nSequence to 1 to spend it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell c753561d24 onchaind: recognize the to-remote output if option_anchor_outputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell a730cbdc96 onchaind: recognize (and ignore) anchor outputs.
Without this, onchaind fails to identify them.  So onchaind now needs
to know the funding pubkeys.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell a5d0c14d4d option_anchor_outputs: wire into all the subdaemons.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 8cf2fff91b pytest: generalize tests.
Anchor outputs break many assumptions in our tests:

1. Remove some hardcoded numbers in favor of a fee calc, so we only have to
   change in one place.
   FIXME: This should also be done for elements!
2. Do binary search to get feerate for a given closing fee.
3. Don't assume output #0: anchor outputs perturb them.
4. Don't assume we can make 1ksat channels (anchors cost 660 sats!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell f0afd060db initial_commit_tx.h: extract base-weight calculation.
This avoids reproducing it inside full_channel.c.

Not sure it does elements correctly though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 3ff8311b40 channeld: change to_remote for option_anchor_outputs.
It's now a P2WSH to incorporate a CSV 1 delay.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell e7423888ba initial_commit_tx, commit_tx: add anchor outputs if needed.
This also means we subtract 660 satoshis more everywhere we subtract
the base fee (except for mutual close, where the base fee is still
used).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 222d018b1a commit_tx: take raw funding keys instead of wscript.
It now needs the keys for anchor outputs, so have it calculate
the 2 of 2 wscript internally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell d8d1014ca4 channeld: implement htlc sig checking changes for option_anchor_outputs.
This is best done by passing `struct bitcoin_signature` around instead
of raw signatures.  We still save raw sigs to the db, and of course the
wire protocol uses them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 85e3b43176 channeld, openingd: take into account option_anchor_outputs for fees.
HTLC fees increase (larger weight), and the fee paid by the opener
has to include the anchor outputs (i.e. 660 sats).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 7a2f964d43 hsmd: take option_anchor_outputs flag for HTLC signatures.
They are to be signed with SIGHASH_SINGLE|SIGHASH_ANYONECANPAY.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 05c3a9bf12 script: change htlc output scripts if option_anchor_outputs.
For the moment, everyone passes "false".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell edf60b4f9e openingd: remember if we negotiated option_anchor_outputs, to put in the db.
And hand it through to channeld just like option_static_remotekey.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell b093856d0c db: add option_anchor_outputs entry to channel table.
This is the same way we handle option_static_remotekey, which
is also sticky (if negotiated at opening time, it always applies).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell ddb4ee296b features: define option_anchor_outputs.
As specified by https://github.com/lightningnetwork/lightning-rfc/pull/688.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell 168009c105 features: require dependent features at init handshake.
This simplifies our test matrix, as we never have to handle talking
to peers that specify one but not the other.

This is particularly important for option_anchor_outputs which
assumes option_static_remotekey.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-14 11:51:14 +09:30
Rusty Russell bcd632f2ae listfunds: add "redeemscript" field.
You'd need this if you ever wanted to make your own PSBT.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `listfunds` now has a `redeemscript` field for p2sh-wrapped outputs.
2020-08-13 12:37:02 -05:00
Rusty Russell cf25ca11b8 sendpsbt: annotate transactions in wallet.
This is what txsend does, only we have a psbt so we have
to change the db interface to take a wally_tx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 12:37:02 -05:00
Rusty Russell 6b39731c6a fundchannel: provide outnum
It's currently always 0, but it won't be once we replace txprepare.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel` has new `outnum` field indicating which output of the transaction funds the channel.
2020-08-13 12:37:02 -05:00
Dave Scotese d2bd1f0fa9 Update TOR.md
My testing shows that I received errors which turned out to be fatal ("/usr/libexec/c-lightning/lightning_connectd: libbacktrace: no debug info in ELF executable") when I had two `bind-addr=...` settings in my config file, which the instructions insinuated since c-lightning would already have been installed and runnable (therefore, already having a bind-addr).
2020-08-13 12:55:33 +02:00
ZmnSCPxj jxPCSnmZ f81611e551 plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress.
Blockheight disagreement is signalled with a permanent failure at the
end node, but is actually a transient failure.
2020-08-13 12:50:16 +02:00
Christian Decker 8769f9ed93 pay: Fix final TLV payload if not going through MPP modifiers
Reported-by: ZmnSCPxj
Signed-off-by: Christian Decker <@cdecker>

Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
2020-08-13 12:50:16 +02:00
ZmnSCPxj jxPCSnmZ 0279be1d13 plugins/libplugin-pay.c: Be less aggressive with advancing through routehints.
Only advance through routehints if no route was found at all, or if the
estimated capacity at the routehint is lower than the amount that we
have to send through the routehint.

Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-13 12:50:16 +02:00
ZmnSCPxj jxPCSnmZ 29df517966 tests/test_pay.py: Provide test showing that blockheight disagreement causes us to advance routehints too aggressively.
The worst effect is that unpublished nodes are harder to pay, but
even published ones make us do unnecessary work, since we are
losing routehints from the published ones that could help us
actually route better to them.
2020-08-13 12:50:16 +02:00
Rusty Russell 2f4c16c388 pytest: fix race in test_bcli.
On my test machine, we queried bitcoind before the close tx was sent:

```
        # When output is spent, it should give us null !
        txo = l1.rpc.call("getutxout", {"txid": txid, "vout": 0})
>       assert txo["amount"] is txo["script"] is None
E       AssertionError: assert '20000000msat' is '00205b8cd3b914cf67cdd8fa6273c930353dd36476734fbd962102c2df53b90880cd'

tests/test_plugin.py:1221: AssertionError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 12:49:49 +02:00
Rusty Russell 185b377551 pytest: don't wait for channel active message in test_closing.py::test_penalty_htlc_tx_timeout
It might have already happened, and anyway, we wait for the actual state
below.

```
        # make database snapshot of l2
        l2.stop()
        l2_db_path = os.path.join(l2.daemon.lightning_dir, chainparams['name'], 'lightningd.sqlite3')
        l2_db_path_bak = os.path.join(l2.daemon.lightning_dir, chainparams['name'], 'lightningd.sqlite3.bak')
        copyfile(l2_db_path, l2_db_path_bak)
        l2.start()
        sync_blockheight(bitcoind, [l2])

        # push some money from l3->l2, so that the commit counter advances
        l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
>       l2.daemon.wait_for_log('now ACTIVE')

tests/test_closing.py:908:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:288: in wait_for_log
    return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyln.testing.utils.LightningD object at 0x7f0c145a32d0>
regexs = ['now ACTIVE'], timeout = 60

    def wait_for_logs(self, regexs, timeout=TIMEOUT):
        """Look for `regexs` in the logs.

        We tail the stdout of the process and look for each regex in `regexs`,
        starting from last of the previous waited-for log entries (if any).  We
        fail if the timeout is exceeded or if the underlying process
        exits before all the `regexs` were found.

        If timeout is None, no time-out is applied.
        """
        logging.debug("Waiting for {} in the logs".format(regexs))
        exs = [re.compile(r) for r in regexs]
        start_time = time.time()
        pos = self.logsearch_start
        while True:
            if timeout is not None and time.time() > start_time + timeout:
                print("Time-out: can't find {} in logs".format(exs))
                for r in exs:
                    if self.is_in_log(r):
                        print("({} was previously in logs!)".format(r))
>               raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E               TimeoutError: Unable to find "[re.compile('now ACTIVE')]" in logs.

contrib/pyln-testing/pyln/testing/utils.py:264: TimeoutError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 10:39:56 +09:30
Rusty Russell fc2fa56b55 pytest: fix temporary memleak found by test_misc.py::test_htlc_out_timeout
```
E           Global errors:
E            - Node /tmp/ltests-o5mr9txw/test_htlc_out_timeout_1/lightning-1/ has memory leaks: [
E               {
E                   "backtrace": [
E                       "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E                       "ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)",
E                       "common/json_helpers.c:182 (json_add_address)",
E                       "common/json_helpers.c:242 (json_add_address_internal)",
E                       "lightningd/peer_control.c:1659 (json_getinfo)",
E                       "lightningd/jsonrpc.c:598 (command_exec)",
E                       "lightningd/jsonrpc.c:708 (rpc_command_hook_callback)",
E                       "lightningd/plugin_hook.c:278 (plugin_hook_call_)",
E                       "lightningd/jsonrpc.c:785 (plugin_hook_call_rpc_command)",
E                       "lightningd/jsonrpc.c:864 (parse_request)",
E                       "lightningd/jsonrpc.c:954 (read_json)",
E                       "ccan/ccan/io/io.c:59 (next_plan)",
E                       "ccan/ccan/io/io.c:435 (io_do_always)",
E                       "ccan/ccan/io/poll.c:300 (handle_always)",
E                       "ccan/ccan/io/poll.c:377 (io_loop)",
E                       "lightningd/io_loop_with_timers.c:24 (io_loop_with_timers)",
E                       "lightningd/lightningd.c:1013 (main)"
E                   ],
E                   "label": "common/json_helpers.c:182:char[]",
E                   "parents": [
E                       "common/json_stream.c:29:struct json_stream",
E                       "ccan/ccan/io/io.c:91:struct io_conn",
E                       "lightningd/lightningd.c:116:struct lightningd"
E                   ],
E                   "value": "0x555e17b303e8"
E               }
E           ]
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 10:39:56 +09:30
Rusty Russell 0fa5cde782 pytest: fix occasional error in test_sign_and_send_psbt:
The modded_psbt is not necessarily invalid:

       # Try a modified (invalid) PSBT string
        modded_psbt = psbt[:-3] + 'A' + psbt[-3:]
        with pytest.raises(RpcError, match=r"should be a PSBT, not"):
>           l1.rpc.signpsbt(modded_psbt)
E           AssertionError: Pattern 'should be a PSBT, not' not found in 'RPC call failed: method: signpsbt, payload: {'psbt': 'cHNidP8BAM0CAAAABJ9446mTRp/ml8OxSLC1hEvrcxG1L02AG7YZ4syHon2sAQAAAAD9////JFJH/NjKwjwrP9myuU68G7t8Q4VIChH0KUkZ5hSAyqcAAAAAAP3///8Uhrj0XDRhGRno8V7qEe4hHvZcmEjt3LQSIXWc+QU2tAEAAAAA/f///wstLikuBrgZJI83VPaY8aM7aPe5U6TMb06+jvGYzQLEAQAAAAD9////AcDGLQAAAAAAFgAUyQltQ/QI6lJgICYsza18hRa5KoEAAAAAAAEBH0BCDwAAAAAAFgAUqc1Qh7Q5kY1noDksmj7cJmHaIbQAAQEfQEIPAAAAAAAWABS3bdYeQbXvBSryHNoyYIiMBwu5rwABASBAQg8AAAAAABepFD1r0NuqAA+R7zDiXrlP7J+/PcNZhwEEFgAUKvGgVL/ThjWE/P1oORVXh/ObucYAAQEgQEIPAAAAAAAXqRRsrE5ugA1VJnAith5+msRMUTwl8ocBBBYAFMrfGCiLi0ZnOCY83ERKJ1sLYMY8AAAA='}, error: {'code': -1, 'message': 'No wallet inputs to sign'}'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-13 10:39:56 +09:30
Christian Decker 3d902501e5 pay: Group payments by their payment_hash, not by bolt11 string
This is quicker and guaranteed to work even for payments that were not
initiated via an invoice.
2020-08-12 19:10:48 +02:00
Christian Decker 894c886bdd pay: Inherit payment label to all children 2020-08-12 19:10:48 +02:00
Christian Decker 0dcd974d97 pytest: Reproduce #3915 2020-08-12 19:10:48 +02:00
Christian Decker 639eaaf2b4 json-rpc: Do not return a null amount_msat to listpays
Suggested-by: Rusty Russell <@rustyrussell>
Changelog-Deprecated: JSON-RPC: `listsendpays` will no longer add `null` if we don't know the `amount_msat` for a payment.
2020-08-11 08:43:18 +09:30
Rusty Russell 624df64330 pyln-client: allow commands and options to mark themselves deprecated.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln-client: commands and options can now mark themselves deprecated.
2020-08-11 08:43:18 +09:30
Rusty Russell fc097b8b48 libplugin: allow commands and options to mark themselves deprecated.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:43:18 +09:30
Rusty Russell 7a7a849fc2 lightningd: allow plugin commands and options to mark themselves deprecated.
This lets us handle it the same way we handle builtin commands, and also
lets us warn if they use deprecated apis and allow-deprecated-apis=false.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: can now mark their options and commands deprecated.
2020-08-11 08:43:18 +09:30
Rusty Russell 4ce8f56327 pyln.client.plugin: set deprcated_apis based on getmanifest.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:43:18 +09:30
Rusty Russell c85b14f06d libplugin: initialize deprecated in getmanifest.
We currently do it by calling listconfigs, but that may be too late.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:43:18 +09:30
Rusty Russell 151bc47583 JSON-RPC: getmanifest passes allow-deprecated-apis flag.
This allows plugins to choose how to present things in getmanifest.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: plugins: `getmanifest` may now include "allow-deprecated-apis" boolean flag.
Changelog-Deprecated: plugins: `getmanifest` without any parameters; plugins should accept any parameters for future use.
2020-08-11 08:43:18 +09:30
Rusty Russell 930e294801 help: better handing of deprecated commands
If allow-deprecated-apis=false, don't mention them at all (we already
disallow calling them in that case).  Otherwise, note that they're
deprecated in the help msg.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:43:18 +09:30
Rusty Russell 0cbc6447b7 doc: refer to correct option name in STYLE.md.
It's --allow-deprecated-apis=false.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:43:18 +09:30
Rusty Russell 601fc805d7 fundpsbt: simplify the logic a little.
It keeps multiple different variables around the loop, but a simple
"are we done yet?" helper makes this clearer and reduces special
cases or all-vs-target.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell a6a8a40594 psbt: implement psbt_txid.
You can't just take the txid of the global tx, if some are p2sh-wrapped.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell 09f065fc45 script: expose script_push_bytes().
We'll need it for creating redeemscripts for PSBTs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell 5e0b03fba9 common: hoist param_bitcoin_address where plugins can use it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell 6966cf99e1 bitcoin: add wally_tx_output helper to create standalone output.
In preparation for when we don't have a tx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell 600d0a4a1d psbt: make psbt_from_b64 more conventional.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell c1df8d586d utxo: remove unused scriptSig field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Christian Decker 03b4662c24 travis: Use a stripped bitcoind bundle
I went and stripped all the parts from the bitcoind tarball and uploaded it to
a GCS bucket. Let's see how much we save.
2020-08-11 08:38:49 +09:30
Christian Decker 33fae1f383 travis: Reorganize tests a bit to move non-integration tests out
We were running the integration tests and the install tests with every
test-group, which doesn't increase test coverage at all. So now we bundle them
with the source check which are short and remove them from the overloaded
configurations.
2020-08-11 08:38:49 +09:30