Commit Graph

7153 Commits

Author SHA1 Message Date
Christian Decker 714b57241b sphinx: Promote TLV payloads to be non-experimental
This is what provides us with the ability to add custom fields in the payload
when using `createonion` so make sure we actually have access to it.

Changelog-Changed: The TLV payloads for the onion packets are no longer considered an experimental feature and generally available.
Changelog-Added: Plugins may now handle modern TLV-style payloads via the `htlc_accepted` hook

Signed-off-by: Christian Decker <@cdecker>
2019-12-01 15:40:47 +01:00
Christian Decker de6bf3e421 json: Add two param parsers for secrets and hex-encoded binary data
These are useful for the `createonion` JSON-RPC we're going to build next. The
secret is used for the optional `session_key` while the hex-encoded binary is
used for the `assocdata` field to which the onion commits. The latter does not
have a constant size, hence the raw binary conversion.
2019-12-01 15:40:47 +01:00
Jarret Dyrbye d712f732d8 pylightning - allow unicode symbols in the description to be passed as-is to the daemon
addresses issue #2753.

Formatting the JSON with the default parameters will escape the unicode
symbols in a way that c-lightning won't allow, leading to an exception.

Changelog-Fixed: `pylightning` now handles unicode characters in JSON-RPC requests and responses correctly.
2019-11-29 21:38:17 +01:00
Christian Decker dc1f27c0b3 pytest: Add failing test for unicode in JSON-RPC through pylightning 2019-11-29 21:38:17 +01:00
Michael Schmoock 895e06122c doc: adds pyln deps to command for running blackbox tests
This one got missing when introducting `pyln` packages as test dependencies.
2019-11-29 21:23:23 +01:00
darosior 36e8d83a62 tools/hsmtool: correct bad access to argv[] 2019-11-29 21:19:05 +01:00
darosior e5e4958909 tool/hsmtool: add a 'guesstoremote' command
This, in the case of data loss on a channel with `option_static_remotekey`
negotiated, allows to likely (if the dbid is not unreasonable) recover
the funds from a remote unilateral close just with the hsm_secret.

Changelog-added: A new command, 'guesstoremote', is added to the hsmtool. It is meant to be used to recover funds after an unilateral close of a channel with `option_static_remotekey` enabled.
2019-11-29 21:19:05 +01:00
darosior cd11c2050c devtools/gossipwith: add a "network" option
If specified, this will add the corresponding chain_hash to the init message.
2019-11-29 21:17:08 +01:00
darosior 3322048774 connectd: add network to init message
Changelog-Added: protocol: We now signal the network we are running on at init.
2019-11-29 21:17:08 +01:00
darosior 4b13b88f6c bitcoin/chainparams: add an utility to retrieve chainparams for all networks 2019-11-29 21:17:08 +01:00
darosior e6b8a02446 connectd: setup chainparams
We are going to signal the genesis block hash in the init message.
2019-11-29 21:17:08 +01:00
Christian Decker c84473f82c hsm: Stabilize the hsm encryption and decryption tests
We were using sleeps to hope we catch the password prompt. This makes the test
flaky. So I added a help text followed by a `fflush` to make sure we catcht he
right moment, instead of guessing. The `fflush` is also useful for debugging
if a user ever pipes the output to a file it'd get buffered and the user would
wait forever. The same applies for automated systems such as `expect` or
`pexpect` based scripts that enter the password on prompt.
2019-11-29 15:06:39 +01:00
Christian Decker 9e59740268 pyln-testing: Do not require a conftest.py to annotate tests
In the c-lightning tests we have `tests/conftest.py` which annotates test
function with the outcome. If we use pyln-testing outside of the c-lightning
tree we cannot rely on that annotation being there, so we assume it passed.
2019-11-29 15:06:39 +01:00
Christian Decker 3c5ed15b2c pyln: Add psycopg2-binary as a dependency
Using the psycopg2-binary package means that the apropriate compiled binary
for the user platform will be shipped alongside the python binaries. Otherwise
the python bindings and the C shims would be shipped which would then require
the postgres development packages as well.

This just makes things easier, since we don't require the build dependencies.
2019-11-29 15:06:39 +01:00
Rusty Russell b9a03a08ef wallet/Makefile: fix dependency line.
The .o files need regen, not the .c files!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-28 11:50:13 +01:00
Rusty Russell 493c2ab1d7 openingd: clean up and fix minor leak.
test_openchannel_hook_1:
MEMLEAK: 0x557593c164e8'
  label=wire/fromwire.c:320:char[]'
   backtrace:'
     ccan/ccan/tal/tal.c:437 (tal_alloc_)'
     ccan/ccan/tal/tal.c:466 (tal_alloc_arr_)'
     wire/fromwire.c:320 (fromwire_wirestring)'
     openingd/gen_opening_wire.c:205 (fromwire_opening_got_offer_reply)'
     openingd/openingd.c:1067 (fundee_channel)'
     openingd/openingd.c:1279 (handle_peer_in)'
     openingd/openingd.c:1535 (main)'
   parents:

fromwire_opening_got_offer_reply() allocates two fields off NULL:
err_reason and our_upfront_shutdown_script.  err_reason is used
immediately afterwards (and was the leak detected here), so fixing
that is easy.

To fix the leak of our_upfront_shutdown_script, it makes sense to simply
make it a member of 'state'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-28 11:50:13 +01:00
darosior 670f92002d tools/gen/impl_template: correct one-line for-loops indentation 2019-11-26 21:30:25 +01:00
darosior 5fb8e0aade tools/gen/impl_template: correct tlvs fromwire's for-loop 2019-11-26 21:30:25 +01:00
gorazdko a3961c4e06 unit test: run sphinx unit test by default 2019-11-26 21:25:25 +01:00
gorazdko d5a6072642 unit test: fix test vectors in run-sphinx 2019-11-26 21:25:25 +01:00
gorazdko e8ce9d0327 unit test: fix run-sphinx dependencies and update mocks 2019-11-26 21:25:25 +01:00
fiatjaf 17bb862cfe don't enforce description_hash checking 2019-11-25 18:15:35 +01:00
Michael Schmoock 6ed320189e feat: restructure plugin and options in listconfigs
This will change the command `listconfigs` output in several ways:

 - Deprecated the duplicated "plugin" JSON output by replacing it with
 - a "plugins" array with substructures for each plugin with:
 - path, name and their options

Changelog-Changed: JSON-RPC: `listconfigs` now structures plugins and include their options
Changelog-Deprecated: JSON-RPC: `listconfigs` duplicated "plugin" paths
2019-11-25 18:15:02 +01:00
Rusty Russell 8393d21503 common/features: add `payment_secret` feature if EXPERIMENTAL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 3b37c9d426 common/features: clean up feature handling for different cases.
The spec is (RSN!) going to explicitly denote where each feature should
be presented, so create that infrastructure.

Incorporate the new proposed bolt11 features, which need this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 9765642bf8 lightningd: use final_tlv if the payment_secret is supplied.
This implies the final node understands TLV onion.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell e5247a68b6 lightningd: check payment secret on htlc receipt.
We don't set the secret to compulsory (yet!) but put code in for the
future.  Meanwhile, if there is a secret, check it is correct.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell c2e8531e8e lightningd: add secrets (and associated feature) to invoices.
We don't need a new db column, since we can just derive the secret from
the preimage as required.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 18e9144675 plugins/pay: hand payment_secret from bolt11 through to sendpay.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell dfb1f6a096 JSON: add `payment_secret` parameter to sendpay.
This is not documented yet, since it's ignored unless EXPERIMENTAL_FEATURES
is set.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 854c64ffee common/bolt11: add secret support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 81c89aaef8 wallet: add invoice features into db.
In a future version, we will use features to insist that payers
provide the secret.  In transition, we may have old invoices which
didn't insist on that, so we need to know this on a per-invoice basis.

Not sure if I got the right syntax for adding an empty blob though!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 09cdbb70eb common: make BOLT11 use the normal feature array.
This was decided at a recent spec meeting: in particular, mpp and
var_onion_optin options will be used here.

We enhanced "features_supported" into "features_unsupported" so it
can return the first un-handlable bit number.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell ebac3d2a85 spec: update to experimental BOLTs with secret/total_amount.
Also pulls in a new onion error (mpp_timeout).  We change our
route_step_decode_end() to always return the total_msat and optional
secret.

We check total_amount (to prohibit mpp), but we do nothing with
secret for now other than hand it to the htlc_accepted hook.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 50d6941e89 lightningd: remove redundant htlc_accepted_hook_payload fields
Now we cache them in the route_step, don't need to copy them here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell fb040bc619 Spec: trivial update to latest.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-24 23:33:17 +00:00
Rusty Russell 14997f6237 lightningd: fix handling of truncated config options.
Do the same thing '--help' does with them; append `...`.

Valgrind noticed that we weren't NUL-terminarting if answer was over
78 characters.

Changelog-Fixed: JSONRPC: listconfigs appends '...' to truncated config options.
2019-11-23 22:42:34 +00:00
Rusty Russell f5831e5d10 config: limit depth on includes.
Don't spend too much effort on it, but this is better than running out
of memory and crashing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell badeaf1a67 pytest: upgrade tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell a06fa7f267 log: print UNUSUAL messages before log initialization too.
Otherwise we don't print out the upgrading messages when we move things!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell 57a32ee044 pytest: test for config parsing errors.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell 747bb99575 lightningd: keep pid files in top-level config dir.
They're already qualified with network name, and there's little point
moving them; it might even be dangerous if multiple are running.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell e3dbd78536 config: explicitly disallow nonsensical options.
1. "conf" can't be specified in a configuration file.
2. "lightning-dir" can't be specified in a configuration file unless the file
   was explicitly set with --conf=.
3. "network" options can't be set in a per-network configuration file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell 36c517bac5 common: change default network from testnet to mainnet for new installs.
Changelog-changed: Default network (new installs) is now bitcoin, not testnet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell a56f2b25b0 common: parse --allow-deprecated-apis extremely early.
We're going to want this for changing the default network.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell d512bcb85f lightningd: automatically move files for existing deployments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell aab83e729b lightningd: change config-dir from plugin / wallet / hsm POV into <network> subdir
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell dc23c308e4 config: Read both top-level and network-subdir config files.
This lets you have a default, but also a network-specific config.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: `config` and <network>/`config` read by default.
2019-11-23 22:42:34 +00:00
Rusty Russell 8b1aa3ef8b lightningd: move basic parameter parsing into common/configdir
lightning-cli is going to need to know what network we're on, so
it will need to parse the config files.  Move the code which does
the initial bootstrap parsing into common, as well as the config
file parsing core.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00
Rusty Russell 5673607ebc lightningd: don't use chainparams before param() call in json_fund_channel_start
With coming changes, this will segfault if we access it when param
code is trying to get usage from functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-23 22:42:34 +00:00