Commit Graph

8416 Commits

Author SHA1 Message Date
Sergi Delgado Segura 3edbacbb3c pyln: hexlify -> bytes.hex() in pyln.proto.wire 2020-08-26 06:09:53 +09:30
Sergi Delgado Segura 27e495efa5 pyln: Updates proto to use coincurve for Public and Private keys 2020-08-26 06:09:53 +09:30
ZmnSCPxj jxPCSnmZ c27d7a3110 plugins/libplugin-pay.c: Store the route description, and re-report on failure. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ 6468616c02 plugins/libplugin-pay.c: Propagate local_id from parent to child payment object. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ d8678467fa plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ d89c77c0ce plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ 0d2d85ab5f plugins/libplugin-pay.c: Also print events that create new sub-payments. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ 98583e84b5 plugins/libplugin-pay.c: Give cmd id and partid for each log message.
Changelog-None: internal debugging

Makes it easier to debug payments with tons of splits.
2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ e80d733399 .gitignore: Add missing gitignores. 2020-08-25 12:10:05 +02:00
Rusty Russell dc8458d1e2 tools: fix comment in template.
It's (usually) a .csv not an _csv file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 3e52d4100d common: convert to new wire generation style.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 398b4806b9 connectd: convert to new wire generation style.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell fe8564555f closingd: convert to new wire generation style.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 6d09c0eba9 openingd: convert wire to new scheme.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell dffbf8de85 gossipd: convert wire to new scheme.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 1702c7a69a hsmd: convert to new wire generation style.
Note that other directories were explicitly depending on the generated
file, instead of relying on their (already existing) dependency on 
$(LIGHTNINGD_HSM_CLIENT_OBJS), so we remove that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 8ae6740bb1 Makefile: create generic wiregen rules, use for channeld.
This means some files get renamed, and I took the opportunity to clarify
our naming (the *d* is important!)

1. channeld/channel_wire.csv -> channeld/channeld_wire.csv
2. channeld/gen_channel_wire.h -> channeld/channeld_wiregen.h
3. enum channel_wire_type -> enum channeld_wire
4. WIRE_CHANNEL_FUNDING_DEPTH -> WIRE_CHANNELD_FUNDING_DEPTH.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 7804d89ef1 Makefile: don't output on stderr if file not yet checking into git.
And if both aren't checked in, consider them changed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell b38b913694 Makefile: reduce clutter if they haven't run ./configure
It's hard to see the error:

Before:

    $ make
    File config.vars not found: you must run ./configure before running make.
    CC: cc -DBINTOPKGLIBEXECDIR="../libexec/c-lightning"    -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/x86_64-linux-gnu/libbacktrace-build -I external/libsodium/src/libsodium/include -I external/libsodium/src/libsodium/include/sodium -I external/x86_64-linux-gnu/libsodium-build/src/libsodium/include -I . -I/usr/local/include    -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS   -DBUILD_ELEMENTS=1 -c -o
    LD: cc  -fsanitize=address     -Lexternal/x86_64-linux-gnu -lwallycore -lsecp256k1 -ljsmn -lbacktrace -lsodium -L/usr/local/lib -lm -lgmp -lsqlite3 -lz  -o

After:

    $ make
    File config.vars not found: you must run ./configure before running make.
    make: *** No rule to make target 'config.vars', needed by 'show-flags'.  Stop.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell b9ac032329 doc: force refresh of all manpages.
The idea is that you regenerate the man pages in the same commit you
alter them: that's how we know whether to try regenerating them or not
(git doesn't store timestamps, so it can't really tell).

Travis will now check this, so force them all to sync to this commit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 67cde96397 Makefile: respect --quiet and V=1 in doc/Makefile.
And don't regen doc/index.rst unless there's good reason.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
niftynei 6f114e06a6 bugfix: save max_witness_len as big-endian 2020-08-25 12:34:56 +09:30
niftynei 0305c0a57e psbt: if psbt is null, return null for get_bytes 2020-08-25 12:34:56 +09:30
niftynei df7c122cb8 json: add "json-to-psbt" helper
Pull a PSBT out of a json buffer
2020-08-25 12:34:56 +09:30
niftynei 10237ccb8e psbt: handle 'NULL' or empty byte PSBTs
Don't crash if the PSBT is null / zero-byte len.
2020-08-25 12:34:56 +09:30
Rusty Russell ef3fbab551 invoice: don't allow zero-value invoices.
You can't pay them anyway, and at least one person used 0 instead of "any".

Closes: #3808
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `invoice` no longer accepts zero amounts (did you mean "any"?)
2020-08-25 12:22:45 +09:30
nicolas.dorier 2e51f23a95 Add python to runtime docker image (arm32/64) 2020-08-24 20:23:25 +02:00
Rusty Russell 57dd5be2fd gossipd: prune channels unless *both* peers have refreshed.
See https://github.com/lightningnetwork/lightning-rfc/pull/767

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: channels now pruned after two weeks unless both peers refresh it (see lightning-rfc#767)
2020-08-24 20:55:47 +09:30
Rusty Russell 7ad8fde060 bolt11: update ctlv expiry, always write it.
As per https://github.com/lightningnetwork/lightning-rfc/pull/785

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: protocol: bolt11 invoices always include CLTV fields (see lightning-rfc#785)
2020-08-24 20:55:47 +09:30
Rusty Russell b09e519669 options: be more conservative with timeouts on mainnet.
As per https://github.com/lightningnetwork/lightning-rfc/pull/785

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: config: the default CLTV expiry is now 34 blocks, and final expiry 18 blocks as per new BOLT recommendations.
2020-08-24 20:55:47 +09:30
Rusty Russell 6e4e267f2c doc: update BOLT now option_anchor_outputs is merged.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-24 20:55:47 +09:30
Rusty Russell 007daf6b9f doc: update bolt version
And sweep through and remove git qualifiers from many BOLT strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-24 20:55:47 +09:30
Rusty Russell 95294e25ea wallet: do valgrind checks when binding db statements.
Otherwise valgrind gets upset when we *run* the statements: better
to get a backtrace when we bind, so we can tell which field it is!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 11:25:51 +09:30
Rusty Russell f762f7e247 db: remove assertion on validity of node_ids read from db.
We've never hit this, we do check them on insert, and it's slowing
down some operations unnecessarily.

$ time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null

Before:
	real	0m1.781s
	user	0m0.127s
	sys	0m0.013s

After:
	real	0m1.545s
	user	0m0.124s
	sys	0m0.024s

Also, the raw listsendpays drops from 0.983s to 0.676s.

(With -O3 -flto, listsendpays is 0.416s, listpays 0.971s).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell 28884864d0 plugins/pay: don't bother parsing nodeid for summary.
We have sanity checks in there that it's a valid point.  Simply store
the JSON token like we do with others.

time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null

Before:
	real	0m2.054s
	user	0m0.114s
	sys	0m0.024s

After:
	real	0m1.781s
	user	0m0.127s
	sys	0m0.013s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell 4ba3797923 plugins/pay: iterate the htable properly, not from start each time.
time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null

Before:
	real	0m12.447s
	user	0m0.143s
	sys	0m0.008s

After:
	real	0m2.054s
	user	0m0.114s
	sys	0m0.024s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell 05636e367e libplugin: do partial parsing instead of memmem hack.
memmem is also O(n^2), though it's faster.  Now we have infrastructure,
let's do incremental parsing.

time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null

Before:
	real	0m13.674s
	user	0m0.131s
	sys	0m0.024s

After:
	real	0m12.447s
	user	0m0.143s
	sys	0m0.008s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell 628ae0a15b libplugin: do incremental parsing on lightningd commands.
This doesn't make any difference, since lightningd generally sends us
short commands (command responses are via the rpc loop, which is
already done), but it's harmless.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell c284b5bbf4 lightningd: always do incremental parsing of JSON inputs.
time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null

Before:
	real	0m42.741s
	user	0m0.149s
	sys	0m0.016s

After:
	real	0m13.674s
	user	0m0.131s
	sys	0m0.024s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: significant speedups for plugins which create large JSON replies (e.g. listpays on large nodes).
2020-08-21 09:52:33 +09:30
Rusty Russell 3ed03f9c8f common: make json_parse_input API retry friendly.
The jsmn parser is a beautiful piece of code.  In particular, you can parse
part of a string, then continue where you left off.

We don't take advantage of this, however, meaning for large JSON objects
we parse them multiple times before finally having enough to complete.

Expose the parser state and tokens through the API, so the caller can pass
them in repeatedly.  For the moment, every caller is allocates each time
(except the unit tests).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell f2d0a1d406 plugins/bcli: use simplified parser, unify bad JSON paths.
They should all show the complete JSON, so unify them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell 80b2e1e298 common: add simple json parse wrapper for the complete cases.
We're going to change the API on the more complete JSON parser, so
make and use a simple API for the easy cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-21 09:52:33 +09:30
Rusty Russell 40f17d994c libplugin: don't parse partial JSON repeatedly.
Tested on a test node which had made 50,000 payment, with no optimization.

For comparison, time for 'listsendpays' was 0.983s.

time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null

Before:
	real	0m52.415s
	user	0m0.127s
	sys	0m0.044s

After:
	real	0m42.741s
	user	0m0.149s
	sys	0m0.016s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: libplugin: significant speedups for reading large JSON replies (e.g. calling listsendpays on large nodes, or listchannels / listnodes).
2020-08-21 09:52:33 +09:30
Rusty Russell 85a90b9e32 bitcoin: bitcoin_tx_add_input use psbt_append_input.
Most of what it does was actually a function of adding the input metadata
to the PSBT, so call that and simply copy out the tx input it creates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-20 21:14:12 +09:30
Rusty Russell a0ede40743 bitcoin: make psbt_append_input more powerful.
It can now handle all the metadata as well as the base tx input.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-20 21:14:12 +09:30
Rusty Russell 172b9d7ae3 bitcoin: add nlocktime arg to create_psbt.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-20 21:14:12 +09:30
Rusty Russell 7435d50970 signpsbt: add signonly parameter to restrict/enforce what inputs to sign.
This is an extra safety check for dual funding, where we only want to sign
the inputs we provided!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `signpsbt` takes an optional `signonly` array to limit what inputs to sign.
2020-08-18 11:38:25 -05:00
Rusty Russell aab3808668 utxopsbt: let caller specify locktime, add tests and python binding.
Changelog-Added: JSON-RPC: `utxopsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-18 11:38:25 -05:00
Rusty Russell 14baaaa8ba fundpsbt: let caller specify locktime.
Required for dual funding where the opener sets it.

Changelog-Added: JSON-RPC: `fundpsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-18 11:38:25 -05:00
Rusty Russell bf5e99403e utxopsbt: make default to only allow unreserved utxos.
This more closely mirrors fundpsbt (which will only select unreserved ones)
but you can turn it off if you want to e.g. rbf in future.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: New low-level command `utxopsbt` to create PSBT from existing utxos.
2020-08-18 12:52:41 +09:30