Commit Graph

8400 Commits

Author SHA1 Message Date
Matt Whitlock c1aa33a62a configure: hoist defaulting of PG_CONFIG var to top of script
See: https://github.com/ElementsProject/lightning/pull/3995#discussion_r478899639
2020-08-30 10:24:37 +09:30
Matt Whitlock 94472f67c5 configure: Use pg_config to locate the library location too
Also, allow the pg_config binary to be specified through the PG_CONFIG
environment variable, defaulting to 'pg_config' if unset. Explicitly
setting PG_CONFIG to an empty string will forcibly disable PostgreSQL
support, even if a PostgreSQL library is installed.

Changelog-Fixed: build: On systems with multiple installed versions of the PostgreSQL client library, C-Lightning might link against the wrong version or fail to find the library entirely. `./configure` now uses `pg_config` to locate the library.
2020-08-30 10:24:37 +09:30
Rusty Russell dd8cd81e91 listpeers: add `features` array using BOLT9 names.
It's actually not possible to currently tell if you're using anchor_outputs
with a peer (since it depends on whether you both supported it at *channel open*).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-added: JSON-RPC: `listpeers` shows `features` list for each channel.
2020-08-28 14:46:09 -05:00
ZmnSCPxj jxPCSnmZ 932709cad9 plugins/libplugin-pay.c: Micro-optimize start_block sampling.
Using `waitblockheight 0` is a very slightly faster query than `getinfo`.
Also, avoid querying blockheight for child payments (allow `waitblockheight`
paymod to provide the blockheight returned from the `waitblockheight`, and
just resample the starting blockheight from the parent).

Changelog-None: pointless micro-optimization
2020-08-28 16:40:27 +02:00
ZmnSCPxj jxPCSnmZ 05daa8e5f3 plugins/libplugin-pay.c: Micro-optimization of plugin_is_finished.
This was checked with `gcc -S -O2` to see how an optimized build
would compile the function.
The original code completed calls into each child (and the `.s`
file showed that GCC 9.x was not smart enough to do early-out).

This modification explicitly does early-out, and avoids call-return
stack overhead for the common case where a payment is an ancestor
of a long line of single-child payments due to retrying.

Changelog-None: pointless micro-optimization
2020-08-28 16:40:27 +02:00
niftynei 90662e03a1 travis: use bitcoind v0.20.1
Prior to v0.20.1, PSBT parsing is stricter and doesn't allow both a
witness_utxo and non_witness_utxo to be set.
2020-08-28 11:45:14 +02:00
ZmnSCPxj jxPCSnmZ a1ba214011 channeld/channeld.c: Log and ignore repeated WIRE_CHANNEL_REESTABLISH.
Fixes: #3608

Changelog-Changed: protocol: Ignore (and log as "unusual") repeated `WIRE_CHANNEL_REESTABLISH` messages, to be compatible with buggy peer software that sometimes does this.
2020-08-28 14:29:35 +09:30
niftynei 26f116041c contrib: start_ln now creates variable number of nodes
Re-write start_ln such that we can create up to 10 nodes locally for
testing. Useful for scenarios where more than two nodes are needed

Changelog-Changed: contrib: startup_regtest.sh `startup_ln` now takes a number of nodes to create as a parameter
2020-08-28 14:27:56 +09:30
Christian Decker 0a501b3646 configure: Use pg_config to locate the header location
Changelog-Fixed: build: On some operating systems the postgresql library would not get picked up. `./configure` now uses `pg_config` to locate the headers.
2020-08-28 11:59:57 +09:30
Rusty Russell 3ae4ce736d Apply @cdecker typo suggestions from code review
Co-authored-by: Christian Decker <decker.christian@gmail.com>
2020-08-28 10:56:50 +09:30
Rusty Russell e065cd6be4 devtool/route: simple routing tool and benchmark.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 5714a8c139 devtools/topology: new tool to explore lightning topology.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 6815a1d9a2 common/route: routing helpers built on gossmap and dijksra.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 59cc2f7559 common/dijkstra: routine to calculate shortest/cheapest path.
The user supplies callbacks to do channel selection and comparison.
Note that this continues to map the entire network; not just to the
source, for use with random routing.

Benchmarks: (using current mainnet gossip store)
	/devtools/route gossip-store-2020-07-27 all 03c981ed4ad15837f29a212dc8cf4b31f274105b7c95274a41449bf496ebd2fe10 | grep 'Time to find path'

With nothing (i.e. DEVELOPER build)
	Averages 17ms

With -Og (i.e. standard non-DEVELOPER build)
	Averages 14ms

With -O3 -flto:
	Averages 4ms

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 5d9d21d64d external/gheap: new submodule.
It's actually entirely in the header file, so no linking needed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell daba3e7deb common/gossmap: helper to map the gossip store.
I went overboard on optimization.  I am so sorry:
1. Squeezed channel min/max into 16 bits.
2. Uses mmap and leaves node_ids in the file.
3. Uses offsets instead of pointers where possible.
4. Uses custom free-list to allocate inside arrays.
5. Ignores our autogenerated marshalling code in favor of direct derefs.
6. Carefully aligns everything so we use minimal ram.

The result is that the current gossip_store:
 - load time (-O3 -flto laptop): 40msec
 - load time (-g laptop i.e. DEVELOPER=0): 60msec
 - load time (-O0 laptop i.e. DEVELOPER=1): 110msec
 - Total memory: 2.6MB:
   - 1.5MB for the array of channels
   - 512k for the channel htable to map scid -> channel.
   - 320k for the node htable to map nodeid -> node.
   - 192k for the array of channels inside each node
   - 94k for the array of nodes

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 7dd6f8f2b5 gossipd: add tombstone when we remove a channel.
For those following along at home.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 496c0dd1e6 common/random_select: central place for reservoir sampling.
Turns out we can make quite a simple API out of it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-28 10:56:50 +09:30
Rusty Russell 12d0d5c185 amount: cleanup usage.
We've got some recently-added primitives which help.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-27 18:16:28 +02:00
Rusty Russell 4a9c162211 pytest: disable test_mpp_interference_2.
We're failing this too often: we'd fail it more but it's disabled
with VALGRIND (it shouldn't be: @slow_test removes VALGRIND if SLOW_MACHINE
is set).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-27 13:51:22 +02:00
niftynei c8579b99d0 pyln-proto: use vals for subtype parsing 2020-08-27 10:20:16 +09:30
Christian Decker 21d87f7075 pay: Implement simple presplit fix for ludicrous amounts
This is the simplest possible fix: increase the target amount until we get
the desired number of parts, while still bucketizing payments together that
are in approximately the same size.

The current logic puts all payments that are in the range x < amount <= 16*x
in the same bucket, making them harder to distinguish.

Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-27 10:19:21 +09:30
Christian Decker 3f6809c934 pytest: Reproduce ludicrous presplit issue
Huge payment + presplit modifier = too many HTLCs...
2020-08-27 10:19:21 +09:30
Christian Decker f5985171bd gossipd: Make gossipd much quieter
We are logging way too much from gossipd, causing noisy logs. This PR reduces
logs for incoming messages to those that actually caused a change in our
internal state (duplicate and old messages are just dropped silently now).

Changelog-Changed: gossipd: The `gossipd` is now a lot quieter, and will log only when a message changed our network topology.
2020-08-27 09:45:35 +09:30
Christian Decker bdad5a6f94 pytest: Stabilize test_onchain_middleman
It wasn't waiting for the tx to be in the mempool
2020-08-27 09:45:35 +09:30
Christian Decker 8f028272f4 pytest: Stabilize test_pay_get_error_with_update 2020-08-27 09:45:35 +09:30
Christian Decker 437c411769 pyln-testing: Have LightningNode.fund_channel poll 2020-08-27 09:45:35 +09:30
Christian Decker 0431a29a26 pyln-testing: Poll for channel active instead of watching logs 2020-08-27 09:45:35 +09:30
Christian Decker 8f05d28049 pytest: Stabilize test_onchaind_replay 2020-08-27 09:45:35 +09:30
ZmnSCPxj jxPCSnmZ 128adf0938 plugins/libplugin-pay.c: Round-robin routehints when splitting.
This improves the success rate of `test_mpp_interference_2`, though
still not quite up to the level that we can remove `@flaky` from it.
2020-08-26 09:29:46 +09:30
ZmnSCPxj jxPCSnmZ ba3f38077e lightningd/invoice.c: Use round-robin channel selection.
Changelog-Changed: We now make MPP-aware routehints in invoices.
2020-08-26 09:29:46 +09:30
ZmnSCPxj jxPCSnmZ a9a11265dd lightningd/lightningd.h: Maintain a round-robin list of channels. 2020-08-26 09:29:46 +09:30
ZmnSCPxj jxPCSnmZ f50951a0d6 tests/test_pay.py: Motivating scenario for round-robin routehinting. 2020-08-26 09:29:46 +09:30
ZmnSCPxj jxPCSnmZ 8c763bb7be contrib/pyln-testing/pyln/testing/utils.py: Add a fundbalancedchannel method to LightningNode, for when you want to Thanos your channels. 2020-08-26 09:29:46 +09:30
Sergi Delgado Segura 3edbacbb3c pyln: hexlify -> bytes.hex() in pyln.proto.wire 2020-08-26 06:09:53 +09:30
Sergi Delgado Segura 27e495efa5 pyln: Updates proto to use coincurve for Public and Private keys 2020-08-26 06:09:53 +09:30
ZmnSCPxj jxPCSnmZ c27d7a3110 plugins/libplugin-pay.c: Store the route description, and re-report on failure. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ 6468616c02 plugins/libplugin-pay.c: Propagate local_id from parent to child payment object. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ d8678467fa plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ d89c77c0ce plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ 0d2d85ab5f plugins/libplugin-pay.c: Also print events that create new sub-payments. 2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ 98583e84b5 plugins/libplugin-pay.c: Give cmd id and partid for each log message.
Changelog-None: internal debugging

Makes it easier to debug payments with tons of splits.
2020-08-25 12:17:18 +02:00
ZmnSCPxj jxPCSnmZ e80d733399 .gitignore: Add missing gitignores. 2020-08-25 12:10:05 +02:00
Rusty Russell dc8458d1e2 tools: fix comment in template.
It's (usually) a .csv not an _csv file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 3e52d4100d common: convert to new wire generation style.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 398b4806b9 connectd: convert to new wire generation style.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell fe8564555f closingd: convert to new wire generation style.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 6d09c0eba9 openingd: convert wire to new scheme.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell dffbf8de85 gossipd: convert wire to new scheme.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 1702c7a69a hsmd: convert to new wire generation style.
Note that other directories were explicitly depending on the generated
file, instead of relying on their (already existing) dependency on 
$(LIGHTNINGD_HSM_CLIENT_OBJS), so we remove that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30