Commit Graph

8897 Commits

Author SHA1 Message Date
Michael Schmoock 2816c08036 openchannel: test new hook chainable mechanics 2020-09-08 06:14:29 +09:30
Michael Schmoock a3e18a6aba openchannel: make openchannel hook chainable
This will make the `openchannel_hook` chainable. Logic is as follows:
 - The first plugin that rejects terminates the chain
 - If more than one plugin uses the `close_to` parameter, take the first
   value and log_unusual for the others.

Changelog-Added: openchannel_hook is now chainable
2020-09-08 06:14:29 +09:30
Vincenzo Palazzo 6e16ee0eaa Alignment doc with the actual implementation
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2020-09-07 11:07:34 +09:30
Christian Decker 037f7201be make: Do not delete deployable-lightning.pdf on maintainer-clean
It is not part of the default documentation generation, thus will show up in
the git diff every time a maintainer-clean is done. We could also add it to
the default documentation generation, however it requires some rather
heavy-weight latex dependencies, and the file hasn't changed in several years.
2020-09-07 11:03:29 +09:30
Christian Decker 0d5066c48b github: Mark some more files as derived / generated
Since we started committing generated files as well we can mark them as
derived so that they appear collapsed in the diff, reducing noise while
reviewing.
2020-09-07 11:03:29 +09:30
Christian Decker 6fc0433536 make: Do not clean sqlgen files in non-maintainer clean
Generating these requires python tooling which may not be present, so only
clean them in maintainer mode. Also remove `wallet/statements.po` which is no
longer used or generated at all.
2020-09-07 11:03:29 +09:30
Christian Decker d7ff74e8f6 json: Only start parsing once we're sure we have a message
PR #3957 improved performance considerably, however we still look over the
entire message for the message separator. If instead we just look in the
incrementally read data, we remove the quadratic behavior for large messages.

This is safe since we then loop over the messages which would drain any
message separator from the buffer before we attempt the next read.

Changelog-Fixed: bcli: Significant speedups for block synchronization
2020-09-02 13:21:32 +02:00
Christian Decker f2797f303a block: Compute the txids only once
We're not going to mutate transactions in a block, so computing the txids
every time we need them is a waste, let's compute them upfront and use them
afterwards.
2020-09-02 13:21:32 +02:00
Christian Decker 31c67f1392 bcli: Avoid tal_fmt when handling a raw hex block
We were using `tal_fmt` to truncate off the last byte of the
response (newline), which required an allocation, a call to `vsnprintf` and a
copy of the block contents. This being >2MB in size on mainnet was rather
expensive.

We now just signal the end of the string by overwriting the trailing newline,
and stealing directly onto the result.
2020-09-02 13:21:32 +02:00
Rusty Russell e5e0ccaba0 wallet/Makefile: fix typo in rule to regen sqlgen files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell 8670cb333f wallet/Makefile: fix wallet_clean vs wallet-clean typo.
And remove toplevel target we no longer try to support.

Reported-byL @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell 82f2f43425 Generated files: more merge fallout.
We changed the generation templates!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell cd4c99d028 .gitignore: don't ignore checked-in files.
It doesn't have any effect, but would confuse if we added one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell e4b5a1e0e8 dev: option to set what features we offer.
And use it to smoke-test !option_static_remotekey.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 10:52:49 +09:30
Rusty Russell 3a2191fcfe wallet: db_bind_talarr and db_column_talarr helpers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 39e4796ae3 json_command: command_fail_badparam helper.
It's common to want to complain a token is not what we expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell aa441e3b27 tools/generate-wire.py: fix loop logic for towire_xxx_array
We have to handle singletons which are arrays of variable-length entries:
this needs to be a ptr-to-ptr.

```C
struct blinded_payinfo {
        u32 fee_base_msat;
        u32 fee_proportional_millionths;
        u16 cltv_expiry_delta;
        u8 *features;
};
```

Before:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo *blindedpay;
```

After:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo **blindedpay;
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 897c53ce1c tools/generate-wire.py: fix loop logic for towire_xxx_array
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong.  This happens in the
the bolt13 patch:

```C
struct tlv_offer_tlvs_blindedpath {
        struct pubkey blinding;
        struct onionmsg_path **path;
};
```

Before:
wire/gen_bolt13_tlv.c:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 76813d6f90 tools/generate-wire.py: fix varsize assignment.
Code like this is suspicious:

	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);

In fact, it is a memory leak since we copy and don't free the fromwire
result.  Really, field_0 should be a pointer.

We don't hit this case (yet!) in spec-generated code, but I did for
bolt13.

Here's the difference in gen_test output:

```patch
  diff -ur /tmp/before/test/gen_test.c /tmp/after/test/gen_test.c
  --- /tmp/before/test/gen_test.c	2020-05-07 16:23:31.651611235 +0930
  +++ /tmp/after/test/gen_test.c	2020-05-07 16:20:54.232574482 +0930
  @@ -214,12 +214,12 @@
   static void towire_subtype_varsize_struct(u8 **p, const struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  -	towire_test_features(p, &subtype_varsize_struct->field_0);
  +	towire_test_features(p, subtype_varsize_struct->field_0);
   }
   static void fromwire_subtype_varsize_struct(const u8 **cursor, size_t *plen, struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  - 	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);
  + 	subtype_varsize_struct->field_0 = fromwire_test_features(subtype_varsize_struct, cursor, plen);
   }
   
   /* SUBTYPE: SUBTYPE_VAR_LEN */
  @@ -373,7 +373,7 @@
   
   	ptr = tal_arr(ctx, u8, 0);
   
  -	towire_test_features(&ptr, &r->tlv3->features);
  +	towire_test_features(&ptr, r->tlv3->features);
   
   	towire_amount_msat(&ptr, r->tlv3->amount_msat_1);
   
  @@ -385,7 +385,7 @@
   	struct tlv_test_n1 *r = vrecord;
   
   	r->tlv3 = tal(r, struct tlv_test_n1_tlv3);
  -	r->tlv3->features = *fromwire_test_features(r->tlv3, cursor, plen);
  +	r->tlv3->features = fromwire_test_features(r->tlv3, cursor, plen);
   	r->tlv3->amount_msat_1 = fromwire_amount_msat(cursor, plen);
   	r->tlv3->amount_msat_2 = fromwire_amount_msat(cursor, plen);
   }
  @@ -824,11 +824,11 @@
   
   	towire_test_short_id(&ptr, &r->tlv3->subtype);
   
  -	towire_subtype_var_len(&ptr, &r->tlv3->varlen_subtype);
  +	towire_subtype_var_len(&ptr, r->tlv3->varlen_subtype);
   
  -	towire_subtype_var_assign(&ptr, &r->tlv3->varlen_assigned);
  +	towire_subtype_var_assign(&ptr, r->tlv3->varlen_assigned);
   
  -	towire_subtype_varlen_varsize(&ptr, &r->tlv3->test_sbt_varlen_varsize);
  +	towire_subtype_varlen_varsize(&ptr, r->tlv3->test_sbt_varlen_varsize);
   
   		for (size_t i = 0; i < 2; i++)
   		towire_u32(&ptr, r->tlv3->arr_assign[i]);
  @@ -868,9 +868,9 @@
   
   	r->tlv3 = tal(r, struct tlv_test_n3_tlv3);
   	fromwire_test_short_id(cursor, plen, &r->tlv3->subtype);
  -	r->tlv3->varlen_subtype = *fromwire_subtype_var_len(r->tlv3, cursor, plen);
  -	r->tlv3->varlen_assigned = *fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  -	r->tlv3->test_sbt_varlen_varsize = *fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_subtype = fromwire_subtype_var_len(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_assigned = fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  +	r->tlv3->test_sbt_varlen_varsize = fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
   		for (size_t i = 0; i < 2; i++) {
   		u32 tmp;
   		tmp = fromwire_u32(cursor, plen);
  diff -ur /tmp/before/test/gen_test.h /tmp/after/test/gen_test.h
  --- /tmp/before/test/gen_test.h	2020-05-07 16:23:30.399617108 +0930
  +++ /tmp/after/test/gen_test.h	2020-05-07 16:20:52.912584680 +0930
  @@ -45,7 +45,7 @@
           struct test_short_id field_1;
   };
   struct subtype_varsize_struct {
  -        struct test_features field_0;
  +        struct test_features *field_0;
   };
   struct subtype_var_len {
           struct test_short_id *field_2;
  @@ -60,15 +60,15 @@
           struct test_short_id field3[2];
   };
   struct tlv_test_n1_tlv3 {
  -        struct test_features features;
  +        struct test_features *features;
           struct amount_msat amount_msat_1;
           struct amount_msat amount_msat_2;
   };
   struct tlv_test_n3_tlv3 {
           struct test_short_id subtype;
  -        struct subtype_var_len varlen_subtype;
  -        struct subtype_var_assign varlen_assigned;
  -        struct subtype_varlen_varsize test_sbt_varlen_varsize;
  +        struct subtype_var_len *varlen_subtype;
  +        struct subtype_var_assign *varlen_assigned;
  +        struct subtype_varlen_varsize *test_sbt_varlen_varsize;
           /*  array assigtest_nable */
           u32 arr_assign[2];
           /*  array structs */
  Binary files /tmp/before/test/gen_test.o and /tmp/after/test/gen_test.o differ
  Binary files /tmp/before/test/run-test-wire and /tmp/after/test/run-test-wire differ
  Binary files /tmp/before/test/run-test-wire.o and /tmp/after/test/run-test-wire.o differ
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell be0038e49b Makefile: include common objects in ALL_C_SOURCES.
They didn't have correct dependencies without this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:30:25 +09:30
Rusty Russell 5bedcbeed7 doc: regenerate lightningd-config man page.
Merged as SHA256STAMP was going in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:30:25 +09:30
Rusty Russell 0c7d04bd98 libplugin-pay: fix default CLTV.
This was changed recently, but without a bolt quote, we didn't find
this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 06:59:13 +09:30
Rusty Russell 74e960c20b pytest: increase channel sizes so they reach min capacity with option_anchor_outputs
With a feerate of 7500perkw and subtracting 660 sats for anchors, a
20,000 sat channel has capacity about 9800 sat, below our default:

	You gave bad parameters: channel capacity with funding 20000sat, reserves 546sat/546sat, max_htlc_value_in_flight_msat is 18446744073709551615msat, channel capacity is 9818sat, which is below 10000000msat

So bump channel amounts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell 9fabf614f4 pytest: run test_opening_tiny_channel with option_anchor_outputs.
It's now easy to fix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell 9b8f08a8a0 lightningd: return min-capacity-sat to its intended purpose.
And document exactly what it does: insist that an HTLC can pass of
this value (module assumptions of feerate).

Note that we remove the "is_opener" test from the capacity calculation
for anchor fees: it doesn't matter which side it is, someone has to pay
for anchor fees to it deducts from capacity.

This change breaks the test, which we rewrite.

Changelog-Changed: config: `min-capacity-sat` is now stricter about checking usable capacity of channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell 005fbbe4a6 travis: don't suppress make output.
It's far less verbose than it used to be!  And it's occasionally useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell b3f6a0e78d Travis: don't install requirements.txt if we're just building.
This tests our instructions in INSTALL.md, which say you only need
requirements.txt for development or running tests, not building.

Also removes the unused SOURCE_CHECK_ONLY flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 390c6470ad configure: we don't need mako any more.
And guard the mako-specific tests with a check instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 2c9ad7ea29 external/Makefile: fix mkdir rule for libbacktrace.a
The $(TARGET_DIR) may not exist: use mkdir -p.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell b3d5220da5 tools/refresh-submodules.sh: don't exit, but wait if called in parallel.
Otherwise make thinks we're done, and we can get errors.  Include
primitive code if we abort build halfway and leave .refresh-submodules dir.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 95a0249272 external/Makefile: fix build from fresh git clone.
```
make: *** No rule to make target 'external/gheap/gheap.h', needed by 'bitcoin/chainparams.o'.  Stop.
make: *** Waiting for unfinished jobs....
```

Just simplify the Makefile to make all the external headers we use
dependent on submodcheck.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 3c6af3efb4 Makefile: commit and preserve all the wiregenerated files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Build: we no longer require extra Python modules to build.
2020-08-31 21:33:26 -05:00
Rusty Russell e9a435ce82 Makefile: protect .po file and gen_db files with SHA256STAMP.
And rename them so they're not cleared by `make clean`.  We leave the
old rules in place so old files get cleaned still.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell bb250358ae Makefile: force regeneration of manpages, wire source.
Now that SHA256STAMP protects us, we can avoid timestamps altogether
so we don't get missing builds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 1746406e41 Makefile: normalize all the Makefiles
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS.  Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 2835cf8a43 tools/check-bolt: don't trigger ourselves.
We're going to make check-source cover every C file soon.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell 8150d28575 Makefile: use generic rules to make spec-derived sources.
Now we use the same Makefile rules for all CSV->C generation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell a00179d557 doc, wiregen: use SHA256 stamps instead of git versions.
This should be more robust in future: we SHA256 all of the deps.
For wiregen we prefix with EXPERIMENTAL_FEATURES, since it can effect them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Christian Decker 71e713c722 pyln-testing: Bump version to 0.9.0 and add Makefile for release 2020-08-30 20:03:42 +02:00
Christian Decker 0b50fd0f96 pyln-testing: Add a dummy test to check functionality 2020-08-30 20:03:42 +02:00
Christian Decker 94d4d20d5f pytest: Stabilize test_mpp_interference_2 2020-08-30 20:03:42 +02:00
Christian Decker 8b080dff90 pytest: Test that amountless invoices show the received amount
This was asked in a chat somewhere, and I just want to make sure it works as
expected.
2020-08-30 20:03:42 +02:00
Christian Decker 81d8525d86 pyln: Make the pytest requirement less stringent 2020-08-30 20:03:42 +02:00
Christian Decker 1884500bf8 pytest: Fix a typo in a test plugin
Reported-by: Henri Beqi
2020-08-30 20:03:42 +02:00
Christian Decker b71f6e8eaa repro: Skip confirmation of package installation in repro-build.sh 2020-08-30 20:03:42 +02:00
Christian Decker e420bfd6a8 pytest: Skip test_lockup_drain if VALGRIND and not DEVELOPER
The test was failing very reliably, by not raising the expected exception. My
best guess is that it needs more synchronization to cause the issue.
2020-08-30 20:03:42 +02:00
Matt Whitlock abbc712afb allow building without sqlite3
Changelog-Changed: build: SQLite3 is no longer a hard build requirement. C-Lightning can now be built to support only the PostgreSQL back-end.
2020-08-30 12:44:56 +02:00
Matt Whitlock c1aa33a62a configure: hoist defaulting of PG_CONFIG var to top of script
See: https://github.com/ElementsProject/lightning/pull/3995#discussion_r478899639
2020-08-30 10:24:37 +09:30
Matt Whitlock 94472f67c5 configure: Use pg_config to locate the library location too
Also, allow the pg_config binary to be specified through the PG_CONFIG
environment variable, defaulting to 'pg_config' if unset. Explicitly
setting PG_CONFIG to an empty string will forcibly disable PostgreSQL
support, even if a PostgreSQL library is installed.

Changelog-Fixed: build: On systems with multiple installed versions of the PostgreSQL client library, C-Lightning might link against the wrong version or fail to find the library entirely. `./configure` now uses `pg_config` to locate the library.
2020-08-30 10:24:37 +09:30
Rusty Russell dd8cd81e91 listpeers: add `features` array using BOLT9 names.
It's actually not possible to currently tell if you're using anchor_outputs
with a peer (since it depends on whether you both supported it at *channel open*).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-added: JSON-RPC: `listpeers` shows `features` list for each channel.
2020-08-28 14:46:09 -05:00