Commit Graph

61 Commits

Author SHA1 Message Date
Rusty Russell 53566c47bd tools: remove optional csv field handling.
There are no feature-dependent fields left in the spec.  (I've also
made a PR for the spec to remove the tooling for it there).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-05-23 09:34:08 +09:30
Rusty Russell 5e76c74622 tools/generate_wire.py: don't declare unused for variable.
Ubuntu clang 15.0.2-1 complains:

```
wire/peer_exp_wiregen.c:257:14: error: variable 'i' set but not used [-Werror,-Wunused-but-set-variable]
        for (size_t i = 0; *plen != 0; i++) {
                    ^
wire/peer_exp_wiregen.c:1373:14: error: variable 'i' set but not used [-Werror,-Wunused-but-set-variable]
        for (size_t i = 0; *plen != 0; i++) {
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-11-18 12:26:59 +01:00
adi2011 286d6c3165 tools/gen: Always return bool! 2022-07-14 12:24:48 -05:00
Rusty Russell b19f3a5e7f devtools/decodemsg: don't require --onion for onion tlvs, fail if unknown tlvname.
Generate a table, let decodemsg sort it out.  Do more up-front work in
argparsing too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 83ee68ab06 common/tlvstream: put TLV checking back in the generic function.
Callers were supposed to call "tlv_fields_valid" after fromwire_tlv,
but few did.  Make this the default, and call the underlying function
directly where we want to be more flexible (one place).

This loses the ability to allow misordered fields, or to pass through
*any* even fields.  We restore that for special cases in the next
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell a770f51d0e tools/generate_wire.py: make functions allocate the TLV.
Requiring the caller to allocate them is ugly, and differs from
other types.

This means we need a context arg if we don't have one already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 04a152fbcf tools/generate-wire.py: make (and expose) individual TLV print functions.
When we actually put bolt12 fields (.e.g tlv_invoice) in onion messages,
that code will try to call printwire_tlv_invoice(), so expose it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 8f39002715 tools/generate-wire.py: have printwire routines do demarshal.
This works better in general: let printwire_x do the work of figuring
out how to demarshal x.  This is particularly important for TLVs, which
require a call to tlv_x_new() first.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 88de64a580 tools/generate-wire.py: simplify printwire routines, fix ... handling.
We make them return bool, and always use names `cursor` and `plen` in
callers, for simplicity.

Also, `...` means "loop until finished" not "loop this many bytes".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 12c7b156c5 tools/generate_wire.py: allow generated C files to have includes added.
We previously ignored --include= for these, but onion is about to start
needing bolt12.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell fa0c29f959 tools/generate_wire.py: tlvs should start with tlv_
No more "towire_offer", but "towire_tlv_offer".

This means we double-up on the unfortunately-named `tlv_payload` inside
the onion, but we should rename that in the spec when we remove
old payloads.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 00a0d09340 tools/check-includes.sh: test that c files include their .h files.
This is best-practice (to ensure prototypes match up), but there were a
few places we didn't (at least, directly).  Make it a requirement,
either of form "foo.h" or <dir/foo.h>.

The noise is the change to our print templates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell 1b8551923d tools: handle generating print templates for CSV without tlvs, messages
We want to use this to handle the simple description for channel_type.

It also needs to handle variable-size types (just like subtypes).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell ceb40dea38 lightningd: don't turn zero-length tlv fields into NULL.
Fixes: #4667
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
Christian Decker b4ead97517 tlv: Allow passing some extra types to accept when parsing the stream 2021-06-26 10:55:13 +09:30
Rusty Russell 2fea448498 gen/impl_template: fix generation of singleton varsize elements.
And as Lisa requested, add testcases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-04 16:13:08 +09:30
Rusty Russell ff8830876d wire/tlvstream: add tlv_make_fields helper to populate ->fields array.
This is vital for calculating merkle trees; I previously used
towire+fromwire to get this!

Requires generation change so we can magic the ARRAY_SIZE var (the C
pre-processor can't uppercase things).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell 5c167d16ab tools/generate-wire.py: use helpers.
This was terrible cut & paste.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell dc83e64003 tools/generate-wire: don't use void * pointers for tlv fromwire.
And fix up the one place which got it wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell 0e805427dc tools/generate-wire.py: strip trailing whitespace on lines, fix bolt quotes.
There's a lot of it, and it means we can't `make check-source` on
these files.

Also bring bolt quotes up-to-date.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 16:23:58 +09:30
Rusty Russell aa441e3b27 tools/generate-wire.py: fix loop logic for towire_xxx_array
We have to handle singletons which are arrays of variable-length entries:
this needs to be a ptr-to-ptr.

```C
struct blinded_payinfo {
        u32 fee_base_msat;
        u32 fee_proportional_millionths;
        u16 cltv_expiry_delta;
        u8 *features;
};
```

Before:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo *blindedpay;
```

After:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo **blindedpay;
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 897c53ce1c tools/generate-wire.py: fix loop logic for towire_xxx_array
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong.  This happens in the
the bolt13 patch:

```C
struct tlv_offer_tlvs_blindedpath {
        struct pubkey blinding;
        struct onionmsg_path **path;
};
```

Before:
wire/gen_bolt13_tlv.c:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 76813d6f90 tools/generate-wire.py: fix varsize assignment.
Code like this is suspicious:

	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);

In fact, it is a memory leak since we copy and don't free the fromwire
result.  Really, field_0 should be a pointer.

We don't hit this case (yet!) in spec-generated code, but I did for
bolt13.

Here's the difference in gen_test output:

```patch
  diff -ur /tmp/before/test/gen_test.c /tmp/after/test/gen_test.c
  --- /tmp/before/test/gen_test.c	2020-05-07 16:23:31.651611235 +0930
  +++ /tmp/after/test/gen_test.c	2020-05-07 16:20:54.232574482 +0930
  @@ -214,12 +214,12 @@
   static void towire_subtype_varsize_struct(u8 **p, const struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  -	towire_test_features(p, &subtype_varsize_struct->field_0);
  +	towire_test_features(p, subtype_varsize_struct->field_0);
   }
   static void fromwire_subtype_varsize_struct(const u8 **cursor, size_t *plen, struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  - 	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);
  + 	subtype_varsize_struct->field_0 = fromwire_test_features(subtype_varsize_struct, cursor, plen);
   }
   
   /* SUBTYPE: SUBTYPE_VAR_LEN */
  @@ -373,7 +373,7 @@
   
   	ptr = tal_arr(ctx, u8, 0);
   
  -	towire_test_features(&ptr, &r->tlv3->features);
  +	towire_test_features(&ptr, r->tlv3->features);
   
   	towire_amount_msat(&ptr, r->tlv3->amount_msat_1);
   
  @@ -385,7 +385,7 @@
   	struct tlv_test_n1 *r = vrecord;
   
   	r->tlv3 = tal(r, struct tlv_test_n1_tlv3);
  -	r->tlv3->features = *fromwire_test_features(r->tlv3, cursor, plen);
  +	r->tlv3->features = fromwire_test_features(r->tlv3, cursor, plen);
   	r->tlv3->amount_msat_1 = fromwire_amount_msat(cursor, plen);
   	r->tlv3->amount_msat_2 = fromwire_amount_msat(cursor, plen);
   }
  @@ -824,11 +824,11 @@
   
   	towire_test_short_id(&ptr, &r->tlv3->subtype);
   
  -	towire_subtype_var_len(&ptr, &r->tlv3->varlen_subtype);
  +	towire_subtype_var_len(&ptr, r->tlv3->varlen_subtype);
   
  -	towire_subtype_var_assign(&ptr, &r->tlv3->varlen_assigned);
  +	towire_subtype_var_assign(&ptr, r->tlv3->varlen_assigned);
   
  -	towire_subtype_varlen_varsize(&ptr, &r->tlv3->test_sbt_varlen_varsize);
  +	towire_subtype_varlen_varsize(&ptr, r->tlv3->test_sbt_varlen_varsize);
   
   		for (size_t i = 0; i < 2; i++)
   		towire_u32(&ptr, r->tlv3->arr_assign[i]);
  @@ -868,9 +868,9 @@
   
   	r->tlv3 = tal(r, struct tlv_test_n3_tlv3);
   	fromwire_test_short_id(cursor, plen, &r->tlv3->subtype);
  -	r->tlv3->varlen_subtype = *fromwire_subtype_var_len(r->tlv3, cursor, plen);
  -	r->tlv3->varlen_assigned = *fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  -	r->tlv3->test_sbt_varlen_varsize = *fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_subtype = fromwire_subtype_var_len(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_assigned = fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  +	r->tlv3->test_sbt_varlen_varsize = fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
   		for (size_t i = 0; i < 2; i++) {
   		u32 tmp;
   		tmp = fromwire_u32(cursor, plen);
  diff -ur /tmp/before/test/gen_test.h /tmp/after/test/gen_test.h
  --- /tmp/before/test/gen_test.h	2020-05-07 16:23:30.399617108 +0930
  +++ /tmp/after/test/gen_test.h	2020-05-07 16:20:52.912584680 +0930
  @@ -45,7 +45,7 @@
           struct test_short_id field_1;
   };
   struct subtype_varsize_struct {
  -        struct test_features field_0;
  +        struct test_features *field_0;
   };
   struct subtype_var_len {
           struct test_short_id *field_2;
  @@ -60,15 +60,15 @@
           struct test_short_id field3[2];
   };
   struct tlv_test_n1_tlv3 {
  -        struct test_features features;
  +        struct test_features *features;
           struct amount_msat amount_msat_1;
           struct amount_msat amount_msat_2;
   };
   struct tlv_test_n3_tlv3 {
           struct test_short_id subtype;
  -        struct subtype_var_len varlen_subtype;
  -        struct subtype_var_assign varlen_assigned;
  -        struct subtype_varlen_varsize test_sbt_varlen_varsize;
  +        struct subtype_var_len *varlen_subtype;
  +        struct subtype_var_assign *varlen_assigned;
  +        struct subtype_varlen_varsize *test_sbt_varlen_varsize;
           /*  array assigtest_nable */
           u32 arr_assign[2];
           /*  array structs */
  Binary files /tmp/before/test/gen_test.o and /tmp/after/test/gen_test.o differ
  Binary files /tmp/before/test/run-test-wire and /tmp/after/test/run-test-wire differ
  Binary files /tmp/before/test/run-test-wire.o and /tmp/after/test/run-test-wire.o differ
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell dc8458d1e2 tools: fix comment in template.
It's (usually) a .csv not an _csv file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Christian Decker c0b30ac907 wiregen: Add enums for TLV types so we can call them by their name
Suggested-by: Lisa Neigut <@niftynei>
Signed-off-by: Christian Decker <@cdecker>
2020-07-01 12:19:02 +02:00
Rusty Russell 197d1bcef2 wire: move towire/fromwire_short_channel_id out to bitcoin/short_channel_id.c
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-18 14:51:12 +02:00
Rusty Russell ebb7daed49 tools/generate-wire.py: don't prettify headers.
The formatting makes it harder for update-mocks, eg:

    /* Generated stub for fmt_wireaddr_without_port */
    char *fmt_wireaddr_without_port(const tal_t *ctx UNNEEDED, const struct wireaddr *a UNNEEDED)
    { fprintf(stderr, "fmt_wireaddr_without_port called!\n"); abort(); }
    /* Could not find declaration for fromwire_onionmsg_path */
    /* Generated stub for json_add_member */

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Rusty Russell b0c9059602 tools/generate-wire: no more lonely messages!
When we have only a single member in a TLV (e.g. an optional u64),
wrapping it in a struct is awkward.  This changes it to directly
access those fields.

This is not only more elegant (60 fewer lines), it would also be
more cache friendly.  That's right: cache hot singles!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-06 14:56:09 -05:00
Rusty Russell 11c21f97a6 devtools/decodemsg: don't crash if we have multiple tlv options.
We call tal_count(msg) after we've moved msg, and that causes an abort.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-07 12:48:13 -05:00
Christian Decker 5325ff6352 json-rpc: Don't let users send messages that are handled internally
We cannot let users use `sendcustommsg` to inject messages that are handled
internally since it could result in our internal state tracking being borked.
2020-01-28 23:50:52 +01:00
Rusty Russell eb6a768741 tools: don't use bytelength as array length for arrays in TLVs.
This matters now we have an array in tlv_init_tlvs!  We were overallocating
in fromwire by 32x!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:40 +01:00
Christian Decker 626675c83c tlv: Migrate tlv serialization to typesafe function 2019-12-03 00:37:15 +00:00
Christian Decker e12b5c3824 tlv: Add a typesafe serialization function for tlv namespaces
This is the counterpart to the typesafe deserialization function implemented
in an earlier commit.
2019-12-03 00:37:15 +00:00
Christian Decker 5a78671d9f wire: Remove unused fromwire_tlvs
We are now using the typesafe variant everywhere.
2019-12-03 00:37:15 +00:00
Christian Decker 69c17d2d31 wire: Let the TLV _is_valid function actually return validity
I got this one wrong myself, since the function name implied a boolean
result. So I changed it to take the optional err_index as argument.
2019-12-03 00:37:15 +00:00
darosior 670f92002d tools/gen/impl_template: correct one-line for-loops indentation 2019-11-26 21:30:25 +01:00
darosior 5fb8e0aade tools/gen/impl_template: correct tlvs fromwire's for-loop 2019-11-26 21:30:25 +01:00
Christian Decker 2519f934aa tlv: Add validity check codegen for the tlv namespaces
Since the parser itself just parses and doesn't include validation anymore we
need to put that functionality somewhere. The validation consists of enforcing
that the types are in monotonically increasing order without duplicates and
that for the even types we know how to handle it.
2019-11-22 04:40:25 +00:00
Christian Decker 5794c83b4d tlv: Add typesafe fromwire codegen for TLV namespaces
We were weaving in and out of generic code through `fromwire_tlvs` with custom
parameters defining the types in that namespace. This hard-wires the parser
with the namespace's types. Slowly trying to deprecate `fromwire_tlvs` in
favor of this typesafe variant.
2019-11-22 04:40:25 +00:00
Christian Decker 2255024ead tlv: Add raw fields so we can store unknown fields as well 2019-11-22 04:40:25 +00:00
lisa neigut 3f1f075421 tools: add ability to wrap wire messages with ifs
Makes it possible to hide wire messages behind EXPERIMENTAL_FEATURES
flag.
2019-10-10 05:57:45 +00:00
lisa neigut e99720344e wire-gen: ensure that tlv messages are correctly ordered
Our TLV serializer relies on TLV outputs to be ordered by type
number. Prior to this commit we relied on 1) the ordering in the
RFC to be correct and 2) users to be using a version of Python that
respects stable ordering of dicts (i.e. Python 3.7+)

Instead of relying on these implicitly, we now explicitly sort messages
by type number when the TLV sets.

Resolves #2956.

Thanks-To: @ScottTre for the sort function
Reported-By: @ZmnSCPxj
2019-08-21 03:53:54 +00:00
Rusty Russell b10e0e08bb tools/generate-wire.py: add option to expose tlv_record_type
Next update adds TLV test vectors: without this, we get a warning
about them being unused.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-02 17:32:48 +02:00
Rusty Russell 3f8600e9c0 tools/generate-wire.py: handle implicit tlv length fields.
TLVs have an implicit `len` field, so allow expressions containing
that (eg. `len-1`), but assume it means "the remainder of the
message".

This means in most places, f.size() needs an fallback for the
implicit-length case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
Rusty Russell 2e68b88be8 tools: fix tlv generation
We need to hand -s to both header and body generation, or neither:

wire/gen_peer_wire.c:53:13: error: static declaration of ‘towire_channel_update_timestamps’ follows non-static declaration
In file included from wire/gen_peer_wire.c:5:
./wire/gen_peer_wire.h:78:6: note: previous declaration of ‘towire_channel_update_timestamps’ was here

We also need it for printwire, otherwise we get static unused functions for subtypes:

devtools/gen_print_wire.c:155:13: error: ‘printwire_channel_update_checksums’ defined but not used [-Werror=unused-function]
 static void printwire_channel_update_checksums(const char *fieldname, const u8 **cursor, size_t *plen)
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
devtools/gen_print_wire.c:133:13: error: ‘printwire_channel_update_timestamps’ defined but not used [-Werror=unused-function]
 static void printwire_channel_update_timestamps(const char *fieldname, const u8 **cursor, size_t *plen)
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
lisa neigut 42251cc5b9 bolt-gen: fixup subtype nested varsize definition
variable length varsized subtypes were being allocated as
structs at not pointers; this fixes that.
2019-07-27 05:18:27 +00:00
lisa neigut b6d4c372bb bolt-gen: fix broken 'exposed subtype' generation
borked in 96bf7aea; fixed here.
2019-07-24 06:31:46 +00:00
lisa neigut 068496298c bolt-gen: rm unused bolt-generator; rename new bolt generator
delete now unused wire-generator and replace it with the newer
version.
2019-07-24 06:31:46 +00:00
lisa neigut 236d26308f bolt-gen: make optional 'assignable' fields work
make assignable 'optional' fields work, add test for them
2019-07-24 06:31:46 +00:00
lisa neigut ff82c918a5 bolt-gen: formatting cleanups
add brackets for clarity when printing subtypes; also remove
unnecessary exception (`find_` will raise an exception if
the key is not present)
2019-07-24 06:31:46 +00:00