Commit Graph

35 Commits

Author SHA1 Message Date
Rusty Russell 5c167d16ab tools/generate-wire.py: use helpers.
This was terrible cut & paste.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell dc83e64003 tools/generate-wire: don't use void * pointers for tlv fromwire.
And fix up the one place which got it wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell 0e805427dc tools/generate-wire.py: strip trailing whitespace on lines, fix bolt quotes.
There's a lot of it, and it means we can't `make check-source` on
these files.

Also bring bolt quotes up-to-date.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 16:23:58 +09:30
Rusty Russell 897c53ce1c tools/generate-wire.py: fix loop logic for towire_xxx_array
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong.  This happens in the
the bolt13 patch:

```C
struct tlv_offer_tlvs_blindedpath {
        struct pubkey blinding;
        struct onionmsg_path **path;
};
```

Before:
wire/gen_bolt13_tlv.c:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell 76813d6f90 tools/generate-wire.py: fix varsize assignment.
Code like this is suspicious:

	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);

In fact, it is a memory leak since we copy and don't free the fromwire
result.  Really, field_0 should be a pointer.

We don't hit this case (yet!) in spec-generated code, but I did for
bolt13.

Here's the difference in gen_test output:

```patch
  diff -ur /tmp/before/test/gen_test.c /tmp/after/test/gen_test.c
  --- /tmp/before/test/gen_test.c	2020-05-07 16:23:31.651611235 +0930
  +++ /tmp/after/test/gen_test.c	2020-05-07 16:20:54.232574482 +0930
  @@ -214,12 +214,12 @@
   static void towire_subtype_varsize_struct(u8 **p, const struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  -	towire_test_features(p, &subtype_varsize_struct->field_0);
  +	towire_test_features(p, subtype_varsize_struct->field_0);
   }
   static void fromwire_subtype_varsize_struct(const u8 **cursor, size_t *plen, struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  - 	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);
  + 	subtype_varsize_struct->field_0 = fromwire_test_features(subtype_varsize_struct, cursor, plen);
   }
   
   /* SUBTYPE: SUBTYPE_VAR_LEN */
  @@ -373,7 +373,7 @@
   
   	ptr = tal_arr(ctx, u8, 0);
   
  -	towire_test_features(&ptr, &r->tlv3->features);
  +	towire_test_features(&ptr, r->tlv3->features);
   
   	towire_amount_msat(&ptr, r->tlv3->amount_msat_1);
   
  @@ -385,7 +385,7 @@
   	struct tlv_test_n1 *r = vrecord;
   
   	r->tlv3 = tal(r, struct tlv_test_n1_tlv3);
  -	r->tlv3->features = *fromwire_test_features(r->tlv3, cursor, plen);
  +	r->tlv3->features = fromwire_test_features(r->tlv3, cursor, plen);
   	r->tlv3->amount_msat_1 = fromwire_amount_msat(cursor, plen);
   	r->tlv3->amount_msat_2 = fromwire_amount_msat(cursor, plen);
   }
  @@ -824,11 +824,11 @@
   
   	towire_test_short_id(&ptr, &r->tlv3->subtype);
   
  -	towire_subtype_var_len(&ptr, &r->tlv3->varlen_subtype);
  +	towire_subtype_var_len(&ptr, r->tlv3->varlen_subtype);
   
  -	towire_subtype_var_assign(&ptr, &r->tlv3->varlen_assigned);
  +	towire_subtype_var_assign(&ptr, r->tlv3->varlen_assigned);
   
  -	towire_subtype_varlen_varsize(&ptr, &r->tlv3->test_sbt_varlen_varsize);
  +	towire_subtype_varlen_varsize(&ptr, r->tlv3->test_sbt_varlen_varsize);
   
   		for (size_t i = 0; i < 2; i++)
   		towire_u32(&ptr, r->tlv3->arr_assign[i]);
  @@ -868,9 +868,9 @@
   
   	r->tlv3 = tal(r, struct tlv_test_n3_tlv3);
   	fromwire_test_short_id(cursor, plen, &r->tlv3->subtype);
  -	r->tlv3->varlen_subtype = *fromwire_subtype_var_len(r->tlv3, cursor, plen);
  -	r->tlv3->varlen_assigned = *fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  -	r->tlv3->test_sbt_varlen_varsize = *fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_subtype = fromwire_subtype_var_len(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_assigned = fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  +	r->tlv3->test_sbt_varlen_varsize = fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
   		for (size_t i = 0; i < 2; i++) {
   		u32 tmp;
   		tmp = fromwire_u32(cursor, plen);
  diff -ur /tmp/before/test/gen_test.h /tmp/after/test/gen_test.h
  --- /tmp/before/test/gen_test.h	2020-05-07 16:23:30.399617108 +0930
  +++ /tmp/after/test/gen_test.h	2020-05-07 16:20:52.912584680 +0930
  @@ -45,7 +45,7 @@
           struct test_short_id field_1;
   };
   struct subtype_varsize_struct {
  -        struct test_features field_0;
  +        struct test_features *field_0;
   };
   struct subtype_var_len {
           struct test_short_id *field_2;
  @@ -60,15 +60,15 @@
           struct test_short_id field3[2];
   };
   struct tlv_test_n1_tlv3 {
  -        struct test_features features;
  +        struct test_features *features;
           struct amount_msat amount_msat_1;
           struct amount_msat amount_msat_2;
   };
   struct tlv_test_n3_tlv3 {
           struct test_short_id subtype;
  -        struct subtype_var_len varlen_subtype;
  -        struct subtype_var_assign varlen_assigned;
  -        struct subtype_varlen_varsize test_sbt_varlen_varsize;
  +        struct subtype_var_len *varlen_subtype;
  +        struct subtype_var_assign *varlen_assigned;
  +        struct subtype_varlen_varsize *test_sbt_varlen_varsize;
           /*  array assigtest_nable */
           u32 arr_assign[2];
           /*  array structs */
  Binary files /tmp/before/test/gen_test.o and /tmp/after/test/gen_test.o differ
  Binary files /tmp/before/test/run-test-wire and /tmp/after/test/run-test-wire differ
  Binary files /tmp/before/test/run-test-wire.o and /tmp/after/test/run-test-wire.o differ
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell dc8458d1e2 tools: fix comment in template.
It's (usually) a .csv not an _csv file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
Rusty Russell 197d1bcef2 wire: move towire/fromwire_short_channel_id out to bitcoin/short_channel_id.c
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-18 14:51:12 +02:00
Rusty Russell b0c9059602 tools/generate-wire: no more lonely messages!
When we have only a single member in a TLV (e.g. an optional u64),
wrapping it in a struct is awkward.  This changes it to directly
access those fields.

This is not only more elegant (60 fewer lines), it would also be
more cache friendly.  That's right: cache hot singles!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-06 14:56:09 -05:00
Christian Decker 5325ff6352 json-rpc: Don't let users send messages that are handled internally
We cannot let users use `sendcustommsg` to inject messages that are handled
internally since it could result in our internal state tracking being borked.
2020-01-28 23:50:52 +01:00
Rusty Russell eb6a768741 tools: don't use bytelength as array length for arrays in TLVs.
This matters now we have an array in tlv_init_tlvs!  We were overallocating
in fromwire by 32x!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:40 +01:00
Christian Decker 626675c83c tlv: Migrate tlv serialization to typesafe function 2019-12-03 00:37:15 +00:00
Christian Decker e12b5c3824 tlv: Add a typesafe serialization function for tlv namespaces
This is the counterpart to the typesafe deserialization function implemented
in an earlier commit.
2019-12-03 00:37:15 +00:00
Christian Decker 5a78671d9f wire: Remove unused fromwire_tlvs
We are now using the typesafe variant everywhere.
2019-12-03 00:37:15 +00:00
Christian Decker 69c17d2d31 wire: Let the TLV _is_valid function actually return validity
I got this one wrong myself, since the function name implied a boolean
result. So I changed it to take the optional err_index as argument.
2019-12-03 00:37:15 +00:00
darosior 670f92002d tools/gen/impl_template: correct one-line for-loops indentation 2019-11-26 21:30:25 +01:00
darosior 5fb8e0aade tools/gen/impl_template: correct tlvs fromwire's for-loop 2019-11-26 21:30:25 +01:00
Christian Decker 2519f934aa tlv: Add validity check codegen for the tlv namespaces
Since the parser itself just parses and doesn't include validation anymore we
need to put that functionality somewhere. The validation consists of enforcing
that the types are in monotonically increasing order without duplicates and
that for the even types we know how to handle it.
2019-11-22 04:40:25 +00:00
Christian Decker 5794c83b4d tlv: Add typesafe fromwire codegen for TLV namespaces
We were weaving in and out of generic code through `fromwire_tlvs` with custom
parameters defining the types in that namespace. This hard-wires the parser
with the namespace's types. Slowly trying to deprecate `fromwire_tlvs` in
favor of this typesafe variant.
2019-11-22 04:40:25 +00:00
Christian Decker 2255024ead tlv: Add raw fields so we can store unknown fields as well 2019-11-22 04:40:25 +00:00
lisa neigut 3f1f075421 tools: add ability to wrap wire messages with ifs
Makes it possible to hide wire messages behind EXPERIMENTAL_FEATURES
flag.
2019-10-10 05:57:45 +00:00
lisa neigut e99720344e wire-gen: ensure that tlv messages are correctly ordered
Our TLV serializer relies on TLV outputs to be ordered by type
number. Prior to this commit we relied on 1) the ordering in the
RFC to be correct and 2) users to be using a version of Python that
respects stable ordering of dicts (i.e. Python 3.7+)

Instead of relying on these implicitly, we now explicitly sort messages
by type number when the TLV sets.

Resolves #2956.

Thanks-To: @ScottTre for the sort function
Reported-By: @ZmnSCPxj
2019-08-21 03:53:54 +00:00
Rusty Russell b10e0e08bb tools/generate-wire.py: add option to expose tlv_record_type
Next update adds TLV test vectors: without this, we get a warning
about them being unused.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-02 17:32:48 +02:00
Rusty Russell 3f8600e9c0 tools/generate-wire.py: handle implicit tlv length fields.
TLVs have an implicit `len` field, so allow expressions containing
that (eg. `len-1`), but assume it means "the remainder of the
message".

This means in most places, f.size() needs an fallback for the
implicit-length case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
lisa neigut 42251cc5b9 bolt-gen: fixup subtype nested varsize definition
variable length varsized subtypes were being allocated as
structs at not pointers; this fixes that.
2019-07-27 05:18:27 +00:00
lisa neigut 068496298c bolt-gen: rm unused bolt-generator; rename new bolt generator
delete now unused wire-generator and replace it with the newer
version.
2019-07-24 06:31:46 +00:00
lisa neigut 236d26308f bolt-gen: make optional 'assignable' fields work
make assignable 'optional' fields work, add test for them
2019-07-24 06:31:46 +00:00
lisa neigut 009e9dc945 bolt-gen: neaten up generated comments
remove extra space from the lead-in for inline comments

so that a provided comment like this

    # This is a comment

will appear like this

    /* This is a comment */
2019-07-24 06:31:46 +00:00
lisa neigut 281b4c241e bolt-gen: fixup the devtool/decodemsg printing facility
Fixup TLV handling in the bolt printing utility, `devtools/decodemsg`
2019-07-24 02:52:53 +00:00
lisa neigut 4261e508a9 bolt-gen: add TLV support
Add in support for buiding TLV's (minus the printing capability)
2019-07-24 02:52:53 +00:00
lisa neigut 12125ec8dc bolt-gen: small cleanups
we were incorrectly printing the references for a subtype case, plus
some errant styling,readability changes
2019-07-24 02:52:53 +00:00
lisa neigut f2819ba7d8 bolt-gen: remove 'is-optional' qualifier from msg field
we now handle optional fields, so we should include them in
the message parsing signatures
2019-07-24 02:52:53 +00:00
lisa neigut 96bf7aead5 bolt-gen: handle variable-sized and optionals
actually do the right thing for variable-sized and optional field
types
2019-07-24 02:52:53 +00:00
lisa neigut ade594e941 bolt-gen: add 'top-line' file comments to output
if there are any comments that aren't "attached" to a message,
print them at the top of the generated file. we need this for
the fancy auto-gen'd dependencies in the tool-wiregen tests.
2019-07-24 02:52:53 +00:00
lisa neigut fe3f4f52a0 bolt-gen: handle csv inline comments
The bolts don't have in-line comments, but the internal wire
message CSVs do. This adds the 'comments' back in to the
generated docs.
2019-07-24 02:52:53 +00:00
lisa neigut 6c240ab589 bolts: new parsing script and templates for new bolt format
the RFC's extract-format.py is switching to a new format.
this script can correctly parse them.

mostly moves logic over from generate-wire.py, uses a
Python formatting libarary called mako, which needs to be
installed prior to running this script.

you can add it to your system with

    sudo apt-get install python3-mako
2019-07-16 06:10:58 +00:00