Commit Graph

232 Commits

Author SHA1 Message Date
trueptolemy cdcafdaf74 API: `txprepare` now support mutiple outputs 2019-09-05 16:05:36 -05:00
Rusty Russell 11833192a1 tools: fix Makefile warning
Makefile:424: warning: overriding recipe for target 'clean'
    tools/Makefile:12: warning: ignoring old recipe for target 'clean'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
darosior 77f34fad2b doc/Makefile: adapt 'check-manpages' to markdown 2019-08-22 01:35:01 +00:00
Rusty Russell 9dfc854d94 tools/Makefile: put all tools/ stuff here.
Generally a Makefile should control make within that directory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell 278b12982d tools/test/Makefile: disable unless we're in DEVELOPER mode.
Because it required update-mocks, which is a hack which relies on the
format of linker errors (!) I'd prefer to make this --enable-developer
only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell c74c1e56cc tools/test/Makefile: fix always-on generation of tools/test/gen_test.c
We need to tell it that the tmp file is an intermediate, so doesn't need
remaking if it doesn't exist.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
lisa neigut e99720344e wire-gen: ensure that tlv messages are correctly ordered
Our TLV serializer relies on TLV outputs to be ordered by type
number. Prior to this commit we relied on 1) the ordering in the
RFC to be correct and 2) users to be using a version of Python that
respects stable ordering of dicts (i.e. Python 3.7+)

Instead of relying on these implicitly, we now explicitly sort messages
by type number when the TLV sets.

Resolves #2956.

Thanks-To: @ScottTre for the sort function
Reported-By: @ZmnSCPxj
2019-08-21 03:53:54 +00:00
Rusty Russell 04a57ae4af doc/MAKING-RELEASES.md: update.
1. These days we delete the [Unreleased] tag during rcs.
2. Make sure we test the release build process during rc1, since I
   screwed that up last release.
3. Add a section on rc2, etc.
4. Do final release via a github PR, since I screwed that up on the
   prior release.
5. Update `tools/build-release.sh` and instructions to show that we now
   make a reproducible build for Ubuntu 18.04 x86-64.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell 303c221641 tools/test/Makefile: we depend on tools/test/gen_test.h
This is because update-mocks actually tries to build it to get link
errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-19 03:43:13 +00:00
Rusty Russell c0ba5879df tools/test: can't generate mocks until submodules checked out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 14:08:43 +09:30
Rusty Russell dbb15b0045 Makefile: fix occasional race where we fail to compile tools/tests.
This simplifies the dependencies:
1. Objs depend on headers, not other objs.
2. Programs depend on objs.
3. A .o file will generally implicitly depend on the .c file it's built from.
4. If a file has a build line, it's often better to list all deps there.
5. I spotted some missing 'make clean' files.

The particular problem in this case seems to be that make would use
tools/test/gen_test.c before it was ready.  It's probably confused by
the use of recursive make via update-mocks, so explicitly split that
into two stages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-17 04:20:05 +00:00
Rusty Russell 4b5cd5ff84 tools/generate-wire.py: only use 'struct node_id' in announcements.
Naturally, it's a struct pubkey.  However, those are large, and take
time to marshal, so gossipd treats them as node_id which is a simple
array.  It adds explicit checks at the right points to make sure
they're valid pubkeys.

However, the next commit adds TLV test vectors, which assumes we treat
node_id as a point (thus catch invalid values when parsing).  The best
solution is to restrain our types here to exactly those we've
optimized for.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-02 17:32:48 +02:00
Rusty Russell b10e0e08bb tools/generate-wire.py: add option to expose tlv_record_type
Next update adds TLV test vectors: without this, we get a warning
about them being unused.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-02 17:32:48 +02:00
Rusty Russell 4f937ea1f0 fixup! FreeBSD: fix build. 2019-07-29 14:24:29 +02:00
Rusty Russell 9e5b9a31c9 FreeBSD: fix build.
This makes it build for me on FreeBSD 11:
1. $(MAKE) has to passed through into update-mocks.
2. FreeBSD sed doesn't turn \n into a newline on RHS.
3. Bash and mako dependencies were missing from INSTALL.md

Fixes: #2850
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-29 14:24:29 +02:00
Rusty Russell 3c3a89c483 tools/generate-wire.py: add test for implicit lengths.
Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
Rusty Russell 3f8600e9c0 tools/generate-wire.py: handle implicit tlv length fields.
TLVs have an implicit `len` field, so allow expressions containing
that (eg. `len-1`), but assume it means "the remainder of the
message".

This means in most places, f.size() needs an fallback for the
implicit-length case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
Rusty Russell c580225c38 wire: handle bigsize/varint fields.
They're currently called varint, but there's a proposal to call them all
bigsize.  Allow both for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
Rusty Russell 2e68b88be8 tools: fix tlv generation
We need to hand -s to both header and body generation, or neither:

wire/gen_peer_wire.c:53:13: error: static declaration of ‘towire_channel_update_timestamps’ follows non-static declaration
In file included from wire/gen_peer_wire.c:5:
./wire/gen_peer_wire.h:78:6: note: previous declaration of ‘towire_channel_update_timestamps’ was here

We also need it for printwire, otherwise we get static unused functions for subtypes:

devtools/gen_print_wire.c:155:13: error: ‘printwire_channel_update_checksums’ defined but not used [-Werror=unused-function]
 static void printwire_channel_update_checksums(const char *fieldname, const u8 **cursor, size_t *plen)
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
devtools/gen_print_wire.c:133:13: error: ‘printwire_channel_update_timestamps’ defined but not used [-Werror=unused-function]
 static void printwire_channel_update_timestamps(const char *fieldname, const u8 **cursor, size_t *plen)
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-27 21:19:57 -05:00
lisa neigut 42251cc5b9 bolt-gen: fixup subtype nested varsize definition
variable length varsized subtypes were being allocated as
structs at not pointers; this fixes that.
2019-07-27 05:18:27 +00:00
Rusty Russell b1738c5b89 tools: fix 32 bit compile error
```
tools/test/enum.c: In function ‘fromwire_test_enum’:
tools/test/enum.c:11:34: error: format ‘%ld’ expects argument of type ‘long int’, but argument 2 has type ‘size_t {aka unsigned int}’ [-Werror=format=]
  printf("fromwire_test_enum at %ld\n", *max);
```

and:

```
devtools/print_wire.c: In function ‘printwire_tlvs’:
devtools/print_wire.c:201:22: error: format ‘%ld’ expects argument of type ‘long int’, but argument 2 has type ‘u64 {aka long long unsigned int}’ [-Werror=format=]
    printf("**TYPE #%ld UNKNOWN for TLV %s**\n", type, fieldname);
                      ^
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-25 11:41:48 +08:00
Rusty Russell c5ae29d2b1 check: only check EXPERIMENTAL-tagged BOLT quotes if EXPERIMENTAL_FEATURES
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-24 06:35:57 +00:00
lisa neigut 46cbca97ea bolt-gen: use 'enum' instead of 'e:' prefix for enums
use 'enum ' in wirespec instead of 'e:' as an indicator
that a field is an enum.
2019-07-24 06:31:46 +00:00
lisa neigut b6d4c372bb bolt-gen: fix broken 'exposed subtype' generation
borked in 96bf7aea; fixed here.
2019-07-24 06:31:46 +00:00
lisa neigut 068496298c bolt-gen: rm unused bolt-generator; rename new bolt generator
delete now unused wire-generator and replace it with the newer
version.
2019-07-24 06:31:46 +00:00
lisa neigut 32eaae0cb9 wire-gen: move in-house wire delcarations to new format
tidying things up!
2019-07-24 06:31:46 +00:00
lisa neigut 236d26308f bolt-gen: make optional 'assignable' fields work
make assignable 'optional' fields work, add test for them
2019-07-24 06:31:46 +00:00
lisa neigut ff82c918a5 bolt-gen: formatting cleanups
add brackets for clarity when printing subtypes; also remove
unnecessary exception (`find_` will raise an exception if
the key is not present)
2019-07-24 06:31:46 +00:00
lisa neigut 316edb39a4 bolt-gen: for wire messages, print out optional fields (if present)
optional fields should be printed, if they exist. so let's print them!
2019-07-24 06:31:46 +00:00
lisa neigut 40154b35f0 bolt-gen: make needs-context not recursive
originally done so that any field within a subtype needed context,
that would be reflected at the top-most layer. in reality, we
allocate subtype fields off of the instance of the subtype, so
there's no need to check beneath the 'top' layer of field/types
in a message to determine whether or not to pass in a context.
2019-07-24 06:31:46 +00:00
lisa neigut b30d7d26ea bolt-gen: handle enums
Add parsing know-how for enum fields. This is necessary for
internally defined wire generators. Enums are denoted by prefixing
the field with an `e:`.

Ex:

   msgdata,msg_name,field_name,e:enum_type,
2019-07-24 06:31:46 +00:00
lisa neigut cfd56d86ee bolt-gen: keep single instance of type_obj per type
we rely, perhaps a bit hackily, on there only being one copy of
each type object floating about. using `deepcopy` on a Message
for message extensions destroys this paradigm, which breaks
things in the case where it's a later defined subtype that contains
variable-length members.

to fix this, we modify `__deepcopy__` on the Field class, such
that it preserves the reference to the original type_obj instance.
2019-07-24 06:31:46 +00:00
lisa neigut 009e9dc945 bolt-gen: neaten up generated comments
remove extra space from the lead-in for inline comments

so that a provided comment like this

    # This is a comment

will appear like this

    /* This is a comment */
2019-07-24 06:31:46 +00:00
lisa neigut 7f5ba35e95 bolt-gen-tests: namespace test tlv's under test_
for posterity.
2019-07-24 02:52:53 +00:00
lisa neigut 281b4c241e bolt-gen: fixup the devtool/decodemsg printing facility
Fixup TLV handling in the bolt printing utility, `devtools/decodemsg`
2019-07-24 02:52:53 +00:00
lisa neigut cedebfd2d9 bolt-gen: add truncated uint support for TLV fields
TLV's have truncated values, add support for them. plus
some 'compilation' tests.
2019-07-24 02:52:53 +00:00
lisa neigut 4261e508a9 bolt-gen: add TLV support
Add in support for buiding TLV's (minus the printing capability)
2019-07-24 02:52:53 +00:00
lisa neigut 79f13fa429 bolt-gen: add compilation tests
Add a test for checking that the bolt-gens do the right thing
for a fairly exhaustive test case set (and that it compiles).

Note that this doesn't check that we've got the memory assignment
pieces worked out.

It's got a kind of exotic reliance on the update-mocks in that in
order to depend on as little of the wire/ code as possible (we
only import wire/wire.h), we include an AUTOGENERATE comment
in the test_cases CSV file, and then run update-mocks as part of
the build for that file.
2019-07-24 02:52:53 +00:00
lisa neigut 12125ec8dc bolt-gen: small cleanups
we were incorrectly printing the references for a subtype case, plus
some errant styling,readability changes
2019-07-24 02:52:53 +00:00
lisa neigut f2819ba7d8 bolt-gen: remove 'is-optional' qualifier from msg field
we now handle optional fields, so we should include them in
the message parsing signatures
2019-07-24 02:52:53 +00:00
lisa neigut 96bf7aead5 bolt-gen: handle variable-sized and optionals
actually do the right thing for variable-sized and optional field
types
2019-07-24 02:52:53 +00:00
lisa neigut 181e1916b2 bolt-gen: add field optional handling
we'll need this for internal wire message formats. also disambiguates
from 'bolt message optional fields', which we rename to extensions here.

example of an optional field declaration (note the ? prefixing the
type):

    msgdata,msg_name,field_name,?type,count

these are handled with either a boolean if they're not present,
or a true value and then the object if they are.
2019-07-24 02:52:53 +00:00
lisa neigut ade594e941 bolt-gen: add 'top-line' file comments to output
if there are any comments that aren't "attached" to a message,
print them at the top of the generated file. we need this for
the fancy auto-gen'd dependencies in the tool-wiregen tests.
2019-07-24 02:52:53 +00:00
lisa neigut fe3f4f52a0 bolt-gen: handle csv inline comments
The bolts don't have in-line comments, but the internal wire
message CSVs do. This adds the 'comments' back in to the
generated docs.
2019-07-24 02:52:53 +00:00
lisa neigut d7a68b75f1 bolt-gen: fixup broken subtype parsing 2019-07-24 02:52:53 +00:00
lisa neigut 1044e37fee bolt-gen: use ordered-dict for fields
Use ordered dict for fields, who's order matters when iterated
through (especially for argument lists).

Reported-By: @mocacinno
2019-07-24 00:43:47 +00:00
Rusty Russell 54790c17ea wire: rename var_int to bigsize, and insist on minimal.
The new TLV spec uses BigSize, like Bitcoin's CompactInt but
*little-endian*.  So change our name for clarity, and insist that
decoding be minimal as the spec requires.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-18 06:51:35 +00:00
lisa neigut 85e325cb1f bolt: update to lightning-rfc:6f6ea63233c new fundamental types
RFC tweaks to the types used to specify message fields
2019-07-16 06:10:58 +00:00
lisa neigut 6c240ab589 bolts: new parsing script and templates for new bolt format
the RFC's extract-format.py is switching to a new format.
this script can correctly parse them.

mostly moves logic over from generate-wire.py, uses a
Python formatting libarary called mako, which needs to be
installed prior to running this script.

you can add it to your system with

    sudo apt-get install python3-mako
2019-07-16 06:10:58 +00:00
lisa neigut 118caae45b tools: simplify check
in this case, we always need a pointer, so remove the check
and just add it to the template
2019-07-16 06:10:58 +00:00