Commit Graph

237 Commits

Author SHA1 Message Date
Rusty Russell ab28972aee tools/generate-wire.py: style cleanups.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell e217bc1220 per-commit-secret is a struct secret, not a sha256.
Well, it's generated by shachain, so technically it is a sha256, but
that's an internal detail.  It's a secret.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell ac4c6b1a82 tools/generate-wire.py: support for optional fields, with ? before typename.
We already work around this by using an array with a 0/1 length convention,
but better to be explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Jan Sarenik 9f519afc5d tools/check-includes.sh: shellcheck recommended fixes
$ shellcheck --version
    ShellCheck - shell script analysis tool
    version: 0.5.0
    license: GNU General Public License, version 3
    website: https://www.shellcheck.net
    $ make check-source
    ...

    In tools/check-includes.sh line 14:
        if [[ $(grep -cE "^#((ifndef|define) ${HEADER_ID}|endif /\* ${HEADER_ID} \*/)$" "${HEADER_FILE}") != 3 ]]; then
                                                                 ^-- SC1117: Backslash is literal in "\*". Prefer explicit escaping: "\\*".

    In tools/check-includes.sh line 28:
        git ls-files | grep -v 'ccan/' | grep -E "\.${1}"'$'
                                                  ^-- SC1117: Backslash is literal in "\.". Prefer explicit escaping: "\\.".
2018-07-04 01:48:21 +00:00
Rusty Russell c02ff11506 print_wire: hand field names to print routines.
This lets us override how we print them.

Also, add dependency on header for devtools/Makefile.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-01 14:55:29 +02:00
nicolas.dorier aeafe4dbe7 [Docker] Fix: Bind socat to the right port 2018-06-27 13:40:02 +02:00
Christian Decker 0d3c3a225e docker: Add a usable dockerfile
This is based on @NicolasDorier's excellent proposal for a Dockerfile, sans the
writing of a config file.

Co-authored-by: Nicolas Dorier <nicolas.dorier@gmail.com>
Co-authored-by: Christian Decker <decker.christian@gmail.com>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-21 23:51:57 +00:00
Rusty Russell f64272caea tools/check-manpage.sh: make it work on FreeBSD.
Extended regular expressions are more compatible, it seems (and simpler!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-21 14:00:23 +02:00
arowser 85e02127a9 align generate wire template 2018-06-20 06:56:42 +00:00
Rusty Russell 76dd1577bd external: fix submodule handling for parallel builds, submodule URL changes.
If we change an upstream URL, all submodules break.  Users would need
to run 'git submodule sync'.  Note that the libbacktrace fix was merged
upstream so this is no longer necessary, but it's good for future changes.

Also, stress-testing reveals that git submodule fails locking
'.git/config' when run in paralell.  It also segfaults and other
problems.

This is my final attempt to fix submodules; I've wasted far too many
days on obscure problems it creates: I've already lost one copy of my
repo to apparently unfixable submodule preoblems.  The next "fix" will
be to simply import the source code so it works properly.

Reported-by: @jsarenik
Fixes: #1543
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 626e9fed16 Makefile: add check that manpage and command options match.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
practicalswift abf510740d Force the use of the POSIX C locale for all commands and their subprocesses 2018-04-27 14:02:59 +02:00
Rusty Russell 5b7fcab766 tools: fix shellcheck errors.
Not sure how this got through Travis.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-06 14:26:53 +02:00
practicalswift 1a55147870 Be consistent in choice of shell across all shell scripts in repo 2018-04-04 15:06:30 +02:00
practicalswift b95d3b8f54 Fix shellcheck warnings 2018-04-04 15:06:30 +02:00
practicalswift e078fcefdd Add check-includes.sh: Find redundant includes. Check ifndef:s. 2018-03-25 23:54:21 +00:00
ZmnSCPxj e6de41b468 tools/rel.sh: Make POSIXLY compatible.
Fixes: #1221

We were using `\x0` to match NUL chars in the input (on the
assumption that NUL chars are "impossible" for decent LFS-compliant
systems).
However `\x0` is a GNUism.
Use the `\n` and the newline character, which is supported by (most)
POSIX sed.
2018-03-16 01:50:26 +00:00
practicalswift 160f9b06a5 Fix variable typo in generate-wire.py 2018-02-24 11:44:32 +01:00
practicalswift 0a20bd44f0 Fix flake8 warnings: W191 indentation contains tabs + E101 indentation contains mixed spaces and tabs 2018-02-22 22:21:31 +01:00
practicalswift e18948a6a9 Remove unused variables. Fix flake8 warnings. 2018-02-22 22:21:31 +01:00
practicalswift ae7d857c44 Make Python code conform to the Python PEP 8 style guide 2018-02-22 10:47:19 +01:00
Rusty Russell e92b710406 tools/generate-wire.py: remove length argument from fromwire_ routines.
We always hand in "NULL" (which means use tal_len on the msg), except
for two places which do that manually for no good reason.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-20 22:36:21 +01:00
Rusty Russell b7db06b577 tools/generate-wire.py: wirestring type for handing strings.
A convenient alias for char *, though we don't allow control characters
so our logs can't be fooled with embedded \n.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 674a2c7554 tools/generate_wire.py: make bitcoin_tx a varlen structs
Now it allocates naturally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 526d3a232e tools/generate_wire.py: generate varlen arrays properly.
These are now logically arrays of pointers.  This is much more natural,
and gets rid of the horrible utxo array converters.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell ad8dfaca1c tools/generate_wire.py: make varlen structs self-allocate.
If we tell it a struct is variable length, make fromwire() allocate
and return it off ctx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 32abb4d928 tools/generate-wire.py: --printwire option to create print routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 00:57:10 +00:00
Rusty Russell 0cef23d33a check-spelling: don't paginate, and give file and line.
The pagination causes it to wait for a keypress even with no output
under emacs (complaining about the terminal); we don't want it anyway.

Example output:

Makefile:228:#lighnting!
Identified a likely misspelling of the word "lightning" (see above). Please fix.
Is this warning incorrect? Please teach tools/check-spelling.sh about the exciting new word.
Makefile:230: recipe for target 'check-spelling' failed

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-01 14:40:31 +01:00
practicalswift 9f47c0431a Fix typos 2018-01-28 13:53:39 +01:00
practicalswift 3e40a48332 Add lightning spell check 2018-01-26 01:02:15 +00:00
practicalswift a08b6fffc6 Add consistency check for dependencies listed in README.md and doc/INSTALL.md 2018-01-19 10:11:19 +00:00
ZmnSCPxj fdbf2f41b4 generate-wire.py: Add scare comments to output.
Discourage developers from modifying files generated by generate-wire.py
2018-01-10 23:53:12 +00:00
ZmnSCPxj 1946f0c288 Makefile: Add BINTOPKGLIBEXECDIR macro. 2018-01-09 14:52:48 +01:00
Christian Decker 59128a86ee hsm: Make sure to pass close_info along with the UTXO
The close_info is needed to re-derive the secret key that is supposed
to be used to sign the input spending the output.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-01-08 17:19:25 +01:00
practicalswift 5123c4e059 Fix access to undefined variable 2017-12-28 16:05:15 +01:00
practicalswift aee3661b35 Fix typos (repeated words) 2017-12-28 16:04:38 +01:00
Rusty Russell 810abb6b21 bitcoin: create new wrapper type bitcoin_blkid, log backward endianness.
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_blkid_to_hex() so it's reversed as people expect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-21 11:05:38 +00:00
Rusty Russell 0237e0b28c bitcoin: create new wrapper type bitcoin_txid, log backward endianness.
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_txid_to_hex() so it's reversed as people expect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-21 11:05:38 +00:00
Rusty Russell 42b9df59c1 update-mocks: allow it to handle missing notleak_
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-20 12:43:10 +01:00
practicalswift 0353ec0983 Remove trailing whitespace 2017-12-11 03:35:59 +00:00
practicalswift 61c47c09d0 Fix typos 2017-12-08 13:07:20 +01:00
Rusty Russell 44e45348f2 option-data-loss-protect: fix generate-wire.py and update.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-16 15:26:41 +01:00
Rusty Russell 5320cab2de generate-wire.py: create structures for optional fields.
If a structure foo has a optional fields opt1 and opt2, this creates
towire_foo, towire_foo_opt1 and towire_foo_opt2 (since opt2 implies opt1),
similarly for fromwire_*.

This requires the callers to be updated to call the correct routines (eg.
try fromwire_foo_opt2, then fromwire_foo_opt1, then finally fromwire_foo),
but this is a minimal change to the generation code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-09 13:55:24 +01:00
Rusty Russell ee1fb07c8e update-mocks: fix stubs generation for fatal().
We're going to need this, and the PRINTF_FMT(1,2) in front of it caused
mockup.sh to miss the declaration.

We also eliminate the obviously-unused fallback case (which referred
to daemon/*.h).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 04:14:33 +00:00
William Casarin 52f741de97 tools/generate-wire.py: generalize python3 path
Generalize generate-wire.py python3 path

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-10 20:12:37 +02:00
Rusty Russell 32631b4278 generate-wire.py: add --bolt arg, use size->type hacks only when that's specified.
For our own internal comms CSVs, we should always name explicit types.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-29 14:40:34 +02:00
Rusty Russell 8a829ba9cb update-mocks: move mock generation into tools/, fix and generalize.
update-mocks was broken, since it assumed the daemon/ directory.

We now use "make" directly to build the test file and harvest errors,
and are more robust if it simply doesn't compile (ie. fails, but no
linker errors).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-03 02:01:54 +02:00
Rusty Russell b3514d3430 generate-wire.py: generate chain_hash fields as sha256_double.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-22 14:56:02 +02:00
Rusty Russell 26ffa8fdec check-bolt: move to tools/
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-22 14:56:02 +02:00
Rusty Russell aa23a2a93f generate-wire.py: handle unmarshalling of a single varlen_struct
We previously only handled it in arrays.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-20 13:06:41 +09:30
Rusty Russell 7884343c15 tools/generate-wire.py: enums are assignable.
Simplfies their marshalling/unmarshalling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell be6347b814 tools/generate-wire.py: unmarshall zero-length arrays as NULL.
This is symmetrical with marshalling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-27 10:25:53 +09:30
Rusty Russell da18645901 tools/generate-wire.py: don't avoice tal_count() on NULL.
It actually works fine with recent ccan/tal (returning 0).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-27 10:25:53 +09:30
Rusty Russell 9868e896d9 tools/generate-wire.py: handle marshal/unmarshal of enum types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell ad3ab53fa4 lightningd/htlc_wire: wire types for sending HTLCs to/from channel daemon.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell a2ffccd740 tools/generate-wire.py: fix generation of array of assignable types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-07 09:19:04 +09:30
Rusty Russell cb5ad91788 tools/generate-wire.py: allow consecutive fields to use same len var.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-07 09:19:04 +09:30
Rusty Russell 6e0e1c7067 Update to latest BOLT (hyphens changed to underscores).
Now in sync with 8ee57b97738b1e9467a1342ca8373d40f0c4aca5.

Our tool doesn't need to convert them any more, but we actually had a
mis-typed field in the HSM which needed fixing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-12 12:59:09 +02:00
Christian Decker 67315be673 travis: Simplified build script
The Dockerfile is now stored in contrib and built using the Docker
Hub. This allows us to simply pull in the finished image from the hub
instead of having to build it ourself. Should shave off about 2
minutes from the build time.

I also switched to running the individual build and check steps in
their own containers, but on the same volume, so travis can group the
commands and run them independently.
2017-04-29 10:29:44 +09:30
Rusty Russell 6f9a7f7aa1 Travis support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-04 15:27:34 +09:30
Rusty Russell 9b6753173c tools/generate-wire: allow constant multiplier in fields.
Not just len*u8, but 1254*u8 for explicit fixed-length arrays. 

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-01 23:59:46 +10:30
Rusty Russell d072f2ec06 wire: understand struct preimage, don't treat it as sha256.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-01 23:59:46 +10:30
Christian Decker b2ea4cfd66 wiregen: Passing ctx to array helpers that require it
Some of the struct array helpers need to allocate data when
deserializing their fields. The `getnodes` reply is one such example
that allocates the hostname. Since the change to calling array helpers
the getnodes call was broken because it was attempting to allocate off
of the entry, which did not have a tal header, thus failing.
2017-03-20 11:18:00 +10:30
Rusty Russell e042198cf8 tools/generate-wire.py: allow typename instead of type sizes.
We use the fourth value (size) to determine the type, unless the fifth
value is suppled.  That's silly: allow the fourth value to be a typename,
since that's the only reason we care about the size at all!

Unfortunately there are places in the spec where we use a raw fieldname
without '*1' for a length, so we have to distingish this from the
typename case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-03-16 14:35:25 +10:30
Rusty Russell 9768e4c907 tools/generate-wire.py: open-code array handling for non-u8.
Except for the trivial case of u8 arrays, have the generator create
the loop code for the array iteration.

This removes some trivial helpers, and avoids us having to write more.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-03-16 14:35:23 +10:30
Christian Decker 4a995a42de gen-wire: Cleanup of the wire generator tool
The wiregen tool was a bit hard to maintain since it was printing all
over the place, mixing template and processing logic. This commit
tears the two apart, externalizes everything that is not a single code
line, and repackages it into templates. Specifically functions are now
their own template and header/implementation files are a template.

Furthermore this simplifies some of the boilerplate of mapping types
to sizes and back again, by extracting them into dicts.

All changes have been verified to produce identical results on the
current wire definitions, except a bit of whitespace changes.
2017-03-06 10:22:02 +10:30
Rusty Russell 7419fde9a0 Update to new spec: differentiate channel_id and short_channel_id.
The spec 4af8e1841151f0c6e8151979d6c89d11839b2f65 uses a 32-byte 'channel-id'
field, not to be confused with the 8-byte short ID used by gossip.  Rename
appropriately, and update to the new handshake protocol.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-03-02 22:51:49 +10:30
Rusty Russell 9fd40da38c generate-wire: don't hand unknown structures specially.
It's awkward to handle them differently.  But this change means we
need to expose them to the generated code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-02-21 15:15:19 +10:30
Rusty Russell fd09345628 generate-wire: handle variable-size arrays of unknown types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-02-21 14:59:14 +10:30
Rusty Russell 1edce4878d bitcoin/signature.h: remove struct signature.
It's a wrapper around secp256k1_ecdsa_signature, so use that directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-25 11:03:55 +10:30
Rusty Russell 2b8c7cc840 tools/generate-wire.py: don't allocate on unknown names.
This introduces a potential leak; use a static buffer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-10 15:20:20 +10:30
Rusty Russell 42f474af18 wire: make lengths of variable fields implied by tal_count()
This is a much nicer interface, and works better in practice too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-10 15:19:25 +10:30
Rusty Russell b6a55a68ea tools/generate-wire.py: handle non-integer enum values.
BOLT 4 uses values like "PERM|1", so allow that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-06 13:40:06 +10:30
Rusty Russell 2057de6d4f tools/generate-wire.py: Apparently OptionParser is deprecated, use argparse.
It is a bit nicer though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-06 13:24:45 +10:30
Rusty Russell 8b72604d4d wire: allow #include directives in csv comments, and unknown structure types.
This lets us marshal and unmarshal undefined complex types; the fromwire
function will allocate it for us, so we don't even need to know the size.

This turns out to be really nice for marshalling 'struct crypto_state'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-06 13:24:45 +10:30
Rusty Russell f50af430e1 wire: add bool routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 937a62100f generate-wire.py: allow optional typename in csv file.
For our internal CSV files, we can specify the type explicitly rather
than trying to guess (eg. bool).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 8ad1298f88 generate-wire.py: add enum->name function.
Pretty!  Takes an int instead of the enum directly, because in the
main daemon we call it via a function pointer, so want them all the
same type.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 54cfbfeba9 generate-wire.py: generate enum in order provided in CSV
This means the comments make more sense.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell ba9e4f9377 generate-wire.py: allow hex values in csv files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell bdaa22e247 generate-wire.py: allow NULL len parameter from fromwire_*
It implies tal_count() gives the length. Great for almost all callers which
don't care if there are extra bytes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell e076d56709 generate-wire.py: include type bytes in towire/fromwire routines.
This removes some redundancy in creating messages, but also allows
a lazy form or parsing without explicitly checking the type.

A helper fromwire_peektype() is added to look up the type and handle
the too-short-for-type problem.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 3e044fdd62 generate-wire.py: allow comments.
We emit them into the generated code at appropriate points, but it
would be better if we simply preserved the order they were given in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell bf7fefdc32 generate-wire.py: add memcheck() to towire functions.
Sanity check that we're not streaming uninitialized bytes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30
Rusty Russell c864b28068 generate-wire.py: don't generate structures, hand in all values.
This is a bit more awkward for large structures, but avoids
indirection for the simpler ones (I copied the structures for the test
code, however).  We also remove explicit padding.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30
Rusty Russell 326a9c9477 tools/generate-wire: node_announcement.alias is not an sha256
It's 32 bytes long, but it's just a string.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30
Rusty Russell a08a2105ea generate-wire.py: generalize, move to tools.
We're going to want to use this for inter-daemon comms, so generalize it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30