Commit Graph

102 Commits

Author SHA1 Message Date
lisa neigut 86a099a62f wire-gen: initial start on subtypes
first pass at adding subtype structs
2019-04-10 23:42:25 +00:00
lisa neigut e4658c241e tlv: break out TLVs into new subclass
make TLV messages their own subclass of Message. this makes
other clean ups easier
2019-04-10 23:42:25 +00:00
Rusty Russell 0b484b111e gossipd: make more compact getchannels entries.
We can save significant space by combining both sides: so much that we
can reduce the WIRE_LEN_LIMIT to something sane again.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:34467-36764(35517.8+/-7.7e+02)
	vsz_kb:2637488
	store_rewrite_sec:35.310000-36.580000(35.816+/-0.44)
	listnodes_sec:1.140000-2.780000(1.596+/-0.6)
	listchannels_sec:55.390000-58.110000(56.998+/-0.99)
	routing_sec:30.330000-30.920000(30.642+/-0.19)
	peer_write_all_sec:50.640000-53.360000(51.822+/-0.91)

MCP notable changes from previous patch (>1 stddev):
	-store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
	+store_rewrite_sec:35.310000-36.580000(35.816+/-0.44)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell 91849dddc4 wire: use struct node_id for node ids.
Don't turn them to/from pubkeys implicitly.  This means nodeids in the store
don't get converted, but bitcoin keys still do.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:33934-35251(34531.4+/-5e+02)
	vsz_kb:2637488
	store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
	listnodes_sec:1.020000-1.290000(1.146+/-0.086)
	listchannels_sec:51.110000-58.240000(54.826+/-2.5)
	routing_sec:30.000000-33.320000(30.726+/-1.3)
	peer_write_all_sec:50.370000-52.970000(51.646+/-1.1)

MCP notable changes from previous patch (>1 stddev):
	-store_load_msec:46184-47474(46673.4+/-4.5e+02)
	+store_load_msec:33934-35251(34531.4+/-5e+02)
	-vsz_kb:2638880
	+vsz_kb:2637488
	-store_rewrite_sec:46.750000-48.280000(47.512+/-0.51)
	+store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
lisa neigut ed1223492b tlvs: add methods for decodemsg utility
fixup printing methods in devtools/decodemsg such that TLV's can
now be printed as well. here's how you'd use it:

   $ ./devtools/decodemsg --tlv opening_tlv 0120001E020202020202020202020202020202020202020202020202020202020202
   > WIRE_OPTION_UPFRONT_SHUTDOWN_SCRIPT (size 32):
   > shutdown_scriptpubkey=[020202020202020202020202020202020202020202020202020202020202]
2019-04-08 00:37:29 +00:00
lisa neigut bad0ac6ed6 tlv: use `var_int`s for size of messages
TLV's use var_int's for messages sizes, both internally and
in the top level (you should really stack a var_int inside a var_int!!)

this updates our automagick generator code to understand 'var_ints'
2019-04-03 03:15:42 +00:00
lisa neigut 1213f44071 tlv: adapt to work with new output format
Updated to match what the CSV generator in the RFC repo actually
outputs, see https://github.com/lightningnetwork/lightning-rfc/pull/597
2019-04-03 03:15:42 +00:00
lisa neigut df9774c2be tlv: fixup FIXME -- remove comments + use includes
include includes for TLV _csv files
2019-04-03 03:15:42 +00:00
lisa neigut 5d8b059ccc tlv: free intermediate messages when they're created
otherwise they'll get cleaned up when the message is free'd.
it's nbd either way, but this seems tighter.
2019-04-03 03:15:42 +00:00
lisa neigut 4afcc2e5a8 tlv: fixup parsing for multi-message tlv's
need to pass in a pointer to the array so that when we advance
the array in the subcalls, it advances in the parent. oops
2019-04-03 03:15:42 +00:00
lisa neigut d0f50c8690 tlv: fail if parsed length doesn't match packet length 2019-04-03 03:15:42 +00:00
lisa neigut 28c3f9ca21 tlv: don't crash when you fail to parse the TLV
passing back a null TLV was crashing here, because we tried to
dereference a null pointer. instead, we put it into a temporary
struct that we can check for NULL-ness, before assigning to the
passed in pointer.
2019-04-03 03:15:42 +00:00
lisa neigut d8738452ed tlv: it's ok to be odd
fail if a message type is even and it's not included. otherwise,
continue with the next message type.
2019-04-03 03:15:42 +00:00
lisa neigut b89ea071e8 tlv: allocate tlv structs from within
let's let the fromwire__tlv methods allocate the tlv-objects and
return them. we also want to initialize all of their underlying
messages to NULL, and fail if we discover a duplicate mesage type.

if parsing fails, instead of returning a struct we return NULL.

Suggested-By: @rustyrussell
2019-04-03 03:15:42 +00:00
lisa neigut 5aea65b463 tlv: make message sizes u8 not u16
Suggested-By: @rustyrussell
2019-04-03 03:15:42 +00:00
lisa neigut d6332997bd tlv: remove leading '_' from things
Suggested-By: @rustyrussell
2019-04-03 03:15:42 +00:00
lisa neigut aba4e161ce tlv: calculate sizeof by measuring message length
much better than statically calculating the sizeof
2019-04-03 03:15:42 +00:00
lisa neigut 9a23a354fd tlv: consolidate basetype parsing
clean up basetype parsing code a bit
2019-04-03 03:15:42 +00:00
lisa neigut 6f2e70a6ac tlv: add fromwire_ methods for TLV structs 2019-04-03 03:15:42 +00:00
lisa neigut ef610dcab3 tlv: build tlv top-level structs
construct structs for the TLV's. these will be the 'return type'
for tlv fields in parent messages (so to speak)
2019-04-03 03:15:42 +00:00
lisa neigut 18a23b31de tlv: add structs for message types to wire format .h's (header files)
Since messages in TLV's are optional, the ideal way to deal with
them is to have a 'master struct' object for every defined tlv, where
the presence or lack of a field can be determined via the presence
(or lack thereof) of a struct for each of the optional message
types.

In order to do this, appropriately, we need a struct for every
TLV message. The next commit will make use of these.

Note that right now TLV message structs aren't namespaced to the
TLV they belong to, so there's the potential for collision. This
should be fixed when/where it occurs (should fail to compile).
2019-04-03 03:15:42 +00:00
lisa neigut 0b12d90c4a tlv: add tlv messages to general message set
Add tlv-messages to the general messages set so that their parsing
messages get printed out.

FIXME: figure out how to account for partial message length processing?
2019-04-03 03:15:42 +00:00
lisa neigut 44d052e6c3 tlv: parse fields for tlv messages 2019-04-03 03:15:42 +00:00
lisa neigut d9904c95ab tlv: add tlv fields to enum declarations in implementation file
'.c' wire format files include case statements to print the names
of enums. Include such methods for the enums pertaining to
tlv's as well.
2019-04-03 03:15:42 +00:00
lisa neigut d51ad50032 tlv: add tlv field's type enums to header file
Version 1.1 of the lightning-rfc spec introduces TLVs for optional
data fields. This starts the process of updating our auto-gen'd
wireformat parsers to be able to understand TLV fields.

The general way to declare a new TLV field is to add a '+' to the
end of the fieldname. All field type declarations for that TLV set
should be added to a file in the same directory by the name
`gen_<field_name>_csv`.

Note that the FIXME included in this commit is difficult to fix, as
we currently pass in the csv files via stdin (so there's no easy
way to ascertain the originating directory of file)
2019-04-03 03:15:42 +00:00
Christian Decker 3ce98ab7de wire: Allow non-u16 length variables in non-bolt wire formats
Otherwise we can't really return a variable sized message with more than 65k
results. This was causing an integer overflow in `listchannels` (see #2504 for
details).

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-03-27 12:48:52 +01:00
Rusty Russell 28f5da7b2f tools/generate-wire: use amount_msat / amount_sat for peer protocol.
Basically we tell it that every field ending in '_msat' is a struct
amount_msat, and 'satoshis' is an amount_sat.  The exceptions are
channel_update's fee_base_msat which is a u32, and
final_incorrect_htlc_amount's incoming_htlc_amt which is also a
'struct amount_msat'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell 7fad7bccba common/amount: new types struct amount_msat and struct amount_sat.
They're generally used pass-by-copy (unusual for C structs, but
convenient they're basically u64) and all possibly problematic
operations return WARN_UNUSED_RESULT bool to make you handle the
over/underflow cases.

The new #include in json.h means we bolt11.c sees the amount.h definition
of MSAT_PER_BTC, so delete its local version.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 00:44:57 +00:00
Rusty Russell 1567238dd9 invoice: option to expose/not-expose private channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 22526911cb tools/generate-wire.py: allocate array members off array in fromwire.
If we have an array of varlen structures (which require a ctx arg), we
should make that arg the array itself (which was tal_arr()), not the
root context.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-22 05:15:42 +00:00
lisa neigut 3ce53ab9ed tools/gen-wire: 3th -> 3rd
Small grammar fix in wire gen tool
2018-09-22 19:06:06 +02:00
Rusty Russell 61c6b8b25a tools/generate-wire.py: mark your_last_per_commitment_secret as a struct secret.
Without an override, the tool assumes it's a sha256.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-23 14:46:22 +02:00
practicalswift 9d9a9523d0 Use snprintf(...) instead of sprintf(...) 2018-08-02 16:14:21 +09:30
Rusty Russell 5cf34d6618 Remove tal_len, use tal_count() or tal_bytelen().
tal_count() is used where there's a type, even if it's char or u8, and
tal_bytelen() is going to replace tal_len() for clarity: it's only needed
where a pointer is void.

We shim tal_bytelen() for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-30 11:31:17 +02:00
Rusty Russell 93cf28553d devtools/decodemsg: add --onion option for decoding onion errors.
This requires a tweak to generate-wire.py too, since it always called the
top-level routine 'print_message'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-27 14:12:00 +02:00
Rusty Russell d241bd762c connectd: don't use gossip_getnodes_entry.
gossip_getnodes_entry was used by gossipd for reporting nodes, and for
reporting peers.  But the local_features field is only available for peers,
and most other fields are only available from node_announcement.

Note that the connectd change actually means we get less information
about peers: gossipd used to do the node lookup for peers and include the
node_announcement information if it had it.

Since generate_wire.py can't create arrays-of-arrays, we add a 'struct
peer_features' to encapsulate the two feature arrays for each peer, and
for convenience we add it to lightningd/gossip_msg.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
Rusty Russell 232b330284 tools/generate-wire.py: simple scheme to autoindent.
I verified that the generated files don't change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell 6758062a56 tools/generate-wire.py: remove ugly blank lines after for loops
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell ab28972aee tools/generate-wire.py: style cleanups.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell e217bc1220 per-commit-secret is a struct secret, not a sha256.
Well, it's generated by shachain, so technically it is a sha256, but
that's an internal detail.  It's a secret.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell ac4c6b1a82 tools/generate-wire.py: support for optional fields, with ? before typename.
We already work around this by using an array with a 0/1 length convention,
but better to be explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell c02ff11506 print_wire: hand field names to print routines.
This lets us override how we print them.

Also, add dependency on header for devtools/Makefile.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-01 14:55:29 +02:00
arowser 85e02127a9 align generate wire template 2018-06-20 06:56:42 +00:00
practicalswift 160f9b06a5 Fix variable typo in generate-wire.py 2018-02-24 11:44:32 +01:00
practicalswift 0a20bd44f0 Fix flake8 warnings: W191 indentation contains tabs + E101 indentation contains mixed spaces and tabs 2018-02-22 22:21:31 +01:00
practicalswift e18948a6a9 Remove unused variables. Fix flake8 warnings. 2018-02-22 22:21:31 +01:00
practicalswift ae7d857c44 Make Python code conform to the Python PEP 8 style guide 2018-02-22 10:47:19 +01:00
Rusty Russell e92b710406 tools/generate-wire.py: remove length argument from fromwire_ routines.
We always hand in "NULL" (which means use tal_len on the msg), except
for two places which do that manually for no good reason.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-20 22:36:21 +01:00
Rusty Russell b7db06b577 tools/generate-wire.py: wirestring type for handing strings.
A convenient alias for char *, though we don't allow control characters
so our logs can't be fooled with embedded \n.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 674a2c7554 tools/generate_wire.py: make bitcoin_tx a varlen structs
Now it allocates naturally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00