Commit Graph

404 Commits

Author SHA1 Message Date
lisa neigut 316edb39a4 bolt-gen: for wire messages, print out optional fields (if present)
optional fields should be printed, if they exist. so let's print them!
2019-07-24 06:31:46 +00:00
lisa neigut 40154b35f0 bolt-gen: make needs-context not recursive
originally done so that any field within a subtype needed context,
that would be reflected at the top-most layer. in reality, we
allocate subtype fields off of the instance of the subtype, so
there's no need to check beneath the 'top' layer of field/types
in a message to determine whether or not to pass in a context.
2019-07-24 06:31:46 +00:00
lisa neigut b30d7d26ea bolt-gen: handle enums
Add parsing know-how for enum fields. This is necessary for
internally defined wire generators. Enums are denoted by prefixing
the field with an `e:`.

Ex:

   msgdata,msg_name,field_name,e:enum_type,
2019-07-24 06:31:46 +00:00
lisa neigut cfd56d86ee bolt-gen: keep single instance of type_obj per type
we rely, perhaps a bit hackily, on there only being one copy of
each type object floating about. using `deepcopy` on a Message
for message extensions destroys this paradigm, which breaks
things in the case where it's a later defined subtype that contains
variable-length members.

to fix this, we modify `__deepcopy__` on the Field class, such
that it preserves the reference to the original type_obj instance.
2019-07-24 06:31:46 +00:00
lisa neigut 009e9dc945 bolt-gen: neaten up generated comments
remove extra space from the lead-in for inline comments

so that a provided comment like this

    # This is a comment

will appear like this

    /* This is a comment */
2019-07-24 06:31:46 +00:00
lisa neigut 7f5ba35e95 bolt-gen-tests: namespace test tlv's under test_
for posterity.
2019-07-24 02:52:53 +00:00
lisa neigut 281b4c241e bolt-gen: fixup the devtool/decodemsg printing facility
Fixup TLV handling in the bolt printing utility, `devtools/decodemsg`
2019-07-24 02:52:53 +00:00
lisa neigut cedebfd2d9 bolt-gen: add truncated uint support for TLV fields
TLV's have truncated values, add support for them. plus
some 'compilation' tests.
2019-07-24 02:52:53 +00:00
lisa neigut 4261e508a9 bolt-gen: add TLV support
Add in support for buiding TLV's (minus the printing capability)
2019-07-24 02:52:53 +00:00
lisa neigut 79f13fa429 bolt-gen: add compilation tests
Add a test for checking that the bolt-gens do the right thing
for a fairly exhaustive test case set (and that it compiles).

Note that this doesn't check that we've got the memory assignment
pieces worked out.

It's got a kind of exotic reliance on the update-mocks in that in
order to depend on as little of the wire/ code as possible (we
only import wire/wire.h), we include an AUTOGENERATE comment
in the test_cases CSV file, and then run update-mocks as part of
the build for that file.
2019-07-24 02:52:53 +00:00
lisa neigut 12125ec8dc bolt-gen: small cleanups
we were incorrectly printing the references for a subtype case, plus
some errant styling,readability changes
2019-07-24 02:52:53 +00:00
lisa neigut f2819ba7d8 bolt-gen: remove 'is-optional' qualifier from msg field
we now handle optional fields, so we should include them in
the message parsing signatures
2019-07-24 02:52:53 +00:00
lisa neigut 96bf7aead5 bolt-gen: handle variable-sized and optionals
actually do the right thing for variable-sized and optional field
types
2019-07-24 02:52:53 +00:00
lisa neigut 181e1916b2 bolt-gen: add field optional handling
we'll need this for internal wire message formats. also disambiguates
from 'bolt message optional fields', which we rename to extensions here.

example of an optional field declaration (note the ? prefixing the
type):

    msgdata,msg_name,field_name,?type,count

these are handled with either a boolean if they're not present,
or a true value and then the object if they are.
2019-07-24 02:52:53 +00:00
lisa neigut ade594e941 bolt-gen: add 'top-line' file comments to output
if there are any comments that aren't "attached" to a message,
print them at the top of the generated file. we need this for
the fancy auto-gen'd dependencies in the tool-wiregen tests.
2019-07-24 02:52:53 +00:00
lisa neigut fe3f4f52a0 bolt-gen: handle csv inline comments
The bolts don't have in-line comments, but the internal wire
message CSVs do. This adds the 'comments' back in to the
generated docs.
2019-07-24 02:52:53 +00:00
lisa neigut d7a68b75f1 bolt-gen: fixup broken subtype parsing 2019-07-24 02:52:53 +00:00
lisa neigut 1044e37fee bolt-gen: use ordered-dict for fields
Use ordered dict for fields, who's order matters when iterated
through (especially for argument lists).

Reported-By: @mocacinno
2019-07-24 00:43:47 +00:00
Rusty Russell 54790c17ea wire: rename var_int to bigsize, and insist on minimal.
The new TLV spec uses BigSize, like Bitcoin's CompactInt but
*little-endian*.  So change our name for clarity, and insist that
decoding be minimal as the spec requires.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-18 06:51:35 +00:00
lisa neigut 85e325cb1f bolt: update to lightning-rfc:6f6ea63233c new fundamental types
RFC tweaks to the types used to specify message fields
2019-07-16 06:10:58 +00:00
lisa neigut 6c240ab589 bolts: new parsing script and templates for new bolt format
the RFC's extract-format.py is switching to a new format.
this script can correctly parse them.

mostly moves logic over from generate-wire.py, uses a
Python formatting libarary called mako, which needs to be
installed prior to running this script.

you can add it to your system with

    sudo apt-get install python3-mako
2019-07-16 06:10:58 +00:00
lisa neigut 118caae45b tools: simplify check
in this case, we always need a pointer, so remove the check
and just add it to the template
2019-07-16 06:10:58 +00:00
Rusty Russell 38d2899fbb common/per_per_state: generalize lightningd/peer_comm Part 1
Encapsulating the peer state was a win for lightningd; not surprisingly,
it's even more of a win for the other daemons, especially as we want
to add a little gossip information.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-04 01:29:39 +00:00
Rusty Russell cb9c44ef27 gossipd: remove unnecessary dev_unknown_channel_satoshis arg.
We now have a test blockchain for MCP which has the correct channels,
so this is not needed.

Also fix a benchmark script bug where 'mv "$DIR"/log
"$DIR"/log.old.$$' would fail if you log didn't exist from a previous run.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 11:28:44 +00:00
Rusty Russell eda5314cb5 tools/generate-wire.py: handle optional variable-length fields.
We generated code which didn't compile (we never had one before though).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-20 20:31:07 -04:00
Rusty Russell 7ede5aac31 gossip_store: change format so we store raw messages.
Save some overhead, plus gets us ready for giving subdaemons direct
store access.  This is the first time we *upgrade* the gossip_store,
rather than just discarding.

The downside is that we need to add an extra message after each
channel_announcement, containing the channel capacity.

After:
  store_load_msec:28337-30288(28975+/-7.4e+02)
  vsz_kb:582304-582316(582306+/-4.8)
  store_rewrite_sec:11.240000-11.800000(11.55+/-0.21)
  listnodes_sec:1.800000-1.880000(1.84+/-0.028)
  listchannels_sec:22.690000-26.260000(23.878+/-1.3)
  routing_sec:2.280000-9.570000(6.842+/-2.8)
  peer_write_all_sec:48.160000-51.480000(49.608+/-1.1)

Differences:
  -vsz_kb:582320
  +vsz_kb:582316
  -listnodes_sec:2.100000-2.170000(2.118+/-0.026)
  +listnodes_sec:1.800000-1.880000(1.84+/-0.028)
  -peer_write_all_sec:51.600000-52.550000(52.188+/-0.34)
  +peer_write_all_sec:48.160000-51.480000(49.608+/-1.1)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-13 05:16:18 +00:00
GreenAddress fb07265663 remove libbase58, use base58 from libwally (#2594)
* remove libbase58, use base58 from libwally

This removes libbase58 and uses libwally instead.

It allocates and then frees some memory, we may want to
add a function in wally that doesn't or override
wally_operations to use tal.

Signed-off-by: Lawrence Nahum lawrence@greenaddress.it
2019-04-30 23:07:31 +02:00
Christian Decker 62e1423968 fixup! tools/bench-gossipd.sh: make it work (where possible) with DEVELOPER=0 2019-04-24 13:46:39 -05:00
Rusty Russell b248bb155a tools/bench-gossipd.sh: make it work (where possible) with DEVELOPER=0
Some tests require dev support, but the rest can run.  We simplify
the gossip_store output so it's the same in non-dev mode too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-24 13:46:39 -05:00
Rusty Russell 1ccfbef4e7 tools/bench-gossipd.sh: make sure we wait for gossipd to startup sync.
It can take a while if bitcoind has the regtest chain, and grossly
distorts our benchmarks!

Reported-by: Joe Netti <jnetti@blockstream.io>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-24 13:46:39 -05:00
Joe Netti 902bb22a92 devtools/create-gossipstore: cleanups
added sanity check to make sure scid of csv is the same as scid in gossip.
Revised style, mem allocation, and error checks

[ Minor fixups, and updated benchmark script -- RR ]

With data.tar.gz: 456609740 Apr  2 12:33

store_load_msec:35300-42354(37118.2+/-2.7e+03)
vsz_kb:582832
store_rewrite_sec:12.700000-13.430000(12.988+/-0.27)
listnodes_sec:3.000000-3.160000(3.076+/-0.057)
listchannels_sec:30.790000-31.690000(31.03+/-0.34)
routing_sec:0.00
peer_write_all_sec:63.640000-67.860000(66.294+/-1.4)
2019-04-24 13:46:39 -05:00
lisa neigut 8326f6a35c wire-gen: put tlv's after subtypes
in the output, we need tlv's to come after the subtype declarations
and structs so that a tlv can use a subtype without problems
2019-04-24 14:16:15 +02:00
lisa neigut 560f45cd7f wire generator: reverse subtype struct order
nested subtypes need to come first in the struct declaration. we
reverse the order that we read them off so that this requirement
is met
2019-04-24 14:16:15 +02:00
lisa neigut 0ce287d52e tlvs: include correct reference for nested objects 2019-04-24 14:16:15 +02:00
Rusty Russell 981fa68fe0 tools/bench-gossipd.sh: fix routing.
Channels have a htlc_minimum_msat of 10000, which is why we didn't
find routes.

This makes a significant speed drop:

-routing_sec:26.940000-27.990000(27.616+/-0.39)
+routing_sec:60.70

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
lisa neigut f2ecf8e9c3 wire-gen: simplify if statement 2019-04-10 23:42:25 +00:00
lisa neigut 88786b8f7a wire gen: add subtypes to printwires
now we print the subtypes out when you call printwire

note that we have to reverse the order the subtypes appear in
because
  1) they're static and,
  2) a few of them are nested
2019-04-10 23:42:25 +00:00
lisa neigut 51438cef21 wire gen, tlv: fixup broken methodname when tlvs are around
including tlv's in the wire docs breaks the printwire because
there's a bad method name. this fixes that
2019-04-10 23:42:25 +00:00
lisa neigut 0443b464ad wire gen: add in correct printing for arrays of subtypes
subtypes don't use the fance type registration that other
'set structs' do, see devtools/printwire.c
2019-04-10 23:42:25 +00:00
lisa neigut 2e8768a279 wire-gen: rename 'is_tlv' to 'is_embedded' etc 2019-04-10 23:42:25 +00:00
lisa neigut e31111d70b subtypes: fixup context reference for subtypes
subtype children should be allocated off of themselves. this was
failing to compile for embedded subtypes (subtype within a subtype).
2019-04-10 23:42:25 +00:00
lisa neigut 97b938a469 gen: fixup unescaped \n in printwire declaration for tlvs 2019-04-10 23:42:25 +00:00
lisa neigut a385d1de4c subtype: update parser to understand non-'$' csv output
the original version of the subtype generator emitted '$'
to designate that a field was a subtype; now it's got a different
format:

	funding_type,8,num_inputs,2
	funding_type,10,input_info,num_inputs*input_info

this patch updates our generator to understand this new format
2019-04-10 23:42:25 +00:00
lisa neigut 803b161d7e subtypes: add flag to include subtype wire functions to header file
This is needed so that some csv's can expose their subtype parsing
functions in their header. This gets used in a later PR where
we start replacing manually created 'subtype' definitions with
generated ones.
2019-04-10 23:42:25 +00:00
lisa neigut e8a10b019d gen: move an error check to catch all instances of failure
`m` might not be set on the optional set as well, so move this check
down so that we now encompass both codepaths
2019-04-10 23:42:25 +00:00
lisa neigut 48078f7572 tlv: fixup deref for embedded structs on fromwire 2019-04-10 23:42:25 +00:00
lisa neigut 37d6545191 subtypes: add some parsing for subtypes, so that it passes
this probably could be consolidated, as it splits
out all the print_to/fromwire method stuff for the Subtype class
2019-04-10 23:42:25 +00:00
lisa neigut 94395c6a9a tlv: remove requirement of having tlv_name 2019-04-10 23:42:25 +00:00
lisa neigut de2fb7c9ef tlv: move tlv-specific message functions 2019-04-10 23:42:25 +00:00
lisa neigut 86a099a62f wire-gen: initial start on subtypes
first pass at adding subtype structs
2019-04-10 23:42:25 +00:00
lisa neigut e4658c241e tlv: break out TLVs into new subclass
make TLV messages their own subclass of Message. this makes
other clean ups easier
2019-04-10 23:42:25 +00:00
Rusty Russell dc6d53e787 lightningd: don't bother pretty-printing JSON.
This doesn't result in a speedup for our benchmark, since we use the
cli tool which does the formatting.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:33422-36830(35196.2+/-1.2e+03)
	vsz_kb:2637488
	store_rewrite_sec:36.030000-37.630000(36.794+/-0.52)
	listnodes_sec:0.720000-0.950000(0.86+/-0.077)
	listchannels_sec:40.300000-41.080000(40.668+/-0.29)
	routing_sec:30.440000-31.030000(30.69+/-0.2)
	peer_write_all_sec:50.060000-52.800000(51.416+/-0.91)

MCP notable changes from 2 patches ago (>1 stddev):
	-listchannels_sec:48.560000-55.680000(52.642+/-2.7)
	+listchannels_sec:40.300000-41.080000(40.668+/-0.29)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell 5009d628a3 lightning-cli: do pretty-printing.
Plugins don't do it right anyway, and we're about to remove it from
lightningd.  Produces same format as json_pp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell 0b484b111e gossipd: make more compact getchannels entries.
We can save significant space by combining both sides: so much that we
can reduce the WIRE_LEN_LIMIT to something sane again.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:34467-36764(35517.8+/-7.7e+02)
	vsz_kb:2637488
	store_rewrite_sec:35.310000-36.580000(35.816+/-0.44)
	listnodes_sec:1.140000-2.780000(1.596+/-0.6)
	listchannels_sec:55.390000-58.110000(56.998+/-0.99)
	routing_sec:30.330000-30.920000(30.642+/-0.19)
	peer_write_all_sec:50.640000-53.360000(51.822+/-0.91)

MCP notable changes from previous patch (>1 stddev):
	-store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
	+store_rewrite_sec:35.310000-36.580000(35.816+/-0.44)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell 91849dddc4 wire: use struct node_id for node ids.
Don't turn them to/from pubkeys implicitly.  This means nodeids in the store
don't get converted, but bitcoin keys still do.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:33934-35251(34531.4+/-5e+02)
	vsz_kb:2637488
	store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
	listnodes_sec:1.020000-1.290000(1.146+/-0.086)
	listchannels_sec:51.110000-58.240000(54.826+/-2.5)
	routing_sec:30.000000-33.320000(30.726+/-1.3)
	peer_write_all_sec:50.370000-52.970000(51.646+/-1.1)

MCP notable changes from previous patch (>1 stddev):
	-store_load_msec:46184-47474(46673.4+/-4.5e+02)
	+store_load_msec:33934-35251(34531.4+/-5e+02)
	-vsz_kb:2638880
	+vsz_kb:2637488
	-store_rewrite_sec:46.750000-48.280000(47.512+/-0.51)
	+store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Rusty Russell fbb494fba3 devtools/create-gossipstore: clean up enough to pass check-source.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00
Rusty Russell 891ee20a59 tools/bench-gossipd.sh: rough benchmark for gossipd and the million channels project
Outputs CSV.  We add some stats for load times in developer mode, so we can
easily read them out.

peer_read_all_sec doesn't work, since we seem to reject about half the
updates for having bad signatures.  It's also very slow...

routing fails, for unknown reasons, so that failure is ignored in routing_sec.

Results from 5 runs, min-max(mean +/- stddev):
	store_load_msec,vsz_kb,store_rewrite_sec,listnodes_sec,listchannels_sec,routing_sec,peer_write_all_sec
	39275-44779(40466.8+/-2.2e+03),2899248,41.010000-44.970000(41.972+/-1.5),2.280000-2.350000(2.304+/-0.025),49.770000-63.390000(59.178+/-5),33.310000-34.260000(33.62+/-0.35),42.100000-44.080000(43.082+/-0.67)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>



Header from folded patch 'fixup!_tools-bench-gossipd.sh__rough_benchmark_for_gossipd_and_the_million_channels_project-2.patch':

fixup! tools/bench-gossipd.sh: rough benchmark for gossipd and the million channels project

Suggested-by: @niftynei



Header from folded patch 'fixup!_tools-bench-gossipd.sh__rough_benchmark_for_gossipd_and_the_million_channels_project-1.patch':

fixup! tools/bench-gossipd.sh: rough benchmark for gossipd and the million channels project

MCP filename change.



Header from folded patch 'tools-bench-gossipd.sh__dont_print_csv_by_default.patch':

tools/bench-gossipd.sh: don't print CSV by default.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>



Header from folded patch 'fixup!_tools-bench-gossipd.sh__rough_benchmark_for_gossipd_and_the_million_channels_project.patch':

fixup! tools/bench-gossipd.sh: rough benchmark for gossipd and the million channels project

Make shellcheck happy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00
lisa neigut ed1223492b tlvs: add methods for decodemsg utility
fixup printing methods in devtools/decodemsg such that TLV's can
now be printed as well. here's how you'd use it:

   $ ./devtools/decodemsg --tlv opening_tlv 0120001E020202020202020202020202020202020202020202020202020202020202
   > WIRE_OPTION_UPFRONT_SHUTDOWN_SCRIPT (size 32):
   > shutdown_scriptpubkey=[020202020202020202020202020202020202020202020202020202020202]
2019-04-08 00:37:29 +00:00
lisa neigut bad0ac6ed6 tlv: use `var_int`s for size of messages
TLV's use var_int's for messages sizes, both internally and
in the top level (you should really stack a var_int inside a var_int!!)

this updates our automagick generator code to understand 'var_ints'
2019-04-03 03:15:42 +00:00
lisa neigut 1213f44071 tlv: adapt to work with new output format
Updated to match what the CSV generator in the RFC repo actually
outputs, see https://github.com/lightningnetwork/lightning-rfc/pull/597
2019-04-03 03:15:42 +00:00
lisa neigut df9774c2be tlv: fixup FIXME -- remove comments + use includes
include includes for TLV _csv files
2019-04-03 03:15:42 +00:00
lisa neigut 5d8b059ccc tlv: free intermediate messages when they're created
otherwise they'll get cleaned up when the message is free'd.
it's nbd either way, but this seems tighter.
2019-04-03 03:15:42 +00:00
lisa neigut 4afcc2e5a8 tlv: fixup parsing for multi-message tlv's
need to pass in a pointer to the array so that when we advance
the array in the subcalls, it advances in the parent. oops
2019-04-03 03:15:42 +00:00
lisa neigut d0f50c8690 tlv: fail if parsed length doesn't match packet length 2019-04-03 03:15:42 +00:00
lisa neigut 28c3f9ca21 tlv: don't crash when you fail to parse the TLV
passing back a null TLV was crashing here, because we tried to
dereference a null pointer. instead, we put it into a temporary
struct that we can check for NULL-ness, before assigning to the
passed in pointer.
2019-04-03 03:15:42 +00:00
lisa neigut d8738452ed tlv: it's ok to be odd
fail if a message type is even and it's not included. otherwise,
continue with the next message type.
2019-04-03 03:15:42 +00:00
lisa neigut b89ea071e8 tlv: allocate tlv structs from within
let's let the fromwire__tlv methods allocate the tlv-objects and
return them. we also want to initialize all of their underlying
messages to NULL, and fail if we discover a duplicate mesage type.

if parsing fails, instead of returning a struct we return NULL.

Suggested-By: @rustyrussell
2019-04-03 03:15:42 +00:00
lisa neigut 5aea65b463 tlv: make message sizes u8 not u16
Suggested-By: @rustyrussell
2019-04-03 03:15:42 +00:00
lisa neigut d6332997bd tlv: remove leading '_' from things
Suggested-By: @rustyrussell
2019-04-03 03:15:42 +00:00
lisa neigut aba4e161ce tlv: calculate sizeof by measuring message length
much better than statically calculating the sizeof
2019-04-03 03:15:42 +00:00
lisa neigut 9a23a354fd tlv: consolidate basetype parsing
clean up basetype parsing code a bit
2019-04-03 03:15:42 +00:00
lisa neigut 6f2e70a6ac tlv: add fromwire_ methods for TLV structs 2019-04-03 03:15:42 +00:00
lisa neigut ef610dcab3 tlv: build tlv top-level structs
construct structs for the TLV's. these will be the 'return type'
for tlv fields in parent messages (so to speak)
2019-04-03 03:15:42 +00:00
lisa neigut 18a23b31de tlv: add structs for message types to wire format .h's (header files)
Since messages in TLV's are optional, the ideal way to deal with
them is to have a 'master struct' object for every defined tlv, where
the presence or lack of a field can be determined via the presence
(or lack thereof) of a struct for each of the optional message
types.

In order to do this, appropriately, we need a struct for every
TLV message. The next commit will make use of these.

Note that right now TLV message structs aren't namespaced to the
TLV they belong to, so there's the potential for collision. This
should be fixed when/where it occurs (should fail to compile).
2019-04-03 03:15:42 +00:00
lisa neigut 0b12d90c4a tlv: add tlv messages to general message set
Add tlv-messages to the general messages set so that their parsing
messages get printed out.

FIXME: figure out how to account for partial message length processing?
2019-04-03 03:15:42 +00:00
lisa neigut 44d052e6c3 tlv: parse fields for tlv messages 2019-04-03 03:15:42 +00:00
lisa neigut d9904c95ab tlv: add tlv fields to enum declarations in implementation file
'.c' wire format files include case statements to print the names
of enums. Include such methods for the enums pertaining to
tlv's as well.
2019-04-03 03:15:42 +00:00
lisa neigut d51ad50032 tlv: add tlv field's type enums to header file
Version 1.1 of the lightning-rfc spec introduces TLVs for optional
data fields. This starts the process of updating our auto-gen'd
wireformat parsers to be able to understand TLV fields.

The general way to declare a new TLV field is to add a '+' to the
end of the fieldname. All field type declarations for that TLV set
should be added to a file in the same directory by the name
`gen_<field_name>_csv`.

Note that the FIXME included in this commit is difficult to fix, as
we currently pass in the csv files via stdin (so there's no easy
way to ascertain the originating directory of file)
2019-04-03 03:15:42 +00:00
Christian Decker 3ce98ab7de wire: Allow non-u16 length variables in non-bolt wire formats
Otherwise we can't really return a variable sized message with more than 65k
results. This was causing an integer overflow in `listchannels` (see #2504 for
details).

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-03-27 12:48:52 +01:00
arowser 04c60175ca compatible posfix sh 2019-03-17 03:47:38 +00:00
Rusty Russell 73c02691a3 tools/built-release.sh: fix archives we produce.
We were tarring up the build dir, not the destination dir!  We did this
for 0.6.3 and nobody noticed :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-03-01 21:38:08 +00:00
Rusty Russell 66efcfa855 tools/repro-build.sh: do not turn on address-sanitizer by default.
@cdecker reports that this gives warnings on exit; and we can't suppress
them by setting ASAN_OPTIONS within the binary itself, unfortunately.

So for 0.7, disable it by default.  I'll work through the errors for 0.7.1.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-28 05:17:21 +00:00
Rusty Russell 1dcc482350 Update CHANGELOG.md for -rc2.
And fix trivial typo in MAKING-RELEASES.md, and date retreival in
build-release.sh and repro-build.sh (real git tags start with v!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-26 04:16:34 +00:00
Rusty Russell dce58393cb build-release.sh: fix zipfile determinism.
I tried building zipfile on a fresh clone inside KVM, and got

1. Different times inside the zipfile, since zip seems to save *local* times.
2. A different zipfile order, since zip seems to use filesystem order.

Fix both of these.  I don't know if LANG=C is necessary for git
ls-files, but it can't hurt.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-25 23:36:47 +00:00
Rusty Russell b2ae4f0fd7 tools/repro-build.sh: script to build an identical binary tarball.
For the moment it's only Ubuntu 18.04.1.

Complete documentation is in the final commit; you can test this using
the prior commit and comparing with my intermediate files and results
at:

	https://ozlabs.org/~rusty/clightning-repro

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-23 10:19:33 +00:00
Rusty Russell 9f6d5a3c47 tools/build-release.sh: make it work in a pristine clone.
We need the submodules to exist.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-22 11:20:51 -08:00
Rusty Russell 6e44073bb0 tools/build-release.sh: make zipfile deterministic.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-22 11:20:51 -08:00
Rusty Russell 91fdfbe2f4 tools/build-release.sh: work around git status bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-22 11:20:51 -08:00
Rusty Russell 28f5da7b2f tools/generate-wire: use amount_msat / amount_sat for peer protocol.
Basically we tell it that every field ending in '_msat' is a struct
amount_msat, and 'satoshis' is an amount_sat.  The exceptions are
channel_update's fee_base_msat which is a u32, and
final_incorrect_htlc_amount's incoming_htlc_amt which is also a
'struct amount_msat'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 08:01:37 +00:00
Rusty Russell 7fad7bccba common/amount: new types struct amount_msat and struct amount_sat.
They're generally used pass-by-copy (unusual for C structs, but
convenient they're basically u64) and all possibly problematic
operations return WARN_UNUSED_RESULT bool to make you handle the
over/underflow cases.

The new #include in json.h means we bolt11.c sees the amount.h definition
of MSAT_PER_BTC, so delete its local version.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 00:44:57 +00:00
Rusty Russell 175c869b6b update-mocks: handle NO_NULL_ARGS and NON_NULL_ARGS functions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-21 00:44:57 +00:00
Rusty Russell 2db84d6653 tools/check-setup_locale.sh: don't get caught by main in non-C files.
We're about to put one in configure.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-08 01:10:17 +00:00
Rusty Russell e857229de4 tools: make build-release more friendly.
You can now override sanity checks for testing, and also
specify exactly what to build.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-02-08 01:10:17 +00:00
Rusty Russell 1567238dd9 invoice: option to expose/not-expose private channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 26dda57cc0 utils: make tal_arr_expand safer.
Christian and I both unwittingly used it in form:

	*tal_arr_expand(&x) = tal(x, ...)

Since '=' isn't a sequence point, the compiler can (and does!) cache
the value of x, handing it to tal *after* tal_arr_expand() moves it
due to tal_resize().

The new version is somewhat less convenient to use, but doesn't have
this problem, since the assignment is always evaluated after the
resize.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 12:01:38 +01:00
Rusty Russell 0d5f0d79da build: allow building from source zip file.
Changes both how we construct it and how we deal with not having git.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-01-15 02:18:30 +00:00
Wladimir J. van der Laan c57e63b31d tools: Add missing unistd.h header
`close` was implicitly defined, include `unistd.h` to define it
explicitly so that the FreeBSD compile succeeds.
2018-12-29 13:34:23 +01:00
Nicolas Dorier f9f4ed8e11 [Docker] Make sure lightningd receive SIGTERM
In conjunction to https://github.com/ElementsProject/lightning/pull/2172 this fix https://github.com/ElementsProject/lightning/issues/2074 if EXPOSE_TCP is false (which is the case in production)

More info https://github.com/ElementsProject/lightning/pull/2172#issuecomment-447727668
2018-12-29 13:29:54 +01:00
Rusty Russell 819078fe18 param: make command_fail/command_success WARN_UNUSED_RESULT.
This causes a compiler warning if we don't do something with the
result (hopefully return immediately!).

We use was_pending() to ignore the result in the case where we
complete a command in a callback (thus really do want to ignore
the result).

This actually fixes one bug: we didn't return after command_fail
in json_getroute with a bad seed value.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-20 03:22:32 +00:00
Christian Decker 05dc095ebc mocks: Fix the mocks generation fix
Turns out that I should have tested these with a new dependency
instead of just submitting. `sed` was missing the s command.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-15 22:46:17 +00:00
Christian Decker c5d77c391a mocks: Drop invalid __sentinel__ value in generated mocks
This was introduced in ed268d6c, which broke the mocks
generation. This just filters out the invalid sentinel value.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-12-15 01:13:02 +00:00
Rusty Russell 31a375af53 lightningd: add runtime checking for all system-provided libs.
And I tested this by rolling my own libz; make indeed detects
the change and fixes it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-14 05:36:34 +00:00
Rusty Russell 2a90325e80 Makefile: add dependency on external header versions.
There were a few reports that upgrading Ubuntu recently caused issues
because we assert that the sqlite3 library version matches the one we
were built with. 'make' doesn't fix this, because it doesn't know the
external libraries have changed.

Fix this harder, with a helper which updates a file every binary depends
on, which gets relinked every time so we detect link changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-12-14 05:36:34 +00:00
Rusty Russell 22526911cb tools/generate-wire.py: allocate array members off array in fromwire.
If we have an array of varlen structures (which require a ctx arg), we
should make that arg the array itself (which was tal_arr()), not the
root context.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-11-22 05:15:42 +00:00
Rusty Russell d4f164eb39 Release: add helper script for release, and checklist for the process.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-31 10:56:44 +10:30
Rusty Russell ad2519a6f4 spelling: Check LockTime Verify.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-10-23 16:55:35 +02:00
Rusty Russell 96f05549b2 common/utils.h: add tal_arr_expand helper.
We do this a lot, and had boutique helpers in various places.  So add
a more generic one; for convenience it returns a pointer to the new
end element.

I prefer the name tal_arr_expand to tal_arr_append, since it's up to
the caller to populate the new array entry.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-09-27 22:57:19 +02:00
lisa neigut 3ce53ab9ed tools/gen-wire: 3th -> 3rd
Small grammar fix in wire gen tool
2018-09-22 19:06:06 +02:00
Rusty Russell 61c6b8b25a tools/generate-wire.py: mark your_last_per_commitment_secret as a struct secret.
Without an override, the tool assumes it's a sha256.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-08-23 14:46:22 +02:00
Fabian Raetz 700a0d702d replace use of non standard mv -n flag
The -n flag is a non-standard flag and should not be used in scripts.
See https://man.openbsd.org/FreeBSD-11.1/mv#COMPATIBILITY.
2018-08-15 06:50:29 +00:00
practicalswift 9d9a9523d0 Use snprintf(...) instead of sprintf(...) 2018-08-02 16:14:21 +09:30
Rusty Russell 5cf34d6618 Remove tal_len, use tal_count() or tal_bytelen().
tal_count() is used where there's a type, even if it's char or u8, and
tal_bytelen() is going to replace tal_len() for clarity: it's only needed
where a pointer is void.

We shim tal_bytelen() for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-30 11:31:17 +02:00
Rusty Russell 93cf28553d devtools/decodemsg: add --onion option for decoding onion errors.
This requires a tweak to generate-wire.py too, since it always called the
top-level routine 'print_message'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-27 14:12:00 +02:00
Rusty Russell d241bd762c connectd: don't use gossip_getnodes_entry.
gossip_getnodes_entry was used by gossipd for reporting nodes, and for
reporting peers.  But the local_features field is only available for peers,
and most other fields are only available from node_announcement.

Note that the connectd change actually means we get less information
about peers: gossipd used to do the node lookup for peers and include the
node_announcement information if it had it.

Since generate_wire.py can't create arrays-of-arrays, we add a 'struct
peer_features' to encapsulate the two feature arrays for each peer, and
for convenience we add it to lightningd/gossip_msg.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-25 02:13:52 +00:00
William Casarin f8fa4213f1 tools: use /usr/bin/env bash instead of /bin/bash
These commands fail on systems that do not have bash under /bin

Signed-off-by: William Casarin <jb55@jb55.com>
2018-07-24 00:25:43 +00:00
Rusty Russell 232b330284 tools/generate-wire.py: simple scheme to autoindent.
I verified that the generated files don't change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell 6758062a56 tools/generate-wire.py: remove ugly blank lines after for loops
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell ab28972aee tools/generate-wire.py: style cleanups.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell e217bc1220 per-commit-secret is a struct secret, not a sha256.
Well, it's generated by shachain, so technically it is a sha256, but
that's an internal detail.  It's a secret.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Rusty Russell ac4c6b1a82 tools/generate-wire.py: support for optional fields, with ? before typename.
We already work around this by using an array with a 0/1 length convention,
but better to be explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-17 12:32:00 +02:00
Jan Sarenik 9f519afc5d tools/check-includes.sh: shellcheck recommended fixes
$ shellcheck --version
    ShellCheck - shell script analysis tool
    version: 0.5.0
    license: GNU General Public License, version 3
    website: https://www.shellcheck.net
    $ make check-source
    ...

    In tools/check-includes.sh line 14:
        if [[ $(grep -cE "^#((ifndef|define) ${HEADER_ID}|endif /\* ${HEADER_ID} \*/)$" "${HEADER_FILE}") != 3 ]]; then
                                                                 ^-- SC1117: Backslash is literal in "\*". Prefer explicit escaping: "\\*".

    In tools/check-includes.sh line 28:
        git ls-files | grep -v 'ccan/' | grep -E "\.${1}"'$'
                                                  ^-- SC1117: Backslash is literal in "\.". Prefer explicit escaping: "\\.".
2018-07-04 01:48:21 +00:00
Rusty Russell c02ff11506 print_wire: hand field names to print routines.
This lets us override how we print them.

Also, add dependency on header for devtools/Makefile.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-07-01 14:55:29 +02:00
nicolas.dorier aeafe4dbe7 [Docker] Fix: Bind socat to the right port 2018-06-27 13:40:02 +02:00
Christian Decker 0d3c3a225e docker: Add a usable dockerfile
This is based on @NicolasDorier's excellent proposal for a Dockerfile, sans the
writing of a config file.

Co-authored-by: Nicolas Dorier <nicolas.dorier@gmail.com>
Co-authored-by: Christian Decker <decker.christian@gmail.com>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-06-21 23:51:57 +00:00
Rusty Russell f64272caea tools/check-manpage.sh: make it work on FreeBSD.
Extended regular expressions are more compatible, it seems (and simpler!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-21 14:00:23 +02:00
arowser 85e02127a9 align generate wire template 2018-06-20 06:56:42 +00:00
Rusty Russell 76dd1577bd external: fix submodule handling for parallel builds, submodule URL changes.
If we change an upstream URL, all submodules break.  Users would need
to run 'git submodule sync'.  Note that the libbacktrace fix was merged
upstream so this is no longer necessary, but it's good for future changes.

Also, stress-testing reveals that git submodule fails locking
'.git/config' when run in paralell.  It also segfaults and other
problems.

This is my final attempt to fix submodules; I've wasted far too many
days on obscure problems it creates: I've already lost one copy of my
repo to apparently unfixable submodule preoblems.  The next "fix" will
be to simply import the source code so it works properly.

Reported-by: @jsarenik
Fixes: #1543
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-06-08 17:56:01 +02:00
Rusty Russell 626e9fed16 Makefile: add check that manpage and command options match.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-05-20 02:32:42 +00:00
practicalswift abf510740d Force the use of the POSIX C locale for all commands and their subprocesses 2018-04-27 14:02:59 +02:00
Rusty Russell 5b7fcab766 tools: fix shellcheck errors.
Not sure how this got through Travis.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-04-06 14:26:53 +02:00
practicalswift 1a55147870 Be consistent in choice of shell across all shell scripts in repo 2018-04-04 15:06:30 +02:00
practicalswift b95d3b8f54 Fix shellcheck warnings 2018-04-04 15:06:30 +02:00
practicalswift e078fcefdd Add check-includes.sh: Find redundant includes. Check ifndef:s. 2018-03-25 23:54:21 +00:00
ZmnSCPxj e6de41b468 tools/rel.sh: Make POSIXLY compatible.
Fixes: #1221

We were using `\x0` to match NUL chars in the input (on the
assumption that NUL chars are "impossible" for decent LFS-compliant
systems).
However `\x0` is a GNUism.
Use the `\n` and the newline character, which is supported by (most)
POSIX sed.
2018-03-16 01:50:26 +00:00
practicalswift 160f9b06a5 Fix variable typo in generate-wire.py 2018-02-24 11:44:32 +01:00
practicalswift 0a20bd44f0 Fix flake8 warnings: W191 indentation contains tabs + E101 indentation contains mixed spaces and tabs 2018-02-22 22:21:31 +01:00
practicalswift e18948a6a9 Remove unused variables. Fix flake8 warnings. 2018-02-22 22:21:31 +01:00
practicalswift ae7d857c44 Make Python code conform to the Python PEP 8 style guide 2018-02-22 10:47:19 +01:00
Rusty Russell e92b710406 tools/generate-wire.py: remove length argument from fromwire_ routines.
We always hand in "NULL" (which means use tal_len on the msg), except
for two places which do that manually for no good reason.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-20 22:36:21 +01:00
Rusty Russell b7db06b577 tools/generate-wire.py: wirestring type for handing strings.
A convenient alias for char *, though we don't allow control characters
so our logs can't be fooled with embedded \n.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 674a2c7554 tools/generate_wire.py: make bitcoin_tx a varlen structs
Now it allocates naturally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 526d3a232e tools/generate_wire.py: generate varlen arrays properly.
These are now logically arrays of pointers.  This is much more natural,
and gets rid of the horrible utxo array converters.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell ad8dfaca1c tools/generate_wire.py: make varlen structs self-allocate.
If we tell it a struct is variable length, make fromwire() allocate
and return it off ctx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-08 19:07:12 +01:00
Rusty Russell 32abb4d928 tools/generate-wire.py: --printwire option to create print routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-02 00:57:10 +00:00
Rusty Russell 0cef23d33a check-spelling: don't paginate, and give file and line.
The pagination causes it to wait for a keypress even with no output
under emacs (complaining about the terminal); we don't want it anyway.

Example output:

Makefile:228:#lighnting!
Identified a likely misspelling of the word "lightning" (see above). Please fix.
Is this warning incorrect? Please teach tools/check-spelling.sh about the exciting new word.
Makefile:230: recipe for target 'check-spelling' failed

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2018-02-01 14:40:31 +01:00
practicalswift 9f47c0431a Fix typos 2018-01-28 13:53:39 +01:00
practicalswift 3e40a48332 Add lightning spell check 2018-01-26 01:02:15 +00:00
practicalswift a08b6fffc6 Add consistency check for dependencies listed in README.md and doc/INSTALL.md 2018-01-19 10:11:19 +00:00
ZmnSCPxj fdbf2f41b4 generate-wire.py: Add scare comments to output.
Discourage developers from modifying files generated by generate-wire.py
2018-01-10 23:53:12 +00:00
ZmnSCPxj 1946f0c288 Makefile: Add BINTOPKGLIBEXECDIR macro. 2018-01-09 14:52:48 +01:00
Christian Decker 59128a86ee hsm: Make sure to pass close_info along with the UTXO
The close_info is needed to re-derive the secret key that is supposed
to be used to sign the input spending the output.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2018-01-08 17:19:25 +01:00
practicalswift 5123c4e059 Fix access to undefined variable 2017-12-28 16:05:15 +01:00
practicalswift aee3661b35 Fix typos (repeated words) 2017-12-28 16:04:38 +01:00
Rusty Russell 810abb6b21 bitcoin: create new wrapper type bitcoin_blkid, log backward endianness.
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_blkid_to_hex() so it's reversed as people expect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-21 11:05:38 +00:00
Rusty Russell 0237e0b28c bitcoin: create new wrapper type bitcoin_txid, log backward endianness.
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_txid_to_hex() so it's reversed as people expect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-21 11:05:38 +00:00
Rusty Russell 42b9df59c1 update-mocks: allow it to handle missing notleak_
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-20 12:43:10 +01:00
practicalswift 0353ec0983 Remove trailing whitespace 2017-12-11 03:35:59 +00:00
practicalswift 61c47c09d0 Fix typos 2017-12-08 13:07:20 +01:00
Rusty Russell 44e45348f2 option-data-loss-protect: fix generate-wire.py and update.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-16 15:26:41 +01:00
Rusty Russell 5320cab2de generate-wire.py: create structures for optional fields.
If a structure foo has a optional fields opt1 and opt2, this creates
towire_foo, towire_foo_opt1 and towire_foo_opt2 (since opt2 implies opt1),
similarly for fromwire_*.

This requires the callers to be updated to call the correct routines (eg.
try fromwire_foo_opt2, then fromwire_foo_opt1, then finally fromwire_foo),
but this is a minimal change to the generation code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-09 13:55:24 +01:00
Rusty Russell ee1fb07c8e update-mocks: fix stubs generation for fatal().
We're going to need this, and the PRINTF_FMT(1,2) in front of it caused
mockup.sh to miss the declaration.

We also eliminate the obviously-unused fallback case (which referred
to daemon/*.h).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 04:14:33 +00:00
William Casarin 52f741de97 tools/generate-wire.py: generalize python3 path
Generalize generate-wire.py python3 path

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-10 20:12:37 +02:00
Rusty Russell 32631b4278 generate-wire.py: add --bolt arg, use size->type hacks only when that's specified.
For our own internal comms CSVs, we should always name explicit types.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-29 14:40:34 +02:00
Rusty Russell 8a829ba9cb update-mocks: move mock generation into tools/, fix and generalize.
update-mocks was broken, since it assumed the daemon/ directory.

We now use "make" directly to build the test file and harvest errors,
and are more robust if it simply doesn't compile (ie. fails, but no
linker errors).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-03 02:01:54 +02:00
Rusty Russell b3514d3430 generate-wire.py: generate chain_hash fields as sha256_double.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-22 14:56:02 +02:00
Rusty Russell 26ffa8fdec check-bolt: move to tools/
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-22 14:56:02 +02:00
Rusty Russell aa23a2a93f generate-wire.py: handle unmarshalling of a single varlen_struct
We previously only handled it in arrays.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-20 13:06:41 +09:30
Rusty Russell 7884343c15 tools/generate-wire.py: enums are assignable.
Simplfies their marshalling/unmarshalling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell be6347b814 tools/generate-wire.py: unmarshall zero-length arrays as NULL.
This is symmetrical with marshalling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-27 10:25:53 +09:30
Rusty Russell da18645901 tools/generate-wire.py: don't avoice tal_count() on NULL.
It actually works fine with recent ccan/tal (returning 0).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-27 10:25:53 +09:30
Rusty Russell 9868e896d9 tools/generate-wire.py: handle marshal/unmarshal of enum types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell ad3ab53fa4 lightningd/htlc_wire: wire types for sending HTLCs to/from channel daemon.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell a2ffccd740 tools/generate-wire.py: fix generation of array of assignable types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-07 09:19:04 +09:30
Rusty Russell cb5ad91788 tools/generate-wire.py: allow consecutive fields to use same len var.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-07 09:19:04 +09:30
Rusty Russell 6e0e1c7067 Update to latest BOLT (hyphens changed to underscores).
Now in sync with 8ee57b97738b1e9467a1342ca8373d40f0c4aca5.

Our tool doesn't need to convert them any more, but we actually had a
mis-typed field in the HSM which needed fixing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-12 12:59:09 +02:00
Christian Decker 67315be673 travis: Simplified build script
The Dockerfile is now stored in contrib and built using the Docker
Hub. This allows us to simply pull in the finished image from the hub
instead of having to build it ourself. Should shave off about 2
minutes from the build time.

I also switched to running the individual build and check steps in
their own containers, but on the same volume, so travis can group the
commands and run them independently.
2017-04-29 10:29:44 +09:30
Rusty Russell 6f9a7f7aa1 Travis support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-04 15:27:34 +09:30
Rusty Russell 9b6753173c tools/generate-wire: allow constant multiplier in fields.
Not just len*u8, but 1254*u8 for explicit fixed-length arrays. 

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-01 23:59:46 +10:30
Rusty Russell d072f2ec06 wire: understand struct preimage, don't treat it as sha256.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-01 23:59:46 +10:30
Christian Decker b2ea4cfd66 wiregen: Passing ctx to array helpers that require it
Some of the struct array helpers need to allocate data when
deserializing their fields. The `getnodes` reply is one such example
that allocates the hostname. Since the change to calling array helpers
the getnodes call was broken because it was attempting to allocate off
of the entry, which did not have a tal header, thus failing.
2017-03-20 11:18:00 +10:30
Rusty Russell e042198cf8 tools/generate-wire.py: allow typename instead of type sizes.
We use the fourth value (size) to determine the type, unless the fifth
value is suppled.  That's silly: allow the fourth value to be a typename,
since that's the only reason we care about the size at all!

Unfortunately there are places in the spec where we use a raw fieldname
without '*1' for a length, so we have to distingish this from the
typename case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-03-16 14:35:25 +10:30
Rusty Russell 9768e4c907 tools/generate-wire.py: open-code array handling for non-u8.
Except for the trivial case of u8 arrays, have the generator create
the loop code for the array iteration.

This removes some trivial helpers, and avoids us having to write more.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-03-16 14:35:23 +10:30
Christian Decker 4a995a42de gen-wire: Cleanup of the wire generator tool
The wiregen tool was a bit hard to maintain since it was printing all
over the place, mixing template and processing logic. This commit
tears the two apart, externalizes everything that is not a single code
line, and repackages it into templates. Specifically functions are now
their own template and header/implementation files are a template.

Furthermore this simplifies some of the boilerplate of mapping types
to sizes and back again, by extracting them into dicts.

All changes have been verified to produce identical results on the
current wire definitions, except a bit of whitespace changes.
2017-03-06 10:22:02 +10:30
Rusty Russell 7419fde9a0 Update to new spec: differentiate channel_id and short_channel_id.
The spec 4af8e1841151f0c6e8151979d6c89d11839b2f65 uses a 32-byte 'channel-id'
field, not to be confused with the 8-byte short ID used by gossip.  Rename
appropriately, and update to the new handshake protocol.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-03-02 22:51:49 +10:30
Rusty Russell 9fd40da38c generate-wire: don't hand unknown structures specially.
It's awkward to handle them differently.  But this change means we
need to expose them to the generated code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-02-21 15:15:19 +10:30
Rusty Russell fd09345628 generate-wire: handle variable-size arrays of unknown types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-02-21 14:59:14 +10:30
Rusty Russell 1edce4878d bitcoin/signature.h: remove struct signature.
It's a wrapper around secp256k1_ecdsa_signature, so use that directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-25 11:03:55 +10:30
Rusty Russell 2b8c7cc840 tools/generate-wire.py: don't allocate on unknown names.
This introduces a potential leak; use a static buffer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-10 15:20:20 +10:30
Rusty Russell 42f474af18 wire: make lengths of variable fields implied by tal_count()
This is a much nicer interface, and works better in practice too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-10 15:19:25 +10:30
Rusty Russell b6a55a68ea tools/generate-wire.py: handle non-integer enum values.
BOLT 4 uses values like "PERM|1", so allow that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-06 13:40:06 +10:30
Rusty Russell 2057de6d4f tools/generate-wire.py: Apparently OptionParser is deprecated, use argparse.
It is a bit nicer though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-06 13:24:45 +10:30
Rusty Russell 8b72604d4d wire: allow #include directives in csv comments, and unknown structure types.
This lets us marshal and unmarshal undefined complex types; the fromwire
function will allocate it for us, so we don't even need to know the size.

This turns out to be really nice for marshalling 'struct crypto_state'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-06 13:24:45 +10:30
Rusty Russell f50af430e1 wire: add bool routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 937a62100f generate-wire.py: allow optional typename in csv file.
For our internal CSV files, we can specify the type explicitly rather
than trying to guess (eg. bool).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 8ad1298f88 generate-wire.py: add enum->name function.
Pretty!  Takes an int instead of the enum directly, because in the
main daemon we call it via a function pointer, so want them all the
same type.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 54cfbfeba9 generate-wire.py: generate enum in order provided in CSV
This means the comments make more sense.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell ba9e4f9377 generate-wire.py: allow hex values in csv files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell bdaa22e247 generate-wire.py: allow NULL len parameter from fromwire_*
It implies tal_count() gives the length. Great for almost all callers which
don't care if there are extra bytes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell e076d56709 generate-wire.py: include type bytes in towire/fromwire routines.
This removes some redundancy in creating messages, but also allows
a lazy form or parsing without explicitly checking the type.

A helper fromwire_peektype() is added to look up the type and handle
the too-short-for-type problem.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell 3e044fdd62 generate-wire.py: allow comments.
We emit them into the generated code at appropriate points, but it
would be better if we simply preserved the order they were given in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:21 +10:30
Rusty Russell bf7fefdc32 generate-wire.py: add memcheck() to towire functions.
Sanity check that we're not streaming uninitialized bytes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30
Rusty Russell c864b28068 generate-wire.py: don't generate structures, hand in all values.
This is a bit more awkward for large structures, but avoids
indirection for the simpler ones (I copied the structures for the test
code, however).  We also remove explicit padding.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30
Rusty Russell 326a9c9477 tools/generate-wire: node_announcement.alias is not an sha256
It's 32 bytes long, but it's just a string.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30
Rusty Russell a08a2105ea generate-wire.py: generalize, move to tools.
We're going to want to use this for inter-daemon comms, so generalize it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-04 14:09:20 +10:30