Simplifying some operations, erroring in some cases and moving to global
defines for constants.
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
'bip70_name' is corresponding to the 'chain' field of
the API 'getblockchaininfo'.
At the beginning of lightningd, we use the 'chain' field of 'getblockchaininfo' to check if we are on right blockchain.
We check that memcmp *isn't* constant time, but that's only true under
-O2 or above: __OPTIMIZE__ doesn't distinguish.
So we need a finer-grained test. Also reduce verbosity by default.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The chainparams are needed to know the prefixes, so instead of passing down
the testnet, we pass the entire params struct.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We were deciding whether an address is a testnet address or not in the parser,
and then checking whether it matches our expectation outside as well. This
just returns the address version instead, and still checks it against our
expectation, but without having the parser need to know about address types.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is an intermediate step since the only difference between p2pkh and p2sh
is the argument that the parsing functions take, and parsing twice for that
reason alone is quite useless.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
* remove libbase58, use base58 from libwally
This removes libbase58 and uses libwally instead.
It allocates and then frees some memory, we may want to
add a function in wally that doesn't or override
wally_operations to use tal.
Signed-off-by: Lawrence Nahum lawrence@greenaddress.it
This fixes block parsing on testnet; specifically, non-standard tx versions.
We hit a type bug in libwally (wallt_get_secp_context()) which I had to
work around for the moment, and the updated libsecp adds an optional hash
function arg to the ECDH function.
Fixes: #2563
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Pubkeys are not not actually DER encoding, but Pieter Wuille corrected
me: it's SEC 1 documented encoding.
Results from 5 runs, min-max(mean +/- stddev):
store_load_msec,vsz_kb,store_rewrite_sec,listnodes_sec,listchannels_sec,routing_sec,peer_write_all_sec
38922-39297(39180.6+/-1.3e+02),2880728,41.040000-41.160000(41.106+/-0.05),2.270000-2.530000(2.338+/-0.097),44.570000-53.980000(49.696+/-3),32.840000-33.080000(32.95+/-0.095),43.060000-44.950000(43.696+/-0.72)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is what all of this has been working towards: ripping out the handwoven
transaction handling. By removing the custom parsing we can finally switch
over to using `wally_tx` as sole representation of transactions in
memory. The commit is a bit larger but it's mostly removing setters and old
references to the input and output fields.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
These are handled internally in the `wally_tx` and do not conform to our usual
tallocated strings that can by inspected using `tal_bytelen`, and we don't
really want to litter our code with whitelisting comments for the
`amount_sat.satoshis` access, so these just do read-only on the fly conversions.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The `wally_tx_input`s do not keep track of their input value, which means we
need to track them ourselves if we try to sign these transactions at a later
point in time.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
These are used when grinding the feerate and signing. These are just simple
facades that keep both wally and old style transactions in sync.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
During the migration to `libwally` we want to make absolutely sure that both
transactions are generated identical, and can eventually be switched over.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We are slowly migrating towards a wally-transactions only world, but to make
this reviewable we start building both old and new style transactions in
parallel. In a second pass we'll then start removing the old ones and use
libwally only.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We set the version BIP32_VER_TEST_PRIVATE for testnet/regtest
BIP32 privkey generation with libwally-core, and set
BIP32_VER_MAIN_PRIVATE for mainnet.
For litecoin, we also set it like bitcoin else.
We need to do it in various places, but we shouldn't do it lightly:
the primitives are there to help us get overflow handling correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to still accept it when parsing the database, but this flag
should allow upgrade testing for devs building on top
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a bug 0ba547ee10 caused by
short_channel_id overflow. If we'd caught this, we'd have terminated
the peer instead of crashing, so add appropriate checks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I upgraded my node with --disable-compat, and a heap of channels closed like:
CHANNELD_NORMAL:We disagree on short_channel_ids: I have 557653x0x1351, you say 557653x2373x1",
This is because the scids are strings in the databases, and it failed to parse
them properly.
Now we'll not start if that happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently only used by gossipd for channel elimination.
Also print them in canonical form (/[01]), so tests need to be
changed.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is prep work for when we sign htlc txs with
SIGHASH_SINGLE|SIGHASH_ANYONECANPAY.
We still deal with raw signatures for the htlc txs at the moment, since
we send them like that across the wire, and changing that was simply too
painful (for the moment?).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently make sure that all the bitcoin_tx input scripts are NULL
and set the input script of the input we're signing, so we can easily
reuse the tx hashing code for signature checks. This means that we
sometimes jump through hoops to make sure input scripts are NULL, and
also means that the tx can't be const.
Put more logic inside bitcoin/tx so it can simply ignore things we
don't want to hash.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To be safe, we should never memcmp secrets. We don't do this
currently outside tests, but we're about to.
The tests to prove this as constant time are the tricky bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
tal_count() is used where there's a type, even if it's char or u8, and
tal_bytelen() is going to replace tal_len() for clarity: it's only needed
where a pointer is void.
We shim tal_bytelen() for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Really, we should have a 'struct point' since we don't use all points
as pubkeys. But this is the minimal fix to avoid type cast nastiness.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
structeq() is too dangerous: if a structure has padding, it can fail
silently.
The new ccan/structeq instead provides a macro to define foo_eq(),
which does the right thing in case of padding (which none of our
structures currently have anyway).
Upgrade ccan, and use it everywhere. Except run-peer-wire.c, which
is only testing code and can use raw memcmp(): valgrind will tell us
if padding exists.
Interestingly, we still declared short_channel_id_eq, even though
we didn't define it any more!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I leave all the now-unnecessary accessors in place to avoid churn, but
the use of bitfields has been more pain than help.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With fallback depending on chainparams: this means the first upgrade
will be slow, but after that it'll be fast.
Fixes: #990
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When a serialized length refers to an array of structures, the trivial
DOS prevention can be out by a factor of sizeof(serialized struct). Use
the size of the serialized structure as a multiplier to prevent this.
Transaction inputs are the motivating example, where the check is out by
a factor of ~40.
If no witnesses are present on any inputs, then extended serialisation
should not be used.
[ Amended to make adding new flags clearer in future -- RR ]
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* Add BITCOIN_TEST_PROGRAMS to ALL_TEST_PROGRAMS
* Refactor bitcoin test make directives into its own Makefile under bitcoin/test
Signed-off-by: William Casarin <jb55@jb55.com>
The deserialization of bitcoin transactions in wire/ is rather
annoying in that we first allocate a new bitcoin_tx, then copy it's
contents onto the destination and then still carry the newly allocated
one around due to the tal-tree. This splits `pull_bitcoin_tx` into
two: one part that does the allocation and another one that proceeds
to parse.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_blkid_to_hex() so it's reversed as people expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_txid_to_hex() so it's reversed as people expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Google lead me to a discussion about litecint, it suggested they would use
'ltc' and I don't really care.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For non-delayed HTLC success spends, we have a similar pattern ("<sig>
<preimage> <wscript>") so a we want to use the same function.
The other routines don't say "witness" in them, and should.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You will want to 'make distclean' after this.
I also removed libsecp; we use the one in in libwally anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also, we split the more sophisticated json_add helpers to avoid pulling in
everything into lightning-cli, and unify the routines to print struct
short_channel_id (it's ':', not '/' too).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To avoid everything pulling in HTLCs stuff to the opening daemon, we
split the channel and commit_tx routines into initial_channel and
initial_commit_tx (no HTLC support) and move full HTLC supporting versions
into channeld.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The signing code asserts these are NULL, and if we unmarshal from the
wire then sign them, it gets upset.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`cli` and `cli_args` were not `const` before since they are added to a
non-`const` array. Using `cast_const` we can keep them `const` without
unsafe cast.
Reported-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We were using the bitcoin genesis blockhash for all networks, which is
not correct, and would result in the open being aborted when talking
to other implementations.
Reported-by: @sstone and @pm47
Signed-off-by: Christian Decker <decker.christian@gmail.com>
So far we always needed to know the public key, which was not the case
for addresses that we don't own. Moving the hashing outside of the
script construction allows us to send to arbitrary addresses. I also
added the hash computation to the pubkey primitives.
We alternated between using a sha256 and using a privkey, but there are
numerous places where we have a random 32 bytes which are neither.
This fixes many of them (plus, struct privkey is now defined in terms of
struct secret).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1) Need config.h before wire/gen_ are compiled.
2) The rule to checkout the libbase58 submodule doesn't work, so use the older
one-depends-on-the-other approach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
aka "BOLT 3: Use revocation key hash rather than revocation key",
which builds on top of lightningnetwork/lightning-rfc#105 "BOLT 2,3,5:
Make htlc outputs of the commitment tx spendable with revocation key".
This affects callers, since they now need to hand us the revocation
pubkey, but commit_tx has that already anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a bit tricky: for our signing code, we don't want scriptsigs,
but to calculate the txid, we need them. For most transactions in lightning,
they're pure segwit so it doesn't matter, but funding transactions can
have P2SH-wrapped P2WPKH inputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a step away from the previous more generic script types into
specific helpers for each transaction type we need.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a hack for 'struct rval' in protobuf_convert.h; make an
explicit header and put it in bitcoin/preimage.h. It's not really
bitcoin-specific, but it's better than having bitcoin/script depend on
an external header.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Encodings are signed: we may need 5 bytes to encode giant u32s.
Reported-by: Fabrice Drouin <fabrice.drouin@acinq.fr>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The object file should not be built inside the submodule, as that can
confuse git.
Not everything depends on the libbase58 header (CCAN doesn't), so
move that to the everything-else depends line.
The BITCOIN_SRC etc should also move to bitcoin/Makefile, but that's
a bigger change.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Added channel announcement serialization and parsing, as well as the
entrypoints for the IRC peer discovery. Announcements are signed by the
sending endpoint and signatures are verified before adding the channels
to the local view of the topology. We do not yet verify the existence of
the anchor transaction.
1. Fix #ifdef DEBUG code in signature.c so it compiles.
2. Don't set peer->closing.our_script in queue_pkt_close_shutdown: it's
assigned in caller already.
3. Wrap setting of htlc's rval in set_htlc_rval() function.
4. Log where we were when unexpected packet comes in.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And use this to resolve old transactions by comparing outputs with
HTLCs.
Rather than remembering the output ordering for every one of their
previous commitment transactions, we just remember the commitment
number for each commitment txid, and when we see it, derive all the
HTLC scriptpubkeys and the to-us and to-them scriptpubkeys, and figure
out which is which.
This avoids us having to save information on disk, except for the
txid->commitment-number mapping (and the shachain).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
From doing a code walkthrough with Christian Decker; unnecessary const in
bitcoin/tx.c, an erroneous FIXME, a missing comment, and an unused struct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Update libsecp256k1 has a normalize function, which allows us to test
if the signature was in low-S form.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use libsecp256k1 to convert signatures to DER; we were creating a
temporary one, but we really should be handing the one we have in dstate
through. This does that, everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now need to use bitcoin_witness_htlc with the r value, so that API
is updated to take 'struct rval' or 'struct sha256'.
We use the nc->delay amount (ie. dstate->config.min_htlc_expiry) to
wait for a timeout refund to be buried before "failing" upstream.
This should probably be made into a clearer parameter rather than
overloading this one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'dont-use-peer-nc-in-onchain-code.patch':
peer: Don't use peer->nc->delay for onchain case.
Use the config var directly. We should be freeing peer->nc when the
connection dies anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to enforce this onchain as we do in the protocol off-chain,
otherwise we can have an onchain redemption we can't redeem upstream
via the protocol. While Laolu points out there's a 520 byte limit on
witness stack element, that can still make for a larger tx and make
problems for the steal tx case.
The downside is that even the timeout transaction, which used to spend
the HTLC with an empty 'secret', now needs a 32-byte secret, making it
a little larger. We create a 'bitcoin_witness_htlc' helper for this
case.
See: http://lists.linuxfoundation.org/pipermail/lightning-dev/2016-May/000529.html
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They could be scriptpubkeys, but they're actually used inside p2wsh,
so they're really witness scripts. We use the term "redeem" elsewhere
from when we were using p2sh, though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
BIP141 indicates that the rule for block size has changed: witness
bytes effectively count for 1, and non-witness bytes count for 4, but
the maximum total has increased to 4,000,000.
This means that fee estimates should use the witness cost (divided by
4), not the raw txlen.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Like txids, we need to reverse them. We didn't, but then we only used them
to pass to/from bitcoind. We're about to get them from the block header,
so we need to fix that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want this because P2SH is something we can tell bitcoind to pay to;
we can't (yet?) do that with "raw" P2WPKH.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently linearize and then measure the string; this is better since
we're about to do it in a second place.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
sign_tx_input() now takes a witness_script arg: P2WPKH doesn't really
have a witness_script, but for signing it behaves as if it does.
This helper constructs that "fake" witness_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need this for signing segwitness txs. Unfortunately, we don't have it
for transactions we received as hex, only ones we created; to make this safe
we use a pointer which is NULL if we don't know, and those will crash if
we try to sign or check their sigs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This could only happen via our RPC interface (bitcoind should not give
us bad txs!) but it's better to be robust.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We got the -> second translation wrong by a factor of 512, and also we
need to move the median time in our tests otherwise bitcoind won't let
us spend the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without Alpha, it's superfluous. We're about to add segwit support,
but linearization requires a more powerful approach, and segwit
signature checking is completely different and really deserves its
own function.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I had already disabled it, and this clears the decks for Segregated Witness
which gives us everything we want.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't want to re-create them internally, ever.
The test-cli tools are patched to generate them all the time, but
they're not performance critical.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
All the members of the transaction should be allocated off the
transaction, as they have the same lifetime.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I got confused navigating these, especially since Alpha and Bitcoin
have diverged (BIP68 was proposed after Elements Alpha).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The format for both the nSequence field and the stack arg for
OP_CHECKSEQUENCEVERIFY is either:
Time-relative: [Bit 22 = 1] 00000 <time-shifted-by-9>
Block-relative: [Bit 22 = 0] 00000 <number of blocks>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happens for CSV, for example (3-byte encoding), and bitcoind treats
too-long encodings as non-standard.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The libsecp change broke signature checking. Disable it for now,
with a big FIXME. The next version should have a method for S value
checking, and also compact serialization.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The latest version of the BIP doesn't use inversion, but does use
bitshifts.
It also uncovered a bug in the test scripts: the block timestamps
creep forward when we generate large numbers of blocks (UpdateTime
insists it be > GetMedianTimePast() so it's valid). We need to take
this into account when waiting for the median to move (reduced it from
60 to 30 seconds, since that adds about 14 seconds).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As documented in the paper; it's also two bytes shorter, and allows
us to use the exact same script for three cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It doesn't matter until we start setting sequence numbers properly,
so hasn't been noticed until now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The even-S check was based on https://github.com/sipa/bitcoin/commit/a81cd9680
which was replaced by a low-S check in commit e0e14e43d9586409e42919f6cb955540134cda2a
Abstract out and fix the check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise valgrind tells you when you test a hash; you want to
know if you hash uninitialized memory long before that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's not clear it's needed, and without it there's a good reason to
delay dumping to the blockchain if a node becomes unreachable (since
you'll get your money faster if it comes back online).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Christian Decker
Karl-Johan Alm
lisa neigut
trueptolemy
Rusty Russell
GreenAddress
Saibato
practicalswift
nicolas.dorier
Luca Vaccaro
Jon Griffiths
William Casarin
ZmnSCPxj
Lucas Betschart