Replace `json_to_double()` (which uses `strtod(3)`) with our own
floating-point parsing function `json_to_millionths()` that
specifically expects to receive such a number that can fit in a
64 bit integer after being multiplied by 1 million.
The main piece of the code in this patch comes from
https://github.com/ElementsProject/lightning/pull/3535#discussion_r381041419
Changelog-None
Before this patch we used to send `double`s over the wire by just
copying them. This is not portable because the internal represenation
of a `double` is implementation specific.
Instead of this, multiply any floating-point numbers that come from
the outside (e.g. JSONs) by 1 million and round them to integers when
handling them.
* Introduce a new param_millionths() that expects a floating-point
number and returns it multipled by 1000000 as an integer.
* Replace param_double() and param_percent() with param_millionths()
* Previously the riskfactor would be allowed to be negative, which must
have been unintentional. This patch changes that to require a
non-negative number.
Changelog-None
Otherwise you can ask for a sub-millisatoshi amount, which is dumb and
violates the spec.
See-also: https://github.com/lightningnetwork/lightning-rfc/pull/736
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: We now reject invoices which ask for sub-millisatoshi amounts
Now that we have json_stream in common/, we can move all the related
helpers from lightningd/json to common/json. This way everyone can
benefit of them (including libplugin, the plugins themselves,
potentially lightning-cli), not lightningd alone!
Note that the Makefile of the common/test/ had to be modified, because
the new helpers make use of common/wireaddr... Which turns out to
\#include <lightingd/lightningd.h> ! So we couldnt just include the .c
and add mocks if we redefined some structs (hello run-param).
GCC 10 defaults to `-fno-common`. no longer automatically sharing
global variable definitions, which makes it important to define
them in only one place (otherwise there will be duplicate definition
errors). Add `extern` qualifiers where (I think) is the best place for
them.
We also update since the merged version sets feature bit 9 (as it's
supposed to now that we tied that to payment_secret).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before this patch we used `int` for error codes. The problem with
`int` is that we try to pass it to/from wire and the size of `int` is
not defined by the standard. So a sender with 4-byte `int` would write
4 bytes to the wire and a receiver with 2-byte `int` (for example) would
read just 2 bytes from the wire.
To resolve this:
* Introduce an error code type with a known size:
`typedef s32 errcode_t`.
* Change all error code macros to constants of type `errcode_t`.
Constants also play better with gdb - it would visualize the name of
the constant instead of the numeric value.
* Change all functions that take error codes to take the new type
`errcode_t` instead of `int`.
* Introduce towire / fromwire functions to send / receive the newly added
type `errcode_t` and use it instead of `towire_int()`.
In addition:
* Remove the now unneeded `towire_int()`.
* Replace a hardcoded error code `-2` with a new constant
`INVOICE_EXPIRED_DURING_WAIT` (903).
Changelog-Changed: The waitinvoice command would now return error code 903 to designate that the invoice expired during wait, instead of the previous -2
This makes it clear we're dealing with a message which is a wrapped error
reply (needing unwrap_onionreply), not an already-wrapped one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now "raw_payload" is always the complete string (including realm or length
bytes at the front).
This has several effects:
1. We can receive an decrypt an onion which is grossly malformed.
2. We can still hand this to the htlc_accepted hook.
3. We then fail it unless the htlc_accepted accepts it manually.
4. The createonion API now takes the raw payload, and does not know
anything about "style".
The only caveat is that the sphinx code needs to know the payload
length: we have a call for that, which simply tells it to copy the
entire onion (and treat us as the final node) if it's invalid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The spec is (RSN!) going to explicitly denote where each feature should
be presented, so create that infrastructure.
Incorporate the new proposed bolt11 features, which need this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was decided at a recent spec meeting: in particular, mpp and
var_onion_optin options will be used here.
We enhanced "features_supported" into "features_unsupported" so it
can return the first un-handlable bit number.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also pulls in a new onion error (mpp_timeout). We change our
route_step_decode_end() to always return the total_msat and optional
secret.
We check total_amount (to prohibit mpp), but we do nothing with
secret for now other than hand it to the htlc_accepted hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We wire in the code-generated function, which removes the upfront validation
and add the validation back after the `htlc_accepted` hook returns. If a
plugin wanted to handle the onion in a special way it'll not have told us to
just continue.
This is ignored in subdaemons which are per-peer, but very useful for
multi-peer daemons like connectd and gossipd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is mainly an internal-only change, especially since we don't
offer any globalfeatures.
However, LND (as of next release) will offer global features, and also
expect option_static_remotekey to be a *global* feature. So we send
our (merged) feature bitset as both global and local in init, and fold
those bitsets together when we get an init msg.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We often want a pointer which will turn to NULL if the pointed-to thing is
freed. This is possible with tal objects, so create it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This patch adds a channel_id parameter to allow for specifying
channels that are lacking a short_channel_id.
Useful in the case where a peer has 1) multiple channels (ONCHAIN etc)
and 2) a channel where the funding transaction hasn't been
broadcast/mined.
Currently the only source for amount_asset is the value getter on a tx output,
and we don't hand it too far around (mainly ignoring it if it isn't the
chain's main currency). Eventually we could bubble them up to the wallet, use
them to select outputs or actually support assets in the channels.
Since we don't hand them around too widely I thought it was ok for them to be
pass-by-value rather than having to allocate them and pass them around by
reference. They're just 41 bytes currently so the overhead should be ok.
Signed-off-by: Christian Decker <@cdecker>
The way we build transactions, serialize them, and compute fees depends on the
chain we are working on, so let's add some context to the transactions.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The `runtest` command takes a JSON onion spec, creates the onion and decodes
it with the provided private keys. It is fully configurable and can be used
for the test-vectors in the spec.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is just taking the existing serialization code and repackaging it in a
more useful form.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It assumes the head of the array is the object/array we want to remove from,
but that's not true if we're trying to remove from a sub-object.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are generalized from our internal implementations.
The main difference is that 'struct json_escaped' is now 'struct
json_escape', so we replace that immediately.
The difference between lightningd's json-writing ringbuffer and the
more generic ccan/json_out is that the latter has a better API and
handles escaping transparently if something slips through (though
it does offer direct accessors so you can mess things up yourself!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This fixes block parsing on testnet; specifically, non-standard tx versions.
We hit a type bug in libwally (wallt_get_secp_context()) which I had to
work around for the moment, and the updated libsecp adds an optional hash
function arg to the ECDH function.
Fixes: #2563
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I tried to just do gossipd, but it was uncontainable, so this ended up being
a complete sweep.
We didn't get much space saving in gossipd, even though we should save
24 bytes per node.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Node ids are pubkeys, but we only use them as pubkeys for routing and checking
gossip messages. So we're packing and unpacking them constantly, and wasting
some space and time.
This introduces a new type, explicitly the SEC1 compressed encoding
(33 bytes). We ensure its validity when we load from the db, or get it
from JSON. We still use 'struct pubkey' for peer messages, which checks
validity.
Results from 5 runs, min-max(mean +/- stddev):
store_load_msec,vsz_kb,store_rewrite_sec,listnodes_sec,listchannels_sec,routing_sec,peer_write_all_sec
39475-39572(39518+/-36),2880732,41.150000-41.390000(41.298+/-0.085),2.260000-2.550000(2.336+/-0.11),44.390000-65.150000(58.648+/-7.5),32.740000-33.020000(32.89+/-0.093),44.130000-45.090000(44.566+/-0.32)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I tried to fundchannel 0.01btc, and of course it wanted 8 decimals exactly.
If I can't get this right, it's probably a bad idea.
I still don't allow whole number of btc though, since that's probably a mistake
and you're not supposed to put that much in c-lightning yet :)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As a side-effect of using amount_msat in gossipd/routing.c, we explicitly
handle overflows and don't need to pre-prune ridiculous-fee channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current param_sat accepts "any": rename and move that to invoice.c
where it's called. We rename it to param_msat_or_any and invoice.c
is our first (trivial) amount_msat user.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They're generally used pass-by-copy (unusual for C structs, but
convenient they're basically u64) and all possibly problematic
operations return WARN_UNUSED_RESULT bool to make you handle the
over/underflow cases.
The new #include in json.h means we bolt11.c sees the amount.h definition
of MSAT_PER_BTC, so delete its local version.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
jsmn would accept invalid JSON objects. This is bad because it would
set ->size incorrectly: we expect to have at least size * 2 tokens (in
pairs). We want to rely on ->size, but this would create an exploitable
buffer overflow!
Fortunately, this is fixed upstream, so we add a test and upgrade to v1.0.0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Wasn't using valid JSON, but worked anyway. This is actually OK
because we don't rely on tok->size, but we want to, so another fix
coming.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The external/jsmn/README.md only says:
int size; // Number of child (nested) tokens
But it only counts *direct* children, or *direct* members for an object.
This test verifies this (the bug proved to be elsewhere: see next patch!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is mainly just copying over the copy-editing from the
lightning-rfc repository.
[ Split to just perform changes after the UNKNOWN_PAYMENT_HASH change --RR ]
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Rusty Russell <@rustyrussell>
Handers of a specific form are both designed to be used as callbacks
for param(), and also dispose of the command if something goes wrong.
Make them return the 'struct command_result *' from command_failed(),
or NULL.
Renaming them just makes sense: json_tok_XXX is used for non-command-freeing
parsers too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These routines free the 'struct command': a common coding error is not
to return immediately.
To catch this, we make them return a non-NULL 'struct command_result
*', and we're going to make the command handlers return the same (to
encourage 'return command_fail(...)'-style usage).
We also provide two sources for external use:
1. command_param_failed() when param() fails.
2. command_its_complicated() for some complex cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was removed (as unused) in 6269a4c55d592e8720b7f2a304c21f61f7931238;
now I've even added tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
json_escaped.[ch], param.[ch] and jsonrpc_errors.h move from lightningd/
to common/. Tests moved too.
We add a new 'common/json_tok.[ch]' for the common parameter parsing
routines which a plugin might want, taking them out of
lightningd/json.c (which now only contains the lightningd-specific
ones).
The rest is mainly fixing up includes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
json_tok* is used with 'struct command', so rename this to match the other
low-level json tok helpers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is prep work for when we sign htlc txs with
SIGHASH_SINGLE|SIGHASH_ANYONECANPAY.
We still deal with raw signatures for the htlc txs at the moment, since
we send them like that across the wire, and changing that was simply too
painful (for the moment?).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We probably also want to call secp_randomise/wally_secp_randomize here
too, and since these calls all call setup_tmpctx, it probably makes
sense to have a helper function to do all that. Until thats done, I
modified the tests so grepping will show the places where the sequence
of calls is repeated.
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
It's the only user of them, and it's going to get optimized.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
gossip.pydiff --git a/common/test/run-json.c b/common/test/run-json.c
index 956fdda35..db52d6b01 100644
Totally forgot to add this test. It just shows how a writer can take
exclusive access of a socket over multiple `io_write` calls, and
queuing all others behind it.
It turns out we were heavily relying on the fact that after each message from
the client there'd be a flush, and that there would not be anything after the
JSON object we read. This will no longer be the case once we start streaming
things or we are very quick in issuing the JSON-RPC requests.
This just takes one of the error paths (incomplete read) and makes it into a
successful path if we have indeed read a full root element.
To be safe, we should never memcmp secrets. We don't do this
currently outside tests, but we're about to.
The tests to prove this as constant time are the tricky bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
tal_count() is used where there's a type, even if it's char or u8, and
tal_bytelen() is going to replace tal_len() for clarity: it's only needed
where a pointer is void.
We shim tal_bytelen() for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Often we only need a single secret, so it's clearer to have routines
to do just that. When we change to the lnd key scheme, there will be
no benefit in calculating them all together.
This also adds a test!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
structeq() is too dangerous: if a structure has padding, it can fail
silently.
The new ccan/structeq instead provides a macro to define foo_eq(),
which does the right thing in case of padding (which none of our
structures currently have anyway).
Upgrade ccan, and use it everywhere. Except run-peer-wire.c, which
is only testing code and can use raw memcmp(): valgrind will tell us
if padding exists.
Interestingly, we still declared short_channel_id_eq, even though
we didn't define it any more!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. We need to test all bits, not all bytes.
2. Both local and global features need to be supported.
3. Untested code is broken code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Risks leakage. We could do lookup via the proxy, but that's a TODO.
There's only one occurance of getaddrinfo (and no gethostbyname), so
we add a flag to the callers.
Note: the use of --always-use-proxy suppresses *all* DNS lookups, even
those from connect commands and the command line.
FIXME: An implicit setting of use_proxy_always is done in gossipd if it
determines that we are announcing nothing but Tor addresses, but that
does *not* suppress 'connect'.
This is fixed in a later patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And use it in wireaddr.
We fix up the double '.onion' in the test case, which seems like an error?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a rebased and combined patch for Tor support. It is extensively
reworked in the following patches, but the basis remains Saibato's work,
so it seemed fairest to begin with this.
Minor changes:
1. Use --announce-addr instead of --tor-external.
2. I also reverted some whitespace and unrelated changes from the patch.
3. Removed unnecessary ';' after } in functions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Someone could try to announce an internal address, and we might probe
it.
This breaks tests, so we add '--dev-allow-localhost' for our tests, so
we don't eliminate that one. Of course, now we need to skip some more
tests in non-developer mode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was something @icota implemented, but it fits logically into this
cleanup series. We create a new type which is the internal generalization
of a wireaddr (which is defined by the spec), and add a case here for
a socket name.
Based-on-the-true-story-by: @icota
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These were so far only used for bolt11 construction, but we'll need them for the
DNS seed as well, so here we just pull them out into their own unit and prefix
them.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Just a small cleanup of the indentation code, so we don't have to reformat all
the issue reports to become readable. This is much closer to what `jq` or
`json_pp` spit out and doesn't have those infinitely long lines.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We can have more than one; eg we might offer both bech32 and a p2sh
address, and in future we might offer v1 segwit, etc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I didn't convert all tests: they can still use a standalone context.
It's just marginally more efficient to share the libwally one for all
our daemons which link against it anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We save wireaddr to databases as a string (which is pretty dumb) but
it turned out that my local node saved '[::ffff:127.0.0.1]:49150'
which our parser can't parse.
Thus I've reworked the parser to make fewer assumptions:
parse_ip_port() is renamed to separate_address_and_port() and is now
far more accepting of different forms, and returns failure only on
grossly malformed strings. Otherwise it overwrites its *port arg only
if there's a port specified. I also made it static.
Then fromwire_wireaddr() hands the resulting address to inet_pton to
figure out if it's actually valid.
Cc: William Casarin <jb55@jb55.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Correctly format ipv6 address with ports. This will also make it more compatible
with the new parse_wireaddr, which has been updated to parse ports. They are
inverses now.
Also add some tests that check this.
Signed-off-by: William Casarin <jb55@jb55.com>
We don't use it yet, but now we'll decode correctly.
See: https://github.com/lightningnetwork/lightning-rfc/pull/317
lightning-rfc commit: ef053c09431442697ab46e83f9d3f86e3510a18e
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rusty Russell
Vasil Dimov
Christian Decker
darosior
Wladimir J. van der Laan
gorazdko
lisa neigut
Lawrence Nahum
Mark Beckwith
Jon Griffiths
practicalswift
Saibato
ZmnSCPxj
Björge Dijkstra
William Casarin