I went overboard on optimization. I am so sorry:
1. Squeezed channel min/max into 16 bits.
2. Uses mmap and leaves node_ids in the file.
3. Uses offsets instead of pointers where possible.
4. Uses custom free-list to allocate inside arrays.
5. Ignores our autogenerated marshalling code in favor of direct derefs.
6. Carefully aligns everything so we use minimal ram.
The result is that the current gossip_store:
- load time (-O3 -flto laptop): 40msec
- load time (-g laptop i.e. DEVELOPER=0): 60msec
- load time (-O0 laptop i.e. DEVELOPER=1): 110msec
- Total memory: 2.6MB:
- 1.5MB for the array of channels
- 512k for the channel htable to map scid -> channel.
- 320k for the node htable to map nodeid -> node.
- 192k for the array of channels inside each node
- 94k for the array of nodes
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The jsmn parser is a beautiful piece of code. In particular, you can parse
part of a string, then continue where you left off.
We don't take advantage of this, however, meaning for large JSON objects
we parse them multiple times before finally having enough to complete.
Expose the parser state and tokens through the API, so the caller can pass
them in repeatedly. For the moment, every caller is allocates each time
(except the unit tests).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to change the API on the more complete JSON parser, so
make and use a simple API for the easy cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's no stable ordering on unknown serialization, so linearizing
identical but mis-ordered unknown data will lead to 'wrong' results.
Instead, we just ignore any data that's in the psbt unknown struct.
There's probably also problems here with other PSBT maps. Really, this
needs a finer grained comparison function .... fuck
We don't preserve detailed asset information at the moment, so provide a
way to convert from a sat to an amount_asset struct.
We also need a way to convert from an 'amount_asset' to a 'value' for
elements, which for explicit (i.e. non-blinded) asssets is a 0x01 prefix
plus the big-endian encoded value.
Our existing coin_moves tracking logic assumed that any tx we had an
input in belonged to *all* of our wallet (not a bad assumption as long
as there was no way to update a tx that spends our wallets)
Now that we've got `signpsbt` implemented, however, we need to be
careful about how we account for withdrawals. For now we do a best guess
at what the feerate is, and lump all of our spent outputs as a
'withdrawal' when it's impossible to disambiguate
The main change here is that the previously-optional open/accept
fields and reestablish fields are now compulsory (everyone was
including them anyway). In fact, the open/accept is a TLV
because it was actually the same format.
For more details, see lightning-rfc/f068dd0d8dfa5ae75feedd99f269e23be4777381
Changelog-Removed: protocol: support for optioned form of reestablish messages now compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Update the `bitcoin_tx_add_input` interface to accept a witness script
and or scriptPubkey.
We save the amount + witness script + witness program (if known) to
the PSBT object for a transaction when creating an input.
It returns NULL, so you can simply `return fromwire_fail(...)`
if you want to return NULL in this case. Use that more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we now over-write the wally malloc/free functions, we need to do
so for tests as well. Here we pull up all of the common setup/teardown
logic into a separate place, and update the tests that use libwally to
use the new common_setup core
Changelog-None
We did this originally because these types are referred to in the bolts, and we
had no way of injecting the correct include lines into those. Now we do, so
there's less excuse for this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only use sizeof(f1->bits).
```
common/test/run-features.c:84:36: error: Uninitialized variable: f1 [uninitvar]
for (size_t i = 0; i < ARRAY_SIZE(f1->bits); i++) {
^
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
common/onion is going to need to use this for the case where it finds a blinding
seed inside the TLV. But how it does ecdh is daemon-specific.
We already had this problem for devtools/gossipwith, which supplied a
special hsm_do_ecdh(). This just makes it more general.
So we create a generic ecdh() interface, with a specific implementation
which subdaemons and lightningd can use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out that unnecessary: all callers can access the feature_set,
so make it much more like a normal primitive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is to prepare for dynamic features, including making plugins first
class citizens at setting them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a common thing to do, so create a macro.
Unfortunately, it still needs the type arg, because the paramter may
be const, and the return cannot be, and C doesn't have a general
"(-const)" cast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Replace `json_to_double()` (which uses `strtod(3)`) with our own
floating-point parsing function `json_to_millionths()` that
specifically expects to receive such a number that can fit in a
64 bit integer after being multiplied by 1 million.
The main piece of the code in this patch comes from
https://github.com/ElementsProject/lightning/pull/3535#discussion_r381041419
Changelog-None
Before this patch we used to send `double`s over the wire by just
copying them. This is not portable because the internal represenation
of a `double` is implementation specific.
Instead of this, multiply any floating-point numbers that come from
the outside (e.g. JSONs) by 1 million and round them to integers when
handling them.
* Introduce a new param_millionths() that expects a floating-point
number and returns it multipled by 1000000 as an integer.
* Replace param_double() and param_percent() with param_millionths()
* Previously the riskfactor would be allowed to be negative, which must
have been unintentional. This patch changes that to require a
non-negative number.
Changelog-None
Otherwise you can ask for a sub-millisatoshi amount, which is dumb and
violates the spec.
See-also: https://github.com/lightningnetwork/lightning-rfc/pull/736
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: We now reject invoices which ask for sub-millisatoshi amounts
Now that we have json_stream in common/, we can move all the related
helpers from lightningd/json to common/json. This way everyone can
benefit of them (including libplugin, the plugins themselves,
potentially lightning-cli), not lightningd alone!
Note that the Makefile of the common/test/ had to be modified, because
the new helpers make use of common/wireaddr... Which turns out to
\#include <lightingd/lightningd.h> ! So we couldnt just include the .c
and add mocks if we redefined some structs (hello run-param).
GCC 10 defaults to `-fno-common`. no longer automatically sharing
global variable definitions, which makes it important to define
them in only one place (otherwise there will be duplicate definition
errors). Add `extern` qualifiers where (I think) is the best place for
them.
We also update since the merged version sets feature bit 9 (as it's
supposed to now that we tied that to payment_secret).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rusty Russell
niftynei
ZmnSCPxj jxPCSnmZ
darosior
Christian Decker
Vasil Dimov
Wladimir J. van der Laan