Commit Graph

8978 Commits

Author SHA1 Message Date
Christian Decker 2740f2eccb pyln: Use pyln-client Makefile for checks
Kinda uses an antipattern of `cd`ing into the directory and calling `make`
there, but this keeps the contrib dirs self-contained so we can split them out
eventually.
2020-09-28 09:19:46 +09:30
Christian Decker 1d2c2b6aaf pyln: Parametrize and unify Makefiles for pyln package 2020-09-28 09:19:46 +09:30
Michael Schmoock 71381eb5be chore: reduce log buffer from 100MB to 10MB
The 100MB log buffer has been the biggest memory footprint for the daemon.
Keeping 10MB for emergency log dumps seems sufficient.
This has been mentioned in the last developer meeting.

Changelog-Changed: In-memory log buffer reduced from 100MB to 10MB
2020-09-26 10:38:44 +09:30
Christian Decker 1afa8b88c9 travis: Re-enable the postgres configuration again 2020-09-26 09:04:35 +09:30
Christian Decker 5809cc41f4 pytest: Reduce the db_provider fixture scope to function
For performance reasons we were starting one for each session, which caused
the same postgres DB to be re-used for multiple tests (all test run in the
same worker process), but this could lead to interactions if there is a
timeout or a test happens to touch the `db_provider`. It turns out that we
were only saving about 15 seconds on a 1250 second run anyway, which is a
small cost for increased test isolation.
2020-09-26 09:04:35 +09:30
Christian Decker 7e99a3b5fd pyln: Allow the base test directory to contain some files
We were not removing the base test directory if we had other files in there,
which was the case for postgres runs. This now explicitly check for `test_*`
directories which are an indicator of a failed test.
2020-09-26 09:04:35 +09:30
Christian Decker aeb5f969e2 pyln: Start each postgres DB in its own sub-directory
We had a couple of issues with workers dying and attempting to re-initialize
the database while it was already initialized. This will look for a free
directory and just start the DB in there, allowing workers to be better
isolated.
2020-09-26 09:04:35 +09:30
Christian Decker 85dd19c9ac travis: Reduce reruns to 2 and early failure on test failure
While continuing to test all tests is good, it also means we potentially wait
much longer to get the final verdict, and it can truncate logs if there are
too many failures.
2020-09-26 09:04:35 +09:30
Jan Sarenik a777d21fb7 pytest: Rename all fund_channel to fundchannel 2020-09-25 21:04:56 +02:00
Rusty Russell 7260d9ea3d plugins: generate list of plugins more atomically, respect V=1 and --quiet.
I got a corrupt file, which looked like multiple concurrent attempts
to build it.  So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).

Also move into plugins/ where it logically belongs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 13:19:59 +02:00
Rusty Russell 488b32b003 build: run update-mocks.
Some declarations are redundant now.  Removing them does nothing, but
it makes other PRs cleaner.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 13:19:59 +02:00
Antoine Poinsot 3382dafaee configure: allow to run clang with ASAN
Prefer adding LSAN_OPTIONS="suppressions=$PWD/tests/sanitizer_suppressions/lsan" when CC=clang instead.

Changelog-Added: build: clang build now supports --enable-address-sanitizer .
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-24 13:57:43 +09:30
Christian Decker 5a87e88ee5 pytest: Add test for failcode conversion
We added a conversion of failcodes that do not have sufficient information in
faac4b28ad. That means that a failcode that'd require additional information
in order to be a correct error to return in an onion is mapped to a generic
one since we can't backfill the information.

This tests that the mapping is performed correctly and replicates the
situation in #4070
2020-09-24 11:23:33 +09:30
Michael Schmoock 980a9517c6 fix: openchannel_hook log BROKEN on dup close_to
Changelog-None
2020-09-24 11:22:47 +09:30
Christian Decker a351b9bf2f pyln: Migrate from binascii.hexlify to bytes.hex
Suggested-by: Lisa <@niftynei>
2020-09-24 11:14:22 +09:30
Christian Decker 04462f6a64 pyln: Add code to unwrap an encrypted onion at the intended node
Changelog-Added: pyln-proto: Added pure python implementation of the sphinx onion creation and processing functionality.
2020-09-24 11:14:22 +09:30
Christian Decker e8dcd59b24 pyln: Add a warning that pyln-proto is not safe for production use 2020-09-24 11:14:22 +09:30
Christian Decker 96b182a084 pyln: Implement sphinx onion packet generation in python
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2020-09-24 11:14:22 +09:30
Jan Sarenik 0b5e6c5be1 nit: doc/lightningd-config.5: Remove redundancy 2020-09-24 11:06:32 +09:30
Jan Sarenik b1355bf61e Update comment regarding signet using tb now 2020-09-24 09:24:14 +09:30
Dr. Maxim Orlovsky 0b01a5d7c4 Updating signet chainparams for the latest PoW fix
Changelog-Fixed: Protocol: `signet` is now compatible with the final bitcoin-core version
2020-09-24 09:24:14 +09:30
Rusty Russell 924cc04bd2 bolt11: have caller supply preferred chain.
This lets us distinguish testnet from signet invoices, since they
have the same prefix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 09:24:14 +09:30
Rusty Russell 1356700e2b bitcoin/chainparams: signet uses the same P2PKH and P2SH version bytes as testnet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 09:24:14 +09:30
Rusty Russell d5cb0d85b5 utils: use a cleaner pattern to capture wally allocations.
We force use of tal_wally_start/tal_wally_end around every wally
allocation, and with "end" make the caller choose where to reparent
everything.

This is particularly powerful where we allocate a tx or a psbt: we
want that tx or psbt to be the parent of the other allocations, so
this way we can reparent the tx or psbt, then reparent everything
else onto it.

Implementing psbt_finalize (which uses a behavior flag antipattern)
was tricky, so I ended up splitting that into 'psbt_finalize' and
'psbt_final_tx', which I think also makes the callers clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:45:53 +02:00
Rusty Russell 66ffd7a73f bitcoin/: neaten use of libwally base58 and bas64 routines.
Now libwally uses tal, we don't need to make copies.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:45:53 +02:00
Rusty Russell 4e67b4fd03 Makefile: work with shasum -a as well as sha256sum.
At least on my Ubuntu box, they're compatible. If they're not, we need
to disable regeneration altogether.

Fixes: #4075
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:34:42 +02:00
Christian Decker d9ba39d2cb pytest: Add test for key-value style postgresql DSNs 2020-09-23 20:20:40 +09:30
Christian Decker 1d9e7cf079 db: Add support for key-value pair DSNs in postgresql
These are simple space-separated key-value pair sets of options instead of the
URI style DSNs, but they are also much more flexible allowing the user to
specify client SSL certificates, server certificates, compression and
encryption levels, and much more (see [1] for more information)

[1]: https://www.postgresql.org/docs/9.1/libpq-connect.html

Changelog-Added: db: Added support for key-value DSNs for postgresql, allowing for a wider variety of configurations and environments.
2020-09-23 20:20:40 +09:30
Antoine Poinsot 15adcc915f Remove varint typedef for bigsize
It's not part of the spec anymore

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-23 16:30:36 +09:30
Rusty Russell 7db8680530 lightningd: don't hand around partially-assembled channel_info.
wallet_commit_channel would fill in the old_remote_per_commit and
fee_states, which is weird since the caller doesn't care.

Make the caller set all the channel_info fields, so wallet_commit_channel
is a simple consumer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 15:51:13 +09:30
Rusty Russell 58f6e316cc lightningd: separate fee_states out of struct channel_info.
It was always kind of weird in there anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 15:51:13 +09:30
niftynei cf2a807d3c dual_open: don't release uc here, it gets freed elsewhere 2020-09-23 15:51:13 +09:30
niftynei db2b013990 dual-fund: call daemon_shutdown only
daemon_shutdown calls common_shutdown internally; calling both frees the
same context twice
2020-09-23 15:51:13 +09:30
niftynei 56b3711bbc dual_open: alignment nit 2020-09-23 15:51:13 +09:30
niftynei ed9f661e07 dual_open: memleak problem with fee_states 2020-09-23 15:51:13 +09:30
niftynei 7d836a5945 dual_open_control: nit, reword comment 2020-09-23 15:51:13 +09:30
niftynei aa6420cb58 dual_open: split logic for deserializing a changed vs signed PSBT
We need to run different checks on these, so split them out for finer
grained control
2020-09-23 15:51:13 +09:30
niftynei 1f165c00ae psbt_txid: it's possible a psbt may already have the finalized scriptsig
If we've already got a scriptSig field filled out for a PSBT input, we
use that instead of 'deriving' the scriptSig from the redeemscript
(finalizing a PSBT removes the redeemscript field)
2020-09-23 15:51:13 +09:30
niftynei c5f748e716 dualopend: tiny billboard capitalization change 2020-09-23 15:51:13 +09:30
niftynei 5f6a2c05bd dualopend: check that we're not exceeding max allowable capacity 2020-09-23 15:51:13 +09:30
niftynei 8761468f42 dualopend: use status failed instead of peer failed 2020-09-23 15:51:13 +09:30
Christian Decker 668debb74f pyln: Rename LightningNode.fund_channel to fundchannel
It is inconsistent with the RPC method, which was bothering me for some time.
2020-09-23 14:45:12 +09:30
Christian Decker 8ecb157af6 pyln: Add type annotations to lightning.py 2020-09-23 14:45:12 +09:30
Christian Decker 49ec800a07 pyln: Add type-annotations to plugin.py
This should help users that have type-checking enabled.
2020-09-23 14:45:12 +09:30
Christian Decker d27da4d152 pyln: Fix two formatting string not matching parameters 2020-09-23 14:45:12 +09:30
Christian Decker 748caf91d3 pyln-client: Add support to monkey patch the JSONEncoder
Several times we had issues with plugins not being able to re-encode an RPC
result because they forgot to use the custom encoder class. This allows us to
patch the JSONEncoder when we start the RPC or the plugin and automagically
support classes that provide a `to_json` method.
2020-09-23 14:45:12 +09:30
Christian Decker 382230509b pyln: Add stubs to generate documentation for pyln-client 2020-09-23 14:45:12 +09:30
Rusty Russell 75ce380e0e Travis: disable postgres test.
I don't have time to diagnose why it's failing all the time right now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:03:03 +09:30
Christian Decker 1cb527d46a topo: Do not keep txids in memory indefinitely
I mistakenly assumed the block would be freed after processing completed. That
is not true since chaintopology keeps headers and stubs around for reorgs. So
we need to remove the precomputed txids along with the full_txs.
2020-09-23 13:52:49 +09:30
Rusty Russell f37f2b6193 common/memleak: simplify and document API.
1. Rename memleak_enter_allocations to memleak_find_allocations.
2. Unify scanning for pointers into memleak_remove_region / memleak_remove_pointer.
3. Document the functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 13:52:49 +09:30