Commit Graph

476 Commits

Author SHA1 Message Date
Christian Decker eacc54646f pyln: Remove deprecated txprepare variant
We promised to deprecate the old "destination satoshi feerate utxos"
variant some time on mid-2020 so let's do just that. `txprepare`
doesn't parse it either anymore.

Changelog-Changed: pyln: `txprepare` no longer supports the deprecated `destination satoshi feerate utxos` call format.
2020-12-09 06:56:21 +10:30
Michael Schmoock 362284981d plyn: use math.floor for msat mul and div 2020-12-08 13:48:14 +01:00
Michael Schmoock 565dc95a8e pyln: fix msat float multiplication 2020-12-08 13:48:14 +01:00
Michael Schmoock 2678cfb364 pyln: extend msat floating testcases
This adds two more xfail'ing testcases to show that the current way of
parsing Millisatoshi decimals is not yet optimal.

Changelog-None
2020-12-08 13:48:14 +01:00
Christian Decker 76124eb800 pyln: Replace undecodeable symbols when tailing logs
Logs may contain non-ASCII and non-UTF8 symbols, which crashes the
tailer. It's better to replace them with a glyph representing
undecodeable symbols instead, and handle the issue further up the
call-chain.
2020-12-04 18:34:12 -06:00
Christian Decker c9d24b1de2 pyln: Set the prefix in TailableProc
We rely on it when logging, but didn't set it in the class, rather we
did in the sub-classes. This proved to be rather annoying to search
since it'd just fail silently and not log any output.
2020-12-04 18:34:12 -06:00
Rusty Russell e4950db9a3 pytest: recreate wallet on bitcoind restart.
Doesn't seem to stick in master.  Andy Chow suggested we
simply turn off wallet on older versions, and always create/load.

```
[gw8] [ 40%] FAILED tests/test_misc.py::test_bitcoind_goes_backwards

============================================================= FAILURES ==============================================================
___________________________________________________ test_bitcoind_goes_backwards ____________________________________________________
[gw8] linux -- Python 3.8.5 /usr/bin/python3

node_factory = <pyln.testing.utils.NodeFactory object at 0x7f931859a760>
bitcoind = <pyln.testing.utils.BitcoinD object at 0x7f931865eee0>

    def test_bitcoind_goes_backwards(node_factory, bitcoind):
        """Check that we refuse to acknowledge bitcoind giving a shorter chain without explicit rescan"""
        l1 = node_factory.get_node(may_fail=True, allow_broken_log=True)

        bitcoind.generate_block(10)
        sync_blockheight(bitcoind, [l1])
        l1.stop()

        # Now shrink chain (invalidateblock leaves 'headers' field until restart)
        bitcoind.rpc.invalidateblock(bitcoind.rpc.getblockhash(105))
        # Restart without killing proxies
        bitcoind.rpc.stop()
        TailableProc.stop(bitcoind)
        bitcoind.start()

        # Will simply refuse to start.
        with pytest.raises(ValueError):
            l1.start()

        # Nor will it start with if we ask for a reindex of fewer blocks.
        l1.daemon.opts['rescan'] = 3

        with pytest.raises(ValueError):
            l1.start()

        # This will force it, however.
        l1.daemon.opts['rescan'] = -100
        l1.start()

        # Now mess with bitcoind at runtime.
>       bitcoind.generate_block(6)

tests/test_misc.py:1307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:399: in generate_block
    return self.rpc.generatetoaddress(numblocks, self.rpc.getnewaddress())
contrib/pyln-testing/pyln/testing/utils.py:322: in f
    return proxy._call(name, *args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bitcoin.rpc.RawProxy object at 0x7f93184f6a30>, service_name = 'getnewaddress', args = ()
postdata = '{"version": "1.1", "method": "getnewaddress", "params": [], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -18, 'message': 'No wallet is loaded. Load a wallet using loadwallet or create a new one with createwallet. (Note: A default wallet is no longer automatically created)'}, 'id': 1, 'result': None}
```
2020-12-03 11:35:30 +01:00
Michael Schmoock 7bfb5f10c7 pyln: failing test msat from float str
We were not able to create pyln Millisatoshi from floats, e.g.:
 - "0.01btc"
 - "0.1sat"
 - ...

This adds a test that makes sure this won't happen again.
2020-12-02 10:37:31 +10:30
Michael Schmoock 83a21138b8 pyln: fix msat from float str
Changelog-fixed: pyln: parsing msat from a float string
2020-12-02 10:37:31 +10:30
Michael Schmoock bdad9fcdd9 pyln: def amount 10**6 for fund- and openchannel
Changelog-None
2020-12-01 11:15:30 +10:30
Antoine Poinsot a561360334 pytest: create a wallet for bitcoind if none is present
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Rusty Russell fa006fd2f7 pyln-testing: require bitcoin v0.20 for PSBT handling, and create wallet for v0.21
With older bitcoind, PSBTs fail:

```
    def test_utxopsbt(node_factory, bitcoind, chainparams):
...
>       psbt = bitcoind.rpc.decodepsbt(funding['psbt'])

tests/test_wallet.py:561:
...
self = <bitcoin.rpc.RawProxy object at 0x7f4ec602e100>, service_name = 'decodepsbt'
args = ('cHNidP8BADMCAAAAAaoMihSVXlpdBHGcJePiroqtwq/b1zu09j8IkTG4OKs7AQAAAAD9////AGYAAAAAAQDeAgAAAAABAefqB6BkZE1/AqXaf36T02a7.../7Stf971PEgvUXgvASECXPTIO6tIVxDih6tfKy6suj6WJhhjycwoaTeuso/AQ8llAAAAAQEfQEIPAAAAAAAWABQB+tkKvNZml+JZIWRyLeSpXr7hZQA=',)
postdata = '{"version": "1.1", "method": "decodepsbt", "params": ["cHNidP8BADMCAAAAAaoMihSVXlpdBHGcJePiroqtwq/b1zu09j8IkTG4OKs7AQ...gvUXgvASECXPTIO6tIVxDih6tfKy6suj6WJhhjycwoaTeuso/AQ8llAAAAAQEfQEIPAAAAAAAWABQB+tkKvNZml+JZIWRyLeSpXr7hZQA="], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -22, 'message': 'TX decode failed PSBT is not sane.: iostream error'}, 'id': 1, 'result': None}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: *** Requires bitcoind v0.20.1 or above ***
2020-11-09 20:05:30 -06:00
Rusty Russell e16ed0e207 pyln: add support for dependent hooks.
And use that to add simple tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Christian Decker 62f7e646ec pyln: Bump cryptography dependency to 3.2 due to upstream bug
https://cryptography.io/en/latest/changelog.html

Changelog-None
2020-10-29 11:11:48 +10:30
Christian Decker 9a0327cd25 pyln-proto: Add compactsize alias for varint_{encode,decode}
We were mistakenly calling it varint, while Bitcoin refers to it as
CompactSize.
2020-10-29 11:11:48 +10:30
Michael Schmoock 6907e85a39 chore: optimize postgres binary lookup
This will change the current hardwired PostgreSQL binary lookup mechanism to
utilizing the `pg_config --bindir` utility that should work better for most
distributions. The old method caused issues at least on Archlinux.

Changelog-None
2020-10-27 10:40:52 +10:30
Rusty Russell 806f208295 pyln: handle (ignore) notifications, and add notify_msg to send them.
We also sanity check that response id matches our request.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: pyln.client handles and can send progress notifications.
2020-10-23 13:53:16 +10:30
Sergi Delgado Segura 9de5f438c4 pyln: Adds type annotations to zbase32 functions 2020-10-22 13:56:57 +02:00
Sergi Delgado Segura 3779c0b6ee pyln: Moves old zbase32 test from primitives 2020-10-22 13:56:57 +02:00
Sergi Delgado Segura f497b90ee8 pyln: Adds tests to zbase32 2020-10-22 13:56:57 +02:00
Sergi Delgado Segura 26f651f71f pyln: Adds one-liner docs to zbase32 functions 2020-10-22 13:56:57 +02:00
Sergi Delgado Segura 1da29305fc pyln: Improve zbase32 encoding / decoding
- Adds bitarray filling so mesages of any length can be encoded, instead of forcing the encoding to be of messages with length multiple of 5.
- Adds checks for encoding / decoding and raises expections if the inputs are not as expected.
- Flags functions that are supposed to be internal as "private".
2020-10-22 13:56:57 +02:00
Antoine Poinsot 31e50c9c25 contrib: remove ASAN suppressions
My bad, i comitted them as part of the LSAN suppressions while this data
race could have and had been fixed.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-10-21 19:34:39 +02:00
niftynei 8858ae4f3d df-open: commands to update a PSBT or submit a signed PSBT
`openchannel_signed` and `openchannel_update` which allow a user to
continue a openchannel or kick off the completion of a openchannel.

`openchannel_update` should be called until it returns with
`commitments_secured`.
2020-10-20 12:50:31 +10:30
niftynei de34f08b82 tests,fundchannel: return the result from the fundchannel rpc call
We need this so we can verify the 'close_to' result
2020-10-16 13:51:57 +10:30
niftynei ee329f08de test, fundchannel: pass kwargs down to fundchannel command
Allows us to more easily pass through args to `fundchannel` while still
using the utility function's funding + open confirmation logics
2020-10-16 13:51:57 +10:30
niftynei 4926c25bb5 fundchannel: take a 'close_to' address
Finally, extends the 'close_to' functionality up to the flagship 'open a
channel' command.

Changelog-Added: JSON-API `fundchannel` now accepts an optional 'close_to' param, a bitcoin address that the channel funding should be sent to on close. Requires `opt_upfront_shutdownscript`
2020-10-16 13:51:57 +10:30
Jan Sarenik 90acf5072d pyln: utils:BitcoinD: add -wallet="test" option
Per https://github.com/bitcoin/bitcoin/pull/15454
2020-10-13 20:54:33 +02:00
Christian Decker b1aed933e6 pyln: Plugin methods and hooks refuse to set results twice
We had a couple of instances where a plugin would be killed by `lightningd`
because we were returning a result of an exception twice, and it was hard to
trace down the logic error in the user plugin that caused that. This patch
adds a traceback the first time we return a result/exception, and raise an
exception with a stacktrace of the first termination when a second one comes
in.

This can still terminate the plugin, but the programmer gets a clear
indication where the result was set, and can potentially even recover from it.

Changelog-Added: pyln: Plugin method and hook requests prevent the plugin developer from accidentally setting the result multiple times, and will raise an exception detailing where the result was first set.
2020-10-13 20:52:48 +02:00
Christian Decker 556725c5ff pyln: Add logging handler that passes records to lightningd
It is often pretty usefuk to use the builtin logging module to debug things,
including libraries that a plugin may use. This adds a simple
`PluginLogHandler` that maps the python logging levels to the `lightningd`
logging levels, and formats the record in a way that it doesn't clutter up the
`lightningd` logs (no duplicate timestamps and levels).

This allow us to tweak the log level that is reported to `lightningd` simply
using the following

```python3
import logging
logging.basicConfig(level=logging.DEBUG)
```

Notice that in order for the logs to be displayed on the terminal or the
logfile, both the logging level in the plugin _and_ the `--log-level`
`lightningd` is running need to be adjusted (the python logging level only
controls which messages get forwarded to `lightningd`, it does not have the
power to overrule `lightningd` about what to actually display).

I chose `logging.INFO` as the default, since libraries have a tendency to spew
out everything in `logging.DEBUG` mode

Changelog-Added: pyln: Plugins have been integrated with the `logging` module for easier debugging and error reporting.
2020-10-13 20:52:14 +02:00
Rusty Russell a297c1b9a5 pytest: fix overzealous removal of directories on failure.
We were always deleting the directories.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-13 17:33:49 +02:00
Christian Decker ae9753df3a travis: Specify pytest-rerunfailures==9.1 to avoid regression
Seems the recently released version 9.1.1 has regressed, and isn't actually
rerunning failed tests. Pinning it to 9.1 seems to work however.
2020-10-13 14:30:45 +02:00
Christian Decker d5d41ef1a0 pyln: Remove any logging handlers at teardown to avoid logging error
Inspired by https://github.com/pytest-dev/pytest/issues/5502#issuecomment-647157873
2020-10-13 14:30:45 +02:00
fiatjaf 9c838cf953 pyln-client: listpayments -> listpays 2020-10-09 13:21:35 +10:30
Christian Decker 6f870dfe39 pytest: Don't give up on the first psql connection error
Since we start a new instance of postgres for each test we may end up swamped
and the startup can take a bit longer. So let's loop until we get a success.
2020-10-07 09:36:30 +10:30
Christian Decker 9021bb26d1 pyln: Decode process output once before storing it 2020-10-07 09:36:30 +10:30
Christian Decker b6053d7cd6 Fix pytest major version
They released version 6, which is causing some trouble apparently. So fix it
to v5.
2020-10-07 09:36:30 +10:30
Christian Decker 66dc3ed665 pyln: Add pytest to type ignores
Reported-by: Rusty Russell <@rustyrussell>
2020-10-07 09:36:30 +10:30
Christian Decker 8d8b807793 pytest: Make LightningNode.fund_channel more resilient
It was really flaky, especially under `test_mpp_interference_2`, most likely
due to multiple calls to `fund_channel`. This commit looks for the specific
txids in the `listfunds` output and the `getrawmempool` output, avoiding
strange artifacts from multiple calls.
2020-10-07 09:36:30 +10:30
Christian Decker cafaad741b pyln: Add type-annotations to pyln-proto 2020-09-28 09:19:46 +09:30
Christian Decker c498f949cc pyln: Add mypy typing exceptions for external dependencies
We don't control them, and starting to write type stubs for them is a
different can of worms.
2020-09-28 09:19:46 +09:30
Christian Decker 1d2c2b6aaf pyln: Parametrize and unify Makefiles for pyln package 2020-09-28 09:19:46 +09:30
Christian Decker 5809cc41f4 pytest: Reduce the db_provider fixture scope to function
For performance reasons we were starting one for each session, which caused
the same postgres DB to be re-used for multiple tests (all test run in the
same worker process), but this could lead to interactions if there is a
timeout or a test happens to touch the `db_provider`. It turns out that we
were only saving about 15 seconds on a 1250 second run anyway, which is a
small cost for increased test isolation.
2020-09-26 09:04:35 +09:30
Christian Decker 7e99a3b5fd pyln: Allow the base test directory to contain some files
We were not removing the base test directory if we had other files in there,
which was the case for postgres runs. This now explicitly check for `test_*`
directories which are an indicator of a failed test.
2020-09-26 09:04:35 +09:30
Christian Decker aeb5f969e2 pyln: Start each postgres DB in its own sub-directory
We had a couple of issues with workers dying and attempting to re-initialize
the database while it was already initialized. This will look for a free
directory and just start the DB in there, allowing workers to be better
isolated.
2020-09-26 09:04:35 +09:30
Antoine Poinsot 3382dafaee configure: allow to run clang with ASAN
Prefer adding LSAN_OPTIONS="suppressions=$PWD/tests/sanitizer_suppressions/lsan" when CC=clang instead.

Changelog-Added: build: clang build now supports --enable-address-sanitizer .
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-24 13:57:43 +09:30
Christian Decker a351b9bf2f pyln: Migrate from binascii.hexlify to bytes.hex
Suggested-by: Lisa <@niftynei>
2020-09-24 11:14:22 +09:30
Christian Decker 04462f6a64 pyln: Add code to unwrap an encrypted onion at the intended node
Changelog-Added: pyln-proto: Added pure python implementation of the sphinx onion creation and processing functionality.
2020-09-24 11:14:22 +09:30
Christian Decker e8dcd59b24 pyln: Add a warning that pyln-proto is not safe for production use 2020-09-24 11:14:22 +09:30
Christian Decker 96b182a084 pyln: Implement sphinx onion packet generation in python
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2020-09-24 11:14:22 +09:30