Commit Graph

517 Commits

Author SHA1 Message Date
Karol Hosiawa 7b72ea7061 Added listforwards test 2021-03-03 09:19:20 +10:30
Rusty Russell 8ad6972b25 contrib: make startup_regtest.sh work for modern bitcoind.
They need us to create a wallet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
niftynei ea95ad9c12 fund/utxopsbt: flag 'excess_as_change' to add a change output for excess
In the case where you want a PSBT and also want the output to be added
as a change address, use `excess_as_change` = true.

Generates a change address to use. If you want to pay the excess
elsewhere, you will have to add separately.

Changelog-Added: JSON-RPC: Add new parameter `excess_as_change` to fundpsbt+utxopsbt
2021-02-04 13:25:34 -06:00
Rusty Russell f3159ec4ac pytest: detect warnings, too.
Since we turned many errors into warnings, we want our tests to fail
when they happen unexpectedly.  We make WARNING clear in the strings
we print, too, to help out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:52 +10:30
Michael Schmoock c8198d9820 pyln-testing: add `scids` param to `wait_for_htlcs` 2021-02-04 12:02:15 +10:30
Michael Schmoock d76ca6ed35 pyln-testing: improve `wait_for` a bit
This 'fixes' the `wait_for` helper by removing a pointless final
`time.sleep()`, thus potentially making the method return quicker.

The old code could have had three final states:
 - success() := True
 - Timeout and success() := True
 - Timeout and success() := False

The new code has just two final state:
 - success() := True
 - Timeout and success() := False

It ensures the final `time.sleep()` is just the right amount before timeout.
And more importantly making it more readable :-)

Changelog-None
2021-02-04 12:02:15 +10:30
Rusty Russell 3c5502426b lightningd: addgossip API to inject gossip messages.
Importantly, this is synchronous, so pay will be able to use it
reliably.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-02 13:44:01 +01:00
Christian Decker a5f16ab5b1 pyln: Catch OSError when cleaning up test directories 2021-01-29 10:29:09 +10:30
Christian Decker ea67710e01 pyln: Pretty print RPC calls in the testing framework
We are printing `repr(obj)` which is not pretty-printed, hard to read,
and can't even be copied and inspected to JSON tools. We now print the
JSONified and indented calls and responses for easier debugging based
on solely the logs (useful for CI!).

Changelog-Added: pyln-testing: The RPC client will now pretty-print requests and responses to facilitate log-based debugging.
2021-01-29 10:29:09 +10:30
Christian Decker cd9aa267b4 pyln: Adjust maximum load allowed by the throttler 2021-01-29 10:29:09 +10:30
Christian Decker 4c3ee04bb7 pyln: Use a fair FS lock to throttle node startups
We were getting a couple of starvations, so we need a fair filelock. I
also wasn't too happy with the lock as is, so I hand-coded it quickly.

Should be correct, but the overall timeout will tell us how well we
are doing on CI.
2021-01-29 10:29:09 +10:30
Christian Decker fc677e331a pyln: Update dependencies for all pyln packages
We were getting a number of incompatibility warning due to the
dependencies being expressed too rigidly. This losens the requirement
definitions to being compatible with a known good version, and while
we're at it we also bump all outdated requirements.
2021-01-29 10:29:09 +10:30
Christian Decker b447944285 gci: Add a tester Dockerfile 2021-01-29 10:29:09 +10:30
Christian Decker 6bed85bac2 pyln: Bump pyln version to 0.9.3 before publishing on PyPI
Let's just keep pyln synched up with the c-lightning version
2021-01-27 11:29:56 +01:00
Christian Decker 483579f8b6 pyln: Pin the mypy dependency to 0.790 since 0.800 has a regression 2021-01-26 15:45:13 +01:00
nicolas.dorier 3105a7a204 Fix dockerfiles 2021-01-26 15:03:35 +01:00
niftynei ea1895fc1e pyln-tests: tweak `fundbalancedchannel` to assume peer will match
we got rid of push_msats for dual funded channels. this assumes that hte
peer will match an equal amount of sats as ours (the df_accepter.py
plugin will do this)
2021-01-10 13:44:04 +01:00
niftynei f4b7904ce5 tests: don't enforce daemon name on opening logline 2021-01-10 13:44:04 +01:00
niftynei 3c1d90086f pyln-testing: add 'config' method to a node
Query for a config's value. If not set, returns 'none'
2021-01-10 13:44:04 +01:00
Christian Decker a91254de11 pyln: Add ammag key to onion keyset
This was missing, and is required to wrap error responses.
2021-01-08 19:28:30 +01:00
Rusty Russell 1a8978100e invoice: add ctlv option.
This is required if we want to create a "bouncer" plugin (in my copious free time!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now takes an optional `cltv` parameter.
2021-01-08 10:45:56 +01:00
Christian Decker b6650425b9 pyln: Add notifications support to LightningRpc
Changelog-Added: pyln-client: Added support for command notifications to LightningRpc via the `notify` context-manager.
2021-01-07 08:32:40 +01:00
Christian Decker 84b3653606 pyln: Add command notification support for plugins
Changelog-Added: pyln-client: Plugin methods can now report progress or status via the `Request.notify` function
2021-01-07 08:32:40 +01:00
Christian Decker a1d33c3781 pytest: Add a test for `listinvoices`
We now have some more logic in the query, so let's test it
exhaustively.
2021-01-06 18:48:06 +01:00
Rusty Russell d41b383cf7 pyln-testing: suppress mypy on FileLock
3.0.12 here:
```
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-06 12:15:39 +01:00
niftynei e3a1d1a7f3 pyln-client: to_whole_satoshi returns the rounded up satoshi value
A fractional satoshi value isn't really useful; rounding up loses
precision but that's why you called "whole satoshi", wasn't it?

Changelog-Changed: pyln-client: Millisatoshi has new method, `to_whole_satoshi`; *rounds value up* to the nearest whole satoshi
2021-01-05 19:12:00 +01:00
niftynei cce0020568 pyln-client: add missing `min_witness_weight` to fundpsbt/utxopsbt
We need to use it for the 'df_accepter' plugin, so we get the feerate
correct.

Changelog-Added: pyln-client: `fundpsbt`/`utxopsbt` now support `min_witness_weight` param
2021-01-05 19:12:00 +01:00
Christian Decker dbb83a1bc3 pyln: Do not rstrip() the return value of .append()
This was causing the following error

```
Exception in thread Thread-553:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/cirrus-ci-build/contrib/pyln-testing/pyln/testing/utils.py", line 232, in tail
    self.err_logs.append(line.rstrip().decode('UTF-8', 'replace')).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'

[gw5] [ 33%] FAILED tests/test_misc.py::test_bitcoin_failure
```

Notice the second call to `.rstrip()` on the return value of `.append()`
2021-01-04 11:25:50 +01:00
Christian Decker 1ff57f07f8 pyln: Check that the stderr line is not none in pyln-testing tail
I stumbled over this in a test run and it seems benign.
2021-01-04 11:25:50 +01:00
Christian Decker 7e867e5ee6 pytest: Add throttler to limit load on the test system
Both my machine and apparently the CI tester machines regularly run
into issues with load on the system, causing timeouts (and
unresponsiveness). The throttler throttles the speed with which new
instances of c-lightning get started to avoid overloading. Since the
plugin used for parallelism when testing spawns multiple processes we
need to lock on the fs. Since we have that file open already, we'll
also write a couple of performance metics to it.
2021-01-04 11:25:50 +01:00
Christian Decker f2a0a4abfc pyln: Add logging to bitcoind RPC calls
Useful if we want to debug a bit better
2021-01-04 11:25:50 +01:00
Christian Decker 8b09ac38aa pyln: Log mempool when we generate blocks in BitcoinD fixture
We sometimes have very specific sequences of tx broadcasts and blocks
being generated to confirm them. If the confirmation is missed the
test can completely get out of sync. Make debugging this easier by
logging what we confirmed.
2021-01-04 11:25:50 +01:00
Karol Hosiawa 4a3dec660f Added spent option to listfunds in pyln-client 2020-12-22 13:00:02 +01:00
Rusty Russell 77478408f9 pyln: add RpcException for finer method failure control.
Allows caller to set code and exact message to be returned.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln-client: plugins can now raise RpcException for finer control over error returns.
2020-12-16 12:37:14 +01:00
Rusty Russell d971e3de98 Plugin: support extra args to "start".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `start` command can now take plugin-specific parameters.
2020-12-15 09:28:56 +10:30
Rusty Russell 8a9976c4c1 plugins: support concatenation of multiple args.
"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
2020-12-15 09:28:56 +10:30
Michael Schmoock 0a01111395 pytest: check millisatoshi by float raises
This only adds a test that currently makes sure its not possible
to init a Millisatoshi by a floating number.

Discussion:
As @gallizoltan points out, initialization with a float should be possible:
https://github.com/ElementsProject/lightning/pull/4273#discussion_r540369093

> Millisatoshi(5) / 2 currently works, and removes the half msat.
> So, I think Millisatoshi(5 / 2) should be the same.
2020-12-14 17:11:04 +01:00
Michael Schmoock bd13726db7 pytest: test mul and div units for Millisatoshi 2020-12-14 17:11:04 +01:00
Michael Schmoock fffc343dd7 pyln: fix Millisatoshi div with msat itself
Before this patch this fails: `Millisatoshi(42) / Millisatoshi(2)`
This is an operation that should return the ratio between the two
operands as a float number. Same goes for __floordiv__ operator `//`.

Changelog-None
2020-12-14 17:11:04 +01:00
Christian Decker f7cdf1dd98 pytest: Debug test_lockup_drain 2020-12-11 16:55:55 -06:00
Christian Decker 68d08fc7d7 pyln: Add TOR and SOCKS5 support in pyln.proto.wire.connect
I wanted to talk to TOR-based nodes, so here comes TOR support :-)
2020-12-11 11:39:18 +10:30
Christian Decker eacc54646f pyln: Remove deprecated txprepare variant
We promised to deprecate the old "destination satoshi feerate utxos"
variant some time on mid-2020 so let's do just that. `txprepare`
doesn't parse it either anymore.

Changelog-Changed: pyln: `txprepare` no longer supports the deprecated `destination satoshi feerate utxos` call format.
2020-12-09 06:56:21 +10:30
Michael Schmoock 362284981d plyn: use math.floor for msat mul and div 2020-12-08 13:48:14 +01:00
Michael Schmoock 565dc95a8e pyln: fix msat float multiplication 2020-12-08 13:48:14 +01:00
Michael Schmoock 2678cfb364 pyln: extend msat floating testcases
This adds two more xfail'ing testcases to show that the current way of
parsing Millisatoshi decimals is not yet optimal.

Changelog-None
2020-12-08 13:48:14 +01:00
Christian Decker 76124eb800 pyln: Replace undecodeable symbols when tailing logs
Logs may contain non-ASCII and non-UTF8 symbols, which crashes the
tailer. It's better to replace them with a glyph representing
undecodeable symbols instead, and handle the issue further up the
call-chain.
2020-12-04 18:34:12 -06:00
Christian Decker c9d24b1de2 pyln: Set the prefix in TailableProc
We rely on it when logging, but didn't set it in the class, rather we
did in the sub-classes. This proved to be rather annoying to search
since it'd just fail silently and not log any output.
2020-12-04 18:34:12 -06:00
Rusty Russell e4950db9a3 pytest: recreate wallet on bitcoind restart.
Doesn't seem to stick in master.  Andy Chow suggested we
simply turn off wallet on older versions, and always create/load.

```
[gw8] [ 40%] FAILED tests/test_misc.py::test_bitcoind_goes_backwards

============================================================= FAILURES ==============================================================
___________________________________________________ test_bitcoind_goes_backwards ____________________________________________________
[gw8] linux -- Python 3.8.5 /usr/bin/python3

node_factory = <pyln.testing.utils.NodeFactory object at 0x7f931859a760>
bitcoind = <pyln.testing.utils.BitcoinD object at 0x7f931865eee0>

    def test_bitcoind_goes_backwards(node_factory, bitcoind):
        """Check that we refuse to acknowledge bitcoind giving a shorter chain without explicit rescan"""
        l1 = node_factory.get_node(may_fail=True, allow_broken_log=True)

        bitcoind.generate_block(10)
        sync_blockheight(bitcoind, [l1])
        l1.stop()

        # Now shrink chain (invalidateblock leaves 'headers' field until restart)
        bitcoind.rpc.invalidateblock(bitcoind.rpc.getblockhash(105))
        # Restart without killing proxies
        bitcoind.rpc.stop()
        TailableProc.stop(bitcoind)
        bitcoind.start()

        # Will simply refuse to start.
        with pytest.raises(ValueError):
            l1.start()

        # Nor will it start with if we ask for a reindex of fewer blocks.
        l1.daemon.opts['rescan'] = 3

        with pytest.raises(ValueError):
            l1.start()

        # This will force it, however.
        l1.daemon.opts['rescan'] = -100
        l1.start()

        # Now mess with bitcoind at runtime.
>       bitcoind.generate_block(6)

tests/test_misc.py:1307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:399: in generate_block
    return self.rpc.generatetoaddress(numblocks, self.rpc.getnewaddress())
contrib/pyln-testing/pyln/testing/utils.py:322: in f
    return proxy._call(name, *args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bitcoin.rpc.RawProxy object at 0x7f93184f6a30>, service_name = 'getnewaddress', args = ()
postdata = '{"version": "1.1", "method": "getnewaddress", "params": [], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -18, 'message': 'No wallet is loaded. Load a wallet using loadwallet or create a new one with createwallet. (Note: A default wallet is no longer automatically created)'}, 'id': 1, 'result': None}
```
2020-12-03 11:35:30 +01:00
Michael Schmoock 7bfb5f10c7 pyln: failing test msat from float str
We were not able to create pyln Millisatoshi from floats, e.g.:
 - "0.01btc"
 - "0.1sat"
 - ...

This adds a test that makes sure this won't happen again.
2020-12-02 10:37:31 +10:30
Michael Schmoock 83a21138b8 pyln: fix msat from float str
Changelog-fixed: pyln: parsing msat from a float string
2020-12-02 10:37:31 +10:30