Commit Graph

829 Commits

Author SHA1 Message Date
Christian Decker c673c092dc py: Update the PyPI job to use poetry and correct versions
Let's use poetry when we already use it for everything else.

Changelog-None
2022-03-30 06:12:40 +10:30
Rusty Russell aad4495f56 delinvoice: allow desconly arg to only remove the description.
Means that field is now optional in JSON output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `delinvoice` has a new parameter `desconly` to remove description.
2022-03-29 10:04:16 +10:30
Rusty Russell ccaf04d268 invoice: add deschashonly parameter.
LNURL wants this so they can include images etc in descriptions.

Replaces: #4892
Changelog-Added: JSON-RPC: `invoice` has a new parameter `deschashonly` to put hash of description in bolt11.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-29 10:04:16 +10:30
Rusty Russell 2424b7dea8 connectd: hold peer until we're interested.
Either because lightningd tells us it wants to talk, or because the peer
says something about a channel.

We also introduce a behavior change: we disconnect after a failed open.
We might want to modify this later, but we it's a side-effect of openingd
not holding onto idle connections.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-23 13:20:12 +10:30
Christian Decker fdd7c6b192 pyln: Remove two more occurences of non-None default args 2022-03-22 19:17:13 +10:30
Christian Decker 722f2911df py: Disentangle pyln dependencies
Turns out that the pyln-proto dependency in the bolt packages is only
needed for testing, not for production. Making it a dev-dependency
means it isn't considered in resolution anymore.

Since the bolt, testing and client packages are to be used outside
from the project we can't use relative dependencies either, so make
then dependent on the version on PyPI. This also means we had to push
a couple of updated to PyPI.

Changelog-None
2022-03-22 19:17:13 +10:30
Rusty Russell 999c734bb5 setchannel: add minhtlc
Suggested by @m-schmook, I realized that if we append it later I'll
never get it right: I expect parameters min and max, not max and min!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: you can now alter the `htlc_minimum_msat` and `htlc_maximum_msat` your node advertizes.
2022-03-22 18:45:41 +10:30
Rusty Russell 66e264d6b3 lightningd: new setchannel command.
Based on setchannelfee, but expanded to allow setting max htlc amount (and others
in future?).

The main differences:
1. It doesn't change values which are not specified (that would be hard to
   add fields to!)
2. It says exactly what all values are in any potentially changed channels.

Changelog-Added: JSON-RPC: new `setchannel` command generalizes `setchannelfee`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell 45143cc731 pytest: Remove onion test vectors containing legacy onions.
I thought about fixing them up, but really these should be in
lnprototest anyway.  Turns out they're from the spec, so we should
actually fix them up there.

I moved the vector files into contrib/pyln-proto, since that still
needs them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-18 09:20:11 +10:30
Christian Decker a8aa9bd5ae cln-rpc: Fixed two minor compiler warnings 2022-03-16 11:11:58 +10:30
Christian Decker 487b5e6169 msggen: Add meta file to keep the field numbers stable
We are inferring the field numbers on the fly, which isn't really
compatible with the way GRPC field numbers work, i.e., they must be
stable while the IDL file evolves. So far when a field was added in
the middle of a struct or removed all subsequent fields would get
renumbered, essentially breaking any client that was using the old
scheme.

We now add a meta file `.msggen.json` that keeps track of the numbers
assigned so far, so they can be reused, and new ones can be generated
not to conflict with existing ones. This file is intentionally kept
generic, so other generators can add more information that has to be
managed across runs.

Changelog-None
2022-03-16 11:11:58 +10:30
Christian Decker e99b11565c pyln: Change the default argument to `exclude` in `pay`
We must use `None` for default arguments since otherwise they aren't
filtered out when serializing the request. In addition default
arguments to functions are initialized once and, if mutable, could
persist internal changes across function calls.

Changelog-None
2022-03-15 12:20:35 -05:00
Michael Schmoock a01e2740ef pyln-proto: fix port typo in example script
Changelog-None
2022-03-11 16:42:45 +10:30
Vincenzo Palazzo c0c826d2ee ci: introduce in ci the compilation testing on different os
Changelog-None: introduce in ci the compilation testing on different os

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-10 09:44:16 +10:30
Christian Decker ad22becb26 pyln: Switch to binary psycopg2 2022-03-08 05:00:36 +10:30
Christian Decker 60135cf9ba pyln: MIgrate the BOLT packages to poetry / PEP 517 2022-03-08 05:00:36 +10:30
Christian Decker 4b9bf22193 pyln: Migrate pyln-testing to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker dd8d2c138c pyln: Migrate pyln-client to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker b0f8a99310 pyln: Migrate pyln-bolt7 to PEP517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker 74fd685219 pyln: Fix the pyln-proto version and migrate to PEP 517 (poetry) 2022-03-08 05:00:36 +10:30
Christian Decker 75f0b8e916 cln-grpc: Add `listpeers` RPC method
This is pretty much the hardest to map, but we map it correctly, with
the exception of the state_changes[] array we truncated out in the
last commit.
2022-02-28 23:25:37 +00:00
Christian Decker b73405a4c2 cln-grpc: Allow fields to be mapped to None to ignore them
`listpeers` is a rather deeply nested structure which has a couple of
caveats, namely that we use the same enum multiple times, which causes
naming clashes. So we truncate the state_changes[]. We can later map
them if needed, but it'll get much easier once we have an abstract
model description that isn't JSON schema, which unrolls all types,
causing us to generate those enums multiple times.
2022-02-28 23:25:37 +00:00
Christian Decker bb4946a6e6 msggen: Support renaming methods in GRPC
There is at least one clash with a built-in for the grpc server trait,
namely `connect` so we add support for renaming a method when
generating the scaffolding
2022-02-28 23:25:37 +00:00
Christian Decker efed7d8617 msggen: Support enums in requests too
They are sent as i32 over protobuf, so we need to convert them into
their enum representation.
2022-02-28 23:25:37 +00:00
Christian Decker 494243d41c msggen: Handle some more types in request conversions 2022-02-28 23:25:37 +00:00
Christian Decker 62dc078271 cln-grpc: Generate server dispatcher
The server doesn't do much more than unwrapping the request from its
grpc envelope, convert it into the matching JSON-RPC binding struct,
initiate the RPC connection (until we have connection pooling), and
then forwards the converted request. The inverse then happens for the
result.
2022-02-28 23:25:37 +00:00
Christian Decker 5d6e9d6dae cln-grpc: Add generation of request conversion
This is taking protobuf requests on one side and converting them into
the JSON-RPC requests.
2022-02-28 23:25:37 +00:00
Christian Decker 8d3871d791 cln-grpc: Add result conversion generator to `msggen`
This takes the Rust bindings and converts them into the generated
protobuf bindings:

> JSON-RPC -> Rust bindings -> grpc bindings -> protobuf
2022-02-28 23:25:37 +00:00
Christian Decker d01b2c21a7 cln-grpc: Add generation of grpc protobuf file from schema 2022-02-28 23:25:37 +00:00
Simon Vrouwe c286eb053f pyln-client: Plugin, improve background comment 2022-03-01 09:38:40 +10:30
Simon Vrouwe ad22535d34 pyln-testing: improve description of wait_for_logs 2022-03-01 09:38:40 +10:30
Simon Vrouwe 84bead9396 pyln-testing: in LightningNode.openchannel, make wait_for_announce more reliable
it now waits for 'alias' in node_announcement, not just block confirms.
more cleanup
2022-03-01 09:38:40 +10:30
Simon Vrouwe f84e1a0536 pyln-testing: in utils, update method fundbalancedchannel, now it also works with dualfund
and some cleanup, fundchannel has been returning tx, txid and outnum for years
2022-03-01 09:38:40 +10:30
azuchi 167fade0fa Add LIGHTNINGD_NETWORK env variable to Dockerfile for ARM
Changelog-Added: Docker build for ARM defaults to `bitcoin`, but can be overridden with the `LIGHTNINGD_NETWORK` envvar.
2022-02-22 09:51:08 +10:30
Michael Schmoock 03a1df074a peer_wire: add remote_addr to init_tlv
Unfortunately we can't do any smart parsing here since
wiregen does not support switch/type cases for different
substructure unions yet. So just give us a pointer we can use.
2022-02-22 05:45:47 +10:30
Rusty Russell d0c7e18995 bitcoind: importmulti fails (bitcoin master), use importdescriptors
But this requires a watch-only wallet, and python-bitcoinlib doesn't support
multiple wallets, so we need to unload the original one, but then we need
to generate a block, so that can't generate a new address, so we need
an address arg to generate_block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-20 19:29:44 +10:30
Christian Decker b0053e2ca2 msggen: Generate the cln-rpc Rust structs
We're generating these structs so we can parse them directly into
native objects.
2022-02-11 16:02:25 +01:00
Christian Decker 0fc0ffc961 msggen: Parse JSON-RPC schemas and build the in-memory model
We build an in-memory model of what the API should look like, which
will later be used to generate a variety of bindings. In this PR we
will use the model to build structs corresponding to the requests and
responses for the various methods.

The JSON-RPC schemas serve as ground-truth, however they are missing a
bit of context: methods, and the request-response matching (as well as
a higher level grouping we'll call a Service). I'm tempted to create a
new document that describes this behavior and we could even generate
the rather repetitive JSON schemas from that document. Furthermore
it'd allow us to add some required metadata such as grpc field
numbering once we generate those bindings.

Changelog-Added: JSON-RPC: A new `msggen` library allows easy generation of language bindings for the JSON-RPC from the JSON schemas
2022-02-11 16:02:25 +01:00
Christian Decker 95eb868047 pyln: Delete psql DBs after testing 2022-02-11 16:02:25 +01:00
Rusty Russell e8d2176e6b pytest: protect against bad gossip messages from mining confirms too fast.
If we fund a channel between two nodes, then mine all the blocks to
announce it, any other nodes may see the announcement before the
blocks, causing CI to complain about "bad gossip":

```
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Ignoring future channel_announcment for 113x1x1 (current block 112)
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 113x1x1/0
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 113x1x1/1
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG   032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_NODE_ANNOUNCEMENT before announcement 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e
```

Add a new helper for this case, and use it where there are more than 2 nodes.

Cleans up test_routing_gossip and a few other places which did this manually.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell 96ff874bd1 pytest: fix race when we mine blocks after pay().
This seems to trigger now, especially on PostgresQL (maybe it's faster
to process blocks?).

e.g. test_closing_simple() hangs in close(), because the close is unilateral
because the HTLC timed out, so it's waiting for a block (other lines removed):

```
lightningd-1: 2022-01-12T00:33:46.258Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_out WIRE_COMMITMENT_SIGNED
lightningd-1: 2022-01-12T00:33:46.278Z DEBUG   lightningd: close_command: timeout = 172800
2022-01-12T01:03:36.9757201Z lightningd-2: 2022-01-12T00:33:46.384Z DEBUG   lightningd: Adding block 104: 73ffa19d27d048613b2731e1682b4efff0dc226807d8cc99d724523c2ea58204
2022-01-12T01:03:36.9759053Z lightningd-2: 2022-01-12T00:33:46.396Z DEBUG   lightningd: Adding block 105: 44fd06ed053a0d0594abcfefcfa69089351fc89080826799fb4b278a68fe5c20
2022-01-12T01:03:36.9760865Z lightningd-2: 2022-01-12T00:33:46.406Z DEBUG   lightningd: Adding block 106: 0fee2dcbd1376249832642079131275e195bba4fb49cc9968df3a899010bba0f
2022-01-12T01:03:36.9762632Z lightningd-2: 2022-01-12T00:33:46.418Z DEBUG   lightningd: Adding block 107: 7f24f2d7d3e83fe3256298bd661e57cdf92b058440738fd4d7e1c8ef4a4ca073
2022-01-12T01:03:36.9773411Z lightningd-2: 2022-01-12T00:33:46.429Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: peer_in WIRE_REVOKE_AND_ACK
2022-01-12T01:03:36.9794707Z lightningd-2: 2022-01-12T00:33:46.437Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: Commits outstanding after recv revoke_and_ack
2022-01-12T01:03:36.9788197Z lightningd-2: 2022-01-12T00:33:46.433Z DEBUG   lightningd: Adding block 108: 283b371fb5d1ef42980ea10ab9f5965a179af8e91ddf31c8176e79820e1ec54d
2022-01-12T01:03:36.9799347Z lightningd-2: 2022-01-12T00:33:46.439Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: HTLC 0[REMOTE] => RCVD_REMOVE_REVOCATION
2022-01-12T01:03:36.9808057Z lightningd-2: 2022-01-12T00:33:46.447Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Peer permanent failure in CHANNELD_NORMAL: Fulfilled HTLC 0 RCVD_REMOVE_REVOCATION cltv 109 hit deadline
```

This is because `pay` returns from l1 when it has the preimage, not
when the HTLC is fully resolved.  Add a helper for this, and call it
at the end of the pay test helper.  We might need this elsewhere
though!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
manreo 73bf4b2c2e Update lightning.py 2022-01-08 14:15:29 +01:00
manreo dded47e332 LightningRpc should accept pathlib Path
Not sure why socket does not accept pathlib, but just added some code to transform the path to string.
It is not the most pretty code, but better than these:
https://stackoverflow.com/questions/58647584/how-to-test-if-object-is-a-pathlib-path
2022-01-08 14:15:29 +01:00
nicolas.dorier d5f2b1126c Fix dockerfile for arm32/64 2021-12-08 12:46:58 +01:00
Andrew Toth 69bc1191cb tests: add pay test for exclude arg 2021-12-04 21:37:42 +10:30
Rusty Russell e6ba66063e pytest: don't valgrind plugins under CI if many nodes.
Just pick one at random to trace fully.

Eg. test_gossip.py::test_getroute_exclude (creates 5 nodes)

Before:
	%MEM VSZ      RSS
	27.6 15941008 5106764

After:
	%MEM VSZ      RSS
	15.6 12234844 3009016

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-27 12:41:37 +00:00
Rusty Russell 67f9a8e4d6 contrib: fix other pyln-spec package requirements too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:50:54 +10:30
Jonas Nick 6409608696 pyln-bolt7: fix requirements.txt 2021-11-17 10:50:54 +10:30
Rusty Russell 65bb989cf1 pytest: don't checksum plugins on startup in VALGRIND developer mode.
This loads up 20MB of plugins temporarily; we seem to be getting OOM
killed under CI and I wonder if this is contributing.

Doesn't significantly reduce runtime here, but I have lots of memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 0c0a301062 fundchannel_complete: remove deprecated txid/txout params.
Changelog-Removed: JSON-RPC: `fundchannel_complete` `txid` and `txout` parameters (deprecated in v0.10.0)
2021-11-14 18:49:46 +01:00
Christian Decker 24701600ee pyln: Add groupid to `sendpay` and `sendonion` 2021-10-13 13:41:18 +10:30
Christian Decker 1cc349ef17 pyln: Add missing requirement from pyln-client to pyln-proto
Gossmap makes use of the proto, this ensures we have it.
2021-10-12 09:15:48 +10:30
Christian Decker cc7985df92 pytest: Use the same `TIMEOUT` in `wait_for_route`
This was getting flaky whenever we had multihop routing syncs and had
`DEVELOPER=0`, which uses the slow gossip propagation.
2021-10-12 09:15:48 +10:30
Rusty Russell 8f582e770c BOLT12: use point32 instead of pubkey32.
That's the modern BOLT12 term.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-08 13:47:30 +02:00
Michael Folkson a418cf5654 pylightning: Make it clear pyln-client should be used instead 2021-10-06 13:44:42 +02:00
Christian Decker b8f79d3e44 gci: Remove push trigger 2021-09-28 18:34:43 +02:00
Christian Decker ef579e7e9f gci: Pin down a couple more dependencies 2021-09-28 18:34:43 +02:00
Christian Decker 478c43cd9c pyln: Derive version from git for pyln-proto 2021-09-28 18:34:43 +02:00
Christian Decker 5efa7659f9 pyln: Derive version from git for pyln-testing 2021-09-28 18:34:43 +02:00
Christian Decker fe9949ce0b pyln: Derive version from git for pyln-client 2021-09-28 18:34:43 +02:00
Christian Decker 8a22cbfce7 pyln: Materialize setup.py and requirements.txt in pyln-spec
When downloading a python package from the PyPI repository the links
where pointing to a non-existent parent directory, thus breaking the
packages. The files don't ever change, and are really simple, so let's
just materialize them.
2021-09-28 18:34:43 +02:00
Rusty Russell 33168fc733 lightningd: provide 10 minutes for channel fee increases to propagate.
This was measured as a 95th percentile in our rough testing, thanks to
all the volunteers who monitored my channels.

Fixes: #4761
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setchannelfee` gives a grace period (`enforcedelay`) before rejecting old-fee payments: default 10 minutes.
2021-09-23 15:05:09 +02:00
Vincenzo Palazzo bea6ea27e8 review 1/2: Fixed the doc missing details about formatting.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo 9d5c992fc8 plugin: Adding status to the pay plugin
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>

Rebase

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo 98623aaff8 pyln-client: Adding parameter to the listpays rpc method.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Rusty Russell 2736e997f6 ccan: remove autodata.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-21 18:04:43 +02:00
Rusty Russell 24536c5561 common/autodata: use instead of ccan/autodata
This means it needs to be linked ~everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-21 18:04:43 +02:00
Rusty Russell 00a0d09340 tools/check-includes.sh: test that c files include their .h files.
This is best-practice (to ensure prototypes match up), but there were a
few places we didn't (at least, directly).  Make it a requirement,
either of form "foo.h" or <dir/foo.h>.

The noise is the change to our print templates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Dustin Dettmer cdb93eefec resolve() appears broken on Python 3.7+
On my machine it should have produced:
/tmp/ltests-1p0v5z1j/.locks/lock-1631090663.1066182
but instead it produced:
/private/tmp/ltests-1p0v5z1j/.locks/lock-1631090663.105848

The mismatch resulted in a hang inside flock.

Path .resolve() seems to be causing problems for others as well. It appears the library was not meant to handle complex path situations and isn't maintained as such (see link reference).

Since we have already built a full path here anyway, the call to .resolve() appears redundant.

Tested on python 3.9.6 on my Mac OS X 11.4 (20F71), Xcode 12.5.1 (12E507)

Relevant discussion: https://discuss.python.org/t/pathlib-absolute-vs-resolve/2573
2021-09-10 11:55:50 +02:00
Rusty Russell 6ee8c40b29 closing: add option to set closing range.
This affects the range we offer even without quick-close, but it's
more critical for quick-close.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSONRPC: `close` now takes a `feerange` parameter to set min/max fee rates for mutual close.
2021-09-09 12:04:48 +09:30
Michael Schmoock e9b801d9c8 pyln-client/gossmap: adds testcase for half channels 2021-09-08 09:34:14 +09:30
Rusty Russell e8e7c4719f pyln-client: fix mypy warnings, fix and test deletion of a channel.
This only happens when a deletion is added by a running gossipd, so
we put a deletion at the end of the store to test it.

mypy noticed that this code was nonsensical, so clearly untested.

The testing noticed that making a nodeid from a string was also buggy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Michael Schmoock 125752118a pyln-client/gossmap: init GossmapNode and Id also with hexstring
also improves test coverage
Changelog-Added: pyln-client: routines for direct access to the gossip store as Gossmap
2021-09-08 09:34:14 +09:30
Michael Schmoock fa8e74a2ad pyln-client/gossmap: make node and node_id comparable with __lt__ and __eq__ 2021-09-08 09:34:14 +09:30
Michael Schmoock 47efea92c6 pyln-client/gossmap: let half channel have source and destination 2021-09-08 09:34:14 +09:30
Michael Schmoock 2bca76b6e9 pyln-client/gossmap: start internal function names with underscore _ 2021-09-08 09:34:14 +09:30
Michael Schmoock de6c593272 pyln-client/gossmap: have channels link to their nodes instead of id
This is likely easier for programmers and does not use more mem as we
already load all this.
2021-09-08 09:34:14 +09:30
Rusty Russell 4ad4ed8d17 pyln-client/gossmap: make test gossip_store include channel_updates, deletions.
Suggested-by: @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 487facf1f0 pyln-client/gossmap: more fixes, make mypy happier.
Mainly fixing type annotations, but some real fixes:

1. GossmapHalfchannel.from_str() should be a classmethod.
2. update_channel had weird, unusable default values (fields can't be NULL,
   since we use it below).

[ There was one more occurence where isinstance should be used above
type() == xyz comparison. -- MS ]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Michael Schmoock ba2bcac530 pyln-client/gossmap: Don't mix bytes and GossmapNodeId
Do not mix bytes and GossmapNodeId when accessing Gossmap.nodes dicts.

Therefore the definion got GossmapNodeId also needed to be pulled to the
beginning of the file.
2021-09-08 09:34:14 +09:30
Michael Schmoock ac27217114 pyln-client/gossmap: adds __repr__ functions 2021-09-08 09:34:14 +09:30
Michael Schmoock b60d3259bf pyln-client/gossmap: extend testcase 2021-09-08 09:34:14 +09:30
Michael Schmoock b6ed405d8c pyln-client/gossmap: adds channel satoshi capacity
This reads the `gossip_store_channel_amount` that always follows the
`channel_announcement` messages. Therefore it uses an internal variable
_last_scid to know what channel has been added last time.
2021-09-08 09:34:14 +09:30
Michael Schmoock c5c909acd3 pyln-client/gossmap: adds helpers to get channels and nodes 2021-09-08 09:34:14 +09:30
Michael Schmoock fd16535f68 pyln-client/gossmap: adds GossmapHalfchannel objects 2021-09-08 09:34:14 +09:30
Rusty Russell 75126a0997 pyln-client/gossmap: add NodeId class.
This is more efficient than converting them all to Pubkeys: about 3.8
seconds vs 5.4 seconds.  Usually treating them as raw bytes is what we
want anyway.

[ Fixup by Michael Schmoock <michael@schmoock.net> ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 887fb38e06 pyln-client/gossmap: use ShortChannelId class from pyln.proto, if available.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 0ce473c656 pyln-proto: expose ShortChannelId and PublicKey.
They're generally useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 4753ca4d35 pyln-client/gossmap: add a little documentation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell e495462c50 pyln-client/gossmap: save deconstructed fields instead of raw msg bytes.
We have to parse them anyway, so why not make them accessible.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 1a5a26d659 pyln.client: new functionality to access Gossmap.
It doesn't do much work, but it does parse the gossmap file and extract
nodes and channels.

[ Fixup by Michael Schmoock <michael@schmoock.net> ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 07d2639f51 pyln.client: don't try to use module inside setup.py.
This fails once it has dependencies (next patch): instead extract version
manually.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-08 09:34:14 +09:30
Rusty Russell 6b8d3a05f5 pyln-spec: update to latest version of the spec.
Changelog-Changed: pyln-spec: updated to latest BOLT versions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-07 17:41:28 +09:30
Vincenzo Palazzo bc1eab8646 Skip following import and skip flask type to mypy code analysis.
As suggested in this issue https://github.com/python/mypy/issues/7484#issuecomment-529363083 we skip following import becuase with the recent version of mypy the __init__.py file make confusione inside the analysis (in the python issue it is unclear the main motivation of this issue. At list unclear to me).

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-08-30 12:12:32 +09:30
Vincenzo Palazzo b40812494d Move to minimum required mypy dependeces
This fix some dependencies error with lnprototest

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-08-30 12:12:32 +09:30
Michael Schmoock da0b159323 pylightning: fix requirements to work with pyln-client
Without this, we cannot pip install pylightning and pyln-client in the
same environment anymore, as it tries to pull in an incompatible version
of pyln-client.

Changelog-None
2021-08-26 10:13:01 +09:30
Christian Decker 0da87e7715 pyln: Fix up the mypy annotation for `init` self-disabling 2021-08-14 12:52:19 +09:30
Rusty Russell f5e08c3dae pyln-client: document and test that init can self-disable.
mypy says it returns None, but it actually doesn't have to!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-14 12:52:19 +09:30
Christian Decker 0112bbbeb7 pyln: Add E741 as excluded flake warnings
This is the ambiguous variable name warning which newer versions will
complain about.
2021-07-30 19:17:47 +02:00
Rusty Russell de12c540ee pyln.{proto,client,testing}: bump version numbers to match release.
I left pyln.proto.spec, since it's unchanged (and completely independent of
c-lightning anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-30 19:17:47 +02:00
Rusty Russell c0eba75b1d pyln.proto.message: add new BOLT12 fundamental types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-30 19:17:47 +02:00
Rusty Russell 85b4400d8d pyln.proto.message: export two more symbols, remove unused __version__ from message.py
It gets the version of pyln.proto.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-30 19:17:47 +02:00
Christian Decker d732fa9724 pyln: Fix backward compat issue in `LightningRpc.close`
The issue is that the new keyword `force_lease_closed` was being set
even if the user didn't specify it, which results in breakage if this
new pyln version talks to older c-lightning nodes that don't have this
keyword yet. By setting the default to `None` it gets filtered out if
the user has not explicitly set it, but still retains the `True` /
`False` values if they did.

Changelog-None Issue is not present in released versions
2021-07-29 16:01:37 +09:30
Vincenzo Palazzo c52af08797 Moving dependencies to the minimum required
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-07-24 14:42:37 +09:30
niftynei 77d2c538b3 queryrates: make it dev-only
Since we now use 'compact_lease' to gate an open (if the rates have
changed, we fail), we no longer need to rely on query rates for figuring
things out, so we make it dev-only.

Changelog-Changed: JSON-API: queryrates is now developer only
2021-07-20 13:28:38 -04:00
niftynei f24bbac8d9 channel leases: pass expected lease rates around in compat form
We need to know what the lease we're expecting is. To do this
we pass around the hex encoded portion of the wire format.

We can use this passed in expected lease rates to confirm that the peer
is, in fact, using the same rates as what we have currently.

Changelog-Added: JSON-RPC: fundchannel, multifundchannel, and openchannel_init now accept a 'compact_lease' for any requested funds
2021-07-20 13:28:38 -04:00
niftynei f3b54a510b close: param to force-close a leased channel
By default, we won't close a channel that we leased to a peer.
You can override this with the `force_lease_closed` flag.

Changelog-Added: JSON-RPC: close now has parameter to force close a leased channel (option_will_fund)
2021-07-20 13:28:38 -04:00
niftynei 9ad40f2544 liquidity-ad: pipe `request_amt` all the way out to fundchannel
Changelog-Added: JSON-RPC: `fundchannel` now takes optional `request_amt` parameter
2021-07-20 13:28:38 -04:00
niftynei e41d2dc0e6 liquidity-ad: request amount, pass through to dualopend
Changelog-Experimental: EXPERIMENTAL-DUAL-FUND: JSON-RPC: openchannel_init now takes a `requested_amt`, which is an amount to request from peer
2021-07-20 13:28:38 -04:00
niftynei 13e4a72188 contrib: offer a liquidity-ad lease in developer mode 2021-07-20 13:28:38 -04:00
niftynei 376e6f8bd1 dual-funding: update fee_step to be a feerate
Using a 'feestep' is more restrictive than you'd want, instead we
enforce that the next feerate must be at least 1/64th more than the
last, but put no upper limit on it

Includes update to lnprototest changes

Contributed-By: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-EXPERIMENTAL: Protocol: Replaces init_rbf's `fee_step` for RBF of v2 opens with `funding_feerate_perkw`, breaking change
2021-07-19 16:13:24 -04:00
nathanael 3ebf2325f3 chore: add 3 gpg pubkeys of maintainers to the repository
changelog-none - closes #4393
2021-07-19 13:22:33 +09:30
Rusty Russell 4e881e56ce pytest: always provide payment_secret when making payments.
They're about to become compulsory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-14 14:38:00 -05:00
Michael Schmoock 89c45b379a pyln: rpc support for listchannels by destination 2021-07-14 14:34:00 -05:00
Rusty Russell 063366ed7e listinvoices: support listing by local offer_id.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Christian Decker 5f9e5d598e pytest: Add test for keysend extra-tlvs 2021-06-26 10:55:13 +09:30
Christian Decker a4e3773408 pyln: Add wrapper for the keysend command 2021-06-26 10:55:13 +09:30
Rusty Russell 74eeaa8213 doc/schemas: getsharedsecret, help, invoice, listchannels, listforwards
And a new "u8" type.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 09:49:33 +09:30
Rusty Russell edee0793d3 pyln.proto: fix receiving unknown fields in TLVs.
We can still convert these (e.g. k=1) to Python, by just using hex strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-24 19:46:38 +09:30
Rusty Russell ca3680246e pyln.proto: fix handling of subtypes in TLVs.
This was revealed by using lnprototest on channel_types.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-24 19:46:38 +09:30
Rusty Russell 35c8247b5f cleanup: remove unused pyln-spec symlinks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-24 19:46:38 +09:30
Antoine Poinsot 1643a61499 pyln-testing: check if process died after having read all logs
We would fail even if a process exited cleanly after having the line we
were looking for, because we checked if it died before reading the logs.

Co-Authored-by: Daniela Brozzoni <daniela@revault.dev>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-06-18 10:40:07 +09:30
Michael Schmoock a53aa1aaef pyln: adds a help text to start plugins manually
Changelog-None
2021-06-18 09:59:54 +09:30
Rusty Russell 50cb810bd0 pyln-client: revert old/new comparisons for getroute, listchannels, listnodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell 9a2d73891b pyln-client: hack in test that listnodes returns the same old/new.
It sometimes fails because the two race, and sometimes because there's
randomness, but it generally works (and doesn't fail systemically).

We remove this before the final merge.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell 79299a4780 pyln-client: hack in test that listchannels returns the same old/new.
It sometimes fails because the two race, and sometimes because there's
randomness, but it generally works (and doesn't fail systemically).

We remove this before the final merge.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell d1ab5e3872 pyln-client: hack in test that getroute returns the same old/new.
It sometimes fails because the two race, and sometimes because there's
randomness, but it generally works (and doesn't fail systemically).

We remove this before the final merge.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell 67c03c02a0 pytest: speed up test_opening.py::test_funder_contribution_limits when !DEVELOPER
This test takes 695 seconds, because fundwallet waits for the wallet to
notice the tx, which takes 60 seconds if not DEVELOPER.  Do all the waiting
at once, and this speeds the test up to 153 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-05 09:47:50 +09:30
Christian Decker 90ced7ecde libhsmd: Do not use the message as the peer ID... 2021-05-30 19:49:22 +09:30
Christian Decker e27ce7d5b6 libhsmd: Add a simple tx signature test 2021-05-30 19:49:22 +09:30
Christian Decker c6a3d30bec libhsmd: Always use gcc to configure and compile 2021-05-30 19:49:22 +09:30
Christian Decker 431683293f libhsmd: Correctly wrap with `tmpctx` management in libhsmd-python 2021-05-30 19:49:22 +09:30
Rusty Russell ea99a05249 pytest: add schema support for JSON responses.
This adds our first (basic) schema, and sews support into pyln-testing
so it will load schemas for any method for doc/schemas/{method}.schema.json.

All JSON responses in a test run are checked against the schema (if any).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-27 20:28:49 +09:30
Rusty Russell efe29c9db8 sendpsbt, fundpsbt, utxopsbt, reserveinputs, unreserveinputs: allow custom number of blocks to reserve.
Not an API break: reserve=true|false still works for fundpsbt and utxopsbt,
but we also allow a raw number in there.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-26 15:08:01 +09:30
Christian Decker 6c67afeb03 make: Add CPPCHECK_OPTS to allow excluding files from the check
With swig we now have C files that are generated with tools that are
not under our control, so provide an escape hatch for them.
2021-05-15 09:31:12 +09:30
Christian Decker a4c19ba6ba libhsmd: Clean up the build descriptors for libhsmd_python 2021-05-15 09:31:12 +09:30
Christian Decker 7f15f86718 libhsmd: First working build script for libhsmd-python
Changelog-Added: libhsmd: Added python bindings for `libhsmd`
2021-05-15 09:31:12 +09:30
niftynei 3a2d602922 tests: add marker for v1/v2 channel opens
Tests that will only run when !EXPERIMENTAL_DUAL_FUND:

	@pytest.marker.openchannel('v1')
	def test_...()

Tests that will only run when EXPERIMENTAL_DUAL_FUND:

	@pytest.marker.openchannel('v2')
	def test_...()
2021-05-12 11:25:41 +09:30
niftynei 484d6bde83 tests: move EXP_DF into the testing utils 2021-05-12 11:25:41 +09:30
niftynei 09b5c906fd funder, startupregtest: fixup default config to be 100
Fails on start otherwise
2021-05-12 11:25:41 +09:30
Rusty Russell d8e68893f5 bcli: become less aggressive with onchain fee levels.
Users are more upset recently with the cost of unilateral closes
than they are the risk of being cheated.  While we complete our
anchor implementation so we can use low fees there, let's
get less aggressive (we already have 34 or 18 blocks to close
in the worst case).

The changes are:

- Commit transactions were "2 CONSERVATIVE" now "6 ECONOMICAL".
- HTLC resolution txs were "3 CONSERVATIVE" now "6 ECONOMICAL".
- Penalty txs were "3 CONSERVATIVE" now "12 ECONOMICAL".
- Normal txs were "4 ECONOMICAL" now "12 ECONOMICAL".

There can be no perfect levels, but we have had understandable
complaints recently about how high our default fee levels are.

Changelog-Changed: Protocol: channel feerates reduced to bitcoind's "6 block ECONOMICAL" rate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-11 11:25:16 +09:30
Christian Decker 083b41f090 plugin: Add a list of notification topics registered by plugin
We will eventually start emitting and dispatching custom notifications
from plugins just like we dispatch internal notifications. In order to
get reasonable error messages we need to make sure that the topics
plugins are asking for were correctly registered. When doing this we
don't really care about whether the plugin that registered the
notification is still alive or not (it might have died, but
subscribers should stay up and running), so we keep a list of all
topics attached to the `struct plugins` which gathers global plugin
information.
2021-05-03 11:20:15 +09:30
niftynei 473067859f pyln-testing: use provided outnum instead of trying to find it
Dual-funded channels won't match the old amount check. Good news is
we're already returning the outnum so we just use that.
2021-05-03 11:06:10 +09:30
niftynei bc7875864b contrib: if you're in dev mode, use dual-funding (with matching)
If you're using the regtest node, turn on dual funding and
automatically attempt to dual fund at a 100% match for every channel
open that you do.
2021-05-03 11:06:10 +09:30
Rusty Russell 32d650f9df lightningd: don't abort on incorrect versions, but try to re-exec.
You still shouldn't do this (you could get some transient failures),
but at least you have a decent chance if you reinstall over a running
daemon, instead of getting confusing internal errors if message
formats have changed.

Changelog-Added: lightningd: we now try to restart if subdaemons are upgraded underneath us.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #4346
2021-04-24 13:56:58 +09:30
Rusty Russell 402f7f90c0 pytest: handle case where funding tx is not tx #1.
e.g. in test_closing_id we can get a spend from the first (closed) channel
in the same block as the open of the second.  Half the time, we'll choose
the wrong one as scid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-04-12 23:03:47 +02:00
Christian Decker 8e0d0c3e7b pyln: Pin mypy to version 0.790 since later has import path issues 2021-04-08 10:34:14 +09:30
Christian Decker 0dd57cbf3c pyln-spec: Clean up the setup.py files
This centralizes the setup.py file, and parametrizes it so it can
auto-detect which bolt we are building. It also uses trick 3 from [1]
to avoid importing the package itself during the manifest creation,
which'd cause an import error due to missing dependencies.

[1] https://packaging.python.org/guides/single-sourcing-package-version/
2021-04-08 10:34:14 +09:30
Christian Decker 6b0a7b173c pyln: Change the setup.py file not to import the package
This would lead to errors about missing dependencies when attempting
to install using `pyhon setup.py install`. This is because the
`setup.py` file effectively is the manifest file used to discover
which dependencies are needed, so when using it to detect dependencies
we obviously don't have them yet.

See https://packaging.python.org/guides/single-sourcing-package-version/
2021-04-08 10:34:14 +09:30
Rusty Russell fad4aee5da pyln: remove deprecated fundchannel/fundchannel_start variants.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: pyln: removed deprecated fundchannel/fundchannel_start `satoshi` arg.
2021-04-07 14:34:39 +09:30
Rusty Russell 0ed614d5c4 pyln: remove deprecated pay/sendpay variants.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: pyln: removed deprecated pay/sendpay `description` arg.
2021-04-07 14:34:39 +09:30
Rusty Russell 10dee030f1 pyln: remove deprecated close variant.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: pyln: removed deprecated close `force` variant.
2021-04-07 14:34:39 +09:30
niftynei cbde1f8158 pyln-proto: make an error message more detailed
Easier to figure out what's going wrong
2021-03-30 13:44:34 +10:30
niftynei 5142dc81f6 pyln-proto: write out length of arrays of subtypes to wire
We weren't writing out the length of a nested subtype's
dynamicarraylenght, now we do. The trick is to iterate through the
fields on a subtype (since the length field is added separately)
and to also iterate down through the otherfield values as we 'descend'
2021-03-30 13:44:34 +10:30
Rusty Russell dc44737fb8 python: handle psbt-version of fundchannel_complete, deprecate old.
I didn't change the callers yet, so I can test the backwards compat code
works.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-03-16 13:10:07 +10:30
Rene Pickhardt da2ee188f3 added the default bitcoin path for macos to startup_regtest.sh
[ Neatened shell script a little, removed unwanted submodule update --RR ]
2021-03-16 10:40:34 +10:30
niftynei 9776e23b83 contrib: add the dev- options to the config iff DEVELOPER=1 configured
We scan config.vars to figure out if you configured developer on or off.
If it's on, we add the dev-only options to the config.

Fixes: #4400
Reported-By: Jonathan Harvey-Buschel @jharveyb
2021-03-16 10:26:51 +10:30
niftynei bec96a6c5b df: add openchannel_abort command
Allows us to clean up an in-progress open that we won't be completing

Changelog-Added: EXPERIMENTAL JSON-RPC: Permit user-initiated aborting of in-progress opens. Only valid for not-yet-committed opens and RBF-attempts
2021-03-15 14:08:44 +10:30
Alexey Zagarin 43c46ceb2d Fix wallet creation in contrib/startup_regtest.sh 2021-03-15 12:16:38 +10:30
Christian Decker 1e6626ff43 pyln: Fix listfunds call defaulting to spent=False
The semantics don't change, since `lightningd` will use false as
default as well, however setting it to something other than `None`
causes the RPC library to include the parameter in the query, and
since the parameter was introduced only in 0.9.3 and pyln may be used
with older versions this then results in an error about an unknown
parameter.
Setting this to `None` makes sure pyln filters out the argument before
calling.

Changelog-Fixed: pyln: Fixed an error when calling `listfunds` with an older c-lightning version causing an error about an unknown `spent` parameter
2021-03-11 14:07:47 +10:30
Christian Decker ba083c8869 pyln: Plugins can be nice if invoked from CLI too
I had way too much fun with this and got a bit carried away with the
letter writing. The idea is to be helpful when users start the plugin
from the command line, rather than run it under the control of
lightningd. We also print detailed information about the user-visible
things such as the methods and options exposed by the plugin.

Changelog-Added: pyln: Plugins that are run from the command line print helpful information on how to configure c-lightning to include them and print metadata about what RPC methods and options are exposed.
Suggested-by: Rusty Russell <@rustyrussell>
2021-03-10 21:02:55 +10:30
niftynei 1e9a900918 df: start of RPC for bumping the fee!
Doesn't call into dualopend yet
2021-03-03 16:19:04 -06:00
Karol Hosiawa 7b72ea7061 Added listforwards test 2021-03-03 09:19:20 +10:30
Rusty Russell 8ad6972b25 contrib: make startup_regtest.sh work for modern bitcoind.
They need us to create a wallet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-25 14:44:03 +10:30
niftynei ea95ad9c12 fund/utxopsbt: flag 'excess_as_change' to add a change output for excess
In the case where you want a PSBT and also want the output to be added
as a change address, use `excess_as_change` = true.

Generates a change address to use. If you want to pay the excess
elsewhere, you will have to add separately.

Changelog-Added: JSON-RPC: Add new parameter `excess_as_change` to fundpsbt+utxopsbt
2021-02-04 13:25:34 -06:00
Rusty Russell f3159ec4ac pytest: detect warnings, too.
Since we turned many errors into warnings, we want our tests to fail
when they happen unexpectedly.  We make WARNING clear in the strings
we print, too, to help out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:52 +10:30
Michael Schmoock c8198d9820 pyln-testing: add `scids` param to `wait_for_htlcs` 2021-02-04 12:02:15 +10:30
Michael Schmoock d76ca6ed35 pyln-testing: improve `wait_for` a bit
This 'fixes' the `wait_for` helper by removing a pointless final
`time.sleep()`, thus potentially making the method return quicker.

The old code could have had three final states:
 - success() := True
 - Timeout and success() := True
 - Timeout and success() := False

The new code has just two final state:
 - success() := True
 - Timeout and success() := False

It ensures the final `time.sleep()` is just the right amount before timeout.
And more importantly making it more readable :-)

Changelog-None
2021-02-04 12:02:15 +10:30
Rusty Russell 3c5502426b lightningd: addgossip API to inject gossip messages.
Importantly, this is synchronous, so pay will be able to use it
reliably.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-02 13:44:01 +01:00
Christian Decker a5f16ab5b1 pyln: Catch OSError when cleaning up test directories 2021-01-29 10:29:09 +10:30
Christian Decker ea67710e01 pyln: Pretty print RPC calls in the testing framework
We are printing `repr(obj)` which is not pretty-printed, hard to read,
and can't even be copied and inspected to JSON tools. We now print the
JSONified and indented calls and responses for easier debugging based
on solely the logs (useful for CI!).

Changelog-Added: pyln-testing: The RPC client will now pretty-print requests and responses to facilitate log-based debugging.
2021-01-29 10:29:09 +10:30
Christian Decker cd9aa267b4 pyln: Adjust maximum load allowed by the throttler 2021-01-29 10:29:09 +10:30
Christian Decker 4c3ee04bb7 pyln: Use a fair FS lock to throttle node startups
We were getting a couple of starvations, so we need a fair filelock. I
also wasn't too happy with the lock as is, so I hand-coded it quickly.

Should be correct, but the overall timeout will tell us how well we
are doing on CI.
2021-01-29 10:29:09 +10:30
Christian Decker fc677e331a pyln: Update dependencies for all pyln packages
We were getting a number of incompatibility warning due to the
dependencies being expressed too rigidly. This losens the requirement
definitions to being compatible with a known good version, and while
we're at it we also bump all outdated requirements.
2021-01-29 10:29:09 +10:30
Christian Decker b447944285 gci: Add a tester Dockerfile 2021-01-29 10:29:09 +10:30
Christian Decker 6bed85bac2 pyln: Bump pyln version to 0.9.3 before publishing on PyPI
Let's just keep pyln synched up with the c-lightning version
2021-01-27 11:29:56 +01:00
Christian Decker 483579f8b6 pyln: Pin the mypy dependency to 0.790 since 0.800 has a regression 2021-01-26 15:45:13 +01:00
nicolas.dorier 3105a7a204 Fix dockerfiles 2021-01-26 15:03:35 +01:00
niftynei ea1895fc1e pyln-tests: tweak `fundbalancedchannel` to assume peer will match
we got rid of push_msats for dual funded channels. this assumes that hte
peer will match an equal amount of sats as ours (the df_accepter.py
plugin will do this)
2021-01-10 13:44:04 +01:00
niftynei f4b7904ce5 tests: don't enforce daemon name on opening logline 2021-01-10 13:44:04 +01:00
niftynei 3c1d90086f pyln-testing: add 'config' method to a node
Query for a config's value. If not set, returns 'none'
2021-01-10 13:44:04 +01:00
Christian Decker a91254de11 pyln: Add ammag key to onion keyset
This was missing, and is required to wrap error responses.
2021-01-08 19:28:30 +01:00
Rusty Russell 1a8978100e invoice: add ctlv option.
This is required if we want to create a "bouncer" plugin (in my copious free time!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now takes an optional `cltv` parameter.
2021-01-08 10:45:56 +01:00
Christian Decker b6650425b9 pyln: Add notifications support to LightningRpc
Changelog-Added: pyln-client: Added support for command notifications to LightningRpc via the `notify` context-manager.
2021-01-07 08:32:40 +01:00
Christian Decker 84b3653606 pyln: Add command notification support for plugins
Changelog-Added: pyln-client: Plugin methods can now report progress or status via the `Request.notify` function
2021-01-07 08:32:40 +01:00
Christian Decker a1d33c3781 pytest: Add a test for `listinvoices`
We now have some more logic in the query, so let's test it
exhaustively.
2021-01-06 18:48:06 +01:00
Rusty Russell d41b383cf7 pyln-testing: suppress mypy on FileLock
3.0.12 here:
```
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-06 12:15:39 +01:00
niftynei e3a1d1a7f3 pyln-client: to_whole_satoshi returns the rounded up satoshi value
A fractional satoshi value isn't really useful; rounding up loses
precision but that's why you called "whole satoshi", wasn't it?

Changelog-Changed: pyln-client: Millisatoshi has new method, `to_whole_satoshi`; *rounds value up* to the nearest whole satoshi
2021-01-05 19:12:00 +01:00
niftynei cce0020568 pyln-client: add missing `min_witness_weight` to fundpsbt/utxopsbt
We need to use it for the 'df_accepter' plugin, so we get the feerate
correct.

Changelog-Added: pyln-client: `fundpsbt`/`utxopsbt` now support `min_witness_weight` param
2021-01-05 19:12:00 +01:00
Christian Decker dbb83a1bc3 pyln: Do not rstrip() the return value of .append()
This was causing the following error

```
Exception in thread Thread-553:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/cirrus-ci-build/contrib/pyln-testing/pyln/testing/utils.py", line 232, in tail
    self.err_logs.append(line.rstrip().decode('UTF-8', 'replace')).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'

[gw5] [ 33%] FAILED tests/test_misc.py::test_bitcoin_failure
```

Notice the second call to `.rstrip()` on the return value of `.append()`
2021-01-04 11:25:50 +01:00
Christian Decker 1ff57f07f8 pyln: Check that the stderr line is not none in pyln-testing tail
I stumbled over this in a test run and it seems benign.
2021-01-04 11:25:50 +01:00
Christian Decker 7e867e5ee6 pytest: Add throttler to limit load on the test system
Both my machine and apparently the CI tester machines regularly run
into issues with load on the system, causing timeouts (and
unresponsiveness). The throttler throttles the speed with which new
instances of c-lightning get started to avoid overloading. Since the
plugin used for parallelism when testing spawns multiple processes we
need to lock on the fs. Since we have that file open already, we'll
also write a couple of performance metics to it.
2021-01-04 11:25:50 +01:00
Christian Decker f2a0a4abfc pyln: Add logging to bitcoind RPC calls
Useful if we want to debug a bit better
2021-01-04 11:25:50 +01:00
Christian Decker 8b09ac38aa pyln: Log mempool when we generate blocks in BitcoinD fixture
We sometimes have very specific sequences of tx broadcasts and blocks
being generated to confirm them. If the confirmation is missed the
test can completely get out of sync. Make debugging this easier by
logging what we confirmed.
2021-01-04 11:25:50 +01:00
Karol Hosiawa 4a3dec660f Added spent option to listfunds in pyln-client 2020-12-22 13:00:02 +01:00
Rusty Russell 77478408f9 pyln: add RpcException for finer method failure control.
Allows caller to set code and exact message to be returned.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln-client: plugins can now raise RpcException for finer control over error returns.
2020-12-16 12:37:14 +01:00
Rusty Russell d971e3de98 Plugin: support extra args to "start".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `start` command can now take plugin-specific parameters.
2020-12-15 09:28:56 +10:30
Rusty Russell 8a9976c4c1 plugins: support concatenation of multiple args.
"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
2020-12-15 09:28:56 +10:30
Michael Schmoock 0a01111395 pytest: check millisatoshi by float raises
This only adds a test that currently makes sure its not possible
to init a Millisatoshi by a floating number.

Discussion:
As @gallizoltan points out, initialization with a float should be possible:
https://github.com/ElementsProject/lightning/pull/4273#discussion_r540369093

> Millisatoshi(5) / 2 currently works, and removes the half msat.
> So, I think Millisatoshi(5 / 2) should be the same.
2020-12-14 17:11:04 +01:00
Michael Schmoock bd13726db7 pytest: test mul and div units for Millisatoshi 2020-12-14 17:11:04 +01:00