Having the grpc bindings in the pyln-testing package was always a bit
strange, however it came with additional issues. Due to the way that
protos are handled by protobuf, any name clash, independently of where
we import the protos, would cause an import error. This usually
happens in diamond-pattern dependencies, and so pull out the generated
files into their own package that everyone else can rely on.
Changelog-None
When we release too fast, the plugin crashes:
```
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: SendError(())', plugins/examples/cln-plugin-reentrant.rs:31:27
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
```
This happens with CI under VALGRIND!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was pointed out by Daywalker [1]: we are synchronously processing
events from `lightningd` which means that if processing one of the
hooks or requests was slow or delayed, we would not get notifications,
hooks, or RPC requests, massively impacting the flexbility.
This highlights the issue with a failing test (it times out), and in
the next commit we will isolate event processing into their own task,
so to free the event loop from having to wait for an eventual
response.
[1] https://community.corelightning.org/c/developers/hold-invoice-plugin#comment_wrapper_16754493
These may eventually end up in pyln-client, as they allow talking to
the GRPC interface exposed by cln-grpc, however for now they are used
for testing only. Once we have sufficient API and test coverage we can
move them and leave imports in their place.
This is consistent with our output changes, and increases consistency.
It also keeps future sanity checks happy, that we only use JSON msat
helpers with '_msat' fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `invoice`, `sendonion`, `sendpay`, `pay`, `keysend`, `fetchinvoice`, `sendinvoice`: `msatoshi` argument is now called `amount_msat` to match other fields.
Changelog-Deprecated: JSON-RPC: `invoice`, `sendonion`, `sendpay`, `pay`, `keysend`, `fetchinvoice`, `sendinvoice` `msatoshi` (use `amount_msat`)
So far we were papering over the actual error with a generic string
that isn't very useful. Now we decode the error and forward it through
the grpc layer as well (though still as a string).
e.g.
```
lightningd-1: 2022-03-28T11:02:12.476Z DEBUG plugin-cln-grpc: add_pem_file processed 1 valid and 0 invalid certs
lightningd-1: 2022-03-28T11:02:12.478Z DEBUG plugin-cln-grpc: Connecting to \"lightning-rpc\" and serving grpc on 0.0.0.0:36331
lightningd-1: 2022-03-28T11:02:12.478Z DEBUG connectd: REPLY WIRE_CONNECTD_ACTIVATE_REPLY with 0 fds
lightningd-1: 2022-03-28T11:02:12.478Z INFO lightningd: --------------------------------------------------
lightningd-1: 2022-03-28T11:02:12.478Z INFO lightningd: Server started with public key
```
Which means we don't see it, since start() swallows it:
```
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('serving grpc on 0.0.0.0:')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we aren't using the correct certificates we should reject the
connections during the mTLS connection setup. This test tries to
connect with the wrong client cert to the node, and the server will
reject it.
For now hooks are treated identically to rpcmethods, with the
exception of not being returned in the `getmanifest` call. Later on we
can add typed handlers as well.
Having a list of very targeted suppressions allows us to still run the
majority of tests with valgrind checking, and not fail when Rust does
some trickery. This is for example the case with `std::sync::Once`
which uses `num_procs` calling out to the cgroups subsystem, sometimes
with a null path.
Suggested-by: Rusty Russell <@rustyrussell>
`valgrind` reports seems to flag some memory accesses that are ok in
the Rust standard library, which we can consider false positives for
our purposes:
```Valgrind error file: valgrind-errors.69147
==69147== Syscall param statx(file_name) points to unaddressable byte(s)
==69147== at 0x4B049FE: statx (statx.c:29)
==69147== by 0x2E2DA0: std::sys::unix::fs::try_statx (weak.rs:139)
==69147== by 0x2D7BD5: <std::fs::File as std::io::Read>::read_to_string (fs.rs:784)
==69147== by 0x2632CE: num_cpus::linux::Cgroup::param (linux.rs:214)
==69147== by 0x263179: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==69147== by 0x263002: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==69147== by 0x262C01: num_cpus::linux::load_cgroups (linux.rs:149)
==69147== by 0x26289D: num_cpus::linux::init_cgroups (linux.rs:129)
==69147== by 0x26BD88: core::ops::function::FnOnce::call_once (function.rs:227)
==69147== by 0x26B749: std::sync::once::Once::call_once::{{closure}} (once.rs:262)
==69147== by 0x139717: std::sync::once::Once::call_inner (once.rs:419)
==69147== by 0x26B6D5: std::sync::once::Once::call_once (once.rs:262)
==69147== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==69147==
==69147== Syscall param statx(buf) points to unaddressable byte(s)
==69147== at 0x4B049FE: statx (statx.c:29)
==69147== by 0x2E2DA0: std::sys::unix::fs::try_statx (weak.rs:139)
==69147== by 0x2D7BD5: <std::fs::File as std::io::Read>::read_to_string (fs.rs:784)
==69147== by 0x2632CE: num_cpus::linux::Cgroup::param (linux.rs:214)
==69147== by 0x263179: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==69147== by 0x263002: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==69147== by 0x262C01: num_cpus::linux::load_cgroups (linux.rs:149)
==69147== by 0x26289D: num_cpus::linux::init_cgroups (linux.rs:129)
==69147== by 0x26BD88: core::ops::function::FnOnce::call_once (function.rs:227)
==69147== by 0x26B749: std::sync::once::Once::call_once::{{closure}} (once.rs:262)
==69147== by 0x139717: std::sync::once::Once::call_inner (once.rs:419)
==69147== by 0x26B6D5: std::sync::once::Once::call_once (once.rs:262)
==69147== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==69147==
```
Christian Decker
Rusty Russell