Commit Graph

986 Commits

Author SHA1 Message Date
Greg Sanders 1037bf3c43 Put bitcoind logging in stdout for pytest 2023-05-25 11:07:45 +09:30
Jon Griffiths 8b4136bad5 build: remove gmp
Core and secp have not used gmp for a very long time now (core disabled
it in 2015).

Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2023-05-24 18:21:43 +09:30
Rusty Russell e7d4c3175a build: remove --enable-experimental-features / EXPERIMENTAL_FEATURES
Changelog-EXPERIMENTAL: Build: all experimental features are now runtime-enabled; no more ./configure --enable-experimental-features
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-05-23 09:34:08 +09:30
Rusty Russell b8aa3a579e listtransactions: remove annotations, which were only in EXPERIMENTAL_FEATURES.
I don't know if anyone was using them, they seem half-hearted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-05-23 09:34:08 +09:30
Shahana Farooqui d1cf88c62e meta: update changelog and pyln version for 23.05 release
Changelog-None
2023-05-10 09:50:38 -07:00
Shahana Farooqui 4f258a9354 meta: Add changelog for 23.05rc4 2023-05-09 14:04:14 -07:00
Matt Morehouse fe5f3cef51 pyln: remove unused variable
Variable inroduced in 708fb17 is making CI fail.
2023-05-09 13:15:36 -07:00
Shahana Farooqui ea23122880 meta: Add changelog for 23.05rc3 2023-05-05 10:20:27 -07:00
Christian Decker 708fb17fa2 pytest: Add helper to get a grpc stub and test decode 2023-05-05 09:24:07 -07:00
Christian Decker acc3bb2276 msggen: Switch signatures to string instead of bytes 2023-05-05 09:24:07 -07:00
Christian Decker ef9e0fcf60 pyln: Set the correct envvar for logging for rust plugins 2023-05-05 09:24:07 -07:00
Christian Decker fc26675336 msggen: Add DecodePay to the mappings
Changelog-Added: grpc: Added mapping for `listpeerchannels`, `listclosedchannels`, `decode` and `decodepay` RPC method
2023-05-05 11:54:41 +09:30
Christian Decker 0031f1160b msggen: Map arrays of hashes and add HtlcState enum 2023-05-05 11:54:41 +09:30
Christian Decker db843159ea msggen: Move overrides into the model itself
We were using per-type overrides which caused some asymmetries, where
conversions could end up dropping fields as we went along. Essentially
each conversion would need to override a superset of the previous one,
which then caused issues when attempting to close the loop. By
overriding on the model level we ensure that all representations are
equivalent and convertible into one another, at the expense of
overriding a bit more aggressively, which should be fine anyway.
2023-05-05 11:54:41 +09:30
Christian Decker b41cb2f005 cln-grpc: Rename the ChannelSide variants
IN/OUT => LOCAL/REMOTE
2023-05-05 11:54:41 +09:30
Christian Decker bff3b1ca8c msggen: Add ListClosedChannels and overrides 2023-05-05 11:54:41 +09:30
Christian Decker d28815f7b8 msggen: Disable grpc response -> json response temporarily
We use overrides that omit fields in some cases, which makes the
conversion lossy. This also means that until we complete the mapping
we can't reconvert back.
2023-05-05 11:54:41 +09:30
Christian Decker e7a96cac11 msggen: Normalize enum and field names if they contain a '/' 2023-05-05 11:54:41 +09:30
Christian Decker 65f5134643 msggen: Add ListPeerChannels to generated interfaces 2023-05-05 11:54:41 +09:30
Christian Decker 2a52e52015 jsonrpc: Add versioning annotation to listpeerchannels 2023-05-05 11:54:41 +09:30
Shahana 21cc16fb5b meta: Add changelog for 23.05rc2 2023-04-26 15:46:02 -07:00
Shahana Farooqui 15795c969a meta: Add changelog for 23.05rc1
Typo correction

Updated delpay changelog
2023-04-15 12:33:06 -07:00
Christian Decker 2e5ad0f417 pyln: Exclude all `cln-` plugins from valgrind 2023-04-13 18:13:28 -07:00
Joel Klabo c0d3eeb789 Fix Typo in startup_regtest.sh 2023-04-11 13:39:51 +09:30
Rusty Russell 89b96e8ac0 pyln-testing: add support to tell bitcoind not to include txs if fee is too low.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell cf80f0520a connectd: dev-report-fds to do file descriptor audit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 09:41:56 +09:30
Rusty Russell a000ee015a lightningd: do RBF again for all the txs.
Now we've set everything up, the replacement code is quite simple.

Some tests now have to deal with RBF though, and our rbf tests need work
since they look for the old onchaind messages.

In particular, when we can't afford the fee we want, we back off to
the next blockcount estimate, rather than spending all on fees
(necessarily).  So test_penalty_rbf_burn no longer applies.

Changelog-Changed: Protocol: spending unilateral close transactions now use dynamic fees based on deadlines (and RBF), instead of fixed fees.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell 3a3370f4c1 feerates: add `floor` field for the current minimum feerate bitcoind will accept
Changelog-Added: JSON-RPC: `feerates`: added `floor` field for current minimum feerate bitcoind will accept
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell c46473e615 lightningd: allow "NNblocks" and "minimum" as feerates.
And consolidate descriptions into lightning-feerates().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` now allow "minimum" and NN"blocks" as `feerate` (`feerange` for `close`).
2023-04-10 07:31:12 +09:30
Rusty Russell 64b1ddd761 lightningd: clean up feerate handling, deprecate old terms.
Drop try_get_feerate() in favor of explicit feerate_for_deadline() and
smoothed_feerate_for_deadline().

This shows us everywhere we deal with old-style feerates by names.

`delayed_to_us` and `htlc_resolution` will be moving to dynamic fees,
so deprecate those.

Note that "penalty" is still used for generating penalty txs for
watchtowers, and "unilateral_close" still used until we get zero-fee
anchors.

Changelog-Added: JSON-RPC: `feerates` `estimates` array shows fee estimates by blockcount from underlying plugin (usually *bcli*).
Changelog-Changed: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) value *slow* is now 100 block-estimate, not half of 100-block estimate.
Changelog-Deprecated: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) expressed as, "delayed_to_us", "htlc_resolution", "max_acceptable" or "min_acceptable".  Use explicit block counts or *slow*/*normal*/*urgent*/*minimum*.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell cdb85d5618 lightningd: handle fees as blockcount + range.
Rather than have specific-purpose levels, have an array of
[blockcount, feerate], and rebuild the specific-purpose levels
for now on top.

We also keep a *separate* smoothed feerate, so you can ask for that
explicitly.

Since all the plugins used the same formula to derive the different
named fee levels, we apply the reverse to return to the underlying
estimates: updating the interface comes next.

This is ugly for now, but various specific-purpose levels will be
going away, as we shift to deadline-driven fees.

This temporarily breaks the floor calculation, so that test is
disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell b53cc69cfd msggen: fix incorrect assertion.
Adding a new field with `added` fails:

```
AssertionError: Field Feerates.perkb.estimates[] does not have an `added` annotation
```

Looks like this assertion is wrong: we should get an added from the field itself or
from the .msggen.json file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell c5b7dbcd98 pytest: clean up wait_for_onchaind_tx interface, remove wait_for_onchaind_broadcast
Using single tuples in Python is ugly, so:
1. Rename wait_for_onchaind_tx to wait_for_onchaind_txs.
2. Make it take tuples explicitly.
3. Make wait_for_onchaind_tx a simpler wrapper/unwrapper.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell 3e83bed460 pyln-testing: adapt wait_for_onchaind_broadcast function for when onchaind uses lightningd for broadcast.
We can no longer grab the tx in one line as we did with
wait_for_onchaind_broadcast, we need to track the broadcast from
lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Christian Decker 168bc54700 msggen: Add VersioningCheck
This is a visitor that ensures every new field has at least an `added`
field, and that we don't change the `added` or `deprecated` annotation
after the fact.
2023-04-06 10:16:19 +09:30
Christian Decker 60b12ec096 msggen: Use the inferred optional field
Changelog-Changed: msggen: The generated interfaces `cln-rpc` anc `cln-grpc` can now work with a range of versions rather than having to match the CLN version
2023-04-06 10:16:19 +09:30
Christian Decker 392cacac81 msggen: Add an optional patch
This patch annotates the fields with a new `optional` attribute which
determines whether the field should be considered an inferred optional
due to being added or deprecated.
2023-04-06 10:16:19 +09:30
Christian Decker 5df469cfca msggen: Add patching system, add `added` and `deprecated` to Field
The patching system allows us to enrich the raw schema with some
additional information. In this specific case we want to backfill the
`added` and `deprecated` fields for the multiversion support.
2023-04-06 10:16:19 +09:30
Rusty Russell 13ae1a5168 pyln.testing: remove Throttler.
CI seems to block; Christian suggests the throttler may be to blame somehow?

Since trying to fix it made it worse, let's just remove it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:10:53 +09:30
Chris Guida 1507e87197 fix helloworld.py example in README for pyln-client 2023-04-05 15:28:32 +09:30
Michael Schmoock 04ea37d88f pygossmap: rename GossipStoreHeader to GossipStoreMsgHeader
Changelog-Added: pyln-client: Improvements on the gossmap implementation
2023-04-05 06:13:08 +09:30
Michael Schmoock 6e46a63c57 pygossmap: adds statistic and filter module
Includes a lot of useful filters and statistical methods.

To see a gossip_store summary:
```
s = GossmapStats(g)
s.print_stats()
```
2023-04-05 06:13:08 +09:30
Michael Schmoock f1b6047d69 pygossmap: store features for nodes and channels
also makes them acessible using bitmask functions
2023-04-05 06:13:08 +09:30
Michael Schmoock 3130f4ec27 pygossmap: read .disabled from channel_flags 2023-04-05 06:13:08 +09:30
Michael Schmoock 6a16a31a98 pygossmap: parse node addresses and other data 2023-04-05 06:13:08 +09:30
Michael Schmoock 9409f2f1ea pygossmap: adds get_neighbors and get_neighbors_hc flodding method 2023-04-05 06:13:08 +09:30
Michael Schmoock 5a9a3d83c9 pygossmap: adds get_halfchannel 2023-04-05 06:13:08 +09:30
Michael Schmoock d50722d26b pygossmap: adds a more complete mesh testcase 2023-04-05 06:13:08 +09:30
Michael Schmoock eb9cb5ef31 pygossmap: adds missing __str__, __eq__ and __hash__
Also caches certain __hash__ and __str__ operations,
This way graph operations can be done quicker.
2023-04-05 06:13:08 +09:30
Michael Schmoock be60f2ac33 pygossmap: adds GossmapHalfchannel to module exports 2023-04-05 06:13:08 +09:30
Michael Schmoock 3f651b08d5 pygossmap: cleanups and optimizations
- moves offset into GossipHeader hdr which is passed to all constuctors
 - reads .flags as u16 instead of extracting it from the .length, see 0274d88ba
 - adds zombie and ratelimit flag to GossipHeader
 - bytes_read start at 0 instead of 1 which is more correct,
   the one byte is then corrected for when setting the offset of new header.
 - bytes_read is increased in pull_bytes as this is the only place where
   something is read
 - use new style for various format-strings
2023-04-05 06:13:08 +09:30
Michael Schmoock 882cafd3c7 pytest: adds skipped test_create_gossip_mesh
This can be adapted and used to create test gossip stores.
The test is just skipped by design as it would fail on intention.
2023-04-05 06:13:08 +09:30
Michael Schmoock fb0027e314 pyln-testing: fundbalancedchannel default total_capacity to FUNDAMOUNT 2023-04-05 06:13:08 +09:30
Michael Schmoock 407d4d2922 pyln-testing: remove deprecated fund_channel
This method is no longer used in cln nor in the plugins repo.

Changelog-None
2023-04-05 06:13:08 +09:30
Michael Schmoock f4b8a401cd pyln-proto: shorten ShortChannelId.from_str() 2023-04-05 06:13:08 +09:30
Christian Decker 7d7b2abd02 msggen: Allow using deprecated fields in the rpc -> grpc conversion
We should rather hand the annotation through to the user code, and
warn there.
2023-03-24 00:23:43 +00:00
Greg Sanders cf662e55a7 Make startup_regtest.sh more robust to bitcoind wallet state 2023-03-23 16:10:55 +10:30
Greg Sanders cb7caa3139 Re-enable PSBT tests for Liquid except test_sign_and_send_psbt 2023-03-23 16:10:55 +10:30
Kristaps Kaupe 906279a46e Output channel_id in listfunds
Changelog-Added: JSON-RPC: `listfunds` now has a `channel_id` field.
2023-03-21 10:29:33 +00:00
Rusty Russell 983542f2a7 global: remove deprecated "msat" suffix on msat fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: the "msat" suffix on millisatoshi fields, as deprecated in v0.12.0.
2023-03-18 15:55:49 +10:30
Rusty Russell 780f32dfc6 global: remove deprecated non-msat-named msat fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: all the non-msat-named millisatoshi fields deprecated in v0.12.0.
2023-03-18 15:55:49 +10:30
Rusty Russell 67f23c19f7 lightningd: remove deprecated local_msat, remote_msat from listpeers.
Changelog-Removed: JSON-RPC: `listpeers`.`local_msat` and `listpeers`.`remote_msat` (deprecated v0.12.0)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-18 15:55:49 +10:30
Alex Myers 538a8d5c57 meta: update changelog and pyln version for 23.02 release
Changelog-None
2023-03-02 12:14:50 -06:00
Alex Myers 0707ffcab4 reprobuild: use pyenv for python installation
python-setuptools installation was flakey on some systems. Installing
with pyenv should provide a more reproducible build.

Changelog-None
2023-03-02 12:11:02 -06:00
Christian Decker c7fd13a460 repro: Add `protoc` dependency to repro-build 2023-03-02 12:11:02 -06:00
Alex Myers b6a7532625 meta: Add changelog for 23.02rc1
Changelog-None
2023-02-08 23:11:42 -06:00
Carl Dong b83a19164c msggen: Regenerate for addition of SendCustomMsg
Performed using:
  PYTHONPATH=contrib/msggen python3 contrib/msggen/msggen/__main__.py
2023-02-08 23:10:53 -06:00
Carl Dong ef51ae3c6d msggen: Enable SendCustomMsg 2023-02-08 23:10:53 -06:00
Riccardo Casatta 510abb934c cln-grpc: add roundtrip tests for test_getinfo and test_listppers 2023-02-08 12:03:27 -06:00
Riccardo Casatta 21a8342289 Implement GRPC -> JSON conversions also for response types 2023-02-08 12:03:27 -06:00
Riccardo Casatta 9c35f9c13a Implement conversion JSON->GRPC also for requests type
This could be useful for existing software with  a JSON interface that want
to mimic the interface
2023-02-08 12:03:27 -06:00
Michael Schmoock 1e951a9479 mssgen: adds num_channels 2023-02-07 14:46:04 -06:00
Christian Decker e5d5737927 grpc: Allow conversion code to use deprecated fields
The warning was rather superfluous, we should rather annotate the
downstream structs so the developer gets a warning, not us library
maintainers.
2023-02-07 08:23:17 -06:00
Carl Dong dc4ae9deb4 msggen: Regenerate for addition of SignInvoice
Performed using:
  PYTHONPATH=contrib/msggen python3 contrib/msggen/msggen/__main__.py
2023-02-06 15:54:32 -06:00
Carl Dong 11227d37ba msggen: Enable SignInvoice 2023-02-06 15:54:32 -06:00
Michael Schmoock 3efbc12706 pyln-client: adds utils cln_parse_rpcversion
This adds the `cln_parse_rpcversion` helper that is already used in
various plugins to pyln-client, so it does not need to be copied
around anymore.

Changelog-None
2023-02-03 16:58:31 +01:00
Michael Schmoock 3dde1ca399 pyln-testing: fix wait_for_htlcs helper
When doing the updates on the plugin repo, I discovered that
this helper function got broken by the `listpeerchannels` upgrade.

Its rarely used in the main repo, just at the end of the pyln-testing
'pay' helper.

If unfixed, this bug may result in test flakes when using the `pay`
helper, because its not correctly waiting for all HTLCs to be resolved
before returning.

Changelog-None
2023-02-02 18:47:08 +01:00
Rusty Russell 83c690fe5f doc: fix listsendpays man page.
We actually had a partid allowed (in the oneOf clauses), but didn''t
document it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9ab488fc41 plugins/topology: add direction field to listchannels.
It's a core concept in the spec which isn't directly exposed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
2023-01-30 15:15:41 -06:00
Rusty Russell 611795beee listtransactions: get rid of per-tx type annotations.
We didn't actually populate them properly, and the real annotations
are on inputs and outputs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `listtransactions` `channel` and `type` field removed at top level.
2023-01-30 15:15:41 -06:00
Rusty Russell 4b9cb7eb76 doc: remove unused offerout schema.
We removed the command for v22.11.

Also, we removed the `refund_for` offer parameter, so remove its description
from the manpage.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Riccardo Casatta 182a9cdcb6 cln-rpc: use serde rename instead of alias
rename is necessary to roundtrip, otherwise the rust name is used.

This also remove the rename if they are not necessary.

Note that:
```
 #[serde(rename="foo", skip_serializing_if=="bar")]
 pub field: bool,
```

is equivalent to:
```
 #[serde(rename="foo")]
 #[serde(skip_serializing_if=="bar")]
 pub field: bool,
```

and for simplicity of construction the latter is used
2023-01-26 15:03:54 +01:00
Rusty Russell 932ca9e91f lightningd: deprecate p2sh-segwit addresses for `newaddr` `addresstype`
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `newaddr`: `addresstype` `p2sh-segwit` (use default, or `bech32`).
2023-01-14 21:17:45 +10:30
Rusty Russell 85fc46f76c pyln-testing: don't default openchannel and fundwallet to p2sh-segwit, use bech32.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-14 21:17:45 +10:30
Vincenzo Palazzo f08d3516f7 contrib/pyln-testing: use listpeerchannels. 2023-01-13 10:42:42 +10:30
Vincenzo Palazzo 6fa904b4fb lightningd: add listpeerchannels command
Changelog-Added: JSON-RPC: new command `listpeerchannels` now contains information on direct channels with our peers.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-01-13 10:42:42 +10:30
Justin Litchfield 879db694e1 Ping request types are changed from number to u16 2023-01-09 14:35:23 +01:00
arowser f918a04738 unify the zlib version 2023-01-05 14:42:00 +01:00
arowser 90956fa947 change zlib download path 2023-01-05 14:42:00 +01:00
ekzyis 87a0cb1603 Replace head -n with sort -R for better random peer selection 2023-01-04 13:29:46 +10:30
Jesse de Wit 241cd8d012 generate composite fields in grpc 2023-01-03 10:56:10 +01:00
Michael Schmoock af502eb625 pytest: adds test for msat to int comparison
Changelog-None
2023-01-02 11:42:17 +01:00
Michael Schmoock 6f46010417 pyln-client: make Millisatoshi comparable to int
This often helps the msat purge and pyln msat replacement mischmasch issues.
I changed it in a way that the `AttributeError: 'int' object has no attribute 'millisatoshis'`
Error will still be raised whever a `Millisatoshi` is compared to
something else then a `Millisatoshi` or `int`.

Changelog-None
2022-12-29 13:00:42 +01:00
Christian Decker 8e75232205 rs: Update outdated dependencies
This commit updates outdated dependencies and hangs all
bitcoin-related dependencies off of the `bitcoin` crate, using its
re-exports. This means that as long as the bitcoin crate matches, all
of its dependents will also match.

Changelog-None
2022-12-22 15:42:12 +01:00
Greg Sanders 3b56e90a13 Turn on logging for key topics in bitcoind for black box tests 2022-12-22 12:05:01 +01:00
Rusty Russell 68f3e3fba9 pyln: add datastore routines.
Without explicit definitions, we don't get proper names args and help
messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-12-22 12:04:30 +01:00
Greg Sanders c79c9c73fc Properly raise ValueError message in wait_for 2022-12-19 13:09:46 +01:00
Rusty Russell 5b3746172f lightningd: remove `setchannelfee`.
Replaced by `setchannel`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changlog-Removed: JSON-RPC: `setchannelfee` (deprecated in v0.11.0)
2022-12-13 08:28:12 +10:30
Christian Decker 6af78c35df misc: Update cdecker's gpg key 2022-12-06 16:37:19 +01:00
nicolas.dorier a96ff3b097 Update the contrib arm32v7 and arm64v8 dockerfiles 2022-12-06 10:48:57 +01:00