Commit Graph

1356 Commits

Author SHA1 Message Date
ShahanaFarooqui 77e3c3a38e plugin/clnrest: Adjusting flask-cors install after rebase 2023-09-29 16:00:05 +09:30
ShahanaFarooqui e010eece1b plugins/clnrest: Updated clnrest's requirements.txt
Even after migrating from pip install to poetry install for clnrest,
we still need to keep `plugins/clnrest/requirements.txt` for Fedora-28-amd64
build. `Dockerfile.builder.fedora` does not have poetry and will need
requirements.txt to install libraries with pip.

It is also required for non-developers who want to build cln from a tagged or master version.
2023-09-29 16:00:05 +09:30
ShahanaFarooqui 6a756256c0 plugins/clnrest: Adding pyproject.toml for Poetry install
Reference links:
https://github.com/ElementsProject/lightning/issues/6637
https://github.com/nix-community/poetry2nix

Changelog-Changed: Upgraded clnrest to poetry project.
2023-09-29 16:00:05 +09:30
Rusty Russell 577161a4fa renepay: remove custom debug, in favor of normal operations.
As a bonus, unit tests no longer leave files in /tmp.

Reported-by: https://github.com/whitslack
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-25 13:59:13 +09:30
Rusty Russell 2bcae6fc76 plugins/renepay: remove DEVELOPER build-time, in favor of runtime.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell f725edad62 plugins: remove #if DEVELOPER.
And rename dev-only-option `use_shadow` to `dev_use_shadow`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell ef87999f9a common: add option for dev-only parameters.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell 1ecf31bae7 db: remove #if DEVELOPER in favor of runtime flag inside db struct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell a9f26b7d07 common/daemon.c: remove #ifdef DEVELOPER in favor of runtime flag.
Also requires us to expose memleak when !DEVELOPER, however we only
ever used the memleak tracking when the LIGHTNINGD_DEV_MEMLEAK
environment variable was set, so keep that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell c8c6dd183d common/bolt12: roll all necessary checks into offer_decode.
Some of these were done by fetchinvoice (and it did some that
offer_decode already did).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 13:31:03 +09:30
ShahanaFarooqui 3108ff93b8 plugins/clnrest: Update content security policy for server 2023-09-19 11:40:15 +09:30
ShahanaFarooqui 51578e1f77 plugins/clnrest: Add `rest-csp` option for config 2023-09-19 11:40:15 +09:30
ShahanaFarooqui eca3a33e69 plugins/clnrest: Update Websocket server to dynamically add CORS settings
Changelog-Added: New configurable Cross-Origin-Resource-Sharing(CSP) header for clnrest
2023-09-19 11:40:15 +09:30
ShahanaFarooqui b0430cc897 plugins/clnrest: Update App server to dynamically add CORS settings 2023-09-19 11:40:15 +09:30
ShahanaFarooqui 632e7abd05 plugins/clnrest: Add `rest-cors-origins` option for config 2023-09-19 11:40:15 +09:30
ShahanaFarooqui 224a5dcbcf plugins/clnrest: Add CORS library 2023-09-19 11:40:15 +09:30
ShahanaFarooqui 6aa697ea3d plugins/clnrest: Rune authentication for websocket server 2023-09-19 11:40:15 +09:30
ShahanaFarooqui 9147996372 plugins/clnrest: Generate all three certs if missing
Generate client, server and CA certs for https and trusted CA update
2023-09-19 11:40:15 +09:30
ShahanaFarooqui 761172001b plugin/clnrest: Deleting redundant README.md 2023-09-12 16:10:15 +09:30
Rusty Russell a4644550a2 clnrest: don't require nodeid (FIXME: needs screenshot updates!)
This was a misunderstanding: nodeid is useful for commando, where it's the
peer's nodeid, and Noise-XK guarantees that we know who that is.  It's
not useful for clnrest, so don't require it (it was our node id, which
is redundant).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-12 15:19:02 +09:30
ShahanaFarooqui e788f76f99 plugin/clnrest: do not read json payload if data length is zero
OpenAPI readme always includes `content-type: application/json` header, even when body parameters are empty.
But the server expects data if the content-type has been sent.
This results in a "Server Error" response for non-param requests from readme doc.
This only affects readme requests as it is designed to send the header by default.

Changelog-None
2023-08-31 16:13:35 +09:30
Rusty Russell f4f4ab34f3 renepay: fix case where shadow increases amount past payment amount.
Without this, sendpay will refuse to pay.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-31 16:13:21 +09:30
Rusty Russell 3285aa0eea renepay: fix shadow route extension logic.
pseudorand(1) is always 0, and capacity test was the wrong way.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-31 16:13:21 +09:30
Rusty Russell 7a92eb2d89 renepay: calculate fee and delay accumulation correctly.
1. When we add a shadow amount, we were using the wrong channel for
   the fee calculation.
2. Similarly, when calculating the delay amount.

The result is that we can get WIRE_INCORRECT_CLTV_EXPIRY repeatedly
from nodes.

Reported-by: https://github.com/Sjors
Fixes: #6620
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changlog-Experimental: Fixed: `renepay` handles ctlv correctly when it varies along a path.
2023-08-31 16:13:21 +09:30
Rusty Russell 14e28a51ce renepay: don't crash if we get failure from sendpay.
This happened while I was hacking (it shouldn't normally!): cmd here is NULL.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-31 16:13:21 +09:30
Christian Decker 8e2e7b1c51 rs: Bump crate versions before publishing
Changelog-None
2023-08-28 11:19:30 +02:00
Dusty Daemon 8d5be61fcb clnrest: Change process method to fork
On mac the clnrest plugin hangs on log messages, causing it to never run.

This guide suggests switching to fork’ing instead of spwan’ing
https://superfastpython.com/multiprocessing-common-errors/#Error_2_print_Does_Not_Work_In_Child_Processes

Switching to fork fixes the hang on Mac.

Changelog-None
2023-08-24 19:36:42 +09:30
Rusty Russell 7cd4359521 renepay: hack in workaround spendable reduction for release.
```
Flow 391: amount=23528000msat prob=0.000 fees=1023msat delay=140 path=-2471854x37x4/1(min=max=23528783msat)->-2414928x98x0/0->
Flow 391: Failure of 23529023msat for 2471854x37x4/1 capacity [23528783msat,23528783msat] -> [23528783msat,23528783msat]
```

We added fees and went over capacity!  This screams of a deeper logic
bug, but renepay is experimental and it's release day so hack around
it for now...

Reported-by: https://github.com/daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 13:22:27 +09:30
Rusty Russell a984202576 plugins/renepay: don't drop min to 0 if it's shown wrong, but drop 50%.
Eduardo is on holiday right now, but he pinged me asking for this.  It
makes some sense, and using half the *failed value* covers the case where
it's less than half what we expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 32f5ae8d1f renepay: remove remaining calls to debug_paynote in favor of payflow_note().
Some are redundant, others simply convert.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 574716dfd5 plugins/renepay: convenience routing to log when we disable a chan.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 00e9af57f5 plugins/renepay: neaten the command notifications.
It now looks like (for test_hardmpp):

```
# we have computed a set of 1 flows with probability 0.328, fees 0msat and delay 23
#   Flow 1: amount=1800000000msat prob=0.328 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0->-103x3x0/1->
#   Flow 1: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 1: Failure of 1800000000msat for 103x5x0/0 capacity [0msat,3000000000msat] -> [0msat,1799999999msat]
# we have computed a set of 2 flows with probability 0.115, fees 0msat and delay 23
#   Flow 2: amount=500000000msat prob=0.475 fees=0msat delay=12 path=-103x6x0/0(min=max=4294967295msat)->-103x1x0/1->-103x4x0/1->
#   Flow 3: amount=1300000000msat prob=0.242 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1799999999msat)->-103x3x0/1->
#   Flow 3: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 3: Failure of 1300000000msat for 103x5x0/0 capacity [0msat,1799999999msat] -> [0msat,1299999999msat]
# we have computed a set of 2 flows with probability 0.084, fees 0msat and delay 23
#   Flow 4: amount=260000000msat prob=0.467 fees=0msat delay=12 path=-103x6x0/0(500000000msat in 1 htlcs,min=max=4294967295msat)->-103x1x0/1(500000000msat in 1 htlcs)->-103x4x0/1(500000000msat in 1 htlcs)->
#   Flow 5: amount=1040000000msat prob=0.179 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1299999999msat)->-103x3x0/1->
#   Flow 5: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 5: Failure of 1040000000msat for 103x5x0/0 capacity [0msat,1299999999msat] -> [0msat,1039999999msat]
# we have computed a set of 2 flows with probability 0.052, fees 0msat and delay 23
#   Flow 6: amount=120000000msat prob=0.494 fees=0msat delay=12 path=-103x6x0/0(760000000msat in 2 htlcs,min=max=4294967295msat)->-103x1x0/1(760000000msat in 2 htlcs)->-103x4x0/1(760000000msat in 2 htlcs)->
#   Flow 7: amount=920000000msat prob=0.105 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1039999999msat)->-103x3x0/1->
#   Flow 7: Success
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 7db6aaa523 renepay: clarify chan_extra_can_send.
I am doing to add more more debugging, but sent here is 0.
Document that clearly, and put a real value in sent.

Also: since we already sub 1 msat from x, amount_msat_less_eq should
be amount_msat_less (it may be equal to our min, in theory).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 7287ade705 plugins/renepay: use struct short_channel_id_dir.
No changes, just use this convenience type for handing
around, and arrays.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell a6c2cd77a1 plugins/renepay: clean up unused fmt_payflows.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell eef57488f9 pay_flow: downgrade messages about shadow route delays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 4196f97571 plugins/renepay: Add infra for per-flow notes.
And set logging levels.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-23 10:47:28 +09:30
Rusty Russell 763ec6ad43 plugins/renepay: don't free waiting-for-addgossip flow!
```
Aug 18 13:45:13  lightningd: 0x7fa921f8ffcf ???
Aug 18 13:45:13  lightningd:         ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
Aug 18 13:45:13  lightningd: 0x55b3bb54e6d3 pay_flow_finished_adding_gossip
Aug 18 13:45:13  lightningd:         plugins/renepay/pay_flow.c:675
Aug 18 13:45:13  lightningd: 0x55b3bb54af25 addgossip_done
Aug 18 13:45:13  lightningd:         plugins/renepay/pay.c:171
```

The assert we fail is almost certainly due to the flow being freed:

```
struct pf_result *pay_flow_finished_adding_gossip(struct pay_flow *pf)
{
	assert(pf->state == PAY_FLOW_FAILED_GOSSIP_PENDING);
```

Reported-by: https://github.com/daywalker90
Fixes: #6567
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-21 21:13:48 +09:30
Vincenzo Palazzo d2e39f7af8 meta: ignore renepay binary
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-08-21 10:35:41 +09:30
Rusty Russell 685c413b61 renepay: reduce minimum "known" if it proves incorrect.
There's a fascinating bug report which suggests this happens on local channels,
implying spendable_msat is wrong?

See-also: #6567
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 20:29:55 +09:30
Rusty Russell b5d1708900 plugins/sql: make testing-only sqlfilename option `dev-sqlfilename`.
dev options don't need to be documented, and this is only for debugging
the plugin itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 20:29:34 +09:30
Rusty Russell bcea0a9580 listchannels: also list splicing channels as active: true.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 17:09:54 +09:30
Rusty Russell 0147b3ba94 pay: don't filter out local channels which are splicing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 17:09:54 +09:30
Rusty Russell 71a09138b6 renepay: splicing channels are also usable for payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 17:09:54 +09:30
Rusty Russell d3c7d48acb pay: don't discard high-htlc_min channels when searching for routehint starts.
As side-effect, getroute(0) is special too.

Reported-by: MiddleW4y in Discord
Fixes: #6577
Changelog-Fixed: `pay` will still use an invoice routehint if path to it doesn't take 1-msat payments.
2023-08-18 14:55:32 +09:30
Rusty Russell af6b535efd pay: still use channels for routehints even if peer says it's disabled.
We have a report that LND said our (unannounced) channel was disabled, so we didn't
use it for routehints.  We're better off ignoring that in this case (if the peer is
actually not connected, the routehint code will check that and ignore anyway).

Fixes: #6555
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: pay: use channels in routehints even if peer says they're "disabled" (LND compat)
2023-08-15 06:22:53 +09:30
Rusty Russell 8dc8416ed7 plugins/renepay: fix gcc-12.3.0 -O3 warning.
Compiler can't tell that we always set have_state[PAY_FLOW_FAILED_FINAL]
when we set this:

```
plugins/renepay/payment.c: In function ‘payment_reconsider’:
plugins/renepay/payment.c:287:25: error: ‘final_error’ may be used uninitialized [-Werror=maybe-uninitialized]
  287 |                         payment_fail(payment, final_error, "%s", final_msg);
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
plugins/renepay/payment.c:194:30: note: ‘final_error’ was declared here
  194 |         enum jsonrpc_errcode final_error, ecode;
      |                              ^~~~~~~~~~~
plugins/renepay/payment.c:287:25: error: ‘final_msg’ may be used uninitialized [-Werror=maybe-uninitialized]
  287 |                         payment_fail(payment, final_error, "%s", final_msg);
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
plugins/renepay/payment.c:195:21: note: ‘final_msg’ was declared here
  195 |         const char *final_msg;
      |                     ^~~~~~~~~
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-15 06:21:23 +09:30
Rusty Russell aac8905afd plugins/renepay/test: fix access-after-free.
We cannot carry pointers into the gossmap across localmod addition
or removal.

We didn't notice because the map->chan_arr is not normally resized,
but if we change gossmap.c line 689 to only allocate 1 to start, we see this:

```
VALGRIND=1 valgrind -q --error-exitcode=7 --track-origins=yes --leak-check=full --show-reachable=yes --errors-for-leak-kinds=all plugins/renepay/test/run-mcf > /dev/null
==2349744== Invalid read of size 4
==2349744==    at 0x1788C2: gossmap_chan_scid (gossmap.c:558)
==2349744==    by 0x1872A2: get_chan_extra_half_by_chan (flow.c:346)
==2349744==    by 0x187797: remove_completed_flow (flow.c:488)
==2349744==    by 0x187927: remove_completed_flow_set (flow.c:518)
==2349744==    by 0x18DF4D: main (run-mcf.c:393)
==2349744==  Address 0x4b80f38 is 88 bytes inside a block of size 136 free'd
==2349744==    at 0x4848C63: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2349744==    by 0x173D71: tal_resize_ (tal.c:744)
==2349744==    by 0x177E36: next_free_chan (gossmap.c:336)
==2349744==    by 0x177ED3: new_channel (gossmap.c:351)
==2349744==    by 0x178441: add_channel (gossmap.c:458)
==2349744==    by 0x1798D4: gossmap_apply_localmods (gossmap.c:904)
==2349744==    by 0x18DEDB: main (run-mcf.c:388)
==2349744==  Block was alloc'd at
==2349744==    at 0x4848C63: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2349744==    by 0x173D71: tal_resize_ (tal.c:744)
==2349744==    by 0x177E36: next_free_chan (gossmap.c:336)
==2349744==    by 0x177ED3: new_channel (gossmap.c:351)
==2349744==    by 0x178441: add_channel (gossmap.c:458)
==2349744==    by 0x178B6D: map_catchup (gossmap.c:635)
==2349744==    by 0x178F45: load_gossip_store (gossmap.c:697)
==2349744==    by 0x179D71: gossmap_load (gossmap.c:978)
==2349744==    by 0x18D22F: main (run-mcf.c:295)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-15 05:32:22 +09:30
Rusty Russell 4f16f50353 renepay: don't free adg twice after adding gossip.
It will usually be freed by the call to pay_flow_finished_adding_gossip.

```
cln-renepay: FATAL SIGNAL 6 (version 862dac2-modded)
0x561f232f32f8 call_error
	/home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:95
0x561f232f3df8 check_bounds
	/home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:169
0x561f232f2281 to_tal_hdr
	/home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:179
0x561f232f2189 tal_free
	/home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:514
0x561f2327f801 addgossip_done
	/home/runner/work/lightning/lightning/plugins/renepay/pay.c:170
0x561f23292d9b handle_rpc_reply
	/home/runner/work/lightning/lightning/plugins/libplugin.c:871
0x561f23292a90 rpc_read_response_one
	/home/runner/work/lightning/lightning/plugins/libplugin.c:1048
0x561f23292811 rpc_conn_read_response
	/home/runner/work/lightning/lightning/plugins/libplugin.c:1072
0x561f232dc0f2 next_plan
	/home/runner/work/lightning/lightning/ccan/ccan/io/io.c:59
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-15 05:31:46 +09:30
Rusty Russell d032a23647 plugins/renepay: don't require raw_message in error reply.
It's not there if it's a local error:

```
{
   "code": 202,
   "message": "Parsing '{message:%,data:{erring_index:%,failcode:%,raw_message:': object does not have member raw_message"
}
```

Reported-by: https://github.com/daywalker90
Fixes: #6553
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-15 05:31:46 +09:30
Rusty Russell efc0264ba2 renepay: add command notifications
These show that we should clean up our notes.  Here's the result from test_hardmpp:

# we have computed a set of 1 flows with probability 0.328, fees 0msat and delay 23
# No MPP, so added 0msat shadow fee
# Shadow route on flow 0/1 added 0 block delay. now 5
# sendpay flow groupid=1, partid=1, delivering=1800000000msat, probability=0.328
# Update chan knowledge scid=103x2x0, dir=0: [0msat,1799999999msat]
# onion error WIRE_TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
# we have computed a set of 2 flows with probability 0.115, fees 0msat and delay 23
# Shadow route on flow 0/2 added 0 block delay. now 5
# Shadow route on flow 1/2 added 0 block delay. now 5
# sendpay flow groupid=1, partid=3, delivering=500000000msat, probability=0.475
# sendpay flow groupid=1, partid=2, delivering=1300000000msat, probability=0.242
# Update chan knowledge scid=103x2x0, dir=0: [0msat,1299999999msat]
# onion error WIRE_TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
# we have computed a set of 2 flows with probability 0.084, fees 0msat and delay 23
# Shadow route on flow 0/2 added 0 block delay. now 5
# Shadow route on flow 1/2 added 0 block delay. now 5
# sendpay flow groupid=1, partid=5, delivering=260000000msat, probability=0.467
# sendpay flow groupid=1, partid=4, delivering=1040000000msat, probability=0.179
# Update chan knowledge scid=103x2x0, dir=0: [0msat,1039999999msat]
# onion error WIRE_TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
# we have computed a set of 2 flows with probability 0.052, fees 0msat and delay 23
# Shadow route on flow 0/2 added 0 block delay. now 5
# Shadow route on flow 1/2 added 0 block delay. now 5
# sendpay flow groupid=1, partid=7, delivering=120000000msat, probability=0.494
# sendpay flow groupid=1, partid=6, delivering=920000000msat, probability=0.105

Ideally it would look something like:

# Computed 1 flows, probability=0.328:
#  Flow 1: 103x2x0 1800000000msat fee=0msat probability=0.328 shadow=+0msat/0blocks
#  Flow 1: FAIL: TEMPORARY_CHANNEL_FAILURE for 103x2x0.
# Computed 2 flows, probability=0.115:
#  Flow 2: XXX->XXX 1300000000msat fee=XXX, probability=0.475 shadow=+0msat/0blocks
#  Flow 3: XXX->XXX 500000000msat fee=XXX, probability=0.475 shadow=+0msat/0blocks
#  Flow 2: FAIL: TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0
# Computed 2 flows (3 total), probability=0.084, fee=0msat, delay=23
...
#  Flow 4: SUCCESS, 2 in progress should succeed soon.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 5c9a1075ec renepay: trivial cleanup to rename `flow` to `pf` everywhere.
Consistency FTW.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 96cace892c renepay: do less work in destroy_pay_flow, and reorder pay_flow.c
Unifies the pay_flow resolve functions, and moves remove_htlc_payflow
and commit_htlc_payflow to the top.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 8e89895f97 renepay: add dummy pf_resuly type to ensure we deal with the flow.
We want to make sure that on every path, we terminate the flow.  The simplest
way to do this is encourage the pattern "return pay_flow_xxx(flow)".

Indeed, this caught a few places I missed!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell d7b3bdd420 renepay: drive *all* progress from termination of struct pay_flow.
The main function here is payment_reconsider:
* Each payment has a list of pay_flow.
* This is populated in try_paying(), calling add_payflows & sendpay_new_flows.
* When we get a notification, we resolve a pay_flow using one of the pay_flow_failedxxx
  or pay_flow_succeeded functions.
* They call payment_reconsider() which cleans up finished flows decides what to do:
  often calling try_paying again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell d75d68cdde renepay: grab update from WIRE_TEMPORARY_CHANNEL_FAILURE if present.
It's not required, but it should be there so we might as well use it
(though we sometimes don't put one in, esp if it's a private channel).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 15c30f4018 renepay: make pay_plugin a tal object.
Avoids a gratuitous "ctx" field, and the simplified declaration
is now understood by `make update-mocks`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 01a29d8453 renepay: fix up handling of errors from final node.
Treat it just like "PAY_TRY_OTHER_ROUTE", except it is from the final node:
this means we correctly process that it "succeeded".

Add a test: this crashes sometimes, but it's cleaned up soon...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 40d93c2af7 renepay: remove always-true "first_time" and "unlikely_ok" flags.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 5e5f592369 renepay: remove unused result member.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell e8cfb40709 renepay: put the entire hash in the key struct.
As recommended by your TODO, a bit simpler: we also make the hash function
return a ptr rather than the (now rather large) struct.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 135180c1a0 renepay: don't re-parse bolt11 to get routehints.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 2ceac6c11d renepay: simplify JSON handling in notification_sendpay_success.
Use json_scan(), and use the new pay_flow_from_notification() routine.

Also, the tal_dup_or_null can be tal_dup, since &preimage is never NULL.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 72b316924d renepay: simplify JSON handling.
We can use json_scan(), and share a routine to map the notification
to the pay_flow.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell fb50474efd renepay: get max group_id in single iteration.
Simplifies the logic since we bail out if there are two different
group ids in progress.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 3b8217bd72 renepay: merge `struct renepay` and `struct payment` into one.
There are a few fields in `struct renepay` which are genuinely
transient, but it makes the code much harder to follow than simply
having a single structure.

More cleanups will follow, but this is the minimal set.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 6d7cd1e729 renepay: make memleak simpler.
Simply tell it to scan the entire object.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 0091dc4e6d renepay: move list_node to first member of struct payment.
Results in payments having a pointer to the start of the object,
which helps our memleak code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 008bd74c93 renepay: use more formal allocator pattern.
The general pattern for xxx_new is that it should populate all
fields, for encapsulation and so you never can have a half-formed
object.

This means a fair bit of work for now, but it pays off in the next
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell ca72a230f3 renepay: fix localmods.
You cannot refresh the gossmap with localmods applied, nor apply localmods
when others have applied localmods in the same process.

There are optimizations we could do, but for now always apply/unapply before
querying gossmap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 0a9b3d6740 renepay: remove attempt limit.
Time is what users care about, so remove this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell b5273610b2 renepay: remove unused all_flows field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell 5f8b77480c plugins: fix error report from bitcoin-cli exec failure.
We've stomped errno, so if exec fails we don't get a reliable result:

```
2023-08-07T17:58:45.713Z **BROKEN** plugin-bcli: bitcoin-cli exec failed: Bad file descriptor
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-09 19:53:47 +09:30
junderw 32b88a2340 Fix: Remove Sync requirements on Futures returned in the Rust plugin library.
See: https://github.com/bitcoindevkit/bdk/issues/1047#issuecomment-1660645669

In general, futures produced by most libraries in the ecosystem of Rust, and bounds placed
on users of famous runtimes like tokio and its spawn method all lack Sync requirements.

Because of this, anyone who creates a callback using any sort of library that returns a
non-Sync future (which most libraries fit this description) inside of it will get some
cryptic error messages (async error messages still leave a lot to be desired).

Removing these Sync requirements will make the library more useful.
2023-08-08 09:45:42 +09:30
Rusty Russell 54bcb10227 lightningd: fix bolt11 parsing in preapproveinvocie, sendonion, listsendpays and renepay
Since bolt11_decode now insists that any `lightning:` prefix be removed, we need
to make sure to use param_invstring not param_string for all bolt11 parameters:

```
2023-08-07T05:55:32.515Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v23.08rc1-21-g0bf5ee6)
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x55dd94934154
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x55dd949342e6
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7f5cf5a3bcef
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:44 (__pthread_kill_implementation) 0x7f5cf5a9226b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:78 (__pthread_kill_internal) 0x7f5cf5a9226b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:89 (__GI___pthread_kill) 0x7f5cf5a9226b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ../sysdeps/posix/raise.c:26 (__GI_raise) 0x7f5cf5a3bc45
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7f5cf5a227fb
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./assert/assert.c:92 (__assert_fail_base) 0x7f5cf5a2271a
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./assert/assert.c:101 (__GI___assert_fail) 0x7f5cf5a33595
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/bolt11.c:734 (bolt11_decode_nosig) 0x55dd94929967
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/bolt11.c:953 (bolt11_decode) 0x55dd9492a44f
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/pay.c:1730 (json_listsendpays) 0x55dd948d7d72
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:658 (command_exec) 0x55dd948b525b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:786 (rpc_command_hook_final) 0x55dd948b5876
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:285 (plugin_hook_call_) 0x55dd948f6446
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:874 (plugin_hook_call_rpc_command) 0x55dd948b5c77
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:984 (parse_request) 0x55dd948b6234
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1090 (read_json) 0x55dd948b670f
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55dd94ac9bf4
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55dd94aca823
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55dd94aca865
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x55dd94accbff
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x55dd948b33c4
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1332 (main) 0x55dd948ba429
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7f5cf5a2350f
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:381 (__libc_start_main_impl) 0x7f5cf5a235c8
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x55dd94881e74
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Fixes: #6524
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: broken in master since last release.
2023-08-07 18:46:48 +09:30
Rusty Russell 2042b50978 plugins/bcli: update minimum required bitcoind version.
Less than 22 is obsolete anyway, so we should increment this from 16.0 at least!

Closes: #6234
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-07 16:04:14 +09:30
Rusty Russell 84fe3dc940 build: more 32-bit printf fixes.
Reported-by: Shahana
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-07 11:52:40 +09:30
ShahanaFarooqui 18b928046b plugin/clnrest: Downgrading pyln-proto versino from 23.08 to 23.5.2
Bug fix: pip install from plugins/clnrest/requirements.txt was unable to find "not yet released" pyln-proto version 23.08.
2023-08-06 12:34:19 +09:30
Dusty Daemon 7c9f5e9847 renepay: Use macros instead of manual format types
Fixes 32-bit builds:

```
In file included from plugins/renepay/pay.c:5:
./plugins/renepay/pay_flow.h: In function 'fmt_payflow_key':
./plugins/renepay/pay_flow.h:54:17: error: format '%ld' expects argument of type 'long int', but argument 4 has type 'u64' {aka 'long long unsigned int'} [-Werror=format=]
   54 |                 "key: groupid=%ld, partid=%ld, payment_hash=%s",
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   55 |                 k->groupid,k->partid,
      |                 ~~~~~~~~~~
      |                  |
      |                  u64 {aka long long unsigned int}
```

etc
2023-08-04 20:33:20 +09:30
Rusty Russell 67e43ea868 renepay: clean up ARC_ADDITIONAL_BITS.
It's expressed in bits, but really it's clearer as a quantity, given
how it's used.

Suggested-by: @Lagrang3
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 11:40:52 +09:30
Rusty Russell 15c8a6f6fe renepay: switch from arc_t to struct arc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 11:40:52 +09:30
Rusty Russell b793dc9224 renepay: implement an alternative to a union with bitsets.
This was actually relying on little-endian; if we want this, we need to be explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 11:40:52 +09:30
Shahana Farooqui c96d0c2643 plugin/clnrest: Updated pyln version to 23.08 in requirements.txt 2023-08-01 11:35:19 +09:30
Dusty Daemon 4628e3ace8 channeld: Code to implement splicing
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.

Implement the channeld splicing protocol leveraging the interactivetx protocol.

Implement lightningd’s channel_control to support channeld in its splicing efforts.

Changelog-Added: Added the features to enable splicing & resizing of active channels.
2023-07-31 21:00:22 +09:30
Dusty Daemon 50fe819f47 splicing: Add channel state AWAITING_SPLICE
Update gossip routiens and various other hecks on the channel state to consider AWAITING_SPLICE to be routable and treated similar to CHANNELD_NORMAL.

Small updates to psbt interface

Changelog-None
2023-07-31 21:00:22 +09:30
Lagrang3 3024afe524 renepay: more cleanups
- adopt "const <type> *"convention
- remove use_shadow option for some pyln tests
- show prob. information of flows into paynotes
- show prob. of success of entire payment flow in paynotes
- minflow: We were not releasing the memory of flow arrays when replacing
  them with a new canditate.
- use memleak_scan_obj in memleak_check
- replace u64 with size_t

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell b5da85e85d plugins/renepay/dijkstra: improve API to remove global.
The global is an *internal* hack because dijkstra_item_mover doesn't
take a context arg!  It should be used with care.

Easy, since all the accessors exist: we just hand in the struct dijkstra.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 12:58:56 +09:30
Lagrang3 79486c1e3b renepay: fixups after comments
- remove internal gheap checks
- add check for arc_t.chanidx overflow
- remove outdated comments
- check the delta flow bounds before augmenting along a path
- get_flow_paths uses a dynamic tal array instead of a list.
- fix a unit test that depended on the order of returned flows
- fix bug: lightnind doesn't like if I reuse the partid of a failed
  flow, therefore use a higher partid than any of the previous attempts.
- plugin_err instead of LOG_BROKEN if sendpay fails and we cannot get a
  an error code.
- fix wrong comments.
- remove the background timer.
- This is a bugfix. Previous to this the MCF network was built using the
knowledge of the min and max liquidity but it didn't take into account
pending HTLCs.
- Also remove the min_prob_success option but hardcode a 90% value.

Removing some options that are not relevant to the user, they're kept
for developer mode only:
- base_fee_penalty
- min_prob_success
- prob_cost_factor
- remove heap.h, not used

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Lagrang3 d46990d301 renepay: add information decay linear in time
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell b8ca193606 renepay: an experimental payment plugin
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
Changelog-Added: Plugins: `renepay`: an experimental pay plugin implementing Pickhardt payments (`renepay` and `renepaystatus`).
2023-07-31 12:58:56 +09:30
Shahana Farooqui 637f314bc6 plugin/clnrest: Improved readme and comments
- Adding runes info in readme
- Added extra comments in the code
- System exit on shutdown
- Replaced single quotes with double quotes

Changelog-None
2023-07-30 16:50:04 +09:30
Shahana Farooqui 3fab4cc4f3 plugin/clnrest: Updating package
- Adding package initializer for future proofing
- Updated Pyln version
2023-07-30 16:50:04 +09:30
niftynei 73a67fbb3e funder: don't re-reserve utxos on retries
This way unreserving the PSBT will work as intended, and we don't have
to keep track of how many times we've called reserved for any one input.

Technically we're supposed to not reserve inputs at *all* while doing
opens, this moves us slightly closer to that.
2023-07-30 15:20:04 +09:30
Christian Decker 5f2072869a pay: Fix a null-dereference introduced in #6428
The alias may not be set for non-alias channels after they
confirm. The other branch is safe because we only consider active
channels.

Changelog-None
Fixes #6450
2023-07-28 10:06:43 +09:30
Rusty Russell b2c1ef73cb build: avoid printing PLUGIN_BASES on every build.
Debugging left over in 94205c094f.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 09:44:58 +09:30
Rusty Russell 0a727ae6c5 plugins/bcli: plug temporary leak on retry.
Caught by leak detection, we just re-assigned this when we retried: sure,
it's temporary, but it's technically a leak.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30
Rusty Russell 7ddebada90 plugins/bcli: fix leak report when bitcoind goes away.
I shut down bitcoind during a test, and bcli leak reports flooded in.
They're all temporary, but this fixes them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30
Rusty Russell 620135335d libplugin/pay: allow shortcut for self-pay.
This is the simplest solution, not the best, but there's significant risk in try to remove the "we have a path" assumption in the code pay code.

Includes removing a `tal_steal` which was incorrect: the buffer has the same lifetime as the plugin, so if we steal it then things get messy when we free the  struct payment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `pay` will now pay your own invoices if you try.
2023-07-26 06:55:48 +09:30
Rusty Russell 981d82c406 common/utils: add tal_strdup_or_null helper.
It's not that uncommon to want to pass NULL through, for optional strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:55:48 +09:30
Rusty Russell b6d347a6d3 db: add optional column string helper, and make db_col amounts return sat/msat.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:55:48 +09:30
Shahana Farooqui 911d2c117e plugins/clnrest: Websocket Server
- Added Dependencies
- Updated WS client instructions
- WS Server
2023-07-25 19:20:16 +09:30
Shahana Farooqui 94205c094f plugins/clnrest: Update clnrest dependencies in scripts and Dockerfiles
Adding python3-json5 python3-flask python3-gunicorn flask_restx pyln-client
Also run `pip3 install -r plugins/clnrest/requirements.txt`
2023-07-25 19:20:16 +09:30
Shahana Farooqui 5e3ffc468b plugins/clnrest: removing notifications queue in favor of websocket server 2023-07-25 19:20:16 +09:30
Rusty Russell 2ab0b4af3e plugins/clnrest: disable ourselves unless rest-port is set.
This mirrors grpc's behavior, and avoids listening on ports without
explicit user permission.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 19:20:16 +09:30
Shahana Farooqui abcfda133f plugin: On-boarding default python plugins
plugin: On-boarding default python plugins
2023-07-25 19:20:16 +09:30
Rusty Russell a6701bf180 plugins/clnrest: fail gracefully if dependencies not installed.
json and sys are Python built-in packages, so we can do this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 19:20:16 +09:30
Shahana Farooqui 21160aa6a7 plugin: clnrest
plugin: clnrest
2023-07-25 19:20:16 +09:30
Rusty Russell a219c99691 commando: pass through commands to lightningd.
The in-core commands are much more helpful with details on why runes failed, too!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell 1f791e29eb commando: remove now-unused internal checking routines.
Separate patch to make the previous diff smaller.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell 38075a95d4 commando: use CLN's checkrune() instead of our own for for rune validation.
This means (temporarily) that blacklisting won't work (fix later), and
means that old-style (commando.py) master-secret-override doesn't work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: `commando` no longer allows datastore ['commando', 'secret'] to override master secret (re-issue runes if you were using that!).
2023-07-25 14:47:12 +09:30
Rusty Russell 3e4c0103a5 plugins/commando: make struct cond_info self-contained, split try_command()
In preparation for going async:
1. Split try_command's tail into a new function called execute_command() after
   the rune checks have succeeded.
2. Put all the info execute_command() needs into struct cond_info, to make it
   a simple callback style.

So we create new_cond_info() which dynamically allocates `struct cond_info`
and sets the destructor.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell b16932697d plugins/commando: try_command can reuse incoming `struct commando`.
We would create a `struct commando` to marshal our incoming messages,
then try_command would create a *new* one.  We can simply reuse, but
when I did I noticed a trick: the new one was not in the `incomings`
array, so didn't work towards the ratelimit.  So we need to remove it
from `incomings` in `try_command`, but at least it's now explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell 8db148871d plugins/commando: create new_commando() allocator, use it.
We allocate one in three places, so at least a partial constructor
is a nice pattern to have.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell cd16690af8 commando: always active.
We used to activate on the first rune creation, but we're no longer in charge
of runes, so we can't make that call.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Christian Decker 2549dd6ea5 pay: Annotate both local alias and real scid with channel hints
Debugging a number of payments showed that we sometimes waste a number
of attempts routing through a channel via its alias, rather than its
scid. This is because while we annotate the scid when it has been set,
we do not do so for the alias. The alias then is picked for routing
despite not having enough capacity, failing the attempt locally.

It can also happen that we alternate between scid and alias, doubling
the number of failed attempts before we can make progress.

This patch sets the hint for the alias to a capacity of 0 and disables
it as if the peer were offline. This means when available we'll always
use the scid, which is also far easier to read in the logs.

Changelog-Fixed: pay: We now track spendable amounts when routing on both the local alias as well as the short channel ID
2023-07-24 09:05:40 +09:30
Christian Decker 22462e1d91 libplugin: Remove presplitter from the API 2023-07-21 07:13:48 +09:30
Christian Decker b768804734 pay: Remove the presplitter modifier
The presplitter modifier would split a payment before trying the first
attempt based on some common sizes. Its goal was to have smaller parts
in flight over different paths, in order to make it more difficult for
a forwarding node to learn payment amount. However it was causing some
issues for direct payments, and estimates on spendable amounts which
considers only the first HTLC being added, but presplitter would
always cause multiple HTLCs to be kicked off, causing the estimate to
be off.

Removing the presplitter fixes this, making draining channels easier,
and worse success rates, due to more HTLCs in flight directly
impacting the changes of getting stuck.

Changelog-Removed: pay: `pay` no longer splits based on common size, as it was causing issues in various scenarios.
2023-07-21 07:13:48 +09:30
Shahana Farooqui 93a3d7f632 commando: added check for empty params
Changelog-Changed: Protocol: commando commands now allow a missing params field, instead of requiring an empty field.

No-schema-diff-check
2023-07-19 05:40:30 +09:30
Rusty Russell 0bcff1e76d db/bindings: now bindings are always in order, remove index.
Simply always bind the next one.  No arithmetic required now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-18 14:24:01 +09:30
Rusty Russell 9af407a47a bkpr: use BIND_NEXT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-18 14:24:01 +09:30
Rusty Russell b7b3cbc84a db: enforce that bindings be done in order.
This is almost always true already; fix up the few non-standard ones.

This is enforced with an assert, and I ran the entire test suite to
double-check.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-18 14:24:01 +09:30
Vincenzo Palazzo 5f6642a6ff fix(jsonrpc): trim the lightning: prefix from invoice
Previously, our code checked for the presence of the `lightning:`
prefix while decoding a bolt11 string. Although this prefix is valid
and accepted by the core lightning pay command, it was causing issues
with how we managed invoices. Specifically, we were skipping the prefix
when creating a copy of the invoice string and storing the raw invoice
(including the prefix) in the database, which caused inconsistencies
in the user experience.

To address this issue, we need to strip the `lightning:` prefix before
calling each core lightning command. In addition, we should
modify the invstring inside the db with the canonical one.

This commit fixes the issue by stripping the `lightning:` prefix
from the `listsendpays` function, which will improve the
user experience and ensure consistency in our invoice management (see
next commit).

Reported-by: @johngribbin
Link: ElementsProject#6207
Fixes: debbdc0
Changelog-Fixes: trim the `lightning:` prefix from invoice everywhere.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-07-14 15:04:26 +09:30
Rusty Russell 4d507065f3 libplugin: support wildcard subscriptions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-13 18:59:37 +09:30
Greg Sanders 4b70736d13 Support p2tr deposit addresses
Changelog-Added: JSON-RPC: newaddr: p2tr option to create taproot addresses.
Changelog-Changed: Wallet: we now use taproot change addresses.
2023-07-11 11:41:15 +09:30
Christian Decker f09871d739 cln-plugin: Remove unused `nonnumericids` field in `Builder`
It wasn't used, and we always return `true` in the manifest, because
we support them.
2023-07-10 18:22:11 +02:00
Rusty Russell b197b713be bcli: don't feed CLN massive feerates.
Stay classy, signet!

Reported-by: Anthony Towns
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-10 13:37:22 +02:00
Rusty Russell 99b885c84c plugins: update for new deprecation API where all fields are nicely wrapped.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-10 14:51:11 +09:30
Rusty Russell 0c4426a349 lightningd: remove deprecated_apis global, put into lightningd.
We usually have access to `ld`, so avoid the global.

The only place generic code needs it is for the json command struct,
and that already has accessors: add one for libplugin and lightningd
to tell it if deprecated apis are OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-09 16:49:48 +09:30
Rusty Russell 04f485aee0 db: add generic warn/error function.
This avoids the mess where we override db_fatal for teqsts, and keeps it
generic.

Also allows us to get rid of one #if DEVELOPER, and an ugly global for
bookkeeper.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-09 16:49:48 +09:30
Rusty Russell bef137209d libplugin: expose plugin_logv.
If you have your own logging infra, you'll want this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-09 16:49:48 +09:30
Rusty Russell 3e57496af9 plugins: don't let multifundchannel open a new anchor channel without reserves.
If we're opening a channel with a peer which support anchors (and
we do), we tell fundpsbt/utxopsbt to enforce the emergency reserve;
this matters, as it doesn't know about the channel yet, and thus
won't (if it's our first anchor channel).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `fundchannel` and `multifundchannel` will refuse to spend funds below `min-emergency-msat` if we have any anchor channels (or are opening one).
2023-06-29 11:28:47 -04:00
Rusty Russell 2f0e13e793 multifundchannel: use `excess_as_change` flag to simplify logic.
This was added to fundpsbt/utxopsbt in v0.10, but the spender plugin
didn't take advantage of it, instead calculating its own change amount
and output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell 37005b93d3 txprepare: use `excess_as_change` flag to simplify withdraw/txprepare logic.
This was added to fundpsbt/utxopsbt in v0.10, but the txprepare plugin
didn't take advantage of it, instead calculating its own change amount
and output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell d882a38619 plugin: make sure multifundchannel / fundchannel unreserve correctly on failure.
We were marking our inputs very late, which means any early failure
would not know to unreserve them.

This becomes particularly bad when we start enforcing emergency reserves.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell dd76d60b0d dual-funding: don't override default if multifundchannel doesn't set feerate.
mfc->feerate_str is *never* NULL, since we set it in getfeerate; this is
confusing, as many places check for NULL.

Indeed, the logic in perform_fundpsbt() was *wrong* in this case: it used
`normal` (if it was NULL, which it never was) instead of `opening` to fundpsbt.

And the correct thing is for multifundchannel to not use a string here at
all, but to use the exact feerate it is counting on (even the same
string may have different values now if a block has come in).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-29 11:28:47 -04:00
Rusty Russell ca9a6b15b5 pay: don't require description for hashdesc invoices (i.e. undeprecate).
Since we didn't hash the descriptions properly (see previous commit), we
cannot immediately deprecate omitting the descriptions (since you'd
have to omit them for backwards compat!).

And move the "must have description or hash" test into bolt11.c core.
Changelog-Deprecated: `pay` has *undeprecated* paying a description-hash invoice without providing the description.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-22 13:23:21 +09:30
Rusty Russell 92ff0fd8c0 pay, decodepay: handle descriptions with " inside them where we use hashed descriptions.
This means we need to push off requring this for another full deprecation cycle!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `pay` and `decodepay` with description now correctly handle JSON escapes (e.g " inside description)
2023-06-22 13:23:21 +09:30
Rusty Russell 263625872e autoclean: allow dynamic changes to autoclean-cycle.
Slightly less trivial: reset timer unless it's currently running callback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell f8f8045a06 autoclean: various configuration options now dynamic.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `autoclean` configuration variables now settable with `setconfig`.
2023-06-20 20:08:25 +09:30
Rusty Russell a65f2ef22a plugins: libplugin support for dynamic config options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell aa5c7e763f libplugin: make set callback for options take plugin ptr, check correct type.
I added a plugin arg and was surprised that compile didn't break.
This is because typesafe_cb et al are conditional casts: if the type
isn't as expected it has no effect, but we're passing plugin_option() through
varargs, so everything is accepted!

Add a noop inline to check type, and fix up the two cases where we
used `const char *` instead of `char *`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell ecc030f12d lightningd, libplugins: allocate opt strings from tmpctx, not NULL.
Previously, if these failed we always exited; once we have dymamic
configs this would be a (tiny) memory leak, so use tmpctx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Vincenzo Palazzo 85992e6e48 feat(bitcoind): pass the current known block height
When core lightning is asking the information about
the blockchain with `getchaininfo` command lightningd
know already the information about the min and max block height.

the problem is when we have a smarter Bitcoin backend that is able
to switch between different clients in some cases is helpful
give the information about current known height by lightningd and
pass it down to the plugin.

In this way, the plugin knows what is the correct known height from lightnind, and can
try to fix some problems if any exit.

This is particularly useful when you are syncing a new backend from scratch
like https://github.com/cloudhead/nakamoto and we avoid returning the
lower height from the known, and avoid the crash of core lightning.

With this information, the plugin can start to sync the chain and return
the answer back only when the chain is in sync with the current status of
lightningd.

Another reason to add this field and not wait the correct block in core
lightning itself is because Bitcoin Core is extremely slow to sync up,
so the question here is, how long should we wait? The time depends
on various factors.

With this approach of informing the plugin about the height, in some cases,
you can start the syncing but move the execution to another backend until
the previous one is ready.

The problem I want to solve is that I don't want to be left in the dark when
we run `getchaininfo`, and I want to have the opportunity to wait for
the blockchain sync or decide to dispatch the request elsewhere.

Changelog-Added: Pass the current known block height down to the getchaininfo call.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-06-13 16:27:10 +02:00
Rusty Russell 33be5c8d80 commando: integer command parameters can be compared with < and >.
Previously any attempt would result in "is not an integer field"; we
now recognize valid JSON integers as integer fields.

Changelog-Fixed: Plugins: `commando` runes can now compare integer parameters using '<' and '>' as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-06 19:47:08 +09:30
Matt Morehouse 3edb092bd8 plugins: initialize destination_reachable
Otherwise we later copy the uninitialized memory to descendants,
triggering undefined behavior:

plugins/libplugin-pay.c:2882:34: runtime error: load of value 190, which is not a valid value for type 'bool'
2023-06-05 16:16:21 +02:00
Rusty Russell 9cb2b2f13a listconfigs: show plugin options in 'configs' with normal options.
This integrates them with configvars properly: they almost "just work"
in listconfigs now, and we don't put them in a special sub-object
under their plugin.

Unfortunately, this means `listconfigs` now has a loose schema: any
plugin can add something to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: reloaded plugins get passed any vars from configuration files.
Changelog-Deprecated: Config: boolean plugin options set to `1` or `0` (use `true` and `false` like non-plugin options).
2023-06-03 10:50:29 +09:30
Rusty Russell 1b252f349f config: replace accept-htlc-tlv-types with accept-htlc-tlv-type
We use multi-specifiable options elsewhere, this is just another.
Otherwise you can't add, you can only set them all.

Changelog-Added: Config: `accept-htlc-tlv-type` (replaces awkward-to-use `accept-htlc-tlv-types`)
Changelog-Deprecated: Config: `accept-htlc-tlv-types` (use `accept-htlc-tlv-type` multiple times)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-03 10:50:29 +09:30
Rusty Russell 9b98f9789e bcli: fix type of rpcport.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-03 10:50:29 +09:30
Christian Decker 1e457bd48d rs: Bump version of cln-plugin 2023-06-02 14:47:13 +02:00
Christian Decker 23088911f7 grpc: The plugin needs to use the `server` feature on `cln-grpc` 2023-06-01 13:41:21 +09:30