Commit Graph

12575 Commits

Author SHA1 Message Date
niftynei 4da0d6230e dual-fund: update to latest, add in updates to rbf amounts
You can now pick a different amount during the RBF phase
2023-02-04 15:31:16 +10:30
Michael Schmoock ad249607d6 dual-fund: update extracted CSVs to latest bolt draft
Changelog-None
2023-02-04 15:31:16 +10:30
Rusty Russell 35acdc112f offers: fix pay where we are using deprecated apis.
In this case, "fee_base_msat" from `listincoming` has `msat` appended.

Fixes: #5850
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 16:59:51 +01:00
Rusty Russell e8f9366a29 pytest: add test for using offers with allow-deprecated-apis=True
Demonstrates that offers doesn't handle msat fields correctly.

Reported-by: @SimonVrouwe
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 16:59:51 +01:00
Michael Schmoock 3efbc12706 pyln-client: adds utils cln_parse_rpcversion
This adds the `cln_parse_rpcversion` helper that is already used in
various plugins to pyln-client, so it does not need to be copied
around anymore.

Changelog-None
2023-02-03 16:58:31 +01:00
niftynei ff0a63a0d7 valgrind-fix: patch valgrind error on log statement in pay plugin
The htlc_budget only exists iff the hint is a 'local' one; we were
failing to write to the htlc_budget field for non-local cases.

To avoid this, we make `local` into a struct that contains the fields
that pertain to local-only payments (in this case, `htlc_budget`).

Valgrind error file: valgrind-errors.1813487
==1813487== Conditional jump or move depends on uninitialised value(s)
==1813487==    at 0x4A9C958: __vfprintf_internal (vfprintf-internal.c:1687)
==1813487==    by 0x4AB0F99: __vsnprintf_internal (vsnprintf.c:114)
==1813487==    by 0x1D2EF9: do_vfmt (str.c:66)
==1813487==    by 0x1D3006: tal_vfmt_ (str.c:92)
==1813487==    by 0x11A60A: paymod_log (libplugin-pay.c:167)
==1813487==    by 0x11B749: payment_chanhints_apply_route (libplugin-pay.c:534)
==1813487==    by 0x11EB36: payment_compute_onion_payloads (libplugin-pay.c:1707)
==1813487==    by 0x12000F: payment_continue (libplugin-pay.c:2135)
==1813487==    by 0x1245B9: adaptive_splitter_cb (libplugin-pay.c:3800)
==1813487==    by 0x11FFB6: payment_continue (libplugin-pay.c:2123)
==1813487==    by 0x1206BC: retry_step_cb (libplugin-pay.c:2301)
==1813487==    by 0x11FFB6: payment_continue (libplugin-pay.c:2123)
==1813487==
{
   <insert_a_suppression_name_here>
   Memcheck:Cond
   fun:__vfprintf_internal
   fun:__vsnprintf_internal
   fun:do_vfmt
   fun:tal_vfmt_
   fun:paymod_log
   fun:payment_chanhints_apply_route
   fun:payment_compute_onion_payloads
   fun:payment_continue
   fun:adaptive_splitter_cb
   fun:payment_continue
   fun:retry_step_cb
[sesh] 0:[tmux]*Z

Suggested-By: @nothingmuch
2023-02-02 18:08:27 -06:00
Rusty Russell 6347ee7308 lightningd: don't run more than one reconnect timer at once.
In various circumstances we can start a reconnection while one is
already going on.  These can stockpile if the node really is unreachable.

Reported-by: @whitslack
Fixes: #5654
Changelog-Fixed: lightningd: we no longer stack multiple reconnection attempts if connections fail.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 09:18:59 +10:30
Rusty Russell ff1d537b87 pytest: neaten test_cln_sendpay_weirdness, rename.
1. Allow 'any' as an option to zeroconf-selective.py plugin so we can use
   it in line_graph where we don't know ids yet.
2. Use modern helpers like line_graph and remove debugging statement.
3. Don't use listchannels(): it's true that it shows local channels as well,
   but that's a quirk I'd like to remove.
4. Make flake8 happy.
5. Rename to be more specific now it's a more narrow test.

I manually tested that the test still failed with the fixes removed, too,
so it is still the same test!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 09:17:33 +10:30
Rusty Russell 9e9686df20 pay: specify the channel to send the first hop.
If we only specify the node_id, we get the "first" channel.

Closes: #5803
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` uses the correct local channel for payments when there are multiple available (not just always the first!)
2023-02-03 09:17:33 +10:30
Rusty Russell ec8aab7cb2 libplugin-pay: fix (transitory) memleak which memleak detection complains about.
We assign this in the loop without freeing it first.

```
 plugin-pay: MEMLEAK: 0x55792b155e18
 plugin-pay:   label=plugins/libplugin-pay.c:3274:struct short_channel_id_dir
 plugin-pay:   backtrace:
 plugin-pay:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
 plugin-pay:     plugins/libplugin-pay.c:3274 (direct_pay_listpeerchannels)
 plugin-pay:     plugins/libplugin.c:860 (handle_rpc_reply)
 plugin-pay:     plugins/libplugin.c:1036 (rpc_read_response_one)
 plugin-pay:     plugins/libplugin.c:1060 (rpc_conn_read_response)
 plugin-pay:     ccan/ccan/io/io.c:59 (next_plan)
 plugin-pay:     ccan/ccan/io/io.c:407 (do_plan)
 plugin-pay:     ccan/ccan/io/io.c:417 (io_ready)
 plugin-pay:     ccan/ccan/io/poll.c:453 (io_loop)
 plugin-pay:     plugins/libplugin.c:1893 (plugin_main)
 plugin-pay:     plugins/pay.c:1294 (main)
 plugin-pay:     ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main)
 plugin-pay:     ../csu/libc-start.c:381 (__libc_start_main_impl)
 plugin-pay:   parents:
 plugin-pay:     plugins/libplugin-pay.c:3308:struct direct_pay_data
 plugin-pay:     plugins/libplugin.c:1775:struct plugin
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 09:17:33 +10:30
Rusty Russell 0cbd9e02de pytest: limit test_cln_sendpay_weirdness to only failures.
We fixed most of them.  Now hone in to the case which fails: `pay`
when it needs to use the direct zero-conf channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 09:17:33 +10:30
Carl Dong 7c7e32b324 tests: add Carl Dong's example exhaustive zeroconf channel pay test.
Modifications from issue #5803 to work here:

1. import json
2. Add xfail
3. Increase channel sizes by 10x so we can open them
4. Fix plugin path

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-03 09:17:33 +10:30
Rusty Russell 38510202c4 pytest: fix flake in test_closing_simple when we mine too fast.
We can actually catch l2 with HTLCs still closing and mine blocks,
then it force closes due to HTLC timeout.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-02 14:56:04 -06:00
Rusty Russell f87c7ed439 plugins/sql: fix foreign keys.
I noticed that our subtables were not being cleaned, despite being "ON
DELETE CASCADE".  This is because foreign keys were not enabled, but
then I got foreign key errors: rowid cannot be a foreign key anyway!

So create a real "rowid" column.  We want "ON DELETE CASCADE" for
nodes and channels (and other tables in future) where we update
partially.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-02 14:56:04 -06:00
Michael Schmoock 3dde1ca399 pyln-testing: fix wait_for_htlcs helper
When doing the updates on the plugin repo, I discovered that
this helper function got broken by the `listpeerchannels` upgrade.

Its rarely used in the main repo, just at the end of the pyln-testing
'pay' helper.

If unfixed, this bug may result in test flakes when using the `pay`
helper, because its not correctly waiting for all HTLCs to be resolved
before returning.

Changelog-None
2023-02-02 18:47:08 +01:00
Alex Myers 4502340dac lightningd: flag false-positive memleak in lightningd
The leak-detector can't find unconnected_htlcs_in on the stack and
incorrectly flags this as a leak.  However, it is appropriately tal
allocated and freed.

Changelog-None
2023-02-02 08:29:25 +01:00
Rusty Russell 6176912683 plugins/pay: fix htlc_budget calc when we get temporary_channel_failure
Valgrind correctly reports it as uninitialized for this log message, and
the only way this can happen is channel_hints_update() when we receive a
temporary_channel_failure.  Put a dummy value here in this case.

```
Valgrind error file: valgrind-errors.23404
==23404== Conditional jump or move depends on uninitialised value(s)
==23404==    at 0x49E4B56: __vfprintf_internal (vfprintf-internal.c:1516)
==23404==    by 0x49F6519: __vsnprintf_internal (vsnprintf.c:114)
==23404==    by 0x1EBCEB: do_vfmt (str.c:66)
==23404==    by 0x1EBDF8: tal_vfmt_ (str.c:92)
==23404==    by 0x11A336: paymod_log (libplugin-pay.c:167)
==23404==    by 0x11B4B2: payment_chanhints_apply_route (libplugin-pay.c:534)
==23404==    by 0x11E999: payment_compute_onion_payloads (libplugin-pay.c:1707)
==23404==    by 0x11FF4C: payment_continue (libplugin-pay.c:2135)
==23404==    by 0x1245C0: adaptive_splitter_cb (libplugin-pay.c:3800)
==23404==    by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404==    by 0x1205FE: retry_step_cb (libplugin-pay.c:2301)
==23404==    by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404==
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-02 10:11:11 +10:30
Alex Myers 12761c38e3 libwally: update to cln_0.8.5_patch
Improves handling of psbts with multisig inputs.

Changelog-None
2023-02-01 10:50:42 +01:00
Rusty Russell 18397c0b87 doc-schema: make address field in getinfo response required
There were cases where address it's empty, and this cases are not right if the
field is considered optional.
This makes it required and add the field also when `--offline` is set.

Changelog-Changed: JSON-RPC: `getinfo` `address` array is always present (though may be empty)
2023-02-01 13:55:00 +10:30
Michael Schmoock dcb9b4b8d1 make: fix make doc error
This seems to fix issue #5941
Certain make version(?) will parse the ')' as function end when not
inside "" but ''.

Changelog-None
2023-02-01 12:46:17 +10:30
Alex Myers a9eb17adf9 db: catch postgres error on uninitialized database
Avoids the following when postgres returns no query result:
==63458== Conditional jump or move depends on uninitialised value(s)
==63458==    at 0x226A1F: db_postgres_step (db_postgres.c:156)
==63458==    by 0x22535B: db_step (utils.c:155)
==63458==    by 0x1E089A: db_data_version_get (exec.c:49)
==63458==    by 0x194F6F: db_setup (db.c:1029)
==63458==    by 0x199A2F: wallet_new (wallet.c:101)
==63458==    by 0x154B70: main (lightningd.c:1035)

Changelog-None
2023-01-31 10:25:25 +10:30
Peter Neuroth 80250f9b60 datastore: Add check for empty key array
We need to check if the key parameter is an empty array in
`listdatastore` as we do assume an array of at least length 1 in
`wallet.c:5306`.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-01-31 10:23:18 +10:30
Adi Shankara 959678244c doc: remove sections on litestream, .dump and vacuum into
Changelog-None
The sections on SQLite Litestream, sqlite3 .dump and VACUUM INTO commands
were to be removed six months after 0.10.3, due to issues observed in #4857.

We now recommend using --wallet=sqlite3://${main}:${backup} instead.
2023-01-31 10:19:45 +10:30
Rusty Russell fea73680d7 typo fixes found by @niftynei
Also, put the "added" lines in the request schemas for new commands:
this doesn't do anything (yet?) but it keeps `make schema-added-check` happy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 259dd2a652 doc: add examples for sql plugin.
Prompted by @shahanafarooqui's playing with examples and finding
common errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 0240c24936 plugins/sql: listsqlschemas command to retrieve schemas.
Good for detection of what fields are present.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell d8320f015f plugins/sql: add bkpr-listaccountevents and bkpr-listincome support.
This *would* be a 1-line change (add it to Makefile) except that we
previously assumed a "list" prefix on commands.

These use the default refreshing, but they could be done better using
the time-range parameters.

Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9a591277f5 plugins/sql: allow some simple functions.
And document them!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 14aac0769c doc: document the sql command.
In particular, we generate the schema part from the plugin itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
2023-01-30 15:15:41 -06:00
Rusty Russell adb8de3e07 plugins/sql: print out part of man page referring to schemas.
We now add tables to the strmap as we allocate them, since we don't
want to call "finish_td" when we're merely invoked for the
documentation, and don't need a database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 20654ebd49 plugins/sql: pay attention to `deprecated` in schema.
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 40fe893172 doc/schemas: fix old deprecations.
`"deprecated": true` is obsolete; we don't document them anyway.

Where it would have otherwise changed the GRPC wrappers, I actually put the
version number in.

We allow "listchannels" to have "satoshis" since we have some tests
that run in deprecated-api mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9b08c4f25a plugins/sql: refresh listnodes and listchannels by monitoring the gossip_store.
It's quite a lot of code, but these are the most expensive commands we
do so it's worth it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell aa3a1131aa plugins/sql: include the obvious indexes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 68370a203e pytest: perform more thorough testing.
Painfully created by hand from the source.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 8a0ee5f56e plugins/sql: add listpeerchannels support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 51ae7118f1 plugins/sql: make tables for non-object arrays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell c230291141 plugins/sql: rework to parse schemas.
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 260643157d plugins/sql: create `struct column` to encode column details.
Rather than two arrays "columns" (for names) and "fieldtypes" (for
types), use a struct.  This makes additions easier for successive
patches.

Also pull process_json_obj() out of the loop in list_done().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 24d86a85c3 plugins/sql: initial commit of new plugin.
This is designed to allow you to perform complex server-side
queries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 2c41c5d52d doc: use specific types in schema rather than "hex".
We have "secret" and "hash" types which are often more appropriate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell fa127a4071 doc/schemas: remove unnecessary length restrictions.
If we have a specific type (not just "hex") the length is implied.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9589ea0240 common: add routine to get double from JSON.
I don't like it, but we do expose some times like this :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell eb6b8551d4 tools/fromschema.py: don't try to handle more complex cases.
We only handle top-level objects with an array of objects:
make sure it is one before we call the routines.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 153b7bf192 common/gossip_store: move subdaemon-only routines to connectd.
connectd is the only one who uses these routines now.  The
rest can be linked into a plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 7e8b93daa1 common/gossip_store: expose routine to read one header.
This is useful when you're writing routines to scan it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 0274d88bad common/gossip_store: clean up header.
It's actually two separate u16 fields, so actually treat it as
such!

Cleans up zombie handling code a bit too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 83c690fe5f doc: fix listsendpays man page.
We actually had a partid allowed (in the oneOf clauses), but didn''t
document it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell d6f46e2373 lightningd: fix type of listhtlcs payment_hash.
`hash` is a tighter requirement than simply `hex`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9ab488fc41 plugins/topology: add direction field to listchannels.
It's a core concept in the spec which isn't directly exposed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
2023-01-30 15:15:41 -06:00