Commit Graph

12911 Commits

Author SHA1 Message Date
Michael Schmoock 3dde1ca399 pyln-testing: fix wait_for_htlcs helper
When doing the updates on the plugin repo, I discovered that
this helper function got broken by the `listpeerchannels` upgrade.

Its rarely used in the main repo, just at the end of the pyln-testing
'pay' helper.

If unfixed, this bug may result in test flakes when using the `pay`
helper, because its not correctly waiting for all HTLCs to be resolved
before returning.

Changelog-None
2023-02-02 18:47:08 +01:00
Alex Myers 4502340dac lightningd: flag false-positive memleak in lightningd
The leak-detector can't find unconnected_htlcs_in on the stack and
incorrectly flags this as a leak.  However, it is appropriately tal
allocated and freed.

Changelog-None
2023-02-02 08:29:25 +01:00
Rusty Russell 6176912683 plugins/pay: fix htlc_budget calc when we get temporary_channel_failure
Valgrind correctly reports it as uninitialized for this log message, and
the only way this can happen is channel_hints_update() when we receive a
temporary_channel_failure.  Put a dummy value here in this case.

```
Valgrind error file: valgrind-errors.23404
==23404== Conditional jump or move depends on uninitialised value(s)
==23404==    at 0x49E4B56: __vfprintf_internal (vfprintf-internal.c:1516)
==23404==    by 0x49F6519: __vsnprintf_internal (vsnprintf.c:114)
==23404==    by 0x1EBCEB: do_vfmt (str.c:66)
==23404==    by 0x1EBDF8: tal_vfmt_ (str.c:92)
==23404==    by 0x11A336: paymod_log (libplugin-pay.c:167)
==23404==    by 0x11B4B2: payment_chanhints_apply_route (libplugin-pay.c:534)
==23404==    by 0x11E999: payment_compute_onion_payloads (libplugin-pay.c:1707)
==23404==    by 0x11FF4C: payment_continue (libplugin-pay.c:2135)
==23404==    by 0x1245C0: adaptive_splitter_cb (libplugin-pay.c:3800)
==23404==    by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404==    by 0x1205FE: retry_step_cb (libplugin-pay.c:2301)
==23404==    by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404==
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-02 10:11:11 +10:30
Alex Myers 12761c38e3 libwally: update to cln_0.8.5_patch
Improves handling of psbts with multisig inputs.

Changelog-None
2023-02-01 10:50:42 +01:00
Rusty Russell 18397c0b87 doc-schema: make address field in getinfo response required
There were cases where address it's empty, and this cases are not right if the
field is considered optional.
This makes it required and add the field also when `--offline` is set.

Changelog-Changed: JSON-RPC: `getinfo` `address` array is always present (though may be empty)
2023-02-01 13:55:00 +10:30
Michael Schmoock dcb9b4b8d1 make: fix make doc error
This seems to fix issue #5941
Certain make version(?) will parse the ')' as function end when not
inside "" but ''.

Changelog-None
2023-02-01 12:46:17 +10:30
Alex Myers a9eb17adf9 db: catch postgres error on uninitialized database
Avoids the following when postgres returns no query result:
==63458== Conditional jump or move depends on uninitialised value(s)
==63458==    at 0x226A1F: db_postgres_step (db_postgres.c:156)
==63458==    by 0x22535B: db_step (utils.c:155)
==63458==    by 0x1E089A: db_data_version_get (exec.c:49)
==63458==    by 0x194F6F: db_setup (db.c:1029)
==63458==    by 0x199A2F: wallet_new (wallet.c:101)
==63458==    by 0x154B70: main (lightningd.c:1035)

Changelog-None
2023-01-31 10:25:25 +10:30
Peter Neuroth 80250f9b60 datastore: Add check for empty key array
We need to check if the key parameter is an empty array in
`listdatastore` as we do assume an array of at least length 1 in
`wallet.c:5306`.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-01-31 10:23:18 +10:30
Adi Shankara 959678244c doc: remove sections on litestream, .dump and vacuum into
Changelog-None
The sections on SQLite Litestream, sqlite3 .dump and VACUUM INTO commands
were to be removed six months after 0.10.3, due to issues observed in #4857.

We now recommend using --wallet=sqlite3://${main}:${backup} instead.
2023-01-31 10:19:45 +10:30
Rusty Russell fea73680d7 typo fixes found by @niftynei
Also, put the "added" lines in the request schemas for new commands:
this doesn't do anything (yet?) but it keeps `make schema-added-check` happy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 259dd2a652 doc: add examples for sql plugin.
Prompted by @shahanafarooqui's playing with examples and finding
common errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 0240c24936 plugins/sql: listsqlschemas command to retrieve schemas.
Good for detection of what fields are present.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell d8320f015f plugins/sql: add bkpr-listaccountevents and bkpr-listincome support.
This *would* be a 1-line change (add it to Makefile) except that we
previously assumed a "list" prefix on commands.

These use the default refreshing, but they could be done better using
the time-range parameters.

Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9a591277f5 plugins/sql: allow some simple functions.
And document them!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 14aac0769c doc: document the sql command.
In particular, we generate the schema part from the plugin itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
2023-01-30 15:15:41 -06:00
Rusty Russell adb8de3e07 plugins/sql: print out part of man page referring to schemas.
We now add tables to the strmap as we allocate them, since we don't
want to call "finish_td" when we're merely invoked for the
documentation, and don't need a database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 20654ebd49 plugins/sql: pay attention to `deprecated` in schema.
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 40fe893172 doc/schemas: fix old deprecations.
`"deprecated": true` is obsolete; we don't document them anyway.

Where it would have otherwise changed the GRPC wrappers, I actually put the
version number in.

We allow "listchannels" to have "satoshis" since we have some tests
that run in deprecated-api mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9b08c4f25a plugins/sql: refresh listnodes and listchannels by monitoring the gossip_store.
It's quite a lot of code, but these are the most expensive commands we
do so it's worth it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell aa3a1131aa plugins/sql: include the obvious indexes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 68370a203e pytest: perform more thorough testing.
Painfully created by hand from the source.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 8a0ee5f56e plugins/sql: add listpeerchannels support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 51ae7118f1 plugins/sql: make tables for non-object arrays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell c230291141 plugins/sql: rework to parse schemas.
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 260643157d plugins/sql: create `struct column` to encode column details.
Rather than two arrays "columns" (for names) and "fieldtypes" (for
types), use a struct.  This makes additions easier for successive
patches.

Also pull process_json_obj() out of the loop in list_done().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 24d86a85c3 plugins/sql: initial commit of new plugin.
This is designed to allow you to perform complex server-side
queries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 2c41c5d52d doc: use specific types in schema rather than "hex".
We have "secret" and "hash" types which are often more appropriate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell fa127a4071 doc/schemas: remove unnecessary length restrictions.
If we have a specific type (not just "hex") the length is implied.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9589ea0240 common: add routine to get double from JSON.
I don't like it, but we do expose some times like this :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell eb6b8551d4 tools/fromschema.py: don't try to handle more complex cases.
We only handle top-level objects with an array of objects:
make sure it is one before we call the routines.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 153b7bf192 common/gossip_store: move subdaemon-only routines to connectd.
connectd is the only one who uses these routines now.  The
rest can be linked into a plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 7e8b93daa1 common/gossip_store: expose routine to read one header.
This is useful when you're writing routines to scan it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 0274d88bad common/gossip_store: clean up header.
It's actually two separate u16 fields, so actually treat it as
such!

Cleans up zombie handling code a bit too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 83c690fe5f doc: fix listsendpays man page.
We actually had a partid allowed (in the oneOf clauses), but didn''t
document it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell d6f46e2373 lightningd: fix type of listhtlcs payment_hash.
`hash` is a tighter requirement than simply `hex`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 9ab488fc41 plugins/topology: add direction field to listchannels.
It's a core concept in the spec which isn't directly exposed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
2023-01-30 15:15:41 -06:00
Rusty Russell 611795beee listtransactions: get rid of per-tx type annotations.
We didn't actually populate them properly, and the real annotations
are on inputs and outputs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `listtransactions` `channel` and `type` field removed at top level.
2023-01-30 15:15:41 -06:00
Rusty Russell 578f075407 wallet: remove unused TX_ANNOTATION type in transaction_annotations table.
We only ever use this table for output and input transactions: indeed, my node
doesn't have any annotation types 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 4b9cb7eb76 doc: remove unused offerout schema.
We removed the command for v22.11.

Also, we removed the `refund_for` offer parameter, so remove its description
from the manpage.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Alex Myers 98bfd23fff pytest: test_channel_resurrection now succeeds 2023-01-30 16:33:03 +10:30
Alex Myers 1bae8cd28a gossipd: zombify inactive channels instead of pruning
Though BOLT 7 says a channel may be pruned when one side becomes inactive
and fails to refresh their channel_update, in practice, the
channel_announcement can be difficult to recover if deleted entirely.
Here the channel_announcement is tagged as zombie such that gossip_store
consumers may safely ignore it, but it may be retained should the channel
come back online in the future. Node_announcements and channel_updates may
also be retained in such a fashion until the channel is ready to be
resurrected.

Changelog-Fixed: Pruned channels are more reliably restored.
2023-01-30 16:33:03 +10:30
Alex Myers 6bff10cd40 gossip_store: add a flag for zombie entries
This will allow gossipd to store and persist gossip for channels rather
than deleting them entirely when the channels are pruned from the
network.
2023-01-30 16:33:03 +10:30
Alex Myers ed4815527a gossipd: avoid gossipd crash due to double freeing timer 2023-01-30 16:33:03 +10:30
Alex Myers f2f05117aa pytest: gossipd: test resumption of pruned channels 2023-01-30 16:33:03 +10:30
Rusty Russell a3ca3fb047 common/bolt11: fix 32-bit compilation.
Fixes d9fed06b900368e59f4d1f432b87d40fd28ce8d3:

```
common/bolt11.c:868:31: error: format specifies type 'size_t' (aka 'unsigned long') but the argument has type 'u64' (aka 'unsigned long long') [-Werror,-Wformat]
                                           bech32_charset[type], field_len);
                                                                 ^~~~~~~~~
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 16:30:46 +10:30
Rusty Russell 2dec805465 decode: fix handling of blinded_payinfo.
Our pay code handles this correctly, but decode was still using an old model
where there was a payinfo per hop, not per path.

Reported-by: @t-bast
See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell 6eb7a4cbf2 plugins: update to match latest offers text.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-offers" BOLTVERSION=guilt/offers
```

In this case, only trivial mods.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell 9aefe3d40a common: update to latest onion-message spec.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-onion-message" BOLTVERSION=guilt/offers
```

Mainly textual, though I neatened the extra fields check for TLVs with
blinding, and implemented the "no other fields" requirement for
non-final onion message hops.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell 8e630e7c53 common/test: fix up name of test file to match latest version.
Otherwise it's skipped!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell e9eb5f493b common: update to latest route-blinding spec.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-route-blinding" BOLTVERSION=guilt/offers
```

Other than textual changes, this does:

1. Ensures we put total_amount_msat in onion final hop (reported by @t-bast).
2. Require that they put total_amount_msat in onion final hop.
3. Return `invalid_onion_blinding` exactly as defined by the spec (i.e. less
   aggressive when we're the final hop) (also reported by @t-bast, but I already
   knew).

See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (total_amount_sat required, Eclair compat)
2023-01-30 13:24:29 +10:30