Commit Graph

1318 Commits

Author SHA1 Message Date
Rusty Russell d3e33cfd14 wallet: fix crash on listtransactions.
We removed the (experimental-only!) annotation output in 611795beee
but we still loaded them from the db.  Turns out that we were putting bogus
annotations into the db, and accessing out of range when loading them.

Consider the following db entry in transaction_annotations:

```
CREATE TABLE transaction_annotations (  txid BLOB, idx INTEGER, location INTEGER, type INTEGER, channel INTEGER REFERENCES channels(id), UNIQUE(txid, idx));
...
INSERT INTO transaction_annotations VALUES(X'19706f9af2875508a06c7db1754ef7ecb3da745ead005992e626441e4e83465f',18,1,129,53699);
```

Here is the corresponding entry in txs:

```
INSERT INTO transactions VALUES(X'19706f9af2875508a06c7db1754ef7ecb3da745ead005992e626441e4e83465f',710327,966,X'02000000000101f2add69112a1d557317826120e1f4ea3bc1cbe4674d720325695b26ecfe8355d120000000000000000013634000000000000160014dca21f104359bbb81e88ed7da985549f2cd0cbc30347304402201cdc854b76c4c7523e3ca09f38a81539d3b2f7fbd9a0de6ae10b7ceaa65ed9d402205a1770058cd1ef081c77c2fe957c07a334cb3a11bc0cc502834a29c59424fe010120589da1f809d955c7af150bf53123c27ffc0a741489b5291f6be811189863ec838576a9142f188d0d973c4ad1865a619d3748340b30746e088763ac672103b7bbcd592197ba6501e7176aabd3f908d94b126ae82ab1e7a4c58f5a789782e57c820120876475527c21029bcf62114eb36758fcb1aead7e67b6f707d32f34e67816894d5211ac9f2d6ce752ae67a9144483a115219ba65c63a3844be8445f739703bea988ac686800000000',NULL,NULL);
```

The annotation refers to output 18 of the tx, but it only has one output!

However, decoding the tx shows that it spent output 18 of a previous tx, so
that's probably where the `18` came from.

Remove this logic: we can remove the remaining (clearly broken!) annotation
adding code in another cleanup commit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-05-29 18:41:46 +09:30
Rusty Russell b8aa3a579e listtransactions: remove annotations, which were only in EXPERIMENTAL_FEATURES.
I don't know if anyone was using them, they seem half-hearted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-05-23 09:34:08 +09:30
Rusty Russell 7acaccfb36 wallet: add channel_type field to db.
This was always the intent, but now we have to reconstruct from the
disparate fields.

This means `option_anchor_outputs` is now redundant.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell f1deeda123 wallet: allow psbt_using_utxos to take a starter psbt.
It will append inputs to this PSBT instead of allocating a new one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell 57b2cbcb32 lightningd: expose default_locktime for wider usage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 17:26:47 +09:30
Rusty Russell df9552bcc1 db: make db_exec_prepared_v2 return void.
It calls db_fatal() if it fails anyway, so don't expect anyone to check.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:50:32 +09:30
Rusty Russell 9bcf28afb3 db: catch SQL errors unless we're expecting them.
I couldn't figure out why my new SQL query was returning 0 rows,
and it was because we were ignoring errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:50:32 +09:30
Rusty Russell f2f02f9de6 chaintopology: allow minblock for broadcast_tx.
Fun story.  We're changing onchaind to hand txs to us, and we will
construct them and do the broadcast for it.  lightningd tells onchaind
the witness it used (with flags to indicate which fields were
signatures so should be ignored) so onchaind can recognize the tx
when/if it is mined.

And when onchaind was waiting for a CLTV delay, it wouldn't tell
lightningd yet, but wait until the parent was sufficiently deep

But this caused bugs!

In particular, on replay, onchaind would see transactions which it
hasn't sent yet.  This was not a problem before, as onchaind had
created the tx, even if it hadn't told lightningd to broadcast it, so
recognized the variant when it came in.  When we're relying on
lightningd to tell us what the tx will look like, this doesn't work
any more.

The cause of this is that we fire off txowatches ("this output was
spent!") while we process blocks, and only fire off txwatches ("this
tx increased depth") once all the current blocks are processed.  Often
this didn't matter, since we replay messages to onchaind from the
database, *but* we trim the last few blocks on restart (or, if there's
a small reorg while we're stopped), and we can hit this misordering.

Changing our topology code to only ever process one block at a time
would be a solution, but slows down catchup (and tests, where we often
mine a run of blocks).

So, this seems like a premature optimization, but it's really
required!  And in future, lightningd can use this knowledge of pending
transactions to combine them in more clever ways.

Note that if a tx is valid at block N, we broadcast it once we see
block N-1, to get it in the mempool for block N.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:01:48 +09:30
Rusty Russell fc54c19716 lightningd: provide callback in broadcast_tx() for refreshing tx.
We'll use this to do RBF.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:01:48 +09:30
Rusty Russell 528f44c2d3 bitcoin: helpers to clone a bitcoin_tx, and format one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:01:48 +09:30
Vincenzo Palazzo b92b9f074d delpay: delete the payment by status from the db
There are cases (difficult to reproduce with a test) where
a payment will fail one time and succeed later.

As far I understand in this case the groupid field of the payment
is the same, and the only thing that change is the status, so
our logic inside the delpay is ambiguous where it is not
possible to delete a payment as described in https://github.com/ElementsProject/lightning/issues/6114

A sequence of commands that explain the problem is

```
$ lc -k listpays payment_hash=H
{
   "pays": [
      {
         "bolt11": "I",
         "destination": "redacted",
         "payment_hash": "H",
         "status": "complete",
         "created_at": redacted,
         "completed_at": redacted,
         "preimage": "P",
         "amount_msat": "redacted",
         "amount_sent_msat": "redacted"
      }
   ]
}
$ lc delpay H complete
{
   "code": 211,
   "message": "Payment with hash H has failed status but it should be complete"
}
```

In this case, the delpay is not able to delete a payment because the
listpays is returning only the succeeded one, so by running the
listsendpays we may see the following result where our delpay logic
will be stuck because it works to ensure that all the payments stored
in the database has the status specified by the user

```
➜  VincentSSD clightning --testnet listsendpays -k payment_hash=7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4
{
   "payments": [
      {
         "id": 322,
         "payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
         "groupid": 1,
         "partid": 1,
         "destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
         "amount_msat": 300,
         "amount_sent_msat": 1664,
         "created_at": 1679510203,
         "completed_at": 1679510205,
         "status": "failed",
         "bolt11": "lntb1pjpkj4xsp52trda39rfpe7qtqahx8jjplhnj3tatxy8rh6sc6afgvmdz7n0llspp50lr5hmdm0re0xvcp2hv3nf2wwvx0r8q3h3e7jmqz0awdfg6w206qdp0w3jhxarfdenjqargv5sxgetvwpshjgrzw4njqun9wphhyaqxqyjw5qcqp2rzjqtp28uqy77te96ylt7ek703h4ayldljsf8rnlztgf3p8mg7pd0qzwf8a3yqqpdqqqyqqqqt2qqqqqqgqqc9qxpqysgqgeya2lguaj6sflc4hx2d89jvah8mw9uax4j77d8rzkut3rkm0554x37fc7gy92ws9l76yprdva2lalrs7fqjp9lcx40zuty8gca0g5spme3dup"
      },
      {
         "id": 323,
         "payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
         "groupid": 1,
         "partid": 2,
         "destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
         "amount_msat": 300,
         "amount_sent_msat": 3663,
         "created_at": 1679510205,
         "completed_at": 1679510207,
         "status": "failed"
      },
      {
         "id": 324,
         "payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
         "groupid": 1,
         "partid": 3,
         "destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
         "amount_msat": 300,
         "amount_sent_msat": 3663,
         "created_at": 1679510207,
         "completed_at": 1679510209,
         "status": "failed"
      },
      {
         "id": 325,
         "payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
         "groupid": 1,
         "partid": 4,
         "destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
         "amount_msat": 300,
         "amount_sent_msat": 4663,
         "created_at": 1679510209,
         "completed_at": 1679510221,
         "status": "complete",
         "payment_preimage": "43f746f2d28d4902489cbde9b3b8f3d04db5db7e973f8a55b7229ce774bf33a7"
      }
   ]
}
```

This commit solves the problem by forcing the delete query in the
database to specify status too, and work around this kind of
ambiguous case.

Fixes: f52ff07558 (lightningd: allow delpay to delete a specific payment.)
Reported-by: Antoine Poinsot <darosior@protonmail.com>
Link: https://github.com/ElementsProject/lightning/issues/6114
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Co-Developed-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: delpay be more pedantic about delete logic by allowing
delete payments by status directly on the database.
2023-04-05 06:15:47 +09:30
Rusty Russell 2cb96a8d77 wallet: don't silently load invalid last_tx psbts.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-31 09:16:25 +10:30
Rusty Russell 5bb0270492 wallet: fix up PSBTs as a migration.
In the now-misnamed "last_tx" field.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-31 09:16:25 +10:30
Rusty Russell f1fa75fa06 wallet/test/run-psbt_fixup.c: test for psbt fixups.
Should do nothing to normal ones, but fix up old invalids ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-31 09:16:25 +10:30
Rusty Russell 7174d06a70 wallet/psbt_fixup: routine to fix invalid PBSTs which modern libwally won't load.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-31 09:16:25 +10:30
Rusty Russell 4b6e9649eb wallet: add accessor for closed channels.
This doesn't restore every bit of information we have, but it does
contain the important ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell 6e1eafbb0b wallet: make it clear that `enum state_change` is in db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell f720e0ff0b wallet: use db_col_optional.
We don't cover three common patterns:
1. Optional integers (db_col_u64 has different form from structs)
2. Optional strings.
3. Optional array fields.

But it does neaten and reduce the scope for cut&paste errors in the
common "if not-NULL, tal and assign".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell d9e274cee2 db_bind_scid: rename to db_bind_short_channel_id
We used to have a text version, so this was named 'scid'.  Fix it now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell ae861d1793 wallet: don't clear reference from channel to peers table when we close channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell 09011177a8 wallet: only delete peer from db if it's unused.
This relaxes the assertion that it won't be used, and renames the
function to be clear.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell 98d425f1f4 wallet: add comment on db noting that `ON DELETE CASCADE` is never used.
We actually have an assertion that there are no channels remaining when
we delete peers, so this is confusing!

Actually removing the constraint is db-specific and deeply non-trivial.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Greg Sanders 887c6f71cf Add PSBT version setting RPC to aid with debugging and compatibility
PSBTv2 support is quite low in the ecosystem, so having a call to convert
log messages and the like should be useful since they'll often be in v2.

Changelog-Added: Added setpsbtversion RPC to aid debugging and compatibility
2023-03-23 16:10:55 +10:30
Greg Sanders 908f834d66 Update libwally to 0.8.8, support PSBTv2
Libwally update breaks compatibility, so
we do this in one large step.

Changelog-Changed: JSON-RPC: elements network PSET now only supports PSETv2.
Changelog-Added: JSON-RPC: PSBTv2 supported for fundchannel_complete, openchannel_update, reserveinputs, sendpsbt, signpsbt, withdraw and unreserveinputs parameter psbt, openchannel_init and openchannel_bump parameter initialpsbt, openchannel_signed parameter signed_psbt and utxopsbt parameter utxopsbt
2023-03-23 16:10:55 +10:30
Greg Sanders 5eddf3cd73 test: add PSBT field that doesn't collide with PSBTv2 fields
Which gets libwally upset post-update to 0.8.8
2023-03-23 16:10:55 +10:30
Rusty Russell df085a8a87 wallet/db: don't use migration_context.
`struct lightningd` is not completely initialized, so we added a
"migration_context" which only had some of the fields.  But we ended
up handing in `struct lightningd` anyway, so I don't think this
complexity is worthwhile.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-22 13:50:32 +10:30
Rusty Russell 3db3dc946f lightningd: move bip32_pubkey here from common/, add hsm check.
At the moment only lightingd needs it, and this avoids missing any
places where we do bip32 derivation.

This uses a hsm capability to mean we're backwards compatible with older
hsmds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: we now always double-check bitcoin addresses are correct (no memory errors!) before issuing them.
2023-03-22 13:50:32 +10:30
Rusty Russell 3f02797e88 lightningd: move bip32_base pointer into struct lightningd.
It's needed as the db and wallet is being set up (db migrations), so
it's simpler this way to always use ld->bip32_base for the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-22 13:50:32 +10:30
Rusty Russell e02f5f5bb8 hsmd: new version, which tells us the HSM version, and capabilities.
Importantly, adds the version number at the *front* to help future
parsing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


Header from folded patch 'fix-hsm-check-pubkey.patch':

fixup! hsmd: capability addition: ability to check pubkeys.
2023-03-22 13:50:32 +10:30
Kristaps Kaupe 906279a46e Output channel_id in listfunds
Changelog-Added: JSON-RPC: `listfunds` now has a `channel_id` field.
2023-03-21 10:29:33 +00:00
Rusty Russell 9366e6b39f cleanup: rename json_add_amount_msat_only to json_add_amount_msat
Now there's no compat variant, we can rename this function.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-18 15:55:49 +10:30
Rusty Russell 780f32dfc6 global: remove deprecated non-msat-named msat fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: all the non-msat-named millisatoshi fields deprecated in v0.12.0.
2023-03-18 15:55:49 +10:30
Rusty Russell 6c4a438afd wallet: really allow broken migrations.
e778ebb9af ("wallet: only log broken if we
have duplicate scids in channels.") downgraded the fatal() to a broken
log message, but the user reports it still won't start up.

Perhaps they're hitting the fatal() outside the loop?  (And we're
not getting that output).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-08 16:20:09 -06:00
niftynei e315f30728 db-fix: update NULL lease_satoshi fields to zero
Missed a DEFAULT in the db clause.

Feb 15 16:02:12 citrine lightningd[902093]: Accessing a null column lease_satoshi/15 in query SELECT  funding_tx_id, funding_tx_outnum, funding_feerate, funding_satoshi, our_funding_satoshi, funding_psbt, last_tx, last_sig, funding_tx_remote_sigs_received, lease_expiry, lease_commit_sig, lease_chan_max_msat, lease_chan_max_ppt, lease_blockheight_start, lease_fee, lease_satoshi FROM channel_funding_inflights WHERE channel_id = ? ORDER BY funding_feerate

Fixes #6016
2023-02-15 18:08:25 -06:00
Vincenzo Palazzo a610f28ad4 add a log message when it is not possible upgrade the db
People are upgrading to 22.11.1 not, and in some configurations like the one
mentioned in the issue, we should
put some info information in the log when we are not able to upgrade.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-02-09 17:40:58 -06:00
niftynei beec517910 df: persist our setting to disk, read back to dualopend at reinit
It's not likely but possible that the node's settings will shift btw a
start and an RBF; we persist the setting to the database so we don't
lose it.

Right now holding onto it forever is kind of extra but maybe we'll
reuse the setting for splices? idk.

Should this be a channel type??
2023-02-07 21:03:36 -06:00
niftynei f05d450098 df: persist channel open preference to database
technically we don't need this info after the channel opens, but for any
subsequent RBF (and maybe splice?) we need to remember what the
open/accept peer signaled
2023-02-07 21:03:36 -06:00
niftynei 679a473f9a fundpsbt: add option to filter out wrapped p2sh inputs
We need to be able to only use non-wrapped inputs for v2/interactive tx
protocol.

Changelog-Added: JSONRPC: `fundpsbt` option `nonwrapped` filters out p2sh wrapped inputs
2023-02-04 15:31:16 +10:30
niftynei 4c46750001 dual-open-rbf: remember the requested lease amount btw restarts
Don't forget the requested lease across restarts.
2023-02-04 15:31:16 +10:30
Rusty Russell 611795beee listtransactions: get rid of per-tx type annotations.
We didn't actually populate them properly, and the real annotations
are on inputs and outputs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `listtransactions` `channel` and `type` field removed at top level.
2023-01-30 15:15:41 -06:00
Rusty Russell 578f075407 wallet: remove unused TX_ANNOTATION type in transaction_annotations table.
We only ever use this table for output and input transactions: indeed, my node
doesn't have any annotation types 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell 0e25d56329 lightningd: use a hash table for peer->dbid.
Otherwise, loading up when we have 100k peers is *painful*!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell cfa632b0e9 lightningd: use hash map for peers instead of linked list.
After connecting 100,000 peers with one channel each (not all at
once!), we see various places where we exhibit O(N^2) behaviour.

Fix these by keeping a map of id->peer instead of a simple
linked-list.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell 0faa8397c3 wallet: add dependency on lightningd/ headers.
wallet should really be a subdir of lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Vincenzo Palazzo 8f94e8b943 comm: make sure that our version check is reliable
Rework the logic of the version check used in the
database migration, and make sure
that it is full functional to avoid confusion
at release time.

Changelog-Fixed: database: Correctly identity official release versions for database upgrade.

Reported-by: @urza
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-01-17 14:15:24 +10:30
Rusty Russell 932ca9e91f lightningd: deprecate p2sh-segwit addresses for `newaddr` `addresstype`
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `newaddr`: `addresstype` `p2sh-segwit` (use default, or `bech32`).
2023-01-14 21:17:45 +10:30
Rusty Russell 1d8b899551 lightningd: prepare internal json routines for listpeerchannels.
We're soon going to call json_add_unsaved_channel and
json_add_uncommitted_channel from a new place, where we want the peer
state directly included.

Based on patch by @vincenzopalazzo.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-13 10:42:42 +10:30
Rusty Russell 5dfcd15782 all: no longer need to call htable_clear to free htable contents.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-12 11:44:10 +10:30
Rusty Russell 763d02e424 lightningd: ensure htlc htables are always tal objects.
We want to change the htable allocator to use tal, which will need
this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-12 11:44:10 +10:30
Christian Decker 845503f72e db: Fix the ordering of `channel_htlcs` in postgres
I discovered this while reworking the CI workflow, and it seems like
the HTLC queries do not order, while the tests assume a specific
order. This matches sqlite3 which without an explicit ORDER clause
will use insertion order, while postgres does not keep things in
insertion order, thus breaking the assumption. Ordering by `id`
re-establishes that implicit assumption

Changelog-Changed: postgres: Ordering of HTLCs in `listhtlcs` are now ordered by time of creation
2023-01-02 14:32:41 +01:00