Commit Graph

10553 Commits

Author SHA1 Message Date
Rusty Russell fcf3d0ce6c db: turn generated queries array into a simple hash table.
Since we have that functionality, let's use it.  Also, make table
const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 53c9d9853d wallet: remove db_column_ functions.
Keep some as internal helpers only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 2e6df89274 wallet: remove redundant field counters, ignored fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell c462ccae1a wallet: have db track what columns are accessed in DEVELOPER mode.
And add db_col_ignore helper for cases where it's deliberate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 7382616513 CCAN: import strset.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 663c8c6c02 wallet: convert db internal routines to db_col_
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell b8a240bd15 wallet: convert wallet.c to db_col_ APIs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell c2a4285a7c wallet: use db_col_ accessors in wallet/invoices.c
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell befab73070 db: improve db_col_* APIs.
1. db_col_text becomes db_col_strdup, which is what is usually wanted.
2. db_col_short_channel_id becomes db_col_short_channel_id_str, to emphasize
   that it stores in string form.  Modern versions should store u64.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 5b482eb04b db: db_col_ variants for accessing SELECT statements by name.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 787fbb1228 db: create simple hashtable of fields in SELECT.
This simplistically maps names to numbers, eg:

	SELECT foo, bar FROM tbl;

'foo' -> 0
'bar' -> 1

If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 57328fe59e tests: use common_setup/common_shutdown to avoid leaks.
It also does more checks (like taken() checks).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 68043c2e8c common: clean up autodata in common_shutdown().
valgrind locally complains about the allocations in autodata leaking:

```
==138200== 16 bytes in 1 blocks are still reachable in loss record 1 of 2
==138200==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==138200==    by 0x10D41A: autodata_register_ (autodata.c:20)
==138200==    by 0x10E7B8: register_autotype_type_to_string (type_to_string.h:79)
==138200==    by 0x10F5CA: register_one_type_to_string0 (block.c:259)
==138200==    by 0x19734C: __libc_csu_init (in /home/rusty/devel/cvs/lightning/common/test/run-route-specific)
==138200==    by 0x4A3D03F: (below main) (libc-start.c:264)
==138200== 
==138200== 176 bytes in 1 blocks are still reachable in loss record 2 of 2
==138200==    at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==138200==    by 0x10D472: autodata_register_ (autodata.c:26)
==138200==    by 0x122D37: register_autotype_type_to_string (type_to_string.h:79)
==138200==    by 0x122F1F: register_one_type_to_string0 (node_id.c:50)
==138200==    by 0x19734C: __libc_csu_init (in /home/rusty/devel/cvs/lightning/common/test/run-route-specific)
==138200==    by 0x4A3D03F: (below main) (libc-start.c:264)
==138200== 
make: *** [Makefile:638: unittest/common/test/run-route-specific] Error 7
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 82c66b10df pytest: allow bad gossip in pruning test.
If we forget a channel, we can get upset when we get an update about it:

```
2021-11-04T00:35:43.8242370Z lightningd-3: 2021-11-04T00:29:22.073Z DEBUG   gossipd: Pruning channel 103x1x1 from network view (ages 61 and 22s)
...
2021-11-04T00:35:43.8263502Z lightningd-3: 2021-11-04T00:29:22.509Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x1/0
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 67f9a8e4d6 contrib: fix other pyln-spec package requirements too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:50:54 +10:30
Jonas Nick 6409608696 pyln-bolt7: fix requirements.txt 2021-11-17 10:50:54 +10:30
Rusty Russell cd36436179 paystatus: remove doubled amount_msat.
It's always been there:

```
{
   "pay": [
      {
         "bolt11": "...",
         "amount_msat": "1000msat",
         "amount_msat": "1000msat",
         "destination": "03...",
         "attempts": [
            {
               "strategy": "Initial attempt",
               "start_time": "2021-11-06T04:20:20.135Z",
               "age_in_seconds": 229032,
               "end_time": "2021-11-06T04:20:27.792Z",
               "state": "completed",
               "success": {
                  "id": 31994,
                  "payment_preimage": "..."
               }
            }
         ]
      }
   ]
}
```

Reported-by: denis2342 on IRC
Changelog-Fixed: JSON-RPC: `paystatus` entries no longer have two identical `amount_msat` entries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:50:33 +10:30
Andrew Toth 1f951a5588 doc: update HACKING.md build dep instructions 2021-11-17 10:49:51 +10:30
Andrew Toth 3210213dc3 doc: add missing dev deps, upgrade and don't use sudo for pip 2021-11-17 10:49:51 +10:30
Andrew Toth ee1749bb5e gitignore: ignore eggs and pyln versions 2021-11-17 10:49:51 +10:30
Rusty Russell 65bb989cf1 pytest: don't checksum plugins on startup in VALGRIND developer mode.
This loads up 20MB of plugins temporarily; we seem to be getting OOM
killed under CI and I wonder if this is contributing.

Doesn't significantly reduce runtime here, but I have lots of memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 09bbda4bca pytest: don't create 5 nodes in test_fetchinvoice.
CI seems to be OOM killing us; 5 may be too many under valgrind.

VALGRIND=1 pytest tests/test_pay.py::test_fetchinvoice 
Before:
	1 passed in 199.33s (0:03:19)

After:
	1 passed in 177.91s (0:02:57)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 35c7f56c73 common: use ccan/base64 instead of libsodium.
For consistency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell ff276e958b common: test that our sodium-based base64 encoding is identitcal to ccan's.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell b2c762969c wireaddr: clean up tor parsing.
blob[] is really a string from the commandline; leave it as a char.

And parsing is much simpler than this code makes it seem!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 9d18180172 lightningd: really do allow two Torv3 addresses.
This surprised me, since the CHANGELOG for [0.8.2] said:

	We now announce multiple addresses of the same type, if given. ([3609](https://github.com/ElementsProject/lightning/pull/3609))

But it lied!

Changelog-Fixed: We really do allow providing multiple addresses of the same type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 3dcab9793d doc/TOR.md: simplify, and don't cover autotor.
autotor is older, but statictor is better.  Your options are really
"use HiddenServiceDir in torrc" vs "use statictor", with the issue
that statictor requires you to configure Tor for control access by
c-lightning.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 2a35f33cb0 doc/TOR.md: Make it clear that Tor == Torv3.
And switch ```` to ``` (emacs colorization was confused!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell 2f247c7bfb torv2: remove support for advertizing and connecting.
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.

I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell ecdc15591b custommsg hook: remove message field.
Changelog-Removed: Plugins:  The `message` field on the `custommsg` hook (deprecated in v0.10.0)
2021-11-14 18:49:46 +01:00
Rusty Russell 0c0a301062 fundchannel_complete: remove deprecated txid/txout params.
Changelog-Removed: JSON-RPC: `fundchannel_complete` `txid` and `txout` parameters (deprecated in v0.10.0)
2021-11-14 18:49:46 +01:00
Rusty Russell c00202a0da pytest: remove test_shutdown_alternate_txid.
We're about to require that fundchannel_complete() take a PSBT, where it
does sanity checks to avoid this error, making this a difficult mistake
to make.

Is it time to remove this functionality anyway?  @cdecker?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
ZmnSCPxj jxPCSnmZ c3847484d8 Makefile: Avoid overriding `PYTHONPATH`.
On some distributions (e.g. Gnu Guix) Python packages are not installed in
some standard directory, rather they are installed in different places and
the `PYTHONPATH` variable is modified to include the different places.
So, we must not use the name `PYTHONPATH` in our `Makefile` since `make`
will replace the `PYTHONPATH` environment variable, preventing e.g.
`tools/generate-wire.py` from finding `python-mako` installed on such
distributions.
2021-11-11 14:49:30 +10:30
Rusty Russell 78ebddeece subd: clean up our fd shuffling logic.
It's both complex and flawed, as ZmnSCPxj points out.  Make a generic
fd ordering routine, and use it.

Plus, test it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-10 10:27:15 +10:30
Christian Decker 12689674c7 doc: Update CHANGELOG.md for v0.10.2 release 2021-11-03 15:11:01 +01:00
Christian Decker 000508ca12 doc: Update CHANGELOG.md for v0.10.2rc2 2021-10-31 17:04:43 +01:00
Christian Decker 3d8293dbf4 gci: Add `-f` flag to git fetch
This avoids the warning of clobbering tags when fetching tags

Changelog-None
2021-10-31 16:57:56 +01:00
Christian Decker d680e3b158 doc: Remove non-ASCII character from docs
mrkd doesn't like non-ASCII characters in ubuntu:18.04, used for
repro-builds. This is the easiest way to sidestep this issue until we
deprecate the old ubuntu version.
2021-10-31 16:57:56 +01:00
Christian Decker 0351940593 peer: Wrap the HTLC out destructor in a DB transaction
Changelog-Fixed: peer: Fixed a crash when a connection is lost outside of a DB transaction
2021-10-31 13:11:34 +01:00
Christian Decker 344579b233 wallet: Re-sync the payments.id sequence with postgresql
Fixes #4883
Fixes #4879
Changelog-None: Unreleased bug being fixed.
2021-10-31 13:11:34 +01:00
Christian Decker 15b403531e db: Allow DB-specific queries in the migrations
Sometimes we need some DB-specific magic, and marking migrations with
`/*PSQL*/` or `/*SQLITE*/` will now give us that.
2021-10-31 13:11:34 +01:00
Vincenzo Palazzo e0745358d1 Fix typo in the error message.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-10-30 11:12:21 +02:00
Vincenzo Palazzo b031f07b8c Ignore all the deamon inside the git history
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-10-28 15:51:38 +02:00
ZmnSCPxj jxPCSnmZ d330e6e5b4 wallet/db_postgres.c: Fix obvious typo in error message.
ChangeLog-None
2021-10-28 15:51:25 +02:00
Rusty Russell ba77fce453 websocket: switch to address type 6, since DNS will take 5.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-28 15:51:11 +02:00
Christian Decker 091a6d9151 meta: Add version 0.10.2rc1 changelog 2021-10-23 13:22:20 +02:00
Christian Decker 012c2ecf17 pytest: Create less dust in `test_delpay_payment_split`
We were triggering the dust panic.
2021-10-23 12:59:13 +02:00
niftynei 303fbce479 tests: raise dust limit on mpp test
Fails liquid-regtest otherwise; liquid tends to hit the dust limit
earlier than non-liquid tx, and MPP exacerbates this by divvying up
payments into dusty bits then attempting to shove them through the same
channel, hitting the dust max. The MPP then fails as not all the parts
were able to arrive at their destination.
2021-10-23 12:59:13 +02:00
niftynei b57fed047a dusty htlcs: don't fail the channel, make it error a whole bunch
Let's make this a softer launch by just warning on the channel til the
feerates go back down.

You can also 'fix' this by upping your dust limit with
the `max-dust-htlc-exposure-msat` config.
2021-10-23 12:59:13 +02:00
niftynei cad082a763 channeld: add in RFC notes for max_htlc_dust_exposure_msat
And update some behavior to check both sides on receipt of a
update_fee, as per the proposed spec.

https://github.com/lightningnetwork/lightning-rfc/pull/919
2021-10-23 12:59:13 +02:00