Commit Graph

12564 Commits

Author SHA1 Message Date
Rusty Russell 9aefe3d40a common: update to latest onion-message spec.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-onion-message" BOLTVERSION=guilt/offers
```

Mainly textual, though I neatened the extra fields check for TLVs with
blinding, and implemented the "no other fields" requirement for
non-final onion message hops.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell 8e630e7c53 common/test: fix up name of test file to match latest version.
Otherwise it's skipped!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell e9eb5f493b common: update to latest route-blinding spec.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-route-blinding" BOLTVERSION=guilt/offers
```

Other than textual changes, this does:

1. Ensures we put total_amount_msat in onion final hop (reported by @t-bast).
2. Require that they put total_amount_msat in onion final hop.
3. Return `invalid_onion_blinding` exactly as defined by the spec (i.e. less
   aggressive when we're the final hop) (also reported by @t-bast, but I already
   knew).

See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (total_amount_sat required, Eclair compat)
2023-01-30 13:24:29 +10:30
Rusty Russell d5c19b23d8 common/onion_decode: put final flag in onion_payload.
You can use rs->nextcase, but we don't always keep that around, so
keep a flag in onion_payload.

We'll use this in the "do we need to return a blinded error code"
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell 885506765e tools/check-bolt.c: don't leak open directory.
Thanks valgrind!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell da3506e6a0 wire: use correct number of update_add_tlvs blinding field.
See: #5823
Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (update_add_tlvs fix, Eclair compat)
2023-01-30 13:24:29 +10:30
Rusty Russell 5958c9c3d6 common/test: remove unused padding in bolt04/blinded-onion-message-onion-test.json
This was reported by @valentinewallace: Dave would only use padding to
make all his own encrypted_recipient_data equal-length.  We did it
across the entire path, which includes the hop added by Alice, which
Dave wouldn't know about.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Christian Decker 13fe27c65f gossipd: Do not send warning when node_announcement parsing fails
Changelog-Fixed: gossip: We removed a warning for old `node_announcement` that was causing LND peers to disconnect
2023-01-27 12:58:49 -06:00
Ken Sedgwick 7b2c5617c1 hsmd: increase HSM_MAX_VERSION to 3 2023-01-26 21:10:15 -06:00
Ken Sedgwick a4dc714cdc hsmd: add hsmd_preapprove_keysend and check_preapprovekeysend pay modifier
Changelog-added: hsmd: A new message `hsmd_preapprove_keysend` is added.
Changelog-added: JSON-RPC: A new command `preapprovekeysend` is added.
2023-01-26 21:10:15 -06:00
Ken Sedgwick f29343d740 hsmd: add hsmd_preapprove_invoice and check_preapproveinvoice pay modifier
Changelog-added: hsmd: A new message `hsmd_preapprove_invoice` is added.
Changelog-added: JSON-RPC: A new command `preapproveinvoice` is added.
2023-01-26 21:10:15 -06:00
Braydon 9482e0619c docker: Install protobuf-compiler for builder 2023-01-26 15:22:45 +00:00
Riccardo Casatta 182a9cdcb6 cln-rpc: use serde rename instead of alias
rename is necessary to roundtrip, otherwise the rust name is used.

This also remove the rename if they are not necessary.

Note that:
```
 #[serde(rename="foo", skip_serializing_if=="bar")]
 pub field: bool,
```

is equivalent to:
```
 #[serde(rename="foo")]
 #[serde(skip_serializing_if=="bar")]
 pub field: bool,
```

and for simplicity of construction the latter is used
2023-01-26 15:03:54 +01:00
Rusty Russell fa4b61d13d common/bolt11: convert to table-driven.
Decode functions are now almost entirely uniform, so just use a table.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-25 13:59:34 -06:00
Rusty Russell cbd0ef4192 common/bolt11: add pull_all helper for common case of entire field.
And make pull_bits return a uniform error message, since that's what
callers want, rather than asserting success.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: pay: don't assert() on malformed BOLT11 strings.
2023-01-25 13:59:34 -06:00
Rusty Russell d9fed06b90 common/bolt11: const cleanup, fix parsing errors.
Also, we don't need to pass the total length to the field parsers,
just the length for this field (confusingly, this was called
"data_length").

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-25 13:59:34 -06:00
Michael Schmoock 1fb1e0eec4 pytest: test ip discovery for custom port 2023-01-25 17:57:04 +01:00
Michael Schmoock a62c74be7b doc: usage of --announce-addr-discovered-port option 2023-01-25 17:57:04 +01:00
Michael Schmoock ca9e3e4cc1 opts: adds --announce-addr-discovered-port config option
This will give the user an option to set a custom port when using
discovered IPs for node_announcents. Without this, only the selected
networks default port can used.

Changelog-Added: Adds --announce-addr-discovered-port config option to set custom port for IP discovery.
2023-01-25 17:57:04 +01:00
Michael Schmoock 34cfc93939 cli: getinfo output to regard --ip-discovery 2023-01-25 14:37:56 +01:00
Michael Schmoock 30dea0a431 opts: deprecate --disable-ip-discovery switch
This switch was not doing anything useful anymore.
We deprecate it anyways to notify the user about the new switch.

Changelog-Deprecated: The old --disable-ip-discovery config switch
2023-01-25 14:37:56 +01:00
Michael Schmoock a2b94f16f8 pytest: fix and adapt test_remote_addr_disabled 2023-01-25 14:37:56 +01:00
Michael Schmoock ee046662a7 doc: announce-addr-discovered config switch 2023-01-25 14:37:56 +01:00
Michael Schmoock 3babbc5201 opts: announce-addr-discovered on/off/auto switch
This adds the option to explicitly enable ip-discovery, which maybe
helpful for example when a user wants TOR announced along with
discovered IPs to improve connectivity and have TOR just as a fallback.

Changelog-Added: Adds config switch 'announce-addr-discovered': on/off/auto
2023-01-25 14:37:56 +01:00
Michael Schmoock bd75f8ea6c opts: adds the autobool on/off/auto feature 2023-01-25 14:37:56 +01:00
Michael Schmoock e932f05bc8 ci: adds git fetch before doing schema checks
This fixes the CI errors when doing `make check-source`
steps 'schema-added-check' and 'schema-removed-check'.
These errors prevented CI from performing these steps correctly:

```
fatal: ambiguous argument 'main': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
```

I changed it so that CI does a `git fetch origin` at first and do
the `git diff` against 'origin/master' (which then exist).
Also fixed a bug in the script that was missing $$master in the same line.

Also I added that the script shows the actual diff before failing,
so the user quickly sees whats wrong.
2023-01-25 14:37:56 +01:00
Rusty Russell 0e25d56329 lightningd: use a hash table for peer->dbid.
Otherwise, loading up when we have 100k peers is *painful*!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell cfa632b0e9 lightningd: use hash map for peers instead of linked list.
After connecting 100,000 peers with one channel each (not all at
once!), we see various places where we exhibit O(N^2) behaviour.

Fix these by keeping a map of id->peer instead of a simple
linked-list.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell 17aa047b17 pytest: fix output order assumption in test_setchannel_all
Hash table coming up makes it random!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell 6a95d3a25e common: expose node_id_hash functions.
They're used in several places, and we're about to add more.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell 0faa8397c3 wallet: add dependency on lightningd/ headers.
wallet should really be a subdir of lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Matt Whitlock acfb63e4bf channeld: remove dead HTLCs from htable and free them (eventually)
The channel->htlcs map was exhibiting unbounded growth, as elements were
never removed from it. This was causing lightning_channeld processes to
consume ever-increasing amounts of memory, and iterating over the map
was causing ever-increasing CPU utilization. There were FIXME comments
suggesting that the intention was to remove HTLCs from the map upon
their deaths. This commit implements that intention.

Changelog-Fixed: channeld no longer retains dead HTLCs in memory.
2023-01-20 14:04:08 -06:00
daywalker90 82c94330a5 add PartialEq to ShortChannelId
PartialEq makes it possible to compare 2 ShortChannelIds for equality which i found often helpful
2023-01-17 14:58:07 +01:00
Vincenzo Palazzo 8f94e8b943 comm: make sure that our version check is reliable
Rework the logic of the version check used in the
database migration, and make sure
that it is full functional to avoid confusion
at release time.

Changelog-Fixed: database: Correctly identity official release versions for database upgrade.

Reported-by: @urza
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-01-17 14:15:24 +10:30
Rusty Russell 8d825ef0b7 lightningd: fix valgrind reported leak when we exit early.
```
E           Global errors:
E            - Node /tmp/ltests-adkwu44c/test_logging_1/lightning-2/ has memory leaks: [
E               {
E                   "backtrace": [
E                       "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E                       "lightningd/peer_control.c:2203 (load_channels_from_wallet)",
E                       "lightningd/lightningd.c:1105 (main)"
E                   ],
E                   "label": "lightningd/peer_control.c:2203:struct htlc_in_map",
E                   "parents": [
E                       "lightningd/lightningd.c:107:struct lightningd"
E                   ],
E                   "value": "0x55d920a345e8"
E               }
E           ]
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-17 14:13:45 +10:30
Rusty Russell 288f5df8d1 ccan: update to fix recent gcc "comparison will always evaluate as 'false'" warning
```
lightningd/jsonrpc.c: In function ‘destroy_json_command’:
lightningd/jsonrpc.c:1180:63: error: the comparison will always evaluate as ‘false’ for the address of ‘canary’ will never be NULL [-Werror=address]
lightningd/jsonrpc.c:108:53: note: ‘canary’ declared here
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-17 14:13:45 +10:30
Christian Decker 1f21796454 gha: Temporarily disable `test_notify`
It is failing due to an error somewhere:

```
*** buffer overflow detected ***: terminated
```
2023-01-16 13:53:48 +10:30
Christian Decker e3a9bda301 ci: Downgrade the upload-artifact action to v2.2.4
There seem to be a couple of issues with Bad Requests with later
versions, trying this :-)
2023-01-16 13:53:48 +10:30
Christian Decker b0e3d483e6 gci: Add a test for the postgres backend 2023-01-16 13:53:48 +10:30
Christian Decker 383fca7d0e gci: Add an explicit name to the matrix for display
This is going to be useful to discern what goal the job has.
2023-01-16 13:53:48 +10:30
Christian Decker e17611c570 gci: Split out installation of elements and bitcoin into a script
It was crowding the github workflow file
2023-01-16 13:53:48 +10:30
Christian Decker 71b581da4d make: Clean up duplicate cargo examples build rules 2023-01-16 13:53:48 +10:30
Christian Decker e76618e2a6 ci: Use bzip2 and release to reduce artifact size 2023-01-16 13:53:48 +10:30
Christian Decker b40fd3efbd ci: Add gather step
We can require a status to be successful in PRs, but they are
referenced by name, and so we'd have to add each matrix job as
required. That's rather cumbersome, so have this artificial gather
step at the end which signals success on the entire run.
2023-01-16 13:53:48 +10:30
Christian Decker ca3053707f ci: Build and test in ubuntu:22.04 and install lowdown 2023-01-16 13:53:48 +10:30
Christian Decker bcc75b6e98 tests: Fix a small memory leak in the onion test vector tester
Changelog-None
2023-01-16 13:53:48 +10:30
Christian Decker 6fe6395678 ci: Add bitcoind to integration test job 2023-01-16 13:53:48 +10:30
Christian Decker 34f0954074 ci: Automatically cancel CI runs if we push a new version to the PR
We often have a number of changes in flight, and we amend PRs before the previous run completes. This allows us to cancel that prior run, preserving our precious runners.
2023-01-16 13:53:48 +10:30
Christian Decker e7ee40e951 ci: Split out the unit tests
They are run in parallel with the integration tests, not on the
critical path, but can be a bit lengthy since they build a log of C
code, and run in valgrind.
2023-01-16 13:53:48 +10:30
Christian Decker a650dcb26d ci: Split the pre-flight checks into separate steps
One of these escapes, and actually builds the entire thing. This
should not actually build the CLN binaries, just check it works.
2023-01-16 13:53:48 +10:30