Commit Graph

311 Commits

Author SHA1 Message Date
Rusty Russell 5c20b8b1a3 devtools/decodemsg: add --list-tlvs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell b19f3a5e7f devtools/decodemsg: don't require --onion for onion tlvs, fail if unknown tlvname.
Generate a table, let decodemsg sort it out.  Do more up-front work in
argparsing too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell a770f51d0e tools/generate_wire.py: make functions allocate the TLV.
Requiring the caller to allocate them is ugly, and differs from
other types.

This means we need a context arg if we don't have one already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 2b422272e5 wire: generate printwire_ routines for bolt12 CSV.
Good exercise, and we need it when bolt12 messages appear in onion
messages anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell b3a2666a3e decodemsg: if decode fails, status 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 8f39002715 tools/generate-wire.py: have printwire routines do demarshal.
This works better in general: let printwire_x do the work of figuring
out how to demarshal x.  This is particularly important for TLVs, which
require a call to tlv_x_new() first.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 88de64a580 tools/generate-wire.py: simplify printwire routines, fix ... handling.
We make them return bool, and always use names `cursor` and `plen` in
callers, for simplicity.

Also, `...` means "loop until finished" not "loop this many bytes".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 1cb93ffc27 devtools: remove blindedpath tool.
It only made obsolete onions anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 726b6878d1 offers: import latest variant from draft.
In particular, this changes the name of a field in invoice_request:
`payer_signature` becomes simply `signature`.  So we allow both for
now, and send the old one unless deprecated_apis is disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell e36d4d1143 devtools/decodemsg: fix printing of wireaddr.
printwire_ routines are supposed to print!  And they're only needed
inside devtools/.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 35cca73d2f devtools/fp16: conversion tool for fp16 format.
We use this in gossmap to represent htlc max/min approximations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell 43a833e405 lightningd: remove support for legacy onion format.
As per proposal in https://github.com/lightning/bolts/pull/962

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
2022-03-18 09:20:11 +10:30
niftynei 7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei ea36c3a938 db: collapse the db_queries files into the generated ones
There's no reason to have these be independent of the generated files,
all the data contained within them is 'formulaic'
2022-03-05 15:03:34 +10:30
niftynei ce12d2b8a9 database: pull out database code into a new module
We're going to reuse the database controllers for the accounting plugin
2022-03-05 15:03:34 +10:30
Simon Vrouwe 760c271381 devtools: in decodemsg, add newline and fix decoding a node_announcement with tlvs
Terminal prompt got messed up because missing newline in case of empty fields.

printwire_addresses expected it to be the last field, which is not
the case of a node_announcement with tlv
2022-03-01 09:38:40 +10:30
Rusty Russell 9c0bb444b7 per_peer_state: remove struct crypto_state
Now that connectd does the crypto, no need to hand around crypto_state.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6bfc3d8692 gossipwith: create our own internal sync_crypto functions.
This avoids changes to crypto_sync which are coming in next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 7e7a63a20d connectd: keep timeout timer around so we can disable it.
connectd will be keeping the conn open, so it needs to free this
"conn_timeout" timer.  Pass it through, so we can do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
ZmnSCPxj jxPCSnmZ d088288daa bitcoin/chainparams.h: Split BIP173 name into onchain and Lightning HRPs.
Fixes: #4937
2021-12-14 11:21:35 +10:30
Rusty Russell f64df6fbb5 devtools: simple script to look for functions which don't seem to be used.
Gives a rough first-pass, at least.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell 4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell 163d3a16f4 doc: clean up offers bolt quotes
As of 2b923a0367c5f9154fcec706e3302cc4658dd889.

Recurrence quotes need to be marked separately, since they're no longer
in offers main bolt.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell b3af5f5a2c spec: import latest onionmessage spec, based on routeblinding.
This is from 6e99c5feaf60cb797507d181fe583224309318e9

We renamed the enctlv field to encrypted_recipient_data in the spec, and the
new onion_message is message 513.  We don't handle it until the next patch.

Two renames:
1. blinding_seed -> blinding_point.
2. enctlv -> encrypted_recipient_data.

We don't do a compat cycle for our JSON APIs for these experimental
features only used by our own plugins, we just rename.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell 1ec6346f3d common: rename current onion message structures to obs2_.
Yes, we changed the spec again.  Hopefully for the last time!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell b74848f6f6 common: remove support for pre v0.10.2 onionmessages.
Temporarily disable sendpay_blinding test which uses obsolete onionmsg;
there's still some debate on the PR about how blinded HTLCs will work.

Changelog-EXPERIMENTAL: onionmessage: removed support for v0.10.1 onion messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Michael Schmoock 01e8a523e9 bolt7: allow announcement of ADDR_TYPE_DNS 2021-11-30 09:38:17 +10:30
Rusty Russell 1cbdb9cc12 devtools/taldump-analyze.py: script to analyze tal_dump() output.
Example output on my production machine, running v0.10.2rc1-9-g3d8293d:

```
Top sizes by name
backtrace: 1172021248 bytes, 4578208 items
wallet/txfilter.c:111:struct bitcoin_outpoint: 49584636 bytes, 1377351 items
lightningd/plugin.c:1651:char[]: 17261056 bytes, 16 items
lightningd/log.c:282:char[]: 16653908 bytes, 838086 items
wally_tal: 13390257 bytes, 122503 items
lightningd/log.c:250:struct log_entry[]: 7340032 bytes, 1 items
lightningd/jsonrpc.c:939:char[]: 6979648 bytes, 214 items
lightningd/log.c:440:u8[]: 3278359 bytes, 1338 items
common/memleak.c:51:struct memleak_notleak: 2216763 bytes, 2216763 items
Top sizes by path
['']: 2579337788 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd']: 831485962 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet']: 756169631 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter']: 756166227 bytes, 2 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter', 'wallet/txfilter.c:134:struct outpointset']: 706581191 bytes, 2 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable']: 457580181 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book']: 430307262 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology']: 48054597 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block']: 47076437 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]']: 47034285 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins']: 17569217 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins', 'lightningd/plugin.c:246:struct plugin', 'lightningd/plugin.c:1651:char[]']: 16784384 bytes, 5 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx']: 13730127 bytes, 452 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/jsonrpc.c:1189:struct jsonrpc']: 7745890 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book', 'lightningd/log.c:250:struct log_entry[]']: 7340032 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx', 'wally_tal']: 4480328 bytes, 166 items
['', 'tmpctx']: 0 bytes, 1 items
```
2021-11-27 10:38:13 +10:30
Rusty Russell fcf3d0ce6c db: turn generated queries array into a simple hash table.
Since we have that functionality, let's use it.  Also, make table
const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 787fbb1228 db: create simple hashtable of fields in SELECT.
This simplistically maps names to numbers, eg:

	SELECT foo, bar FROM tbl;

'foo' -> 0
'bar' -> 1

If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 2f247c7bfb torv2: remove support for advertizing and connecting.
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.

I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Christian Decker 15b403531e db: Allow DB-specific queries in the migrations
Sometimes we need some DB-specific magic, and marking migrations with
`/*PSQL*/` or `/*SQLITE*/` will now give us that.
2021-10-31 13:11:34 +01:00
niftynei 1fe829c546 lightningd: new option for htlc dust limit
To reduce the surface area of amount of a channel balance that can be
eaten up as htlc dust, we introduce a new config
'--max-dust-htlc-exposure-msat', which sets the max amount that any
channel's balance can be added as dust

Changelog-Added: config: new option --max-dust-htlc-exposure-msat, which limits the total amount of sats to be allowed as dust on a channel
2021-10-23 12:59:13 +02:00
Rusty Russell b013b3ab0c patch websocket-address-support.patch 2021-10-22 11:56:30 +02:00
Rusty Russell c503232cde common: use bitcoin_outpoint.
I started pulling this thread, and the entire codebase got unravelled.

Oh well, it's done now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell 36b66871a2 db: add min/max commitnum fields to channel_htlcs.
And initialize max to current height max when htlcs are already dead.
Turns out (thanks CI!) that MAX() of multiple columns is GREATEST() in
Postgres.  That's clearer (MAX is used elsewhere for single columns),
so translate on the sqlite3 side.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell c92ce59892 BOLT 12: switch invoice_request/invoice to singular `chain` field.
We keep the now-removed chains field, and in deprecated mode, we set it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: bolt12: `chains` in invoice_request and invoice is deprecated, `chain` is used instead.
2021-10-08 13:47:30 +02:00
Rusty Russell 45bf7a3974 bolt12: update to latest spec.
Main changes are:
1. Uses point32 instead of pubkey32.
2. Uses issuer instead of vendor.
3. Uses byte instead of u8.
4. blinded_path num_hops is now a byte, not u16 (we don't use that yet!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: bolt12: `vendor` is deprecated: the field is now called `issuer`.
2021-10-08 13:47:30 +02:00
Rusty Russell 8f582e770c BOLT12: use point32 instead of pubkey32.
That's the modern BOLT12 term.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-08 13:47:30 +02:00
Rusty Russell 7ef2f4d7fb devtools/features: tool to convert feature bitmap to names.
For example:

```
$ ./devtools/features 80008008226aa2
option_data_loss_protect/odd (optional)
option_upfront_shutdown_script/odd (optional)
option_gossip_queries/odd (optional)
option_var_onion_optin/odd (optional)
option_gossip_queries_ex/odd (optional)
option_static_remotekey/odd (optional)
option_payment_secret/even (compulsory)
option_basic_mpp/odd (optional)
option_anchor_outputs/odd (optional)
option_shutdown_anysegwit/odd (optional)
option_onion_messages/odd (optional)
option_unknown_54/odd (optional)
```


Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-23 15:02:05 +02:00
Rusty Russell e3ae7883bb channeld: rename onion_message to obs_onion_message.
This splits the existing old-spec pathways to prepare for the
new ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-22 06:44:26 +09:30
Rusty Russell 24536c5561 common/autodata: use instead of ccan/autodata
This means it needs to be linked ~everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-21 18:04:43 +02:00
Rusty Russell 7401b26824 cleanup: remove unneeded includes in C files.
Before:
 Ten builds, laptop -j5, no ccache:

```
real	0m36.686000-38.956000(38.608+/-0.65)s
user	2m32.864000-42.253000(40.7545+/-2.7)s
sys	0m16.618000-18.316000(17.8531+/-0.48)s
```

 Ten builds, laptop -j5, ccache (warm):

```
real	0m8.212000-8.577000(8.39989+/-0.13)s
user	0m12.731000-13.212000(12.9751+/-0.17)s
sys	0m3.697000-3.902000(3.83722+/-0.064)s
```

After:
 Ten builds, laptop -j5, no ccache: 8% faster

```
real	0m33.802000-35.773000(35.468+/-0.54)s
user	2m19.073000-27.754000(26.2542+/-2.3)s
sys	0m15.784000-17.173000(16.7165+/-0.37)s
```

 Ten builds, laptop -j5, ccache (warm): 1% faster

```
real	0m8.200000-8.485000(8.30138+/-0.097)s
user	0m12.485000-13.100000(12.7344+/-0.19)s
sys	0m3.702000-3.889000(3.78787+/-0.056)s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell ea30c34d82 cleanup: remove unneeded includes in header files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell 1ab2b4a3fb devtools/reduce-includes.sh: brute-force script to remove includes.
Tries one at a time.  Slow, but doesn't need to be run often.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell 740afb822c common/initial_channel: use channel_type instead of individual option bools.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell cb22015b2a common/channel_type: wrapper for generated 'struct channel_type'.
We make it a first-class citizen internally, even though we won't use
it over the wire (at least, non-experimental builds).  This scheme
follows the latest draft, in which features are flagged compulsory.

We also add several helper functions.

Since uses the *even* bits (as per latest spec), not the *odd* bits,
we have some other fixups.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell fb4edc2938 Makefile: update bolt version to include option_anchors_zero_fee_htlc_tx.
This touches a lot of text, mainly to change "if `option_anchor_outputs`"
to "if `option_anchors`"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-09 12:04:48 +09:30
Rusty Russell 1b2c6964fd Makefile: update spec version
This includes the new bolt11 test vectors, and also removes the
requirement that HTLCs be less than 2^32 msat.  We keep that for now
because Electrum enforced it on receive: in two releases we will stop
that too.

So no longer warn about needing mpp in that case either.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Protocol: No longer restrict HTLCs to
2021-09-09 12:04:48 +09:30
Rusty Russell cf1dd779d1 gossmap: don't spew to stderr, include counter for callers.
Fixes: #4722
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: don't drop complaints about silly channels to stderr.
2021-08-24 06:42:32 +09:30