Commit Graph

330 Commits

Author SHA1 Message Date
niftynei fdfca9e721 sqlite3: no NULLS FIRST
sqlite3 added NULLS FIRST to their schema as of v3.30.1 but some older
boxes aren't updated (notably the author's own node box).

The default behavior for sqlite3 is to sort nulls first however, this
is *really* only needed for postgres.

As a simple fix (which will allow older boxes to not upgrade their
sqlite yet), simply strip out the NULLS FIRST keywords from sqlite3
queries.

Reported-By: Simon Vrouwe @SimonVrouwe
Fixes: #5517
2022-08-16 12:32:08 -05:00
niftynei 75ccce7808 devtools: if there's a message in the API call, print and exit
If you've got bad credentials, you'll get an error message.
2022-08-02 10:00:26 -07:00
niftynei 967c56859f sql: use last " as " to find name token for column
We were using the first ' as ', which causes problems for the following
types of lines

	CAST(SUM(of.debit) AS BIGINT) as debit
2022-07-28 12:08:18 +09:30
adi2011 5abed486d0 Add rune and commando to gitignore.
Changelog-None: Small fix
2022-07-19 15:29:26 +09:30
Rusty Russell cf4374c4ed devtools/rune: simple decode tool.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 3e672b784d Makefile: use a library archive for CCAN
The linker discards whole files in an archive if it doesn't need them,
so saves a bit of space (and time).  Also allows us to add more niche
things to CCAN (e.g. runes support!) without bloating all the binaries.

We also had many places which depended on $(CCAN_FILES), but that was
already a dependent of $(ALL_PROGRAMS) and $(ALL_TEST_PROGRAMS).

Before:

```
$ size lightningd/lightning*d
   text	   data	    bss	    dec	    hex	filename
2247683	   8696	  39008	2295387	 23065b	lightningd/lightning_channeld
2086607	   7432	  38880	2132919	 208bb7	lightningd/lightning_closingd
2227916	   8056	  39200	2275172	 22b764	lightningd/lightning_connectd
3369236	 119288	  39240	3527764	 35d454	lightningd/lightningd
2183551	   8352	  38880	2230783	 2209ff	lightningd/lightning_dualopend
2196389	   8024	  39136	2243549	 223bdd	lightningd/lightning_gossipd
2086216	   7488	  39264	2132968	 208be8	lightningd/lightning_hsmd
2134396	   8136	  39424	2181956	 214b44	lightningd/lightning_onchaind
2133391	   8352	  38880	2180623	 21460f	lightningd/lightning_openingd
1512168	   2136	  34384	1548688	 17a190	lightningd/lightning_websocketd
```

After:
```
text	   data	    bss	    dec	    hex	filename
2192065	   8488	  38912	2239465	 222be9	lightningd/lightning_channeld
2030957	   7224	  38816	2076997	 1fb145	lightningd/lightning_closingd
2179571	   7968	  39104	2226643	 21f9d3	lightningd/lightning_connectd
3354296	 119288	  39208	3512792	 3599d8	lightningd/lightningd
2127933	   8144	  38816	2174893	 212fad	lightningd/lightning_dualopend
2141699	   7856	  39072	2188627	 216553	lightningd/lightning_gossipd
2024482	   7288	   5240	2037010	 1f1512	lightningd/lightning_hsmd
2072074	   7920	   5400	2085394	 1fd212	lightningd/lightning_onchaind
2077773	   8144	  38816	2124733	 206bbd	lightningd/lightning_openingd
1408958	   1752	    344	1411054	 1587ee	lightningd/lightning_websocketd
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
Rusty Russell 401f1debc5 common: clean up json routine locations.
We have them split over common/param.c, common/json.c,
common/json_helpers.c, common/json_tok.c and common/json_stream.c.

Change that to:
* common/json_parse (all the json_to_xxx routines)
* common/json_parse_simple (simplest the json parsing routines, for cli too)
* common/json_stream (all the json_add_xxx routines)
* common/json_param (all the param and param_xxx routines)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 12:24:00 -05:00
Rusty Russell dcf2916dcb devtools: add --print-timestamps to dump-gossipstore.
Allowed me to sanity check the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-15 21:18:29 +09:30
Alex Myers 9dc794dba8 gossipd: make use of new ratelimit bit in gossip_store length mask
routing.c now flags rate-limited gossip as it enters the gossip_store but
makes use of it in updating the routing graph. Flagged gossip is not
rebroadcast to gossip peers.

Changelog-Changed: gossipd: now accepts spam gossip, but squelches it for
peers.
2022-07-06 14:31:19 +09:30
Vincenzo Palazzo 2754e30269 devtools: revert changes and make sure chainparams always set.
[Also fixes crash with -h, since chainparams was not set! --RR]
2022-06-28 06:09:01 +09:30
Vincenzo Palazzo 7ff62b4a00 lightnind: remove`DEFAULT_PORT` global definition
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-06-28 06:09:01 +09:30
Rusty Russell 36a2491a89 json: fix up msat amounts in non-_msat fields.
We had json_add_amount_msat_only(), which was designed to be used to
print out msat fields, if we had sats.

However, we misused it, so split it into the three different cases:
1. json_add_amount_sat_msat: We are using it correctly, with a field called
   xxx_msat.
2. json_add_amount_sats_deprecated: We were using it wrong, so deprecate
   the old field and create a new one which does end in _msat.
3. json_add_sats: we were using it to hand sats as a JSON parameter to an
   interface, where "XXXsat".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `rbf_channel` and `openchannel2` hooks `their_funding` (use `their_funding_msat`)
Changelog-Deprecated: Plugins: `openchannel2` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `funding_satoshis` (use `funding_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `channel_reserve_satoshis` (use `channel_reserve_msat`)
Changelog-Deprecated: Plugins: `channel_opened` notification `amount` (use `funding_msat`)
Changelog-Deprecated: JSON-RPC: `listtransactions` `msat` (use `amount_msat`)
Changelog-Deprecated: Plugins: `htlc_accepted` `forward_amount` (use `forward_msat`)
2022-06-21 06:52:35 +09:30
Rusty Russell 7c8dc62035 channeld: take over gossip_rcvd_filter.c and is_msg_gossip_broadcast.
channeld is the only user of these functions, since it now streams
all gossip itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-17 14:14:02 +09:30
Rusty Russell bf040c398b Makefile: update to BOLTs without zlib.
This contains a typo fix and a clarification on channel_type, but also
removes ZLIB.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-05-19 09:47:32 +09:30
Vincenzo Palazzo f90ee0ecfd ignore generic binary
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-05-09 04:04:25 +00:00
Michael Dance f067e8c909 Changed external/libwally-core to test_build_fix
Combined with the following commit which is required to
update against changed libsecp256k1 APIs:

Updated deprecated function calls
2022-04-12 15:17:52 +09:30
Rusty Russell 7f89763f9e sendpay: add payment_metadata argument.
And document the missing arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell e01abf0b34 bolt11: support payment_metadata.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell 116a77f1be lightningd: Restore forwarding of legacy onions.
Partial revert of 43a833e405
"lightningd: remove support for legacy onion format."; we restore the
ability to decode legacy onions for forwarding, but not to generate them.
(We don't accept them properly since making payment_secret compulsory
anyway, so no real change there!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: ... but we still forward legacy HTLC onions for now.
2022-03-31 19:38:05 +10:30
Rusty Russell 5c20b8b1a3 devtools/decodemsg: add --list-tlvs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell b19f3a5e7f devtools/decodemsg: don't require --onion for onion tlvs, fail if unknown tlvname.
Generate a table, let decodemsg sort it out.  Do more up-front work in
argparsing too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell a770f51d0e tools/generate_wire.py: make functions allocate the TLV.
Requiring the caller to allocate them is ugly, and differs from
other types.

This means we need a context arg if we don't have one already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 2b422272e5 wire: generate printwire_ routines for bolt12 CSV.
Good exercise, and we need it when bolt12 messages appear in onion
messages anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell b3a2666a3e decodemsg: if decode fails, status 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 8f39002715 tools/generate-wire.py: have printwire routines do demarshal.
This works better in general: let printwire_x do the work of figuring
out how to demarshal x.  This is particularly important for TLVs, which
require a call to tlv_x_new() first.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 88de64a580 tools/generate-wire.py: simplify printwire routines, fix ... handling.
We make them return bool, and always use names `cursor` and `plen` in
callers, for simplicity.

Also, `...` means "loop until finished" not "loop this many bytes".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 1cb93ffc27 devtools: remove blindedpath tool.
It only made obsolete onions anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 726b6878d1 offers: import latest variant from draft.
In particular, this changes the name of a field in invoice_request:
`payer_signature` becomes simply `signature`.  So we allow both for
now, and send the old one unless deprecated_apis is disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell e36d4d1143 devtools/decodemsg: fix printing of wireaddr.
printwire_ routines are supposed to print!  And they're only needed
inside devtools/.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell 35cca73d2f devtools/fp16: conversion tool for fp16 format.
We use this in gossmap to represent htlc max/min approximations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell 43a833e405 lightningd: remove support for legacy onion format.
As per proposal in https://github.com/lightning/bolts/pull/962

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
2022-03-18 09:20:11 +10:30
niftynei 7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei ea36c3a938 db: collapse the db_queries files into the generated ones
There's no reason to have these be independent of the generated files,
all the data contained within them is 'formulaic'
2022-03-05 15:03:34 +10:30
niftynei ce12d2b8a9 database: pull out database code into a new module
We're going to reuse the database controllers for the accounting plugin
2022-03-05 15:03:34 +10:30
Simon Vrouwe 760c271381 devtools: in decodemsg, add newline and fix decoding a node_announcement with tlvs
Terminal prompt got messed up because missing newline in case of empty fields.

printwire_addresses expected it to be the last field, which is not
the case of a node_announcement with tlv
2022-03-01 09:38:40 +10:30
Rusty Russell 9c0bb444b7 per_peer_state: remove struct crypto_state
Now that connectd does the crypto, no need to hand around crypto_state.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 6bfc3d8692 gossipwith: create our own internal sync_crypto functions.
This avoids changes to crypto_sync which are coming in next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell 7e7a63a20d connectd: keep timeout timer around so we can disable it.
connectd will be keeping the conn open, so it needs to free this
"conn_timeout" timer.  Pass it through, so we can do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
ZmnSCPxj jxPCSnmZ d088288daa bitcoin/chainparams.h: Split BIP173 name into onchain and Lightning HRPs.
Fixes: #4937
2021-12-14 11:21:35 +10:30
Rusty Russell f64df6fbb5 devtools: simple script to look for functions which don't seem to be used.
Gives a rough first-pass, at least.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell 4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell 163d3a16f4 doc: clean up offers bolt quotes
As of 2b923a0367c5f9154fcec706e3302cc4658dd889.

Recurrence quotes need to be marked separately, since they're no longer
in offers main bolt.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell b3af5f5a2c spec: import latest onionmessage spec, based on routeblinding.
This is from 6e99c5feaf60cb797507d181fe583224309318e9

We renamed the enctlv field to encrypted_recipient_data in the spec, and the
new onion_message is message 513.  We don't handle it until the next patch.

Two renames:
1. blinding_seed -> blinding_point.
2. enctlv -> encrypted_recipient_data.

We don't do a compat cycle for our JSON APIs for these experimental
features only used by our own plugins, we just rename.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell 1ec6346f3d common: rename current onion message structures to obs2_.
Yes, we changed the spec again.  Hopefully for the last time!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell b74848f6f6 common: remove support for pre v0.10.2 onionmessages.
Temporarily disable sendpay_blinding test which uses obsolete onionmsg;
there's still some debate on the PR about how blinded HTLCs will work.

Changelog-EXPERIMENTAL: onionmessage: removed support for v0.10.1 onion messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Michael Schmoock 01e8a523e9 bolt7: allow announcement of ADDR_TYPE_DNS 2021-11-30 09:38:17 +10:30
Rusty Russell 1cbdb9cc12 devtools/taldump-analyze.py: script to analyze tal_dump() output.
Example output on my production machine, running v0.10.2rc1-9-g3d8293d:

```
Top sizes by name
backtrace: 1172021248 bytes, 4578208 items
wallet/txfilter.c:111:struct bitcoin_outpoint: 49584636 bytes, 1377351 items
lightningd/plugin.c:1651:char[]: 17261056 bytes, 16 items
lightningd/log.c:282:char[]: 16653908 bytes, 838086 items
wally_tal: 13390257 bytes, 122503 items
lightningd/log.c:250:struct log_entry[]: 7340032 bytes, 1 items
lightningd/jsonrpc.c:939:char[]: 6979648 bytes, 214 items
lightningd/log.c:440:u8[]: 3278359 bytes, 1338 items
common/memleak.c:51:struct memleak_notleak: 2216763 bytes, 2216763 items
Top sizes by path
['']: 2579337788 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd']: 831485962 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet']: 756169631 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter']: 756166227 bytes, 2 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter', 'wallet/txfilter.c:134:struct outpointset']: 706581191 bytes, 2 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable']: 457580181 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book']: 430307262 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology']: 48054597 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block']: 47076437 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]']: 47034285 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins']: 17569217 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins', 'lightningd/plugin.c:246:struct plugin', 'lightningd/plugin.c:1651:char[]']: 16784384 bytes, 5 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx']: 13730127 bytes, 452 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/jsonrpc.c:1189:struct jsonrpc']: 7745890 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book', 'lightningd/log.c:250:struct log_entry[]']: 7340032 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx', 'wally_tal']: 4480328 bytes, 166 items
['', 'tmpctx']: 0 bytes, 1 items
```
2021-11-27 10:38:13 +10:30
Rusty Russell fcf3d0ce6c db: turn generated queries array into a simple hash table.
Since we have that functionality, let's use it.  Also, make table
const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 787fbb1228 db: create simple hashtable of fields in SELECT.
This simplistically maps names to numbers, eg:

	SELECT foo, bar FROM tbl;

'foo' -> 0
'bar' -> 1

If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell 2f247c7bfb torv2: remove support for advertizing and connecting.
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.

I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00