Commit Graph

11828 Commits

Author SHA1 Message Date
niftynei d9ea21a40d
remove all mrkd references from project 2022-09-22 12:55:54 -05:00
niftynei 4e87e100e8
Changelog for v0.12.1 2022-09-21 21:44:55 -05:00
niftynei b381da6a2e
pyln: update to v0.12.1 2022-09-21 21:44:54 -05:00
Rusty Russell 838400c702
libplugin: handle JSON reply after command freed.
This can happen, and in fact does below in our test_autoclean_once
test where we update the datastore, and return from the cmd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-21 21:44:54 -05:00
Rusty Russell 4249c6176a
libplugin: forget pending requests if associated command freed.
This is usually fine, but without this, commando (another branch!) has
a race:

1. A command has multiple parts.
2. We start sending them out.
3. We get a response, which completes the cmd.
4. We go to send the next one out, but it's been freed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-21 21:44:54 -05:00
Rusty Russell 1781fad8b8
contrib/pyln-client: allow lightningd to give us non-numeric ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-21 21:44:53 -05:00
niftynei 8055a2228a
build-release: configure before submodcheck
We added a submod dep (lowdown) that requires config to run first.
2022-09-21 20:49:56 -05:00
Michael Schmoock e1290b496b
peer_control: getinfo show correct port on discovered IPs
Changelog-Fixed: peer_control: getinfo shows the correct port on discovered IPs
2022-09-19 17:02:41 -05:00
Michael Schmoock 9f4dba14dc
cleanup: fix mixed indentation of json_getinfo
This one got badly messed up over time. I know we usually don't fix
these to have easier git-bisect. I can remove this commit if required.
2022-09-19 17:02:41 -05:00
Michael Schmoock 095f65043a
peer_control: fix getinfo showing unannounced addr
Currently discovered IPs are only announced when we don't have any
usable addresses detected or configured already. However, the cli
command `getinfo` still showed theses unannounced addr as if they
were announced.

Changelog-Fixed: peer_control: getinfo showing unannounced addresses.
2022-09-19 17:02:40 -05:00
Dustin Dettmer 2853da1715
connectd+: Flake/race fix for new channels
1) dualopen has fd to connectd
2) channeld needs to take over
3) dualopen passes fd that leads to a connectd over for channeld to use
4) lightningd must receive the fd transfer request and process
5) dualopen shuts down and closes everything it owns

4 & 5 end up in a race. If 5 happens before 4, channeld ends up with an invalid fd for connectd — leaving it in a position to not receive messages.

Lingering for a second makes 4 win the race. Since the daemon is closing anyway, waiting for a second should be alright.

Changelog-Fixed: Fixed a condition for newly created channels that could trigger a need for reconnect.
2022-09-18 10:03:47 -05:00
Rusty Russell 7b506d573a
gossmap: don't crash if we see a duplicate channel_announce.
Apparently we had two private channel announcements (the !private assert
failed).  While this shouldn't happen, don't crash because of it.

Fixes: #5578
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: topology plugin could crash when it sees duplicate private channel announcements.
2022-09-14 14:05:36 -05:00
Alex Myers 1cd6e1d41f
gossip_store: fix offset error
The gossip_store version byte was unaccounted for in the initial traversal
of gossip_store_end. This lead to an offset and a bogus message length
field. As a result, an early portion of the gossip_store could have been
skipped, potentially leading to gossip propagation issues downstream.

Fixes #5572 #5565

Changelog-fixed: proper gossip_store operation may resolve some previous gossip propagation issues
2022-09-14 14:05:35 -05:00
niftynei 0a4c030b6f
bkpr-test: maybe fix race in test_bookkeeping_closing_trimmed_htlcs
test_bookkeeping_closing_trimmed_htlcs fails to find 'all outputs
resolved' occassionally, seems like it's because the
OUR_DELAYED_TO_WALLET doesn't make it into the mempool before we start
mining blocks?

So here make sure there's something in the mempool before before we
start making new blocks.
2022-09-14 14:05:35 -05:00
niftynei c83b9a2987
bkpr: confirm that replaying the open+lock-in txs at start is ok
Make sure that we're not issuing duplicate lease_fee events!
2022-09-14 14:05:35 -05:00
niftynei 52f1938c57
bkpr: migration to delete any duplicate lease_fee entries
Clean up for #5557.

If you've got duplicate 'lease_fee' entries, we delete them!
2022-09-14 14:05:34 -05:00
niftynei af41c7c883
notif: dont send balance snapshot for not yet opened channel
We were double counting channel lease fees because we were double firing
the channel open event sequence (so to speak). If we don't report
balances for unopened channels, we don't have this problem?

Changelog-Changed: Plugins: `balance_snapshot` notification does not send balances for channels that aren't locked-in/opened yet
2022-09-14 14:05:34 -05:00
niftynei b2708cfe36
bkpr: failing test for bookkeeper crash
Reproduce crash for #5557!

If we record the channel open because bookkeeper was added after the
channel open request started but the channel confirms later, we end up
with re-recording any associated push or leased fees (paid or rcvd).

In the case where you've paid for these fees, your channel balance goes
negative and the node crashes the next time you call `listbalances`.

Reported-by: @chrisguida
2022-09-14 14:05:34 -05:00
niftynei 7cb754cb02
builds: cleanup duplicate and unused code, fix spelling
Few extremely minor updates to the ubuntu dockerfile and ci builds
2022-09-13 12:30:49 -05:00
niftynei 6285110c75
build: use ubuntu 22.04 LTS
Requires us to update to latest lnproto which is now using the most up
to date python-bitcoinlib, as well as updating our python lock files
(which pin the grpcio deps, because of locking problems h/t @cdecker)
2022-09-13 12:30:38 -05:00
Rusty Russell 3763c5007b
connectd: better diagnostics on invalid gossip_store entries.
Should help diagnose https://github.com/ElementsProject/lightning/issues/5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:30:28 -05:00
Rusty Russell 67b75b1f59
doc: document that we can build lowdown, remove from Alpine.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:30:21 -05:00
Rusty Russell 5c8edadcc6
external: build lowdown if not already found.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:30:13 -05:00
Rusty Russell db1cd3bd58
external/lowdown: local import of lowdown source.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:30:01 -05:00
Rusty Russell 2536fd5b21
doc: always escape underscores in property names
If there's only a single underscore, lowdown ignores it, but if there are multiple
(see min_final_cltv_expiry) it decides we're trying to highlight part of the word.

Reported-by: @wtogami
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:29:55 -05:00
Rusty Russell 0145e8ecec
Makefile: Revert ba7d4a8f6b (make-schema: don't include tools/fromschema.py in SHASUMS)
1. If the tool changes, you need to regenerate since the output may
   change.

2. This didn't just filter that out, ignored all but the first
   dependency, which made bisecting the bookkeeper plugin a nightmare:
   it didn't regenerate the .po file, causing random crashes.

If we want this, try $(filter-out tools/fromschema.py) instead.  But I
don't think we want that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:29:46 -05:00
Rusty Russell 9ce6b6ce85
doc: generate correct markdown from schemas.
You can't start a list without a paragraph separator.

```diff
 --- /tmp/before	2022-07-20 22:02:23.485372596 +0930
 +++ /tmp/after	2022-07-20 22:02:33.745528456 +0930
 @@ -21,12 +21,16 @@
         On startup of the daemon, no autoclean is set up.
  
  RETURN VALUE
 -       On success, an object is returned,  containing:  -  enabled  (boolean):
 -       whether invoice autocleaning is active
 +       On success, an object is returned, containing:
  
 -       If enabled is true: - expired_by (u64): how long an invoice must be ex‐
 -       pired (seconds) before we delete it - cycle_seconds (u64): how long  an
 -       invoice must be expired (seconds) before we delete it
 +       • enabled (boolean): whether invoice autocleaning is active
 +
 +       If enabled is true:
 +
 +       • expired_by (u64): how long an invoice must be expired  (seconds)  be‐
 +         fore we delete it
 +       • cycle_seconds  (u64):  how  long an invoice must be expired (seconds)
 +         before we delete it
  
  AUTHOR
         ZmnSCPxj <ZmnSCPxj@protonmail.com> is mainly responsible.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:29:32 -05:00
Rusty Russell 237d16fdbf
doc: format markdown correctly.
There are no definition lists in Markdown, and lists get mangled if
they follow immediately: they need a line between them.

So use bullets for options, and use an indent so the text gets in the
line below.

Here's a before-and-after example:

```diff
 --- /tmp/after	2022-07-20 21:55:54.355487769 +0930
 +++ /tmp/after2	2022-07-20 21:58:17.305642576 +0930
 @@ -10,38 +10,71 @@
         lightning-cli sends commands to the lightning daemon.

  OPTIONS
 -       --lightning-dir=DIR Set the directory for the lightning daemon we're talking to; defaults to $HOME/.lightning.
 +       • --lightning-dir=DIR

 -       --conf=PATH Sets configuration file (default: lightning-dir/config ).
 +         Set the directory for the lightning daemon we're talking to; defaults to $HOME/.lightning.

 -       --network=network --mainnet --testnet --signet Sets network explicitly.
 +       • --conf=PATH

 -       --rpc-file=FILE Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
 +         Sets configuration file (default: lightning-dir/config ).

 -       --keywords/-k Use format key=value for parameters in any order
 +       • --network=network

 -       --order/-o Follow strictly the order of parameters for the command
 +       • --mainnet

 -       --json/-J Return result in JSON format (default unless help command, or result contains a format-hint field).
 +       • --testnet

 -       --raw/-R Return raw JSON directly as lightningd replies; this can be faster for large requests.
 +       • --signet

 -       --human-readable/-H Return result in human-readable output.
 +         Sets network explicitly.

 -       --flat/-F  Return  JSON result in flattened one-per-line output, e.g. { "help": [ { "command": "check" } ] } would become
 +       • --rpc-file=FILE
 +
 +         Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
 +
 +       • --keywords/-k
 +
 +         Use format key=value for parameters in any order
 +
 +       • --order/-o
 +
 +         Follow strictly the order of parameters for the command
 +
 +       • --json/-J
 +
 +         Return result in JSON format (default unless help command, or result contains a format-hint field).
 +
 +       • --raw/-R
 +
 +         Return raw JSON directly as lightningd replies; this can be faster for large requests.
 +
 +       • --human-readable/-H
 +
 +         Return result in human-readable output.
 +
 +       • --flat/-F
 +
 +         Return  JSON  result  in  flattened  one-per-line  output,  e.g.  {  "help":  [ { "command": "check" } ] } would become
         help[0].command=check.  This is useful for simple scripts which want to find a  specific  output  field  without  parsing
         JSON.

 -       --notifications/-N=LEVEL  If  LEVEL is 'none', then never print out notifications.  Otherwise, print out notifications of
 -       LEVEL or above (one of io, debug, info (the default), unusual or broken: they are prefixed with # .
 +       • --notifications/-N=LEVEL
 +
 +         If  LEVEL  is 'none', then never print out notifications.  Otherwise, print out notifications of LEVEL or above (one of
 +         io, debug, info (the default), unusual or broken: they are prefixed with # .
 +
 +       • --help/-h
 +
 +         Pretty-print summary of options to standard output and exit.  The format can be changed using -F, -R, -J, -H etc.
 +
 +       • --version/-V

 -       --help/-h Pretty-print summary of options to standard output and exit.  The format can be changed using -F,  -R,  -J,  -H
 -       etc.
 +         Print version number to standard output and exit.

 -       --version/-V Print version number to standard output and exit.
 +       • allow-deprecated-apis=BOOL

 -       allow-deprecated-apis=BOOL Enable deprecated options. It defaults to true, but you should set it to false when testing to
 -       ensure that an upgrade won't break your configuration.
 +         Enable deprecated options. It defaults to true, but you should set it to false when testing to ensure that  an  upgrade
 +         won't break your configuration.

  COMMANDS
         lightning-cli simply uses the JSON RPC interface to talk to lightningd, and prints the results. Thus the commands  avail‐
 @@ -60,13 +93,13 @@
         this is not encouraged.

  EXAMPLES
 -       1.     List commands
 +       1.     List commands:

 -       lightning-cli help
 +              • lightning-cli help

 -       2.     Fund a 10k sat channel using uncomfirmed outputs
 +       2.     Fund a 10k sat channel using uncomfirmed outputs:

 -       lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0
 +              • lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0

  BUGS
         This manpage documents how it should work, not how it does work. The pretty printing of results isn't pretty.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:29:20 -05:00
Rusty Russell 76b57a221d
Makefile: replace mrkd with lowdown(1).
Here's the before-vs-after comparison (ignoring whitespace changes):

```diff
 --- /tmp/before	2022-07-20 21:52:44.336641810 +0930
 +++ /tmp/after	2022-07-20 21:55:54.355487769 +0930
 @@ -1,7 +1,7 @@
 -LIGHTNING-CLI(1)                                          lightning-cli                                         LIGHTNING-CLI(1)
 +LIGHTNING-CLI(1)                                                                                                LIGHTNING-CLI(1)
  
  NAME
 -       lightning-cli - Control lightning daemon
 +       lightning-cli -- Control lightning daemon
  
  SYNOPSIS
         lightning-cli [OPTIONS] command
 @@ -14,10 +14,7 @@
  
          --conf=PATH Sets configuration file (default: lightning-dir/config ).
  
 -        --network=network
 -        --mainnet
 -        --testnet
 -        --signet Sets network explicitly.
 +       --network=network --mainnet --testnet --signet Sets network explicitly.
  
          --rpc-file=FILE Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
  
 @@ -43,8 +40,8 @@
  
          --version/-V Print version number to standard output and exit.
  
 -        allow-deprecated-apis=BOOL  Enable  deprecated options. It defaults to true, but you should set it to false when testing
 -       to ensure that an upgrade won't break your configuration.
 +       allow-deprecated-apis=BOOL Enable deprecated options. It defaults to true, but you should set it to false when testing to
 +       ensure that an upgrade won't break your configuration.
  
  COMMANDS
         lightning-cli simply uses the JSON RPC interface to talk to lightningd, and prints the results. Thus the commands  avail‐
 @@ -67,7 +64,7 @@
  
         lightning-cli help
  
 -              1.     Fund a 10k sat channel using uncomfirmed outputs
 +       2.     Fund a 10k sat channel using uncomfirmed outputs
  
         lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0
  
 @@ -84,4 +81,4 @@
         Note: the modules in the ccan/ directory have their own licenses, but the rest of the code is covered  by  the  BSD-style
         MIT license.
  
 -                                                                                                                LIGHTNING-CLI(1)
 +Core Lightning v0.11.0.1-350-gac2e137                                                                           LIGHTNING-CLI(1)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5437
2022-09-13 12:29:06 -05:00
Rusty Russell 5865fce831
tools: add md2man.sh tool, using lowdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-13 12:28:57 -05:00
Alex Myers 0798a5dab0
pytest: Add debugging to test_gossip_store_compact_on_load
This flake has been difficult to reproduce, so let's dump the gossip
store to aid in debugging. See issue #5410.

Changelog-None
2022-09-13 12:27:33 -05:00
niftynei 3ce26735e5 v0.12.0 release changelog 2022-08-23 14:13:25 -04:00
Orb fd8b667e6a use 'postgres' driver (not postgresql). Include plain url to guide, as link is broken on RTD. 2022-08-22 11:21:59 -04:00
greggzigler ce9232af37 Changelog-None 2022-08-22 11:21:19 -04:00
greggzigler c13dffe980 fix MD link where text and url are swapped 2022-08-22 11:21:19 -04:00
niftynei 8cdbe97d91 v0.12.0rc3 - Release Candidate numero tres
Release Candidate 3 for v0.12.0.

Now with more bugfixes!
2022-08-19 18:05:56 -04:00
Rusty Russell 7c9985fd9c channeld: correct reversed shutdown message in billboard.
It was backwards, which made #5496 confusing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `listpeers` `status` "They've sent shutdown" and "We've sent shutdown" were backwards.
2022-08-19 11:04:37 -04:00
Christian Decker aabf38a11d ci: Fix the Mac OS build test 2022-08-19 10:44:21 -04:00
adi2011 8f78a76d1a tests/test_misc.py: check logs for already existing channel. 2022-08-19 10:27:09 -04:00
adi2011 0b737fd7e5 doc: Typo 2022-08-19 10:27:09 -04:00
adi2011 23b675922f lightningd/opening_control.c: Skip over channels which are already stored, and don't create new peer if it already exits. Changelog-None 2022-08-19 10:27:09 -04:00
Rusty Russell 9219e778a1 doc: update usage of lightning-invoice.
The new parameter name (this version) is amount_msat: msatoshi is
deprecated.   The doc/schemas/invoice.request.json has this correct
(but we don't generate teh *usage* line from that yet!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-18 15:53:14 -04:00
Ken Sedgwick 82f7035524 Renumber hsmd_derive_secret for consistency with other hsmd messages 2022-08-16 15:04:49 -05:00
Christian Decker 65549a2931 ld: Fix a log message assuming that the `channel->scid` was set
This is no longer true after we introduce `zeroconf`, so use the alias
local alias instead if not set.

Signed-off-by: Christian Decker <@cdecker>
2022-08-16 14:52:52 -05:00
Christian Decker ab1ca7f159 pytest: Reproduce a crash when we have multiple channels and 0conf
This is a case where we assume that the `channel->scid` is set, which
is no longer true with `zeroconf`

Changelog-None: Zeroconf was not yet released at the time the discovery was made

Reported-by: Yaacov Akiba Slama <@yaslama>
Reported-by: Roei Erez <@roeierez>
2022-08-16 14:52:52 -05:00
niftynei 1a25651182 bkpr-recorder: fatal if there's an error with a database statement
We were silently failing when NULLS FIRST wasn't found on older boxes
(running sqlite3 < v3.30.1)
2022-08-16 12:32:08 -05:00
niftynei fdfca9e721 sqlite3: no NULLS FIRST
sqlite3 added NULLS FIRST to their schema as of v3.30.1 but some older
boxes aren't updated (notably the author's own node box).

The default behavior for sqlite3 is to sort nulls first however, this
is *really* only needed for postgres.

As a simple fix (which will allow older boxes to not upgrade their
sqlite yet), simply strip out the NULLS FIRST keywords from sqlite3
queries.

Reported-By: Simon Vrouwe @SimonVrouwe
Fixes: #5517
2022-08-16 12:32:08 -05:00
niftynei 4e7f89f211 signpsbt: don't crash if HSM doesn't like your psbt, just return err
Changelog-Changed: JSON-RPC: `signpsbt` no longer crashes if it doesn't like what your PSBT is
2022-08-13 12:57:03 -05:00
niftynei 4984014578 signpsbt: add utxo info to inputs
If you build a PSBT externally from CLN and attempt to sign for the
output, we would crash. Now we don't crash.

Changelog-Changed: JSON-RPC: `signpsbt` will now add redeemscript + witness-utxo to the PSBT for an input that we can sign for, before signing it.

Fixes #5499 ?
2022-08-13 12:57:03 -05:00
niftynei 63f8c74da9 psbt: dont crash when printing psbt to log
We call `tal_wally_start` and then `tal_wally_start` again in
`type_to_string` for psbt.

end the last tal before calling type to string.

Fixes: #5499
Reported-By: @fiatjaf

lightning_hsmd: FATAL SIGNAL 6 (version v0.11.0.1-231-gddf8fbd-modded)
0x5574ca0d87ef send_backtrace
	common/daemon.c:33
0x5574ca0d8877 crashdump
	common/daemon.c:46
0x7f76ef63e8df ???
	???:0
0x7f76ef68e36c ???
	???:0
0x7f76ef63e837 ???
	???:0
0x7f76ef628534 ???
	???:0
0x5574ca0df55e tal_wally_start
	common/utils.c:27
0x5574ca0e4024 psbt_to_b64
	bitcoin/psbt.c:687
0x5574ca0e4093 fmt_wally_psbt_
	bitcoin/psbt.c:694
0x5574ca0df4b9 type_to_string_
	common/type_to_string.c:32
0x5574ca0d5139 sign_our_inputs
	hsmd/libhsmd.c:486
0x5574ca0d5206 handle_sign_withdrawal_tx
	hsmd/libhsmd.c:1029
0x5574ca0d63c4 hsmd_handle_client_message
	hsmd/libhsmd.c:1575
0x5574ca0ce763 handle_client
	hsmd/hsmd.c:671
0x5574ca100032 next_plan
	ccan/ccan/io/io.c:59
0x5574ca1004b9 do_plan
	ccan/ccan/io/io.c:407
0x5574ca100552 io_ready
	ccan/ccan/io/io.c:417
0x5574ca101daf io_loop
	ccan/ccan/io/poll.c:453
0x5574ca0ceb7b main
	hsmd/hsmd.c:739
0x7f76ef62928f ???
	???:0
0x7f76ef629349 ???
	???:0
0x5574ca0cda04 ???
	../sysdeps/x86_64/start.S:115
2022-08-13 12:57:03 -05:00