Commit Graph

7816 Commits

Author SHA1 Message Date
Rusty Russell 3c625b367d doc: update cli documentation.
1. help now uses a boutique format, not -H.
2. document the -F option.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 15:25:44 +02:00
Rusty Russell abb16b4226 cli: implement new 'flattened JSON' mode.
Much nicer for grepping, since `{ "foo": { "bar": [7] } }` is turned into
`foo.bar[0]=7`.

Changelog-Added: cli: New `--flat` mode for easy grepping.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 15:25:44 +02:00
Rusty Russell 679d3494b4 cli: tweak -H output to remove format-hint fields.
-H removes the top-level if there's only one, and 'format-hint'
breaks this heuristic, so we end up with:

```
help=command=autocleaninvoice [cycle_seconds] [expired_by]
category=plugin
description=Set up autoclean of expired invoices.
verbose=Perform cleanup every {cycle_seconds} (default 3600), or disable autoclean if 0. Clean up expired invoices that have expired for {expired_by} seconds (default 86400).
command=check command_to_check
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 15:25:44 +02:00
mb300sd b6285ffa5d Fix bash completion.
At some point lightning-cli help defaulted to human readable format, the additional -H broke the bash completion.

Changelog-Fixed: bash completion on lightning-cli now works again
2020-05-15 15:25:33 +02:00
Rusty Russell 97e3d61748 pyln: fix incorrect python syntax.
contrib/pyln-proto/pyln/proto/bech32.py:120
  /home/rusty/devel/cvs/lightning/contrib/pyln-proto/pyln/proto/bech32.py:120: SyntaxWarning: "is not" with a literal. Did you mean "!="?
    assert decode(hrp, ret) is not (None, None)

I think this warning is correct (though I don't see the warning once I installed coincurve:
are we suppressing warnings?)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 15:21:50 +02:00
Rusty Russell 4bcac3032c bitcoin: remove unused struct bitcoin_tx_input.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 09:55:59 +09:30
Rusty Russell 88402bc234 Makefile: fix clean and distclean for external libs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 09:55:59 +09:30
William Casarin d6ede8db1b build: cleanup compiler output
Hide CFLAGS and LDFLAGS line noise each time an object file is compiled
or linked.

Also add a `make show-flags` command for displaying CC, LD, CFLAGS and
LDFLAG information. This is shown at the start of each build.

Use `V=1 make` to restore original output

make

  CC: gcc -DBINTOPKGLIBEXECDIR="../libexec/c-lightning" -Wall [..]
  LD: gcc   -Og    -Lexternal -lwallycore -lsecp256k1 -ljsmn  [..]
  ...
  cc wallet/test/run-db.c
  cc lightningd/test/run-jsonrpc.c
  cc lightningd/test/run-invoice-select-inchan.c
  cc lightningd/test/run-log-pruning.c
  cc lightningd/test/run-find_my_abspath.c
  cc cli/test/run-large-input.c
  cc cli/test/run-remove-hint.c
  ld lightningd/lightning_hsmd
  ld lightningd/lightning_gossipd
  ld lightningd/lightning_openingd
  ld lightningd/lightning_channeld
  ld lightningd/lightning_closingd
  ...

Signed-off-by: William Casarin <jb55@jb55.com>
Changelog-Changed: build: default compile output is prettier and much less verbose
2020-05-14 17:44:01 +02:00
Rusty Russell 75a93ec32f lightningd: EXPERIMENTAL_FEATURES: fix crash caused by test_sendonionmessage
Commit b0c9059602 broke the case where
next_node_id is NULL:

[libsecp256k1] illegal argument: pubkey != NULL
lightning_channeld: FATAL SIGNAL 6 (version 13d9c27)
0x55b4cd261b64 send_backtrace
	common/daemon.c:39
0x55b4cd261c0e crashdump
	common/daemon.c:52
0x7fc60307746f ???
	???:0
0x7fc6030773eb ???
	???:0
0x7fc603056898 ???
	???:0
0x55b4cd2c7cee ???
	???:0
0x55b4cd2d74d4 ???
	???:0
0x55b4cd26ac62 node_id_from_pubkey
	common/node_id.c:12
0x55b4cd24e194 handle_onion_message
	channeld/channeld.c:1890
0x55b4cd24e697 peer_in
	channeld/channeld.c:2001
0x55b4cd2521f4 main
	channeld/channeld.c:3419
0x7fc6030581e2 ???
	???:0
0x55b4cd24881d ???
	???:0
0xffffffffffffffff ???
	???:0

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-14 12:55:42 +02:00
Candle fbfe0e6c0b Comment optimisations
I quickly ran through the comments in lightning.py and saw a few small inconsistencies:

- upper/lower case for the "B" in "Bitcoin" unified (see https://github.com/lnbook/lnbook/pull/98)
- added missing "." after a complete sentence
- removed unnecessary double spaces
2020-05-12 22:19:34 +02:00
Rusty Russell 1fb505d118 pytest: extend timeout for test_payment_duplicate_uncommitted
We've been seeing some Travis timeouts under VALGRIND, with the
10 second timeout here: use TIMEOUT as per standard.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-12 21:57:43 +02:00
Sebastian Falbesoner 0bf93137f6 devtools/credit: check if tag exists first
Changelog-None
2020-05-12 19:48:05 +02:00
lisa neigut b881f7337e make the experimental features branch work ok 2020-05-12 15:46:58 +09:30
lisa neigut d35149e5ca try to fix travis test flakes 2020-05-12 15:46:58 +09:30
lisa neigut 487771aee5 tests: skip longer htlc tests on valgring+slow machines
These guys take a while to run, so let's just skip them on the
valgrind/slow-machine combos :/
2020-05-12 15:46:58 +09:30
lisa neigut 6e4b4c50f0 coin tests: try to make a bit more travis robust 2020-05-12 15:46:58 +09:30
lisa neigut 8acbbca05d coins: use the chain's BIP173 name instead of a 'unit of account'
Updates the unit of account to be the chain_id, which is the BIP173 name
of the chain that the coins moved on.

Suggested-By: @rustyrussell
2020-05-12 15:46:58 +09:30
lisa neigut de065580f6 coins: update API surface for creating coin movements
Canonicalize the signature for the 'tag-type' of coin moves by unique
constructor/method calls.

Suggested-By: @rustyrussell
2020-05-12 15:46:58 +09:30
lisa neigut aab9893661 coins: have `we_fulfilled` be fully 'ternary'
note that 'null' 'we_fulfilled's are going to be legacy from this
release forward.
2020-05-12 15:46:58 +09:30
lisa neigut 8537e77ac7 coins: re-write API interface for htlc notices
Wrap up more logic internally to the method call for htlcs. Also, don't
touch part id if we're not the 'origin'

Suggested-By: @rustyrussell
2020-05-12 15:46:58 +09:30
lisa neigut 4bfbb58c56 coins: fix feerate for withdraw so test work? 2020-05-12 15:46:58 +09:30
lisa neigut 087ab166b3 coins, fix: don't crash if the to_us amount is greater than our_msat
It's possible for our peer to publish a commitment tx that has already
updated our balance for an htlc before we've completed removing it from
our commitment tx (aka before we've updated our balance). This used to
crash, now we just update our balance (and the channel balance logs!)
and keep going.

If they've removed anything from our balance, we'll end up counting it
as chain_fees below. Not ideal but fine... probably.
2020-05-12 15:46:58 +09:30
lisa neigut ffd9467f14 coin moves: persist the coin movement index counter to disk
Should make it easier to track when coin moves in the plugin are
disjoint from what c-lightning says it's broadcast already.
2020-05-12 15:46:58 +09:30
lisa neigut 44f747b29a coin moves: update PLUGINS.md with details about `coin_movement`s
Changelog-Added: Plugins: new notification type, `coin_movement`, which tracks all fund movements for a node
2020-05-12 15:46:58 +09:30
lisa neigut de86e29e16 coin moves: log all withdrawals when confirmed in a block
This moves the notification for our coin spends from when it's
successfully submited to the mempool to when they're confirmed in a
block.

We also add an 'informational' notice tagged as `spend_track` which
can be used to track which transaction a wallet output was spent in.
2020-05-12 15:46:58 +09:30
lisa neigut e9d26a46e0 coin moves: actually record the blockheight for all chain moves
Previously we were annotating every movement with the blockheight of
lightningd at notification time. Which is lossy in terms of info, and
won't be helpful for reorg reconciliation. Here we switch over to
logging chain moves iff they've been confirmed.

Next PR will fix this up for withdrawals, which are currently tagged
with a blockheight of zero, since we log on successful send.
2020-05-12 15:46:58 +09:30
lisa neigut 9caf20f636 coin moves: don't log coin moves in onchaind if we're replaying
On node start we replay onchaind's transactions from the database/from
our loaded htlc table.  To keep things tidy, we shouldn't notify the
ledger about these, so we wrap pretty much everything in a flag that
tells us whether or not this is a replay.

There's a very small corner case where dust transactions will get missed
if the node crashes after the htlc has been added to the database but
before we've successfully notified onchaind about it.

Notably, most of the obtrusive updates to onchaind wrappings are due to
the fact that we record dust (ignored outputs) before we receive
confirmation of its confirmation.
2020-05-12 15:46:58 +09:30
lisa neigut 1b5221cbf5 coin moves tests: push_msat and the wallet withdrawal
Check that we account for push_msat and wallet withdrawal/deposits
correctly
2020-05-12 15:46:58 +09:30
lisa neigut 8c986d67db coin moves: test all of the onchaind variants
Mostly we update existing tests to account for channel balances. In a
few places, new tests were needed as there wasn't an existing pathway
that tested the chain-fees for a few penalty cases
2020-05-12 15:46:58 +09:30
lisa neigut 41d3471c7f coin moves tests: save updates to disk
If we don't save to disk, if the node restarts we'll lose them all
and the resulting balance check at the end will be incorrect.
2020-05-12 15:46:58 +09:30
lisa neigut 5d58f125c5 coin moves: record withdrawals
For every withdrawal transaction emitted, we record each of the outputs
plus the fees paid for this transaction.
2020-05-12 15:46:58 +09:30
lisa neigut fc54bfc488 coin moves: record wallet deposits 2020-05-12 15:46:58 +09:30
lisa neigut e623143c9f fix: the tx types for element fee updates were annotated wrong
Fixes the tx type annotation in a few places.
2020-05-12 15:46:58 +09:30
lisa neigut 9b429cdf4f coin moves: some additional htlc handling
We record htlcs when they're fulfilled as 'withdrawals' that are
onchain. This should make use of the payment_hash that we stashed.

Additionally, if an htlc spend comes through that's not ours, it's
probably them resolving our attempted cheat; we should allow it to
proceed without bombing, and just do our accounting as necessary. It'll
all come out in the wash.
2020-05-12 15:46:58 +09:30
lisa neigut 6ee6cdc280 coin moves: record their attempted cheat (and our handling thereof)
For cheats, we do a little bit of weird accounting. First we 'update'
our on-ledger balance to be the entirety of the channel's balance. Then,
as outputs get resolved, we record the fees and outputs as withdrawals
from this amount.

It's possible that they might successfully 'cheat', in which case we
record those as 'penalty' but debits (not credits).
2020-05-12 15:46:58 +09:30
lisa neigut 007a62a369 coin moves: handle ignored outputs
Ignored outputs don't end up in the same 'resolved' pathway as other
tracked outputs do, so we mark them as moved when proposed/broadcast
instead of when resolved (since they'll never flow through as resolved)
2020-05-12 15:46:58 +09:30
lisa neigut 19268afba5 coin moves: record moves for unilateral closes 2020-05-12 15:46:58 +09:30
lisa neigut c215a00c45 coin moves: record cheats / loss due to 'unknown' txs
Whenever we detect an 'unknown' tx is published, we should count that as
a loss, as needed.
2020-05-12 15:46:58 +09:30
lisa neigut 034b2c7ee4 coin moves: add handling for mutual closure case 2020-05-12 15:46:58 +09:30
lisa neigut 1f0cfa71b0 coin moves: pass our currently known channel balance to onchaind
We'll need it to do reconciliation for unexpected/penalty closures and
to compute fees paid / outputs trimmed.
2020-05-12 15:46:58 +09:30
lisa neigut 15e4e922c9 wire: serialize the amounts for a bitcoin tx over the wire
we'll need this for calculating fees etc in onchaind
2020-05-12 15:46:58 +09:30
lisa neigut e8d10edbe5 coin moves: record onchain movements after they've been resolved
We pass them back to lightningd who sends out a notification for them
2020-05-12 15:46:58 +09:30
lisa neigut 175515b914 coin moves: stash the payment hash for onchaind in tracked_output
We'll use it when we record 'onchain' htlc movements that are fulfilled
2020-05-12 15:46:58 +09:30
lisa neigut 6914adf861 coin moves: record new wallet utxos as deposits
These are incoming from onchaind, so the result of any transactions
we've created or outputs we own as a result of a channel closure. These
go into the 'wallet' account.
2020-05-12 15:46:58 +09:30
lisa neigut c6e0cf9279 coin moves: record the fees and outputs for any wallet withdrawal
Here we record coin moves that are withdrawals and any chain fees we pay
for those outgoing transactions.
2020-05-12 15:46:58 +09:30
lisa neigut dcde37581e coin moves: add wire handlers for chain coin moves
onchaind is the only daemon that emits coin events, and those are all
onchain (ha!), so the only 'wire' facility we need for coin moves are
for the 'chain' type.
2020-05-12 15:46:58 +09:30
lisa neigut 3a4a3597a3 amount: add a helper for msat == sat
Handy dandy method for verifying that a millisatoshi-typed amount is
equivalent to a satoshi-typed amount.
2020-05-12 15:46:58 +09:30
lisa neigut 824e8fa72b tx: split 'compute fee' into two, with one that takes an input value
Transactions that we 'get' from bitciond don't have the input values
available on the transaction; for these cases we'll sum up the inputs
amounts using a different data source than the transaction's
`input_amounts`. So we need to expose it here.
2020-05-12 15:46:58 +09:30
lisa neigut 3d241bc261 tx: expose elements 'is fee' calculation
we'll use it when figuring out what outputs to account for during
a withdrawal.
2020-05-12 15:46:58 +09:30
lisa neigut 9c4c0f10fb coin_mvt: add integration tests for in-channel htlc movements 2020-05-12 15:46:58 +09:30