For the moment it's a complete tx, but in future designs we might only
be given the specific input which closes the channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It returns NULL, so you can simply `return fromwire_fail(...)`
if you want to return NULL in this case. Use that more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We did not take the value of --commit-fee into account : this removes
the unused option from lightningd and instead registers it in bcli,
where we set the actual feerate of commitment transactions. This also
corrects the documentation.
Changelog-Fixed: config: we now take the --commit-fee parameter into account.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Since we now over-write the wally malloc/free functions, we need to do
so for tests as well. Here we pull up all of the common setup/teardown
logic into a separate place, and update the tests that use libwally to
use the new common_setup core
Changelog-None
This moves the notification for our coin spends from when it's
successfully submited to the mempool to when they're confirmed in a
block.
We also add an 'informational' notice tagged as `spend_track` which
can be used to track which transaction a wallet output was spent in.
Previously we were annotating every movement with the blockheight of
lightningd at notification time. Which is lossy in terms of info, and
won't be helpful for reorg reconciliation. Here we switch over to
logging chain moves iff they've been confirmed.
Next PR will fix this up for withdrawals, which are currently tagged
with a blockheight of zero, since we log on successful send.
On node start we replay onchaind's transactions from the database/from
our loaded htlc table. To keep things tidy, we shouldn't notify the
ledger about these, so we wrap pretty much everything in a flag that
tells us whether or not this is a replay.
There's a very small corner case where dust transactions will get missed
if the node crashes after the htlc has been added to the database but
before we've successfully notified onchaind about it.
Notably, most of the obtrusive updates to onchaind wrappings are due to
the fact that we record dust (ignored outputs) before we receive
confirmation of its confirmation.
These are incoming from onchaind, so the result of any transactions
we've created or outputs we own as a result of a channel closure. These
go into the 'wallet' account.
HTLCs trigger a coin movement only when their final form (state) is
reached. This prevents us from needing to concern ourselves with
retries, as well as being the absolutely most correct in terms of
answering the question 'when has the money irrevocably changed hands'.
All coin movements should pass this bar, for ultimate accounting
correctness
Adds a new plugin notification for getting information about coin
movements. Also includes two 'helper' notification methods that can be
called from within lightningd. Separated from the 'common' set because
the lightningd struct is required to finalize the blockheight etc
Changelog-Added: Plugins: new notification type 'coin_movement'
The current plan for coin movements involves tagging
origination/destination htlc's with a separate tag from 'routed' htlcs
(which pass through our node). In order to do this, we need a persistent flag on
incoming htlcs as to whether or not we are the final destination.
`lightningd` passes in all the known penalty_bases when starting a new
`channeld` instance, which tracks them internally, eventually matching them
with revocations and passing them back to `lightningd` so it can create the
penalty transaction. From here it is just a small step to having `channeld`
also generate the penalty transaction if desired.
When we have only a single member in a TLV (e.g. an optional u64),
wrapping it in a struct is awkward. This changes it to directly
access those fields.
This is not only more elegant (60 fewer lines), it would also be
more cache friendly. That's right: cache hot singles!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's no reason to assign the plugin vars inside the callback, so do
that outside, and the tal_steal() is redundant (the plugin is already
the conn parent).
And reduce duplication by making plugin_conn_finish call plugin_kill:
just make sure we don't call plugin_conn_finish again if plugin_kill
is called externally.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The previous implementation was a bit lazy: in particular, since we didn't
remember the disabled plugins, we would load them on rescan.
Changelog-Changed: config: the `plugin-disable` option works even if specified before the plugin is found.
1. Make the destructor call check_plugins_resolved(),
unless it was uninitialized (`opt_disable_plugin`).
2. Remove redundant list_del (destructor already does it).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
That's more convenient for most callers, which don't need a fmt.
Fixed-by: Darosior <darosior@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is what I expected from plugin_kill, and now all the callers do the
equivalent anywat, it's easy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of calling plugin_kill() and returning, have them
uniformly return an error string or NULL, and have the top
level (plugin_read_json) do the plugin_kill() call.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we now clean up options in startup plugins (that was only
done by dynamic code!), and now they both share the 60 second timeout
instead of 20 seconds for dynamic.
For the dynamic case though, it's 60 seconds to both complete
getmanifest and init, which seems fair.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will allow the dynamic starting code to use them too.
Also lets us move dev_debug_subprocess under #if DEVELOPER.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will let us unify the startup and runtime-started infrastructure.
Note that there are two kinds of notifications:
1. Starting a single plugin (i.e. `plugin start`)
2. Starting multiple plugins (i.e. `plugin rescan` or `plugin startdir`).
In the latter case, we want the command to complete only once *all*
the plugins are dead/finished.
We also call plugin_kill() in all cases, and correctly return afterwards
(it matters once we use the same paths for dynamic plugins, which don't
cause a fatal error if they don't startup).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we know whether the command completed or not, we can correctly
call command_still_pending() if it didn't complete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The symptom (under heavy load and valgrind) in test_plugin_command:
lightningd: common/json_stream.c:237: json_stream_output_: Assertion `!js->reader' failed.
This is because we try to call `getmanifest` again on `pay` which has not yet
responded to init.
The minimal fix for this is to keep proper state, so we can tell the
difference between "not yet called getmanifest" and "not yet finished
init".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed the following in logs for tests/test_connection.py::test_feerate_stress:
```
DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Failing HTLC 18446744073709551615 due to peer death
DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: local_routing_failure: 8194 (WIRE_TEMPORARY_NODE_FAILURE)
```
This is because it reports the (transient) node_failure error, because
our channel_failure message is incomplete. Fix this wart up.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously we've used the term 'funder' to refer to the peer
paying the fees for a transaction; v2 of openchannel will make
this no longer true. Instead we rename this to 'opener', or the
peer sending the 'open_channel' message, since this will be universally
true in a dual-funding world.
This in addition removes the init fixed timeout hack.
Changelog-fixed: We now *always* die if our Bitcoin backend failed unexpectedly.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Commit 9aedb0c61f changed this from allocating off `c` to allocating
off NULL, knowing that it's tal_steal() in the callback. But before
that, it can be detected as a mem leak:
```
@pytest.fixture
def teardown_checks(request):
"""A simple fixture to collect errors during teardown.
We need to collect the errors and raise them as the very last step in the
fixture tree, otherwise some fixtures may not be cleaned up
correctly. Require this fixture in all other fixtures that need to either
cleanup before reporting an error or want to add an error that is to be
reported.
"""
errors = TeardownErrors()
yield errors
if errors.has_errors():
# Format a nice list of everything that went wrong and raise an exception
request.node.has_errors = True
> raise ValueError(str(errors))
E ValueError:
E Node errors:
E Global errors:
E - Node /tmp/ltests-iz9y1chb/test_hsmtool_secret_decryption_1/lightning-1/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E "lightningd/jsonrpc.c:848 (parse_request)",
E "lightningd/jsonrpc.c:941 (read_json)",
E "ccan/ccan/io/io.c:59 (next_plan)",
E "ccan/ccan/io/io.c:407 (do_plan)",
E avis/build/ElementsProject/lightning/lightningd/../plugins/pay
```
Reported-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The plugin can basically return whatever it thinks the preimage is, but we
weren't handling the case in which it doesn't actually match the hash. If it
doesn't match now we just return an error claiming we don't have any matching
invoice.
We use the new function `plugins_free` to define the correct deallocation
order on shutdown, since under normal operation the allocation tree is
organized to allow plugins to terminate and automatically free all dependent
resources. During shutdown the deallocation order is under-defined since
siblings may get freed in any order, but we implicitly rely on them staying
around.
One is called on every plugin return, and tells us whether to continue;
the other is only called if every plugin says ok.
This works for things like payload replacement, where we need to process
the results from each plugin, not just the final one!
We should probably turn everything into a chained callback next
release.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They callback must take ownership of the payload (almost all do, but
now it's explicit).
And since the payload and cb_arg arguments to plugin_hook_call_() are
always identical, make them a single parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have several of these, and they're not always called obvious things like
"delete" or "free". `STEALS` provides a strong hint here.
I only added it to a couple I knew about off the top of my head.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes testing easier, and makes sense: lightningd might not
*know* about other connected channels, depending on gossip, but if the
user specifies it we should obey it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON: `invoice` `exposeprivatechannels` now includes explicitly named channels even if they seem like dead-ends.
This is what actually lets us pay blinded invoices.
Unfortunately, our internal logic assumes every hop in a path has a
next `short_channel_id`, so we have to use a dummy. This is
sufficient for testing, however.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will be used when we want to specify these in a route. But for now, they
only alter gossipd, which always sets them to NULL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that it's channeld which calculates the shared secret, too. This
minimizes the work that lightningd has to do, at cost of passing this
through.
We also don't yet save the blinding field(s) to the database.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This requires us to call ecdh() in the corner case where the blinding seed
is in the TLV itself (which is the case for the start of a blinded route).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now track all pending RPC passthrough calls, and terminate them with an
error if the plugin dies.
Changelog-Fixed: JSON-RPC: Pending RPC method calls are now terminated if the handling plugin exits prematurely.
Use `LC_ALL=C sort` instead of `sort` so that mocks get sorted in
the same way on all developers' environments.
Re-record the result of `make update-mocks`.
Changelog-None
This happened on my testnet node because I've been failing to reconnect to
a node which created a channel and never exchanged announcement sigs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
common/onion is going to need to use this for the case where it finds a blinding
seed inside the TLV. But how it does ecdh is daemon-specific.
We already had this problem for devtools/gossipwith, which supplied a
special hsm_do_ecdh(). This just makes it more general.
So we create a generic ecdh() interface, with a specific implementation
which subdaemons and lightningd can use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently abuse the added_htlc and failed_htlc messages to tell channeld
about existing htlcs when it restarts. It's clearer to have an explicit
'existing_htlc' type which contains all the information for this case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's almost always "their_features" and "our_features" respectively, so
make those names clear.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that now we check capacity once we've figured out which peer, which
broke a test (we returned "unknown peer" instead of "capacity exceeded"),
so we rework that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is useful in general, but in particular it allows fundchannel to avoid YA
query to figure out if it can wumbo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON: `connect` returns `features` of the connected peer on success.
Shows what features we use in various contexts, including those added
by plugins in getmanifest.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugin: `feature_set` object added to `init`
Turns out that unnecessary: all callers can access the feature_set,
so make it much more like a normal primitive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This cleans up the boutique handling of features, and importantly, it
means that if a plugin says to offer a feature in init, we will now
*accept* that feature.
Changelog-Fixed: Plugins: setting an 'init' feature bit allows us to accept it from peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is to prepare for dynamic features, including making plugins first
class citizens at setting them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON: `listnodes` `globalfeatures` output (`features` since in 0.7.3).
Changelog-Removed: JSON: `listpeers` `localfeatures` and `globalfeatures` output (`features` since in 0.7.3).
Changelog-Removed: JSON: `peer_connected` hook `localfeatures` and `globalfeatures` output (`features` since in 0.7.3).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON: `fundchannel` and `fundchannel_start` `satoshi` parameter removed (renamed to `amount` in 0.7.3).
This adapts our fee estimations requests to the Bitcoin backend to the
new semantic, and batch the requests.
This makes our request for fees much simpler, and leaves some more
flexibility for a plugin to do something smart (it could still lie before
but now it's explicit, at least.) as we don't explicitly request
estimation for a specific mode and a target.
Changelog-Changed: We now batch the requests for fee estimation to our Bitcoin backend.
Changelog-Changed: We now get more fine-grained fee estimation from our Bitcoin backend.
We kept track of an URGENT, a NORMAL, and a SLOW feerate. They were used
for opening (NORMAL), mutual (NORMAL), UNILATERAL (URGENT) transactions
as well as minimum and maximum estimations, and onchain resolution.
We now keep track of more fine-grained feerates:
- `opening` used for funding and also misc transactions
- `mutual_close` used for the mutual close transaction
- `unilateral_close` used for unilateral close (commitment transactions)
- `delayed_to_us` used for resolving our output from our unilateral close
- `htlc_resolution` used for resolving onchain HTLCs
- `penalty` used for resolving revoked transactions
We don't modify our requests to our Bitcoin backend, as the next commit
will batch them !
Changelog-deprecated: The "urgent", "slow", and "normal" field of the `feerates` command are now deprecated.
Changelog-added: The fields "opening", "mutual_close", "unilateral_close", "delayed_to_us", "htlc_resolution" and "penalty" have been added to the `feerates` command.
This allows us to set more fine-grained feerate for onchain resolution.
We still give it the same feerate for all types, but this will change as
we move feerates to bcli.
My node crashed as follows:
lightningd: lightningd/peer_control.c:957: peer_connected: Assertion `!peer->uncommitted_channel' failed.
In the logs I found:
Running lightning_openingd: Cannot allocate memory
Which reveals that we're not freeing uc in that path!
Changelog-Fixed: Fix assertion on reconnect if we fail to run openingd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For messages, we use the onion but payload lengths 0 and 1 aren't special.
Create a flag to disable that logic.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
also: convert the stored int value from 'int' to 's64'
atoi fails silently, returning a zero. instead we use the more robust
strtoll which will allow us fail with an error.
we also make the parsing for bools stricter, only allowing plausibly
boolean values to parse.
We were nesting like the following:
```json
{"params": {
"rpc_command": {
"rpc_command": {
}
}
}
```
This is really excessive, so we unwrap once, and now have the following:
```json
{"params": {
"rpc_command": {
}
}
```
Still more wrapping than necessary (the method is repeated in the `params`
object), but it's getting closer.
Changelog-Deprecated: JSON-RPC: Removed double wrapping of `rpc_command` payload in `rpc_command` JSON field.
Suggested-by: @fiatjaf
Signed-off-by: Christian Decker <@cdecker>
Before this patch we would only update `channel->last_tx` with the newly
proposed closure tx from the peer if the fee of the new one was lower.
In negotiations where we are at the higher end and the peer starts
lower, all peer's subsequent proposals will be higher than his initial
proposal and in this case we would never update `channel->last_tx`
and would wrongly broadcast his initial proposal at the end of the
negotiation.
Fixes https://github.com/ElementsProject/lightning/issues/3549
Changelog-Fixed: Always broadcast the latest close transaction at the end of the close fee negotiation, instead of sometimes broadcasting the peer's initial closing proposal.
Does the allocation and copying; this is useful because we can
avoid being fooled into doing giant allocations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ChangeLog-Added: New `getsharedsecret` command, which lets you compute a shared secret with this node knowing only a public point. This implements the BOLT standard of hashing the ECDH point, and is incompatible with ECIES.
Even without optimization, it's faster to walk all the channels than
ping another daemon and wait for the response.
Changelog-Changed: Forwarding messages is now much faster (less inter-daemon traffic)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of saving a stripped_update, we use the new
local_fail_in_htlc_needs_update.
One minor change: we return the more correct
towire_temporary_channel_failure when the node is still syncing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The idea is that gossipd can give us the cupdate we need for an error, and
we wire things up so that we ask for it (async) just before we send the
error to the subdaemon.
I tried many other things, but they were all too high-risk.
1. We need to ask gossipd every time, since it produces these lazily
(in particular, it doesn't actually generate an offline update unless
the channel is used).
2. We can't do async calls in random places, since we'll end up with
an HTLC in limbo. What if another path tries to fail it at the same time?
3. This allows us to use a temporary_node_failure error, and upgrade it
when gossipd replies. This doesn't change any existing assumptions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a common thing to do, so create a macro.
Unfortunately, it still needs the type arg, because the paramter may
be const, and the return cannot be, and C doesn't have a general
"(-const)" cast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
common should not include specific per-daemon files. Turns out this
caused a lot of indirect includes to be exposed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was a pointer into the list of plugins for the hook, but it was rather
unstable: if a plugin exits after handling the event we could end up skipping
a later plugin. We now rely on the much more stable `call_chain` list, so we
can clean up that useless field.
We are attaching the destructor to notify us when the plugin exits, but we
also need to clear them once the request is handled correctly, so we don't
call the destructor when it exits later.
We make the current state of `lightningd` explicit so we don't have to
identify a shutdown by its side-effects. We then use this in order to prevent
the killing and freeing of plugins to continue down the chain of registered
plugins.
We were waiting for both stdin and stdout to close, however that resulted in
us deferring cleanup indefinitely since we did not poll stdout for being
writable most of the time. On the other hand we are almost always polling
the plugin's stdout, so that notifies us as soon as the plugin stops.
Changelog-Fixed: plugin: Plugins no longer linger indefinitely if their process terminates
Before this patch we used to send `double`s over the wire by just
copying them. This is not portable because the internal represenation
of a `double` is implementation specific.
Instead of this, multiply any floating-point numbers that come from
the outside (e.g. JSONs) by 1 million and round them to integers when
handling them.
* Introduce a new param_millionths() that expects a floating-point
number and returns it multipled by 1000000 as an integer.
* Replace param_double() and param_percent() with param_millionths()
* Previously the riskfactor would be allowed to be negative, which must
have been unintentional. This patch changes that to require a
non-negative number.
Changelog-None
This completes the conversion; any in-flight HTLC failures get turned into temporary_node_failures.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This cleans up the "local failure" callers for incoming HTLCs to hand
an onionreply instead of making us generate it from the code inside
make_failmsg.
(The db path still needs make_failmsg, so that's next).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-deprecated: Plugins: htlc_accepted_hook "failure_code" only handles simple cases now, use "failure_message".
Unfortunately the invoice_payment_hook can give us a failcode, so I simply
restrict it to the two sensible ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-deprecated: plugins: invoice_payment_hook "failure_code" only handles simple cases now, use "failure_message".
We tell channeld that an htlc is bad by sending it a 'struct
failed_htlc'. This usually contains an onionreply to forward, but for
the case where the onion itself was bad, it contains a failure code
instead.
This makes the "send a failed_htlc for a bad onion" a completely
separate code path, then we can work on removing failcodes from the
other path.
In several places 'failcode' is now changed to 'badonion' to reflect
that it can only be a BADONION failcode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
At the moment, we store e.g. WIRE_TEMPORARY_CHANNEL_FAILURE, and then
lightningd has a large demux function which turns that into the correct
error message.
Such an enum demuxer is an anti-pattern.
Instead, store the message directly for output HTLCs; channeld now
sends us an error message rather than an error code.
For input HTLCs we will still need the failure code if the onion was
bad (since we need to prompt channeld to send a completely different
message than normal), though we can (and will!) eliminate its use in
non-BADONION failure cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to change our internal structure next, so this is preparation.
We populate existing errors with temporary node failures, for simplicity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of making it ourselves, lightningd does it. Now we only have
two cases of failed htlcs: completely malformed (BADONION), and with
an already-wrapped onion reply to send.
This makes channeld's job much simpler.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I hadn't realized that lightningd asks gossipd every time we forward
a payment. But I'm going to abuse it here to get the latest channel_update,
otherwise (as lightningd takes over error message generation) lightningd
needs to do an async request at various painful points.
So have gossipd tell us the lastest update (stripped so compatible with
the strange in-onion-error format).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turn it into temporary node failure: this only happens if we restart
with a failed htlc in, but it's clearer and more robust to handle it
generically.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For incoming htlcs, we need failure details in case we need to
re-xmit them. But for outgoing htlcs, lightningd is telling us it
already knows they've failed, so we just need to flag them failed
and don't need the details.
Internally, we set the ->fail to a dummy non-NULL value; this is
cleaned up next.
This matters for the next patch, which moves onion handling into
lightningd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. forward_htlc sets hout to NULL.
2. forward_htlc passes &hout to send_htlc_out.
3. forward_htlc checks the failcode and frees(NULL) and sets hout to NULL
(again). This in fact covers every failcode which send_htlc_out returns.
We should ensure send_htlc_out sets *houtp to NULL on failure; in fact,
both callers pass houtp, so we can make it unconditional.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't compile with NDEBUG defined, but if we did, this code would
vanish. I did a quick audit, inspired by @ZmnSCPxj.
I actually hacked up something to compile with NDEBUG (many unused vars
resulted, and of course unit tests are allowed to rely on assert()), and
after this the testsuite still passes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I reproduced this by putting a sleep(60) in the pay plugin, then
'lightning-cli pay', 'lightning-cli plugin stop pay' and then ^C
the `lightning-cli pay`:
2020-02-14T00:33:11.217Z INFO plugin-pay: Killing plugin: pay stopped by lightningd via RPC
2020-02-14T00:33:15.250Z DEBUG lightningd: Still waiting for initial block download
==5157== Invalid read of size 8
==5157== at 0x12A29C: destroy_jcon (jsonrpc.c:149)
==5157== by 0x1C6F2A: notify (tal.c:235)
==5157== by 0x1C7441: del_tree (tal.c:397)
==5157== by 0x1C7493: del_tree (tal.c:407)
==5157== by 0x1C77DD: tal_free (tal.c:481)
==5157== by 0x1B7380: io_close (io.c:450)
==5157== by 0x1B71A7: do_plan (io.c:401)
==5157== by 0x1B7214: io_ready (io.c:417)
==5157== by 0x1B94AC: io_loop (poll.c:445)
==5157== by 0x1291C9: io_loop_with_timers (io_loop_with_timers.c:24)
==5157== by 0x12EC7E: main (lightningd.c:928)
==5157== Address 0x4ebab98 is 40 bytes inside a block of size 88 free'd
==5157== at 0x483BA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==5157== by 0x1C750F: del_tree (tal.c:416)
==5157== by 0x1C7493: del_tree (tal.c:407)
==5157== by 0x1C77DD: tal_free (tal.c:481)
==5157== by 0x153856: clear_plugin (plugin_control.c:209)
==5157== by 0x1538FF: plugin_dynamic_stop (plugin_control.c:225)
==5157== by 0x153C51: json_plugin_control (plugin_control.c:295)
==5157== by 0x12B4EC: command_exec (jsonrpc.c:588)
==5157== by 0x12B8AB: rpc_command_hook_callback (jsonrpc.c:679)
==5157== by 0x154575: plugin_hook_call_ (plugin_hook.c:170)
==5157== by 0x12BCD3: plugin_hook_call_rpc_command (jsonrpc.c:756)
==5157== by 0x12BD04: call_rpc_command_hook (jsonrpc.c:764)
==5157== Block was alloc'd at
==5157== at 0x483A7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==5157== by 0x1C6F98: allocate (tal.c:245)
==5157== by 0x1C7559: tal_alloc_ (tal.c:423)
==5157== by 0x15135A: plugin_rpcmethod_add (plugin.c:706)
==5157== by 0x151600: plugin_rpcmethods_add (plugin.c:756)
==5157== by 0x151BDD: plugin_parse_getmanifest_response (plugin.c:893)
==5157== by 0x151C9C: plugin_manifest_cb (plugin.c:915)
==5157== by 0x14FFB9: plugin_response_handle (plugin.c:258)
==5157== by 0x150165: plugin_read_json_one (plugin.c:356)
==5157== by 0x1502BC: plugin_read_json (plugin.c:388)
==5157== by 0x1B65ED: next_plan (io.c:59)
==5157== by 0x1B71D2: do_plan (io.c:407)
Fixes: #3509
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If the peer is not connected, or other error which means we don't
actually create an outgoing HTLC, we don't record the
short_channel_id. This is unhelpful!
Pass the scid down to the wallet code, and explicitly hand the
scid and amount down to the notification code rather than handing it
the htlc_out (which it doesn't need).
Changelog-Changed: JSON API: `listforwards` now shows `out_channel` even if we couldn't forward.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids the getblockhash+getblock, and more importantly that was the
last functionality making use of bitcoind_getrawblock() and bitcoin_getblockhash(),
so we can also get rid of them.
This adds `getchaininfo` and `getrawblockbyheight` handling lightningd-side,
and use them in setup_topology().
We then remove legacy bitcoind_getblockcount() (we already get the count in
`getchaininfo`), bitcoind_getblockchaininfo() (it was only used in setup_topology()),
and wait_for_bitcoind() (this was specific to bitcoin-core and we assume our Bitcoin
backend to be functional if the plugin responds to `init`).
Add new check if we're funder trying to add HTLC, keeping us
with enough extra funds to pay for another HTLC the peer might add.
We also need to adjust the spendable_msat calculation, and update
various tests which try to unbalance channels. We eliminate
the now-redundant test_channel_drainage entirely.
Changelog-Fixed: Corner case where channel could become unusable (https://github.com/lightningnetwork/lightning-rfc/issues/728)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make the `htlc_accepted` hook the first chained hook in our repertoire. The
plugins are called one after the other in order until we have no more plugins
or the HTLC was handled by one of the plugins. If no plugins handles the HTLC
we continue to handle it internally like always.
Handling in this case means the plugin returns either `{"result": "resolve",
...}` or `{"result": "fail", ...}`.
Changelog-Changed: plugin: Multiple plugins can now register for the htlc_accepted hook.
This used to be necessary because we allocated the `plugin_hook_request` off
of the plugin instance (only tal allocated object we could grab at that
time. Now the plugin was replaced by a list, which itself is tal-allocated,
making that workaround pointless, or even wrong once we have multiple plugins
registering for that hook.
We will be using `plugin_hook_call_next` as part of the loop to traverse all
plugins that registered the hook, so group initialization in the init function
and move per-plugin logic into `plugin_hook_call_next`
We are about to call multiple plugins, and we'll have to pass the payload into
each call. Sadly the serialized stream gets consumed during the call, so keep
the unserialized payload around.
Switch from having a single plugin to a list of plugins. If the hook is of
type single we will enforce that constraint on the number of registered
plugins when attempting to add.
The newly introduced type is used to determine what the call semantics of the
hook are. We have `single` corresponding to the old behavior, as well as
`chain` which allows multiple plugins to register for the hook, and they are
then called sequentially (if all plugins return `{"result": "continue"}`) or
exit the chain if the hook event was handled.
This is the last venue we need to add custom featurebits to, so we also unmark
the test as xfail.
Changelog-Added: plugin: Plugins can now signal support for experimental protocol extensions by registering featurebits for `node_announcement`s, the connection handshake, and for invoices. For now this is limited to non-dynamic plugins only
The `init_featurebits` are computed at startup, and then cached
indefinitely. They are then used whenever a new `init` handshake is performed.
We could add a new message to push updates to `connectd` whenever a plugin is
added or removed, but that's up for discussion.
Break out a method for canceling a channel that will either
loop through contacting the peer to tell them of the error or
just directly cleans up if the peer is currently disconnected.
restrict fundchannel_cancel usage to only the opener side
Changelog-Changed: Only the opener of a fundchannel can cancel the channel open with fundchannel_cancel
As a separated commit because it was pre-existent (changelog + xfail test).
This also fix a logical problem in lightningd/plugin_control: we were
assuming a plugin started with 'plugin start' but which did not comport
a 'dynamic' entry in its manifest to be dynamic, though it should have
been treated as static.
Changelog-fixed: plugins: Dynamic C plugins can now be managed when lightningd is up
Now that we have json_stream in common/, we can move all the related
helpers from lightningd/json to common/json. This way everyone can
benefit of them (including libplugin, the plugins themselves,
potentially lightning-cli), not lightningd alone!
Note that the Makefile of the common/test/ had to be modified, because
the new helpers make use of common/wireaddr... Which turns out to
\#include <lightingd/lightningd.h> ! So we couldnt just include the .c
and add mocks if we redefined some structs (hello run-param).
Changelog-Changed: The hooks `db_write`, `invoice_payment`, and `rpc_command` now accept `{ "result": "continue" }` to mean "do default action", in addition to `true` (`db_write`), `{}` (`invoice_payment`), and `{"continue": true}` (`rpc_command`). The older "default" indicators are now deprecated and are now recognized only if `--deprecated-apis` is set.
Changelog-Added: lightningd: Added --subdaemon command to allow alternate subdaemons.
[ Wow, that was mammoth; 44 comments over 12 commits. Feels almost unfair to squash it into one commit, so I wanted to note @ksedgwic's perseverence here! --RR ]
Using it with a different value to the amount sent causes a crash in 0.8.0,
which is effectively deprecating it, so let's disallow it now.
Changelog-Changed: If the optional `msatoshi` param to sendpay for non-MPP is set, it must be the exact amount sent to the final recipient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We tag them with specific versions when they're experimental,
but do a poor job of cleaning them up (and thus ensuring they're
checked!) afterwards.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before this patch we used `int` for error codes. The problem with
`int` is that we try to pass it to/from wire and the size of `int` is
not defined by the standard. So a sender with 4-byte `int` would write
4 bytes to the wire and a receiver with 2-byte `int` (for example) would
read just 2 bytes from the wire.
To resolve this:
* Introduce an error code type with a known size:
`typedef s32 errcode_t`.
* Change all error code macros to constants of type `errcode_t`.
Constants also play better with gdb - it would visualize the name of
the constant instead of the numeric value.
* Change all functions that take error codes to take the new type
`errcode_t` instead of `int`.
* Introduce towire / fromwire functions to send / receive the newly added
type `errcode_t` and use it instead of `towire_int()`.
In addition:
* Remove the now unneeded `towire_int()`.
* Replace a hardcoded error code `-2` with a new constant
`INVOICE_EXPIRED_DURING_WAIT` (903).
Changelog-Changed: The waitinvoice command would now return error code 903 to designate that the invoice expired during wait, instead of the previous -2
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Relative plugin paths are not relative to startup (deprecated v0.7.2.1)
Changelog-Removed: Dummy fields in listforwards (deprecated v0.7.2.1)
If the same memory gets reallocated, our "has the tip changed?" test
gets a false negative. This happened for me about one time in 10,
causing tests/test_misc.py::test_funding_reorg_remote_lags to fail.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This shouldn't happen if channeld is working properly, but I'm going to
change that, and this current code means we stop responding at that point
(not every failpath in peer_accepted_htlc() called channel_internal_error).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. We asserted that there wouldn't be a raw failcode.
2. We didn't pass the failure information via JSON in this case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We could use sendonion to do this, but it actually takes a different path through
pay, and I wanted to test all of it, so I made a new dev flag.
We currently get upset with the response:
lightningd/pay.c:556: payment_failed: Assertion `!hout->failcode' failed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This solves a couple of issues with the need to synchronously drop the
connection in case we were required to understand what the peer was talking
about while still allowing users to experiment, just not kill connections.
We cannot let users use `sendcustommsg` to inject messages that are handled
internally since it could result in our internal state tracking being borked.
This command injects a custom message into the encrypted transport stream to
the peer, allowing users to build custom protocols on top of c-lightning
without requiring any changes to c-lightning itself.
This is currently in opening_control since that's the only part that has
access to the uncommitted_channel internals. Otherwise it's independent from
the specific daemon.
These messages may be exchanged between the master and any daemon. For now
these are just the daemons that a peer may be attached to at any time since
the first example of this is the custommsg infrastructure.
Fixes: #3192
Changelog-Added: `waitanyinvoice` now supports a `timeout` parameter, which when set will cause the command to fail when the timeout is reached; can set this to 0 to fail immediately if no new invoice has been paid yet.
json_listconfigs() returns in the middle; the name0 is not always freed.
It will be freed later with the response, but our memleak detection doesn't
know that, and Travis caught it:
Global errors:
E - Node /tmp/ltests-5mfrzh5v/test_hsmtool_secret_decryption_1/lightning-1/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:437 (tal_alloc_)",
E "ccan/ccan/tal/tal.c:466 (tal_alloc_arr_)",
E "ccan/ccan/tal/tal.c:794 (tal_dup_)",
E "ccan/ccan/tal/str/str.c:32 (tal_strndup_)",
E "lightningd/options.c:1122 (add_config)",
E "lightningd/options.c:1282 (json_listconfigs)",
E "lightningd/jsonrpc.c:588 (command_exec)",
E "lightningd/jsonrpc.c:679 (rpc_command_hook_callback)",
E "lightningd/plugin_hook.c:123 (plugin_hook_call_)",
E "lightningd/jsonrpc.c:729 (plugin_hook_call_rpc_command)",
E "lightningd/jsonrpc.c:736 (call_rpc_command_hook)",
E "common/timeout.c:39 (timer_expired)",
E "lightningd/io_loop_with_timers.c:32 (io_loop_with_timers)",
E "lightningd/lightningd.c:871 (main)"
E ],
E "label": "lightningd/options.c:1122:char[]",
E "parents": [
E "lightningd/json_stream.c:49:struct json_stream",
E "ccan/ccan/io/io.c:91:struct io_conn",
E "lightningd/lightningd.c:104:struct lightningd"
E ],
E "value": "0x5569ada057a8"
E }
E ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Generally I prefer structures over u8, since the size is enforced at
runtime; and in several places we were doing conversions as the code
using Sphinx does treat struct secret as type of the secret.
Note that passing an array is the same as passing the address, so
changing from 'u8 secret[32]' to 'struct secret secret' means various
'secret' parameters change to '&secret'. Technically, '&secret' also
would have worked before, since '&' is a noop on array, but that's
always seemed a bit weird.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes it clear we're dealing with a message which is a wrapped error
reply (needing unwrap_onionreply), not an already-wrapped one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is needed to fully implement handling of blockheight disagreements
between us and payee.
If payee believes the blockheight is higher than ours, then `pay`
should wait for our node to achieve that blockheight.
Changelog-Add: Implement `waitblockheight` to wait for a specific blockheight.
Add towire_int() and fromwire_int() functions to "(de)serialize"
"int". This will only work as long as both the caller of towire_int()
and the caller of fromwire_int() use the same in-memory representation
of signed integers and have the same sizeof(int).
Changelog-None
Add "peer not connected" and "unknown peer" as error codes, so that
users can check against numeric error codes instead of textual error
messages.
Will ease https://github.com/ElementsProject/lightning/issues/3366
Changelog-None
`wallet_payment_store` would free the `wallet_payment` instance which would
then cause us to reload it from the DB. Instead of doing the store->free->load
dance we now tell `wallet_payment_store` whether it should take ownership and
leave it alone if not.
Passing the payment around instead of referencing it through payment_hash and
partid is a nice side-effect.
`wallet_payment_store` frees the unstored payment after it has stored it, but
we still need that instance for our notifications. This is the smallest
possible fix, but I plan to refactor this out.
Changelog-Changed: plugin: `notify_sendpay_success` and `notify_sendpay_failure` are now always called, even if there is no command waiting on the result.
This lets us do more flexible filtering in the next patch. But it also
keeps some weird logic out of gossipd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This increments the `data_version` upon committing dirty transactions, reads
the last data_version upon startup, and tracks the number in memory in
parallel to the DB (see next commit for rationale).
Changelog-Changed: JSON-RPC: Added a `data_version` field to the `db_write` hook which returns a numeric transaction counter.
it's that time of year (merry xmas!)
enables the ability to push_msat on fundchannel
Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
Thanks to @t-bast, who made this possible by interop testing with Eclair!
Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise tests for hold_invoice fail on Travis (they use 180 / 2 as
the timeout, and we free it after 70 seconds).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the final step: we pass the complete fee_states to and from
channeld.
Changelog-Fixed: "Bad commitment signature" closing channels when we sent back-to-back update_fee messages across multiple reconnects.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Bastien TEINTURIER <bastien@acinq.fr> writes:
> One thing I noticed but didn't investigate much: after sending the two
> payments, I tried using `waitsendpay` and it reported an error *208*
> (*"Never attempted payment for
> '98ee736d29d860948e436546a88b0cc84f267de8818531b0fdbe6ce3d080f22a'"*).
>
> I was expecting the result to be something like: "payment succeeded for
> that payment hash" (the HTLCs were correctly settled).
Indeed, if you waitsendpay without specifying a partid, you are waiting
for 0, which may not exist. Clarify the error msg.
Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Bastien TEINTURIER <bastien@acinq.fr> writes:
> It looks like the split on c-lightning side is quite limited at the moment:
> the only option is to split a payment in exactly its two halves,
> otherwise I get rejected because of the rule of overpaying more than
> twice the amount?
We only tested exactly two equal-size payments; indeed, our finalhop
test was backwards. We only complain if the final hop pays more than
twice msat (technically, this test is still too loose for mpp: the
spec says we should sum to the exact amount).
Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The invoice_try_pay code now takes a set, rather than a single htlc, but
it's basically the same thing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
AFAICT this only "worked" previously because replay htlc simply failed
them all (no peers are currently connected). With upcoming changes
(foreshadowed by the comment) this is no longer true:
Attempting to prepare a db_stmt outside of a transaction: wallet/invoices.c:373
lightningd: FATAL SIGNAL 6 (version v0.7.3-188-g45b0af4-modded)
0x55b475590a73 send_backtrace
common/daemon.c:41
0x55b475590b1d crashdump
common/daemon.c:54
0x7f16c557b46f ???
???:0
0x7f16c557b3eb ???
???:0
0x7f16c555a898 ???
???:0
0x55b475564c8f fatal
lightningd/log.c:814
0x55b4755c3ed5 db_prepare_v2_
wallet/db.c:605
0x55b4755c76b5 invoices_find_unpaid
wallet/invoices.c:373
0x55b4755ce91c wallet_invoice_find_unpaid
wallet/wallet.c:1990
0x55b47555861f invoice_check_payment
lightningd/invoice.c:257
0x55b475557a7c htlc_add_set
lightningd/htlc_set.c:112
0x55b47557b294 handle_localpay
lightningd/peer_htlcs.c:332
0x55b47557c63c htlc_accepted_hook_callback
lightningd/peer_htlcs.c:857
0x55b475585573 plugin_hook_call_
lightningd/plugin_hook.c:118
0x55b47557c747 plugin_hook_call_htlc_accepted
lightningd/peer_htlcs.c:882
0x55b47557ca3e peer_accepted_htlc
lightningd/peer_htlcs.c:991
0x55b47557ffb9 htlcs_resubmit
lightningd/peer_htlcs.c:2131
0x55b4755620f7 main
lightningd/lightningd.c:801
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This isn't plumbed in yet, but the idea is that every htlc gets put
into a "set" and then we process them once the set is satisfied. For
the !EXPERIMENTAL_FEATURES, the set is simply always size 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now return the same error for various "does not match this
invoice", so it makes sense to encapsulate these checks. We'll also
want to expose this for multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Explicit #if EXPERIMENTAL_FEATURES check in case we enable them at different
times, but it requires a payment_secret since we put them in the same field.
This incidently stops it working on legacy nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
msatoshi was used to indicate the amount the invoice asked for, but
for parallel sendpay it's required, as it allows our sanity check of
limiting the total payments in flight, ie. it becomes
'total_msat'.
There's a special case for sendonion, which always tells us the value is 0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently refuse a payment if one is already in flight. For parallel
payments, it's a bit more subtle: we want to refuse if it we already have
the total-amount-of-invoice in flight.
So we get all the current payments, and sum the pending ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, we're about to do surgery on the detection-of-previous-payments
logic, and we should not do this in two places.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a transient field, so rework things so we don't leave it in
struct htlc_out. Instead, load htlc_in first and connect htlc_out to
them as we go.
This also changes one place where we use it instead of the am_origin
flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is in preparation for partial payments. For existing payments,
partid is 0 (to match the corresponding payment).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is in preparation for partial payments. For existing payments,
partid is 0 (arbitrarity) and total_msat is msatoshi.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Because my node runs under valgrind, it can take quite a while to
sync; nodes tend to disconnect and reconnect if you block too long.
This is particularly problematic since we often update fees: when the
other side sends its commitment_signed we block.
In particular, this triggers the corner case we have where we
update_fee twice, disconnecting each time, and our state machine gets
confused (which is why we never saw this exact corner case before this
change in 0.7.3!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now "raw_payload" is always the complete string (including realm or length
bytes at the front).
This has several effects:
1. We can receive an decrypt an onion which is grossly malformed.
2. We can still hand this to the htlc_accepted hook.
3. We then fail it unless the htlc_accepted accepts it manually.
4. The createonion API now takes the raw payload, and does not know
anything about "style".
The only caveat is that the sphinx code needs to know the payload
length: we have a call for that, which simply tells it to copy the
entire onion (and treat us as the final node) if it's invalid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular:
1. It must redirect to an existing command.
2. It must contain method, params and id.
And update the docs to show the id, which is vital.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to have a static Tor service created from a blob bound to
our node on cmdline
Changelog-added: persistent Tor address support
Changelog-added: allow the Tor inbound service port differ from 9735
Signed-off-by: Saibato <saibato.naga@pm.me>
Add base64 encode/decode to common
We need this to encode the blob for the tor service
Signed-off-by: Saibato <saibato.naga@pm.me>
cppcheck found this:
[lightningd/options.c:1137] -> [lightningd/options.c:1120] -> [lightningd/options.c:1193]: (error) Using pointer to local variable 'buf' that is out of scope.
Indeed, answer can point into buf, which is no longer in scope at the end.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we initiated the payment using an externally generated onion we don't know
what the final hop gets, or even who it is, so we don't display the amount in
these cases. I chose to show `null` instead in order not to break dependees
that rely on the value being there.
If we can't decode the onion, because the onion got corrupted or we used
`sendonion` without specifying the `shared_secrets` used, the best we can do
is tell the caller instead.
This means that c-lightning can now internally decrypt an eventual error
message, and not force the caller to implement the decryption. The main
difficulty was that we now have a new state (channels and nodes not specified,
while shared_secrets are specified) which needed to be handled.
When using `sendonion` with `shared_secrets` we may be able to decode the
onioned error message but we cannot infer which node reported the failure
since we don't know which nodes where involved.
We are breaking with a couple of assumptions, namely that we have the
`path_secrets` to decode the error onion. If this happens we just want it to
error out.
These are useful for the `createonion` JSON-RPC we're going to build next. The
secret is used for the optional `session_key` while the hex-encoded binary is
used for the `assocdata` field to which the onion commits. The latter does not
have a constant size, hence the raw binary conversion.
We were using sleeps to hope we catch the password prompt. This makes the test
flaky. So I added a help text followed by a `fflush` to make sure we catcht he
right moment, instead of guessing. The `fflush` is also useful for debugging
if a user ever pipes the output to a file it'd get buffered and the user would
wait forever. The same applies for automated systems such as `expect` or
`pexpect` based scripts that enter the password on prompt.
This will change the command `listconfigs` output in several ways:
- Deprecated the duplicated "plugin" JSON output by replacing it with
- a "plugins" array with substructures for each plugin with:
- path, name and their options
Changelog-Changed: JSON-RPC: `listconfigs` now structures plugins and include their options
Changelog-Deprecated: JSON-RPC: `listconfigs` duplicated "plugin" paths
We don't set the secret to compulsory (yet!) but put code in for the
future. Meanwhile, if there is a secret, check it is correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In a future version, we will use features to insist that payers
provide the secret. In transition, we may have old invoices which
didn't insist on that, so we need to know this on a per-invoice basis.
Not sure if I got the right syntax for adding an empty blob though!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also pulls in a new onion error (mpp_timeout). We change our
route_step_decode_end() to always return the total_msat and optional
secret.
We check total_amount (to prohibit mpp), but we do nothing with
secret for now other than hand it to the htlc_accepted hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Do the same thing '--help' does with them; append `...`.
Valgrind noticed that we weren't NUL-terminarting if answer was over
78 characters.
Changelog-Fixed: JSONRPC: listconfigs appends '...' to truncated config options.
They're already qualified with network name, and there's little point
moving them; it might even be dangerous if multiple are running.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. "conf" can't be specified in a configuration file.
2. "lightning-dir" can't be specified in a configuration file unless the file
was explicitly set with --conf=.
3. "network" options can't be set in a per-network configuration file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets you have a default, but also a network-specific config.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: `config` and <network>/`config` read by default.
lightning-cli is going to need to know what network we're on, so
it will need to parse the config files. Move the code which does
the initial bootstrap parsing into common, as well as the config
file parsing core.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With coming changes, this will segfault if we access it when param
code is trying to get usage from functions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This function ensures we have all the infos we need to continue if the
htlc_accepted hook tells us to. It also enforces well-formedness of the TLV
payload if we have a TLV payload.
Suggested-by: List Neigut <@niftynei>
Signed-off-by: Christian Decker <@cdecker>
We wire in the code-generated function, which removes the upfront validation
and add the validation back after the `htlc_accepted` hook returns. If a
plugin wanted to handle the onion in a special way it'll not have told us to
just continue.
Rounds out the application of `upfront_shutdown_script`, allowing
an accepting node to specify a close_to address.
Prior to this, only the opening node could specify one.
Changelog-Added: Plugins: Allow the 'accepter' to specify an upfront_shutdown_script for a channel via a `close_to` field in the openchannel hook result
This leads to all sorts of problems; in particular it's incredibly
slow (days, weeks!) if bitcoind is a long way back. This also changes
the behaviour of a rescan argument referring to a future block: we will
also refuse to start in that case, which I think is the correct behavior.
We already ignore bitcoind if it goes backwards while we're running.
Also cover a false positive memleak.
Changelog-Fixed: If bitcoind goes backwards (e.g. reindex) refuse to start (unless forced with --rescan).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Spaces just make life a little harder for everyone.
(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The tal overhead of 5 pointers, the linked list node is 2; and we also
tal'd the string. That's 96 bytes per entry.
Use a simple array instead, though it means more work on deletion
since each log_entry is no longer a tal object.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Don't prune the last 10%.
2. Be more aggressive on pruning IO and DEBUG.
3. Account for skipped entries correctly across multiple prunes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.
Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows finegrained logging control of particular subdaemons or
subsystems.
To do this, we defer setting the logging levels for each log object
until after early argument parsing (since e.g. "bitcoind" log object
is created early).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: log-level can now specify different levels for different subsystems.
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
We had a separate logbook for each peer, and copy log entries above
the printable log level into the master logbook. This didn't always
work well, since we didn't dump it on crash for example.
Keep a single global logbook instead, and remove this infrastructure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Simply better encapsulation. We still need to expose log_entry, since the
notification hook uses it though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is ignored in subdaemons which are per-peer, but very useful for
multi-peer daemons like connectd and gossipd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A log can have a default node_id, which can be overridden on a per-entry
basis. This changes the format of logging, so some tests need rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
==1310== Conditional jump or move depends on uninitialised value(s)
==1310== at 0x127C7F: io_loop_with_timers (io_loop_with_timers.c:30)
==1310== by 0x14F0E1: plugins_init (plugin.c:1019)
==1310== by 0x12E4B1: main (lightningd.c:694)
==1310==
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Default is legacy. If we have future styles, new strings can be defined,
but for now it's "tlv" or "legacy".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: JSON API: `htlc_accepted` hook has `type` (currently `legacy` or `tlv`) and other fields directly inside `onion`.
Changelog-deprecated: JSON API: `htlc_accepted` hook `per_hop_v0` object deprecated, as is `short_channel_id` for the final hop.
--dev-force-tmp-channel-id flag takes a 64-character hex string
to use as the temporary channel id. Useful for spec tests
[ Fixed crash in non-DEVELOPER mode --RR ]
Changelog-None
If a 'upfront_shutdown_script' was specified, show the address +
scriptpubky in `listpeers`
Changelog-added: JSON API: `listpeers` channels now include `close_to` and `close_to_addr` iff a `close_to` address was specified at channel open
The 'rpc_command' hook allows a plugin to take over any RPC command.
It sends the complete JSONRPC request to the plugin, which can then respond
with :
- {'continue'}: executes the command normally
- {'replace': {a_jsonrpc_request}}: replaces the request made
- {'return': {'result': {}}}: send a custom response
- {'return': {'error': {}}}: send a custom error
This way, a plugin can modify (/reimplement) or restrict the usage of
any of `lightningd`'s commands.
Changelog-Added: Plugin: A new plugin hook, `rpc_command` allows a plugin to take over `lightningd` for any RPC command.
It's less helpful, sure, but it's far better than someone
sending me their output and leaking this information.
Fixes: #3242
Reported-by: @JavierRSobrino
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I was wondering why TAGS was missing some functions, and finally
tracked it down: PRINTF_FMT() confuses etags if it's at the start
of a function, and it ignores the rest of the file.
So we put PRINTF_FMT at the end, but that doesn't work for
*definitions*, only *declarations*. So we remove it from definitions
and add gratuitous declarations in the few static places.1
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Feerate changes are asymmetric, as they can only be sent by the funder.
For FUNDER, the remote feerate is set when upon send of
commitment_signed, and the local feerate is set on receipt of
revoke_and_ack.
For non-funder, the local feerate is set on receipt of
commitment_signed, and the remote feerate set on send of
revoke_and_ack. In our code, these two happen together.
channeld gets this right, but lightningd ignored the funder/fundee
distinction, and as a result, receipt of a commitment_signed by the
funder altered fees in the database. If there was a reconnection
event or restart, then these (incorrect) values would be used, causing
us to complain about a 'Bad commit_sig signature' and close the
channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A 'Bad commit_sig signature' was reported by @Javier on Telegram and
@DarthCoin. This was between two c-lightning peers, so definitely our fault.
Analysis of this message revealed the signature was using the wrong
feerate. I finally managed to make a test case which triggered this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Takes advantage of upfront-shutdown-script to permit users to
specify the close-to address for a channel at open, by adding
a `close_to` field to `fundchannel_start`.
Note that this only is in effect if `fundchannel_start` returns
with `close_to` set -- otherwise, peer doesn't
support `option_upfront_shutdown_script`.
*If* we know the key has signed something else (as is the case for
channel_announcement) then we can effectively trust the key derivation.
This matches how LND's VerifyMessage works, though in the next patch
we will document it exactly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I wanted to call it verifymessage, but then I read the LND API for that
and wanted nothing to do with it!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I had a report of a 0.7.2 user whose node hadn't appeared on 1ml. Their
node_announcement wasn't visible to my node, either.
I suspect this is a consequence of recent version reducing the amount of
gossip they send, as well as large nodes increasingly turning off gossip
altogether from some peers (as we do). We should ignore timestamp filters
for our own channels: the easiest way to do this is to push them out
directly from gossipd (other messages are sent via the store).
We change channeld to wrap the local channel_announcements: previously
we just handed it to gossipd as for any other gossip message we received
from our peer. Now gossipd knows to push it out, as it's local.
This interferes with the logic in tests/test_misc.py::test_htlc_send_timeout
which expects the node_announcement message last, so we generalize
that too.
[ Thanks to @trueptolmy for bugfix! ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is mainly an internal-only change, especially since we don't
offer any globalfeatures.
However, LND (as of next release) will offer global features, and also
expect option_static_remotekey to be a *global* feature. So we send
our (merged) feature bitset as both global and local in init, and fold
those bitsets together when we get an init msg.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I thought LND had a bug, but turns out it doesn't like out-of-order
short_channel_ids: in fact, the spec says they have to be in order!
This means we use uintmap instead of a htable for unknown_scids and
stale_scids so they're nicely ordered.
But our nodes-missing-announcements probe is harder since they can
also contain duplicates: we switch that to iterate through channels
rather than nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This patch adds a channel_id parameter to allow for specifying
channels that are lacking a short_channel_id.
Useful in the case where a peer has 1) multiple channels (ONCHAIN etc)
and 2) a channel where the funding transaction hasn't been
broadcast/mined.
This splits maybe_create_hsm_secret() in two parts (either encrypted
or in clear) for clarity, and adds an encryption detection in load_hsm().
There are actually three cases if an encryption key is passed:
- There is no hsm_secret => just create it and store the encrypted seed
- There is an encrypted hsm_secret => the provided key should be able to
decrypt the seed, if the wrong key is passed libsodium will nicely error
and hsmd will exit() to not throw a backtrace (using status_failed() as for
other errors) at the face of an user who mistyped its password.
- There is a non-encrypted hsm_secret => load the seed, delete the
hsm_secret, create the hsm_secret, store the encrypted seed.
Add a new startup option which will, if set, prompt the user for a
password to derive a key from. This key will later be used to encrypt
and/or decrypt `hsm_secret`.
This was made a noarg option even if it would have been preferable to
let the user the choice of how to specify the password. Since we have
to chose, better to not let the password in the commands history.
Command format: close id [unilateraltimeout] [destination]
Close the channel with peer {id}, forcing a unilateral
close after {unilateraltimeout} seconds if non-zero, and
the to-local output will be sent to {destination}. If
{destination} isn't specified, the default is the address
of lightningd.
Also change the pylightning:
update the `close` API to support `destination` parameter
`shutdown_scriptpubkey[REMOTE]` is original remote_shutdown_scriptpubkey;
`shutdown_scriptpubkey[LOCAL]` is the script used for "to-local" output when `close`. Add the default is generated form `final_key_idx`;
Store `shutdown_scriptpubkey[LOCAL]` into wallet;
We have split the iteration over the txs and the output in different
functions, so pushing the annotation down, while keeping the transaction
addition atop. This showcases the need to not have the txid reference the
transactions.id in the DB: we annotate in a function that doesn't have the tx
index context, but only add the TX after we have finished extracting.
WIRE_REQUIRED_CHANNEL_FEATURE_MISSING anticipates a glorious Wumbo future,
and is closer to correct (it's a PERM failure).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make --announce-addr with autotor: also
a meaningful use case.
The option --announce-addr=autotor: is more
intuitive than to use the --addr=autotor: option
Signed-off-by: Saibato <saibato.naga@pm.me>
Declare opt_add_addr at top of option.c
We we use opt_add_addr and opt_announce_addr vice versa.
To make compiler happy, we declare it at top.
Signed-off-by: Saibato <saibato.naga@pm.me>
Currently the only source for amount_asset is the value getter on a tx output,
and we don't hand it too far around (mainly ignoring it if it isn't the
chain's main currency). Eventually we could bubble them up to the wallet, use
them to select outputs or actually support assets in the channels.
Since we don't hand them around too widely I thought it was ok for them to be
pass-by-value rather than having to allocate them and pass them around by
reference. They're just 41 bytes currently so the overhead should be ok.
Signed-off-by: Christian Decker <@cdecker>
We now have a pointer to chainparams, that fails valgrind if we do anything
chain-specific before setting it.
Suggested-by: Rusty Russell <@rustyrussell>
The fundchannel plugin needs to know how to build a transaction, so we need to
tell it which chainparams to use. Also adds `chainparams` as a global, since
that seems to be the way to do things in plugins.
Turns out that if we have the init message contain both the chainparams as
well as a transaction that needs to be parsed we need to set the parser to
elements mode before we reach the transaction...
Skipping coinbase transactions and ensuring that the transaction is serialized
correctly when sending it onwards.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The header is not a contiguous section of memory in elements, and it is of
variable length, so the simple trick of hashing in-memory data won't work
anymore. Some of the datafields would have been wrong on big-endian machines
anyway, so this is better anyway.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Using a global variable is a bit lazy, but weaving the network type through
the entire stack is a daunting task. Maybe we can make that happen at a later
stage.
Most of the changes in `chainparams.c` are just formatting the
`genesis_blockhash` a bit nicer (`clang-format` to the rescue).
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The Fairy Tail version of .onion option calls on cmdline
reroute the call to --announce-addr or just drop it in case of bind.
Signed-off-by: Saibato <saibato.naga@pm.me>
And make a function for the 'rescan' subcommand so that we are
consistent and have only subcommand logic in the main function,
which return command_still_pending() in any case (but 'stop').
patched-by: @rustyrussell
This does the same as for the 'start' subcommand for the 'startdir'
one.
Note that we could fail to start the last plugin of a directory, but
have succesfully started the precedent plugins. This will make us return
an error to the user while some of the plugins have been started, but we
still don't end up in a transient state with
half-configured-half-errored plugins.
This remove the reliance on startup plugins' function "plugin_start" in
order to use a distinct runtime path for a dynamically started plugin,
which will allow startup plugins' code to be (almost) agnostic of
dynamic plugins.
This also makes the 'start' subcommand return only if the plugin is
either started, or killed : no weird middle state where the plugin
mishbehaving could crash lightningd.
With enable-autotor-v2 defined in cmdline the default behavior to create
v3 onions with the tor service call, is set to v2 onions.
Signed-off-by: Saibato <saibato.naga@pm.me>
I was seeing some accidental pruning under load / Travis, and in
particular we stopped accepting channel_updates because they were 103
seconds old. But making it too long makes the prune test untenable,
so restore a separate flag that this test can use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We should never open more than 1024 file descriptors anyway, and under some
situations, namely running as root or in docker, would give us huge
allowances. This then results in a huge, unneeded, cleanup for subprocesses,
which we use a lot.
Fixes#2977
The math is a bit tricky, so encapsulate it.
Includes the extra 'e' in 'announcable' as noted by @cdecker :)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will let gossipd be more intelligent about gossiping before we're
synced, and also it might know how far behind we are.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Will be demuxed into starting the selected DB backend in one of the next
commits. Defaults to the old database location.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It's generally clearer to have simple hardcoded numbers with an
#if DEVELOPER around it, than apparent variables which aren't, really.
Interestingly, our pruning test was always kinda broken: we have to pass
two cycles, since l2 will refresh the channel once to avoid pruning.
Do the more obvious thing, and cut the network in half and check that
l1 and l3 time out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The simplest case is to explicitly load it when we see it's been
set.
This involves neatening the default config setup, to remove it from
opt_parse_from_config() and into the caller. It also seems we don't
need to call it anymore before parsing early options: none of them
need ld->config set.
Closes: #3030
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are some more #if DEVELOPER one-liners coming, this makes them
clear, but still lets them stand out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(The json when sendpay successes is too different when sendpay fails, so
divide the sendpay result into two notifications: `sendpay_success` and
`sendpay_failure`)
`sendpay_failure`
A notification for topic `sendpay_failure` is sent every time a sendpay
success(with `failed` status). The json is same as the return value of
command `sendpay`/`waitsendpay` when this cammand fails.
```json
{
"sendpay_failure": {
"code": 204,
"message": "failed: WIRE_UNKNOWN_NEXT_PEER (reply from remote)",
"data": {
"id": 2,
"payment_hash": "9036e3bdbd2515f1e653cb9f22f8e4c49b73aa2c36e937c926f43e33b8db8851",
"destination": "035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d",
"msatoshi": 100000000,
"amount_msat": "100000000msat",
"msatoshi_sent": 100001001,
"amount_sent_msat": "100001001msat",
"created_at": 1561395134,
"status": "failed",
"erring_index": 1,
"failcode": 16394,
"failcodename": "WIRE_UNKNOWN_NEXT_PEER",
"erring_node": "022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59",
"erring_channel": "103x2x1",
"erring_direction": 0
}
}
}
```
`sendpay` doesn't wait for the result of sendpay and `waitsendpay`
returns the result of sendpay in specified time or timeout, but
`sendpay_failure` will always return the result anytime when sendpay
fails if is was subscribed.
pPayment field includes the basic information of the payment, so the return valves of 'sendpay_success()' and 'sendpay_fail()' should include this field.
Note "immediate_routing_failure" is before payment creation, and for this case, return won't include payment fields.
`sendpay_success`
A notification for topic `sendpay_success` is sent every time a sendpay
success(with `complete` status). The json is same as the return value of
command `sendpay`/`waitsendpay` when these cammand succeeds.
```json
{
"sendpay_success": {
"id": 1,
"payment_hash": "5c85bf402b87d4860f4a728e2e58a2418bda92cd7aea0ce494f11670cfbfb206",
"destination": "035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d",
"msatoshi": 100000000,
"amount_msat": "100000000msat",
"msatoshi_sent": 100001001,
"amount_sent_msat": "100001001msat",
"created_at": 1561390572,
"status": "complete",
"payment_preimage": "9540d98095fd7f37687ebb7759e733934234d4f934e34433d4998a37de3733ee"
}
}
```
`sendpay` doesn't wait for the result of sendpay and `waitsendpay`
returns the result of sendpay in specified time or timeout, but
`sendpay_success` will always return the result anytime when sendpay
successes if is was subscribed.
531c8d7d9b
In this one, we always send my_current_per_commitment_point, though it's
ignored. And we have our official feature numbers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This moves field initialization into plugins_new(), and
adds a memleak helper to search the request map:
=================================== ERRORS ====================================
___________________ ERROR at teardown of test_plugin_command ___________________
[gw0] linux -- Python 3.7.1 /opt/python/3.7.1/bin/python3.7
> lambda: ihook(item=item, **kwds),
when=when,
)
../../../.local/lib/python3.7/site-packages/flaky/flaky_pytest_plugin.py:306:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/fixtures.py:112: in node_factory
ok = nf.killall([not n.may_fail for n in nf.nodes])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <utils.NodeFactory object at 0x7f873b245278>, expected_successes = [True]
def killall(self, expected_successes):
"""Returns true if every node we expected to succeed actually succeeded""
unexpected_fail = False
for i in range(len(self.nodes)):
leaks = None
# leak detection upsets VALGRIND by reading uninitialized mem.
# If it's dead, we'll catch it below.
if not VALGRIND:
try:
# This also puts leaks in log.
leaks = self.nodes[i].rpc.dev_memleak()['leaks']
except Exception:
pass
try:
self.nodes[i].stop()
except Exception:
if expected_successes[i]:
unexpected_fail = True
if leaks is not None and len(leaks) != 0:
raise Exception("Node {} has memory leaks: {}".format(
self.nodes[i].daemon.lightning_dir,
> json.dumps(leaks, sort_keys=True, indent=4)
))
E Exception: Node /tmp/ltests-qm87my20/test_plugin_command_1/lightnng-1/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:437 (tal_alloc_)",
E "lightningd/jsonrpc.c:1112 (jsonrpc_request_start_)",
E "lightningd/plugin.c:1041 (plugin_config)",
E "lightningd/plugin.c:1072 (plugins_config)",
E "lightningd/plugin.c:846 (plugin_manifest_cb)",
E "lightningd/plugin.c:252 (plugin_response_handle)",
E "lightningd/plugin.c:342 (plugin_read_json_one)",
E "lightningd/plugin.c:367 (plugin_read_json)",
E "ccan/ccan/io/io.c:59 (next_plan)",
E "ccan/ccan/io/io.c:407 (do_plan)",
E "ccan/ccan/io/io.c:417 (io_ready)",
E "ccan/ccan/io/poll.c:445 (io_loop)",
E "lightningd/io_loop_with_timers.c:24 (io_loop_with_tiers)",
E "lightningd/lightningd.c:840 (main)"
E ],
E "label": "lightningd/jsonrpc.c:1112:struct jsonrpc_reques",
E "parents": [
E "lightningd/plugin.c:66:struct plugin",
E "lightningd/lightningd.c:103:struct lightningd"
E ],
E "value": "0x55d6385e4088"
E },
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:437 (tal_alloc_)",
E "lightningd/jsonrpc.c:1112 (jsonrpc_request_start_)",
E "lightningd/plugin.c:1041 (plugin_config)",
E "lightningd/plugin.c:1072 (plugins_config)",
E "lightningd/plugin.c:846 (plugin_manifest_cb)",
E "lightningd/plugin.c:252 (plugin_response_handle)",
E "lightningd/plugin.c:342 (plugin_read_json_one)",
E "lightningd/plugin.c:367 (plugin_read_json)",
E "ccan/ccan/io/io.c:59 (next_plan)",
E "ccan/ccan/io/io.c:407 (do_plan)",
E "ccan/ccan/io/io.c:417 (io_ready)",
E "ccan/ccan/io/poll.c:445 (io_loop)",
E "lightningd/io_loop_with_timers.c:24 (io_loop_with_tiers)",
E "lightningd/lightningd.c:840 (main)"
E ],
E "label": "lightningd/jsonrpc.c:1112:struct jsonrpc_reques",
E "parents": [
E "lightningd/plugin.c:66:struct plugin",
E "lightningd/lightningd.c:103:struct lightningd"
E ],
E "value": "0x55d6386529d8"
E }
E ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than reaching into data structures, let them register their own
callbacks. This avoids us having to expose "memleak_remove_xxx"
functions, and call them manually.
Under the hood, this is done by having a specially-named tal child of
the thing we want to assist, containing the callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now have a much stronger consistency check from the combination of
transaction wrapping, tal memory leak detection. Tramsaction wrapping ensures
that each statement is executed before the transaction is committed. The
commit is also driven by the `io_loop`, which means that it is no longer
possible for us to have statements outside of transactions and transactions
are guaranteed to commit at the round's end.
By adding the tal-awareness we can also get a much better indication as to
whether we have un-freed statements flying around, which we can test at the
end of the round as well.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We will soon generalize the DB, so directly reaching into the `struct db`
instance to talk to the sqlite3 connection is bad anyway. This increases
flexibility and allows us to tailor the actual implementation to the
underlying DB.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This removes the WIRE_FINAL_EXPIRY_TOO_SOON which leaked too much info,
and adds the blockheight to WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently end up sleeping for 1 second for channeld and gossipd:
better to use a normal blocking waitpid and an alarm to wake us in
case they don't exit.
This speeds up `lightning-cli stop` on my machine from 2.008s to 0.008s:
a 286 times speedup!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During sync it is highly likely that we can coalesce multiple calls and share
results among them. We also report back failures for non-existing blocks early
on, so we don't run into issues with blocks that our bitcoind doesn't have
yet.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was caused by us not checking against the max_blockheight, but rather the
min_blockheight which can be negative with a newly created node. This is still
safe since we check for duplicates anyway in `wallet_filteredblock_add`.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is probably worth preventing.
1. Our depth estimate would be inaccurate possibly leading to us
timing out too early.
2. If we're not up-to-date our onchain funds are unknown.
3. We wouldn't be able to send or receive HTLCs until we're synced anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to still allow incoming connections, and reestablishment of
channels, but if one tries to give us an HTLC, stall until we're
synced.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't know block height, we shouldn't be sending HTLCs. This
stops us forwarding HTLCs as well as new payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I suspect multiple plugins trying to connect at the same
time are overrunning the 1-deep listen queue:
From man listen(2):
The backlog argument defines the maximum length to which the queue of
pending connections for sockfd may grow. If a connection request ar‐
rives when the queue is full, the client may receive an error with an
indication of ECONNREFUSED
Fixes: #2922
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call. `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.
Timing out JSON calls is generally deprecated: that's the job of the
client. And the semantics of this are confusing, even to me! A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.
The transition code is awkward, but we'll manage for the three
releases until we can remove it.
The new defaults are to unilaterally close after 48 hours.
Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we were to just insert filtered blocks in the range that we will scan later
we'd be hitting the uniqueness constraints later.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Instead of allowing all calls to `getfilteredblock` to be scheduled on the
`bitcoind` queue right away we instead add them in a separate queue, and
process a single call at a time. This limits the concurrency and avoids
thrashing `bitcoind`. At the same time we dispatch incoming results back to
all calls that were queued for that particular blockheight, reducing the
overall number of calls and an increase in overall speed.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We will be calling the callback out of order once we fan out the results of a
single lookip to multiple calls, so being sure that everything is allocated
ahead of time is necessary.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Since we now check all P2WSH outputs in a block, this is getting quite a
common occurence, so logging just produces lots of noise.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This will eventually replace the multi-step `getblockhash` + `getblock` +
`gettxout` mechanism, and return entire filtered blocks which can be added to
the DB, and represent the full set of P2WSH UTXOs.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was causing `--help` to fail if we already had a `lightningd` running
with the same `--lightning-dir`.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
1. Now checking the pid file really does precede touching the db and
starting plugins, which is far safer.
2. Crashlog is now activated just after daemon parent release, and just
before the main loop, which means no "crash" on startup if we call fatal().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Dumb programs which have a --daemon option call fork() early. This is
terrible UX since startup errors get lost: the program exits with
"success" immediately then you discover via the logs that it didn't
start at all.
However, forking late introduced a heap of problems with changing
pids. Instead, fork early but keep stderr and the parent around: if
we fail early on, the parent fails with us. We release our parent
with an explicit action just before the main loop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we are walking the entire allocation tree anyway, and access the tal
metadata anyway, we can just as well also track the size of the memory
allocations to simplify debugging of memory use.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Otherwise it creates the lightning-dir. This can't be helped for --help
(at least, if plugins are present), but --version simply prints and exits.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that we move adding the plugin to the plugins list to the end, otherwise
the hook from logging can examine the (uninitialized) plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is easy since we did the option parsing cleanup, but it has the
effect that plugins are launched from the lightning-dir. Now
we have dynamic plugins, this means startup and post-startup plugins
experience the same environment.
This is absolutely a desirable thing: they can just drop files in
their cwd rather than having to move (including, I might note, core
files!).
We also highlight the change in various places (and a drive-up update
of PLUGINS.md which says you have to use --plugin).
The next patch adds a backwards compatibility wedge for old users of
relative plugin paths.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently send channel_announcement as soon as we and our
peer agree it's 6 blocks deep. In theory, our other peers might
not have seen that block yet though, so delay a little.
This is mitigated by two factors:
1. lnd will stash any "not ready yet" channel_announcements anyway.
2. c-lightning doesn't enforce the 6 depth minimum at all.
We should not rely on other nodes' generosity or laxity, however!
Next release, we can start enforcing the depth limit, and maybe stashing
ones which don't quite make it (or simply enforce depth 5, not 6).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In file included from wallet/test/run-wallet.c:15:0:
./lightningd/peer_htlcs.c: In function ‘htlcs_reconnect’:
./lightningd/peer_htlcs.c:2060:15: error: ‘failcode’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
} else if (failcode) {
^~~~~~~~
./lightningd/peer_htlcs.c:2056:19: error: ‘failcode’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
failcode != 0
~~~~~~~~~^~~~
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`forward_event`
A notification for topic `forward_event` is sent every time the status
of a forward payment is set. The json format is same as the API
`listforwards`.
```json
{
"forward_event": {
"payment_hash": "f5a6a059a25d1e329d9b094aeeec8c2191ca037d3f5b0662e21ae850debe8ea2",
"in_channel": "103x2x1",
"out_channel": "103x1x1",
"in_msatoshi": 100001001,
"in_msat": "100001001msat",
"out_msatoshi": 100000000,
"out_msat": "100000000msat",
"fee": 1001,
"fee_msat": "1001msat",
"status": "settled",
"received_time": 1560696342.368,
"resolved_time": 1560696342.556
}
}
```
or
```json
{
"forward_event": {
"payment_hash": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
"in_channel": "103x2x1",
"out_channel": "110x1x0",
"in_msatoshi": 100001001,
"in_msat": "100001001msat",
"out_msatoshi": 100000000,
"out_msat": "100000000msat",
"fee": 1001,
"fee_msat": "1001msat",
"status": "local_failed",
"failcode": 16392,
"failreason": "WIRE_PERMANENT_CHANNEL_FAILURE",
"received_time": 1560696343.052
}
}
```
- The status includes `offered`, `settled`, `failed` and `local_failed`,
and they are all string type in json.
- When the forward payment is valid for us, we'll set `offered`
and send the forward payment to next hop to resolve;
- When the payment forwarded by us gets paid eventually, the forward
payment will change the status from `offered` to `settled`;
- If payment fails locally(like failing to resolve locally) or the
corresponding htlc with next hop fails(like htlc timeout), we will
set the status as `local_failed`. `local_failed` may be set before
setting `offered` or after setting `offered`. In fact, from the
time we receive the htlc of the previous hop, all we can know the
cause of the failure is treated as `local_failed`. `local_failed`
only occuors locally or happens in the htlc between us and next hop;
- If `local_failed` is set before `offered`, this
means we just received htlc from the previous hop and haven't
generate htlc for next hop. In this case, the json of `forward_event`
sets the fields of `out_msatoshi`, `out_msat`,`fee` and `out_channel`
as 0;
- Note: In fact, for this case we may be not sure if this incoming
htlc represents a pay to us or a payment we need to forward.
We just simply treat all incoming failed to resolve as
`local_failed`.
- Only in `local_failed` case, json includes `failcode` and
`failreason` fields;
- `failed` means the payment forwarded by us fails in the
latter hops, and the failure isn't related to us, so we aren't
accessed to the fail reason. `failed` must be set after
`offered`.
- `failed` case doesn't include `failcode` and `failreason`
fields;
- `received_time` means when we received the htlc of this payment from
the previous peer. It will be contained into all status case;
- `resolved_time` means when the htlc of this payment between us and the
next peer was resolved. The resolved result may success or fail, so
only `settled` and `failed` case contain `resolved_time`;
- The `failcode` and `failreason` are defined in [BOLT 4][bolt4-failure-codes].
Warp this process as a new function: 'void json_format_forwarding_object()'. This function will be used in 'forward_event' next, and can ensure the consistent json object structure for forward_payment between 'listforwards' API and 'forward_event' notification.
The reason lnd was sending sync error was that we were taking more than
30 seconds to send the channel_reestablish after connect. That's
understandable on my test node under valgrind, but shouldn't happen normally.
However, it seems it has at least once,
(see https://github.com/ElementsProject/lightning/issues/2847)
: space out startup so it's less likely to happen.
Suggested-by: @cfromknecht
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
niftynei
Rusty Russell
Antoine Poinsot
Christian Decker
rbndg
Sebastian Falbesoner
Vasil Dimov
Michael Schmoock
ZmnSCPxj jxPCSnmZ
Ken Sedgwick
nicolas.dorier
Saibato
fiatjaf
gorazdko
trueptolemy
Yash Bhutwala
trueptolemy
Simon Vrouwe
Rene Pickhardt