Commit Graph

7324 Commits

Author SHA1 Message Date
lisa neigut 963a1da958 addr: handle P2SH/P2PKH in scriptpubkey encoding
Previously, returned null if a scriptpubkey was not Segwit; now
handles encoding to Base58 for other types.
2019-11-13 03:31:20 +00:00
lisa neigut 6f215a70e5 contrib: fixup documentation of l1/2-log 2019-11-13 03:31:20 +00:00
Rusty Russell 7679f25aec devtools: fix mkcommit crash.
Needs to initialize global now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-13 01:45:32 +00:00
Christian Decker 9378be72a9 pyln: Allow users to override the LightningNode class
Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.
2019-11-12 21:23:55 +01:00
Christian Decker bc9b1c405e pyln: Move RPC and daemon instantiation into the LightningNode
We were relying heavily on NodeFactory to do some magic before instantiating
the Node with rpc and daemon initialized, that meant that we'd have to replace
all 3 classes when customizing the node to our needs. Moving that
initialization into the node itself means that the LightningNode class now can
be swapped out and customized, without having to wire everything else through.
2019-11-12 21:23:55 +01:00
Christian Decker 53a2789547 pyln: Default to DEVELOPER=0 when running outside of tree
`DEVELOPER=1` assumes that the binary has been compiled with developer set to
true, which might not be the case for plugin developers. Setting this to 0 by
default has no effect in c-lightning since we always at least set it in
`config.vars` but may prevent some issues outside.
2019-11-12 21:23:55 +01:00
Christian Decker d58339b872 pytest: Migrate to in-tree pyln-testing 2019-11-12 21:23:55 +01:00
Christian Decker b7e7e535c8 pytest: Migrate `env` to pyln-testing 2019-11-12 21:23:55 +01:00
Christian Decker b31defdf96 make: Add in-tree pyln-testing to python path
We'll start relying on the in-tree version pretty soon, so we better make
sure we find it.
2019-11-12 21:23:55 +01:00
Christian Decker f6a016c17d pytest: Consolidate access to environment variables
We use `env()` to look up configuration variables and allow them to be
overridden by the environment.
2019-11-12 21:23:55 +01:00
Christian Decker a2a3d33802 pyln-testing: Copy basic support infrastructure into pyln.testing
We'll rewrite the tests to use this infrastructure in the next commit.

Changelog-Added: The new `pyln-testing` package now contains the testing infrastructure so it can be reused to test against c-lighting in external projects
2019-11-12 21:23:55 +01:00
Christian Decker 81f7978cc8 pyln: Migrate implementation from pylightning to pyln-client
This should not affect any consumer of the API since we just shift the actual
implementation from one side to the other, and keep aliases in place so
scripts don't break.

We also bump the version number from 0.0.7.3 to 0.7.4 which allows us to be in
sync with c-lightning itself, and remove the superfluous `0` in front.
2019-11-12 21:23:55 +01:00
darosior de91eda36d hsmtool: add a tool to dump commitment points and secrets
This takes a dbid, a "depth" (how many points to dump), the hsm_secret
path, and a potential password to dump informations about all
commitments until the depth.

Co-Authored-By: Sjors Provoost <sjors@sprovoost.nl>
2019-11-12 05:29:19 +00:00
darosior c61b60b8fb README: move hsm_secret encryption infos into a dedicated part
And tell about decryption/encryption with hsmtool
2019-11-12 05:29:19 +00:00
darosior f97d548582 pytest: test hsm_secret management with hsmtool 2019-11-12 05:29:19 +00:00
darosior dccad7784f tools/hsmtool: add a tool for encrypting hsm_secret 2019-11-12 05:29:19 +00:00
darosior 04762a1d38 tools/hsmtool: add a tool for decrypting hsm_secret
A general one, for all things hsm_secret.
2019-11-12 05:29:19 +00:00
darosior 1171b5dba2 pytest: clean up hsm_secret_encryption test 2019-11-12 05:29:19 +00:00
darosior 6427ccba50 Document the 'rpc_command' hook 2019-11-11 22:30:01 +01:00
darosior 95c6513ff3 pytest: test the 'rpc_command' hook 2019-11-11 22:30:01 +01:00
darosior a0df49718a lightningd/jsonrpc: Add a 'rpc_command' hook
The 'rpc_command' hook allows a plugin to take over any RPC command.
It sends the complete JSONRPC request to the plugin, which can then respond
with :
- {'continue'}: executes the command normally
- {'replace': {a_jsonrpc_request}}: replaces the request made
- {'return': {'result': {}}}: send a custom response
- {'return': {'error': {}}}: send a custom error

This way, a plugin can modify (/reimplement) or restrict the usage of
any of `lightningd`'s commands.

Changelog-Added: Plugin: A new plugin hook, `rpc_command` allows a plugin to take over `lightningd` for any RPC command.
2019-11-11 22:30:01 +01:00
darosior dd10b543da lightningd/json: Add a json helper to append any jsmn token to a stream 2019-11-11 22:30:01 +01:00
Saibato 187d2e0f26 We state not to do 'any local DNS' if --always-use-proxy flag is set, so we should this
Even if it is on startup only once ...
Like @bitcoin-software indicated the expected UX should be in
line with what a user expects the software will do
so we should not dns if we say so with a flag that suggest that.

Changelog-Fixed: We disable all dns even on startup the scan for bogus dns servers, if --always-use-proxy is set true

Signed-off-by: Saibato <saibato.naga@pm.me>
2019-11-11 00:04:07 +00:00
Rusty Russell 5df9e5b7b4 gossipd: allow node_announcements and channel_announcements with unsupported features.
The flat feature PR changes the rules so these are OK to propagate.
That makes sense: the unsupported features means there's something
unsupported about the *node* or *channel*, not the msg itself
(for that we'd use a different message type).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-10 10:42:29 +01:00
Rusty Russell c62f0cb6ff sphinx: fix potential data leak.
https://github.com/lightningnetwork/lightning-rfc/pull/697
https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002288.html

We generate it from an hmac using the session secret.  It's not
clear that this will be useful for reproducing test vectors though,
since we don't generate the first 66 bytes, which is what the
spec says to do.

Reported-by: @roasbeef
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-08 13:05:20 +01:00
darosior 208123771e pytest: test shadow routing in pay plugin 2019-11-08 03:27:58 +00:00
darosior 7dac7a8c52 pytest: deactivate shadow routing for some tests that use 'pay'
So that we can assert payment values
2019-11-08 03:27:58 +00:00
darosior f3d304901b pay: add a dev-only parameter to deactivate shadow routing
Had to make a special pylightning function to avoid rewriting all calls
to 'pay()' with 'rpc.call()' in the next commit..
2019-11-08 03:27:58 +00:00
darosior 16f5af00c7 common/json: add a helper for json to u16
As Rusty pointed out to me, the gossip protocol restricts cltvs to u16
so at least we'll use this helper for them.
2019-11-08 03:27:58 +00:00
darosior 5766231d98 pay: restore payment value randomization through shadow routing
When doing the random walk through the channel, we now add the fees
(both the base and the proportional one) for that channel in addition to
the cltv delta.

Changelog-Added: Payment amount fuzzing is restored, but through shadow routing.
2019-11-08 03:27:58 +00:00
Christian Decker 290b4d68b3 changelog: Add a tool to extract changelog entries from the commits
Since the `CHANGELOG.md` file is a major source for merge conflicts I decided
to build a tiny tool that generates the entries for the changelog
automatically from the commit messages.
2019-11-08 00:23:09 +00:00
gorazdko 122fc1f26f config file: fix line count in error message 2019-11-08 00:22:03 +00:00
Rusty Russell a70f2dc650 pytest: two more timeouts on Travis.
Reduce test_feerate_stress iterations, and simply don't run
test_pay_retry under VALGRIND with SLOW_MACHINE at all.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 5a8677edc6 gossipd: add txout_failure when a close is seen.
This prevents a gratuitous lookup of we get a late channel_announce,
but even better, it suppresses the "bad gossip" messages in case of
a late channel_update, which have plagued Travis (especially since we
got aggressive in pushing our own updates).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell abe7133bd5 gossipd: use in_txout_failures to do lookup in channel_announcement.
This correctly refreshes the txout entry against aging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 40d34fed9e pytest: clean up test_gossip_notices_close now that gossipwith has more options.
And drive-by fix: document that you can now (since
e40f07803c) use --max-messages=0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 30634aa837 pytest: fix test_gossip_notices_close where we really do inject bad gossip!
It currently works because we inject it so fast that it's still doing the
txout lookup, but that's about to change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell c96cee9b8d channeld: fix invalid assumption in htlc restore.
A long time ago (93dcd5fed7), I
simplified the htlc reload code so it adjusted the amounts for HTLCs
in id order.  As we presumably allowed them to be added in that order,
this avoided special-casing overflow (which was about to deliberately
be made harder by the new amount_msat code).

Unfortunately, htlc id order is not canonical, since htlc ids are
assigned consecutively in both directions!  Concretely, we can have two HTLCs:

	HTLC #0 LOCAL->REMOTE: 500,000,000 msat, state RCVD_REMOVE_REVOCATION
	HTLC #0 REMOTE->LOCAL: 10,000 msat, state SENT_ADD_COMMIT

On a new remote-funded channel, in which we have 0 balance, these
commits *only* work in this order.  Sorting by HTLC ID is not enough!
In fact, we'd have to worry about redemption order as well, as that
matters.

So, regretfully, we offset the balances halfway to UINT64_MAX, then check
they didn't underflow at the end.  This loses us this one sanity check,
but that's probably OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 22:38:07 +01:00
Rusty Russell 7b6a1c8c87 pytest: add test for bug found by Travis
We fail to restore HTLCs on reconnect sometimes, depending on traverse order:

	2019-10-30T18:39:40.012Z **BROKEN** lightningd(7652): lightning_channeld-0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518 chan #1: Cannot add htlc #0 10000msat to LOCAL
	2019-10-30T18:39:40.024Z **BROKEN** lightningd(7652): lightning_channeld-0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518 chan #1: Could not restore HTLCs (version v0.7.3-12-ga0a271a)

Or, alternatively:

lightning_channeld: Could not restore HTLCs (version v0.7.3-11-gd7838db-modded)
0x564d1c1b53bd send_backtrace
	common/daemon.c:41
0x564d1c1c23c9 status_failed
	common/status.c:199
0x564d1c1a7509 init_channel
	channeld/channeld.c:3073
0x564d1c1a7959 main
	channeld/channeld.c:3165
0x7fdc73be01e2 ???
	???:0
0x564d1c19ee5d ???
	???:0
0xffffffffffffffff ???
	???:0

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 22:38:07 +01:00
Rusty Russell de65369e28 channeld: if we can't restore HTLCs, log at level broken, not debug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 22:38:07 +01:00
Rusty Russell f4d888ec7e lightningd: obscure sensitive bitcoin args when bitcoind unreachable.
It's less helpful, sure, but it's far better than someone
sending me their output and leaking this information.

Fixes: #3242
Reported-by: @JavierRSobrino
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 16:47:52 +01:00
lisa neigut 181e0d4982 test: amend 'behind sync' tests
Check explicitly for both 'fundchannel' case and funding a transaction;
remove unnecessary 'second funding'.
2019-11-04 17:52:48 +01:00
lisa neigut f0f47ce153 warnings: if behind blockchain, don't show cannot afford
If you're replaying or syncing with the blockchain, show that error
instead of 'cannot afford', in the case of not having enough utxos
to pay for a transaction. This is the 'more correct' error to show, as
there's a chance that the funds you're expecting to spend are in the
portion of the blockchain that hasn't been synced yet.
2019-11-04 17:52:48 +01:00
Rusty Russell 7f45e55d84 gossipd: set the push marker for our own messages.
This is a better fix than doing it manually, which turned out
to do it in the wrong order (node_announcement followed by
channel_announcement) anyway.

Should fix many "Bad gossip" messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Rusty Russell bb370e66a8 gossipd: handle a "push" marker into the gossip_store.
This tells clients to ignore any timestamp_filter and always
send this message when it sees it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Rusty Russell 2d8e93687d pytest: prepare test_gossip_timestamp_filter to be spammed.
We're about to change it so we always send our local messages, which
breaks this test.  Add a new node which doesn't have any local
messages, so the test works correctly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Rusty Russell 04403ed59f pytest: fix flaky 'Bad gossip' error in test_block_backfill
Sometimes the l3 seeker asks for scids, and the reply contains the
channel which is then closed by the time it checks, so it considers
the updates bad gossip.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Sebastian Geisler 4c2f51ce13 Fix syntax error in sql-rewrite.py
On python 3.4.2 having kwargs after unpacking
a dict in a function call seems to be a syntax
error.
2019-11-02 16:11:28 +01:00
Rusty Russell fe17acf07b TAGS: reformat to fix when PRINTF_FMT() used.
I was wondering why TAGS was missing some functions, and finally
tracked it down: PRINTF_FMT() confuses etags if it's at the start
of a function, and it ignores the rest of the file.

So we put PRINTF_FMT at the end, but that doesn't work for
*definitions*, only *declarations*.  So we remove it from definitions
and add gratuitous declarations in the few static places.1

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-01 17:27:20 -05:00
lisa neigut 7374134dab doc: add reminder to close out Milestone to Making-Releases 2019-11-01 18:55:54 +01:00