Commit Graph

7396 Commits

Author SHA1 Message Date
darosior 16f5af00c7 common/json: add a helper for json to u16
As Rusty pointed out to me, the gossip protocol restricts cltvs to u16
so at least we'll use this helper for them.
2019-11-08 03:27:58 +00:00
darosior 5766231d98 pay: restore payment value randomization through shadow routing
When doing the random walk through the channel, we now add the fees
(both the base and the proportional one) for that channel in addition to
the cltv delta.

Changelog-Added: Payment amount fuzzing is restored, but through shadow routing.
2019-11-08 03:27:58 +00:00
Christian Decker 290b4d68b3 changelog: Add a tool to extract changelog entries from the commits
Since the `CHANGELOG.md` file is a major source for merge conflicts I decided
to build a tiny tool that generates the entries for the changelog
automatically from the commit messages.
2019-11-08 00:23:09 +00:00
gorazdko 122fc1f26f config file: fix line count in error message 2019-11-08 00:22:03 +00:00
Rusty Russell a70f2dc650 pytest: two more timeouts on Travis.
Reduce test_feerate_stress iterations, and simply don't run
test_pay_retry under VALGRIND with SLOW_MACHINE at all.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 5a8677edc6 gossipd: add txout_failure when a close is seen.
This prevents a gratuitous lookup of we get a late channel_announce,
but even better, it suppresses the "bad gossip" messages in case of
a late channel_update, which have plagued Travis (especially since we
got aggressive in pushing our own updates).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell abe7133bd5 gossipd: use in_txout_failures to do lookup in channel_announcement.
This correctly refreshes the txout entry against aging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 40d34fed9e pytest: clean up test_gossip_notices_close now that gossipwith has more options.
And drive-by fix: document that you can now (since
e40f07803c) use --max-messages=0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell 30634aa837 pytest: fix test_gossip_notices_close where we really do inject bad gossip!
It currently works because we inject it so fast that it's still doing the
txout lookup, but that's about to change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-07 03:50:53 +00:00
Rusty Russell c96cee9b8d channeld: fix invalid assumption in htlc restore.
A long time ago (93dcd5fed7), I
simplified the htlc reload code so it adjusted the amounts for HTLCs
in id order.  As we presumably allowed them to be added in that order,
this avoided special-casing overflow (which was about to deliberately
be made harder by the new amount_msat code).

Unfortunately, htlc id order is not canonical, since htlc ids are
assigned consecutively in both directions!  Concretely, we can have two HTLCs:

	HTLC #0 LOCAL->REMOTE: 500,000,000 msat, state RCVD_REMOVE_REVOCATION
	HTLC #0 REMOTE->LOCAL: 10,000 msat, state SENT_ADD_COMMIT

On a new remote-funded channel, in which we have 0 balance, these
commits *only* work in this order.  Sorting by HTLC ID is not enough!
In fact, we'd have to worry about redemption order as well, as that
matters.

So, regretfully, we offset the balances halfway to UINT64_MAX, then check
they didn't underflow at the end.  This loses us this one sanity check,
but that's probably OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 22:38:07 +01:00
Rusty Russell 7b6a1c8c87 pytest: add test for bug found by Travis
We fail to restore HTLCs on reconnect sometimes, depending on traverse order:

	2019-10-30T18:39:40.012Z **BROKEN** lightningd(7652): lightning_channeld-0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518 chan #1: Cannot add htlc #0 10000msat to LOCAL
	2019-10-30T18:39:40.024Z **BROKEN** lightningd(7652): lightning_channeld-0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518 chan #1: Could not restore HTLCs (version v0.7.3-12-ga0a271a)

Or, alternatively:

lightning_channeld: Could not restore HTLCs (version v0.7.3-11-gd7838db-modded)
0x564d1c1b53bd send_backtrace
	common/daemon.c:41
0x564d1c1c23c9 status_failed
	common/status.c:199
0x564d1c1a7509 init_channel
	channeld/channeld.c:3073
0x564d1c1a7959 main
	channeld/channeld.c:3165
0x7fdc73be01e2 ???
	???:0
0x564d1c19ee5d ???
	???:0
0xffffffffffffffff ???
	???:0

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 22:38:07 +01:00
Rusty Russell de65369e28 channeld: if we can't restore HTLCs, log at level broken, not debug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 22:38:07 +01:00
Rusty Russell f4d888ec7e lightningd: obscure sensitive bitcoin args when bitcoind unreachable.
It's less helpful, sure, but it's far better than someone
sending me their output and leaking this information.

Fixes: #3242
Reported-by: @JavierRSobrino
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-05 16:47:52 +01:00
lisa neigut 181e0d4982 test: amend 'behind sync' tests
Check explicitly for both 'fundchannel' case and funding a transaction;
remove unnecessary 'second funding'.
2019-11-04 17:52:48 +01:00
lisa neigut f0f47ce153 warnings: if behind blockchain, don't show cannot afford
If you're replaying or syncing with the blockchain, show that error
instead of 'cannot afford', in the case of not having enough utxos
to pay for a transaction. This is the 'more correct' error to show, as
there's a chance that the funds you're expecting to spend are in the
portion of the blockchain that hasn't been synced yet.
2019-11-04 17:52:48 +01:00
Rusty Russell 7f45e55d84 gossipd: set the push marker for our own messages.
This is a better fix than doing it manually, which turned out
to do it in the wrong order (node_announcement followed by
channel_announcement) anyway.

Should fix many "Bad gossip" messages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Rusty Russell bb370e66a8 gossipd: handle a "push" marker into the gossip_store.
This tells clients to ignore any timestamp_filter and always
send this message when it sees it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Rusty Russell 2d8e93687d pytest: prepare test_gossip_timestamp_filter to be spammed.
We're about to change it so we always send our local messages, which
breaks this test.  Add a new node which doesn't have any local
messages, so the test works correctly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Rusty Russell 04403ed59f pytest: fix flaky 'Bad gossip' error in test_block_backfill
Sometimes the l3 seeker asks for scids, and the reply contains the
channel which is then closed by the time it checks, so it considers
the updates bad gossip.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-04 17:50:58 +01:00
Sebastian Geisler 4c2f51ce13 Fix syntax error in sql-rewrite.py
On python 3.4.2 having kwargs after unpacking
a dict in a function call seems to be a syntax
error.
2019-11-02 16:11:28 +01:00
Rusty Russell fe17acf07b TAGS: reformat to fix when PRINTF_FMT() used.
I was wondering why TAGS was missing some functions, and finally
tracked it down: PRINTF_FMT() confuses etags if it's at the start
of a function, and it ignores the rest of the file.

So we put PRINTF_FMT at the end, but that doesn't work for
*definitions*, only *declarations*.  So we remove it from definitions
and add gratuitous declarations in the few static places.1

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-11-01 17:27:20 -05:00
lisa neigut 7374134dab doc: add reminder to close out Milestone to Making-Releases 2019-11-01 18:55:54 +01:00
lisa neigut 16fde5c79d releases: break up first git tag for verification purposes
I'm not sure what went wrong, but the first RC I cut had some trouble
with the tag being picked up with `git describe`, I think it was missing
a 'tag message'. I'm not sure what caused this.

This commit breaks up the first git tag procedure to have the releaser
verify that the tag command works as intended (and sensitizes them to
checking this for subsequent release cuts, if necessary)
2019-11-01 18:55:54 +01:00
lisa neigut c581fd9956 releases: update with niftynei's notes from release 0.7.3 2019-11-01 18:55:54 +01:00
arowser 82a2c6b02d change psycopg2 to psycopg2-binary 2019-11-01 18:54:57 +01:00
Jorge Timón 61383408a4 pay: Return error when trying to pay to an invoice from unkown or different chain 2019-10-30 13:06:24 -05:00
Rusty Russell 6fa965c6b5 CHANGELOG.md: reset to Unreleased.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-30 12:59:16 -05:00
gorazdko a3851f2943 wallet: always create signatures with low r-value 2019-10-29 18:51:09 -05:00
arowser 67ce27aac7 remove repeat line in doc/requirements.txt 2019-10-29 12:18:45 -05:00
arowser cd894468e7 remove repeat install dependencies 2019-10-29 12:18:45 -05:00
arowser d1060e1b80 .gitignore: Remove devtools/create-gossipstore 2019-10-29 12:18:27 -05:00
arowser 90667a24b6 devtools/.gitignore: Ignore create-gossipstore checkchannels mkquery lightning-checkmessage 2019-10-29 12:18:27 -05:00
gorazdko 35ee800b6e json-rpc: show lightning-dir in getinfo 2019-10-29 12:18:06 -05:00
darosior f690c35883 cleanup lightning-pay
No json.dumps, make bolt11 a required argument (better usage output instead of assertion error)
2019-10-29 12:15:29 -05:00
darosior 8a9650c887 pylightning: reorder imports 2019-10-29 12:15:29 -05:00
lisa neigut 5bc2de8997 update version to 0.7.3 2019-10-28 15:23:37 -05:00
lisa neigut 0cee553ed7 remove reverted feature desc 2019-10-28 15:23:37 -05:00
Rusty Russell 21d2cc663b lightningd: apply feerate changes correctly.
Feerate changes are asymmetric, as they can only be sent by the funder.

For FUNDER, the remote feerate is set when upon send of
commitment_signed, and the local feerate is set on receipt of
revoke_and_ack.

For non-funder, the local feerate is set on receipt of
commitment_signed, and the remote feerate set on send of
revoke_and_ack.  In our code, these two happen together.

channeld gets this right, but lightningd ignored the funder/fundee
distinction, and as a result, receipt of a commitment_signed by the
funder altered fees in the database.  If there was a reconnection
event or restart, then these (incorrect) values would be used, causing
us to complain about a 'Bad commit_sig signature' and close the
channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-28 13:07:41 -05:00
Rusty Russell 61e1d6431c pytest: stress fee_update code, trigger bug.
A 'Bad commit_sig signature' was reported by @Javier on Telegram and
@DarthCoin.  This was between two c-lightning peers, so definitely our fault.

Analysis of this message revealed the signature was using the wrong
feerate.  I finally managed to make a test case which triggered this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-28 13:07:41 -05:00
Rusty Russell f6571460ff configure: fix FreeBSD which has sqlite3 in /usr/local
Fixes: #3080
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-27 21:42:54 -05:00
arowser 0985c6e219 Fix build fail on 32bit OS. 2019-10-23 07:23:33 +11:00
Rusty Russell 4d0c2e93bf common: remove spammy debug msg.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-22 07:05:47 -07:00
lisa neigut d5706b80c0 rc3 2019-10-21 15:29:43 +02:00
Rusty Russell 35bbba68a5 Revert "gossipd: query_messages: fail the connection if peer says it does not have up-to-date infos"
This reverts commit 7fd2f6db6d.

We want to set 'complete' to false in future if we don't know
everything!
2019-10-21 15:28:42 +02:00
Rusty Russell 2801d98e34 pytest: allow bad gossip in test_pay_direct.
Whenever we have multi-connected nodes, out-of-order gossip is possible.
In particular, if a node_announcement is 1 second fresher than the
channel_announcement, a timestamp_filter might get one and not the
other.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-21 14:08:05 +02:00
Rusty Russell bc430cced3 gossipd: fix false-positive memleak detection in pending_node_map.
lightning_gossipd(17421): MEMLEAK: 0x564b4b17b5a8
    ligtning_gossipd(17421):   label=gossipd/routing.c:1490:struct pending_node_announce
    lightning_gossipd(17421):   backtrace:
    lightning_gossipd(17421):     ccan/ccan/tal/tal.c:437 (tal_alloc_)
    lightning_gossipd(17421):     gossipd/routing.c:1490 (catch_node_announcement)
    lightning_gossipd(17421):     gossipd/routing.c:1837 (handle_channel_announcement)
    lightning_gossipd(17421):     gossipd/gossipd.c:238 (handle_channel_announcement_msg)
    lightning_gossipd(17421):     gossipd/gossipd.c:461 (peer_msg_in)
    lightning_gossipd(17421):     common/daemon_conn.c:31 (handle_read)
    lightning_gossipd(17421):     ccan/ccan/io/io.c:59 (next_plan)
    lightning_gossipd(17421):     ccan/ccan/io/io.c:407 (do_plan)
    lightning_gossipd(17421):     ccan/ccan/io/io.c:417 (io_ready)
    lightning_gossipd(17421):     ccan/ccan/io/poll.c:445 (io_loop)
    lightning_gossipd(17421):     gossipd/gossipd.c:1700 (main)
    lightning_gossipd(17421):   parents:
    lightning_gossipd(17421):     gossipd/routing.c:294:struct routing_state

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-10-21 14:08:05 +02:00
Kristaps Kaupe e16ac08f6f Change order of paragraphs in manpage of fundchannel 2019-10-21 14:01:49 +02:00
Christian Decker 21c8eaf105 pytest: Check for null access warnings in tests 2019-10-21 13:56:10 +02:00
Christian Decker 396e8224fa db: Add a migration to set received_time on forwards in rare cases
The case where this is needed is when the wallet had a forwarded payment
somewhere between commits 66a47d2 (which started tracking forwardings) and
d901304 (which added the `received_time` column). This just emulates the
behavior of sqlite3 for postgres as well.

Signed-off-by: Christian Decker <@cdecker>
2019-10-21 13:56:10 +02:00
Christian Decker 1ecad0cc53 db: Maybe a bit too pedantic?
Checking on whether we access a null field is ok, but should we crash right
away? Probably not. This reduces the access to a warning on sqlite3 and let's
it continue. We can look for occurences and fix them as they come up and then
re-arm the asserts once we addressed all cases.
2019-10-21 13:56:10 +02:00