Commit Graph

7187 Commits

Author SHA1 Message Date
Vasil Dimov b2c4d5e952 build: fix developer mode compilation on FreeBSD
Recent FreeBSD versions use LLVM's lld as a linker, not the GNU ld.
Their behavior slightly differs, so adapt the build system to handle
either one.

* The LLVM's linker prints "undefined symbol:" instead of
  "undefined reference to". Tweak tools/mockup.sh to also look for that
  message.

* The LLVM's linker may only print the first dozen errors (omitting
  the rest to avoid flooding the screen). tools/update-mocks.sh relies
  on getting all errors as it extracts the missing symbols' names from
  the error output and creates mocks for them. Detect if errors were
  omitted and re-run, telling the linker to not omit any. The GNU linker
  does not support -error-limit=0, so unfortunately we can't just run
  with that option unconditionally from the first attempt.

* Nit: FreeBSD's sed(1) prints "t" for "\t" instead of a horizontal tab.
  Use a verbatim tab in the command, instead of "\t" which works on
  both.

Changelog-Fixed: Developer mode compilation on FreeBSD.
2020-01-02 16:56:20 +01:00
minidisk1147 f22595596a removed duplicate log-level subsystem 2020-01-02 16:40:24 +01:00
minidisk1147 1715b3d004 updated FreeBSD install instructions 2020-01-02 16:40:24 +01:00
Christian Decker b1e18e16a5 docs: Fix the `htlc_accepted` JSON payload in the docs
Had a spurious `}` in the example.
2020-01-02 16:05:52 +01:00
Christian Decker 483b00392e pytest: Skip test_closing_torture under valgrind
It's really too slow under valgrind, so skip it. Otherwise it'll just fail the
entire run in most cases.
2020-01-02 16:05:52 +01:00
Christian Decker 8cad3ffeac pyln: Work around the socket path length on Linux OSs
Some Linux OSs impose a length limit on the path a Unix socket may have. This
is not an issue in `lightningd` since we `chdir()` into that directory before
opening the socket, however in pyln this became a problem for some tests,
since we use absolute paths in the testing framework. It's also a rather
strange quirk to expose to users.

This patch introduces a `UnixSocket` abstraction that attempts to work around
these limitations by aliasing the directory containing the socket into
`/proc/self/fd` and then connecting using that alias.

It was inspired by Open vSwitch code here https://github.com/openvswitch/ovs/blob/master/python/ovs/socket_util.py

Signed-off-by: Christian Decker <@cdecker>
2020-01-02 16:05:52 +01:00
Christian Decker de619b1754 pyln: Check to see if we have annotations on funcs before accessing
We were indiscriminately accessing the `__annotations__` which could cause
issues if the function had been wrapped by some functions such as
`functools.partial`. This just checks that the access is safe before doing it.

Suggested-by: jarret <@jarret>
Signed-off-by: Christian Decker <@cdecker>
2020-01-02 16:05:52 +01:00
Christian Decker 8c387932c0 pyln: Bump versions of pyln-client and pyln-testing 2019-12-31 16:43:04 +01:00
darosior 841fbf54ea plugin_control: spawn plugin processes with a non-0 umask
Changelog-Added: JSONRPC: 'plugin start' now restores initial umask before spawning the plugin process
2019-12-29 19:37:06 +01:00
darosior 1cfb8425f5 plugin_control: more descriptive key for 'plugin stop' result
Changelog-Changed: JSONRPC: 'plugin stop' result is now accessible using the 'result' key instead of the empty ('') key.
2019-12-29 06:44:22 +08:00
darosior 77eb9e45b2 connectd: return error when no common chain at init
This is useful for testing.
2019-12-28 14:22:05 +01:00
darosior 1e15e13de0 connectd: clean up a redundant check 2019-12-28 14:22:05 +01:00
darosior 9dcbf20a7a connectd: check all chain hashs of the TLV array 2019-12-28 14:22:05 +01:00
darosior 4f3e8d461e cli: make '--rpc-file' take over other config options if absolute
If absolute, we can deduce the lightning_dir, the network, and the
socket filename out of it.
This allows to be able to only specify the 'rpc-file', as before.

Changelog-Changed: Usage of `lightning-cli` by specifying only `--rpc-file` has been restored.
2019-12-26 12:49:23 +08:00
lisa neigut 689dd28ddd funding: enable push_msat
it's that time of year (merry xmas!)

enables the ability to push_msat on fundchannel

Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
2019-12-24 12:04:01 -06:00
darosior 051b5bd1bf Update bitcoin-core version
Bitcoin-core v0.18.X is now the more widely used.
2019-12-22 12:42:56 +01:00
Michael Schmoock c77a085ca4 fix: update pip dependencies version in dockerfile 2019-12-19 15:17:25 +01:00
Michael Schmoock dc5a1ad664 fix: normalize pip requirements
This will change some `requirements.txt` of pyln-testing in a way that
it does not require different package version i.e. to `tests/requirements.txt`.

The reason for this is that users are not forced to hassle with pyenv
or virtualenv and could just use `--user`.

```bash
pip install --user -r tests/requirements.txt -r contrib/pyln-testing/requirements.txt
```

Changelog-None
2019-12-19 15:17:25 +01:00
Michael Schmoock 43b5abeb22 fix: pip dependency version for psycopg2-binary
Trying to `pip install psycopg2-binary==2.8.3` raised an error for a
long time. Since version `2.8.4` is recent and also seem to work I
suggest updating the requirements so other users dont run into the
following error, where pip tries to use `pg_config` for the BINARY
package:

Collecting psycopg2-binary==2.8.3 (from -r contrib/pyln-testing/requirements.txt (line 6))
  Using cached 91911be018dd876c1e0f12e1b5fb90/psycopg2-binary-2.8.3.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-zapr0fhs/psycopg2-binary/
    Complete output (23 lines):
    running egg_info
    creating pip-egg-info/psycopg2_binary.egg-info
    writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Changelog-None
2019-12-19 13:26:13 +01:00
Christian Decker bb2733a226 travis: Make python environment selection less picky
Travis seems to have upgraded to a newer minor version of python3, so let's be
way less picky about which exact version we want to run against.
2019-12-19 10:35:45 +08:00
Vasil Dimov b14b2b0490 doc: fix typos and minor rewording in doc/PLUGINS.md
Changelog-None
2019-12-17 23:21:51 +08:00
Christian Decker c15169b463 release: Updating changelog to v0.8.0 final 2019-12-16 21:25:35 +01:00
Vasil Dimov bf1f768a82 doc: fix the path to the sqlite database file
The sqlite database location has been changed in aab83e72.

Changelog-None
2019-12-16 15:31:42 +01:00
Vasil Dimov 1fe8cd8149 doc: fix typos in doc/TOR.md 2019-12-16 13:02:27 +01:00
Vasil Dimov 250fc79e93 doc: remove trailing whitespace from doc/TOR.md 2019-12-16 13:02:27 +01:00
Christian Decker eb2b225c9f changelog: Update the changelog for 0.8.0rc2 2019-12-13 22:46:30 +01:00
Rusty Russell b315a29202 wire: perform corruption test on init_msg.
Fixes: #3301
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:40 +01:00
Rusty Russell eb6a768741 tools: don't use bytelength as array length for arrays in TLVs.
This matters now we have an array in tlv_init_tlvs!  We were overallocating
in fromwire by 32x!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:40 +01:00
Rusty Russell 1d0c433dc4 channeld: treat all incoming errors as "soft", so we retry.
We still close the channel if we *send* an error, but we seem to have hit
another case where LND sends an error which seems transient, so this will
make a best-effort attempt to preserve our channel in that case.

Some test have to be modified, since they don't terminate as they did
previously :(

Changelog-Changed: quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-13 16:36:18 +01:00
Rusty Russell 839909d2cf Protocol: make var_onion, payment_secret and basic_mpp non-EXPERIMENTAL.
Thanks to @t-bast, who made this possible by interop testing with Eclair!

Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:16:03 +01:00
Rusty Russell 2e97afd1ac lightningd: remove htlc_set destruction timer once we head to invoice.
Otherwise tests for hold_invoice fail on Travis (they use 180 / 2 as
the timeout, and we free it after 70 seconds).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:16:03 +01:00
Christian Decker fb7c67889e htlcs: Make sure htlc_sets is not erroneously flagged as a memleak 2019-12-12 22:16:03 +01:00
Rusty Russell e521aaec8e devtools: fix example at top of mkcommit.
The norm for channels is a 1% reserve.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell 72aa315b5e lightningd: save the fee_states into the database.
This is the final step: we pass the complete fee_states to and from
channeld.

Changelog-Fixed: "Bad commitment signature" closing channels when we sent back-to-back update_fee messages across multiple reconnects.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell 4270031d75 wallet: add per-channel feerate_state to the database.
The upgrade here is a bit tricky: we map the two values into the
feerate_state.  This is trivial if they're both the same, but if
they're different we don't know exactly what state they're in (this
being the source of the bug!).

So, we assume that the have received the update and not acked it,
as that would be the normal case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell d2673a4e6f channeld: remove changes_pending flags.
These used to be necessary as we could have feerate changes which
we couldn't track: now we do, we don't need these flags.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell 24d54f98ad channeld: use fee_states internally.
This is an intermediary step: we still don't save it to the database,
but we do use the fee_states struct to track it internally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell acb89debdd channeld: generate revoke_and_ack before telling lightningd about commitment.
The `channel_got_commitsig` we send the lightningd also implies we sent
the revoke_and_ack, as an optimization.  It doesn't currently matter,
since channel_sending_revoke_and_ack doesn't do anything important to the
state, but that changes once we start uploading the entire fee_states.

So now we move our state machine *before* sending to lightningd, in
preparation for sending fee_states too.

Unfortunately, we need to marshall the info to send before we
increment the state, as lightningd expects that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell 8ffd9d570b channeld: cleanup: use the channel_feerate() accessor everywhere.
And also move it to initial_channel, so we can use it there.

This saves churn in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell b9d6e8bbaa common: add fee_states state machine for update_fee.
This uses the same state machine as HTLCs, but they're only
ever added, not removed.  Since we can only have one in each
state, we use a simple array; mostly NULL.

We could make this more space-efficient by folding everything into the
first 5 states, but that would be more complex than just using the
identical state machine.

One subtlety: we don't send uncommitted fee_states over the wire.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 22:15:48 +01:00
Rusty Russell ec1505199e lightningd: fix failure message in waitsendpay with multi-part payments.
Bastien TEINTURIER <bastien@acinq.fr> writes:
> One thing I noticed but didn't investigate much: after sending the two
> payments, I tried using `waitsendpay` and it reported an error *208*
> (*"Never attempted payment for
> '98ee736d29d860948e436546a88b0cc84f267de8818531b0fdbe6ce3d080f22a'"*).
> 
> I was expecting the result to be something like: "payment succeeded for
> that payment hash" (the HTLCs were correctly settled).

Indeed, if you waitsendpay without specifying a partid, you are waiting
for 0, which may not exist.  Clarify the error msg.

Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell edab0df611 lightningd: fix spurious "more than twice final" error.
Bastien TEINTURIER <bastien@acinq.fr> writes:
> It looks like the split on c-lightning side is quite limited at the moment:
> the only option is to split a payment in exactly its two halves,
> otherwise I get rejected because of the rule of overpaying more than
> twice the amount?

We only tested exactly two equal-size payments; indeed, our finalhop
test was backwards.  We only complain if the final hop pays more than
twice msat (technically, this test is still too loose for mpp: the
spec says we should sum to the exact amount).

Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 7fb4efd98a lightningd: require payment_secret for MPP.
It makes sense, and it's been proposed for addition to the spec to
broad agreement:

	https://github.com/lightningnetwork/lightning-rfc/pull/712

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 326ceab8ad pytest: add more multi-part-payment tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 5d413e0f79 common: offer option_basic_mpp for EXPERIMENTAL_FEATURES.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 207689c274 plugins: listpays will now consolidate multi-part payments.
This won't usually be visible to the end-user, since the pay plugin doesn't
do multi-part yet (and mpp requires EXPERIMENTAL_FEATURES), but we're ready
once it does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell f1bc0b21f1 plugins: listpays ignores pre-0.7.0 or manual sendpay payments w/ no bolt11.
The pay plugin has been supplying the bolt11 string since 0.7.0, so only
ancient "pay" commands would be omitted by this change.

You can create a no-bolt11 "sendpay" manually, but then you'll find it
in 'listsendpays'.

Changelog-Removed: JSON: `listpays` won't shown payments made via sendpay without a bolt11 string, or before 0.7.0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 8e3234e67a lightningd: sew in htlc set.
The invoice_try_pay code now takes a set, rather than a single htlc, but
it's basically the same thing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell c2b357b916 lightningd: wrap htlc replay in a database transaction.
AFAICT this only "worked" previously because replay htlc simply failed
them all (no peers are currently connected).  With upcoming changes
(foreshadowed by the comment) this is no longer true:

Attempting to prepare a db_stmt outside of a transaction: wallet/invoices.c:373
lightningd: FATAL SIGNAL 6 (version v0.7.3-188-g45b0af4-modded)
0x55b475590a73 send_backtrace
	common/daemon.c:41
0x55b475590b1d crashdump
	common/daemon.c:54
0x7f16c557b46f ???
	???:0
0x7f16c557b3eb ???
	???:0
0x7f16c555a898 ???
	???:0
0x55b475564c8f fatal
	lightningd/log.c:814
0x55b4755c3ed5 db_prepare_v2_
	wallet/db.c:605
0x55b4755c76b5 invoices_find_unpaid
	wallet/invoices.c:373
0x55b4755ce91c wallet_invoice_find_unpaid
	wallet/wallet.c:1990
0x55b47555861f invoice_check_payment
	lightningd/invoice.c:257
0x55b475557a7c htlc_add_set
	lightningd/htlc_set.c:112
0x55b47557b294 handle_localpay
	lightningd/peer_htlcs.c:332
0x55b47557c63c htlc_accepted_hook_callback
	lightningd/peer_htlcs.c:857
0x55b475585573 plugin_hook_call_
	lightningd/plugin_hook.c:118
0x55b47557c747 plugin_hook_call_htlc_accepted
	lightningd/peer_htlcs.c:882
0x55b47557ca3e peer_accepted_htlc
	lightningd/peer_htlcs.c:991
0x55b47557ffb9 htlcs_resubmit
	lightningd/peer_htlcs.c:2131
0x55b4755620f7 main
	lightningd/lightningd.c:801

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00
Rusty Russell 36b1e884dd lightningd: implement htlc sets.
This isn't plumbed in yet, but the idea is that every htlc gets put
into a "set" and then we process them once the set is satisfied.  For
the !EXPERIMENTAL_FEATURES, the set is simply always size 1.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-12-12 15:03:53 +01:00