This lets us do more flexible filtering in the next patch. But it also
keeps some weird logic out of gossipd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The optimistic lock prevents multiple instances of c-lightning making
concurrent modifications to the database. That would be unsafe as it messes up
the state in the DB. The optimistic lock is implemented by checking whether a
gated update on the previous value of the `data_version` actually results in
an update. If that's not the case the DB has been changed under our feet.
The lock provides linearizability of DB modifications: if a database is
changed under the feet of a running process that process will `abort()`, which
from a global point of view is as if it had crashed right after the last
successful commit. Any process that also changed the DB must've started
between the last successful commit and the unsuccessful one since otherwise
its counters would not have matched (which would also have aborted that
transaction). So this reduces all the possible timelines to an equivalent
where the first process died, and the second process recovered from the DB.
This is not that interesting for `sqlite3` where we are also protected via the
PID file, but when running on multiple hosts against the same DB, e.g., with
`postgres`, this protection becomes important.
Changelog-Added: DB: Optimistic logging prevents instances from running concurrently against the same database, providing linear consistency to changes.
Some Linux OSs impose a length limit on the path a Unix socket may have. This
is not an issue in `lightningd` since we `chdir()` into that directory before
opening the socket, however in pyln this became a problem for some tests,
since we use absolute paths in the testing framework. It's also a rather
strange quirk to expose to users.
This patch introduces a `UnixSocket` abstraction that attempts to work around
these limitations by aliasing the directory containing the socket into
`/proc/self/fd` and then connecting using that alias.
It was inspired by Open vSwitch code here https://github.com/openvswitch/ovs/blob/master/python/ovs/socket_util.py
Signed-off-by: Christian Decker <@cdecker>
it's that time of year (merry xmas!)
enables the ability to push_msat on fundchannel
Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
Trying to `pip install psycopg2-binary==2.8.3` raised an error for a
long time. Since version `2.8.4` is recent and also seem to work I
suggest updating the requirements so other users dont run into the
following error, where pip tries to use `pg_config` for the BINARY
package:
Collecting psycopg2-binary==2.8.3 (from -r contrib/pyln-testing/requirements.txt (line 6))
Using cached 91911be018dd876c1e0f12e1b5fb90/psycopg2-binary-2.8.3.tar.gz
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-zapr0fhs/psycopg2-binary/
Complete output (23 lines):
running egg_info
creating pip-egg-info/psycopg2_binary.egg-info
writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Changelog-None
We still close the channel if we *send* an error, but we seem to have hit
another case where LND sends an error which seems transient, so this will
make a best-effort attempt to preserve our channel in that case.
Some test have to be modified, since they don't terminate as they did
previously :(
Changelog-Changed: quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Thanks to @t-bast, who made this possible by interop testing with Eclair!
Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an intermediary step: we still don't save it to the database,
but we do use the fee_states struct to track it internally.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Bastien TEINTURIER <bastien@acinq.fr> writes:
> It looks like the split on c-lightning side is quite limited at the moment:
> the only option is to split a payment in exactly its two halves,
> otherwise I get rejected because of the rule of overpaying more than
> twice the amount?
We only tested exactly two equal-size payments; indeed, our finalhop
test was backwards. We only complain if the final hop pays more than
twice msat (technically, this test is still too loose for mpp: the
spec says we should sum to the exact amount).
Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This won't usually be visible to the end-user, since the pay plugin doesn't
do multi-part yet (and mpp requires EXPERIMENTAL_FEATURES), but we're ready
once it does.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The invoice_try_pay code now takes a set, rather than a single htlc, but
it's basically the same thing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Explicit #if EXPERIMENTAL_FEATURES check in case we enable them at different
times, but it requires a payment_secret since we put them in the same field.
This incidently stops it working on legacy nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
msatoshi was used to indicate the amount the invoice asked for, but
for parallel sendpay it's required, as it allows our sanity check of
limiting the total payments in flight, ie. it becomes
'total_msat'.
There's a special case for sendonion, which always tells us the value is 0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently refuse a payment if one is already in flight. For parallel
payments, it's a bit more subtle: we want to refuse if it we already have
the total-amount-of-invoice in flight.
So we get all the current payments, and sum the pending ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were hardcoding the chainparams->chain_hash which caused the query to
return an empty result. By parametrizing the test we can make it work on
elements.
Now "raw_payload" is always the complete string (including realm or length
bytes at the front).
This has several effects:
1. We can receive an decrypt an onion which is grossly malformed.
2. We can still hand this to the htlc_accepted hook.
3. We then fail it unless the htlc_accepted accepts it manually.
4. The createonion API now takes the raw payload, and does not know
anything about "style".
The only caveat is that the sphinx code needs to know the payload
length: we have a call for that, which simply tells it to copy the
entire onion (and treat us as the final node) if it's invalid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular:
1. It must redirect to an existing command.
2. It must contain method, params and id.
And update the docs to show the id, which is vital.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
cheroot release(/"changes"?) notes:
#218 via PR #219: Fix HTTP parser to return 400 on invalid major-only HTTP version in Request-Line.
#198 via 9f7affe: Fix race condition when toggling stats counting in the middle of request processing.
Improve post Python 3.9 compatibility checks.
Fix support of abstract namespace sockets.
#222 via 621f4ee: Fix socket.SO_PEERCRED constant fallback value under PowerPC
Revisit PR #85 under PR #221. Now backports.functools_lru_cache is only required on Python 3.2 and earlier.
CherryPy #1206 via PR #204: Fix race condition in threadpool shrink code.
PR #224: Refactored “open URL” behavior in webtest to rely on retry_call. Callers can no longer pass raise_subcls or ssl_context positionally, but must pass them as keyword arguments.
#231 via PR #232: Remove custom setup.cfg parser handling, allowing the project (including sdist) to build/run on setuptools 41.4. Now building cheroot requires setuptools 30.3 or later (for declarative config support) and preferably 34.4 or later (as indicated in pyproject.toml).
Workers are now request-based, addressing the long-standing issue with keep-alive connections (#91 via PR #199).
Deprecated use of negative timeouts as alias for infinite timeouts in ThreadPool.stop.
CherryPy #1662 via PR #74: For OPTION requests, bypass URI as path if it does not appear absolute.
CherryPy #1818: Restore support for None default argument to WebCase.getPage().
https://github.com/cherrypy/cheroot/blob/master/CHANGES.rst
flaky changes:
Bugfixes - Reraise KeyboardInterrupt when running tests under pytest.
https://github.com/box/flaky/blob/v3.6.1/HISTORY.rst#361-2019-08-06
python-bitcoinlib:
New RPC `generatetoaddress(self,numblocks,addr)`.
Fixed Python 2.7 incompatibility.
Various OpenSSL fixes, including a memory leak.
https://github.com/petertodd/python-bitcoinlib/blob/python-bitcoinlib-v0.10.2/release-notes.md#v0102
pytest release notes:
A lot of misc fixes, see https://docs.pytest.org/en/latest/changelog.html.
If we can't decode the onion, because the onion got corrupted or we used
`sendonion` without specifying the `shared_secrets` used, the best we can do
is tell the caller instead.
This means that c-lightning can now internally decrypt an eventual error
message, and not force the caller to implement the decryption. The main
difficulty was that we now have a new state (channels and nodes not specified,
while shared_secrets are specified) which needed to be handled.
We are breaking with a couple of assumptions, namely that we have the
`path_secrets` to decode the error onion. If this happens we just want it to
error out.
addresses issue #2753.
Formatting the JSON with the default parameters will escape the unicode
symbols in a way that c-lightning won't allow, leading to an exception.
Changelog-Fixed: `pylightning` now handles unicode characters in JSON-RPC requests and responses correctly.
We were using sleeps to hope we catch the password prompt. This makes the test
flaky. So I added a help text followed by a `fflush` to make sure we catcht he
right moment, instead of guessing. The `fflush` is also useful for debugging
if a user ever pipes the output to a file it'd get buffered and the user would
wait forever. The same applies for automated systems such as `expect` or
`pexpect` based scripts that enter the password on prompt.
This will change the command `listconfigs` output in several ways:
- Deprecated the duplicated "plugin" JSON output by replacing it with
- a "plugins" array with substructures for each plugin with:
- path, name and their options
Changelog-Changed: JSON-RPC: `listconfigs` now structures plugins and include their options
Changelog-Deprecated: JSON-RPC: `listconfigs` duplicated "plugin" paths
The spec is (RSN!) going to explicitly denote where each feature should
be presented, so create that infrastructure.
Incorporate the new proposed bolt11 features, which need this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't set the secret to compulsory (yet!) but put code in for the
future. Meanwhile, if there is a secret, check it is correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Do the same thing '--help' does with them; append `...`.
Valgrind noticed that we weren't NUL-terminarting if answer was over
78 characters.
Changelog-Fixed: JSONRPC: listconfigs appends '...' to truncated config options.
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets you have a default, but also a network-specific config.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: `config` and <network>/`config` read by default.
Postgresql does not guarantee insertion order when querying, so we need to
actually go and look for the correct transaction.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Rounds out the application of `upfront_shutdown_script`, allowing
an accepting node to specify a close_to address.
Prior to this, only the opening node could specify one.
Changelog-Added: Plugins: Allow the 'accepter' to specify an upfront_shutdown_script for a channel via a `close_to` field in the openchannel hook result
This leads to all sorts of problems; in particular it's incredibly
slow (days, weeks!) if bitcoind is a long way back. This also changes
the behaviour of a rescan argument referring to a future block: we will
also refuse to start in that case, which I think is the correct behavior.
We already ignore bitcoind if it goes backwards while we're running.
Also cover a false positive memleak.
Changelog-Fixed: If bitcoind goes backwards (e.g. reindex) refuse to start (unless forced with --rescan).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So far we've only handled legacy payloads, which meant we could drop the realm
byte since it was always 0x00. Once we start handling TLV payloads the first
byte, i.e., the former realm byte, is important since it gives us the length
of the payload. This is a breaking change, however I don't think there's
anyone using the `raw_payload` as of yet.
Changelog-Changed: JSON-RPC: the `raw_payload` now includes the first byte, i.e., the realm byte, of the payload as well. This allows correct decoding of a TLV payload in the plugins.
Spaces just make life a little harder for everyone.
(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can definitely get a pong from l1 (should test be slow enough):
it's l3 we are concerned about.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.
Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
A log can have a default node_id, which can be overridden on a per-entry
basis. This changes the format of logging, so some tests need rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: JSON API: `htlc_accepted` hook has `type` (currently `legacy` or `tlv`) and other fields directly inside `onion`.
Changelog-deprecated: JSON API: `htlc_accepted` hook `per_hop_v0` object deprecated, as is `short_channel_id` for the final hop.
If a 'upfront_shutdown_script' was specified, show the address +
scriptpubky in `listpeers`
Changelog-added: JSON API: `listpeers` channels now include `close_to` and `close_to_addr` iff a `close_to` address was specified at channel open
Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.
Reduce test_feerate_stress iterations, and simply don't run
test_pay_retry under VALGRIND with SLOW_MACHINE at all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It currently works because we inject it so fast that it's still doing the
txout lookup, but that's about to change.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A long time ago (93dcd5fed7), I
simplified the htlc reload code so it adjusted the amounts for HTLCs
in id order. As we presumably allowed them to be added in that order,
this avoided special-casing overflow (which was about to deliberately
be made harder by the new amount_msat code).
Unfortunately, htlc id order is not canonical, since htlc ids are
assigned consecutively in both directions! Concretely, we can have two HTLCs:
HTLC #0 LOCAL->REMOTE: 500,000,000 msat, state RCVD_REMOVE_REVOCATION
HTLC #0 REMOTE->LOCAL: 10,000 msat, state SENT_ADD_COMMIT
On a new remote-funded channel, in which we have 0 balance, these
commits *only* work in this order. Sorting by HTLC ID is not enough!
In fact, we'd have to worry about redemption order as well, as that
matters.
So, regretfully, we offset the balances halfway to UINT64_MAX, then check
they didn't underflow at the end. This loses us this one sanity check,
but that's probably OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to change it so we always send our local messages, which
breaks this test. Add a new node which doesn't have any local
messages, so the test works correctly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes the l3 seeker asks for scids, and the reply contains the
channel which is then closed by the time it checks, so it considers
the updates bad gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Feerate changes are asymmetric, as they can only be sent by the funder.
For FUNDER, the remote feerate is set when upon send of
commitment_signed, and the local feerate is set on receipt of
revoke_and_ack.
For non-funder, the local feerate is set on receipt of
commitment_signed, and the remote feerate set on send of
revoke_and_ack. In our code, these two happen together.
channeld gets this right, but lightningd ignored the funder/fundee
distinction, and as a result, receipt of a commitment_signed by the
funder altered fees in the database. If there was a reconnection
event or restart, then these (incorrect) values would be used, causing
us to complain about a 'Bad commit_sig signature' and close the
channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A 'Bad commit_sig signature' was reported by @Javier on Telegram and
@DarthCoin. This was between two c-lightning peers, so definitely our fault.
Analysis of this message revealed the signature was using the wrong
feerate. I finally managed to make a test case which triggered this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Christian Decker
Vasil Dimov
Rusty Russell
darosior
lisa neigut
Michael Schmoock
Saibato
fiatjaf
Jarret Dyrbye
arowser