Commit Graph

7906 Commits

Author SHA1 Message Date
Rusty Russell 85bb93618b new modules: pyln.proto.message.{bolt1,bolt2,bolt4,bolt7}
These are autogenerated, but now they export their own
MessageNamespace, as well as the raw csv.

They also expose their SubtypeTypes, MessageTypes and TlvStreamTypes,
though in theory these could clash (they don't for now, and it'd be
kinda awkward if they did).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell 59bb6b90ac pyln: new module pyln.proto.message.bolts
This contains the CSVs for the current bolts (autogenerated).  It's a
separate module because I expect it to be updated alongside the spec.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message.bolts
2020-06-12 15:35:35 +02:00
Rusty Russell daa707d213 pyln.proto.message: separate fundamental types from other subtypes.
This will be useful for the next patch, which introduces per-bolt
modules.  This makes it easier for them generate variables for each
field type they parse (they don't want to export u16, for example)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell e4b5679f5d message: support option fields.
These are (probably) going away soon, but just tag them for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell 9992a577b4 pyln: add (undocumented) u8 fundamental type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell 42aab89b9f pyln.proto.message: expose fundamental MessageTypes as variables.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell 47631cc23c pyln.proto.message: use BufferedIOBase instead of bytes for binary ops.
Instead of val_to_bin/val_from_bin which deal with bytes, we implement
read and write which use streams.  This simplifies the API. 

Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell ed4eadc8f3 patch message-export-types.patch 2020-06-12 15:35:35 +02:00
Rusty Russell eb73a0dd8f pyln: add pyln.proto.message.
This supports infrasructure for creating messages.  In particular, it
can be fed CSV from the spec's `tools/extract-formats.py` and then convert
them all to and from strings and binary formats.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message
2020-06-12 15:35:35 +02:00
Rusty Russell 5b4be02ed3 pyln: add Makefile
This runs flake8 and the python tests.  Helps me, at least!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
niftynei 9e0ef45048 psbt: handle 'unsetting' final witness stack
Prior to this commit, passing a NULL stack to `bitcoin_tx_input_set_witness`
unsets the witness stack on the bitcoin_tx's wally_tx but leaves the
final witness on the PSBT unchanged.

at the moment, libwally's `wally_psbt_input_set_final_witness` will blow
up if you attempt to set a NULL witness -- instead we manually remove it
if the passed in stack is NULL. previously we would leave the PSBT's
witness unchanged.
2020-06-11 13:13:13 +02:00
niftynei 58282819a9 psbt: if a transaction has witnesses/scriptSig set, add it to psbt
For any transaction that's got 'finalized' signature data for an input,
we should add this information to the psbt also
2020-06-11 13:13:13 +02:00
niftynei 891f61ad48 channel_tx: add the commitment sig and pubkey data to the commit tx
needs to be update elsewhere too!
2020-06-11 13:13:13 +02:00
niftynei 971f615695 psbt: affirm database upgrade works for last_tx -> psbt
We use a database snapshot with 3 channels -- two of which have HTLCs
dangling and one is an initial open channel tx in the 'old' tx hex
format in last_tx and confirm that they are successfully updated to PSBT
format on start.
2020-06-11 13:13:13 +02:00
niftynei bb589e0eaf psbt: database migration for converting last_tx to a psbt
We update the `last_tx` in `channels` to be psbt format, instead
of a linearized transaction.

We need the amount of the input populated, which we have since
this is the 'funding' amount. Ideally we'd also populate the funding
scriptPubkey, but to do that we'd need to access the HSM module to fetch
our local funding pubkey, which isn't initialized at the time that the
database migrations are run.

Since the only field the HSM uses currently when signing these is the
amount field, it's ok to just leave it out.

needs a test!
2020-06-11 13:13:13 +02:00
niftynei 57488cde13 hsm: decouple hsm from wallet; init before wallet
We're going to use the hsm for a migration, so we need to set up the HSM
before we get to the wallet migration code.

All that this requires is removing the places in HSM init that we touch
the database struct -- easy enough to accomplish by passing the required
field back out from init, and then associating it onto the wallet after
it's been initialized.
2020-06-11 13:13:13 +02:00
niftynei 8fa04a710a psbt: move `channels.last_tx` field to be a psbt
note: missing migration at the moment lol
2020-06-11 13:13:13 +02:00
niftynei 052d40ae98 psbt: add method to confirm 'finalized' status of psbt
calling `wally_psbt_finalize` doesn't return a status indicator; instead
you must call `psbt_is_finalized` to check that it's eligible for
'extraction' -- extraction will fail if the psbt is not in a finalized
state.
2020-06-11 13:13:13 +02:00
niftynei 000ef2079c psbt: helpers for adding a pubkey or signature to a psbt
we'll use these for the commitment txs!
2020-06-11 13:13:13 +02:00
niftynei 3c6e3eecee psbt: add 'wally_psbt_clone' function, to clone a psbt
We'll need this for settng the tx correctly, for reasons
2020-06-11 13:13:13 +02:00
Rusty Russell d737b54625 bitcoin/tx: implement wally_tx_clone (badly) for now.
When libwally exposes it, we can use theirs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-11 13:13:13 +02:00
niftynei f9300e8480 tx: add setter for tx locktime
We need to update the psbt's global transaction simultaneously, so we
wrap access to the locktime in a method which will handle both
2020-06-11 13:13:13 +02:00
niftynei db8ef922ed psbt: add to/from byte helpers
We'll need these for the database methods we're going to add shortly
2020-06-11 13:13:13 +02:00
niftynei 894a46e8e3 psbt: populate last commitment transaction's input info at db
when re-populating a channel's data from the database, since we don't
store the psbt data (with input scripts + amounts), we need to
re-populate it.

the right solution is to patch the psbt into the database; for now we
'monkey-patch' it in.
2020-06-11 13:13:13 +02:00
niftynei 7969f61ab3 tx: update comment
Make comment a bit more descriptive / usefu
2020-06-11 13:13:13 +02:00
niftynei dc868630a8 tx-psbt: pass in the witness script (if known) when adding an input
Update the `bitcoin_tx_add_input` interface to accept a witness script
and or scriptPubkey.

We save the amount + witness script + witness program (if known) to
the PSBT object for a transaction when creating an input.
2020-06-11 13:13:13 +02:00
niftynei a04f0fe250 psbt: remove input_amounts from bitcoin tx
Instead we will stash them into the PSBT as a utxo/witness record (which
includes the amount)
2020-06-11 13:13:13 +02:00
niftynei 85f395f7d4 utxo: fill in scriptPubkey to NULL
We need this so we can access it when populating bitcoin inputs in the
next commit
2020-06-11 13:13:13 +02:00
niftynei cc6eba1d72 psbt: try one big alloc and fail instead of incremental buffer increases
was running into buffer overrun errors? something about the iteration
method was broken
2020-06-11 13:13:13 +02:00
niftynei 6197bd2b82 psbt: check return value on init for wally input 2020-06-11 13:13:13 +02:00
niftynei 1d56a4eda1 libwally: update to latest commit 2020-06-11 13:13:13 +02:00
Rusty Russell 0f568e1861 hsmd: remove unused sign_funding_tx.
We always treat it as a withdrawl.

Reported-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-02 08:47:31 +02:00
niftynei 27e83f075f doc: remove bitcoin PPA from install instructions, use snap
the PPA is no longer maintained, the new hotness is the snapd install.
except that snapd does some weirdness with the binary names, calling it
bitcoin-core.daemon and bitcoin-core.cli. To get around this, you can
create a symbolic link to the snap binaries, which is what we outline
here.
2020-06-01 23:18:15 +02:00
niftynei 185fe722be update-mocks: make it a bit easier to tell what step is happening
Adding a small explainer before printing the filename makes it a bit
clearer what's going on when parsing make logs
2020-05-30 15:36:56 +02:00
niftynei 76c57595c3 tools-make: add mock parser for clang ld output
According to #3226, it looks like clang's LD error format has changed.
This patch adds the new format so we can parse the mocks successfully.

Apple clang version 11.0.3 (clang-1103.0.32.29)

```
checking for ANSI C header files... Undefined symbols for architecture x86_64:
  "_fromwire_amount_msat", referenced from:
      _fromwire_tlv_test_n1_tlv3 in ccj4zKdV.o
  "_fromwire_bool", referenced from:
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_test_enum", referenced from:
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_tlvs", referenced from:
      _fromwire_test_tlv1 in ccj4zKdV.o
      _fromwire_test_tlv2 in ccj4zKdV.o
      _fromwire_test_tlv3 in ccj4zKdV.o
  "_fromwire_tu32", referenced from:
      _fromwire_tlv_test_n2_tlv2 in ccj4zKdV.o
  "_fromwire_tu64", referenced from:
      _fromwire_tlv_test_n1_tlv1 in ccj4zKdV.o
      _fromwire_tlv_test_n2_tlv1 in ccj4zKdV.o
  "_fromwire_u16", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_arrays in ccj4zKdV.o
      _fromwire_tlv_test_n1_tlv4 in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_tlv1 in ccj4zKdV.o
      ...
  "_fromwire_u32", referenced from:
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_u64", referenced from:
      _fromwire_test_short_id in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_fromwire_u8", referenced from:
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_var_len in ccj4zKdV.o
      _fromwire_subtype_varlen_varsize in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_fromwire_u8_array", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_arrays in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
      ...
```

Changelog-Fixed: Build for macOS Catalina / Apple clang v11.0.3 fixed
2020-05-30 15:36:56 +02:00
niftynei 3640befc10 tools-make: also be quiet if --quiet is flagged
Missed a update-mocks.sh call
2020-05-30 15:36:56 +02:00
niftynei 24e186bc20 test: use correct deps for channeld tests
LIGHTNING_CHANNELD_* doesn't exist; the actual name is
LIGHTNINGD_CHANNEL_*
2020-05-29 16:20:23 -05:00
niftynei 78d95b51aa nit: align spacing for SQL stmts 2020-05-29 16:20:23 -05:00
fiatjaf 96452eafb7 sort listinvoices and listsendpays by order of creation. 2020-05-29 15:30:59 +02:00
Rusty Russell ce9e559aed features: do not set option_support_large_channel in channel_announcement.
Spec is wrong (it says it should be compulsory), and Eclair doesn't set it
at all, leading to an error when they send their announcement_signatures.

Fixes: #3703
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: large-channels: negotiate successfully with Eclair nodes.
2020-05-26 19:57:29 -05:00
Rusty Russell b3eef81f19 pytest: add test for channel_announcement feature bits.
And neaten current feature mangling code now we have a wumbo flag for
expected_peer_features().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-26 19:57:29 -05:00
Rusty Russell 4d917e2566 gossipd: add test showing how we and Eclair disagree on channel_announcement features for wumbo channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-26 19:57:29 -05:00
Rusty Russell f4f8a363dd pytest: fix feature mask for EXPERIMENTAL_FEATURES, add wumbo support.
There are various places where our tests failed with
--enable-expimental-features.  And our plugin test overlapped an
existing feature.

We make our expected_feature functions more generic, and use them
everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-26 19:57:29 -05:00
Rusty Russell 0b89664076 gossipd: fix cut & paste in error messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-26 19:57:29 -05:00
Rusty Russell 0b3040b9a6 bitcoin/tx: insist input amounts all be populated across the wire.
If you need to send a tx where you don't know this info, send a tx_parts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30
Rusty Russell 1e889eeaf7 lightningd: have sign_last_tx populate the input amounts.
With this change, all bitcoin_tx we send across the wire have
their inputs_amounts populated.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30
Rusty Russell 48853ec954 onchaind: use tx_parts for initial tx.
For the moment it's a complete tx, but in future designs we might only
be given the specific input which closes the channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30
Rusty Russell abba59057c onchaind: remove chainparams args in favor of global.
Otherwise this creates noise for the next patch which switches the initial
`struct bitcoin_tx` into a `struct tx_parts`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30
Rusty Russell 48f397ee19 onchaind: receive a tx_parts instead of a tx when a tx is seen onchain.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30
Rusty Russell dafaf854c5 bitcoin/tx_parts: infrastructure for partial bitcoin txs.
`struct tx_parts` is just a txid and a bunch of inputs and outputs,
some of which may be NULL.

This is both a nod towards a future where we (or our peer) can combine
HTLCs or (in an eltoo world) commitments, although for the moment all
our tx_parts will be complete.

It also matches our plan to split `bitcoin_tx` into two types: this
`struct tx_parts` where we don't know input amounts etc, and `psbt`
where we do.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30