store exception in variable instead of using a bool flag
add default str to routing exceptions
Add separate exception class to handle fee related payment errors
Ideally, given an on-chain backup, after the remote force-closes, we should be able to spend our anchor output,
to CPFP the remote commitment tx (assuming the channel used OPTION_ANCHORS).
To spend the anchor output, we need to be able to sign with the local funding_privkey.
Previously we derived the funding_key from the channel_seed (which comes from os.urandom).
Prior to anchors, there was no use case for signing with the funding_key given a channel backup.
Now with anchors, we should make its derivation deterministic somehow, in a way so that it can
be derived given just an on-chain backup.
- one way would be to put some more data into the existing OP_RETURN
- uses block space
- the OP_RETURNs can be disabled via "use_recoverable_channels"
- only the initiator can use OP_RETURNs (so what if channel is in incoming dir?)
- instead, new scheme for our funding_key:
- we derive the funding_privkey from the lnworker root secret (derived from our bip32 seed)
- for outgoing channels:
- lnworker_root_secret + remote_node_id + funding_tx_nlocktime
- for incoming channels:
- lnworker_root_secret + remote_node_id + remote_funding_pubkey
- a check is added to avoid reusing the same key between channels:
not letting to user open more than one channel with the same peer in a single block
- only the first 16 bytes of the remote_node_id are used, as the onchain backup OP_RETURNs only contain that
- as the funding_privkey cannot be derived from the channel_seed anymore, it is included in the
imported channel backups, which in turn need a new version defined
- a wallet db upgrade is used to update already stored imported cbs
- alternatively we could keep the imported cbs as-is, so no new version, no new funding_privkey field, as it is clearly somewhat redundant given on-chain backups can reconstruct it
- however adding the field seems easier
- otherwise the existing code would try to derive the funding_privkey from the channel_seed
- also note: atm there is no field in the imported backups to distinguish anchor channels vs static-remotekey channels
* sets the weight of htlc transactions to zero, thereby putting a zero
fee for the htlc transactions
* add inputs to htlc-tx for fee bumping
* switches feature flags
* disable anchor test vectors, which are now partially invalid
* add a method for backups to sweep to_remote
* to_remote sweeping needs the payment_basepoint's private key
to sign the sweep transaction
* we restore the private key from our funding multisig pubkey
(pubished with the closing transaction) and a static payment key secret
* to_remote has now an additional csv lock of 1
* anchor outputs are added if to_local/remote outputs are present
* funder balance is reduced to accomodate anchors
* changes the htlc outputs' witness script to have a csv lock of 1
* send signatures for remote ctx with ANYONECANPAY|SINGLE
* refactor htlc weight (useful for zero-fee-htlc)
* in order to be able to sweep to_remote in an onchain backup scenario
we need to retain the private key for the payment_basepoint
* to facilitate the above, we open a channel derived from a static
secret (tied to the wallet seed), the static_payment_key combined with
the funding pubkey (multisig_key), which we can restore from the channel
closing transaction
- Separation between SwapManager and its transport:
Legacy transpport uses http, Nostr uses websockets
- The transport uses a context to open/close connections.
This context is not async, because it needs to be called
from the GUI
- Swapserver fees values are initialized to None instead
of 0, so that any attempt to use them before the swap
manager is initialized will raise an exception.
- Remove swapserver fees disk caching (swap_pairs file)
- Regtests use http transport
- Android uses http transport (until QML is ready)
This will be useful if we decide to ship lntransport as a separate
package. It is also a conceptual cleanup.
Notes:
- lntransport still requires crypto.py
- parsing node id from a bolt11 invoice is not supported.
Add support for key-path-spending taproot utxos into transaction.py.
- no wallet support yet
- add some psbt, and minimal descriptor support
- preliminary work towards script-path spends
get_preimage_script should really have been private API...
looks like everywhere it is used outside of transaction.py, it is actually abused :/
Other existing usages in plugin code I don't dare to touch without lots of manual testing...
Instead of some functions operating with hex strings,
and others using bytes, this consolidates most things to use bytes.
This mainly focuses on bitcoin.py and transaction.py,
and then adapts the API usages in other files.
Notably,
- scripts,
- pubkeys,
- signatures
should be bytes in almost all places now.
- all forwarding types use the same flow
- forwarding callback returns a htlc_key or None
- forwarding info is persisted in lnworker:
- ongoing_forwardings
- downstream to upstream htlc_key
- htlc_key -> error_bytes
- introduce PaymentFeeBudget, which contains limits for fee budget and cltv budget
- when splitting a payment,
- the fee budget is linearly distributed between the parts
- this resolves a FIXME in lnrouter ("FIXME in case of MPP")
- the cltv budget is simply copied
- we could also add other kinds of budgets later, e.g. for the num in-flight htlcs
- resolves TODO in lnworker ("todo: compare to the fee of the actual route we found")
follow-up https://github.com/spesmilo/electrum/pull/8536
This replaces 69336befee, which was insufficient.
#8536 added a new field into the struct, which older versions do not ignore but raise:
opening a wallet file with new code updated the struct to include it,
after which old code could no longer open the wallet file.
i.e. #8536 was an invisible wallet upgrade, breaking compat.
This commit simply formalises the wallet upgrade: old code will now show
an understandable error when trying to open new files.
LnFeatures.supports() is became part of the hot path of
LNPathFinder.find_path_for_payment()
in 6b43eac6fd,
which made find_path_for_payment considerably slower than before.
It used to take around 0.8 sec originally, then after linked commit went to ~9 sec,
and now this takes it down to ~1.1 sec (on my laptop).
follow-up https://github.com/spesmilo/electrum/pull/8536
```
1.52 | E | gui.qt.ElectrumGui |
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/gui/qt/__init__.py", line 342, in start_new_window
wallet = self.daemon.load_wallet(path, None)
File "/home/user/wspace/electrum/electrum/daemon.py", line 469, in func_wrapper
return func(self, *args, **kwargs)
File "/home/user/wspace/electrum/electrum/daemon.py", line 479, in load_wallet
wallet = self._load_wallet(path, password, manual_upgrades=manual_upgrades, config=self.config)
File "/home/user/wspace/electrum/electrum/util.py", line 466, in do_profile
o = func(*args, **kw_args)
File "/home/user/wspace/electrum/electrum/daemon.py", line 504, in _load_wallet
db = WalletDB(storage.read(), manual_upgrades=manual_upgrades)
File "/home/user/wspace/electrum/electrum/wallet_db.py", line 117, in __init__
self._after_upgrade_tasks()
File "/home/user/wspace/electrum/electrum/wallet_db.py", line 247, in _after_upgrade_tasks
self._load_transactions()
File "/home/user/wspace/electrum/electrum/util.py", line 466, in do_profile
o = func(*args, **kw_args)
File "/home/user/wspace/electrum/electrum/wallet_db.py", line 1536, in _load_transactions
self.data = StoredDict(self.data, self, [])
File "/home/user/wspace/electrum/electrum/json_db.py", line 117, in __init__
self.__setitem__(k, v)
File "/home/user/wspace/electrum/electrum/json_db.py", line 49, in wrapper
return func(self, *args, **kwargs)
File "/home/user/wspace/electrum/electrum/json_db.py", line 135, in __setitem__
v = self.db._convert_dict(self.path, key, v)
File "/home/user/wspace/electrum/electrum/json_db.py", line 247, in _convert_dict
v = dict((k, constructor(**x)) for k, x in v.items())
File "/home/user/wspace/electrum/electrum/json_db.py", line 247, in <genexpr>
v = dict((k, constructor(**x)) for k, x in v.items())
TypeError: ImportedChannelBackupStorage.__init__() missing 1 required positional argument: 'local_payment_pubkey'
```
scenario:
- user opens a lightning channel and exports an "imported channel backup"
- user closes channel via local-force-close
- local ctx is published, to_local output has user's funds and they are CSV-locked for days
- user restores wallet file from seed and imports channel backup
- new wallet file should be able to sweep coins from to_local output (after CSV expires)
This was not working previously, as the local_payment_basepoint was not included in the
imported channel backups, and the code was interpreting the lack of this as the channel not
having option_static_remotekey enabled. This resulted in lnutil.extract_ctn_from_tx
using an incorrect funder_payment_basepoint, and lnsweep not recognising the ctx due to
the garbage ctn value.
The imported channel backup serialisation format is slightly changed to include the
previously missing field, and its version number is bumped (0->1). We allow importing
both version 0 and version 1 backups, however v0 backups cannot handle the above
described scenario (they can only be used to request a remote-force-close).
Note that we were/are setting the missing local_payment_basepoint to the pubkey of
one of the wallet change addresses, which is bruteforceable if necessary, but I
think it is not worth the complexity to add this bruteforce logic. Also note
that the bruteforcing could only be done after the local-force-close was broadcast.
Ideally people with existing channels and already exported v0 backups should re-export
v1 backups... Not sure how to handle this.
closes https://github.com/spesmilo/electrum/issues/8516
decorators (instead of overloading JsonDB._convert_dict and
_convert_value)
- stored_in for elements of a StoreDict
- stored_as for singletons
- extra register methods are defined for key conversions
This commit was adapted from the jsonpatch branch
closes https://github.com/spesmilo/electrum/issues/8403
> In Python 3.10 that worked fine, however in Python 3.11 large integer check https://github.com/python/cpython/issues/95778, so now this throws an error.
Apparently this change was deemed a security fix and was backported to all supported branches of CPython (going back to 3.7). i.e. it affects ~all versions of python (if sufficiently updated with bugfix patches), not just 3.11
> Some offending node aliases:
> ```
> ergvein-fiatchannels
> test-mainnet
> arakis
> ```
The features bits set by some of these nodes:
```
(1, 7, 8, 11, 13, 14, 17, 19, 23, 27, 45, 32973, 52973)
(1, 7, 8, 11, 13, 14, 17, 19, 23, 27, 39, 45, 55, 32973, 52973)
```
> P.S. I see there are a lot of nodes with 253 bytes in their feature vectors. Any idea why that could happen?
Note that the valid [merged-into-spec features](50b2df24a2/09-features.md) currently only go as high as ~51.
However the spec does not specify how to choose feature bits for experimental stuff, so I guess some people are using values in the 50k range. The only limit imposed by the spec on the length of the features bitvector is an implicit one due to the max message size: every msg must be smaller than 65KB, and the features bitvector needs to fit inside the init message, hence it can be up to ~524K bits.
(note that the features are not stored in a sparse representation in the init message and in gossip messages, so if many nodes set such high feature bits, that would noticably impact the size of the gossip).
-----
Anyway, our current implementation of LnFeatures is subclassing IntFlag, and it looks like it does not work well for such large integers. I've managed to make IntFlags reasonably in python 3.11 by overriding __str__ and __repr__ (note that in cpython it is apparently only the base2<->base10 conversions that are slow, power-of-2 conversions are fast, so we can e.g. use `hex()`). However in python 3.10 and older, enum.py itself seems really slow for bigints, e.g. enum._decompose in python 3.10.
Try e.g. this script, which is instant in py3.11 but takes minutes in py3.10:
```py
from enum import IntFlag
class c(IntFlag):
known_flag_1 = 1 << 0
known_flag_2 = 1 << 1
known_flag_3 = 1 << 2
if hasattr(IntFlag, "_numeric_repr_"): # python 3.11+
_numeric_repr_ = hex
def __repr__(self):
return f"<{self._name_}: {hex(self._value_)}>"
def __str__(self):
return hex(self._value_)
a = c(2**70000-1)
q1 = repr(a)
q2 = str(a)
```
AFAICT we have two options: either we rewrite LnFeatures so that it does not use IntFlag (and enum.py), or, for the short term as workaround, we could just reject very large feature bits.
For now, I've opted to the latter, rejecting feature bits over 10k.
(note that another option is bumping the min required python to 3.11, in which case with the overrides added in this commit the performance looks perfectly fine)
In the binary serialised format, replace all instances of int16 with uint16.
In particular, this allows port>32767.
Fixes https://github.com/spesmilo/electrum/issues/8264
I think this is backwards compatible, as in, any existing channel backup already out there,
should be properly parsed with the new code. (new code however can serialise cbs that old
code deserialises incorrectly)
```
>>> struct.pack('<h', 258)
b'\x02\x01'
>>> struct.pack('<H', 258)
b'\x02\x01'
```
