If the full tx is missing, we should force mempool/confirmed txs to be LOCAL height,
however future txs should not be forced to LOCAL, they should remain FUTURE.
follow-up 197933debf
get_tx_height previously did not consider whether the walletDB has the full tx for
the corresponding txid, and could consider a tx even "mined" and spv-ed, even if
we were missing the raw tx.
Now if the full tx is missing or the tx in the db is partial,
get_tx_height considers it to be LOCAL.
In particular the txbatcher, in `run_iteration`,
- first saves a tx as local *unsigned* (to reserve UTXOs),
- then signs it and tries to broadcast it.
The history tx will later transition to local and signed,
after we get back the broadcasted tx via the Synchronizer dance.
In the meantime there is a race where we have an unsigned tx in the history,
but the txid could already transition to mempool or even to mined,
before we download the full raw tx from the server.
During that time window, it makes the adb state more consistent
to just consider the history tx to be local, as done here.
tx_height comes from the `get_history` RPC, we then call the `get_transaction` RPC.
By the time the `get_transaction` RPC returns, we might have received another
scripthash status update, called `get_history` again, and updated height for the txid.
Synchronizer._get_transaction() should not call adb.receive_tx_callback() with
the old tx_height (but it was doing exactly that).
Patch to trigger, e.g. regtest failures:
(e.g. for tests.regtest.TestLightningAB.test_extract_preimage)
```
diff --git a/electrum/interface.py b/electrum/interface.py
index 8649652b9c..fce7a1c6de 100644
--- a/electrum/interface.py
+++ b/electrum/interface.py
@@ -991,6 +991,7 @@ class Interface(Logger):
return res
async def get_transaction(self, tx_hash: str, *, timeout=None) -> str:
+ await asyncio.sleep(3)
if not is_hash256_str(tx_hash):
raise Exception(f"{repr(tx_hash)} is not a txid")
raw = await self.session.send_request('blockchain.transaction.get', [tx_hash], timeout=timeout)
```
Adds a new config option: `WALLET_FREEZE_REUSED_ADDRESS_UTXOS`.
This is based on Bitcoin Core's "avoid_reuse" wallet flag. [0]
This opt-in feature, if enabled:
> Automatically freeze coins received to already used addresses.
> This can eliminate a serious privacy issue where a malicious user can track your spends by sending small payments
> to a previously-paid address of yours that would then be included with unrelated inputs in your future payments.
Note that currently we only have a single coinchooser policy, `CoinChooserPrivacy`,
which interacts well with this option, as it spends all coins from any selected address.
However, if we later add a different coinchooser policy, which allowed "partial spends",
care should be taken re e.g. disallowing using that when this option is set.
Also note that this PR adds this as a config option, but arguably it could be wallet-specific instead,
such as `use_change`.
[0]: https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.19.0.1.md#wallet
closes https://github.com/spesmilo/electrum/issues/7497
Previously calling add_transaction with a malformed Transaction obj could
result in an exception late in the flow, after the walletdb was already side-effected.
Rollback of such side-effects is not implemented :/
but this small patch should at least cover and prevent some common cases.
```
File "/opt/electrum/electrum/address_synchronizer.py", line 358, in add_transaction
self.db.add_transaction(tx_hash, tx)
File "/opt/electrum/electrum/json_db.py", line 42, in wrapper
return func(self, *args, **kwargs)
File "/opt/electrum/electrum/wallet_db.py", line 1434, in add_transaction
tx = tx_from_any(str(tx))
File "/opt/electrum/electrum/transaction.py", line 1339, in tx_from_any
raise SerializationError(f"Failed to recognise tx encoding, or to parse transaction. "
```
Instead of some functions operating with hex strings,
and others using bytes, this consolidates most things to use bytes.
This mainly focuses on bitcoin.py and transaction.py,
and then adapts the API usages in other files.
Notably,
- scripts,
- pubkeys,
- signatures
should be bytes in almost all places now.
> These kind of checks look incorrect and confused/confusing for me:
> 43a5d03426/electrum/gui/qml/qeinvoice.py (L388)
> 43a5d03426/electrum/gui/qt/main_window.py (L1801)
> For example, consider creating a local tx that spends all UTXOs in the wallet, and consolidates them to another ismine address.
> Such a tx basically does not change what wallet.get_balance() returns.
> but to the checks listed above, the confirmed balance of the wallet I guess should be 0?
I am only making this change as it makes it simpler to do manual testing of CPFP-ing *local* txs.
That is, by changing a few easy-to-find lines of code, I can make the GUI allow CPFP-ing a local tx, but
without this change it errors out with "unknown fee for parent transaction".
If one has an incoming local tx saved in the wallet, adb.get_tx_fee(txid) returns None
(if the tx gets into the mempool, it will use the server-reported value instead).
In contrast, wallet.get_tx_info(tx).fee calls both wallet.get_wallet_delta(tx) and adb.get_tx_fee(txid),
making it more expensive but more complete.
A new config API is introduced, and ~all of the codebase is adapted to it.
The old API is kept but mainly only for dynamic usage where its extra flexibility is needed.
Using examples, the old config API looked this:
```
>>> config.get("request_expiry", 86400)
604800
>>> config.set_key("request_expiry", 86400)
>>>
```
The new config API instead:
```
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS
604800
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS = 86400
>>>
```
The old API operated on arbitrary string keys, the new one uses
a static ~enum-like list of variables.
With the new API:
- there is a single centralised list of config variables, as opposed to
these being scattered all over
- no more duplication of default values (in the getters)
- there is now some (minimal for now) type-validation/conversion for
the config values
closes https://github.com/spesmilo/electrum/pull/5640
closes https://github.com/spesmilo/electrum/pull/5649
Note: there is yet a third API added here, for certain niche/abstract use-cases,
where we need a reference to the config variable itself.
It should only be used when needed:
```
>>> var = config.cv.WALLET_PAYREQ_EXPIRY_SECONDS
>>> var
<ConfigVarWithConfig key='request_expiry'>
>>> var.get()
604800
>>> var.set(3600)
>>> var.get_default_value()
86400
>>> var.is_set()
True
>>> var.is_modifiable()
True
```
Previously if a local tx got broadcast, it was still displayed as local
in the history until it got mined (or some other action forced a full refresh).
- they are now distinguished from local by the status text "in %d blocks"
- this status text needs updating occasionally: on new blocks,
and some lnwatcher interactions, hence new event: "adb_set_future_tx"
- wallet.add_input_info() previously had a fallback to download parent
prev txs from the network (after a lookup in wallet.db failed).
wallet.add_input_info() is not async, so the network request cannot
be done cleanly there and was really just a hack.
- tx.add_info_from_wallet() calls wallet.add_input_info() on each txin,
in which case these network requests were done sequentially, not concurrently
- the network part of wallet.add_input_info() is now split out into new method:
txin.add_info_from_network()
- in addition to tx.add_info_from_wallet(), there is now also tx.add_info_from_network()
- callers of old tx.add_info_from_wallet() should now called either
- tx.add_info_from_wallet(), then tx.add_info_from_network(), preferably in that order
- tx.add_info_from_wallet() alone is sufficient if the tx is complete,
or typically when not in a signing context
- callers of wallet.bump_fee and wallet.dscancel are now expected to have already
called tx.add_info_from_network(), as it cannot be done in a non-async context
(but for the common case of all-inputs-are-ismine, bump_fee/dscancel should work regardless)
- PartialTxInput.utxo was moved to the baseclass, TxInput.utxo
fixes https://github.com/spesmilo/electrum/issues/8240#8240 was triggering an AssertionError in wallet.get_invoice_status,
as code there was assuming conf >= 0. To trigger, force-close
a LN channel, and while the sweep is waiting on the CSV, try to
make a payment in the Send tab to the ismine change address used
for the sweep in the future_tx. (order of events can also be reversed)
- add a new event, 'adb_removed_tx'
- new wallet method: get_tx_parents
- number of parents is shown in coins tab
- detailed list of parents is shown in dialog
ShortIDs were originally designed for lightning channels, and are now
understood by some block explorers.
This allows to remove one column in the UTXO tab (height is redundant).
In the transaction dialog, the space saving ensures that all inputs fit
into one line (it was not the case previously with p2wsh addresses).
For clarity and consistency, the ShortID is displayed for both inputs
and outputs in the transaction dialog.
- AddressSynchronizer no longer has its own state re up_to_date,
it defers to Synchronizer/Verifier instead
- Synchronizer is now tracking address sync states throughout their lifecycle:
and Synchronizer.is_up_to_date() checks all states
- Synchronizer.add_queue (internal) is removed as it was redundant
- should fix wallet.is_up_to_date flickering during sync due to race
related:
dc6c4814061c20a29a22
This reverts commit dc6c481406 as it introduced its own issue:
while add_address was running on one thread, synchronizer._reset could be running on another,
and by the time the "enqueue" coro would run, it would use a new add_queue and
addr would not be in requested_addrs anymore...
```
I/w | wallet.Standard_Wallet.[test_segwit_2] | starting taskgroup.
I | lnworker.LNWallet.[test_segwit_2] | starting taskgroup.
E/i | interface.[testnet.qtornado.com:51002] | Exception in run: KeyError('tb1q3wmgf8n5eettnj50pzgnfrrpdpjmwn37x7nzsc5780kk4je9v4hspym8mu')
Traceback (most recent call last):
File ".../electrum/electrum/util.py", line 1243, in wrapper
return await func(*args, **kwargs)
File ".../electrum/electrum/interface.py", line 506, in wrapper_func
return await func(self, *args, **kwargs)
File ".../electrum/electrum/interface.py", line 529, in run
await self.open_session(ssl_context)
File ".../electrum/electrum/interface.py", line 679, in open_session
async with self.taskgroup as group:
File ".../aiorpcX/aiorpcx/curio.py", line 304, in __aexit__
await self.join()
File ".../electrum/electrum/util.py", line 1339, in join
task.result()
File ".../electrum/electrum/synchronizer.py", line 80, in _run_tasks
async with taskgroup as group:
File ".../aiorpcX/aiorpcx/curio.py", line 304, in __aexit__
await self.join()
File ".../electrum/electrum/util.py", line 1339, in join
task.result()
File ".../electrum/electrum/synchronizer.py", line 127, in subscribe_to_address
self.requested_addrs.remove(addr)
KeyError: 'tb1q3wmgf8n5eettnj50pzgnfrrpdpjmwn37x7nzsc5780kk4je9v4hspym8mu'
```
AddressSynchronizer.add_address called synchronizer.add, which would only
schedule adding the addr to the Synchronizer in the next event loop iter.
If during that time, the synchronizer called adb.set_up_to_date(True),
the wallet would falsely believe and advertise itself as up_to_date
(as the wallet would see wallet.synchronize not creating new addresses,
and adb (via synchronizer) telling it is up_to_date).
Moments later, the synchronizer._add_address is finally executed and
up_to_date=False propagates out synchronizer->adb->wallet.
- revert the logic of do_breach_remedy to what it was
before 0ca3d66d15,
but now calling self.maybe_redeem unconditionally.
- replace mempool transactions only if the fee increases
- do not notify the GUI if a local tx is replaced
- delete labels when replacing
The wallet needs its own up_to_date logic:
- the adb being up_to_date means all its addresses are synced
- but an HD wallet might decide to roll the gap limit and generate new addresses
- the adb does not know about this...
- the wallet should be considered *not* up_to_date
- relatedly, it is now the wallet that decides to reset the network request counters
- note that wallet.main() was never cleaned up previously.
- now wallet gets its own taskgroup, which is cleaned up in wallet.stop.
Follow-up to adb refactor: 121d8732f1
