We added some code in 0b3a283586
to explicitly hold strong refs for all tasks/futures. At the time I was uncertain if that also solves
GC issues with asyncio.run_coroutine_threadsafe.
ref https://github.com/spesmilo/electrum/pull/9608#issuecomment-2703681663
Looks like it does. run_coroutine_threadsafe *is* going through the custom task factory.
See the unit test.
The somewhat confusing thing is that we need a few event loop iterations for the task factory to run,
due to how run_coroutine_threadsafe is implemented. And also, the task that we will hold as strong ref
in the global set is not the concurrent.futures.Future that run_coroutine_threadsafe returns.
So this commit simply "fixes" the unit test so that it showcases this, and removes related, older, plumbing
from util.py that we now know is no longer needed because of this.
channel_db.load_data() is slow, slowing down startup time (when trampoline is disabled).
util.list_enabled_bits() is one of the main contributors to the slowness, called by validate_features().
One could argue that we could even simply *not* call validate_features for gossip messages as part of load_data,
as they have already been validated before storing them in the db. However re-validating them there is a good
clean-up/sanity check IMO. Note that what is considered "valid" can change over time, so just because validate_features
passed when we originally received and stored a gossip message, it might no longer be valid a year later if the bolts change.
This caching decreases the time needed for load_data on two different machines / gossip dbs as below:
47 sec -> 10 sec
18 sec -> 6 sec
If instead of caching, I just rm the validate_features() calls, the benchmarks are almost identical, within noise.
That is, the cache looks really effective.
(the rest of the slowness is mostly due to lnmsg.decode_msg)
```
>>> lnutil.validate_features.cache_info()
CacheInfo(hits=172674, misses=287, maxsize=1000, currsize=277)
```
-----
We could alternatively directly cache util.list_enabled_bits (instead of validate_features).
That would be a bit slower and might end up using a lot more memory in some cases I think, but maybe conceptually would be cleaner.
Also note that if validate_features() raises an exception, that is not cached.
```
$ export ELECTRUM_LINTERS=E9,E101,E129,E273,E274,E703,E71,E722,F5,F6,F7,F8,W191,W29,B
$ export ELECTRUM_LINTERS_IGNORE=B007,B009,B010,B019,B036,F541,F841
$ flake8 . --count --select="$ELECTRUM_LINTERS" --ignore="$ELECTRUM_LINTERS_IGNORE" --show-source --statistics --exclude "*_pb2.py,electrum/_vendor/"
./electrum/commands.py:98:1: F811 redefinition of unused 'format_satoshis' from line 48
def format_satoshis(x):
^
./electrum/commands.py:437:9: F811 redefinition of unused 'Mnemonic' from line 62
from .mnemonic import Mnemonic
^
./electrum/gui/qt/wizard/wallet.py:37:5: F811 redefinition of unused 'Daemon' from line 14
from electrum.daemon import Daemon
^
./electrum/lntransport.py:14:1: F811 redefinition of unused 'Optional' from line 12
from typing import NamedTuple, List, Tuple, Mapping, Optional, TYPE_CHECKING, Union, Dict, Set, Sequence
^
./electrum/lntransport.py:14:1: F811 redefinition of unused 'TYPE_CHECKING' from line 12
from typing import NamedTuple, List, Tuple, Mapping, Optional, TYPE_CHECKING, Union, Dict, Set, Sequence
^
./electrum/plugin.py:966:13: F811 redefinition of unused 'hid' from line 593
import hid
^
./electrum/plugin.py:1040:13: F811 redefinition of unused 'hid' from line 593
import hid
^
./electrum/util.py:44:1: F811 redefinition of unused 'json' from line 26
import json
^
./electrum/util.py:46:1: F811 redefinition of unused 'NamedTuple' from line 29
from typing import NamedTuple, Optional
^
./electrum/util.py:46:1: F811 redefinition of unused 'Optional' from line 29
from typing import NamedTuple, Optional
^
./electrum/util.py:1456:56: F811 redefinition of unused 'traceback' from line 34
async def __aexit__(self, exc_type, exc_value, traceback):
^
./electrum/wallet_db.py:536:9: F811 redefinition of unused 'LOCAL' from line 46
LOCAL = 1
^
./electrum/wallet_db.py:537:9: F811 redefinition of unused 'REMOTE' from line 46
REMOTE = -1
^
./tests/test_bitcoin.py:28:1: F811 redefinition of unused 'bitcoin' from line 9
from electrum import crypto, constants, bitcoin
^
./tests/test_txbatcher.py:11:1: F811 redefinition of unused 'Transaction' from line 7
from electrum.transaction import Transaction, PartialTxInput, PartialTxOutput, TxOutpoint
^
./tests/test_wallet_vertical.py:20:1: F811 redefinition of unused 'Transaction' from line 10
from electrum.transaction import Transaction, PartialTxOutput, tx_from_any, Sighash
^
16 F811 redefinition of unused 'format_satoshis' from line 48
16
```
```
./electrum/commands.py:144:9: F824 `global known_commands` is unused: name is never assigned in scope
global known_commands
^
./electrum/commands.py:1916:9: F824 `global known_commands` is unused: name is never assigned in scope
global known_commands
^
./electrum/gui/qt/main_window.py:2405:13: F824 `nonlocal done` is unused: name is never assigned in scope
nonlocal done
^
./electrum/i18n.py:52:5: F824 `global language` is unused: name is never assigned in scope
global language
^
./electrum/plugin.py:189:9: F824 `global _root_permission_cache` is unused: name is never assigned in scope
global _root_permission_cache
^
5 F824 `global known_commands` is unused: name is never assigned in scope
5
```
With `txbatcher.SLEEP_INTERVAL = 0.01`, on my laptop, the batcher called try_broadcasting() even before next_tx() effectively awaited _tx_event.
This resulted in the test failing due to timeout.
Basically, if the txbatcher.SLEEP_INTERVAL was too low, the 2-4 event loop iterations needed to await _tx_event.wait() took too long.
(note that the exact number of event loop iterations needed depends on the python version and the OS)
