Adds a new configvar `WALLET_PARTIAL_WRITES` to enable/disable partial writes for the walletDB.
This is a further restriction on top of the existing restrictions,
e.g. wallet files still need to have file encryption disabled for partial writes.
It defaults to off, so even for unencrypted wallets we disable partial writes for now.
This is used as a stopgap measure until we fix the issues found with the partial writes impl
(see https://github.com/spesmilo/electrum/issues/10000).
The hack needs to be applied before we try importing electrum_ecc, i.e. it needs to be in the main script.
However, it should also be applied if the main script is not invoked directly, but e.g. the user imports electrum directly.
Hence the duplication.
- CURRENT_WALLET is set when a single wallet is loaded in memory, and it
remains set after Electrum stops running.
- If several wallets are loaded at the same time, CURRENT_WALLET is unset,
and RPCs must specify the wallet explicitly (using --wallet for the CLI)
- The fallback to 'default_wallet' essentially only applies when
creating a new wallet file
This config var should have never been saved to the config file, therefore it is safe to remove
instead of setting to '' to overwrite any existing value.
This module tries to restrict the ability of other processes to access the memory of our process.
Traditionally, on Linux, one process can access the memory of another arbitrary process
if both are running as the same user (uid). (Root can ofc access the memory of ~any process)
Programs can opt-out from this by setting `prctl(PR_SET_DUMPABLE, 0);`
also see https://man.archlinux.org/man/PR_SET_DUMPABLE.2const.en
-----
Also, from https://unix.stackexchange.com/a/518452 :
In a terminal window:
```
% echo $$ # show our pid
6744
% read -sp 'secret password: '; echo
secret password:
%
```
Then in another terminal window:
```
% grep heap /proc/6744/maps
01bb7000-01c3e000 rw-p 00000000 00:00 0 [heap]
% dd if=/proc/6744/mem bs=1 skip=$((0x01bb7000)) count=$((0x01c3e000-0x01bb7000)) status=none |
strings | less
...
% dd if=/proc/6744/mem bs=1 skip=$((0x01bb7000)) count=$((0x01c3e000-0x01bb7000)) status=none |
strings | grep obiwan
obiwan_kenobi # "secret password"
```
- Separation between SwapManager and its transport:
Legacy transpport uses http, Nostr uses websockets
- The transport uses a context to open/close connections.
This context is not async, because it needs to be called
from the GUI
- Swapserver fees values are initialized to None instead
of 0, so that any attempt to use them before the swap
manager is initialized will raise an exception.
- Remove swapserver fees disk caching (swap_pairs file)
- Regtests use http transport
- Android uses http transport (until QML is ready)
```
$ ./run_electrum --testnet signmessage -w /home/user/.electrum/testnet/wallets/test_trezor_white_bip84 tb1q5pguna9y2g9y2gsu8r8gmxeye2cefvyly8dg02 heyheyhey -o
0.84 | W | plugins.jade.jadepy.jade | No module named 'electrum.plugins.jade.jadepy.jade_ble'
0.84 | W | plugins.jade.jadepy.jade | BLE scanning/connectivity will not be available
3.73 | E | __main__ | error running command (without daemon)
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/plugins/trezor/clientbase.py", line 151, in get_xpub
node = trezorlib.btc.get_public_node(self.client, address_n).node
File "/home/user/.local/lib/python3.10/site-packages/trezorlib/tools.py", line 274, in wrapped_f
ret = f(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/trezorlib/btc.py", line 125, in get_public_node
return client.call(
File "/home/user/.local/lib/python3.10/site-packages/trezorlib/tools.py", line 297, in wrapped_f
return f(client, *args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/trezorlib/client.py", line 260, in call
resp = self._callback_pin(resp)
File "/home/user/.local/lib/python3.10/site-packages/trezorlib/client.py", line 186, in _callback_pin
pin = self.ui.get_pin(msg.type)
File "/home/user/wspace/electrum/electrum/plugins/trezor/clientbase.py", line 308, in get_pin
pin = self.handler.get_pin(msg.format(self.device), show_strength=show_strength)
AttributeError: 'NoneType' object has no attribute 'get_pin'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/wspace/electrum/./run_electrum", line 540, in handle_cmd
result = fut.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/user/wspace/electrum/./run_electrum", line 227, in run_offline_command
password = get_password_for_hw_device_encrypted_storage(plugins)
File "/home/user/wspace/electrum/./run_electrum", line 212, in get_password_for_hw_device_encrypted_storage
return client.get_password_for_storage_encryption()
File "/home/user/wspace/electrum/electrum/plugin.py", line 523, in wrapper
return run_in_hwd_thread(partial(func, *args, **kwargs))
File "/home/user/wspace/electrum/electrum/plugin.py", line 516, in run_in_hwd_thread
return fut.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/wspace/electrum/electrum/plugins/hw_wallet/plugin.py", line 260, in get_password_for_storage_encryption
xpub = self.get_xpub(derivation, "standard")
File "/home/user/wspace/electrum/electrum/plugin.py", line 523, in wrapper
return run_in_hwd_thread(partial(func, *args, **kwargs))
File "/home/user/wspace/electrum/electrum/plugin.py", line 513, in run_in_hwd_thread
return func()
File "/home/user/wspace/electrum/electrum/plugins/trezor/clientbase.py", line 150, in get_xpub
with self.run_flow(creating_wallet=creating):
File "/home/user/wspace/electrum/electrum/plugins/trezor/clientbase.py", line 89, in __exit__
self.end_flow()
File "/home/user/wspace/electrum/electrum/plugins/trezor/clientbase.py", line 82, in end_flow
self.handler.finished()
AttributeError: 'NoneType' object has no attribute 'finished'
```
Previously, generally, in case of any error, commands would raise a generic "Exception()" and the CLI/RPC would convert that and return it as `str(e)`.
With this change, we now distinguish "user-facing exceptions" (e.g. "Password required" or "wallet not loaded") and "internal errors" (e.g. bugs).
- for "user-facing exceptions", the behaviour is unchanged
- for "internal errors", we now pass around the traceback (e.g. from daemon server to rpc client) and show it to the user (previously, assuming there was a daemon running, the user could only retrieve the exception from the log of that daemon). These errors use a new jsonrpc error code int (code 2).
As the logic only changes for "internal errors", I deem this change not to be compatibility-breaking.
----------
Examples follow.
Consider the following two commands:
```
@command('')
async def errorgood(self):
from electrum.util import UserFacingException
raise UserFacingException("heyheyhey")
@command('')
async def errorbad(self):
raise Exception("heyheyhey")
```
----------
(before change)
CLI with daemon:
```
$ ./run_electrum --testnet daemon -d
starting daemon (PID 9221)
$ ./run_electrum --testnet errorgood
heyheyhey
$ ./run_electrum --testnet errorbad
heyheyhey
$ ./run_electrum --testnet stop
Daemon stopped
```
CLI without daemon:
```
$ ./run_electrum --testnet -o errorgood
heyheyhey
$ ./run_electrum --testnet -o errorbad
heyheyhey
```
RPC:
```
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorgood","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 1, "message": "heyheyhey"}}
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorbad","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 1, "message": "heyheyhey"}}
```
----------
(after change)
CLI with daemon:
```
$ ./run_electrum --testnet daemon -d
starting daemon (PID 9254)
$ ./run_electrum --testnet errorgood
heyheyhey
$ ./run_electrum --testnet errorbad
(inside daemon): Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/daemon.py", line 254, in handle
response['result'] = await f(*params)
File "/home/user/wspace/electrum/electrum/daemon.py", line 361, in run_cmdline
result = await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 163, in func_wrapper
return await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 217, in errorbad
raise Exception("heyheyhey")
Exception: heyheyhey
internal error while executing RPC
$ ./run_electrum --testnet stop
Daemon stopped
```
CLI without daemon:
```
$ ./run_electrum --testnet -o errorgood
heyheyhey
$ ./run_electrum --testnet -o errorbad
0.78 | E | __main__ | error running command (without daemon)
Traceback (most recent call last):
File "/home/user/wspace/electrum/./run_electrum", line 534, in handle_cmd
result = fut.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/user/wspace/electrum/./run_electrum", line 255, in run_offline_command
result = await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 163, in func_wrapper
return await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 217, in errorbad
raise Exception("heyheyhey")
Exception: heyheyhey
```
RPC:
```
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorgood","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 1, "message": "heyheyhey"}}
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorbad","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 2, "message": "internal error while executing RPC", "data": {"exception": "Exception('heyheyhey')", "traceback": "Traceback (most recent call last):\n File \"/home/user/wspace/electrum/electrum/daemon.py\", line 254, in handle\n response['result'] = await f(*params)\n File \"/home/user/wspace/electrum/electrum/commands.py\", line 163, in func_wrapper\n return await func(*args, **kwargs)\n File \"/home/user/wspace/electrum/electrum/commands.py\", line 217, in errorbad\n raise Exception(\"heyheyhey\")\nException: heyheyhey\n"}}}
```
Plugins.stop was never called, so the Plugins thread only stopped
because of the is_running() check in run(), which triggers too late:
the Plugins thread was stopping after the main thread stopped.
E.g. playing around in the qt wizard with wallet creation for a Trezor,
and closing the wizard (only window):
``` 24.85 | E | p/plugin.Plugins |
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/util.py", line 386, in run_jobs
job.run()
File "/home/user/wspace/electrum/electrum/plugin.py", line 430, in run
client.timeout(cutoff)
File "/home/user/wspace/electrum/electrum/plugin.py", line 363, in wrapper
return run_in_hwd_thread(partial(func, *args, **kwargs))
File "/home/user/wspace/electrum/electrum/plugin.py", line 355, in run_in_hwd_thread
fut = _hwd_comms_executor.submit(func)
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
```
if qt is not installed, e.g. on a server, was getting:
```
$ ./run_electrum sadasdasddsa
Traceback (most recent call last):
File "/home/user/wspace/electrum/./run_electrum", line 532, in <module>
main()
File "/home/user/wspace/electrum/./run_electrum", line 383, in main
lang = get_default_language(gui_name=gui_name)
File "/home/user/wspace/electrum/electrum/gui/default_lang.py", line 23, in get_default_language
from PyQt5.QtCore import QLocale
ModuleNotFoundError: No module named 'PyQt5.QtCore'
```
Without this,
`$ python3 -m unittest electrum.tests.regtest.TestUnixSockets.test_unixsockets`
was failing on my machine but succeeding on CI, due to timing differences.
I have reconsidered and now think that we should always hard-fail
if asserts asserts are disabled. It is just easier to reason about
the code knowing that asserts are evaluated.
If an end-user or library user has a concrete use case where this is
a problem, please open an issue and let us know.
follow-up
0f541be6f10e5464ca13
A new config API is introduced, and ~all of the codebase is adapted to it.
The old API is kept but mainly only for dynamic usage where its extra flexibility is needed.
Using examples, the old config API looked this:
```
>>> config.get("request_expiry", 86400)
604800
>>> config.set_key("request_expiry", 86400)
>>>
```
The new config API instead:
```
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS
604800
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS = 86400
>>>
```
The old API operated on arbitrary string keys, the new one uses
a static ~enum-like list of variables.
With the new API:
- there is a single centralised list of config variables, as opposed to
these being scattered all over
- no more duplication of default values (in the getters)
- there is now some (minimal for now) type-validation/conversion for
the config values
closes https://github.com/spesmilo/electrum/pull/5640
closes https://github.com/spesmilo/electrum/pull/5649
Note: there is yet a third API added here, for certain niche/abstract use-cases,
where we need a reference to the config variable itself.
It should only be used when needed:
```
>>> var = config.cv.WALLET_PAYREQ_EXPIRY_SECONDS
>>> var
<ConfigVarWithConfig key='request_expiry'>
>>> var.get()
604800
>>> var.set(3600)
>>> var.get_default_value()
86400
>>> var.is_set()
True
>>> var.is_modifiable()
True
```
