When replaying messages during channel-reestablishment,
previously we first resent all update messages, along with potential commitment_signed messages,
and then we potentially resent a single revoke_and_ack.
This can result in incorrect behaviour in case both a commitment_signed and a revoke_and_ack needs to be resent.
When replaying messages, the relative order of commitment_signed and revoke_and_messages needs to be preserved.
(the order of updates (htlc/fee) in relation to the revack messages does not matter)
implements https://github.com/lightning/bolts/pull/810
The logic here is somewhat based on what c-lightning does:
01e5f1886e/channeld/channeld.c (L3059)
Scenario:
- 2of2 multisig wallet with device1 and device2
- disconnect all devices
- open wallet file
- fail all pairings at wallet-open
- connect device2
- try to sign a tx
At this point Electrum will try to find the device for keystore1 first, and there is only a single unpaired device: device2.
Automatic pairing of keystore1 and device2 will fail, due to device id mismatching compared to what is persisted on disk for keystore1, so the user is prompted for manual selection. The selection dialog is somewhat confusing as it is not clear that the app is asking to select a device for keystore1. Pairing would fail, so the user is expected to cancel the dialog. If they cancel, keystore1 is skipped, and we try to pair for keystore2 now, and device2 will pair with it automatically.
fixes https://github.com/spesmilo/electrum/issues/4199#issuecomment-1112552416
I had a ledger nano S and a ledger nano S plus connected at the same time,
and the "id_"s were colliding resulting in weird behaviour. Multisig was pretty
much not usable with both devices connected simultaneously.
Example dicts returned by `hid.enumerate(0, 0)`:
{'path': b'\\\\?\\hid#vid_2c97&pid_1015&mi_00#a&2a30{REDACTED}&0&0000#{REDACTED_UUID}', 'vendor_id': 11415, 'product_id': 4117, 'serial_number': '0001', 'release_number': 513, 'manufacturer_string': 'Ledger', 'product_string': 'Nano S', 'usage_page': 65440, 'usage': 1, 'interface_number': 0},
{'path': b'\\\\?\\hid#vid_2c97&pid_5011&mi_00#a&28d{REDACTED}&0&0000#{REDACTED_UUID}', 'vendor_id': 11415, 'product_id': 20497, 'serial_number': '0001', 'release_number': 513, 'manufacturer_string': 'Ledger', 'product_string': 'Nano S Plus', 'usage_page': 65440, 'usage': 1, 'interface_number': 0}
- the mac build broke with 184e122c36
- upstream fix is at https://github.com/pyinstaller/pyinstaller/pull/6701
however, we are using an old version of pyinstaller atm, so this workaround is easier
log excerpt of failed build:
```
💬 INFO: Installing PyInstaller.
Processing ./contrib/osx/.cache/pyinstaller
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
Traceback (most recent call last):
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
self.run_setup()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 281, in run_setup
super(_BuildMetaLegacyBackend,
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 75, in <module>
setup(
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 122, in setup
dist.parse_config_files()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/dist.py", line 850, in parse_config_files
setupcfg.parse_configuration(
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 167, in parse_configuration
meta.parse()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 446, in parse
section_parser_method(section_options)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 417, in parse_section
self[name] = value
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 238, in __setitem__
value = parser(value)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 552, in _parse_version
return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 372, in _parse_attr
return expand.read_attr(attr_desc, package_dir, root_dir)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/expand.py", line 194, in read_attr
module = _load_spec(spec, module_name)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/expand.py", line 214, in _load_spec
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/user/wspace/electrum/contrib/osx/.cache/pyinstaller/PyInstaller.py", line 16, in <module>
from PyInstaller.__main__ import run
ModuleNotFoundError: No module named 'PyInstaller.__main__'; 'PyInstaller' is not a package
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```
This allows more freedom than using releases from PyPI.
(atm there is no released version that fixes https://github.com/pyinstaller/pyinstaller/pull/6701 )
Also, we now build the pyinstaller bootloader, just like in the windows build:
one fewer binary blob to trust.
- the windows build broke with 184e122c36
- upstream fix being pulled in is https://github.com/pyinstaller/pyinstaller/pull/6701
log excerpt of failed build:
```
💬 INFO: Installing PyInstaller.
Processing c:\electrum\contrib\build-wine\.cache\win32\pyinstaller
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
Traceback (most recent call last):
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 363, in <
module>
main()
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 345, in m
ain
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 164, in p
repare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 188, in prepare_metadata_for_
build_wheel
self.run_setup()
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 281, in run_setup
super(_BuildMetaLegacyBackend,
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 174, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 249, in <module>
setup(
File "C:\python3\lib\site-packages\setuptools\__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "C:\python3\lib\site-packages\setuptools\_distutils\core.py", line 122, in setup
dist.parse_config_files()
File "C:\python3\lib\site-packages\setuptools\dist.py", line 850, in parse_config_files
setupcfg.parse_configuration(
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 167, in parse_configurat
ion
meta.parse()
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 446, in parse
section_parser_method(section_options)
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 417, in parse_section
self[name] = value
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 238, in __setitem__
value = parser(value)
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 552, in _parse_version
return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 372, in _parse_attr
return expand.read_attr(attr_desc, package_dir, root_dir)
File "C:\python3\lib\site-packages\setuptools\config\expand.py", line 194, in read_attr
module = _load_spec(spec, module_name)
File "C:\python3\lib\site-packages\setuptools\config\expand.py", line 214, in _load_spec
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\electrum\contrib\build-wine\.cache\win32\pyinstaller\PyInstaller.py", line 15, in <modu
le>
from PyInstaller.__main__ import run
ModuleNotFoundError: No module named 'PyInstaller.__main__'; 'PyInstaller' is not a package
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
🗯 ERROR: prepare-wine failed
```
The old and new behaviour is as follows:
1. "pyinstaller" case: portable `.exe`, other `.exe`s with `--portable`, and `.dmg` with `--portable`
- uses `$PWD`
- note that when you double-click the portable `.exe` on Windows, `$PWD` is set to the parent folder, i.e. the datadir gets put next to the `.exe`
2. appimage `--portable`
- was broken (see https://github.com/spesmilo/electrum/issues/5551)
- (CHANGED NOW to) uses `$PWD`
3. git clone
- next to `run_electrum`
4. unpacking `tar.gz` and running locally from it
- next to `run_electrum`
5. `pip install *.tar.gz`, and calling "electrum --portable" from terminal
- used python's user script directory
- `~/.local/bin/electrum_data`
- `$VIRTUAL_ENV/bin/electrum_data`
- (CHANGED NOW to) uses `$PWD`
That is, we now almost always put the datadir in `$PWD`,
except for the local source case, where we put it next to `run_electrum`.
The "appimage" case (2) is now fixed.
The only breaking change is re case 5 which previously behaved completely
unintuitively and most likely not in a useful way.
closes https://github.com/spesmilo/electrum/issues/7732
fixes https://github.com/spesmilo/electrum/issues/5551
pyinstaller 4.3 changed the value of `__file__`.
This change makes our behaviour independent of that pyinstaller change
(we always behave like old versions of pyinstaller).
fixes https://github.com/spesmilo/electrum/issues/7729
regression was introduced by b5951adc29
- rename `is_bundle` to `is_pyinstaller` (no semantic changes, just to clearer name)
- define `is_appimage`
- add comment to explain `is_local`
- its value is the same as before (but more explicit definition)
- define `is_git_clone`, and restrict DeprecationWarnings to that case
fixes https://github.com/spesmilo/electrum/issues/7447
Consider this trace for 4.2.0:
```
Traceback (most recent call last):
File "electrum/gui/qt/__init__.py", line 332, in start_new_window
File "electrum/gui/qt/__init__.py", line 363, in _start_wizard_to_select_or_create_wallet
File "electrum/gui/qt/installwizard.py", line 302, in select_storage
File "electrum/util.py", line 504, in get_new_wallet_name
PermissionError: [Errno 1] Operation not permitted: '/Users/admin/Documents/Peach/MS'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "electrum/gui/qt/__init__.py", line 426, in main
File "electrum/gui/qt/__init__.py", line 307, in wrapper
File "electrum/gui/qt/__init__.py", line 349, in start_new_window
File "electrum/util.py", line 504, in get_new_wallet_name
PermissionError: [Errno 1] Operation not permitted: '/Users/admin/Documents/Peach/MS'
```
Note that `get_new_wallet_name` (os.listdir) can raise OSError,
and we were calling that on the main entrypoint codepath without exception-handling.
We were also calling it in the fallback codepath without exception-handling.
i.e. the GUI errored out on every startup for affected users, and without CLI usage
it was not possible to recover.
See testcase:
- imported wallet with addr1 and addr2
- three txs: tx1 funds addr1, tx2 funds addr2, tx3 spends all
- if we rm addr1 from the wallet,
- previously both tx1 and tx3 was removed (as tx3 is a child of tx1)
- now only tx1 is removed (tx3 still relates to the wallet via addr2)
fixes https://github.com/spesmilo/electrum/issues/7587
see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1072669548
-----
pyinstaller 4.2 failed (during its runtime) to create exes (worked with cpython 3.9.10, but not with cpython 3.9.11):
```
80572 INFO: Processing module hooks...
80573 INFO: Loading module hook 'hook-certifi.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
80618 INFO: Loading module hook 'hook-cryptography.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
82879 INFO: Loading module hook 'hook-dns.rdata.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84147 INFO: Loading module hook 'hook-mnemonic.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84207 INFO: Loading module hook 'hook-pycparser.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84212 INFO: Loading module hook 'hook-usb1.py' from 'C:\\python3\\lib\\site-packages\\usb1\\__pyinstaller'...
84215 INFO: --- libusb1 pyinstaller hook ---
84226 INFO: Added libusb binaries: [('C:\\python3\\lib\\site-packages\\usb1\\libusb-1.0.dll', 'usb1')]
84228 INFO: Loading module hook 'hook-difflib.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
84237 INFO: Excluding import of doctest from module difflib
84237 INFO: Loading module hook 'hook-distutils.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
Unable to find "C:\python3\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't
come with proper library files. This usually happens by missing development
package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
* apt-get install python3-dev
* apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)
🗯 ERROR: build-electrum-git failed
```
Looks like this might be fixed in pyinstaller 4.3 (we are using 4.2):
https://github.com/pyinstaller/pyinstaller/pull/5218
-----
While trying to update pyinstaller to have that fix, several issues found with versions 4.3-4.10,
now reported upstream and already fixed:
https://github.com/pyinstaller/pyinstaller/issues/6338https://github.com/pyinstaller/pyinstaller/issues/6339https://github.com/pyinstaller/pyinstaller/issues/6686
So the pyinstaller commit pinned here is from the stable "v4" branch, some commits after the 4.10 tag,
which has fixes for the above issues.
with wine 6.0.2 and 6.0.3, cpython 3.9.11 fails to install (but cpython 3.9.10 worked)
```
010c:err:virtual:virtual_setup_exception stack overflow 1220 bytes in thread 010c addr 0x7bc6713d stack 0x440b3c (0x440000-0x441000-0x640000)
🗯 ERROR: wine msiexec failed for dev.msi
🗯 ERROR: prepare-wine failed
```
-----
btw, related note:
After changing the Dockerfile, building the docker image from cache failed. Setting ELECBUILD_NOCACHE=1 fixed it:
```
E: Could not configure 'libc6:i386'.
E: Could not perform immediate configuration on 'libgcc-s1:i386'. Please see man 5 apt.conf under APT::Immediate-Configure for details. (2)
$ ELECBUILD_NOCACHE=1 ./contrib/build-wine/build.sh
```
