The old and new behaviour is as follows:
1. "pyinstaller" case: portable `.exe`, other `.exe`s with `--portable`, and `.dmg` with `--portable`
- uses `$PWD`
- note that when you double-click the portable `.exe` on Windows, `$PWD` is set to the parent folder, i.e. the datadir gets put next to the `.exe`
2. appimage `--portable`
- was broken (see https://github.com/spesmilo/electrum/issues/5551)
- (CHANGED NOW to) uses `$PWD`
3. git clone
- next to `run_electrum`
4. unpacking `tar.gz` and running locally from it
- next to `run_electrum`
5. `pip install *.tar.gz`, and calling "electrum --portable" from terminal
- used python's user script directory
- `~/.local/bin/electrum_data`
- `$VIRTUAL_ENV/bin/electrum_data`
- (CHANGED NOW to) uses `$PWD`
That is, we now almost always put the datadir in `$PWD`,
except for the local source case, where we put it next to `run_electrum`.
The "appimage" case (2) is now fixed.
The only breaking change is re case 5 which previously behaved completely
unintuitively and most likely not in a useful way.
closes https://github.com/spesmilo/electrum/issues/7732
fixes https://github.com/spesmilo/electrum/issues/5551
pyinstaller 4.3 changed the value of `__file__`.
This change makes our behaviour independent of that pyinstaller change
(we always behave like old versions of pyinstaller).
fixes https://github.com/spesmilo/electrum/issues/7729
regression was introduced by b5951adc29
- rename `is_bundle` to `is_pyinstaller` (no semantic changes, just to clearer name)
- define `is_appimage`
- add comment to explain `is_local`
- its value is the same as before (but more explicit definition)
- define `is_git_clone`, and restrict DeprecationWarnings to that case
fixes https://github.com/spesmilo/electrum/issues/7447
Consider this trace for 4.2.0:
```
Traceback (most recent call last):
File "electrum/gui/qt/__init__.py", line 332, in start_new_window
File "electrum/gui/qt/__init__.py", line 363, in _start_wizard_to_select_or_create_wallet
File "electrum/gui/qt/installwizard.py", line 302, in select_storage
File "electrum/util.py", line 504, in get_new_wallet_name
PermissionError: [Errno 1] Operation not permitted: '/Users/admin/Documents/Peach/MS'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "electrum/gui/qt/__init__.py", line 426, in main
File "electrum/gui/qt/__init__.py", line 307, in wrapper
File "electrum/gui/qt/__init__.py", line 349, in start_new_window
File "electrum/util.py", line 504, in get_new_wallet_name
PermissionError: [Errno 1] Operation not permitted: '/Users/admin/Documents/Peach/MS'
```
Note that `get_new_wallet_name` (os.listdir) can raise OSError,
and we were calling that on the main entrypoint codepath without exception-handling.
We were also calling it in the fallback codepath without exception-handling.
i.e. the GUI errored out on every startup for affected users, and without CLI usage
it was not possible to recover.
See testcase:
- imported wallet with addr1 and addr2
- three txs: tx1 funds addr1, tx2 funds addr2, tx3 spends all
- if we rm addr1 from the wallet,
- previously both tx1 and tx3 was removed (as tx3 is a child of tx1)
- now only tx1 is removed (tx3 still relates to the wallet via addr2)
fixes https://github.com/spesmilo/electrum/issues/7587
see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1072669548
-----
pyinstaller 4.2 failed (during its runtime) to create exes (worked with cpython 3.9.10, but not with cpython 3.9.11):
```
80572 INFO: Processing module hooks...
80573 INFO: Loading module hook 'hook-certifi.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
80618 INFO: Loading module hook 'hook-cryptography.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
82879 INFO: Loading module hook 'hook-dns.rdata.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84147 INFO: Loading module hook 'hook-mnemonic.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84207 INFO: Loading module hook 'hook-pycparser.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84212 INFO: Loading module hook 'hook-usb1.py' from 'C:\\python3\\lib\\site-packages\\usb1\\__pyinstaller'...
84215 INFO: --- libusb1 pyinstaller hook ---
84226 INFO: Added libusb binaries: [('C:\\python3\\lib\\site-packages\\usb1\\libusb-1.0.dll', 'usb1')]
84228 INFO: Loading module hook 'hook-difflib.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
84237 INFO: Excluding import of doctest from module difflib
84237 INFO: Loading module hook 'hook-distutils.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
Unable to find "C:\python3\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't
come with proper library files. This usually happens by missing development
package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
* apt-get install python3-dev
* apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)
🗯 ERROR: build-electrum-git failed
```
Looks like this might be fixed in pyinstaller 4.3 (we are using 4.2):
https://github.com/pyinstaller/pyinstaller/pull/5218
-----
While trying to update pyinstaller to have that fix, several issues found with versions 4.3-4.10,
now reported upstream and already fixed:
https://github.com/pyinstaller/pyinstaller/issues/6338https://github.com/pyinstaller/pyinstaller/issues/6339https://github.com/pyinstaller/pyinstaller/issues/6686
So the pyinstaller commit pinned here is from the stable "v4" branch, some commits after the 4.10 tag,
which has fixes for the above issues.
with wine 6.0.2 and 6.0.3, cpython 3.9.11 fails to install (but cpython 3.9.10 worked)
```
010c:err:virtual:virtual_setup_exception stack overflow 1220 bytes in thread 010c addr 0x7bc6713d stack 0x440b3c (0x440000-0x441000-0x640000)
🗯 ERROR: wine msiexec failed for dev.msi
🗯 ERROR: prepare-wine failed
```
-----
btw, related note:
After changing the Dockerfile, building the docker image from cache failed. Setting ELECBUILD_NOCACHE=1 fixed it:
```
E: Could not configure 'libc6:i386'.
E: Could not perform immediate configuration on 'libgcc-s1:i386'. Please see man 5 apt.conf under APT::Immediate-Configure for details. (2)
$ ELECBUILD_NOCACHE=1 ./contrib/build-wine/build.sh
```
This is bumping the python versions bundled inside our binaries.
For macOS and AppImage, from 3.9.10 to 3.9.11.
For Android, from 3.8.12 to 3.8.13.
Windows is left untouched as I am having issues with the wine build when using 3.9.11.
(see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1071901116 )
- rm the `_get_channel_ids` abstraction as each of its usages needs subtle differences.
Some code duplication is preferable in this case.
- raise exceptions in `wait_for_message`, so that callers such as the GUI can show user-feedback
- on_error/on_warning were dropping messages with temp_chan_ids if they were not stored in
`temp_id_to_id` - which was only done once the mapping was known (so the normal chan_id was known).
To fix this, we now store temp_chan_ids into `temp_id_to_id` early.
- `schedule_force_closing` only works if the chan_id is already in `channels`
related:
https://github.com/spesmilo/electrum/pull/7645 (and related commits)
-----
example before commit:
```
D/P | lnpeer.Peer.[LNWallet, 03933884aa-3b53e4ab] | Sending OPEN_CHANNEL
D/P | lnpeer.Peer.[LNWallet, 03933884aa-3b53e4ab] | Received ERROR
I/P | lnpeer.Peer.[LNWallet, 03933884aa-3b53e4ab] | remote peer sent error [DO NOT TRUST THIS MESSAGE]: invalid funding_satoshis=10000 sat (min=400000 sat max=1500000000 sat)
E | gui.qt.main_window.[test_segwit_2] | Could not open channel
Traceback (most recent call last):
File "...\electrum\electrum\util.py", line 1160, in wrapper
return await func(*args, **kwargs)
File "...\electrum\electrum\lnpeer.py", line 661, in wrapper
return await func(self, *args, **kwargs)
File "...\electrum\electrum\lnpeer.py", line 742, in channel_establishment_flow
payload = await self.wait_for_message('accept_channel', temp_channel_id) #
File "...\electrum\electrum\lnpeer.py", line 315, in wait_for_message
name, payload = await asyncio.wait_for(q.get(), LN_P2P_NETWORK_TIMEOUT)
File "...\Python39\lib\asyncio\tasks.py", line 468, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "...\Python39\lib\asyncio\tasks.py", line 492, in wait_for
fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\electrum\electrum\gui\qt\util.py", line 914, in run
result = task.task()
File "...\electrum\electrum\gui\qt\main_window.py", line 1875, in task
return self.wallet.lnworker.open_channel(
File "...\electrum\electrum\lnworker.py", line 1075, in open_channel
chan, funding_tx = fut.result()
File "...\Python39\lib\concurrent\futures\_base.py", line 445, in result
return self.__get_result()
File "...\Python39\lib\concurrent\futures\_base.py", line 390, in __get_result
raise self._exception
File "...\electrum\electrum\util.py", line 1160, in wrapper
return await func(*args, **kwargs)
File "...\electrum\electrum\lnworker.py", line 1006, in _open_channel_coroutine
chan, funding_tx = await asyncio.wait_for(coro, LN_P2P_NETWORK_TIMEOUT)
File "...\Python39\lib\asyncio\tasks.py", line 494, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
```
example after commit:
```
D/P | lnpeer.Peer.[LNWallet, 03933884aa-ff3a866f] | Sending OPEN_CHANNEL
D/P | lnpeer.Peer.[LNWallet, 03933884aa-ff3a866f] | Received ERROR
I/P | lnpeer.Peer.[LNWallet, 03933884aa-ff3a866f] | remote peer sent error [DO NOT TRUST THIS MESSAGE]: invalid funding_satoshis=10000 sat (min=400000 sat max=1500000000 sat). chan_id=124ca21fa6aa2993430ad71f465f0d44731ef87f7478e4b31327e4459b5a3988
E | lnworker.LNWallet.[test_segwit_2] | Exception in _open_channel_coroutine: GracefulDisconnect('remote peer sent error [DO NOT TRUST THIS MESSAGE]: invalid funding_satoshis=10000 sat (min=400000 sat max=1500000000 sat)')
Traceback (most recent call last):
File "...\electrum\electrum\util.py", line 1160, in wrapper
return await func(*args, **kwargs)
File "...\electrum\electrum\lnworker.py", line 1006, in _open_channel_coroutine
chan, funding_tx = await asyncio.wait_for(coro, LN_P2P_NETWORK_TIMEOUT)
File "...\Python39\lib\asyncio\tasks.py", line 481, in wait_for
return fut.result()
File "...\electrum\electrum\lnpeer.py", line 673, in wrapper
return await func(self, *args, **kwargs)
File "...\electrum\electrum\lnpeer.py", line 755, in channel_establishment_flow
payload = await self.wait_for_message('accept_channel', temp_channel_id)
File "...\electrum\electrum\lnpeer.py", line 326, in wait_for_message
raise GracefulDisconnect(
electrum.interface.GracefulDisconnect: remote peer sent error [DO NOT TRUST THIS MESSAGE]: invalid funding_satoshis=10000 sat (min=400000 sat max=1500000000 sat)
I/P | lnpeer.Peer.[LNWallet, 03933884aa-ff3a866f] | Disconnecting: GracefulDisconnect()
```
Which means we need *both* xcode cli tools and full xcode.
Document this weirdness, including the exact paths they should be at.
excerpt from terminal when running make_osx:
```
Warning: You are using macOS 10.14.
We (and Apple) do not provide support for this old version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels. You are responsible for resolving
any issues you experience while you are running this
old version.
==> Downloading https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.xz
Already downloaded: /Users/vagrant/Library/Caches/Homebrew/downloads/5744bb33344b6180adca9d909a87e830d55982a1b3229c61e9dc0e35cfacbf25--coreutils-9.0.tar.xz
Error: Xcode alone is not sufficient on Mojave.
Install the Command Line Tools for Xcode 11.3.1 from:
https://developer.apple.com/download/all/
```
Some exceptions were just killing the gui silently and not even logged.
E.g.:
```
E | gui.qt.ElectrumGui | error loading wallet (or creating window for it)
Traceback (most recent call last):
File "/opt/electrum/electrum/gui/qt/__init__.py", line 433, in main
if not self.start_new_window(path, self.config.get('url'), app_is_starting=True):
File "/opt/electrum/electrum/gui/qt/__init__.py", line 307, in wrapper
return func(self, *args, **kwargs)
File "/opt/electrum/electrum/gui/qt/__init__.py", line 332, in start_new_window
wallet = self._start_wizard_to_select_or_create_wallet(path)
File "/opt/electrum/electrum/gui/qt/__init__.py", line 377, in _start_wizard_to_select_or_create_wallet
db = WalletDB(storage.read(), manual_upgrades=False)
File "/opt/electrum/electrum/wallet_db.py", line 73, in __init__
self.load_data(raw)
File "/opt/electrum/electrum/wallet_db.py", line 104, in load_data
self._after_upgrade_tasks()
File "/opt/electrum/electrum/wallet_db.py", line 202, in _after_upgrade_tasks
self._load_transactions()
File "/opt/electrum/electrum/util.py", line 439, in <lambda>
return lambda *args, **kw_args: do_profile(args, kw_args)
File "/opt/electrum/electrum/util.py", line 435, in do_profile
o = func(*args, **kw_args)
File "/opt/electrum/electrum/wallet_db.py", line 1310, in _load_transactions
self.data = StoredDict(self.data, self, [])
File "/opt/electrum/electrum/json_db.py", line 79, in __init__
self.__setitem__(k, v)
File "/opt/electrum/electrum/json_db.py", line 44, in wrapper
return func(self, *args, **kwargs)
File "/opt/electrum/electrum/json_db.py", line 97, in __setitem__
v = self.db._convert_dict(self.path, key, v)
File "/opt/electrum/electrum/wallet_db.py", line 1361, in _convert_dict
v = dict((k, SwapData(**x)) for k, x in v.items())
```
If the user had `config.get("show_crash_reporter")==False`, they would
never get to see excs collected via `send_exception_to_crash_reporter(exc)`.
E.g.:
```
E | gui.qt.ElectrumGui | error loading wallet (or creating window for it)
Traceback (most recent call last):
File "/opt/electrum/electrum/gui/qt/__init__.py", line 433, in main
if not self.start_new_window(path, self.config.get('url'), app_is_starting=True):
File "/opt/electrum/electrum/gui/qt/__init__.py", line 307, in wrapper
return func(self, *args, **kwargs)
File "/opt/electrum/electrum/gui/qt/__init__.py", line 332, in start_new_window
wallet = self._start_wizard_to_select_or_create_wallet(path)
File "/opt/electrum/electrum/gui/qt/__init__.py", line 377, in _start_wizard_to_select_or_create_wallet
db = WalletDB(storage.read(), manual_upgrades=False)
File "/opt/electrum/electrum/wallet_db.py", line 73, in __init__
self.load_data(raw)
File "/opt/electrum/electrum/wallet_db.py", line 104, in load_data
self._after_upgrade_tasks()
File "/opt/electrum/electrum/wallet_db.py", line 202, in _after_upgrade_tasks
self._load_transactions()
File "/opt/electrum/electrum/util.py", line 439, in <lambda>
return lambda *args, **kw_args: do_profile(args, kw_args)
File "/opt/electrum/electrum/util.py", line 435, in do_profile
o = func(*args, **kw_args)
File "/opt/electrum/electrum/wallet_db.py", line 1310, in _load_transactions
self.data = StoredDict(self.data, self, [])
File "/opt/electrum/electrum/json_db.py", line 79, in __init__
self.__setitem__(k, v)
File "/opt/electrum/electrum/json_db.py", line 44, in wrapper
return func(self, *args, **kwargs)
File "/opt/electrum/electrum/json_db.py", line 97, in __setitem__
v = self.db._convert_dict(self.path, key, v)
File "/opt/electrum/electrum/wallet_db.py", line 1361, in _convert_dict
v = dict((k, SwapData(**x)) for k, x in v.items())
```
This is the new minimum the google play store requires.
note: the newer android command-line tools use a tiny bit different paths,
hence the `mv "${ANDROID_SDK_HOME}/cmdline-tools" "${ANDROID_SDK_HOME}/tools"` rename
see https://github.com/kivy/python-for-android/issues/2540
