- `bitstring` started depending on `bitarray` in version 4.1 [0]
- that would mean one additional dependency for us (from yet another maintainer), which is not even pure python
- we only use bitstring for bolt11-parsing
- hence this PR rewrites the bolt11-parsing and removes `bitstring` as dependency
- note: I benchmarked lndecode using [1], and the new code performs better,
taking around 80% time needed for old code (when using bitstring 3.1.9, pure python).
Though the variance is quite large in both cases.
[0]: 95ee533ee4/release_notes.txt (L108)
[1]: d7597d96d0
does not work with current beta:
```
d142c5534f4da4a7f0a1f52676716908f674ff9e06b87f7b153384b161296eaf /home/user/wspace/electrum/contrib/android/../../dist/Electrum-4.5.0.0-armeabi-v7a-release-unsigned.apk
+ mv dist/Electrum-4.5.0b0.0-armeabi-v7a-release-unsigned.apk dist/Electrum-4.5.0b0.0-armeabi-v7a-release.apk
mv: cannot stat 'dist/Electrum-4.5.0b0.0-armeabi-v7a-release-unsigned.apk': No such file or directory
```
- For the notarisation, we were using "altool" (part of xcode), but Apple now fully deprecated that.
- need to migrate from altool to notarytool
https://developer.apple.com/news/?id=y5mjxqmn
- currently using macOS 10.14.6
and xcode 11.3.1
- notarytool requires xcode 13+
- xcode 13 requires macOS 11.3
https://en.wikipedia.org/wiki/Xcode#Version_comparison_table
==> created new build VM with macOS 11.7.10 and xcode 13.2
With newer xcode (12+, I think), the default behaviour is to build
universal binaries that include native code for both arm64 and x86_64.
On older xcode, the default was to only target x86_64.
I am not sure why, but "hid.cpython-310-darwin.so" is not built
reproducibly when it is "universal".
(Even for consecutive builds done on the same machine.)
```
- + diff='Files /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/hid.cpython-310-darwin.so and /tmp/electrum_compare_dmg/dmg2/Electrum.app/Contents/MacOS/hid.cpython-310-darwin.so differ
```
This commit works around the reproducibly issue by making clang only
build for the x86_64 target. The binary should still be able to run
on arm64 macs using rosetta (same with previous versions of Electrum).
-----
The `lipo` tool can tell what archs a shared object is built for, e.g.:
```
$ lipo -info /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so
Non-fat file: /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so is architecture: x86_64
```
For quick testing, we don't need to build the whole .app, only hid.so:
```
source /Users/vagrant/electrum/contrib/osx/build-venv/bin/activate
export CFLAGS="-g0"
python3 -m pip install -I --no-build-isolation --no-dependencies --no-binary :all: --no-warn-script-location hidapi==0.14.0
strip -x /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so
cp /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so /Users/vagrant/wspace/tmp/try7/hid10.so
```
Re ARCHFLAGS env var, see https://stackoverflow.com/a/5808548
-----
Note: I've found several xcode build settings that looked relevant
but in the end were not useful, e.g. ARCHS, ONLY_ACTIVE_ARCH,
EXCLUDED_ARCHS, VALID_ARCHS.
- see https://developer.apple.com/documentation/technotes/tn3117-resolving-build-errors-for-apple-silicon
- see https://stackoverflow.com/a/64422757
This should fix an issue when running with python 3.11 (possibly only 3.11.5<= ).
```
47.45 | I | exchange_rate.CoinGecko | getting fx quotes for EUR
48.18 | E | exchange_rate.CoinGecko | failed fx quotes: ClientOSError('Cannot write to closing transport')
Traceback (most recent call last):
File "...\electrum\env11\Lib\site-packages\aiohttp\client.py", line 599, in _request
resp = await req.send(conn)
^^^^^^^^^^^^^^^^^^^^
File "...\electrum\env11\Lib\site-packages\aiohttp\client_reqrep.py", line 712, in send
await writer.write_headers(status_line, self.headers)
File "...\electrum\env11\Lib\site-packages\aiohttp\http_writer.py", line 130, in write_headers
self._write(buf)
File "...\electrum\env11\Lib\site-packages\aiohttp\http_writer.py", line 75, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\electrum\electrum\exchange_rate.py", line 85, in update_safe
self._quotes = await self.get_rates(ccy)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "...\electrum\electrum\exchange_rate.py", line 345, in get_rates
json = await self.get_json('api.coingecko.com', '/api/v3/exchange_rates')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...\electrum\electrum\exchange_rate.py", line 69, in get_json
async with session.get(url) as response:
File "...\electrum\env11\Lib\site-packages\aiohttp\client.py", line 1187, in __aenter__
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File "...\electrum\env11\Lib\site-packages\aiohttp\client.py", line 613, in _request
raise ClientOSError(*exc.args) from exc
aiohttp.client_exceptions.ClientOSError: Cannot write to closing transport
```
related:
https://github.com/romis2012/aiohttp-socks/issues/27https://github.com/python/cpython/issues/109321
Ah ok, I give up for now... the prev does not really work.
The prior commit works on Windows but not on macOS.
On Windows, it would package all plugins as code and only as code.
On MacOS however, it would not package any plugins at all. And with this commit,
where I mark the plugins folder to be packaged as *data*, it packages all plugins as *both* code and data.
Not sure why.
Let's just package all plugins as both code and data, and ignore the code instances explicitly...
The hw_wallet and jade plugins were being included twice in the pyinstaller binaries.
They were apparently the only two plugins being picked as "modules" (a.pure),
which by default are included as compiled-bytecode.
In addition, we included "electrum/plugins" as data (a.data), which meant all
plugins in source form.
Instead of hacking around to fix the specific issue, this attempts a larger clean-up.
```
602 WARNING: Failed to collect submodules for 'pkg_resources._vendor.pyparsing.diagram' because importing 'pkg_resources._vendor.pyparsing.diagram' raised: ModuleNotFoundError: No module named 'railroad'
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
5921 WARNING: collect_data_files - skipping data collection for module 'electrum.plugins' as it is not a package.
```
pyinstaller tries to import electrum and its different submodules at build-time
during the "Analysis" phase. sys._GUI_QT_VERSION was not getting set there,
and the resulting exception was blocking pyinstaller from discovering that
gui/common_qt is being used.
at runtime:
```
$ ./dist/Electrum.app/Contents/MacOS/run_electrum
1.53 | E | daemon.Daemon | GUI raised exception: Exception('Error loading trustedcoin plugin: ModuleNotFoundError("No module named \'electrum.gui.common_qt\'")'). shutting down.
1.53 | E | __main__ | daemon.run_gui errored
Traceback (most recent call last):
File "electrum/plugin.py", line 135, in load_plugin
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/Users/vagrant/electrum/dist/Electrum.app/Contents/MacOS/electrum/plugins/trustedcoin/qt.py", line 51, in <module>
from .common_qt import TrustedcoinPluginQObject
File "/Users/vagrant/electrum/dist/Electrum.app/Contents/MacOS/electrum/plugins/trustedcoin/common_qt.py", line 16, in <module>
from electrum.gui.common_qt.plugins import PluginQObject
ModuleNotFoundError: No module named 'electrum.gui.common_qt'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "run_electrum", line 456, in handle_cmd
d.run_gui()
File "electrum/daemon.py", line 617, in run_gui
self.gui_object = gui.ElectrumGui(config=self.config, daemon=self, plugins=self._plugins)
File "electrum/util.py", line 473, in do_profile
o = func(*args, **kw_args)
File "electrum/gui/qt/__init__.py", line 153, in __init__
self.plugins.load_plugin('trustedcoin')
File "electrum/plugin.py", line 138, in load_plugin
raise Exception(f"Error loading {name} plugin: {repr(e)}") from e
Exception: Error loading trustedcoin plugin: ModuleNotFoundError("No module named 'electrum.gui.common_qt'")
```
