note: python 3.9.x is now in source-only mode, so could not update to latest...
it is time to investigate upgrading to python 3.10 in the win and mac binaries
fixes https://github.com/spesmilo/electrum/issues/8011
> Upgraded from Appimage 4.1.5 to 4.3.2 on MX-Linux 19.
> The new version fails to start with:
>
> ```
> $ electrum
> qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
> This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may > fix this problem.
>
> Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.
>
> Aborted (core dumped)
> ```
debian 10 only distributes libxcb-util0.
debian 11 distributes libxcb-util1.
our base image atm is debian 10, so we build the package from source.
Change Docker base images from Ubuntu to Debian, and use `snapshot.debian.org` as apt source list.
Ubuntu occasionally removes version-pinned packages from apt (see #7484), which
- breaks historical reproducible builds
- introduces maintenance burden as we have to update the version pins
Hopefully this change fixes both issues.
merges https://github.com/spesmilo/electrum/pull/7926
closes https://github.com/spesmilo/electrum/issues/7484
This reverts commit 0c2a885c66.
Reverting for now due to reproducibility issues:
```
error: /Users/vagrant/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate: can't open file: /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/libffi.8.dylib (No such file or directory)
```
`Electrum.app/Contents/MacOS/libffi.8.dylib` is included when building on one machine, but missing on the other...
Will have to investigate later.
related https://github.com/spesmilo/electrum/pull/7918
4.1.5->4.2.1, the appimage grew 54M->66M. This change shrinks it back to 58M.
```
$ ls -lah
total 224M
drwxrwxr-x 5 user user 4.0K Mar 27 18:18 .
drwxrwxr-x 16 user user 4.0K Mar 23 16:02 ..
-rwxrw-r-- 1 user user 48M Dec 18 2020 electrum-4.0.9-x86_64.AppImage
-rwxrw-r-- 1 user user 54M Jan 19 14:25 electrum-4.1.5-x86_64.AppImage
-rwxr-xr-x 1 user user 58M Mar 27 18:12 electrum-4.2.1-dirty-x86_64.AppImage
-rwxrw-r-- 1 user user 66M Mar 27 15:00 electrum-4.2.1-x86_64.AppImage
```
I've used the great `ncdu` tool to investigate file sizes.
```
$ du squashfs-root-415/usr/lib/python3.7/ --max-depth=1 | sort -nr | head -n8
154608 squashfs-root-415/usr/lib/python3.7/
138864 squashfs-root-415/usr/lib/python3.7/site-packages
4720 squashfs-root-415/usr/lib/python3.7/lib-dynload
1744 squashfs-root-415/usr/lib/python3.7/encodings
664 squashfs-root-415/usr/lib/python3.7/pydoc_data
460 squashfs-root-415/usr/lib/python3.7/distutils
460 squashfs-root-415/usr/lib/python3.7/asyncio
436 squashfs-root-415/usr/lib/python3.7/email
$ du squashfs-root-421/usr/lib/python3.9/ --max-depth=1 | sort -nr | head -n8
194088 squashfs-root-421/usr/lib/python3.9/
143512 squashfs-root-421/usr/lib/python3.9/site-packages
33824 squashfs-root-421/usr/lib/python3.9/config-3.9-x86_64-linux-gnu
5244 squashfs-root-421/usr/lib/python3.9/lib-dynload
1720 squashfs-root-421/usr/lib/python3.9/encodings
696 squashfs-root-421/usr/lib/python3.9/pydoc_data
520 squashfs-root-421/usr/lib/python3.9/asyncio
464 squashfs-root-421/usr/lib/python3.9/distutils
```
We should delete `usr/lib/python3.9/config-3.9-x86_64-linux-gnu/` (which is 33M unpacked)
With py3.7 (electrum 4.1.5), this folder was named `config-3.7m-x86_64-linux-gnu`,
presumably because the default config to compile py3.7 was `--with-pymalloc`,
but maybe it is not for py3.9... ? not sure. (see https://peps.python.org/pep-3149/ )
This is bumping the python versions bundled inside our binaries.
For macOS and AppImage, from 3.9.10 to 3.9.11.
For Android, from 3.8.12 to 3.8.13.
Windows is left untouched as I am having issues with the wine build when using 3.9.11.
(see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1071901116 )
This includes two logically separate changes:
- on the host, try not to require sudo when running the build scripts
- namely when interacting with the docker daemon, this requires
the unix user on the host to be part of the `docker` group
- this solves part of https://github.com/spesmilo/electrum/issues/7602
- while running inside the docker containers, do not run as root
- this means that e.g. files created in mounted folders should
no longer be owned by root on the host
- there is some code duplication involved here - not sure
how it could be deduped.
The latest release of appimagetool bundles a new enough version of
mksquashfs. We had been building a fork of mksquashfs but all the
relevant patches there had been upstreamed.
Note: we still need a wrapper when calling mksquashfs, as appimagetool
calls it with "-mkfs-time 0" and we have the SOURCE_DATE_EPOCH env var
set; and these two would conflict.
Two ways to fix: either unset SOURCE_DATE_EPOCH for that context, or
build a wrapper that removes the "-mkfs-time 0". The former would be
cleaner but for some reason I did not manage to build reproducibly
that way. The latter seems to work.
related:
- https://github.com/AppImage/AppImageKit/issues/929#issuecomment-580769875
> Now official squashfs 4.4 makes reproducible images by default
- https://github.com/AppImage/AppImageKit/pull/996
fixes#7198
We recently bumped the bundled version of PyQt (5.14.2->5.15.4).
It seems the new version has a lot more dynamic dependencies.
From https://doc.qt.io/qt-5/linux-requirements.html :
> From Qt 5.15 onwards, Qt does require libxcb 1.11. Also, the -qt-xcb
> configure option got removed that was bundling some of the libs below.
We are using the prebuilt wheels for PyQt5 from PyPI. Presumably those
had previously been built using the -qt-xcb option.
Ubuntu 20.04 needs `libxcb-xinerama.so.0`
and Debian 10 needs `libxcb-util.so.1` for the Qt `xcb` plugin.
-----
ported from 380f04a805
see https://github.com/Electron-Cash/Electron-Cash/issues/2196
-----
In particular, see this comment:
>> I confirmed that both Ubuntu 20.04 and Debian 10.8 clean install have an issue with the 4.2.4 AppImage.
>
> Any idea what broke it? I assume it used to work.
Yes, it did work in 4.2.3 which used PyQt5 5.13.2. I just had a look at the xcb plugin and it definitely has less dependencies:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.6/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007f0ec9d07000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f0ec9cdd000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007f0ec6ead000)
```
compared to PyQt5 5.15.2 in 4.2.4:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.8/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007fcb5715f000)
libxcb-icccm.so.4 => /lib/x86_64-linux-gnu/libxcb-icccm.so.4 (0x00007fcb57158000)
libxcb-image.so.0 => /lib/x86_64-linux-gnu/libxcb-image.so.0 (0x00007fcb56f51000)
libxcb-shm.so.0 => /lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fcb56f4c000)
libxcb-util.so.1 => /lib/x86_64-linux-gnu/libxcb-util.so.1 (0x00007fcb56d46000)
libxcb-keysyms.so.1 => /lib/x86_64-linux-gnu/libxcb-keysyms.so.1 (0x00007fcb56d41000)
libxcb-randr.so.0 => /lib/x86_64-linux-gnu/libxcb-randr.so.0 (0x00007fcb56d2e000)
libxcb-render-util.so.0 => /lib/x86_64-linux-gnu/libxcb-render-util.so.0 (0x00007fcb56d27000)
libxcb-render.so.0 => /lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fcb56d16000)
libxcb-shape.so.0 => /lib/x86_64-linux-gnu/libxcb-shape.so.0 (0x00007fcb56d11000)
libxcb-sync.so.1 => /lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007fcb56d07000)
libxcb-xfixes.so.0 => /lib/x86_64-linux-gnu/libxcb-xfixes.so.0 (0x00007fcb56cfd000)
libxcb-xinerama.so.0 => /lib/x86_64-linux-gnu/libxcb-xinerama.so.0 (0x00007fcb56cf8000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007fcb56cda000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fcb56cae000)
```
