4.1.5->4.2.1, the appimage grew 54M->66M. This change shrinks it back to 58M.
```
$ ls -lah
total 224M
drwxrwxr-x 5 user user 4.0K Mar 27 18:18 .
drwxrwxr-x 16 user user 4.0K Mar 23 16:02 ..
-rwxrw-r-- 1 user user 48M Dec 18 2020 electrum-4.0.9-x86_64.AppImage
-rwxrw-r-- 1 user user 54M Jan 19 14:25 electrum-4.1.5-x86_64.AppImage
-rwxr-xr-x 1 user user 58M Mar 27 18:12 electrum-4.2.1-dirty-x86_64.AppImage
-rwxrw-r-- 1 user user 66M Mar 27 15:00 electrum-4.2.1-x86_64.AppImage
```
I've used the great `ncdu` tool to investigate file sizes.
```
$ du squashfs-root-415/usr/lib/python3.7/ --max-depth=1 | sort -nr | head -n8
154608 squashfs-root-415/usr/lib/python3.7/
138864 squashfs-root-415/usr/lib/python3.7/site-packages
4720 squashfs-root-415/usr/lib/python3.7/lib-dynload
1744 squashfs-root-415/usr/lib/python3.7/encodings
664 squashfs-root-415/usr/lib/python3.7/pydoc_data
460 squashfs-root-415/usr/lib/python3.7/distutils
460 squashfs-root-415/usr/lib/python3.7/asyncio
436 squashfs-root-415/usr/lib/python3.7/email
$ du squashfs-root-421/usr/lib/python3.9/ --max-depth=1 | sort -nr | head -n8
194088 squashfs-root-421/usr/lib/python3.9/
143512 squashfs-root-421/usr/lib/python3.9/site-packages
33824 squashfs-root-421/usr/lib/python3.9/config-3.9-x86_64-linux-gnu
5244 squashfs-root-421/usr/lib/python3.9/lib-dynload
1720 squashfs-root-421/usr/lib/python3.9/encodings
696 squashfs-root-421/usr/lib/python3.9/pydoc_data
520 squashfs-root-421/usr/lib/python3.9/asyncio
464 squashfs-root-421/usr/lib/python3.9/distutils
```
We should delete `usr/lib/python3.9/config-3.9-x86_64-linux-gnu/` (which is 33M unpacked)
With py3.7 (electrum 4.1.5), this folder was named `config-3.7m-x86_64-linux-gnu`,
presumably because the default config to compile py3.7 was `--with-pymalloc`,
but maybe it is not for py3.9... ? not sure. (see https://peps.python.org/pep-3149/ )
This is bumping the python versions bundled inside our binaries.
For macOS and AppImage, from 3.9.10 to 3.9.11.
For Android, from 3.8.12 to 3.8.13.
Windows is left untouched as I am having issues with the wine build when using 3.9.11.
(see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1071901116 )
This includes two logically separate changes:
- on the host, try not to require sudo when running the build scripts
- namely when interacting with the docker daemon, this requires
the unix user on the host to be part of the `docker` group
- this solves part of https://github.com/spesmilo/electrum/issues/7602
- while running inside the docker containers, do not run as root
- this means that e.g. files created in mounted folders should
no longer be owned by root on the host
- there is some code duplication involved here - not sure
how it could be deduped.
The latest release of appimagetool bundles a new enough version of
mksquashfs. We had been building a fork of mksquashfs but all the
relevant patches there had been upstreamed.
Note: we still need a wrapper when calling mksquashfs, as appimagetool
calls it with "-mkfs-time 0" and we have the SOURCE_DATE_EPOCH env var
set; and these two would conflict.
Two ways to fix: either unset SOURCE_DATE_EPOCH for that context, or
build a wrapper that removes the "-mkfs-time 0". The former would be
cleaner but for some reason I did not manage to build reproducibly
that way. The latter seems to work.
related:
- https://github.com/AppImage/AppImageKit/issues/929#issuecomment-580769875
> Now official squashfs 4.4 makes reproducible images by default
- https://github.com/AppImage/AppImageKit/pull/996
fixes#7198
We recently bumped the bundled version of PyQt (5.14.2->5.15.4).
It seems the new version has a lot more dynamic dependencies.
From https://doc.qt.io/qt-5/linux-requirements.html :
> From Qt 5.15 onwards, Qt does require libxcb 1.11. Also, the -qt-xcb
> configure option got removed that was bundling some of the libs below.
We are using the prebuilt wheels for PyQt5 from PyPI. Presumably those
had previously been built using the -qt-xcb option.
Ubuntu 20.04 needs `libxcb-xinerama.so.0`
and Debian 10 needs `libxcb-util.so.1` for the Qt `xcb` plugin.
-----
ported from 380f04a805
see https://github.com/Electron-Cash/Electron-Cash/issues/2196
-----
In particular, see this comment:
>> I confirmed that both Ubuntu 20.04 and Debian 10.8 clean install have an issue with the 4.2.4 AppImage.
>
> Any idea what broke it? I assume it used to work.
Yes, it did work in 4.2.3 which used PyQt5 5.13.2. I just had a look at the xcb plugin and it definitely has less dependencies:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.6/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007f0ec9d07000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f0ec9cdd000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007f0ec6ead000)
```
compared to PyQt5 5.15.2 in 4.2.4:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.8/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007fcb5715f000)
libxcb-icccm.so.4 => /lib/x86_64-linux-gnu/libxcb-icccm.so.4 (0x00007fcb57158000)
libxcb-image.so.0 => /lib/x86_64-linux-gnu/libxcb-image.so.0 (0x00007fcb56f51000)
libxcb-shm.so.0 => /lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fcb56f4c000)
libxcb-util.so.1 => /lib/x86_64-linux-gnu/libxcb-util.so.1 (0x00007fcb56d46000)
libxcb-keysyms.so.1 => /lib/x86_64-linux-gnu/libxcb-keysyms.so.1 (0x00007fcb56d41000)
libxcb-randr.so.0 => /lib/x86_64-linux-gnu/libxcb-randr.so.0 (0x00007fcb56d2e000)
libxcb-render-util.so.0 => /lib/x86_64-linux-gnu/libxcb-render-util.so.0 (0x00007fcb56d27000)
libxcb-render.so.0 => /lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fcb56d16000)
libxcb-shape.so.0 => /lib/x86_64-linux-gnu/libxcb-shape.so.0 (0x00007fcb56d11000)
libxcb-sync.so.1 => /lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007fcb56d07000)
libxcb-xfixes.so.0 => /lib/x86_64-linux-gnu/libxcb-xfixes.so.0 (0x00007fcb56cfd000)
libxcb-xinerama.so.0 => /lib/x86_64-linux-gnu/libxcb-xinerama.so.0 (0x00007fcb56cf8000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007fcb56cda000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fcb56cae000)
```
Had to also bump ubuntu version 18.04->20.04,
as was getting errors running the self-compiled pyinstaller otherwise (weird...):
```
from .utils.git import get_repo_revision
ModuleNotFoundError: No module named 'PyInstaller.utils'
```
(similar to https://github.com/pyinstaller/pyinstaller/issues/4403 )
-----
Note re appimage: when trying to compile python 3.8.x on ubuntu 16.04, I am getting:
./Modules/posixmodule.c: In function ‘os_copy_file_range_impl’:
./Modules/posixmodule.c:10351:15: error: implicit declaration of function ‘copy_file_range’ [-Werror=implicit-function-declaration]
ret = copy_file_range(src, p_offset_src, dst, p_offset_dst, count, flags);
This is because ubuntu 16.04 has too old glibc.
If there is a collision between a branch name and a commit hash, git
will choose the branch, even if the full 40-hex-long commit hash is
given. GitHub disallows branches/tags with such a name but git itself
does not. By adding the `^{commit}` syntax sugar after a ref name,
we can tell git that we want the commit hash to be preferred,
and hence we don't need to trust GitHub (only git).
see https://security.stackexchange.com/questions/225411/
