from https://stackoverflow.com/a/69088164
I am trying to run the build.sh scripts directly on a cirrus CI runner, and "docker run -it" is erroring due to lack of a tty.
Updates the appimage build scripts to use the newer
https://github.com/AppImage/appimagetool tool to bundle the appimage
instead of the discontinued https://github.com/AppImage/AppImageKit.
To prevent the new appimagetool from downloading a random "latest"
appimage runtime (`type2-runtime`) binary this PR also adds
functionality to clone and build
https://github.com/AppImage/type2-runtime from source. This is done
using the build scripts provided in the `type2-runtime` repository,
however the Dockerfile they use for building is replaced by a copy with
pinned package versions to prevent issues with reproducibility.
This should fix the issue of missing libfuse2 which users of the appimage
have on "modern" distributions.
The new `type2-runtime` is statically linked and includes the required
dependencies now instead of relying on the host to provide it.
- repro builds to use fixed uid=1000 inside the container
- in case the file permissions leak into the binaries, they are still reproducible
- chown 1000:1000 fresh_clone
- repro builds to create fresh_clone dir outside git clone
- otherwise the local dev build would still interact with the fresh_clone dir
- due to e.g. recursive "find -exec touch",
- and even the "docker build" cmd itself would try to stat/read it
- see https://github.com/docker/for-linux/issues/380
- and "rm -rf fresh_clone" needs sudo if the host uid is not 1000
- this way the local dev build does not need sudo
to recap:
- local dev builds use the host userid inside the container, directly operate on the project dir
- does not need sudo
- repro builds create a fresh git clone, chown it to 1000, and use userid=1000 inside the container
- if the host userid is 1000, does not need sudo
- otherwise, needs sudo
closes https://github.com/spesmilo/electrum/issues/8261
reproducibility probably needs a hardcoded userid
Also, move the UID arg later in the dockerfiles, for better caching.
(if local dev build and repro build set different UIDs, the build caches
will diverge at that step)
This includes two logically separate changes:
- on the host, try not to require sudo when running the build scripts
- namely when interacting with the docker daemon, this requires
the unix user on the host to be part of the `docker` group
- this solves part of https://github.com/spesmilo/electrum/issues/7602
- while running inside the docker containers, do not run as root
- this means that e.g. files created in mounted folders should
no longer be owned by root on the host
- there is some code duplication involved here - not sure
how it could be deduped.
Ubuntu 20.04 needs `libxcb-xinerama.so.0`
and Debian 10 needs `libxcb-util.so.1` for the Qt `xcb` plugin.
-----
ported from 380f04a805
see https://github.com/Electron-Cash/Electron-Cash/issues/2196
-----
In particular, see this comment:
>> I confirmed that both Ubuntu 20.04 and Debian 10.8 clean install have an issue with the 4.2.4 AppImage.
>
> Any idea what broke it? I assume it used to work.
Yes, it did work in 4.2.3 which used PyQt5 5.13.2. I just had a look at the xcb plugin and it definitely has less dependencies:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.6/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007f0ec9d07000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f0ec9cdd000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007f0ec6ead000)
```
compared to PyQt5 5.15.2 in 4.2.4:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.8/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007fcb5715f000)
libxcb-icccm.so.4 => /lib/x86_64-linux-gnu/libxcb-icccm.so.4 (0x00007fcb57158000)
libxcb-image.so.0 => /lib/x86_64-linux-gnu/libxcb-image.so.0 (0x00007fcb56f51000)
libxcb-shm.so.0 => /lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fcb56f4c000)
libxcb-util.so.1 => /lib/x86_64-linux-gnu/libxcb-util.so.1 (0x00007fcb56d46000)
libxcb-keysyms.so.1 => /lib/x86_64-linux-gnu/libxcb-keysyms.so.1 (0x00007fcb56d41000)
libxcb-randr.so.0 => /lib/x86_64-linux-gnu/libxcb-randr.so.0 (0x00007fcb56d2e000)
libxcb-render-util.so.0 => /lib/x86_64-linux-gnu/libxcb-render-util.so.0 (0x00007fcb56d27000)
libxcb-render.so.0 => /lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fcb56d16000)
libxcb-shape.so.0 => /lib/x86_64-linux-gnu/libxcb-shape.so.0 (0x00007fcb56d11000)
libxcb-sync.so.1 => /lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007fcb56d07000)
libxcb-xfixes.so.0 => /lib/x86_64-linux-gnu/libxcb-xfixes.so.0 (0x00007fcb56cfd000)
libxcb-xinerama.so.0 => /lib/x86_64-linux-gnu/libxcb-xinerama.so.0 (0x00007fcb56cf8000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007fcb56cda000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fcb56cae000)
```
Had to also bump ubuntu version 18.04->20.04,
as was getting errors running the self-compiled pyinstaller otherwise (weird...):
```
from .utils.git import get_repo_revision
ModuleNotFoundError: No module named 'PyInstaller.utils'
```
(similar to https://github.com/pyinstaller/pyinstaller/issues/4403 )
-----
Note re appimage: when trying to compile python 3.8.x on ubuntu 16.04, I am getting:
./Modules/posixmodule.c: In function ‘os_copy_file_range_impl’:
./Modules/posixmodule.c:10351:15: error: implicit declaration of function ‘copy_file_range’ [-Werror=implicit-function-declaration]
ret = copy_file_range(src, p_offset_src, dst, p_offset_dst, count, flags);
This is because ubuntu 16.04 has too old glibc.
If there is a collision between a branch name and a commit hash, git
will choose the branch, even if the full 40-hex-long commit hash is
given. GitHub disallows branches/tags with such a name but git itself
does not. By adding the `^{commit}` syntax sugar after a ref name,
we can tell git that we want the commit hash to be preferred,
and hence we don't need to trust GitHub (only git).
see https://security.stackexchange.com/questions/225411/
All (incl indirect) dependencies are already listed in deterministic-build/requirements*.txt.
This option makes it easier to manually rm a dependency from that list for e.g. testing.
Slightly reduces file size, improves build speed and makes build more
reproducible.
The .comment section contained GCC version information which could cause
different build output from just a minor update in GCC. The information is not
needed so we strip this.
The strip command was invoked using xargs, spawning a new process for each file.
This is inefficient as xargs can correctly run the strip command with multiple
file names.
-----
taken from 43aaf9572f
On Ubuntu host, build succeeded; but e.g. on Manjaro host, it failed with:
```
./build.sh: line 233: /opt/electrum/contrib/build-linux/appimage/../../../contrib/build-linux/appimage/.cache/appimage/appimagetool: No such file or directory
```
This slightly increases the AppImage size but allows us to be more
compatible with older distributions.
-----
taken from Electron-Cash/Electron-Cash@96644acd6f
It warns about scripts being installed in a location that is not on the
path, but that is inconsequential as they are not used.
-----
taken from Electron-Cash/Electron-Cash@9a29017c5d
When building in docker on macOS, python builds with .exe extension
because the case insensitive file system of macOS leaks into docker.
This causes the build to result in a different output on macOS compared
to Linux. We simply patch sysconfigdata to remove the extension.
Some more info: https://bugs.python.org/issue27631
There are a lot of dupliacted files, testing files and unused libraries
present in the AppImage. Removing these reduces the AppImage size
significantly.
-----
taken from Electron-Cash/Electron-Cash@cff5fb1289
There was a problem where Python would not properly include the faketime
timestamp sometimes. This patch replaces faketime with a patch that is
used by Ubuntu for reproducible builds by exporting BUILD_DATE and
BUILD_TIME with the desired values.
-----
taken from Electron-Cash/Electron-Cash@9532508a3f
We build our own mksquashfs from squashfskit which supports generating
reproducible squashfs images. We use a small wrapper script to remove
the -mkfs-fixed-time which appimagekit passes but squashfskits
mksquashfs does not support.
-----
taken from Electron-Cash/Electron-Cash@dd1f106f4f
see AppImage/AppImageKit#929
