1
0
Commit Graph

168 Commits

Author SHA1 Message Date
SomberNight
c52a29fc22 build: appimage: fix trezor plugin for new trezorlib
- similar to prev commit that fixes it for the pyinstaller builds
- note that .dist-info/RECORD files are still not reproducible for many packages :((((

```
$ cd dist/
$ ./electrum-*-x86_64.AppImage1 --appimage-extract
$ mv squashfs-root/ squashfs-root1/
$ ./electrum-*-x86_64.AppImage2 --appimage-extract
$ mv squashfs-root/ squashfs-root2/
$ $(cd squashfs-root1; find -type f -exec sha256sum '{}' \; > ./../sha256sum1)
$ $(cd squashfs-root2; find -type f -exec sha256sum '{}' \; > ./../sha256sum2)
$ diff sha256sum1 sha256sum2 > d
$ cat d
507c507
< 269a133bd0d3c85265219c14154323ed934ac36ef9b389da609e43788de2bbcd  ./usr/lib/python3.12/site-packages/cffi-1.17.1.dist-info/RECORD
---
> d2c08987f207eed8e90476c576eeff67c0809d1bfd35234e583a2dc6895c5ff1  ./usr/lib/python3.12/site-packages/cffi-1.17.1.dist-info/RECORD
518c518
< d08bd76099191932f3cd289c19427ccab51d9c00a6c2e5126c57f83a84ec2246  ./usr/lib/python3.12/site-packages/aiohttp-3.12.4.dist-info/RECORD
---
> 281c5c7e2b28e56134fa8690a2a446e3ebc515f8cb3705cd5d06e83c53812fab  ./usr/lib/python3.12/site-packages/aiohttp-3.12.4.dist-info/RECORD
630c630
< 33665bc46c14e2c0f409ef2627170b093f6d7741e9516928928acbeeb717f155  ./usr/lib/python3.12/site-packages/propcache-0.3.1.dist-info/RECORD
---
> 4fddb29289418edb3ec392a09b8efa826333a60cf508055cac809375e497174e  ./usr/lib/python3.12/site-packages/propcache-0.3.1.dist-info/RECORD
1497c1497
< f80c1b47d15a7262ab2b81bcfc1f69816961b9f6b0fd3775ee5b5eb40d4b7a5b  ./usr/lib/python3.12/site-packages/pyqt6_sip-13.10.2.dist-info/RECORD
---
> 1b8fb8ea4d50d401cd4f6689531ab4922cb064b9caaec1aae6bd885c3a387eca  ./usr/lib/python3.12/site-packages/pyqt6_sip-13.10.2.dist-info/RECORD
3108c3108
< ca06bbe1dba05cd647e89411dd301fad15f42d0ff53ac2c509391cf0303c15c4  ./usr/lib/python3.12/site-packages/multidict-6.4.4.dist-info/RECORD
---
> 400741699b95d3c6bde3c64c7a4bb04b26cbdaee437a17b398d202794a1bd4cd  ./usr/lib/python3.12/site-packages/multidict-6.4.4.dist-info/RECORD
3233c3233
< f67e8f271725676df296fab7fab72099f3b853905f9ff677b5de261a52412411  ./usr/lib/python3.12/site-packages/hidapi-0.14.0.post4.dist-info/RECORD
---
> f4d43d5e4c3c2aa3e69fa40c7307b23b82ce4c5e4d5fb041a75b11ddc2ed7423  ./usr/lib/python3.12/site-packages/hidapi-0.14.0.post4.dist-info/RECORD
3271c3271
< aaa92bb84d0f56e82832903f1109dd4362c09ea4af0fefe3d0c95509bc0920ed  ./usr/lib/python3.12/site-packages/cbor2-5.6.5.dist-info/RECORD
---
> 8a21af0fadf22898266581df9987c97ee737a670bef9d71d5f53427619466b88  ./usr/lib/python3.12/site-packages/cbor2-5.6.5.dist-info/RECORD
```

-----

Example RECORD files:

```
$ diff squashfs-root1/./usr/lib/python3.12/site-packages/aiohttp-3.12.4.dist-info/RECORD squashfs-root2/./usr/lib/python3.12/site-packages/aiohttp-3.12.4.dist-info/RECORD
64c64
< aiohttp/_http_parser.cpython-312-x86_64-linux-gnu.so,sha256=SjcWt8faDDTOxBHetnDdlQU31sPj0Suc8pwY1_UnKkE,2868384
---
> aiohttp/_http_parser.cpython-312-x86_64-linux-gnu.so,sha256=xdf5Uy_NnFLmSmqNmeY7wytgwFdY7pHxZLOFFfl2Aq8,2868384
66c66
< aiohttp/_http_writer.cpython-312-x86_64-linux-gnu.so,sha256=g6PDJVuICGYq8rywuI6XFWLP6tw1KLWLpwSvJODWo18,510280
---
> aiohttp/_http_writer.cpython-312-x86_64-linux-gnu.so,sha256=U_Y0F0oYClrGRIKO_agXtAohjJ_jqfG375vESBZ56CY,510280
80c80
< aiohttp/_websocket/mask.cpython-312-x86_64-linux-gnu.so,sha256=u0PluBaJntMBktyD1AYv267FvbUO6gpj-Nt7erLj3Qg,257384
---
> aiohttp/_websocket/mask.cpython-312-x86_64-linux-gnu.so,sha256=tEnvrEzSPEtaFI8V_Ey08zy4SBmDhnm-QnxTxgDSYM4,257384
85c85
< aiohttp/_websocket/reader_c.cpython-312-x86_64-linux-gnu.so,sha256=wfcclW8jYOTVeLDRPO87dPvOBrtXiqxBsnEbP0yyGpI,1816496
---
> aiohttp/_websocket/reader_c.cpython-312-x86_64-linux-gnu.so,sha256=jfey8U3MSIp04r0bG_a1TFkIKSzMQ-FLSAWD3nY4wgg,1816496
```

```
$ diff squashfs-root1/./usr/lib/python3.12/site-packages/hidapi-0.14.0.post4.dist-info/RECORD squashfs-root2/./usr/lib/python3.12/site-packages/hidapi-0.14.0.post4.dist-info/RECORD
1c1
< hid.cpython-312-x86_64-linux-gnu.so,sha256=JefmLwo-XKZZmno3PBdmHnREed0Yw7IaPYCI2k1UmbM,1152024
---
> hid.cpython-312-x86_64-linux-gnu.so,sha256=38htTAcIC4nnTPIIjD0jkdtdwtGyWlBIhQwQ-g0K0iI,1152024
12c12
< hidraw.cpython-312-x86_64-linux-gnu.so,sha256=dJ4kVI0jXUj_Ba2cMcAL7Wjy8fkO21VPbhAmR-5n30Y,1156768
---
> hidraw.cpython-312-x86_64-linux-gnu.so,sha256=7908Jgbg7A-3JRtfKabPr_WJkMIjYgd5pX78_Yajo54,1156768
```

-----

error at runtime:
```
 22.51 | E | plugins.trezor.trezor | error importing trezor plugin deps
Traceback (most recent call last):
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/importlib/metadata/__init__.py", line 397, in from_name
    return next(cls.discover(name=name))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/site-packages/electrum/plugins/trezor/trezor.py", line 29, in <module>
    from .clientbase import TrezorClientBase, RecoveryDeviceInputMethod
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/site-packages/electrum/plugins/trezor/clientbase.py", line 18, in <module>
    import trezorlib.device
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/site-packages/trezorlib/device.py", line 27, in <module>
    from slip10 import SLIP10
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/site-packages/slip10/__init__.py", line 6, in <module>
    __version__ = importlib.metadata.version(__package__ or __name__)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/importlib/metadata/__init__.py", line 889, in version
    return distribution(distribution_name).version
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/importlib/metadata/__init__.py", line 862, in distribution
    return Distribution.from_name(distribution_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.mount_electrteDkNu/usr/lib/python3.12/importlib/metadata/__init__.py", line 399, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for slip10

```
2025-06-05 18:46:12 +00:00
SomberNight
d17c4beee9 appimage build: add comment we should update to new appimagetool toolchain 2025-06-05 16:52:06 +00:00
SomberNight
fa271e3958 build: bump python versions in binaries
note: 3.12 just transitioned to security-only status,
so can't bump win/mac binaries without switching to 3.13
(as we don't compile our own cpython for those)
2025-06-05 16:51:56 +00:00
SomberNight
0c42dfc5f1 build: rerun freeze_packages 2025-06-05 16:50:55 +00:00
SomberNight
608fedd7cd build: libsecp: rm hardcoded ABI version 2025-06-05 16:50:51 +00:00
SomberNight
bf0ad20c60 build: bump python versions in binaries 2025-06-05 16:50:35 +00:00
SomberNight
37ca5f7eff build: appimage: bump base from debian buster(2019) to bullseye(2021) 2025-06-05 16:50:31 +00:00
SomberNight
31b176169a contrib: mv locale-related scripts to contrib/locale/ 2025-04-14 17:18:37 +00:00
SomberNight
1144d9b8ea build: add script build_cleanlocale.sh 2025-04-14 17:18:34 +00:00
SomberNight
3567a4cfb0 mv git submodule electrum-locale from contrib to electrum/locale
- this merges `contrib/deterministic-build/locale` and `electrum/locale`
- it is now once again possible have translations when running from a local git clone
    - which was already possible in the past before crowdin removed their unauthenticated APIs
        - see https://github.com/spesmilo/electrum/issues/9531
    - however, the translations available are the often-old frozen strings from electrum-locale
        - while previously one could just download the latest strings from crowdin
2025-04-14 17:18:30 +00:00
SomberNight
b70da1d3b9 build: set YARL_NO_EXTENSIONS for all builds
due to cython version mismatch -- easier to just disable the compilation for now
2025-04-01 16:59:26 +00:00
SomberNight
450768ee6c sdist build: bump base image to debian 12 to have py3.10+
- debian 11 only has python 3.9, deb12 has py3.11
- pip install pip is no longer needed, atm apt has new enough pip
  - and on deb12, started getting "error: externally-managed-environment"
- faketime does not seem to work properly on debian 12
    (getting reproducibility issues for the tarball)
  - so instead we untar, fix the timestamps manually, and re-tar
2025-01-10 18:58:02 +00:00
SomberNight
3aac8ff0b8 sdist build: simplify sourceonly handling 2025-01-10 18:51:04 +00:00
SomberNight
f35437f03c build: set ELECTRUM_ECC_DONT_COMPILE=1, instead manually build lib
Haven't checked if electrum-ecc compiles libsecp reproducibly.
For now let's just keep the old flow.
(but if we spent time on making that compilation reproducible,
the appimage and the macos builds could use it directly)
2024-10-10 15:46:18 +00:00
SomberNight
276488e3d0 binaries: document min requirements for target systems 2024-10-04 14:02:51 +00:00
SomberNight
cfe8502f96 qt desktop gui: upgrade qt5->qt6
closes https://github.com/spesmilo/electrum/issues/8007
2024-09-18 15:48:38 +00:00
SomberNight
6b4ba1f757 sdist README: document differences between sourceonly and normal tar.gz 2024-09-11 14:55:22 +00:00
SomberNight
0d0aa0f04a build: appimage: dl xcb-util as debian src pkg instead of upstream git
The upstream git repo has been unreachable for at least days.
2024-09-10 15:45:07 +00:00
SomberNight
9508f76b7f appimage build: bump python version (3.10.13->3.11.9) 2024-04-18 17:18:44 +00:00
SomberNight
5369be66da appimage build: build zbar ourselves
to be able to package new version, which has security fixes
2024-01-10 20:48:27 +00:00
SomberNight
615cb66692 appimage build: bump python version (3.10.11->3.10.13) 2023-12-24 05:15:39 +00:00
SomberNight
1ff4130804 contrib/docker_notes.md: add notes re debian apt mirror, and envvars
related https://github.com/spesmilo/electrum/issues/8496
2023-06-22 15:45:30 +00:00
SomberNight
16c3c3c4c0 appimage build: bump python version (3.9.15->3.10.11) 2023-06-02 17:17:36 +00:00
SomberNight
cede16a522 libsecp256k1: update hardcoded .so lib name in binaries
follow-up 2a2b683d23

TODO: maybe we should drop the version number in the lib name we bundle...
2023-04-21 14:42:51 +00:00
SomberNight
16fcfe34a7 build: rm "contrib" from debian apt sources lists 2023-04-18 16:09:50 +00:00
SomberNight
f25e384654 build: fail if not inside git clone
related: https://github.com/spesmilo/electrum/issues/8284
2023-03-28 22:35:19 +00:00
SomberNight
c9b6a6c01e build: fix repro builds where host userid != 1000
- repro builds to use fixed uid=1000 inside the container
  - in case the file permissions leak into the binaries, they are still reproducible
  - chown 1000:1000 fresh_clone
- repro builds to create fresh_clone dir outside git clone
  - otherwise the local dev build would still interact with the fresh_clone dir
    - due to e.g. recursive "find -exec touch",
    - and even the "docker build" cmd itself would try to stat/read it
      - see https://github.com/docker/for-linux/issues/380
  - and "rm -rf fresh_clone" needs sudo if the host uid is not 1000
  - this way the local dev build does not need sudo

to recap:
- local dev builds use the host userid inside the container, directly operate on the project dir
  - does not need sudo
- repro builds create a fresh git clone, chown it to 1000, and use userid=1000 inside the container
  - if the host userid is 1000, does not need sudo
  - otherwise, needs sudo

closes https://github.com/spesmilo/electrum/issues/8261
2023-03-20 23:32:59 +00:00
SomberNight
6e472efd5f build: follow-up prev: only use host userid for local dev builds
reproducibility probably needs a hardcoded userid

Also, move the UID arg later in the dockerfiles, for better caching.
(if local dev build and repro build set different UIDs, the build caches
will diverge at that step)
2023-03-20 02:06:54 +00:00
Sander van Grieken
ab073827cf build: use uid of user building the build containers 2023-03-20 02:06:50 +00:00
SomberNight
08ae0a73b2 build: add separate .dockerignore files
The .dockerignore symlink in the project root dir is only picked up by the android build.
The android build has the project root as its build context for "docker build" --
the other builds have their own subdirectories as build context, e.g. contrib/build-linux/appimage.
2023-03-20 02:05:40 +00:00
SomberNight
d8abab34d8 build: rm "non-free" from debian apt sources lists
was not needed, and better to avoid :)
2023-03-18 06:22:38 +00:00
Sander van Grieken
65abb90049 build: update build container base image versions, update apt sources to current 2023-02-27 11:16:04 +01:00
ghost43
0eea47c78d libsecp256k1: update hardcoded .so lib name in binaries (#8186)
follow-up 7d83335e34
2023-02-04 01:36:19 +00:00
SomberNight
65ae281180 build: don't hardcode num worker threads "make -j4" 2022-12-03 22:50:18 +00:00
Sander van Grieken
acde8cd0b7 add apt --allow-downgrades in more places.
build sdist also from debian bullseye container
2022-11-06 10:55:10 +01:00
Sander van Grieken
c0772019f1 rerun freeze_container_distro.sh and update docker base images for debian buster and bullseye 2022-11-06 10:42:39 +01:00
SomberNight
1e404f4e30 binaries: update python and openssl
note: python 3.9.x is now in source-only mode, so could not update to latest...
it is time to investigate upgrading to python 3.10 in the win and mac binaries
2022-11-05 14:41:23 +00:00
SomberNight
6f311c8930 appimage build: better caching, both locally and on CI 2022-11-04 15:49:16 +00:00
SomberNight
34488478e6 appimage build: build and include libxcb-util1
fixes https://github.com/spesmilo/electrum/issues/8011

> Upgraded from Appimage 4.1.5 to 4.3.2 on MX-Linux 19.
> The new version fails to start with:
>
> ```
> $ electrum
> qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
> This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may > fix this problem.
>
> Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.
>
> Aborted (core dumped)
> ```

debian 10 only distributes libxcb-util0.
debian 11 distributes libxcb-util1.
our base image atm is debian 10, so we build the package from source.
2022-10-11 18:24:02 +00:00
SomberNight
b5900eae98 contrib: reformat most shell scripts
Mostly just indentations.
For consistency, to conform to .editorconfig.
2022-09-27 14:55:22 +00:00
SomberNight
1cecd2c6e8 contrib: rename some shell scripts to have ".sh" extension
The extension gives formatting hints to some editors. (especially if they support .editorconfig)
2022-09-27 13:34:03 +00:00
SomberNight
0abecffe6d mac build: force using source dist for most of our python dependencies
We compile from tar.gz, instead of using pre-built binary wheels from PyPI.
(or if the dep is pure-python, use tar.gz instead of "source-only" wheel)

-----
Some unorganised things below for future reference.

```
$ dsymutil -dump-debug-map dist1/hid.cpython-39-darwin.so
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hid.o unable to open object file: No such file or directory
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hidapi/mac/hid.o unable to open object file: No such file or directory
---
triple:          'x86_64-apple-darwin'
binary-path:     'dist1/hid.cpython-39-darwin.so'
...
```

```
$ nm -pa dist1/hid.cpython-39-darwin.so
```

- https://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable
- https://github.com/pypa/pip/issues/6505
- https://github.com/pypa/pip/issues/7808#issuecomment-770275723
- https://github.com/NixOS/nixpkgs/pull/91272
- https://github.com/cython/cython/pull/1576
- 9d2ba1611b/Cython/Compiler/ModuleNode.py (L913)
2022-09-26 12:16:02 +00:00
SomberNight
4f574afe5a Merge remote-tracking branch 'remotes/spesmilo/appimage-debian-base'
Change Docker base images from Ubuntu to Debian, and use `snapshot.debian.org` as apt source list.
Ubuntu occasionally removes version-pinned packages from apt (see #7484), which
- breaks historical reproducible builds
- introduces maintenance burden as we have to update the version pins
Hopefully this change fixes both issues.

merges https://github.com/spesmilo/electrum/pull/7926
closes https://github.com/spesmilo/electrum/issues/7484
2022-08-22 13:10:25 +00:00
SomberNight
019d213325 Revert "mac build: force using source dist for most of our python dependencies"
This reverts commit 0c2a885c66.

Reverting for now due to reproducibility issues:
```
error: /Users/vagrant/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate: can't open file: /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/libffi.8.dylib (No such file or directory)
```
`Electrum.app/Contents/MacOS/libffi.8.dylib` is included when building on one machine, but missing on the other...
Will have to investigate later.

related https://github.com/spesmilo/electrum/pull/7918
2022-08-17 17:07:22 +00:00
SomberNight
c2b5e3ec15 build: use build-locale.sh in all build scripts 2022-08-16 19:06:47 +00:00
SomberNight
d0de44a794 build: also build a "source-only" linux-packager-friendly tarball
We now distribute two tarballs, a "normal" one (the default, recommended for users),
and a new strictly source-only one (for Linux distro packagers).
The normal tarball, in addition to including everything from
the source-only one, also includes:
- compiled (`.mo`) locale files (in addition to source `.po` locale files)
- compiled (`_pb2.py`) protobuf files (in addition to source `.proto` files)
- the `packages/` folder containing source-only pure-python runtime dependencies

merges https://github.com/spesmilo/electrum/pull/7594
with follow-ups
2022-08-16 17:28:41 +00:00
SomberNight
7d36f2ba3a build: follow-up source-only tarballs 2022-08-16 17:23:08 +00:00
yanmaani
9a37184a57 build: create source-only tarball in release.sh 2022-08-15 12:00:00 +00:00
Sander van Grieken
f63df43614 wine: rename freeze_appimage_distro.sh to freeze_containers_distro.sh and
adapt build-wine container to Debian+snapshots base
2022-08-11 12:36:48 +02:00
Sander van Grieken
a1975e2449 appimage: add freeze_appimage_distro.sh and adapt Dockerfile to use debian buster base 2022-08-11 12:15:22 +02:00