build: also build a "source-only" linux-packager-friendly tarball

We now distribute two tarballs, a "normal" one (the default, recommended for users),
and a new strictly source-only one (for Linux distro packagers).
The normal tarball, in addition to including everything from
the source-only one, also includes:
- compiled (`.mo`) locale files (in addition to source `.po` locale files)
- compiled (`_pb2.py`) protobuf files (in addition to source `.proto` files)
- the `packages/` folder containing source-only pure-python runtime dependencies

merges https://github.com/spesmilo/electrum/pull/7594
with follow-ups

contrib/build-linux/sdist/README.md

@@ -1,5 +1,4 @@
-Source tarballs
-===============
+# Source tarballs
 
 ✓ _This file should be reproducible, meaning you should be able to generate
    distributables that match the official releases._
@@ -7,18 +6,31 @@ Source tarballs
 This assumes an Ubuntu (x86_64) host, but it should not be too hard to adapt to another
 similar system.
 
+We distribute two tarballs, a "normal" one (the default, recommended for users),
+and a strictly source-only one (for Linux distro packagers).
+The normal tarball, in addition to including everything from
+the source-only one, also includes:
+- compiled (`.mo`) locale files (in addition to source `.po` locale files)
+- compiled (`_pb2.py`) protobuf files (in addition to source `.proto` files)
+- the `packages/` folder containing source-only pure-python runtime dependencies
+
+
+## Build steps
+
 1. Install Docker
 
     See `contrib/docker_notes.md`.
 
-2. Build source tarball
+2. Build tarball
 
+    (set envvar `OMIT_UNCLEAN_FILES=1` to build the "source-only" tarball)
     ```
     $ ./build.sh
     ```
     If you want reproducibility, try instead e.g.:
     ```
     $ ELECBUILD_COMMIT=HEAD ELECBUILD_NOCACHE=1 ./build.sh
+    $ ELECBUILD_COMMIT=HEAD ELECBUILD_NOCACHE=1 OMIT_UNCLEAN_FILES=1 ./build.sh
     ```
 
 3. The generated distributables are in `./dist`.

contrib/build-linux/sdist/build.sh

@@ -47,6 +47,7 @@ docker run -it \
     -v "$PROJECT_ROOT_OR_FRESHCLONE_ROOT":/opt/electrum \
     --rm \
     --workdir /opt/electrum/contrib/build-linux/sdist \
+    --env OMIT_UNCLEAN_FILES \
     electrum-sdist-builder-img \
     ./make_sdist.sh
 

contrib/build-linux/sdist/make_sdist.sh

@@ -6,7 +6,7 @@ PROJECT_ROOT="$(dirname "$(readlink -e "$0")")/../../.."
 CONTRIB="$PROJECT_ROOT/contrib"
 CONTRIB_SDIST="$CONTRIB/build-linux/sdist"
 DISTDIR="$PROJECT_ROOT/dist"
-LOCALE="$PROJECT_ROOT/electrum/locale/"
+LOCALE="$PROJECT_ROOT/electrum/locale"
 
 . "$CONTRIB"/build_tools_util.sh
 
@@ -19,35 +19,55 @@ break_legacy_easy_install
 # (make_packages will later install a pinned version of pip in a venv)
 python3 -m pip install --upgrade pip
 
-"$CONTRIB"/make_packages || fail "make_packages failed"
+rm -rf "$PROJECT_ROOT/packages/"
+if ([ "$OMIT_UNCLEAN_FILES" != 1 ]); then
+  "$CONTRIB"/make_packages || fail "make_packages failed"
+fi
 
 git submodule update --init
 
 (
-    cd "$CONTRIB/deterministic-build/electrum-locale/"
-    if ! which msgfmt > /dev/null 2>&1; then
-        echo "Please install gettext"
-        exit 1
-    fi
-    # We include both source (.po) and compiled (.mo) locale files in the source dist.
-    # Maybe we should exclude the compiled locale files? see https://askubuntu.com/a/144139
-    # (also see MANIFEST.in)
+    # By default, include both source (.po) and compiled (.mo) locale files in the source dist.
+    # Set option OMIT_UNCLEAN_FILES=1 to exclude the compiled locale files
+    # see https://askubuntu.com/a/144139 (also see MANIFEST.in)
     rm -rf "$LOCALE"
-    for i in ./locale/*; do
-        dir="$PROJECT_ROOT/electrum/$i/LC_MESSAGES"
-        mkdir -p "$dir"
-        msgfmt --output-file="$dir/electrum.mo" "$i/electrum.po" || true
-        cp $i/electrum.po "$PROJECT_ROOT/electrum/$i/electrum.po"
-    done
+    cp -r "$CONTRIB/deterministic-build/electrum-locale/locale/" "$LOCALE/"
+    if ([ "$OMIT_UNCLEAN_FILES" != 1 ]); then
+      "$CONTRIB/build_locale.sh" "$LOCALE"
+    fi
 )
 
+if ([ "$OMIT_UNCLEAN_FILES" = 1 ]); then
+    # FIXME side-effecting repo... though in practice, this script probably runs in fresh_clone
+    rm -f "$PROJECT_ROOT/electrum/paymentrequest_pb2.py"
+fi
+
 (
     cd "$PROJECT_ROOT"
 
     find -exec touch -h -d '2000-11-11T11:11:11+00:00' {} +
 
     # note: .zip sdists would not be reproducible due to https://bugs.python.org/issue40963
-    TZ=UTC faketime -f '2000-11-11 11:11:11' python3 setup.py --quiet sdist --format=gztar
+    if ([ "$OMIT_UNCLEAN_FILES" = 1 ])
+        then PY_DISTDIR="dist/_sourceonly" # The DISTDIR variable of this script is only used to find where the output is *finally* placed.
+        else PY_DISTDIR="dist"
+    fi
+    TZ=UTC faketime -f '2000-11-11 11:11:11' python3 setup.py --quiet sdist --format=gztar --dist-dir="$PY_DISTDIR"
+    if ([ "$OMIT_UNCLEAN_FILES" = 1 ]); then
+        python3 <<EOF
+import importlib.util
+import os
+
+# load version.py; needlessly complicated alternative to "imp.load_source":
+version_spec = importlib.util.spec_from_file_location('version', 'electrum/version.py')
+version_module = importlib.util.module_from_spec(version_spec)
+version_spec.loader.exec_module(version_module)
+
+VER = version_module.ELECTRUM_VERSION
+os.rename(f"dist/_sourceonly/Electrum-{VER}.tar.gz", f"dist/Electrum-sourceonly-{VER}.tar.gz")
+EOF
+        rmdir "$PY_DISTDIR"
+    fi
 )