Merge pull request #8738 from SomberNight/202312_mac_build

mac build: migrate to new VM (macOS 10.14 -> 11)
2023-12-12 12:08:40 +01:00
parent dd3fc041f9 c98830d091
commit c1b359d43f
3 changed files with 56 additions and 93 deletions
--- a/contrib/osx/README.md
+++ b/contrib/osx/README.md
@@ -11,7 +11,7 @@ This guide explains how to build Electrum binaries for macOS systems.

 This needs to be done on a system running macOS or OS X.

-The script is only tested on Intel-based Macs, and the binary built
+The script is only tested on Intel-based (x86_64) Macs, and the binary built
 targets `x86_64` currently.

 Notes about compatibility with different macOS versions:
@@ -22,40 +22,50 @@ Notes about compatibility with different macOS versions:
  imposes a minimum supported macOS version.
 - If you want to build binaries that conform to the macOS "Gatekeeper", so as to
  minimise the warnings users get, the binaries need to be codesigned with a
-  certificate issued by Apple, and starting with macOS 10.15 the binaries also
-  need to be notarized by Apple's central server. The catch is that to be able to build
-  binaries that Apple will notarise (due to the requirements on the binaries themselves,
+  certificate issued by Apple, and starting with macOS 10.15 (targets) the binaries also
+  need to be notarized by Apple's central server. To be able to build
+  binaries that Apple will notarize (due to the requirements on the binaries themselves,
  e.g. hardened runtime) the build machine needs at least macOS 10.14.
  See [#6128](https://github.com/spesmilo/electrum/issues/6128).
+  - There are two tools that can be used to notarize a binary, both part of Xcode:
+    the old `altool` and the newer `notarytool`. `altool`
+    [was deprecated](https://developer.apple.com/news/?id=y5mjxqmn) by Apple.
+    `notarytool` requires Xcode 13+, and that in turn requires macOS 11.3+.

-We currently build the release binaries on macOS 10.14.6, and these seem to run on
-10.13 or newer.
-
-Before starting, you should install [`brew`](https://brew.sh/).
+We currently build the release binaries on macOS 11.7.10, and these seem to run on
+10.14 or newer. (note: 10.13 might also work, haven't tested)


 #### Notes about reproducibility

 - We recommend creating a VM with a macOS guest, e.g. using VirtualBox,
  and building there.
- The guest should run macOS 10.14.6 (that specific version).
+- The guest should run macOS 11.7.10 (that specific version).
 - The unix username should be `vagrant`, and `electrum` should be cloned directly
  to the user's home dir: `/Users/vagrant/electrum`.
 - Builders need to use the same version of Xcode; and note that
  full Xcode and Xcode commandline tools differ!
-  - Xcode CLI tools are sufficient for everything, except it is missing `altool`,
-    which is needed for the release-manager to notarise the `.dmg`.
-  - so full Xcode is needed, to have `altool`.
-  - however, brew now consider macOS 10.14 too old, and for some reason it
-    requires Xcode CLI tools. (`Error: Xcode alone is not sufficient on Mojave.`)
+  We use the Xcode CLI tools as installed by brew. (version 13.2)

-  So, we need *both* full Xcode and Xcode CLI tools. Both with version 11.3.1.
-  The two Xcodes should be located exactly as follows:
+  Sanity checks:
    ```
    $ xcode-select -p
-    /Users/vagrant/Downloads/Xcode.app/Contents/Developer
+    /Library/Developer/CommandLineTools
    $ xcrun --show-sdk-path
-    /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
+    /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
+    $ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
+    package-id: com.apple.pkg.CLTools_Executables
+    version: 13.2.0.0.1.1638488800
+    volume: /
+    location: /
+    install-time: XXXXXXXXXX
+    groups: com.apple.FindSystemFiles.pkg-group
+    $ gcc --version
+    Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
+    Apple clang version 13.0.0 (clang-1300.0.29.30)
+    Target: x86_64-apple-darwin20.6.0
+    Thread model: posix
+    InstalledDir: /Library/Developer/CommandLineTools/usr/bin
    ```
 - Installing extraneous brew packages can result in build differences.
  For example, pyinstaller seems to pick up and bundle brew-installed `libffi`.
@@ -64,41 +74,12 @@ Before starting, you should install [`brew`](https://brew.sh/).
  (or run e.g. `git clean -ffxd` to rm all local changes).


-#### 1. Get Xcode
+#### 1. Install brew

-Notarizing the application requires full Xcode
-(not just command line tools as that is missing `altool`).
+Install [`brew`](https://brew.sh/).

-Get it from [here](https://developer.apple.com/download/more/).
-Unfortunately, you need an "Apple ID" account.
+Let brew install the Xcode CLI tools.

-(note: the last Xcode that runs on macOS 10.14.6 is Xcode 11.3.1)
-
-Install full Xcode:
-```
-$ shasum -a 256 "$HOME/Downloads/Xcode_11.3.1.xip"
-9a92379b90734a9068832f858d594d3c3a30a7ddc3bdb6da49c738aed9ad34b5  /Users/vagrant/Downloads/Xcode_11.3.1.xip
-$ xip -x "$HOME/Downloads/Xcode_11.3.1.xip"
-$ sudo xcode-select -s "$HOME/Downloads/Xcode.app/Contents/Developer/"
-$ # agree with licence
-$ sudo xcodebuild -license
-```
-
-(note: unsure if needed:)
-```
-$ # try this to install additional component:
-$ sudo xcodebuild -runFirstLaunch
-```
-
-Install Xcode CLI tools:
-```
-$ sudo rm -rf /Library/Developer/CommandLineTools
-$ shasum -a 256 "$HOME/Downloads/Command_Line_Tools_for_Xcode_11.3.1.dmg"
-1c4b477285641cca5313f456b712bf726aca8db77f38793420e1d451588673f9  /Users/vagrant/Downloads/Command_Line_Tools_for_Xcode_11.3.1.dmg
-$ hdiutil attach "$HOME/Downloads/Command_Line_Tools_for_Xcode_11.3.1.dmg"
-$ sudo installer -package "/Volumes/Command Line Developer Tools/Command Line Tools.pkg" -target /
-$ hdiutil detach "/Volumes/Command Line Developer Tools"
-```

 #### 2. Build Electrum

@@ -107,14 +88,19 @@ $ hdiutil detach "/Volumes/Command Line Developer Tools"

 This creates both a folder named Electrum.app and the .dmg file.

-If you want the binaries codesigned for MacOS and notarised by Apple's central server,
+##### 2.1. For release binaries, here be dragons
+
+If you want the binaries codesigned for macOS and notarised by Apple's central server,
 provide these env vars to the `make_osx.sh` script:

    CODESIGN_CERT="Developer ID Application: Electrum Technologies GmbH (L6P37P7P56)" \
+    APPLE_TEAM_ID="L6P37P7P56" \
    APPLE_ID_USER="me@email.com" \
    APPLE_ID_PASSWORD="1234" \
    ./contrib/osx/make_osx.sh

+(note: `APPLE_ID_PASSWORD` is an app-specific password, *not* the account password)
+

 ## Verifying reproducibility and comparing against official binary

--- a/contrib/osx/make_osx.sh
+++ b/contrib/osx/make_osx.sh
@@ -101,6 +101,10 @@ source $VENV_DIR/bin/activate
 #       see additional "strip" pass on built files later in the file.
 export CFLAGS="-g0"

+# Do not build universal binaries. The default on macos 11+ and xcode 12+ is "-arch arm64 -arch x86_64"
+# but with that e.g. "hid.cpython-310-darwin.so" is not reproducible as built by clang.
+export ARCHFLAGS="-arch x86_64"
+
 info "Installing build dependencies"
 # note: re pip installing from PyPI,
 #       we prefer compiling C extensions ourselves, instead of using binary wheels,
--- a/contrib/osx/notarize_app.sh
+++ b/contrib/osx/notarize_app.sh
@@ -7,7 +7,7 @@ if [ -z "$1" ]; then
    exit 1
 fi

-if [ -z "$APPLE_ID_USER" ] || [ -z "$APPLE_ID_PASSWORD" ]; then
+if [ -z "$APPLE_ID_USER" ] || [ -z "$APPLE_ID_PASSWORD" ] || [ -z "$APPLE_TEAM_ID" ]; then
    echo "You need to set your Apple ID credentials with \$APPLE_ID_USER and \$APPLE_ID_PASSWORD."
    exit 1
 fi
@@ -23,12 +23,14 @@ ditto -c -k --rsrc --keepParent "$APP_BUNDLE" "${APP_BUNDLE}.zip"

 # Submit for notarization
 echo "Submitting $APP_BUNDLE for notarization..."
-RESULT=$(xcrun altool --notarize-app --type osx \
-    --file "${APP_BUNDLE}.zip" \
-    --primary-bundle-id org.electrum.electrum \
-    --username $APPLE_ID_USER \
-    --password @env:APPLE_ID_PASSWORD \
-    --output-format xml
+RESULT=$(xcrun notarytool submit \
+    --team-id "$APPLE_TEAM_ID" \
+    --apple-id "$APPLE_ID_USER" \
+    --password "$APPLE_ID_PASSWORD" \
+    --output-format plist \
+    --wait \
+    --timeout 10m \
+    "${APP_BUNDLE}.zip"
 )

 if [ $? -ne 0 ]; then
@@ -37,46 +39,17 @@ if [ $? -ne 0 ]; then
    exit 1
 fi

-REQUEST_UUID=$(echo "$RESULT" | xpath \
-    "//key[normalize-space(text()) = 'RequestUUID']/following-sibling::string[1]/text()" 2>/dev/null
-)
+STATUS=$(echo "$RESULT" | xpath -e \
+  "//key[normalize-space(text()) = 'status']/following-sibling::string[1]/text()" 2> /dev/null)

-if [ -z "$REQUEST_UUID" ]; then
-    echo "Submitting $APP_BUNDLE failed:"
+if [ "$STATUS" = "Accepted" ]; then
+    echo "Notarization of $APP_BUNDLE succeeded!"
+else
+    echo "Notarization of $APP_BUNDLE failed:"
    echo "$RESULT"
    exit 1
 fi

-echo "$(echo "$RESULT" | xpath \
-    "//key[normalize-space(text()) = 'success-message']/following-sibling::string[1]/text()" 2> /dev/null)"
-
-# Poll for notarization status
-echo "Submitted notarization request $REQUEST_UUID, waiting for response..."
-sleep 60
-while :
-do
-    RESULT=$(xcrun altool --notarization-info "$REQUEST_UUID" \
-        --username "$APPLE_ID_USER" \
-        --password @env:APPLE_ID_PASSWORD \
-        --output-format xml
-    )
-    STATUS=$(echo "$RESULT" | xpath \
-        "//key[normalize-space(text()) = 'Status']/following-sibling::string[1]/text()" 2>/dev/null
-    )
-
-    if [ "$STATUS" = "success" ]; then
-        echo "Notarization of $APP_BUNDLE succeeded!"
-        break
-    elif [ "$STATUS" = "in progress" ]; then
-        echo "Notarization in progress..."
-        sleep 20
-    else
-        echo "Notarization of $APP_BUNDLE failed:"
-        echo "$RESULT"
-        exit 1
-    fi
-done
-
 # Staple the notary ticket
 xcrun stapler staple "$APP_BUNDLE"