b0b68a5601
## Summary This PR migrates our release workflow to [`cargo-dist`](https://github.com/axodotdev/cargo-dist). The primary motivation here is that we want to ship dedicated installers for Ruff that work across platforms, and `cargo-dist` gives us those installers out-of-the-box. The secondary motivation is that `cargo-dist` formalizes some of the patterns that we've built up over time in our own release process. At a high level: - The `release.yml` file is generated by `cargo-dist` with `cargo dist generate`. It doesn't contain any modifications vis-a-vis the generated file. (If it's edited out of band from generation, the release fails.) - Our customizations are inserted as custom steps within the `cargo-dist` workflow. Specifically, `build-binaries` builds the wheels and packages them into binaries (as on `main`), while `build-docker.yml` builds the Docker image. `publish-pypi.yml` publishes the wheels to PyPI. This is effectively our `release.yaml` (on `main`), broken down into individual workflows rather than steps within a single workflow. ### Changes from `main` The workflow is _nearly_ unchanged. We kick off a release manually via the GitHub Action by providing a tag. If the tag doesn't match the `Cargo.toml`, the release fails. If the tag matches an already-existing release, the release fails. The release proceeds by (in order): 0. Doing some upfront validation via `cargo-dist`. 1. Creating the wheels and archives. 2. Building and pushing the Docker image. 3. Publishing to PyPI (if it's not a "dry run"). 4. Creating the GitHub Release (if it's not a "dry run"). 5. Notifying `ruff-pre-commit` (if it's not a "dry run"). There are a few changes in the workflow as compared to `main`: - **We no longer validate the SHA** (just the tag). It's not an input to the job. The Axo team is considering whether / how to support this. - **Releases are now published directly** (rather than as draft). Again, the Axo team is considering whether / how to support this. The downside of drafts is that the URLs aren't stable, so the installers don't work _as long as the release is in draft_. This is fine for our workflow. It seems like the Axo team will add it. - Releases already contain the latest entry from the changelog (we don't need to copy it over). This "Just Works", which is nice, though we'll still want to edit them to add contributors. There are also a few **breaking changes** for consumers of the binaries: - **We no longer include the version tag in the file name**. This enables users to install via `/latest` URLs on GitHub, and is part of the cargo-dist paradigm. - **Archives now include an extra level of nesting,** which you can remove with `--strip-components=1` when untarring. Here's an example release that I created -- I omitted all the artifacts since I was just testing a workflow, so none of the installers or links work, but it gives you a sense for what the release looks like: https://github.com/charliermarsh/cargodisttest/releases/tag/0.1.13. ### Test Plan I ran a successful release to completion last night, and installed Ruff via the installer:   The piece I'm least confident about is the Docker push. We build the image, but the push fails in my test repo since I haven't wired up the credentials.
471 lines
15 KiB
YAML
471 lines
15 KiB
YAML
# Build ruff on all platforms.
|
|
#
|
|
# Generates both wheels (for PyPI) and archived binaries (for GitHub releases).
|
|
#
|
|
# Assumed to run as a subworkflow of .github/workflows/release.yml; specifically, as a local
|
|
# artifacts job within `cargo-dist`.
|
|
name: "Build binaries"
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
plan:
|
|
required: true
|
|
type: string
|
|
pull_request:
|
|
paths:
|
|
# When we change pyproject.toml, we want to ensure that the maturin builds still work.
|
|
- pyproject.toml
|
|
# And when we change this workflow itself...
|
|
- .github/workflows/build-binaries.yml
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
env:
|
|
PACKAGE_NAME: ruff
|
|
MODULE_NAME: ruff
|
|
PYTHON_VERSION: "3.11"
|
|
CARGO_INCREMENTAL: 0
|
|
CARGO_NET_RETRY: 10
|
|
CARGO_TERM_COLOR: always
|
|
RUSTUP_MAX_RETRIES: 10
|
|
|
|
jobs:
|
|
sdist:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build sdist"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
command: sdist
|
|
args: --out dist
|
|
- name: "Test sdist"
|
|
run: |
|
|
pip install dist/${{ env.PACKAGE_NAME }}-*.tar.gz --force-reinstall
|
|
${{ env.MODULE_NAME }} --help
|
|
python -m ${{ env.MODULE_NAME }} --help
|
|
- name: "Upload sdist"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-sdist
|
|
path: dist
|
|
|
|
macos-x86_64:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: macos-12
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
architecture: x64
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels - x86_64"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: x86_64
|
|
args: --release --locked --out dist
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-macos-x86_64
|
|
path: dist
|
|
- name: "Archive binary"
|
|
run: |
|
|
TARGET=x86_64-apple-darwin
|
|
ARCHIVE_NAME=ruff-$TARGET
|
|
ARCHIVE_FILE=$ARCHIVE_NAME.tar.gz
|
|
|
|
mkdir -p $ARCHIVE_NAME
|
|
cp target/$TARGET/release/ruff $ARCHIVE_NAME/ruff
|
|
tar czvf $ARCHIVE_FILE $ARCHIVE_NAME
|
|
shasum -a 256 $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-macos-x86_64
|
|
path: |
|
|
*.tar.gz
|
|
*.sha256
|
|
|
|
macos-aarch64:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: macos-14
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
architecture: arm64
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels - aarch64"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: aarch64
|
|
args: --release --locked --out dist
|
|
- name: "Test wheel - aarch64"
|
|
run: |
|
|
pip install dist/${{ env.PACKAGE_NAME }}-*.whl --force-reinstall
|
|
ruff --help
|
|
python -m ruff --help
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-aarch64-apple-darwin
|
|
path: dist
|
|
- name: "Archive binary"
|
|
run: |
|
|
TARGET=aarch64-apple-darwin
|
|
ARCHIVE_NAME=ruff-$TARGET
|
|
ARCHIVE_FILE=$ARCHIVE_NAME.tar.gz
|
|
|
|
mkdir -p $ARCHIVE_NAME
|
|
cp target/$TARGET/release/ruff $ARCHIVE_NAME/ruff
|
|
tar czvf $ARCHIVE_FILE $ARCHIVE_NAME
|
|
shasum -a 256 $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-aarch64-apple-darwin
|
|
path: |
|
|
*.tar.gz
|
|
*.sha256
|
|
|
|
windows:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: windows-latest
|
|
strategy:
|
|
matrix:
|
|
platform:
|
|
- target: x86_64-pc-windows-msvc
|
|
arch: x64
|
|
- target: i686-pc-windows-msvc
|
|
arch: x86
|
|
- target: aarch64-pc-windows-msvc
|
|
arch: x64
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
architecture: ${{ matrix.platform.arch }}
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: ${{ matrix.platform.target }}
|
|
args: --release --locked --out dist
|
|
env:
|
|
# aarch64 build fails, see https://github.com/PyO3/maturin/issues/2110
|
|
XWIN_VERSION: 16
|
|
- name: "Test wheel"
|
|
if: ${{ !startsWith(matrix.platform.target, 'aarch64') }}
|
|
shell: bash
|
|
run: |
|
|
python -m pip install dist/${{ env.PACKAGE_NAME }}-*.whl --force-reinstall
|
|
${{ env.MODULE_NAME }} --help
|
|
python -m ${{ env.MODULE_NAME }} --help
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-${{ matrix.platform.target }}
|
|
path: dist
|
|
- name: "Archive binary"
|
|
shell: bash
|
|
run: |
|
|
ARCHIVE_FILE=ruff-${{ matrix.platform.target }}.zip
|
|
7z a $ARCHIVE_FILE ./target/${{ matrix.platform.target }}/release/ruff.exe
|
|
sha256sum $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-${{ matrix.platform.target }}
|
|
path: |
|
|
*.zip
|
|
*.sha256
|
|
|
|
linux:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
matrix:
|
|
target:
|
|
- x86_64-unknown-linux-gnu
|
|
- i686-unknown-linux-gnu
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
architecture: x64
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: ${{ matrix.target }}
|
|
manylinux: auto
|
|
args: --release --locked --out dist
|
|
- name: "Test wheel"
|
|
if: ${{ startsWith(matrix.target, 'x86_64') }}
|
|
run: |
|
|
pip install dist/${{ env.PACKAGE_NAME }}-*.whl --force-reinstall
|
|
${{ env.MODULE_NAME }} --help
|
|
python -m ${{ env.MODULE_NAME }} --help
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-${{ matrix.target }}
|
|
path: dist
|
|
- name: "Archive binary"
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
TARGET=${{ matrix.target }}
|
|
ARCHIVE_NAME=ruff-$TARGET
|
|
ARCHIVE_FILE=$ARCHIVE_NAME.tar.gz
|
|
|
|
mkdir -p $ARCHIVE_NAME
|
|
cp target/$TARGET/release/ruff $ARCHIVE_NAME/ruff
|
|
tar czvf $ARCHIVE_FILE $ARCHIVE_NAME
|
|
shasum -a 256 $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-${{ matrix.target }}
|
|
path: |
|
|
*.tar.gz
|
|
*.sha256
|
|
|
|
linux-cross:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
matrix:
|
|
platform:
|
|
- target: aarch64-unknown-linux-gnu
|
|
arch: aarch64
|
|
# see https://github.com/astral-sh/ruff/issues/3791
|
|
# and https://github.com/gnzlbg/jemallocator/issues/170#issuecomment-1503228963
|
|
maturin_docker_options: -e JEMALLOC_SYS_WITH_LG_PAGE=16
|
|
- target: armv7-unknown-linux-gnueabihf
|
|
arch: armv7
|
|
- target: s390x-unknown-linux-gnu
|
|
arch: s390x
|
|
- target: powerpc64le-unknown-linux-gnu
|
|
arch: ppc64le
|
|
# see https://github.com/astral-sh/ruff/issues/10073
|
|
maturin_docker_options: -e JEMALLOC_SYS_WITH_LG_PAGE=16
|
|
- target: powerpc64-unknown-linux-gnu
|
|
arch: ppc64
|
|
# see https://github.com/astral-sh/ruff/issues/10073
|
|
maturin_docker_options: -e JEMALLOC_SYS_WITH_LG_PAGE=16
|
|
- target: arm-unknown-linux-musleabihf
|
|
arch: arm
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: ${{ matrix.platform.target }}
|
|
manylinux: auto
|
|
docker-options: ${{ matrix.platform.maturin_docker_options }}
|
|
args: --release --locked --out dist
|
|
- uses: uraimo/run-on-arch-action@v2
|
|
if: matrix.platform.arch != 'ppc64'
|
|
name: Test wheel
|
|
with:
|
|
arch: ${{ matrix.platform.arch == 'arm' && 'armv6' || matrix.platform.arch }}
|
|
distro: ${{ matrix.platform.arch == 'arm' && 'bullseye' || 'ubuntu20.04' }}
|
|
githubToken: ${{ github.token }}
|
|
install: |
|
|
apt-get update
|
|
apt-get install -y --no-install-recommends python3 python3-pip
|
|
pip3 install -U pip
|
|
run: |
|
|
pip3 install ${{ env.PACKAGE_NAME }} --no-index --find-links dist/ --force-reinstall
|
|
ruff --help
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-${{ matrix.platform.target }}
|
|
path: dist
|
|
- name: "Archive binary"
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
TARGET=${{ matrix.platform.target }}
|
|
ARCHIVE_NAME=ruff-$TARGET
|
|
ARCHIVE_FILE=$ARCHIVE_NAME.tar.gz
|
|
|
|
mkdir -p $ARCHIVE_NAME
|
|
cp target/$TARGET/release/ruff $ARCHIVE_NAME/ruff
|
|
tar czvf $ARCHIVE_FILE $ARCHIVE_NAME
|
|
shasum -a 256 $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-${{ matrix.platform.target }}
|
|
path: |
|
|
*.tar.gz
|
|
*.sha256
|
|
|
|
musllinux:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
matrix:
|
|
target:
|
|
- x86_64-unknown-linux-musl
|
|
- i686-unknown-linux-musl
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
architecture: x64
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: ${{ matrix.target }}
|
|
manylinux: musllinux_1_2
|
|
args: --release --locked --out dist
|
|
- name: "Test wheel"
|
|
if: matrix.target == 'x86_64-unknown-linux-musl'
|
|
uses: addnab/docker-run-action@v3
|
|
with:
|
|
image: alpine:latest
|
|
options: -v ${{ github.workspace }}:/io -w /io
|
|
run: |
|
|
apk add python3
|
|
python -m venv .venv
|
|
.venv/bin/pip3 install ${{ env.PACKAGE_NAME }} --no-index --find-links dist/ --force-reinstall
|
|
.venv/bin/${{ env.MODULE_NAME }} --help
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-${{ matrix.target }}
|
|
path: dist
|
|
- name: "Archive binary"
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
TARGET=${{ matrix.target }}
|
|
ARCHIVE_NAME=ruff-$TARGET
|
|
ARCHIVE_FILE=$ARCHIVE_NAME.tar.gz
|
|
|
|
mkdir -p $ARCHIVE_NAME
|
|
cp target/$TARGET/release/ruff $ARCHIVE_NAME/ruff
|
|
tar czvf $ARCHIVE_FILE $ARCHIVE_NAME
|
|
shasum -a 256 $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-${{ matrix.target }}
|
|
path: |
|
|
*.tar.gz
|
|
*.sha256
|
|
|
|
musllinux-cross:
|
|
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-build') }}
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
matrix:
|
|
platform:
|
|
- target: aarch64-unknown-linux-musl
|
|
arch: aarch64
|
|
maturin_docker_options: -e JEMALLOC_SYS_WITH_LG_PAGE=16
|
|
- target: armv7-unknown-linux-musleabihf
|
|
arch: armv7
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- name: "Prep README.md"
|
|
run: python scripts/transform_readme.py --target pypi
|
|
- name: "Build wheels"
|
|
uses: PyO3/maturin-action@v1
|
|
with:
|
|
target: ${{ matrix.platform.target }}
|
|
manylinux: musllinux_1_2
|
|
args: --release --locked --out dist
|
|
docker-options: ${{ matrix.platform.maturin_docker_options }}
|
|
- uses: uraimo/run-on-arch-action@v2
|
|
name: Test wheel
|
|
with:
|
|
arch: ${{ matrix.platform.arch }}
|
|
distro: alpine_latest
|
|
githubToken: ${{ github.token }}
|
|
install: |
|
|
apk add python3
|
|
run: |
|
|
python -m venv .venv
|
|
.venv/bin/pip3 install ${{ env.PACKAGE_NAME }} --no-index --find-links dist/ --force-reinstall
|
|
.venv/bin/${{ env.MODULE_NAME }} --help
|
|
- name: "Upload wheels"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-${{ matrix.platform.target }}
|
|
path: dist
|
|
- name: "Archive binary"
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
TARGET=${{ matrix.platform.target }}
|
|
ARCHIVE_NAME=ruff-$TARGET
|
|
ARCHIVE_FILE=$ARCHIVE_NAME.tar.gz
|
|
|
|
mkdir -p $ARCHIVE_NAME
|
|
cp target/$TARGET/release/ruff $ARCHIVE_NAME/ruff
|
|
tar czvf $ARCHIVE_FILE $ARCHIVE_NAME
|
|
shasum -a 256 $ARCHIVE_FILE > $ARCHIVE_FILE.sha256
|
|
- name: "Upload binary"
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: artifacts-${{ matrix.platform.target }}
|
|
path: |
|
|
*.tar.gz
|
|
*.sha256
|