Auto build + release workflow of supercollider plugins

Hello everyone

I just wanted to share this little project (after inspiration from yesterday’s SC meetup):

Automatically building, compiling and releasing SuperCollider plugins using Github Actions. I just set this up on my little personal plugins repo.

Here is the yaml file I used:

.github/workflows/cmake.yml:

on:
  push:
    # Sequence of patterns matched against refs/tags
    tags:
      - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10

jobs:
  build:
    # The CMake configure and build commands are platform agnostic and should work equally
    # well on Windows or Mac.  You can convert this to a matrix build if you need
    # cross-platform coverage.
    # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    
    - name: Get SC source code
      run: git clone https://github.com/supercollider/supercollider.git ${{github.workspace}}/supercollider

    - name: Create Build Environment
      # Some projects don't allow in-source building, so create a separate build directory
      # We'll use this as our working directory for all subsequent commands
      run: cmake -E make_directory ${{github.workspace}}/build

    - name: Configure CMake
      # Use a bash shell so we can use the same syntax for environment variable
      # access regardless of the host operating system
      shell: bash
      working-directory: ${{github.workspace}}/build
      # Note the current convention is to use the -S and -B options here to specify source 
      # and build directories, but this is only available with CMake 3.13 and higher.  
      # The CMake binaries on the Github Actions machines are (as of this writing) 3.12
      run: cmake .. -DCMAKE_BUILD_TYPE='Release' -DSC_PATH=${{github.workspace}}/supercollider -DCMAKE_INSTALL_PREFIX=/${{github.workspace}}/build/install

    

    - name: Build
      working-directory: ${{github.workspace}}/build
      shell: bash
      # Execute the build.  You can specify a specific target with "--target <NAME>"
      run: cmake --build . --config "Release" --target install

      # Gather all files in a zip
    - name: Zip up build
      shell: bash
      working-directory: ${{github.workspace}}/build
      run: zip -r MKPlugins install/MKPlugins

      # Publish build
    - name: Create Release
      id: create_release
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
        with:
          tag_name: ${{ github.ref }}
          release_name: ${{ github.ref }}
          draft: false
          prerelease: false
          
    - name: Upload Release Asset
      id: upload-release-asset 
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }} 
        # This pulls from the CREATE RELEASE step above, referencing it's ID to get its outputs object, which include a `upload_url`. See this blog post for more info: https://jasonet.co/posts/new-features-of-github-actions/#passing-data-to-future-steps 
        asset_path: ./${{github.workspace}}/build/install/MKPlugins.zip
        asset_name: MKPlugins.zip
        asset_content_type: application/zip

With this, whenever I push to my github repo with a new tag prefixed v Github will trigger the build workflow and create a new release.

So, this local command:

git tag v0.0.108 && git push origin --tags

Becomes MKPlugins-v0.0.108 after about 45 seconds of Github chewing on the build.

The yaml syntax for these builds is a PITA and a bit underdocumented IMO (like, it really surprised me how much chaos wrong indentation causes (and how bad the resulting error messages are)).

I know SC core has moved to this workflow as well.

Cross platform?

The yaml file above only makes a Linux build.

Can someone give me pointers on how to make this build cross platform to build macos and windows plugins as well? This really could be an awesome future for plugin developers because it would make it easy to have personal repos with easily downloadable prebuilds for the users. Woohoo.

1 Like

The travis file for sc3-plugins may be a good start for cross-platform (can GHA not do OSX builds yet?).

IIRC this should work for personal plugin builds, once you adjust the build commands appropriately, remove the aws deploy section, and add your $GITHUB_KEY as an env variable in travis. Also, if anyone wants to push build artifacts to the s3 store that we use for regular SuperCollider builds, there’s plenty of space, I’m happy to get this set up for you - DM me.

MacOS wasn’t hard to add, here is a modified version with macOS builds as well:

on:
  push:
    # Sequence of patterns matched against refs/tags
    tags:
      - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10

jobs:
  build:
    # The CMake configure and build commands are platform agnostic and should work equally
    # well on Windows or Mac.  You can convert this to a matrix build if you need
    # cross-platform coverage.
    # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    
    - name: Get SC source code
      run: git clone https://github.com/supercollider/supercollider.git ${{github.workspace}}/supercollider

    - name: Create Build Environment
      # Some projects don't allow in-source building, so create a separate build directory
      # We'll use this as our working directory for all subsequent commands
      run: cmake -E make_directory ${{github.workspace}}/build

    - name: Configure CMake
      # Use a bash shell so we can use the same syntax for environment variable
      # access regardless of the host operating system
      shell: bash
      working-directory: ${{github.workspace}}/build
      # Note the current convention is to use the -S and -B options here to specify source 
      # and build directories, but this is only available with CMake 3.13 and higher.  
      # The CMake binaries on the Github Actions machines are (as of this writing) 3.12
      run: cmake .. -DCMAKE_BUILD_TYPE='Release' -DSC_PATH=${{github.workspace}}/supercollider -DCMAKE_INSTALL_PREFIX=/${{github.workspace}}/build/install

    

    - name: Build
      working-directory: ${{github.workspace}}/build
      shell: bash
      # Execute the build.  You can specify a specific target with "--target <NAME>"
      run: cmake --build . --config "Release" --target install

      # Gather all files in a zip
    - name: Zip up build
      shell: bash
      working-directory: ${{github.workspace}}/build
      run: zip -r MKPlugins install/MKPlugins

      # Publish build
    - name: Create Release
      id: create_release
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
        with:
          tag_name: ${{ github.ref }}
          release_name: ${{ github.ref }}
          draft: false
          prerelease: false
          
    - name: Upload Release Asset
      id: upload-release-asset 
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }} 
        # This pulls from the CREATE RELEASE step above, referencing it's ID to get its outputs object, which include a `upload_url`. See this blog post for more info: https://jasonet.co/posts/new-features-of-github-actions/#passing-data-to-future-steps 
        asset_path: ./${{github.workspace}}/build/install/MKPlugins.zip
        asset_name: MKPlugins.zip
        asset_content_type: application/zip

this is great! would you want to find a way to combine this with the cookiecutter repo? if I recall it currently sets you up to use Travis CI which is not going to work much longer.

My thughts exactly and some more characters!

1 Like

Okay trying to simoultaneously figure oout the GH actions syntax and Windows stuff was annoying, but with this script the github action will now do cross platform builds for MacOS, Linux and Windows. Only thing missing is Raspberry Pi/Arm but it doesn’t seem supported.

The file is a bit messy but it works:

on:
  push:
    tags:
      - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10

jobs:
  build:

    runs-on: ${{matrix.os}}
    strategy:
      matrix:
        os: [macos-latest, ubuntu-18.04, windows-latest]

    steps:
    - uses: actions/checkout@v2
    
    - name: Install 7Zip (Windows)
      if: matrix.os == 'windows-latest'
      shell: powershell
      run: Install-Module 7Zip4PowerShell -Force -Verbose

    - name: Get SC source code
      run: git clone https://github.com/supercollider/supercollider.git ${{github.workspace}}/supercollider

    - name: Create Build Environment
      # Some projects don't allow in-source building, so create a separate build directory
      # We'll use this as our working directory for all subsequent commands
      run: cmake -E make_directory ${{github.workspace}}/build

    - name: Configure CMake (Unix)
      shell: bash
      if: matrix.os != 'windows-latest'
      working-directory: ${{github.workspace}}/build
      run: cmake .. -DCMAKE_BUILD_TYPE='Release' -DSC_PATH=${{github.workspace}}/supercollider -DCMAKE_INSTALL_PREFIX=${{github.workspace}}/build/install

    - name: Configure CMake (Windows)
      if: matrix.os == 'windows-latest'
      shell: pwsh
      working-directory: ${{github.workspace}}\build
      run: cmake .. -DCMAKE_BUILD_TYPE='Release' -DSC_PATH=${{github.workspace}}\supercollider -DCMAKE_INSTALL_PREFIX=${{github.workspace}}\build\install

    - name: Build (Unix)
      if: matrix.os != 'windows-latest'
      working-directory: ${{github.workspace}}/build
      shell: bash
      run: cmake --build . --config "Release" --target install

    - name: Build (Windows)
      working-directory: ${{github.workspace}}\build
      if: matrix.os == 'windows-latest'
      shell: pwsh
      run: cmake --build . --config "Release" --target install

      # Gather all files in a zip
    - name: Zip up build (Unix)
      if: matrix.os != 'windows-latest'
      shell: bash
      working-directory: ${{github.workspace}}/build
      run: zip -r MKPlugins-${{runner.os}} install/MKPlugins
      
      # Gather all files in a zip
    - name: Zip up build (Windows)
      if: matrix.os == 'windows-latest'
      shell: pwsh
      working-directory: ${{github.workspace}}\build
      run: Compress-7Zip "install\MKPlugins" -ArchiveFileName "MKPlugins-${{runner.os}}.zip" -Format Zip

      # Publish build
    - name: Create Release
      id: create_release
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
      with:
        tag_name: ${{ github.ref }}-${{matrix.os}}
        release_name: MKPlugins-${{ github.ref }}-${{matrix.os}}
        draft: false
        prerelease: false
          
    - name: Upload Release Asset (Unix)
      id: upload-release-asset-unix 
      if: matrix.os != 'windows-latest'
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }} 
        # This pulls from the CREATE RELEASE step above, referencing it's ID to get its outputs object, which include a `upload_url`. See this blog post for more info: https://jasonet.co/posts/new-features-of-github-actions/#passing-data-to-future-steps 
        asset_path: ${{github.workspace}}/build/MKPlugins-${{runner.os}}.zip
        asset_name: MKPlugins-${{runner.os}}.zip
        asset_content_type: application/zip

    - name: Upload Release Asset (Windows)
      id: upload-release-asset-windows 
      if: matrix.os == 'windows-latest'
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }} 
        asset_path: ${{github.workspace}}\build\MKPlugins-${{runner.os}}.zip
        asset_name: MKPlugins-${{runner.os}}.zip
        asset_content_type: application/zip

another problem with this one is that it creates release tags for each OS, so they come up as seperate releases. Not sure how to fix that yet but if anyone has tips please let me know :slight_smile:

i have never looked into it much, but i’m pretty certain there should be a way to do cross-compilation for RPi on an Ubuntu system. if we ever get around to automating linux packaging through our build system, i think this might make it possible to easily create release packages for RPi. take that with a grain of salt though, my knowledge here is pretty spotty.

i’m not quite sure i understand why you are putting the OS name into the tag. for SuperCollider (and most projects I’ve seen) the typical strategy is to have a single tag, and then include the OS/other metadata in the filename of the artifact. so your config would change to:

-        tag_name: ${{ github.ref }}-${{matrix.os}}
-        release_name: MKPlugins-${{ github.ref }}-${{matrix.os}}
+        tag_name: ${{ github.ref }}
+        release_name: MKPlugins-${{ github.ref }}

would you like to make a PR to the cookiecutter repo? i would happily review it. or are there still things you need to figure out?

I can absolutely make a PR once I’ve figured out the last little issues here.

The tag thing is one of those things. Using the action I have been using, I get an error if I don’t create unique tags because it can’t upload

Validation Failed: {"resource":"Release","code":"already_exists","field":"tag_name"} 

But maybe I should try out another action that allows to append to releases in a better way.

Alright I finally figured it out. It was quite simple actually: Adding a conditional to the release creation stage to check if the run is the first run, and then only create a release if it is, otherwise skip that stage and just upload to the release. Now all seems to work:

on:
  push:
    tags:
      - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10

jobs:
  build:

    runs-on: ${{matrix.os}}
    strategy:
      matrix:
        os: [macos-latest, ubuntu-18.04, windows-latest]

    steps:
    - uses: actions/checkout@v2
    
    - name: Install 7Zip (Windows)
      if: matrix.os == 'windows-latest'
      shell: powershell
      run: Install-Module 7Zip4PowerShell -Force -Verbose

    - name: Get SC source code
      run: git clone https://github.com/supercollider/supercollider.git ${{github.workspace}}/supercollider

    - name: Create Build Environment
      # Some projects don't allow in-source building, so create a separate build directory
      # We'll use this as our working directory for all subsequent commands
      run: cmake -E make_directory ${{github.workspace}}/build

    - name: Configure CMake (Unix)
      shell: bash
      if: matrix.os != 'windows-latest'
      working-directory: ${{github.workspace}}/build
      run: cmake .. -DCMAKE_BUILD_TYPE='Release' -DSC_PATH=${{github.workspace}}/supercollider -DCMAKE_INSTALL_PREFIX=${{github.workspace}}/build/install

    - name: Configure CMake (Windows)
      if: matrix.os == 'windows-latest'
      shell: pwsh
      working-directory: ${{github.workspace}}\build
      run: cmake .. -DCMAKE_BUILD_TYPE='Release' -DSC_PATH=${{github.workspace}}\supercollider -DCMAKE_INSTALL_PREFIX=${{github.workspace}}\build\install

    - name: Build (Unix)
      if: matrix.os != 'windows-latest'
      working-directory: ${{github.workspace}}/build
      shell: bash
      run: cmake --build . --config "Release" --target install

    - name: Build (Windows)
      working-directory: ${{github.workspace}}\build
      if: matrix.os == 'windows-latest'
      shell: pwsh
      run: cmake --build . --config "Release" --target install

      # Gather all files in a zip
    - name: Zip up build (Unix)
      if: matrix.os != 'windows-latest'
      shell: bash
      working-directory: ${{github.workspace}}/build
      run: zip -r MKPlugins-${{runner.os}} install/MKPlugins
      
      # Gather all files in a zip
    - name: Zip up build (Windows)
      if: matrix.os == 'windows-latest'
      shell: pwsh
      working-directory: ${{github.workspace}}\build
      run: Compress-7Zip "install\MKPlugins" -ArchiveFileName "MKPlugins-${{runner.os}}.zip" -Format Zip
      
    # Publish build
    - name: Create Release
      if: github.run_number == 1
      id: create_release
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
      with:
        tag_name: ${{ github.ref }}
        release_name: MKPlugins-${{ github.ref }}
        draft: false
        prerelease: false
          
    - name: Upload binaries to release
      uses: svenstaro/upload-release-action@v2
      with:
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        file: ${{github.workspace}}/build/MKPlugins-${{runner.os}}.zip
        asset_name: MKPlugins-${{runner.os}}.zip
        tag: ${{ github.ref }}

And this is what the resulting release looks like:
https://github.com/madskjeldgaard/mkplugins/releases/tag/v0.0.11184

@MarcinP ultimately opted for https://github.com/softprops/action-gh-release for SuperCollider because of https://github.com/actions/create-release/issues/119

See https://github.com/supercollider/supercollider/blob/develop/.github/workflows/actions.yml#L433-L440

1 Like

I did a writeup of the process here, if anyone’s interested and will do a PR for the cookiecutter repo ASAP as well :slight_smile:

5 Likes

This is great ! Thanks Mads !

Several people are reporting issues on MacOS with regards to code signing when using this method for building and releasing using GHA. I don’t have a Mac at hand and know next to nothing about this. Does anyone here have any ideas on what to do to solve the issue?

Here’s an example from the real world: Signing did not work on Catalina · Issue #2 · redFrik/f0plugins · GitHub

thanks!

Unquarantining the files might help if people haven’t tried that already.

1 Like