Favour ruamel.yaml over PyYAML in tools

PyYAML is currently not packaged for MSYS2 UCRT64, nor it is for
brew. This forces those platform to install a YAML library manually
in addition to the dependencies defined in the ngdevkit package.

ruamel.yaml is available in MSYS2, Ubuntu, Debian and Fedora. The
missing package in brew is likely not going to change [1], but there
are means to download this dependency without using pip directly.

Plus, ruamel.yaml is supposed to be better at round-trip parsing
and keeps comments metadata, so for the time being, import it
in priority in the tools, and fall back to PyYAML when the former
is not available, as both libraries share a common API.

[1] https://docs.brew.sh/Homebrew-and-Python

Author: dciabrin

.github/workflows/build-tests.yaml

@@ -40,14 +40,14 @@ jobs:
 
       - name: Install dependencies
         run: |
-          sudo apt-get install -y ngdevkit-toolchain python3 zip pkg-config
+          sudo apt-get install -y ngdevkit-toolchain python3 python3-pil python3-ruamel.yaml zip pkg-config
 
       - name: Build ngdevkit
         run: ./.github/scripts/build.sh
 
       - name: Install dependencies for ngdevkit-examples
         run: >
-          sudo apt-get install -y python3-yaml ngdevkit-gngeo python3-pil imagemagick
+          sudo apt-get install -y ngdevkit-gngeo imagemagick
           sox libsox-fmt-mp3 libglew-dev libsdl2-dev
 
       - name: Test by compiling ngdevkit-examples
@@ -73,24 +73,23 @@ jobs:
           msystem: UCRT64
           update: true
           install:
-            git autoconf automake make zip
-            mingw-w64-ucrt-x86_64-pkg-config mingw-w64-ucrt-x86_64-python
-            mingw-w64-ucrt-x86_64-python-pygame
+            git autoconf automake make zip rsync pactoys
+            mingw-w64-ucrt-x86_64-pkg-config
+            mingw-w64-ucrt-x86_64-python
+            mingw-w64-ucrt-x86_64-python-pillow
+            mingw-w64-ucrt-x86_64-python-ruamel-yaml
 
       - name: Install ngdevkit dependencies
         run: |
           echo -e "[ngdevkit]\nSigLevel = Optional TrustAll\nServer = https://dciabrin.net/msys2-ngdevkit/\$arch" >> /etc/pacman.conf
           pacman -Sy
           pacman -S --disable-download-timeout --noconfirm mingw-w64-ucrt-x86_64-ngdevkit-toolchain
-
       - name: Build
         run: ./.github/scripts/build-msys2.sh
 
       - name: Install dependencies for ngdevkit-examples
         run: |
-          pacman -S --noconfirm pactoys rsync
-          pacboy -S --noconfirm python:u python-pillow:u toolchain:u zlib:u sox:u imagemagick:u
-          pip install python-yaml
+          pacboy -S --noconfirm ngdevkit-gngeo:u toolchain:u zlib:u sox:u imagemagick:u
 
       - name: Test by compiling ngdevkit-examples
         run: ./.github/scripts/test.sh
@@ -112,14 +111,14 @@ jobs:
           brew install python3 --overwrite
           brew install pillow
           brew install automake ngdevkit-toolchain zip pkg-config sdl2 sdl2_image
+          $(brew --prefix python)/libexec/bin/pip install ruamel.yaml --break-system-packages
 
       - name: Build
         run: ./.github/scripts/build.sh
 
       - name: Install dependencies for ngdevkit-examples
         run: |
           brew install ngdevkit-gngeo imagemagick sox glew make
-          $(brew --prefix python)/libexec/bin/pip install pyyaml --break-system-packages
 
       - name: Test by compiling ngdevkit-examples
         run: ./.github/scripts/test.sh

README-linux.md

@@ -5,7 +5,7 @@
 In order to build the devkit you need autoconf, autoconf-archive and
 GNU Make 4.x, and various additional dependencies to build the
 toolchain modules such as GCC and SDCC. The devkit tools uses Python 3
-and various dependencies (such as pillow and PyYAML).
+and various dependencies (such as pillow and yaml).
 The emulator requires SDL 2 and optionally OpenGL libraries.
 
 Note that older version of the devkit and its example ROMs depended
@@ -19,25 +19,26 @@ For example, on a recent Ubuntu version, you can install the dependencies with:
 
     apt-get install autoconf autoconf-archive automake gcc curl zip unzip
     apt-get install libsdl2-dev
-    apt-get install python3-pil
+    apt-get install python3-pil python3-ruamel.yaml
     apt-get install libreadline-dev
-    GCC_VERSION_PKG=$(apt-cache depends gcc | awk '/Depends.*gcc/ {print $2}')
-    # make sure you have src packages enabled for dependency information
-    echo "deb-src http://archive.ubuntu.com/ubuntu/ noble main restricted" > /etc/apt/sources.list.d/ngdevkit.list
-    echo "deb-src http://archive.ubuntu.com/ubuntu/ noble universe" >> /etc/apt/sources.list.d/ngdevkit.list
+    DISTRIB=$(. /etc/os-release; echo $VERSION_CODENAME)
+    echo "deb-src http://archive.ubuntu.com/ubuntu/ $DISTRIB main restricted" > /etc/apt/sources.list.d/ngdevkit.list
+    echo "deb-src http://archive.ubuntu.com/ubuntu/ $DISTRIB universe" >> /etc/apt/sources.list.d/ngdevkit.list
     apt-get update
     # install build-dependency packages
+    # make sure you have src packages enabled for dependency information
+    GCC_VERSION_PKG=$(apt-cache depends gcc | awk '/Depends.*gcc/ {print $2}')
     apt-get build-dep $GCC_VERSION_PKG
     apt-get build-dep --arch-only sdcc
     # optional: install GLEW for OpenGL+GLSL shaders in GnGeo
     apt-get install libglew-dev
     # dependencies for the example ROMs
     apt-get install imagemagick sox libsox-fmt-mp3
 
-Debian users (for example Buster) would install the dependencies above
+Debian users (for example Trixie) would install the dependencies above
 by updating the src packages URL as follows:
 
-    echo "deb-src http://deb.debian.org/debian trixie main" > /etc/apt/sources.list.d/ngdevkit.list
+    echo "deb-src http://deb.debian.org/debian $DISTRIB main" > /etc/apt/sources.list.d/ngdevkit.list
     # the remaining dependencies are the same
 
 ## Building the toolchain

README-macos.md

@@ -25,7 +25,7 @@ first in your PATH.
 In order to build the devkit you need autoconf, autoconf-archive and
 GNU Make 4.x, and various additional dependencies to build the
 toolchain modules such as GCC and SDCC. The devkit tools uses Python 3
-and various dependencies (such as pillow and PyYAML). The emulator
+and various dependencies (such as pillow and yaml). The emulator
 requires SDL 2 and optionally OpenGL libraries.
 
 The examples require ImageMagick for all the graphics
@@ -44,6 +44,8 @@ Then, ngdevkit's dependencies are installed as follows:
     brew update
     brew install gmake
     brew install python3 pillow
+    # you need a package from pip, either in a virtual env or system-wide, e.g.:
+    # $(brew --prefix python)/libexec/bin/pip install ruamel.yaml --break-system-packages
     brew deps gcc | xargs brew install
     brew deps sdcc | xargs brew install
     # dependencies for the example ROMs

README-msys2.md

@@ -63,7 +63,7 @@ ngdevkit dependencies:
     pacboy -S autoconf autoconf-archive automake pkgconf make tar zip unzip
     pacboy -S git flex bison expat gettext ncurses-devel zlib-devel
     pacboy -S gmp-devel isl-devel mpc-devel mpfr-devel texinfo
-    pacboy -S python:u python-yaml python-pillow:u toolchain:u zlib:u
+    pacboy -S python:u python-ruamel-yaml:u python-pillow:u toolchain:u zlib:u
     # dependencies for the example ROMs
     pacboy -S rsync sox:u imagemagick:u SDL2:u glew:u boost:u
 

README.md

@@ -55,17 +55,17 @@ of dependencies for the examples ROMs:
 
     add-apt-repository -y ppa:dciabrin/ngdevkit
     apt update
-    apt install ngdevkit ngdevkit-gngeo
+    apt install ngdevkit
     # the remaining packages are only required for the examples
-    apt install pkg-config autoconf zip imagemagick sox libsox-fmt-mp3
+    apt install pkg-config autoconf zip imagemagick sox libsox-fmt-mp3 ngdevkit-gngeo
 
 If you are running a Fedora distribution, pre-built packages are
 available in COPR, and can be installed with the following commands:
 
     dnf copr enable dciabrin/ngdevkit
-    dnf install ngdevkit ngdevkit-gngeo
+    dnf install ngdevkit
     # the remaining packages are only requred for the examples
-    dnf install install pkg-config autoconf zip ImageMagick sox
+    dnf install install pkg-config autoconf zip ImageMagick sox ngdevkit-gngeo
 
 #### macOS
 
@@ -78,9 +78,9 @@ packages, available in the ngdevkit tap:
     eval $(/opt/homebrew/bin/brew shellenv)
     # install ngdevkit
     brew tap dciabrin/ngdevkit
-    brew install ngdevkit ngdevkit-gngeo
+    brew install ngdevkit
     # the remaining packages are only required for the examples
-    brew install pkg-config autoconf automake zip imagemagick sox
+    brew install pkg-config autoconf automake zip imagemagick sox ngdevkit-gngeo
 
 Brew bottles are available for the last two versions of macOS for ARM.
 Intel brew bottles are no longer prebuilt. Older versions of macOS are
@@ -98,9 +98,9 @@ configure the pacman repository as follows:
     echo -e "\n[ngdevkit]\nSigLevel = Optional TrustAll\nServer = https://dciabrin.net/msys2-ngdevkit/\$arch" >> /etc/pacman.conf
     # install pacboy with `pacman -S pactoys` if necessary
     pacboy -Sy
-    pacboy -S ngdevkit:u ngdevkit-gngeo:u
+    pacboy -S ngdevkit:u
     # the remaining packages are only required for the examples
-    pacboy -S autoconf automake make zip imagemagick:u sox:u
+    pacboy -S autoconf automake make zip imagemagick:u sox:u ngdevkit-gngeo:u
 
 An old version of ngdevkit supported Windows 10 via [WSL][wsl], but it
 is now deprecated in favour of the native MSYS2 environment. Likewise,

tools/soundtool.py

@@ -20,11 +20,39 @@
 import argparse
 import re
 import sys
-import yaml
 import glob
 import os
 import wave
 
+# Special handling for YAML: try to import ruamel.yaml first,
+# because it's better at keeping comments in YAML files, and
+# default to yaml if not found, as they are API-compatible.
+try:
+    import ruamel.yaml
+    _yaml = "ruamel"
+except ImportError:
+    import yaml
+    _yaml = "PyYAML"
+
+def yaml_load_all(f):
+    if _yaml == "ruamel":
+        return ruamel.yaml.YAML().load_all(f)
+    else:
+        return yaml.safe_load_all(f)
+
+def yaml_load(f):
+    if _yaml == "ruamel":
+        return ruamel.yaml.YAML().load(f)
+    else:
+        return yaml.safe_load(f)
+
+def yaml_dump(data, end, f):
+    if _yaml == "ruamel":
+        return ruamel.yaml.YAML().dump(data, stream=f)
+    else:
+        return yaml.dump(data, end=end, stream=f)
+
+
 VERBOSE = False
 
 
@@ -44,7 +72,7 @@ def load_sound_desc(descfile):
     descdata = open(descfile,'r').read()
     comments = [l for l in descdata.split('\n') if re.search(r'^#', l)]
     header = "\n".join(comments)
-    yamldata = sum(list(yaml.safe_load_all(descdata)), [])
+    yamldata = sum(list(yaml_load_all(descdata)), [])
     return header+"\n", yamldata
 
 
@@ -117,7 +145,7 @@ def detect_adpcm_type(f):
 
     # save sound map description
     print(header, end='', file=output)
-    print(yaml.dump(result_data), end='', file=output)
+    print(yaml_dump(result_data), end='', file=output)
 
 
 
@@ -405,7 +433,7 @@ def main():
     # Otherwise load a sound map from the input yaml file
     for smap in arguments.soundmap:
         with open(smap, 'r') as f:
-            desc.extend(yaml.safe_load(f.read()))
+            desc.extend(yaml_load(f.read()))
 
     # generate z80 macros
     if arguments.action == 'z80':

tools/vromtool.py

@@ -27,7 +27,22 @@
 import wave
 from adpcmtool import ym2610_adpcma, ym2610_adpcmb
 
-import yaml
+# Special handling for YAML: try to import ruamel.yaml first,
+# because it's better at keeping comments in YAML files, and
+# default to yaml if not found, as they are API-compatible.
+try:
+    import ruamel.yaml
+    _yaml = "ruamel"
+except ImportError:
+    import yaml
+    _yaml = "PyYAML"
+
+def yaml_load_all(f):
+    if _yaml == "ruamel":
+        return ruamel.yaml.YAML().load_all(f)
+    else:
+        return yaml.safe_load_all(f)
+
 
 VERBOSE = False
 
@@ -162,7 +177,7 @@ def load_sample_map_file(filenames):
     for filename in filenames:
         ysamples = []
         with open(filename, "rb") as f:
-            yamlblocks = list(yaml.load_all(f, yaml.Loader))
+            yamlblocks = list(yaml_load_all(f))
         for b in yamlblocks:
             ysamples.extend(b)
         all_ysamples.extend([filename, y] for y in ysamples)