Use of defines in reproducible builds

[RobTillaart]

Often builds have command line defines

As these defines are part of the build they must be includable in the profile. e.g.

profiles:
  nanorp:
    fqbn: arduino:mbed_nano:nanorp2040connect
    platforms:
      - platform: arduino:mbed_nano (2.1.0)
    libraries:
      - ArduinoIoTCloud (1.0.2)
    defines:
      - SIZE: 128
  
  other:
    fqbn: ...
    platforms:
      - platform: ....
    libraries:
      - ....
    defines:
      - SIZE: 64

If the user defines the define on the command line, it overrules the value in the profile

arduino-cli compile [sketch] --DSIZE=256--profile myProject

[cmaglie]

Yep, this opens another can of worm :-) but we are definitely going to implement this one in a later revision of the profiles.

The main issue here is that there isn't a common pattern in the platforms.txt to add the extra -D properties, so we should probably create a new platofrm.txt "variable" for this specific purpose and wait for various 3rd party cores maintainers to adopt it (BTW they revealed to be pretty quick, so this may get through in a reasonable time).

[DeeEmm]

The main issue here is that there isn't a common pattern in the platforms.txt to add the extra -D properties,

Is this not already defined within the CLI as custom board options as a parameter of the FQBN?

All of these parameters follow typical key->value format so it should be trivial to allow these within the platforms.txt

i.e.

profiles:
  ESPduino:
    fqbn: esp32:esp32:esp32
        FlashMode qio
        UploadSpeed 115200
        PartitionScheme no_ota

With strict YAML conventions, indenting makes these parameters a subset of the fqbn, so we do not need to define additional variables to store them in unless we need to access these parameters publicly.

so we should probably create a new platofrm.txt "variable" for this specific purpose and wait for various 3rd party cores maintainers to adopt it (BTW they revealed to be pretty quick, so this may get through in a reasonable time).

using the above format is transparent to third party maintainers as it follows the current convention of adding parameter values to the fqbn. It also degrades nicely to the current system as using no platform.txt file just falls back to the current way of doing things.

In addition to this, it should be possible to do the following in addition to the above,

profiles:
  ESPduino:
    fqbn: esp32:esp32:esp32:FlashMode=qio:UploadSpeed=115200:PartitionScheme=no_ota

Which is essentially the same as we currently have. It also make an easy interim solution and should be dead simple to implement

[egnor]

This effort appears to be stalled? It's a quite commonly needed feature:

• Command Line -D defines Arduino#6344
• Build flags set in my boards.txt file not used in the compilation command arduino-cli#210
• Please add a way to include project wide #defines within the IDE arduino-ide#1837
• https://forum.arduino.cc/t/arduino-cli-pass-flags-to-sketch/680474
• https://forum.arduino.cc/t/how-to-add-compile-options-to-arduino-ide-1-8-13/692640/5
• https://forum.arduino.cc/t/how-to-specify-unique-compiling-options-for-multiple-sketches/679370
• https://forum.arduino.cc/t/ide-to-command-line-build-support-d/682481/4
• https://forum.arduino.cc/t/need-mechanism-to-pass-defines-to-library-cpp-files/227179
• https://forum.arduino.cc/t/need-to-set-a-define-in-a-sketch-so-a-library-can-compile-correctly/140113
• https://forum.arduino.cc/t/using-a-macro-defined-in-the-ino-file-from-a-custom-library/1118529
• https://forum.arduino.cc/t/using-build-property-to-set-optimize-and-define-symbols/1254465/4
• https://ecampusontario.pressbooks.pub/rwsnotes/back-matter/advanced-arduino-ide-configuration/
• the MANY references in Provide dedicated sets of "extra flags" properties for platform developers and users arduino-cli#846
• and all the references above in this issue

Major use cases include

• compile-time configuration of libraries (choose features, set buffer size, pins, etc)
• compile-time configuration of platform code (ditto)
• multiple variants of application firmware (esp in sketch profiles)
• switching debug/release code, and compiling out debug code in production
• tweaking logging parameters, minimizing runtime impact when logging is off
• specifying secrets without checking them into a repository
• etc etc

Workarounds that are commonly offered, and why they're sad

• use board options, via IDE menus or as an add-on to the FQBN.
  • however, these can only be options anticipated by the board author and with an enumerated set of choices, so it's not helpful for most of the use cases above
• use --pref (arduino IDE) or --build-properties) (arduino CLI) to set build.extra_flags=-DMYSYMBOL=myvalue or similar.
  • but, the properties to use are not well defined across platform types, you have to learn the "hooks" any given platform/board offers. furthermore, this extra command line has to be remembered somewhere-- do you check in a script to start the IDE with the right flags??
• use a custom board file, perhaps board.local.txt (or use platform.local.txt) which defines the symbol.
  • but this can't be placed with the sketch that uses it, it has to be stashed in your Arduino/hardware directory, it's subject to get moved or ignored whenever a board or platform file is upgraded, it's hard to check into source control, and it's also platform/board-specific
• use #define in your .ino (or .cpp) file.
  • but this doesn't take effect for anything not in a header file being included right there, which is most of the use cases above

(Notably, I have never met a build environment that doesn't let you set defines on a project level before. Admittedly Arduino is coming from a bit of a different perspective, prioritizing ease of getting started above all else. And people do find workarounds. But still it's pretty silly, feels like it should be as easy as adding an entry to sketch.yaml.)

And yes, this requires some way to add some consistent way for platform/build definitions to handle defines, with a consistent syntax. (arduino/arduino-cli#846 is about attacking this subproblem.) Other proposals have suggested "prepend this include file" instead of "define these symbols", which sidesteps some issues but creates others.

[Torkin1]

Maybe I'm not educated enough on the matter, since I started to work on my first Arduino project this week, but this seems a non-problem to me?

arduino-cli already offers the --build-property flag to override build properties defined in the various *.txt files (boards, platform, ...). IMHO, this behaviour should be driven by the sketch.yaml profiles as well to at least keep feature parity between the cli options and the profile specification.

This seems very trivial to me, it should be enough to extract the value from the profile and pass it to the --build-property flag handler.

I reckon that this is not an ideal way to allow users to tamper with the build internals, but at least allows power users to define reproducible builds. Plans like the one described in arduino/arduino-cli#355 can be advanced as further improvement on this matter. But the problem addressed here can be delivered much earlier since it seems like only a matter of feature parity.