Adding IPOPT as an alternative solver for MAgPIE by georg-schroeter · Pull Request #871 · magpiemodel/magpie

georg-schroeter · 2026-03-13T14:34:01Z

🐦 Description of this PR 🐦

While MAgPIE and all pre- and post-processing libraries are open source, executing the MAgPIE core model so far relies on CONOPT, a proprietary solver inside the GAMS framework. This isn't ideal in several ways:

executing MAgPIE currently requires both a GAMS and a CONOPT4 license, raising the barrier for external users to use the model or validate results.
having only one solver available makes the validation of numerical results difficult, since we can't easily distinguish between changes in the model and the solver behavior.
relying on a single solver during development contributes to a multi-faceted lock-in effect, where e.g. problematic formulations are "ok" if the particular solver can resolve them, or potential multiple optima aren't discovered if the solver reliably picks the same local optimum.

This PR tries to alleviate parts of this by adding the option to solve MAgPIE with the open source nlp solver IPOPT instead of the proprietory CONOPT4. Note however that this isn't (yet) an equivalent drop-in solution without any downsides. First, this required several changes in the core model and post-processing formulations (see #869). Second, while we ultimately got it to work properly, it takes much longer than CONOPT4 (by about a factor 12x right now) which can be partially attributed to CONOPT's extensive preprocessing of the model, cutting its size by about 60% after removing the pre- and post-triangular parts. We might be able to cut that in the future by improving the model formulation, but it might also be increased further by future changes in the input our output related parts.

🔧 Checklist for PR creator 🔧

If a point is not applicable, check the checkbox anyway and write "non-applicable" next to the checkbox.

Label pull request from the label list.
- Low risk: Simple bugfixes (missing files, updated documentation, typos) or changes in start or output scripts
- Medium risk: Uncritical changes in the model core (e.g. moderate modifications in non-default realizations)
- High risk: Critical changes in model core or default settings (e.g. changing a model default or adjusting a core mechanic in the model)
Self-review own code
- No hard coded numbers and cluster/country/region names.
- The new code doesn't contain declared but unused parameters or variables.
- magpie4 R library has been updated accordingly and backwards compatible where necessary.
- scenario_config.csv has been updated accordingly (important if default.cfg has been updated)
Document changes
- Add changes to CHANGELOG.md
- Where relevant, put In-code documentation comments
- Properly address updates in interfaces in the module documentations
- run goxygen::goxygen() and verify the modified code is properly documented
Perform test runs
- Low risk:
  - Run a compilation check via Rscript start.R --> "compilation check"
- Medium risk:
  - Run default run via Rscript start.R --> "default"
  - Check logs for errors/warnings
  - Fill in performance section below
- High risk:
  - Run test runs via Rscript start.R --> "test runs"
  - Check logs for errors/warnings
  - Default run from the PR target branch for comparison
  - Provide relevant comparison plots (land-use, emissions, food prices, land-use intensity,...)
  - Fill in performance section below
Reporting produces no errors and no new warnings
Get two approving reviews (at least one from RSE)

📉 Performance 📈

This PR's default : 29.1 min (CONOPT), 416 min (IPOPT)
For comparison: runtimes of weekly test: 35.2 min

🚨 Checklist for reviewer 🚨

PR is labeled correctly
Code changes look reasonable
- No hard coded numbers and cluster/country/region names.
- No unnecessary increase in module interfaces
- model behavior/performance is satisfactory.
Changes are properly documented
- CHANGELOG is updated correctly
- Updates in interfaces have been properly addressed in the module documentations
- In-code documentation looks appropriate

georg-schroeter · 2026-03-18T16:28:22Z

Now to the actual interesting part, the comparison plots between CONOPT and IPOPT solves. Separate post to keep the top about the technical changes and don't clutter the PR template.

The "usual" tau and area plots:

Productivity_Landuse_Intensity_Indicator_Tau(32)

Resources_Land_Cover_Pastures_and_Rangelands(15)

Emissions:

Costs and food expenditure shares:

Prices:

Prices_Index2020_Agriculture_Producer_All_products

same plot, but starting in 2015:

Prices_Index2020_Agriculture_Producer_All_products(1)

Prices_Index2020_Agriculture_Producer_Sugar_cane

Prices_Index2020_Agriculture_Producer_Temperate_cereals

Prices_Index2020_Agriculture_Producer_Maize

georg-schroeter · 2026-03-18T18:36:42Z

Observations from the plots:

Pasture (and forest/other land as a side effect because total area is constant) dynamics show the only visible difference in the "usual" plots, while croparea and tau are identical. The difference in pasture area actually increases with higher numerical tolerance for IPOPT, which in my eyes points towards an issue with the implementation due to the lack of a clear driver:

Resources_Land_Cover_Pastures_and_Rangelands(14)

Comparing recent release versions / weekly tests shows a somewhat similar pattern, where pasture area is much more volatile than other areas between input data revisions and scenarios.
From inspecting the code around pasture, the issue might be that in the absence of a strong demand for pasture in any timestep, there isn't really a difference between converting pasture or other land to cropland/forest, while there is a steep cost associated with establishing pasture when needed again.
The other observation is around prices and price indices. As can be seen from the plots, prices (especially producer prices) are reproducible by IPOPT whenever the production volume is high enough, however our current postprocessing and reporting seems to rely a bit too much on CONOPT's behavior for low volume commodities. This is especially the case for aggregated price indices and for the historical period (which I left out in most of the above plots because they're way too high), due to what I assume as low/no croparea clusters. Before the postprocessing/reporting isn't made more robust to extreme prices for low volume commodities, price indices can't be used from IPOPT runs.

In total I have to say that I'm positively surprised how little differences there are, which I think is great news :)

pascal-sauer

Seeing the code duplication now I think we need a better solution for the food demand model.

modules/80_optimization/nlp_ipopt/input.gms

pascal-sauer · 2026-03-19T08:19:43Z

modules/80_optimization/nlp_ipopt/not_used.txt

@@ -0,0 +1,2 @@
+name,type,reason
+vm_landdiff,input,questionnaire


This was just copied from the other realization (nlp_apr17). Does anyone know where this is coming from and if it's still needed?

The reason is that the lp_nlp_apr17 realization minimizes vm_landdiff instead of vm_cost_glo in its second solve while respecting the vm_cost_glo optimum as an upper bound.

modules/80_optimization/nlp_ipopt/realization.gms

modules/80_optimization/nlp_ipopt/solve.gms

modules/80_optimization/nlp_ipopt/input.gms

Co-authored-by: Pascal Sauer <156898545+pascal-sauer@users.noreply.github.com>

georg-schroeter · 2026-03-26T15:43:44Z

The implementation settings were simplified, and unused options (additional optfile, conditional solve settings) removed.

The duplication of the food demand model with ipopt as a solver was reverted, instead an upcoming rewrite of the module will change the food demand model realization to use the general optimization settings.

pascal-sauer

Nice and simple PR, looks good! Thanks @georg-schroeter !

modules/80_optimization/nlp_ipopt/solve.gms

pascal-sauer · 2026-03-27T09:32:25Z

modules/80_optimization/nlp_ipopt/solve.gms

+);
+
+p80_modelstat(t) = magpie.modelstat;
+p80_num_nonopt(t) = magpie.numNOpt;


What is happening here, what are these variables? I assume this was copied from the conopt realization, so maybe @flohump knows?

Okay found it in the gams docs:

numNOpt (integer): Number of nonoptimalities Available: Attribute statement (use after solve) This model attribute returns the number of nonoptimalities after a solve.

A quick search on github indicates that we never check p80_num_nonopt or do anything with it, do we actually need? If p80_num_nonopt > 0 then this would be reflected in p80_modelstat, no?

From inspecting the full.lst of several IPOPT runs, "nonopt" entries do appear, and can e.g. happen when values slightly out of bound are moved in-bound again, causing a solved equation to be slightly above the given tolerance which makes them count as non-optimal.

Okay, so you think there's enough upside to keep this in the code? My tendency is rather to delete if in doubt, but no strong feelings here

pascal-sauer

How would the user switch to optfile2? Would that realistically ever be done? My feeling is no, so I'd much prefer to agree on/find the one configuration that works best

pascal-sauer · 2026-04-01T12:35:27Z

modules/80_optimization/nlp_ipopt/solve.gms

+put 'quality_function_max_section_steps 4' /;
+put 'nlp_scaling_method gradient-based' /;
+put 'nlp_scaling_max_gradient 100' /;
+put 'acceptable_tol 1e-6' /;


Huh, I wasn't even aware of the acceptable concept/feature of Ipopt. Did someone (not a LLM) think this through? This only makes sense if acceptable_tol > tol, right? Our usual tol (which is feasible) is 1e-7, so 1e-6 does not make sense here. Also, 1e-6 is the default, I'd not specify default settings explicitly unless there is a reason for it.

I'm in favor of dropping the acceptable stuff

Our tol is 1e-8 so considerably lower. "Solved to acceptable level" followed by a "warm start" from that point actually happened sometimes with my other IPOPT runs as well. Since it's the default value, it should be removed anyway.

Co-authored-by: Pascal Sauer <156898545+pascal-sauer@users.noreply.github.com>

georg-schroeter · 2026-04-02T07:25:39Z

More Plots, now also comparing the two optfiles. Generally speaking, IPOPT with optfile 2 seems to have the better agreement with the CONOPT results, likely due to adaptive mu picking lower mu values.

Resources_Land_Cover_Cropland_Croparea_Bioenergy_crops

Resources_Land_Cover_Pastures_and_Rangelands(16)

Productivity_Landuse_Intensity_Indicator_Tau(33)

Prices_Index2020_Agriculture_Food_products

Resources_Timber_operations_Harvested_area_for_timber_Total

Prices_Index2020_Agriculture_Producer_All_products(2)

pascal-sauer · 2026-04-02T09:09:19Z

Not related to this PR, but It's really hard to see the red line in the plots. Maybe we should think about making the lines a little transparent, so it's easier to see overlaps?

pascal-sauer

I suggest some cleanup for optfile2, but good to go otherwise :)

pascal-sauer · 2026-04-02T11:56:16Z

modules/80_optimization/nlp_ipopt/solve.gms

 put 'quality_function_max_section_steps 4' /;
 put 'nlp_scaling_method gradient-based' /;
 put 'nlp_scaling_max_gradient 100' /;
-put 'acceptable_tol 1e-6' /;


The other acceptable_* should also be deleted, no?

pascal-sauer · 2026-04-02T11:56:48Z

modules/80_optimization/nlp_ipopt/solve.gms

@@ -50,9 +47,6 @@ put 'honor_original_bounds yes' /;
 put 'max_iter 10000' /;
 put 'linear_solver mumps' /;


This is the default, no need to specify

pascal-sauer · 2026-04-02T11:57:00Z

modules/80_optimization/nlp_ipopt/solve.gms

@@ -50,9 +47,6 @@ put 'honor_original_bounds yes' /;
 put 'max_iter 10000' /;


Is this relevant enough to keep?

pascal-sauer · 2026-04-02T11:58:46Z

modules/80_optimization/nlp_ipopt/solve.gms

@@ -40,7 +38,6 @@ put 'mu_oracle quality-function' /;
 put 'quality_function_max_section_steps 4' /;
 put 'nlp_scaling_method gradient-based' /;


Delete as this is the default

pascal-sauer · 2026-04-02T11:59:04Z

modules/80_optimization/nlp_ipopt/solve.gms

@@ -40,7 +38,6 @@ put 'mu_oracle quality-function' /;
 put 'quality_function_max_section_steps 4' /;
 put 'nlp_scaling_method gradient-based' /;
 put 'nlp_scaling_max_gradient 100' /;


Delete as this is the default

pascal-sauer · 2026-04-02T12:01:39Z

Oh and as I was heavily involved in this PR I think we should get another RSE review, can you have a look @tscheypidi ?

georg-schroeter added 5 commits March 13, 2026 14:11

created ipopt realizations for 80_optimization and 15_food

98594e7

reverted solprint change

3dc9045

CHANGELOG.md

2672af5

choose ipopt food demand realization in ipopt start script

19c8a46

default title in ipopt.R start script

f911a5f

georg-schroeter added enhancement New feature or request Medium risk labels Mar 13, 2026

georg-schroeter added 2 commits March 18, 2026 09:40

Merge branch 'develop' into ipopt_part2

53efb6c

magpie4 version required for more robust reporting

e42e13c

georg-schroeter marked this pull request as ready for review March 18, 2026 16:28

pascal-sauer reviewed Mar 19, 2026

View reviewed changes

georg-schroeter and others added 6 commits March 20, 2026 12:15

Apply suggestions from code review

de19c18

Co-authored-by: Pascal Sauer <156898545+pascal-sauer@users.noreply.github.com>

Merge branch 'develop' into ipopt_part2

3e183d0

Remove unused options from code review

0879009

Co-authored-by: Pascal Sauer <156898545+pascal-sauer@users.noreply.github.com>

deduplicate code by removing the ipopt food demand realization again.

2972a23

automatically generated files

22c162b

remove unused options from nlp_ipopt realization

dc8c126

georg-schroeter requested review from flohump and pascal-sauer March 26, 2026 15:41

pascal-sauer approved these changes Mar 27, 2026

View reviewed changes

added alternative optfile

34595e2

pascal-sauer requested changes Apr 1, 2026

View reviewed changes

georg-schroeter and others added 2 commits April 1, 2026 14:44

Apply suggestions from code review

4dec88f

Co-authored-by: Pascal Sauer <156898545+pascal-sauer@users.noreply.github.com>

removed setting default options, added optfile switch to start script

2dc5e4e

pascal-sauer approved these changes Apr 2, 2026

View reviewed changes

pascal-sauer requested a review from tscheypidi April 2, 2026 12:02

		@@ -0,0 +1,2 @@
		name,type,reason
		vm_landdiff,input,questionnaire

		@@ -50,9 +47,6 @@ put 'honor_original_bounds yes' /;
		put 'max_iter 10000' /;
		put 'linear_solver mumps' /;

		@@ -40,7 +38,6 @@ put 'mu_oracle quality-function' /;
		put 'quality_function_max_section_steps 4' /;
		put 'nlp_scaling_method gradient-based' /;

Conversation

georg-schroeter commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐦 Description of this PR 🐦

🔧 Checklist for PR creator 🔧

📉 Performance 📈

🚨 Checklist for reviewer 🚨

Uh oh!

georg-schroeter commented Mar 18, 2026

Uh oh!

georg-schroeter commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pascal-sauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

georg-schroeter commented Mar 26, 2026

Uh oh!

pascal-sauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pascal-sauer Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pascal-sauer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georg-schroeter commented Apr 2, 2026

Uh oh!

pascal-sauer commented Apr 2, 2026

Uh oh!

pascal-sauer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pascal-sauer commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

georg-schroeter commented Mar 13, 2026 •

edited

Loading

georg-schroeter commented Mar 18, 2026 •

edited

Loading

pascal-sauer Mar 27, 2026 •

edited

Loading

pascal-sauer commented Apr 2, 2026 •

edited

Loading