replace.xlsx and KeyError: (['USA'], ['SFH'], ['RES2.1'])

[CBreton026]

Hi everyone,
I've been trying to launch BuildME for Canadian archetypes (based on US proxies):

combinations = \
    {
        'CA':
            {'occupation': ['RT', 'MFH', 'SFH'],
             'energy standard': ['non-standard', 'standard', 'efficient', 'ZEB'],
             'RES': ['RES0', 'RES2.1', 'RES2.2', 'RES2.1+RES2.2'],
             'climate_region':
                 ['CA5A', 'CA6A', 'CA7'],
             'climate_scenario': ['2015'],
             'cooling': ['HVAC'] 
             }
  
    }

As discussed in #48, I replaced all climate files using publicly available ones from the US DOE. The calculations start to run, but then:

1. simulate.py lines 129 and 130 throw the following warning : FutureWarning: The behavior of indexing on a MultiIndex with a nested sequence of labels is deprecated and will change in a future version. series.loc[label, sequence] will raise if any members of 'sequence' or not present in the index's second level. To retain the old behavior, use series.index.isin(sequence, level=1)

2. As soon as an invalid combination is called by simulate.py lines 129 or 130, a KeyError is thrown (see below). I received no errors for RT, but I did receive some for MFH and SFH (e.g code below)

Traceback (most recent call last):

  File ".\BuildME\main.py", line 21, in <module>
    simulate.copy_scenario_files(simulation_files, run)

  File ".\BuildME\BuildME\simulate.py", line 181, in copy_scenario_files
    idf_f = apply_rule_from_excel(idf_f, res_str, res_replace, mmv_en_replace)

  File ".\BuildME\BuildME\simulate.py", line 129, in apply_rule_from_excel
    xls_values1 = en_replace.loc(axis=0)[[res[0]], [res[1]], [res[2]]]

  File "~\.conda\envs\BuildME\lib\site-packages\pandas\core\indexing.py", line 961, in __getitem__
    return self._getitem_tuple(key)

  File "~\.conda\envs\BuildME\lib\site-packages\pandas\core\indexing.py", line 1140, in _getitem_tuple
    return self._getitem_lowerdim(tup)

  File "~\.conda\envs\BuildME\lib\site-packages\pandas\core\indexing.py", line 839, in _getitem_lowerdim
    return self._getitem_axis(tup, axis=axis)

  File "~\.conda\envs\BuildME\lib\site-packages\pandas\core\indexing.py", line 1195, in _getitem_axis
    locs = labels.get_locs(key)

  File "~\.conda\envs\BuildME\lib\site-packages\pandas\core\indexes\multi.py", line 3401, in get_locs
    indexer = _update_indexer(indexers, indexer=indexer)

  File "~\.conda\envs\BuildME\lib\site-packages\pandas\core\indexes\multi.py", line 3327, in _update_indexer
    raise KeyError(seq)

KeyError: (['USA'], ['SFH'], ['RES2.1'])

A simple fix was to add "Dummy" combinations in the replace.xlsx file, by providing a region / occupation / RES / idfobject / Name, and making sure that "idfobject" was not in "Name". However, this seems a little odd.

• Is there a reason why only some combinations are available in the replace.xlsx file?
• Is "simulate.py" working as intended, or is the KeyError caused by changes in how ".loc" does its indexing?

I'm curious to know if any of you have encountered this error before. If not, I can look into it to fix 1. and 2.
Thanks!

[nheeren]

Hi @CBreton026. Thanks for your active interaction with BuildME. I will check if I can reproduce these errors the next days. In short the explanation is probably that we are currently developing a number of things and pushed a few things directly on master. Probably that way we also broke some parts, such as replace.xlsx.

[CBreton026]

Hi @nheeren, thank you for your quick reply! That could be the case; here is the replace.xlsx file that I modified. Before adding the red text on the 'RES' tab, I would encounter the KeyError.
replace.xlsx

[kamilakrych]

Hi! I just tested it, indeed we get an error for anything other than RES0, no matter the region (USA gives the same error). The reason is that the sheet 'RES' in replace.xlsx doesn't contain anything other than RES0.

This is a bug that must have come from loooong ago. I have always used just RES0 so I never realized this... The solution is to add more lines in this sheet (to contain other RES), or to change the code, so that it skips this line of code if no matches are found in the excel sheet.

@nheeren I would suggest that you decide, I'm not really aware why we're changing the number of people for different RES in the first place...

[nheeren]

Sorry, I haven't caught up yet. Just a quick explanation: We added the occupation parameter variable, because it was an important discussion in the selection of the SSP scenarios. We wanted to be able to change that across scenarios, but never did in the end, I believe. The obvious solution is to copy the for all RES scenarios.

As you say @kamilitsa, somebody apparently pushed an incomplete replace.xlsx file at some point.