Skip to content

fix: replace pandas APIs removed in 2.0 that leave four entry points … - #1320

Open
EnvDroneSense wants to merge 1 commit into
CLIMADA-project:developfrom
EnvDroneSense:fix/1319-pandas2-removals
Open

fix: replace pandas APIs removed in 2.0 that leave four entry points …#1320
EnvDroneSense wants to merge 1 commit into
CLIMADA-project:developfrom
EnvDroneSense:fix/1319-pandas2-removals

Conversation

@EnvDroneSense

Copy link
Copy Markdown

Changes proposed in this PR:

  • Replace the four calls to DataFrame.append and Series.iteritems, removed in pandas 2.0, that make their enclosing functions raise AttributeError: litpop.py:640, impact_data.py:422, calibration_opt.py:406 and calibration_opt.py:94.
  • Add one regression test per site. None of the four lines was previously executed by the test suite.

This PR fixes #1319 and #826

The affected entry points are LitPop.from_shape_and_countries (with a GeoSeries or list shape), impact_data.hit_country_per_hazard, calibration_opt.calib_all, and calibration_opt.calib_instance (with a multi-row df_out and yearly_impact=True). pyproject.toml declares pandas with no version constraint, so a fresh install always resolves to an affected version.

litpop.py:640 is the cause of #826; that issue's traceback names this line, and its reporter proposed the same patch.

The fix differs by site. litpop.py and calibration_opt.py:406 accumulate DataFrames, so they collect into a list and pd.concat once, which also removes the quadratic copy the loops paid. impact_data.py:422 appends a dict rather than a frame, so it collects records and builds one pd.DataFrame(records, columns=[...]). calibration_opt.py:94 is a rename.

Two behaviour details worth review:

  • calib_all returns the single frame directly when there is exactly one parameter combination. The old code deep-copied the first df_out and only applied ignore_index=True from the second onward, so a caller-supplied impact_data_source index survived; pd.concat(..., ignore_index=True) alone would silently replace it.
  • In litpop.py the type check moved out of the loop, ahead of any selection. Previously [Polygon, Point] ran a full point-in-polygon pass for the polygon before raising on the point.

Also calibration_opt.py:406 passed the builtin input as the second positional argument (ignore_index). bool(input) is True, so it happened to mean ignore_index=True.

Each test was checked by reverting its own fix alone, with the other three in place:

reverted                    failing test                          error
litpop.py                   test_geoseries_shape                  'GeoDataFrame' has no attribute 'append'
impact_data.py              test_hit_country_per_hazard_pass      'DataFrame' has no attribute 'append'
calibration_opt.py:406      test_calib_all_multiple_params        'DataFrame' has no attribute 'append'
calibration_opt.py:94       test_calib_instance_yearly_multirow   'Series' has no attribute 'iteritems'

Note there is precedent for this bug class here: test_emdat_impact_yearlysum_no_futurewarning was added when the same .append problem was fixed in emdat_impact_yearlysum. This brings the remaining four sites in line with that.

PR Author Checklist

PR Reviewer Checklist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant