Skip to content

fix(migrations): keep fields whose source_field matches a field name in CreateModel - #2303

Open
belomaxorka wants to merge 1 commit into
tortoise:developfrom
belomaxorka:fix/migrations-inherited-fields-2251
Open

belomaxorka wants to merge 1 commit into
tortoise:developfrom
belomaxorka:fix/migrations-inherited-fields-2251

Conversation

@belomaxorka

Copy link
Copy Markdown

Description

MigrationWriter._format_create_model() skipped every field whose name showed up as some field's source_field. The intent was to drop the generated <fk>_id backing field of a ForeignKeyField, but source_field is just the column name, so a field like

id = fields.BigIntField(primary_key=True, source_field="id")

matched itself and was left out of CreateModel. In #2251 every field of the abstract base declares source_field="<own name>", so they all went missing. Inheritance itself isn't involved: the same thing happens on a plain model.

The writer now skips only the <name>_id backing fields of ForeignKeyField/OneToOneField fields in the operation, which is how Apps._init_relations() names them. A model can't declare its own field with that name (add_field raises on duplicates), so nothing legitimate is dropped. test_writer_excludes_fk_source_field still passes unchanged.

Motivation and Context

Fixes #2251.

With the old writer the first makemigrations produced a CreateModel without those fields. Since there was no PK, the model got the default id IntField. The second run then emitted RemoveField('id') + AddField('id', BigIntField(...)) + AddField for the rest. On PostgreSQL that drops the SERIAL column and adds "id" BIGINT NOT NULL PRIMARY KEY without a sequence, which is the "PK not generated" part of the issue. With the fix the initial migration creates "id" BIGSERIAL NOT NULL PRIMARY KEY, and a second run reports no changes.

The auto_now/auto_now_add part of the issue is not a bug. Since #2104 those flags are Python-side only and emit no DEFAULT, in generate_schemas() too. db_default=Now() is the database-level option.

Migrations that were already generated with the old writer are not rewritten, so tables created through them keep the column without a sequence.

Related: #2195 fixes the same filter for PK fields only (for #2176). This is the more general version suggested in that PR's review. It sidesteps the test_writer_excludes_fk_source_field conflict mentioned there by recognising the backing field by name instead of by source_field. The generated=False serialization part of #2195 is not touched here.

How Has This Been Tested?

  • test_writer_keeps_fields_with_own_source_field: writer output keeps a PK and a non-PK field whose source_field equals the field name.
  • test_autodetector_keeps_fields_with_own_source_field: uses the model shape from the issue (abstract base with source_field fields). It writes the initial migration and checks that a second autodetector run finds no changes. It also renders the migration with the PostgreSQL schema editor (fake client) and checks for "id" BIGSERIAL NOT NULL PRIMARY KEY.
  • Both tests fail on develop and pass with the fix.
  • Generating an initial migration for examples/comprehensive_migrations_project gives identical output before and after.
  • Full SQLite suite: 1966 passed, 154 skipped. ruff format --check, ruff check, bandit and codespell are clean. mypy reports the same 2 errors as on develop, in files not touched here.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added the changelog accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

…in CreateModel

The writer skipped every CreateModel field whose name appeared as some
field's source_field. That was meant to drop the generated "<fk>_id"
backing field of a ForeignKeyField, but it also dropped any field with
source_field equal to its own name (e.g. id = BigIntField(primary_key=True,
source_field="id")). The next makemigrations then added those fields back
and dropped/re-added the primary key, which on PostgreSQL left it without
a sequence.

Skip only the "<name>_id" backing fields of ForeignKeyField/OneToOneField
fields instead.
@codspeed

codspeed Bot commented Oct 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 24 untouched benchmarks


Comparing belomaxorka:fix/migrations-inherited-fields-2251 (4514c15) with develop (5815573)

Open in CodSpeed

@waketzheng

Copy link
Copy Markdown
Contributor

Thanks for the fix — the issue with source_field matching a field name is real, and skipping every field whose name appears as some field's source_field was too broad.

However, I have a concern about the new approach of skipping f"{name}_id" for every ForeignKeyFieldInstance.

The backing field of a FK is generated as <field_name>_id, but this only holds when the FK's source_field is left at its default. If a user sets a custom source_field on the FK, the backing field's name is still <field_name>_id (its source_field is what the user set), so the current check still works. The problem is the reverse: a user could explicitly define a field named <fk_field_name>_id that is not a backing field, and it would now be silently skipped.

Example:

class MyModel(Model):
    author = ForeignKeyField("models.Author")
    author_id = IntField()  # explicit field, not a backing field

With the new logic, author_id is in fk_backing_fields and gets skipped, so the migration omits it. The previous logic would also skip it if the FK's source_field was author_id, but if the user set a different source_field on the explicit field, it would have been kept. So there's a potential regression here.

Could you clarify how backing fields are identified elsewhere in the codebase? If there's a reliable way to tell a generated backing field apart from a user-defined field, using that would be safer. Otherwise, at minimum, the test coverage should include:

  • A FK whose source_field is customized, ensuring its backing field is still skipped.
  • An explicit field named <fk_name>_id, ensuring it is kept.

Also, does OneToOneFieldInstance need to be included in the check? OneToOneField likely subclasses ForeignKeyField, but if not, it should be added explicitly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrations not adding inherited fields until 2nd migration, and not making PKs generated

2 participants