* Raise on database error in recorder.migration._drop_foreign_key_constraints
* Fix test
* Fix test
* Revert "Fix test"
This reverts commit 940b8cb506e912826d43d09d7697c10888bdf685.
* Update test
* Improve test coverage
* Disable test for SQLite
* Drop violating rows before adding foreign constraints
* Don't delete rows with null-references
* Only delete rows when integrityerror is caught
* Move restore of dropped foreign key constraints to a separate migration step
* Use aliases for tables
* Update homeassistant/components/recorder/migration.py
* Update test
* Don't use alias for table we're deleting from, improve test
* Fix MySQL
* Update instead of deleting in case of self references
* Improve log messages
* Batch updates
* Add workaround for unsupported LIMIT in PostgreSQL
* Simplify
---------
Co-authored-by: J. Nick Koston <nick@koston.org>
* Ensure legacy event foreign key is removed from the states table
If the system ran out of disk space removing the FK, it would
fail. #121938 fixed that to try again, however that PR was made
ineffective by #122069 since it will never reach the check.
To solve this, the migration version is incremented to 2, and
the migration is no longer marked as done unless the rebuild
/fk removal is successful.
* fix logic for mysql
* fix test
* asserts
* coverage
* coverage
* narrow test
* fixes
* split tests
* should have skipped
* fixture must be used
* Refactor recorder data migration
* Fix stale docstrings
* Don't store a session object in BaseRunTimeMigration instances
* Simplify logic in EntityIDMigration.migration_done
* Fix tests
* Record state.last_reported
* Include last_reported in parts of the history API
* Use a bulk update
* fix refactoring error
---------
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Sid <27780930+autinerd@users.noreply.github.com>
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
* Handle statistics columns being unmigrated from previous downgrades
If the user downgraded HA from 2023.3.x to an older version without
restoring the database and they upgrade again with the same database
they will have unmigrated statistics columns since we only migrate them
once.
As its expensive to check, we do not want to check every time
at startup, so we will only do this one more time since the
risk that someone will downgrade to an older version is very
low at this point.
* add guard to sqlite to prevent re-migrate
* test
* move test to insert with old schema
* use helper
* normalize timestamps
* remove
* add check
* add fallback migration
* add fallback migration
* commit
* remove useless logging
* remove useless logging
* do the other columns at the same time
* coverage
* dry
* comment
* Update tests/components/recorder/test_migration_from_schema_32.py
* Fallback to generating a new ULID on migraiton if context is missing or invalid
It was discovered that postgresql will do a full scan if
there is a low cardinality on the index because of missing
context ids. We will now generate a ULID for the timestamp
of the row if the context data is missing or invalid
fixes#91514
* tests
* tweak
* tweak
* preen
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* not working as expected
* override the type compiler
* override the type compiler
* override the type compiler
* override the type compiler
* Apply suggestions from code review
* pgsql char1
* make entity filter test setup with old schema
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix more dbstate mutations
* add shim for older tests
* split migration tests
* add coverage for purging legacy data
* tweak
* more fixes
* drop some legacy
* fix another test
* fix a few more
* add casts for postgresql in case someone deletes the schema changes table
* dry
* dry
* dry
* Restart entity id post migration after a restart
If the entity migration finished and Home Assistant was
restarted during the post migration it would never be resumed
which means the old index and space would never be recovered
* add migration resume test
