* Raise on database error in recorder.migration._drop_foreign_key_constraints
* Fix test
* Fix test
* Revert "Fix test"
This reverts commit 940b8cb506e912826d43d09d7697c10888bdf685.
* Update test
* Improve test coverage
* Disable test for SQLite
* Drop violating rows before adding foreign constraints
* Don't delete rows with null-references
* Only delete rows when integrityerror is caught
* Move restore of dropped foreign key constraints to a separate migration step
* Use aliases for tables
* Update homeassistant/components/recorder/migration.py
* Update test
* Don't use alias for table we're deleting from, improve test
* Fix MySQL
* Update instead of deleting in case of self references
* Improve log messages
* Batch updates
* Add workaround for unsupported LIMIT in PostgreSQL
* Simplify
---------
Co-authored-by: J. Nick Koston <nick@koston.org>
* Ensure legacy event foreign key is removed from the states table
If the system ran out of disk space removing the FK, it would
fail. #121938 fixed that to try again, however that PR was made
ineffective by #122069 since it will never reach the check.
To solve this, the migration version is incremented to 2, and
the migration is no longer marked as done unless the rebuild
/fk removal is successful.
* fix logic for mysql
* fix test
* asserts
* coverage
* coverage
* narrow test
* fixes
* split tests
* should have skipped
* fixture must be used
* Narrow sqlite database corruption check to ensure disk image is malformed
The database corruption check would also replace the database when it
locked externally instead of only when its malformed.
This was discovered in https://github.com/home-assistant/core/issues/121909#issuecomment-2227409124
when a user did a manual index creation while HA was online
* tweak
* tweak
* fix
* fix
* Add test fixture to control recorder migration
* Update tests/components/recorder/conftest.py
Co-authored-by: J. Nick Koston <nick@koston.org>
* Update tests/components/recorder/conftest.py
---------
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Sid <27780930+autinerd@users.noreply.github.com>
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
Almost 99% of items that are put into the recorder queue are
Events. Avoid wrapping them in tasks since we have to unwrap
them right away and its must faster to check for both RecorderTask
and Events since events are the common case.
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* not working as expected
* override the type compiler
* override the type compiler
* override the type compiler
* override the type compiler
* Apply suggestions from code review
* pgsql char1
* make entity filter test setup with old schema
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix more dbstate mutations
* add shim for older tests
* split migration tests
* add coverage for purging legacy data
* tweak
* more fixes
* drop some legacy
* fix another test
* fix a few more
* add casts for postgresql in case someone deletes the schema changes table
* dry
* dry
* dry
* Fix index not being dropped on postgresql databases with a schema prefix
Added logging in case index drops fail so we can tell
why in the future
* coverage
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* reduce
* comments
* Split context id migration into states and events tasks
Since events can finish much earlier than states we
would keep looking at the table because states as not
done. Make them seperate tasks
* add retry dec
* fix migration happening twice
* another case
* Deduplicate event_types in the events table
* Deduplicate event_types in the events table
* more fixes
* adjust
* adjust
* fix product
* fix tests
* adjust
* migrate
* migrate
* migrate
* more test fixes
* more test fixes
* fix
* migration test
* adjust
* speed up
* fix index
* fix more tests
* handle db failure
* preload
* tweak
* adjust
* fix stale docs strings, remove dead code
* refactor
* fix slow tests
* coverage
* self join to resolve query performance
* fix typo
* no need for quiet
* no need to drop index already dropped
* remove index that will never be used
* drop index sooner as we no longer use it
* Revert "remove index that will never be used"
This reverts commit 461aad2c52.
* typo
* Load pending state attributes and event data ids at startup
Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries
* lru
* fix hang
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* cleanup
