Commit graph

19 commits

Author SHA1 Message Date
J. Nick Koston
e156d3132e
Ensure purge can cleanup old format detached states in the database (#92145) 2023-04-28 21:03:59 +02:00
J. Nick Koston
adc472862b
Fallback to generating a new ULID on migraiton if context is missing or invalid (#91704)
* Fallback to generating a new ULID on migraiton if context is missing or invalid

It was discovered that postgresql will do a full scan if
there is a low cardinality on the index because of missing
context ids. We will now generate a ULID for the timestamp
of the row if the context data is missing or invalid

fixes #91514

* tests

* tweak

* tweak

* preen
2023-04-19 21:56:07 -04:00
J. Nick Koston
19f71b3bb9
Fix entity_id migration query failing with MySQL 8.0.30 (#90895) 2023-04-06 10:34:13 +02:00
J. Nick Koston
b1f64de6ce
Remove the old ix_states_event_id index if its no longer being used (#89901)
* Remove the old ix_states_event_id index if its no longer being used

* cover it

* fixes

* fixup

* Update homeassistant/components/recorder/tasks.py
2023-03-17 20:27:33 -04:00
J. Nick Koston
e379aa23bd
Migrate StateAttributes to use a table manager (#89760)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2023-03-15 15:26:29 -10:00
J. Nick Koston
59de7f3057
Migrate EventData management to a table manager (#89716) 2023-03-14 21:40:59 -10:00
J. Nick Koston
c41f91be89
Deduplicate entity_id in the states table (#89557) 2023-03-12 10:01:58 -10:00
J. Nick Koston
8bd43760b6
Deduplicate event_types in the events table (#89465)
* Deduplicate event_types in the events table

* Deduplicate event_types in the events table

* more fixes

* adjust

* adjust

* fix product

* fix tests

* adjust

* migrate

* migrate

* migrate

* more test fixes

* more test fixes

* fix

* migration test

* adjust

* speed up

* fix index

* fix more tests

* handle db failure

* preload

* tweak

* adjust

* fix stale docs strings, remove dead code

* refactor

* fix slow tests

* coverage

* self join to resolve query performance

* fix typo

* no need for quiet

* no need to drop index already dropped

* remove index that will never be used

* drop index sooner as we no longer use it

* Revert "remove index that will never be used"

This reverts commit 461aad2c52.

* typo
2023-03-11 14:54:55 -05:00
J. Nick Koston
170a13302c
Reduce overhead to store context ids in the database (#88942) 2023-03-08 14:51:45 -10:00
J. Nick Koston
c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
J. Nick Koston
b8a1537b58
Improve performance of fetching and storing history and events with the database (#84870) 2023-01-02 13:26:08 -10:00
Erik Montnemery
5f2b4001f3
Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston
d8b0376942
Fix purge of legacy database events that are not state changed (#72815) 2022-06-01 11:56:06 +02:00
J. Nick Koston
a70e2a33dc
Fixing purging legacy rows and improve performance (#71916) 2022-05-15 21:25:07 -07:00
J. Nick Koston
896bf986eb
Speed up nightly database purges with lambda_stmt (#71537) 2022-05-08 12:47:12 -07:00
J. Nick Koston
188040b8bb
Use lambda_stmt for recorder queries and migrate them to queries module (#71219) 2022-05-02 15:17:21 -07:00