J. Nick Koston
e136070718
Speed up purging the database ( #92247 )
2023-04-29 18:45:14 +02:00
J. Nick Koston
e156d3132e
Ensure purge can cleanup old format detached states in the database ( #92145 )
2023-04-28 21:03:59 +02:00
J. Nick Koston
96225bb287
Rename recorder run_history to table_managers.recorder_runs_manager ( #90070 )
2023-03-21 18:38:33 -10:00
J. Nick Koston
aebe4c66a6
Fix cpu thrashing during purge after all legacy events were removed ( #89923 )
...
* Fix cpu thrashing during purge after all legacy events were removed
We now remove the the index of of event ids on the states table when its
all NULLs to save space. The purge path needs to avoid checking for legacy
rows to purge if the index has been removed since it will result in a full
table scan each purge cycle that will always find no legacy rows to purge
* one more place
* drop the key constraint as well
* fixes
* more sqlite
2023-03-19 22:04:24 -04:00
J. Nick Koston
99d6b1fa57
Migrate States to use a table manager ( #89769 )
2023-03-15 16:19:43 -10:00
J. Nick Koston
e379aa23bd
Migrate StateAttributes to use a table manager ( #89760 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2023-03-15 15:26:29 -10:00
J. Nick Koston
59de7f3057
Migrate EventData management to a table manager ( #89716 )
2023-03-14 21:40:59 -10:00
J. Nick Koston
b906d67c1e
Fix filtered purge not removing newer events ( #89721 )
2023-03-14 16:33:19 -10:00
J. Nick Koston
c41f91be89
Deduplicate entity_id in the states table ( #89557 )
2023-03-12 10:01:58 -10:00
J. Nick Koston
8bd43760b6
Deduplicate event_types in the events table ( #89465 )
...
* Deduplicate event_types in the events table
* Deduplicate event_types in the events table
* more fixes
* adjust
* adjust
* fix product
* fix tests
* adjust
* migrate
* migrate
* migrate
* more test fixes
* more test fixes
* fix
* migration test
* adjust
* speed up
* fix index
* fix more tests
* handle db failure
* preload
* tweak
* adjust
* fix stale docs strings, remove dead code
* refactor
* fix slow tests
* coverage
* self join to resolve query performance
* fix typo
* no need for quiet
* no need to drop index already dropped
* remove index that will never be used
* drop index sooner as we no longer use it
* Revert "remove index that will never be used"
This reverts commit 461aad2c52
2023-03-11 14:54:55 -05:00
J. Nick Koston
c2b770bcb9
Load pending state attributes and event data ids at startup ( #88444 )
...
* Load pending state attributes and event data ids at startup
Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries
* lru
* fix hang
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* cleanup
2023-02-19 21:26:38 -05:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 ( #86436 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions ( #87409 )
...
* Speed up purge time with newer MariaDB versions
* fix
* document
* document
* document
* rename
* self review
* Update homeassistant/components/recorder/util.py
* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof
5e81d28116
Update black to 23.1.0 ( #87188 )
2023-02-02 18:35:24 +01:00
Franck Nijhof
c5dedb7a79
Code styling tweaks to the recorder integration ( #86030 )
2023-01-16 19:51:11 +01:00
J. Nick Koston
b8a1537b58
Improve performance of fetching and storing history and events with the database ( #84870 )
2023-01-02 13:26:08 -10:00
Franck Nijhof
94755a5773
String formatting and max line length - Part 4 ( #84445 )
...
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 13:27:27 +01:00
Erik Montnemery
5f2b4001f3
Separate recorder database schema from other classes ( #72977 )
...
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
Marc Mueller
3c246b7800
Update mypy to 0.960 ( #72481 )
2022-05-25 11:42:14 -07:00
J. Nick Koston
a70e2a33dc
Fixing purging legacy rows and improve performance ( #71916 )
2022-05-15 21:25:07 -07:00
J. Nick Koston
896bf986eb
Speed up nightly database purges with lambda_stmt ( #71537 )
2022-05-08 12:47:12 -07:00
J. Nick Koston
a8aa0e1cca
Add Estimated Database Size to the recorder system health ( #71463 )
2022-05-07 21:02:54 -07:00
J. Nick Koston
eb77f8db85
Complete strict typing for recorder ( #71274 )
...
* Complete strict typing for recorder
* update tests
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Remove the asserts
* remove ignore comments
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
J. Nick Koston
188040b8bb
Use lambda_stmt for recorder queries and migrate them to queries module ( #71219 )
2022-05-02 15:17:21 -07:00
J. Nick Koston
5db014666c
Avoid recording state_changed events in the events table ( #71165 )
...
* squash
fix mypy
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* pass all columns
* fix commented out code
* reduce logbook query complexity
* merge
* comment
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
J. Nick Koston
c23866e5e5
De-duplicate event data into a new event_data table ( #71135 )
2022-05-01 21:01:17 -05:00
J. Nick Koston
b9c7a89b47
Prevent sqlalchemy Transparent SQL Compilation Caching from filling up during purge ( #71015 )
2022-04-28 23:48:28 -07:00
J. Nick Koston
6ce768465f
Improve purge performance for non-sqlite databases ( #70576 )
2022-04-26 13:11:57 -07:00
J. Nick Koston
f073f17040
Refactor tracking of the recorder run history ( #70456 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
J. Nick Koston
d75f577b88
Additional strict typing for recorder ( #68860 )
2022-03-30 09:20:44 -07:00
Marc Mueller
53110f8cb7
Update pylint to 2.13.2 ( #68704 )
2022-03-27 16:08:24 +02:00
Marc Mueller
911b159281
Cleanup after pylint update ( #68657 )
2022-03-26 00:34:12 +01:00
J. Nick Koston
7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes ( #68364 )
2022-03-19 14:21:28 -07:00
J. Nick Koston
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
Marc Mueller
cb736eaeaf
Add type ignore error codes [recorder] ( #66780 )
2022-02-18 10:37:38 +01:00
Erik Montnemery
b301ab25a3
Purge short term statistics ( #58028 )
...
* Purge short term statistics
* Less meep
* Add tests
2021-10-20 16:00:59 +02:00
Erik Montnemery
8567aa9e13
Evict purged states from recorder's old_state cache ( #56877 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
Marc Mueller
565a9fea6b
Import Callable from collections.abc (2) ( #56776 )
2021-09-29 14:06:51 +02:00
PeteBa
42c4317628
Avoid drift in recorder purge cut-off ( #52135 )
2021-06-25 11:29:38 -10:00
PeteBa
aa9b99713c
Add purge_entities service call to recorder ( #48069 )
2021-05-22 10:30:05 -05:00
Erik Montnemery
89dd3292ba
Initial draft of statistics ( #49852 )
2021-05-16 10:23:37 -07:00
J. Nick Koston
6d137d2316
Increase recorder test coverage ( #49362 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston
9368891b1b
Live db migrations and recovery ( #49036 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Marc Mueller
92852b9c10
Add apply_filter attribute to recorder.purge service ( #45826 )
2021-03-11 17:03:30 -10:00
Marc Mueller
10848b9bdf
Recorder improvements ( #47739 )
2021-03-11 07:52:07 -10:00
J. Nick Koston
a060acc2b1
Fix recorder with MSSQL ( #46678 )
...
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
J. Nick Koston
2cbb93be43
Always keep the current recorder run when purging ( #43733 )
2020-11-30 09:13:50 +01:00
Franck Nijhof
1c2ebdf307
Upgrade black to 20.8b1 ( #39287 )
2020-08-27 13:56:20 +02:00
Franck Nijhof
53545c984b
Log lines do not end with a full stop ( #37527 )
2020-07-05 23:04:19 +02:00
Franck Nijhof
79f131066c
Ensure recorder data integrity and MySQL lock error handling ( #37228 )
2020-06-29 16:23:11 -07:00
