J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns ( #87321 )
2023-02-09 12:24:19 -06:00
J. Nick Koston
db503c0daa
Always use UTC time for the MariaDB/MySQL session ( #87583 )
2023-02-09 11:39:45 -06:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 ( #86436 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
Franck Nijhof
3d557b5583
Collection of code styling tweaks ( #87381 )
2023-02-04 12:52:26 -05:00
J. Nick Koston
f874258e7e
Silence spurious warnings about removing ix_states_entity_id with newer installs ( #86961 )
...
* Silence spurious warnings about removing ix_states_entity_id with newer installs
https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465
* Silence spurious warnings about removing ix_states_entity_id with newer installs
https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465
2023-01-30 12:38:33 -05:00
J. Nick Koston
4e9bd09d39
Fix old indices not being removed in schema migration leading to slow MySQL queries ( #86917 )
...
fixes #83787
2023-01-29 21:33:23 -05:00
J. Nick Koston
58de7b8df0
Fix v32 schema migration when MySQL global.time_zone is configured with non-UTC timezone ( #86867 )
...
* Fix v32 schema migration when MySQL timezone is not UTC
* tweak
2023-01-28 22:06:07 -05:00
J. Nick Koston
d97a061285
Chunk MariaDB and Postgresql data migration to avoid running out of buffer space ( #86680 )
...
* Chunk MariaDB data migration to avoid running out of buffer space
This will make the migration slower but since the innodb_buffer_pool_size
is using the defaul to 128M and not tuned to the db size there is a
risk of running out of buffer space for large databases
* Update homeassistant/components/recorder/migration.py
* hard code since bandit thinks its an injection
* Update homeassistant/components/recorder/migration.py
* guard against manually modified data/corrupt db
* adjust to 10k per chunk
* adjust to 50k per chunk
* memory still just fine at 250k
* but slower
* commit after each chunk to reduce lock pressure
* adjust
* set to 0 if null so we do not loop forever (this should only happen if the data is missing)
* set to 0 if null so we do not loop forever (this should only happen if the data is missing)
* tweak
* tweak
* limit cleanup
* lower limit to give some more buffer
* lower limit to give some more buffer
* where required for sqlite
* sqlite can wipe as many as needed with no limit
* limit on mysql only
* chunk postgres
* fix limit
* tweak
* fix reference
* fix
* tweak for ram
* postgres memory reduction
* defer cleanup
* fix
* same order
2023-01-27 22:39:45 -05:00
Franck Nijhof
a79885ceaf
Enable Ruff SIM117 ( #86783 )
2023-01-27 11:52:49 +01:00
Franck Nijhof
c5dedb7a79
Code styling tweaks to the recorder integration ( #86030 )
2023-01-16 19:51:11 +01:00
J. Nick Koston
b8a1537b58
Improve performance of fetching and storing history and events with the database ( #84870 )
2023-01-02 13:26:08 -10:00
Franck Nijhof
8819634b61
String formatting and max line length - Part 6 ( #84525 )
2022-12-24 13:19:51 +01:00
Franck Nijhof
94755a5773
String formatting and max line length - Part 4 ( #84445 )
...
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 13:27:27 +01:00
Erik Montnemery
587381440f
Fix repairing datetime precision for PostgreSQL ( #83351 )
2022-12-05 22:33:06 -05:00
Erik Montnemery
f869ce9d06
Validate common statistics DB schema errors on start ( #79707 )
...
* Validate common statistics db schema errors on start
* Fix test
* Add tests
* Adjust tests
* Disable statistics schema validation in tests
* Update after rebase
2022-11-29 10:16:08 +01:00
Erik Montnemery
04cc2ae264
Correct initialization of new databases ( #80234 )
2022-10-13 13:01:27 +02:00
Erik Montnemery
466c4656ca
Refactor recorder migration ( #80175 )
...
* Refactor recorder migration
* Improve test coverage
2022-10-13 08:11:54 +02:00
Erik
4a1c40f09b
Revert "Refactor recorder migration"
...
This reverts commit 69e10e5982
2022-10-12 15:12:12 +02:00
Erik
69e10e5982
Refactor recorder migration
2022-10-12 15:00:28 +02:00
Erik Montnemery
577f7904b5
Minor improvements of recorder typing ( #80165 )
...
* Minor improvements of recorder typing
* Only allow specifying statistic_ids as lists
2022-10-12 14:59:10 +02:00
Erik Montnemery
0fdb7052e9
Add comment in recorder about dropping column ( #79523 )
...
Add comment in recorder
2022-10-03 11:40:11 +02:00
Erik Montnemery
2de273500e
Remove state_unit_of_measurement from metadata DB table ( #79370 )
...
* Remove state_unit_of_measurement from metadata DB table
* Adjust test
2022-10-01 18:55:00 +02:00
Erik Montnemery
dd20a7ea62
Display statistics in the source's unit ( #78031 )
2022-09-15 12:01:24 -04:00
Erik Montnemery
eb28d7188b
Fix DB migration to schema version 29 ( #78037 )
...
* Fix DB migration to schema version 29
* Fix misspelled constants
2022-09-09 08:06:14 +02:00
J. Nick Koston
20b6c4c48e
Fix recorder hanging at start ( #75627 )
2022-07-22 12:37:25 -05:00
Erik Montnemery
fd6ffef52f
Support non-live database migration ( #72433 )
...
* Support non-live database migration
* Tweak startup order, add test
* Address review comments
* Fix typo
* Clarify comment about promoting dependencies
* Tweak
* Fix merge mistake
* Fix some tests
* Fix additional test
* Fix additional test
* Adjust tests
* Improve test coverage
2022-07-22 15:11:34 +02:00
Erik Montnemery
5f2b4001f3
Separate recorder database schema from other classes ( #72977 )
...
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston
cd590c79e2
Fix migration of MySQL data when InnoDB is not being used ( #72893 )
...
Fixes #72883
2022-06-02 14:01:06 +02:00
J. Nick Koston
856e1144c9
Ensure the statistics_meta table is using the dynamic row format ( #72784 )
2022-05-31 14:35:28 -07:00
Erik Montnemery
23bd64b7a2
Prevent duplication of statistics metadata ( #71637 )
...
* Prevent duplication of statistics metadata
* Add models_schema_28.py
* Handle entity renaming as a recorder job
* Improve tests
2022-05-24 15:34:46 +02:00
J. Nick Koston
a8aa0e1cca
Add Estimated Database Size to the recorder system health ( #71463 )
2022-05-07 21:02:54 -07:00
J. Nick Koston
eb77f8db85
Complete strict typing for recorder ( #71274 )
...
* Complete strict typing for recorder
* update tests
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Remove the asserts
* remove ignore comments
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
J. Nick Koston
5db014666c
Avoid recording state_changed events in the events table ( #71165 )
...
* squash
fix mypy
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* pass all columns
* fix commented out code
* reduce logbook query complexity
* merge
* comment
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
J. Nick Koston
c23866e5e5
De-duplicate event data into a new event_data table ( #71135 )
2022-05-01 21:01:17 -05:00
J. Nick Koston
6ce768465f
Improve purge performance for non-sqlite databases ( #70576 )
2022-04-26 13:11:57 -07:00
Dmitry Katsubo
521579fc6a
Fixed syntax error in ALTER TABLE statement ( #70304 ) ( #70336 )
2022-04-22 21:29:25 -10:00
Erik Montnemery
eb3458a3d2
Add MutexPool for recorder tests ( #69410 )
...
* Add MutexPool for recorder tests
* Fix get_schema_version
* Update test test_last_run_was_recently_clean
* Update test test_shutdown_before_startup_finishes
* Revert comments in test_write_lock_db
* Make the MutexPool lock a class variable
* Remove stale comment
* Move MutexPool
* Tweak debug prints
2022-04-12 07:41:46 -10:00
J. Nick Koston
d75f577b88
Additional strict typing for recorder ( #68860 )
2022-03-30 09:20:44 -07:00
J. Nick Koston
3c10ac308d
Fix migration to schema v25 with Postgresql ( #68426 )
2022-03-20 21:49:11 +01:00
J. Nick Koston
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
Erik Montnemery
39ed628cca
Suppress unwanted error messages during recorder migration ( #66004 )
2022-02-07 15:46:40 -08:00
Erik Montnemery
1f8e8926fe
Only remove duplicated statistics on error ( #65653 )
2022-02-04 11:31:12 -08:00
Erik Montnemery
020953e943
Improve recorder migration for PostgreSQL when columns already exist ( #65680 )
2022-02-04 10:55:28 -08:00
Erik Montnemery
41ab12cb88
Don't use shared session during recorder migration ( #65672 )
2022-02-04 09:55:11 -08:00
Erik Montnemery
bceeaec2f8
Remove duplicated statistics rows ( #61146 )
...
* Remove duplicated statistics
* Fix misleading docstring
* Pylint knows best
* Correct test
* Oops
* Prevent insertion of duplicated statistics
* Tweak
* pylint
* Add models_schema_23.py
* Tweak
2021-12-13 14:15:36 +01:00
Erik Montnemery
d980ca7e04
Correct recorder migration._add_columns for PostgreSQL ( #60547 )
2021-11-29 18:33:25 +01:00
Erik Montnemery
2f3dea30e2
Correct migration to recorder schema 22 ( #59048 )
2021-11-04 10:29:10 +01:00
Erik Montnemery
f594bc353b
Add support for external statistics ( #56607 )
...
* Support external statistics
* Update tests
* Lint
* Adjust code after rebase
* Separate external statistic_id with :, add name to metadata
* Adjust tests
* Simplify get_metadata_with_session
* Address review comments
* Allow updating external statistics
* Validate input
* Adjust tests after rebase
* Pylint
* Adjust tests
* Improve test coverage
2021-10-26 10:26:50 +02:00
Erik Montnemery
e3534eec87
Report orphaned statistics in statistic validation ( #57324 )
2021-10-20 15:57:22 +02:00
Erik Montnemery
8337baa354
Correct migration to recorder schema 18 ( #57165 )
2021-10-06 13:29:42 +02:00
