Commit graph

117 commits

Author SHA1 Message Date
Erik Montnemery
86ae70780c
Refactor recorder retryable_database_job decorator (#125306) 2024-09-05 13:09:27 +02:00
Erik Montnemery
984eba809c
Simplify generic decorators in recorder (#125301)
* Simplify generic decorators in recorder

* Remove additional case
2024-09-05 10:16:44 +02:00
Erik Montnemery
a8f2204f4f
Teach recorder data migrator base class to update MigrationChanges (#125214)
* Teach recorder data migrator base class to update MigrationChanges

* Bump migration version

* Improve test coverage

* Update migration.py

* Revert migrator version bump

* Remove unneeded change
2024-09-05 08:56:18 +02:00
J. Nick Koston
a35fa0e95a
Warn that the minimum SQLite version will change to 3.40.1 as of 2025.2 (#104298)
Co-authored-by: Robert Resch <robert@resch.dev>
2024-07-31 08:13:04 -05:00
Erik Montnemery
5dbd7684ce
Fail tests if recorder creates nested sessions (#122579)
* Fail tests if recorder creates nested sessions

* Adjust import order

* Move get_instance
2024-07-25 21:18:55 +02:00
Marc Mueller
ca55986057
Import Generator from collections.abc (1) (#120914) 2024-07-01 11:51:51 +02:00
Marc Mueller
fe21e2b8ba
Import Generator from typing_extensions (1) (#118986) 2024-06-06 17:02:13 +02:00
J. Nick Koston
588380392d
Small speed up to read-only database sessions (#118674) 2024-06-03 19:50:05 +02:00
J. Nick Koston
099ad77078
Migrate recorder instance to use HassKey (#118673) 2024-06-03 11:19:00 -05:00
Jan Bouwhuis
c616fc036e
Move recorder chunk utils to shared collection utils (#118065) 2024-05-24 12:49:39 -10:00
Marc Mueller
907b9c42e5
Use PEP 695 for decorator typing with type aliases (2) (#117663) 2024-05-18 11:41:46 +02:00
Sid
2cc916db6d
Replace pylint broad-except with Ruff BLE001 (#116250) 2024-05-07 14:00:27 +02:00
Sid
6587ee20db
Enable Ruff TRY300 (#114437)
* Enable Ruff TRY300

* Update validation.py

* Address review comments
2024-03-30 10:37:59 +01:00
Sid
a5b609f081
Enable ruff TRY401 (#114395)
* Enable ruff TRY401

* fix tests
2024-03-29 07:20:36 +01:00
Sid
727581eea3
Add first batch of Ruff PT rules (#113665)
* Add first batch of Ruff PT rules

* fix weather test

* Fix pilight test

* Update test_intent.py

* Update pilight test_init.py

* Update test_init.py
2024-03-18 13:26:14 +01:00
Sid
d5fd005db8
Add ruff rule PIE808 (#113621) 2024-03-17 09:56:26 +01:00
J. Nick Koston
af6f2a516e
Guard against db corruption when renaming entities (#112718) 2024-03-08 11:34:07 -10:00
Marc Mueller
a033574ee2
Add empty line after module docstring [r] (#112703) 2024-03-08 15:05:07 +01:00
J. Nick Koston
a22244707b
Create an issue when database backups fail because the system runs out of resources (#109020) 2024-01-30 10:23:58 -10:00
Ville Skyttä
24b1e01d71
Update Ruff to 0.1.8, avoid linter/formatter conflicts (#106080)
* Disable Ruff rules that may conflict with the formatter

* Upgrade Ruff to 0.1.8

- https://github.com/astral-sh/ruff/releases/tag/v0.1.7
- https://github.com/astral-sh/ruff/releases/tag/v0.1.8

* Format with Ruff 0.1.8
2023-12-20 23:55:09 +01:00
J. Nick Koston
9ed745638d
Chunk purging attributes and data ids for old SQLite versions (#104296) 2023-11-24 11:46:02 +01:00
J. Nick Koston
da1d5fc862
Increase max bind vars based on database version (#101464) 2023-10-06 07:59:38 +02:00
Ville Skyttä
34b47a2597
Remove unnnecessary pylint configs from components [m-r]* (#98924) 2023-08-24 00:56:50 +02:00
Ville Skyttä
3094991236
Upgrade ruff to 0.0.285 (#98647) 2023-08-19 08:17:17 -04:00
Franck Nijhof
3e85a29b86
Move overlapping pylint rules to ruff, disable mypy overlap (#94359) 2023-06-27 17:42:46 +02:00
Ville Skyttä
ca936d0b38
Replace bandit with ruff (#93200) 2023-06-08 22:46:04 +02:00
J. Nick Koston
95fcdc5684
Speed up sql queries where ORM rows are not needed (#91839)
* Speed up logbook and history queries where ORM rows are not needed

This avoids having sqlalchemy wrap Result in ChunkedIteratorResult
which has additional overhead we do not need for these cases

* more places

* anything that uses _sorted_statistics_to_dict does not need orm rows either
2023-04-21 23:28:07 -04:00
J. Nick Koston
e40a373c4b
Call sqlite pragma optimize during periodic cleanup task (#91245)
https://www.sqlite.org/pragma.html#pragma_optimize

> To achieve the best long-term query performance without the need to do a detailed engineering analysis of the application schema and SQL, it is recommended that applications run "PRAGMA optimize" (with no arguments) just before closing each database connection. Long-running applications might also benefit from setting a timer to run "PRAGMA optimize" every few hours.

> This pragma is usually a no-op or nearly so and is very fast.

Since we keep the recorder connection open for the entire time HA
is running we fall into the long-running application bucket
2023-04-11 22:39:19 -04:00
J. Nick Koston
49079691d4
Reduce overhead of legacy database columns on new installs (#90246)
* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* not working as expected

* override the type compiler

* override the type compiler

* override the type compiler

* override the type compiler

* Apply suggestions from code review

* pgsql char1

* make entity filter test setup with old schema

* fix some more tests that were mutating state

* fix some more tests that were mutating state

* fix some more tests that were mutating state

* fix more dbstate mutations

* add shim for older tests

* split migration tests

* add coverage for purging legacy data

* tweak

* more fixes

* drop some legacy

* fix another test

* fix a few more

* add casts for postgresql in case someone deletes the schema changes table

* dry

* dry

* dry
2023-04-10 10:08:46 -04:00
J. Nick Koston
b1f64de6ce
Remove the old ix_states_event_id index if its no longer being used (#89901)
* Remove the old ix_states_event_id index if its no longer being used

* cover it

* fixes

* fixup

* Update homeassistant/components/recorder/tasks.py
2023-03-17 20:27:33 -04:00
J. Nick Koston
85ca94e9d4
Mark database sessions that do not write data as read_only (#89600)
* Mark sessions that do not write data as read_only

* Mark sessions that do not write data as read_only
2023-03-12 21:33:28 -04:00
J. Nick Koston
1bd9767d8c
Handle InnoDB deadlocks during migration (#89073)
* Handle slow InnoDB rollback when encountering duplicates during migration

fixes #89069

* adjust

* fix mock

* tests

* return on success
2023-03-03 22:00:13 -05:00
J. Nick Koston
69a3738bdb
Fix migration failing when existing data has duplicates (#88712) 2023-02-24 11:41:44 -06:00
J. Nick Koston
c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston
3533e77ec9
Fix using MariaDB slow range select workaround with Postgresql (#88459)
* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253

* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253
2023-02-19 21:08:43 -05:00
J. Nick Koston
b9f29c27ab
Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
J. Nick Koston
db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
Marc Mueller
b478b4fa16
Add ParamSpec TypeAliases [mypy 1.0] (#87597) 2023-02-07 15:56:26 +01:00
J. Nick Koston
3aa744ed19
Fix indent on slow_range_in_select for MySQL/MariaDB (#87581)
We would calculate this value and throw it away
because it was only used on first_connection
2023-02-06 23:59:54 -05:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
J. Nick Koston
5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
Franck Nijhof
d4955a3d87
Replace assert False by raising an error (#86686) 2023-01-26 16:51:43 +01:00
Marc Mueller
45b4b0e990
Import ParamSpec from typing [Py310] (#86413)
* Import ParamSpec from typing [Py310]

* Update additional imports
2023-01-23 07:28:43 +01:00
Franck Nijhof
79b52a2b41
Stricter pylint message control (#86154) 2023-01-20 13:47:55 +01:00
Franck Nijhof
c5dedb7a79
Code styling tweaks to the recorder integration (#86030) 2023-01-16 19:51:11 +01:00
Ville Skyttä
ae302bbec0
Make use of str.removeprefix and .removesuffix (#85584) 2023-01-13 12:19:38 +01:00
Franck Nijhof
fed8f905c8
Correct return type of two raise-only methods in recorder (#85048) 2023-01-03 12:15:45 +01:00
Franck Nijhof
94755a5773
String formatting and max line length - Part 4 (#84445)
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 13:27:27 +01:00
Erik Montnemery
2fe8e95309
Add helper to calculate statistic period start and end (#82493)
* Add helper to calculate statistic period start and end

* Don't parse values in resolve_period

* Add specific test for resolve_period

* Improve typing

* Move to recorder/util.py

* Extract period schema
2022-11-26 13:00:40 -05:00