Commit graph

334 commits

Author SHA1 Message Date
J. Nick Koston
c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston
3533e77ec9
Fix using MariaDB slow range select workaround with Postgresql (#88459)
* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253

* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253
2023-02-19 21:08:43 -05:00
J. Nick Koston
a9731a7b26
Make the recorder LRU adjustment threadsafe (#88443) 2023-02-19 19:30:08 +01:00
J. Nick Koston
9df00bd285
Adjust recorder LRU cache to align with the number of entities (#88350)
* Adjust size of recorder LRU based on number of entities

If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently

* add a test

* do not actually record 4096 states

* patch target

* patch target
2023-02-19 00:02:36 -05:00
J. Nick Koston
3ca9f3c0d5
Only build the recorder SupportedDialect enum once (#88402)
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
Erik Montnemery
83e5bf7ae8
Use entity_sources to determine integration in recorder platforms (#88382) 2023-02-18 07:21:41 -06:00
J. Nick Koston
289bab6f87
Make recorder platform attribute exclude integration aware (#88357) 2023-02-18 10:08:59 +01:00
epenet
a16e298599
Adjust json imports in tests (#88248) 2023-02-17 19:13:48 +01:00
epenet
185cd61cbd
Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
J. Nick Koston
b9f29c27ab
Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
Franck Nijhof
d3ed8de232
Update Ruff to v0.0.247 (#88210)
* Update Ruff to v0.0.247

* Fix new found violations
2023-02-15 16:46:03 -05:00
Franck Nijhof
ed79265843
Enable Ruff PT006 (#88165)
* Enable Ruff PT006

* Adjust existing cases

* Fix tests

* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
epenet
efef915fc7
Add mariadb 10.3.32 to database tests (#87741)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 16:28:39 -06:00
Erik Montnemery
4ca0a24f87
Increase test scope of MariaDB + PostgreSQL tests (#87019)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 15:58:13 -06:00
J. Nick Koston
7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field (#87937) 2023-02-12 14:51:10 -06:00
J. Nick Koston
c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston
cbaf4764e7
Ensure MySQL tests cleanup connections and raise an exception if they do not (#87767)
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2023-02-09 16:27:53 -06:00
J. Nick Koston
509de02044
Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery
2fae82a9ef
Really ensure recorder test fixture is setup before hass fixture (#87796) 2023-02-09 11:43:14 -06:00
J. Nick Koston
db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
J. Nick Koston
c05a7b29e6
Terminate stale MySQL connections at the end of test runs (#87794) 2023-02-09 11:13:13 -06:00
J. Nick Koston
03b47f1931
Disable test_entity_ids on MariaDB (#87766)
Dropping the database after this test will fail on
MySQL and hang forever because it causes an InnoDB deadlock

```
| 2042 | root   | localhost:52698 | NULL               | Query   |   41 | Waiting for table metadata lock | DROP DATABASE `homeassistant-test` |    0.000 |
```
2023-02-09 15:30:35 +01:00
J. Nick Koston
f4a81a8b5b
Disable test_migration_in_progress on MySQL (#87765)
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753

Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
2023-02-09 15:27:27 +01:00
J. Nick Koston
483b0cd017
Skip db locking test with postgresql/MySQL (#87760)
* Skip db locking test with postgresql/MySQL

https://github.com/home-assistant/core/pull/87756#issuecomment-1423828389

* switch to the guards instead
2023-02-09 13:58:32 +01:00
J. Nick Koston
77cdde94f3
Turn back off test_shutdown_before_startup_finishes test on MySQL (#87761) 2023-02-09 03:12:10 -06:00
Erik Montnemery
58618b2978
Improve recorder tests to use MariaDB / PostgreSQL (#87756) 2023-02-09 02:18:20 -06:00
Erik Montnemery
b0f5bdd504
Improve migration tests to use MariaDB / PostgreSQL database (#87753) 2023-02-09 08:42:32 +01:00
epenet
ba85fdcd61
Add type hints to integration tests (q-s) (#87706) 2023-02-08 08:12:54 -08:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
Erik Montnemery
720f51657d
Add CI job which runs recorder tests on PostgreSQL (#80614)
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
epenet
59ca7780fa
Add typing to tests with single hass argument (#87631) 2023-02-07 15:01:16 +01:00
J. Nick Koston
0a1d5c85ad
Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
epenet
4142f0d15d
Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
epenet
aa00114c2f
Add return type to async tests without arguments (#87612) 2023-02-07 10:26:56 +01:00
Marc Mueller
342b406dc0
Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
J. Nick Koston
c7980bda82
Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
epenet
19e27f4f52
Ensure recorder api aligns with sensor converters (#87257) 2023-02-04 14:28:47 -06:00
J. Nick Koston
389fc515a1
Prevent overly large attributes from being stored in the database (#87105) 2023-02-02 11:49:33 -06:00
Franck Nijhof
5e81d28116
Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet
58596d205c
Ensure sensor and recorder converters align (#87138) 2023-02-02 10:36:03 +01:00
J. Nick Koston
1fd58b6cb7
Fix a few cartesian products in recorder tests (#87106)
Fixes
```
recorder/test_init.py:251: SAWarning: SELECT statement has a cartesian product between FROM element(s) "states" and FROM element "state_attributes".  Apply join condition(s) between each element to resolve.
```
2023-02-02 07:47:09 +01:00
J. Nick Koston
fb55933b0a
Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
J. Nick Koston
5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
Erik Montnemery
53c5f02ca2
Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston
c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
Franck Nijhof
a79885ceaf
Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck Nijhof
e4a78420b8
Enable Ruff PT013 (#86757)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2023-01-27 09:09:46 +01:00
Erik Montnemery
fea30c1ce9
Terminate strings at NUL when recording states and events (#86687) 2023-01-26 11:11:03 +01:00