Commit graph

151 commits

Author SHA1 Message Date
J. Nick Koston
c94b054d75
Retain history when renaming an entity_id (#89963)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2023-03-19 17:33:21 -10:00
J. Nick Koston
817ba97227
Remove unneeded lambda_stmt in place add in statistics (#89943)
We can generate this entire query in a single lambda_stmt
so there is no need to add two which increases the size
of the cache key
2023-03-19 22:05:39 -04:00
J. Nick Koston
5ffb233004
Avoid database executor job to fetch statistic metadata on cache hit (#89960)
* Avoid database executor job to fetch statistic metadata on cache hit

Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job

* Avoid database executor job to fetch statistic metadata on cache hit

Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job

* Avoid database executor job to fetch statistic metadata on cache hit

Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job

* remove exception catch since the threading.excepthook will actually catch this in production

* fix a few missed ones

* threadsafe

* Update homeassistant/components/recorder/table_managers/statistics_meta.py

* coverage and optimistic caching
2023-03-19 22:01:16 -04:00
J. Nick Koston
dbb2706c76
Reduce number of tasks created by compiling missing statistics (#89835) 2023-03-16 21:07:14 -10:00
J. Nick Koston
f6f3565796
Reduce latency to find stats metadata (#89824) 2023-03-16 19:00:02 -10:00
J. Nick Koston
a6d6807dd0
Add typing to statistics results (#89118) 2023-03-14 09:06:56 -10:00
J. Nick Koston
85ca94e9d4
Mark database sessions that do not write data as read_only (#89600)
* Mark sessions that do not write data as read_only

* Mark sessions that do not write data as read_only
2023-03-12 21:33:28 -04:00
Marc Mueller
bfb89fd8f2
Update pylint to 2.17.0 (#89377)
* Update pylint to 2.17.0

* Remove unused pylint disable comments
2023-03-08 10:54:19 -05:00
J. Nick Koston
3c70dd9b42
Make sql subqueries threadsafe (#89254)
* Make sql subqueries threadsafe

fixes #89224

* fix join outside of lambda

* move statement generation into a seperate function to make it easier to test

* add cache key tests

* no need to mock hass
2023-03-06 20:44:11 -05:00
J. Nick Koston
1bd9767d8c
Handle InnoDB deadlocks during migration (#89073)
* Handle slow InnoDB rollback when encountering duplicates during migration

fixes #89069

* adjust

* fix mock

* tests

* return on success
2023-03-03 22:00:13 -05:00
epenet
f2b736fad0
Adjust entity registry access in core platforms (#88944)
* Adjust entity registry access in platforms

* Adjust more core components
2023-03-01 08:02:16 +01:00
J. Nick Koston
9c82cfecca
Avoid asking recorder platforms for list_statistic_ids when already complete (#88495)
* Avoid asking recorder platforms for list_statistic_ids when already complete

If we already had all the data needed for list_statistic_ids, we would
still query recorder platforms and throw away the results

* Update homeassistant/components/recorder/statistics.py
2023-02-20 12:39:42 -05:00
J. Nick Koston
c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston
0b311c8c7f
Fix statistics_at_time query not using index (#88460)
* Fix statistics_at_time query not using index

fixes #82411

* fix refactoring error

* fix query so sqlalc does not get confused

* split it

* write as subquery

* reduce

* cleanup

* reduce

* Revert "reduce"

This reverts commit 43b4b55778.
2023-02-19 21:07:31 -05:00
J. Nick Koston
d4376b2041
Use a single session to compile statistics (#88405)
We had two session_scope contexts in compile_statistics which causes
a commit to happen twice during the compile.
2023-02-18 20:23:47 -05:00
J. Nick Koston
5fe8829cf6
Speed up compiling hourly statistics (#88225) 2023-02-16 12:55:25 -06:00
J. Nick Koston
6c542bd314
Speed up formatting statistics data (#88228)
* Speed up fetching statistics by using attrgetter

Uses the faster native code vs a lambda
4aeae28671/Modules/_operator.c (L1406)

* avoid dict lookups

* avoid dict lookups

* naming

* split

* Revert "split"

This reverts commit 0ead89603d.

* Revert "Revert "split""

This reverts commit 20014af168.

* tweak

* tweak

* tweak

* tweak

* Revert "tweak"

This reverts commit d5e4aac7c5.

* Revert "tweak"

This reverts commit 0cebae33f8.

* Revert "tweak"

This reverts commit 0184e47e24.

* comment

* comment

* comment

* reduce local vars

* less
2023-02-16 12:34:20 -05:00
J. Nick Koston
32a6280139
Optimize _sorted_statistics_to_dict to avoid checking which types each loop (#87901)
* Optimize _sorted_statistics_to_dict to avoid checking which types each loop

* tweak

* avoid some more dict lookups
2023-02-11 23:21:40 -05:00
J. Nick Koston
86a93e9fce
Optimize fetching statistics by avoiding recalculating time boundaries (#87859)
* predictive

* fix conversion error

* fix conversion error

* fix conversion error

* convert day to use new algo

* reduce
2023-02-11 11:41:57 -05:00
J. Nick Koston
c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston
ea356ad260
Optimize fetching weekly/monthly/yearly statistics (#87747)
* Optimize fetching statistics

* speed up

* avoid double groupby

* avoid another loop

* tweak flow

* fixes

* tweak

* avoid a new dt object in the cache for week/month

* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet
6806e85eab
Add missing converters to recorder statistics (#87137) 2023-02-02 09:44:26 +01:00
Franck Nijhof
a79885ceaf
Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck Nijhof
c5dedb7a79
Code styling tweaks to the recorder integration (#86030) 2023-01-16 19:51:11 +01:00
Erik Montnemery
b0d4b73874
Add unitless unit converter (#85694)
* Add unitless unit converter

* Adjust type hints

* Adjust tests

* Rename to UnitlessRatioConverter
2023-01-12 09:20:00 +01:00
Franck Nijhof
f39f3b612a
String formatting and max line length - Part 5 (#84501)
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 16:43:17 +01:00
Franck Nijhof
94755a5773
String formatting and max line length - Part 4 (#84445)
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 13:27:27 +01:00
Erik Montnemery
587381440f
Fix repairing datetime precision for PostgreSQL (#83351) 2022-12-05 22:33:06 -05:00
Erik Montnemery
f869ce9d06
Validate common statistics DB schema errors on start (#79707)
* Validate common statistics db schema errors on start

* Fix test

* Add tests

* Adjust tests

* Disable statistics schema validation in tests

* Update after rebase
2022-11-29 10:16:08 +01:00
Erik Montnemery
58b3a00b16
Remove unnecessary DB access from statistic_during_period (#82871) 2022-11-28 13:48:06 -05:00
Erik Montnemery
a038314d8b
Add display unit to WS recorder/get_statistics_metadata (#82870) 2022-11-28 13:46:57 -05:00
Erik Montnemery
b94e1e9ef8
Fire events when long term statistics is updated (#82492)
* Fire events when long term statistics is updated

* Allow the new events to be subscribed to by anyone

* Address review comments

* Finish renaming events

* Finish renaming events

* Fix do_adhoc_statistics

* Adjust tests

* Adjust tests
2022-11-24 22:01:36 +01:00
Erik Montnemery
607a0e7697
Reduce size of get_statistics_during_period WS API response (#82131) 2022-11-16 17:36:30 +01:00
Erik Montnemery
9b8f94363c
Fix statistic_during_period for data with holes (#81847) 2022-11-16 12:46:29 +01:00
Erik Montnemery
68346599d2
Add WS API recorder/statistic_during_period (#80663) 2022-10-27 15:51:09 -04:00
Michael
8175dab7ab
Add week period to recorder statistics api (#80784)
* add week period to get statistics api

* add test
2022-10-25 20:07:28 +02:00
Erik Montnemery
b42e26fbdd
Use SupportedDialect enum in recorder (#80319) 2022-10-14 08:45:57 -04:00
Erik Montnemery
2e261d5dc2
Allow specifying the target table when importing statistics (#80230)
Allow specifying the table when importing statistics
2022-10-14 08:32:19 +02:00
Erik Montnemery
577f7904b5
Minor improvements of recorder typing (#80165)
* Minor improvements of recorder typing

* Only allow specifying statistic_ids as lists
2022-10-12 14:59:10 +02:00
Erik Montnemery
92ca95ca81
Fix preserving long term statistics when entity_id is changed (#79556) 2022-10-03 21:13:48 -04:00
Erik Montnemery
a2e3978d53
Don't normalize units of long term statistics (#79320)
* Don't normalize units of long term statistics

* Update statistics.py
2022-10-03 21:42:44 +02:00
Erik Montnemery
2de273500e
Remove state_unit_of_measurement from metadata DB table (#79370)
* Remove state_unit_of_measurement from metadata DB table

* Adjust test
2022-10-01 18:55:00 +02:00
Erik Montnemery
bc2dffabc4
Improve naming of units used in statistics (#79276) 2022-09-30 08:38:44 +02:00
epenet
5438552d4a
Cleanup unit conversion (#79197)
* Move UNIT_RATIO to BaseUnitConverter

* Make UNIT_CONVERSION private

* Remove STATISTIC_UNIT_TO_UNIT_CLASS constant

* Cleanup websocket_api

* Imrpove valid_units check
2022-09-28 13:49:46 +02:00
epenet
5389ff3253
Add new weight device class (#79185) 2022-09-28 12:13:49 +02:00
epenet
7c448416e1
Add speed to SensorDeviceClass (#77953)
* Add speed to SensorDeviceClass

* Adjust recorder

* Adjust tests

* Adjust sensor UNIT_CONVERTERS

* Add tests

* Add websocket tests

* Update strings.json
2022-09-27 17:19:34 +01:00
epenet
bfcc18e5b8
Add distance to SensorDeviceClass (#77951)
* Add distance to SensorDeviceClass

* Adjust recorder

* Adjust tests

* Adjust recorder

* Update __init__.py

* Update test_websocket_api.py

* Update test_websocket_api.py

* Update test_websocket_api.py

* Update strings.json

* Fix tests

* Adjust docstring
2022-09-27 15:34:00 +01:00
Erik Montnemery
2167cf540d
Drop some unused constants from recorder (#79138) 2022-09-27 10:52:16 +02:00