Commit graph

778 commits

Author SHA1 Message Date
J. Nick Koston
d4376b2041
Use a single session to compile statistics (#88405)
We had two session_scope contexts in compile_statistics which causes
a commit to happen twice during the compile.
2023-02-18 20:23:47 -05:00
J. Nick Koston
3ca9f3c0d5
Only build the recorder SupportedDialect enum once (#88402)
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
Erik Montnemery
1eb20affa1
Update use of sqlalchemy with_variant (#88395) 2023-02-18 09:18:01 -06:00
Erik Montnemery
83e5bf7ae8
Use entity_sources to determine integration in recorder platforms (#88382) 2023-02-18 07:21:41 -06:00
J. Nick Koston
0a7cf2b88c
Speed up formatting history data (#88230) 2023-02-18 10:34:44 +01:00
J. Nick Koston
289bab6f87
Make recorder platform attribute exclude integration aware (#88357) 2023-02-18 10:08:59 +01:00
J. Nick Koston
b9f29c27ab
Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
J. Nick Koston
5fe8829cf6
Speed up compiling hourly statistics (#88225) 2023-02-16 12:55:25 -06:00
J. Nick Koston
6c542bd314
Speed up formatting statistics data (#88228)
* Speed up fetching statistics by using attrgetter

Uses the faster native code vs a lambda
4aeae28671/Modules/_operator.c (L1406)

* avoid dict lookups

* avoid dict lookups

* naming

* split

* Revert "split"

This reverts commit 0ead89603d.

* Revert "Revert "split""

This reverts commit 20014af168.

* tweak

* tweak

* tweak

* tweak

* Revert "tweak"

This reverts commit d5e4aac7c5.

* Revert "tweak"

This reverts commit 0cebae33f8.

* Revert "tweak"

This reverts commit 0184e47e24.

* comment

* comment

* comment

* reduce local vars

* less
2023-02-16 12:34:20 -05:00
epenet
ba23816a0c
Inverse json import logic (#88099)
* Fix helpers and util

* Adjust components

* Move back errors

* Add report

* mypy

* mypy

* Assert deprecation messages

* Move test_json_loads_object

* Adjust tests

* Fix rebase

* Adjust pylint plugin

* Fix plugin

* Adjust references

* Adjust backup tests
2023-02-16 11:37:57 +01:00
J. Nick Koston
484027023c
Fix new stats rows creating data in the old created column (#88013) 2023-02-13 21:53:37 -06:00
J. Nick Koston
7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field (#87937) 2023-02-12 14:51:10 -06:00
J. Nick Koston
32a6280139
Optimize _sorted_statistics_to_dict to avoid checking which types each loop (#87901)
* Optimize _sorted_statistics_to_dict to avoid checking which types each loop

* tweak

* avoid some more dict lookups
2023-02-11 23:21:40 -05:00
J. Nick Koston
85649ec589
Reduce overhead to see if an entity is recorded (#87912)
A significant chunk of list_statistic_ids was checking if the entity
was recorded because it had to get the recorder instance over and
over
2023-02-11 23:21:16 -05:00
J. Nick Koston
e899754919
Bump sqlalchemy to 2.0.3 (#87913)
changes: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.3
2023-02-11 23:20:22 -05:00
J. Nick Koston
86a93e9fce
Optimize fetching statistics by avoiding recalculating time boundaries (#87859)
* predictive

* fix conversion error

* fix conversion error

* fix conversion error

* convert day to use new algo

* reduce
2023-02-11 11:41:57 -05:00
J. Nick Koston
c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston
ea356ad260
Optimize fetching weekly/monthly/yearly statistics (#87747)
* Optimize fetching statistics

* speed up

* avoid double groupby

* avoid another loop

* tweak flow

* fixes

* tweak

* avoid a new dt object in the cache for week/month

* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
J. Nick Koston
509de02044
Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
J. Nick Koston
db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
Erik Montnemery
5dfb9cddd6
Sort manifests 8 (#87029) 2023-02-08 20:13:46 +01:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet
a202588fd2
Add return type to json_loads (#85672)
* Add JSON type definitions

* Sample use

* Keep mutable for a follo-up PR (avoid dead code)

* Use list/dict

* Remove JsonObjectType

* Remove reference to Union

* Cleanup

* Improve rest

* Rename json_dict => json_data

* Add docstring

* Add type hint to json_loads

* Add cast

* Move type alias to json helpers

* Cleanup

* Create and use json_loads_object

* Make error more explicit and add tests

* Use JsonObjectType in conversation

* Remove quotes
2023-02-07 17:21:55 +01:00
Marc Mueller
b478b4fa16
Add ParamSpec TypeAliases [mypy 1.0] (#87597) 2023-02-07 15:56:26 +01:00
J. Nick Koston
0a1d5c85ad
Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
J. Nick Koston
3aa744ed19
Fix indent on slow_range_in_select for MySQL/MariaDB (#87581)
We would calculate this value and throw it away
because it was only used on first_connection
2023-02-06 23:59:54 -05:00
Marc Mueller
342b406dc0
Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
Franck Nijhof
939eef3b28
Remove translations from Core (#87543)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2023-02-06 15:06:43 +01:00
J. Nick Koston
c7980bda82
Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
GitHub Action
f475446ebf [ci skip] Translation update 2023-02-06 00:23:07 +00:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof
3d557b5583
Collection of code styling tweaks (#87381) 2023-02-04 12:52:26 -05:00
Franck Nijhof
ca1a12898c
Enable Ruff D212 (#87347) 2023-02-03 23:08:48 +01:00
epenet
810367b757
Fix can not typo (#87254) 2023-02-03 11:37:16 +01:00
J. Nick Koston
22698b1cc5
Fix statistics graphs not loading with data_rate, electric_current, voltage, information, and unitless units (#87202)
* Add missing converts to recorder/statistics_during_period API

This was resulting in the stats graphs not loading on the frontend

* its in two places
2023-02-02 14:52:55 -05:00
J. Nick Koston
389fc515a1
Prevent overly large attributes from being stored in the database (#87105) 2023-02-02 11:49:33 -06:00
Franck Nijhof
5e81d28116
Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet
6806e85eab
Add missing converters to recorder statistics (#87137) 2023-02-02 09:44:26 +01:00
GitHub Action
0f80932709 [ci skip] Translation update 2023-02-02 00:25:01 +00:00
J. Nick Koston
fb55933b0a
Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
GitHub Action
be73e43989 [ci skip] Translation update 2023-02-01 00:27:42 +00:00
J. Nick Koston
5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
J. Nick Koston
00118a6f96
Speed up live history setup if there is no pending data to commit (#86942) 2023-01-30 19:00:34 +01:00
J. Nick Koston
f874258e7e
Silence spurious warnings about removing ix_states_entity_id with newer installs (#86961)
* Silence spurious warnings about removing ix_states_entity_id with newer installs

1069648035

* Silence spurious warnings about removing ix_states_entity_id with newer installs

1069648035
2023-01-30 12:38:33 -05:00
Erik Montnemery
53c5f02ca2
Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston
4e9bd09d39
Fix old indices not being removed in schema migration leading to slow MySQL queries (#86917)
fixes #83787
2023-01-29 21:33:23 -05:00
J. Nick Koston
c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
J. Nick Koston
58de7b8df0
Fix v32 schema migration when MySQL global.time_zone is configured with non-UTC timezone (#86867)
* Fix v32 schema migration when MySQL timezone is not UTC

* tweak
2023-01-28 22:06:07 -05:00
J. Nick Koston
d97a061285
Chunk MariaDB and Postgresql data migration to avoid running out of buffer space (#86680)
* Chunk MariaDB data migration to avoid running out of buffer space

This will make the migration slower but since the innodb_buffer_pool_size
is using the defaul to 128M and not tuned to the db size there is a
risk of running out of buffer space for large databases

* Update homeassistant/components/recorder/migration.py

* hard code since bandit thinks its an injection

* Update homeassistant/components/recorder/migration.py

* guard against manually modified data/corrupt db

* adjust to 10k per chunk

* adjust to 50k per chunk

* memory still just fine at 250k

* but slower

* commit after each chunk to reduce lock pressure

* adjust

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* tweak

* tweak

* limit cleanup

* lower limit to give some more buffer

* lower limit to give some more buffer

* where required for sqlite

* sqlite can wipe as many as needed with no limit

* limit on mysql only

* chunk postgres

* fix limit

* tweak

* fix reference

* fix

* tweak for ram

* postgres memory reduction

* defer cleanup

* fix

* same order
2023-01-27 22:39:45 -05:00