Commit graph

666 commits

Author SHA1 Message Date
J. Nick Koston
32a6280139
Optimize _sorted_statistics_to_dict to avoid checking which types each loop (#87901)
* Optimize _sorted_statistics_to_dict to avoid checking which types each loop

* tweak

* avoid some more dict lookups
2023-02-11 23:21:40 -05:00
J. Nick Koston
85649ec589
Reduce overhead to see if an entity is recorded (#87912)
A significant chunk of list_statistic_ids was checking if the entity
was recorded because it had to get the recorder instance over and
over
2023-02-11 23:21:16 -05:00
J. Nick Koston
e899754919
Bump sqlalchemy to 2.0.3 (#87913)
changes: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.3
2023-02-11 23:20:22 -05:00
J. Nick Koston
86a93e9fce
Optimize fetching statistics by avoiding recalculating time boundaries (#87859)
* predictive

* fix conversion error

* fix conversion error

* fix conversion error

* convert day to use new algo

* reduce
2023-02-11 11:41:57 -05:00
J. Nick Koston
c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston
ea356ad260
Optimize fetching weekly/monthly/yearly statistics (#87747)
* Optimize fetching statistics

* speed up

* avoid double groupby

* avoid another loop

* tweak flow

* fixes

* tweak

* avoid a new dt object in the cache for week/month

* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
J. Nick Koston
509de02044
Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
J. Nick Koston
db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
Erik Montnemery
5dfb9cddd6
Sort manifests 8 (#87029) 2023-02-08 20:13:46 +01:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet
a202588fd2
Add return type to json_loads (#85672)
* Add JSON type definitions

* Sample use

* Keep mutable for a follo-up PR (avoid dead code)

* Use list/dict

* Remove JsonObjectType

* Remove reference to Union

* Cleanup

* Improve rest

* Rename json_dict => json_data

* Add docstring

* Add type hint to json_loads

* Add cast

* Move type alias to json helpers

* Cleanup

* Create and use json_loads_object

* Make error more explicit and add tests

* Use JsonObjectType in conversation

* Remove quotes
2023-02-07 17:21:55 +01:00
Marc Mueller
b478b4fa16
Add ParamSpec TypeAliases [mypy 1.0] (#87597) 2023-02-07 15:56:26 +01:00
J. Nick Koston
0a1d5c85ad
Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
J. Nick Koston
3aa744ed19
Fix indent on slow_range_in_select for MySQL/MariaDB (#87581)
We would calculate this value and throw it away
because it was only used on first_connection
2023-02-06 23:59:54 -05:00
Marc Mueller
342b406dc0
Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
Franck Nijhof
939eef3b28
Remove translations from Core (#87543)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2023-02-06 15:06:43 +01:00
J. Nick Koston
c7980bda82
Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
GitHub Action
f475446ebf [ci skip] Translation update 2023-02-06 00:23:07 +00:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof
3d557b5583
Collection of code styling tweaks (#87381) 2023-02-04 12:52:26 -05:00
Franck Nijhof
ca1a12898c
Enable Ruff D212 (#87347) 2023-02-03 23:08:48 +01:00
epenet
810367b757
Fix can not typo (#87254) 2023-02-03 11:37:16 +01:00
J. Nick Koston
22698b1cc5
Fix statistics graphs not loading with data_rate, electric_current, voltage, information, and unitless units (#87202)
* Add missing converts to recorder/statistics_during_period API

This was resulting in the stats graphs not loading on the frontend

* its in two places
2023-02-02 14:52:55 -05:00
J. Nick Koston
389fc515a1
Prevent overly large attributes from being stored in the database (#87105) 2023-02-02 11:49:33 -06:00
Franck Nijhof
5e81d28116
Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet
6806e85eab
Add missing converters to recorder statistics (#87137) 2023-02-02 09:44:26 +01:00
GitHub Action
0f80932709 [ci skip] Translation update 2023-02-02 00:25:01 +00:00
J. Nick Koston
fb55933b0a
Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
GitHub Action
be73e43989 [ci skip] Translation update 2023-02-01 00:27:42 +00:00
J. Nick Koston
5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
J. Nick Koston
00118a6f96
Speed up live history setup if there is no pending data to commit (#86942) 2023-01-30 19:00:34 +01:00
J. Nick Koston
f874258e7e
Silence spurious warnings about removing ix_states_entity_id with newer installs (#86961)
* Silence spurious warnings about removing ix_states_entity_id with newer installs

https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465

* Silence spurious warnings about removing ix_states_entity_id with newer installs

https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465
2023-01-30 12:38:33 -05:00
Erik Montnemery
53c5f02ca2
Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston
4e9bd09d39
Fix old indices not being removed in schema migration leading to slow MySQL queries (#86917)
fixes #83787
2023-01-29 21:33:23 -05:00
J. Nick Koston
c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
J. Nick Koston
58de7b8df0
Fix v32 schema migration when MySQL global.time_zone is configured with non-UTC timezone (#86867)
* Fix v32 schema migration when MySQL timezone is not UTC

* tweak
2023-01-28 22:06:07 -05:00
J. Nick Koston
d97a061285
Chunk MariaDB and Postgresql data migration to avoid running out of buffer space (#86680)
* Chunk MariaDB data migration to avoid running out of buffer space

This will make the migration slower but since the innodb_buffer_pool_size
is using the defaul to 128M and not tuned to the db size there is a
risk of running out of buffer space for large databases

* Update homeassistant/components/recorder/migration.py

* hard code since bandit thinks its an injection

* Update homeassistant/components/recorder/migration.py

* guard against manually modified data/corrupt db

* adjust to 10k per chunk

* adjust to 50k per chunk

* memory still just fine at 250k

* but slower

* commit after each chunk to reduce lock pressure

* adjust

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* tweak

* tweak

* limit cleanup

* lower limit to give some more buffer

* lower limit to give some more buffer

* where required for sqlite

* sqlite can wipe as many as needed with no limit

* limit on mysql only

* chunk postgres

* fix limit

* tweak

* fix reference

* fix

* tweak for ram

* postgres memory reduction

* defer cleanup

* fix

* same order
2023-01-27 22:39:45 -05:00
Franck Nijhof
a79885ceaf
Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck Nijhof
d4955a3d87
Replace assert False by raising an error (#86686) 2023-01-26 16:51:43 +01:00
Erik Montnemery
fea30c1ce9
Terminate strings at NUL when recording states and events (#86687) 2023-01-26 11:11:03 +01:00
Marc Mueller
45b4b0e990
Import ParamSpec from typing [Py310] (#86413)
* Import ParamSpec from typing [Py310]

* Update additional imports
2023-01-23 07:28:43 +01:00
J. Nick Koston
52ea64d1d0
Fix repr for States and Events without a timestamp (#86391) 2023-01-22 08:11:42 -10:00
J. Nick Koston
4c84824ac8
Increase default recorder commit interval to 5 seconds (#86115) 2023-01-21 18:09:00 -10:00
Franck Nijhof
79b52a2b41
Stricter pylint message control (#86154) 2023-01-20 13:47:55 +01:00
Franck Nijhof
c5dedb7a79
Code styling tweaks to the recorder integration (#86030) 2023-01-16 19:51:11 +01:00
J. Nick Koston
233332c3a0
Fix fetching history include_start_time_state when timezone is not UTC (#85983) 2023-01-16 09:01:32 +01:00
J. Nick Koston
43fb68ed79
Avoid linear searches for excluded events (#85851)
If the there are a lot of excluded events for the recorder, it
can have a performance impact as the list has to be searched
every time an event fires in HA
2023-01-14 14:25:03 +02:00
Ville Skyttä
ae302bbec0
Make use of str.removeprefix and .removesuffix (#85584) 2023-01-13 12:19:38 +01:00
Erik Montnemery
b0d4b73874
Add unitless unit converter (#85694)
* Add unitless unit converter

* Adjust type hints

* Adjust tests

* Rename to UnitlessRatioConverter
2023-01-12 09:20:00 +01:00