J. Nick Koston
eac9ad8437
Fix most recent states query not using the index for group by ( #88461 )
...
* Fix most recent states query not using the index for group by
fixes #87851
* Apply suggestions from code review
* reduce
2023-02-19 21:05:45 -05:00
J. Nick Koston
9a6bcc2b63
Fix incorrect created and created_ts columns in statistics database schema ( #88462 )
...
* Remove default from created statistics schema
We were still inserting created times because even though
None was passed when creating the object explictly, the
default would still be used
* adjust column
* preserve original pre sql alc 2.0 behavior
2023-02-19 21:00:49 -05:00
J. Nick Koston
a9731a7b26
Make the recorder LRU adjustment threadsafe ( #88443 )
2023-02-19 19:30:08 +01:00
J. Nick Koston
08cdb0e3ee
Fix recorder LRU being destroyed if event session is reopened ( #88448 )
2023-02-19 19:20:01 +01:00
J. Nick Koston
9df00bd285
Adjust recorder LRU cache to align with the number of entities ( #88350 )
...
* Adjust size of recorder LRU based on number of entities
If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently
* add a test
* do not actually record 4096 states
* patch target
* patch target
2023-02-19 00:02:36 -05:00
J. Nick Koston
d4376b2041
Use a single session to compile statistics ( #88405 )
...
We had two session_scope contexts in compile_statistics which causes
a commit to happen twice during the compile.
2023-02-18 20:23:47 -05:00
J. Nick Koston
3ca9f3c0d5
Only build the recorder SupportedDialect enum once ( #88402 )
...
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
Erik Montnemery
1eb20affa1
Update use of sqlalchemy with_variant ( #88395 )
2023-02-18 09:18:01 -06:00
Erik Montnemery
83e5bf7ae8
Use entity_sources to determine integration in recorder platforms ( #88382 )
2023-02-18 07:21:41 -06:00
J. Nick Koston
0a7cf2b88c
Speed up formatting history data ( #88230 )
2023-02-18 10:34:44 +01:00
J. Nick Koston
289bab6f87
Make recorder platform attribute exclude integration aware ( #88357 )
2023-02-18 10:08:59 +01:00
J. Nick Koston
b9f29c27ab
Cleanup stale recorder code ( #88275 )
...
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
J. Nick Koston
5fe8829cf6
Speed up compiling hourly statistics ( #88225 )
2023-02-16 12:55:25 -06:00
J. Nick Koston
6c542bd314
Speed up formatting statistics data ( #88228 )
...
* Speed up fetching statistics by using attrgetter
Uses the faster native code vs a lambda
4aeae28671/Modules/_operator.c (L1406)0ead89603d20014af168d5e4aac7c50cebae33f80184e47e24
2023-02-16 12:34:20 -05:00
epenet
ba23816a0c
Inverse json import logic ( #88099 )
...
* Fix helpers and util
* Adjust components
* Move back errors
* Add report
* mypy
* mypy
* Assert deprecation messages
* Move test_json_loads_object
* Adjust tests
* Fix rebase
* Adjust pylint plugin
* Fix plugin
* Adjust references
* Adjust backup tests
2023-02-16 11:37:57 +01:00
J. Nick Koston
484027023c
Fix new stats rows creating data in the old created column ( #88013 )
2023-02-13 21:53:37 -06:00
J. Nick Koston
7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field ( #87937 )
2023-02-12 14:51:10 -06:00
J. Nick Koston
32a6280139
Optimize _sorted_statistics_to_dict to avoid checking which types each loop ( #87901 )
...
* Optimize _sorted_statistics_to_dict to avoid checking which types each loop
* tweak
* avoid some more dict lookups
2023-02-11 23:21:40 -05:00
J. Nick Koston
85649ec589
Reduce overhead to see if an entity is recorded ( #87912 )
...
A significant chunk of list_statistic_ids was checking if the entity
was recorded because it had to get the recorder instance over and
over
2023-02-11 23:21:16 -05:00
J. Nick Koston
e899754919
Bump sqlalchemy to 2.0.3 ( #87913 )
...
changes: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.3
2023-02-11 23:20:22 -05:00
J. Nick Koston
86a93e9fce
Optimize fetching statistics by avoiding recalculating time boundaries ( #87859 )
...
* predictive
* fix conversion error
* fix conversion error
* fix conversion error
* convert day to use new algo
* reduce
2023-02-11 11:41:57 -05:00
J. Nick Koston
c35661947a
Optimize fetching statistics by avoiding timestamp conversions ( #87845 )
...
* Optimize fetching statistics by avoiding timestamp conversions
* remove accidential unrelated change
* fix test
* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston
ea356ad260
Optimize fetching weekly/monthly/yearly statistics ( #87747 )
...
* Optimize fetching statistics
* speed up
* avoid double groupby
* avoid another loop
* tweak flow
* fixes
* tweak
* avoid a new dt object in the cache for week/month
* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
J. Nick Koston
509de02044
Ensure recorder still shuts down if the final commit fails ( #87799 )
2023-02-09 15:12:40 -06:00
J. Nick Koston
abf0c87e40
Migrate statistics to use timestamp columns ( #87321 )
2023-02-09 12:24:19 -06:00
J. Nick Koston
db503c0daa
Always use UTC time for the MariaDB/MySQL session ( #87583 )
2023-02-09 11:39:45 -06:00
Erik Montnemery
5dfb9cddd6
Sort manifests 8 ( #87029 )
2023-02-08 20:13:46 +01:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 ( #86436 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet
a202588fd2
Add return type to json_loads ( #85672 )
...
* Add JSON type definitions
* Sample use
* Keep mutable for a follo-up PR (avoid dead code)
* Use list/dict
* Remove JsonObjectType
* Remove reference to Union
* Cleanup
* Improve rest
* Rename json_dict => json_data
* Add docstring
* Add type hint to json_loads
* Add cast
* Move type alias to json helpers
* Cleanup
* Create and use json_loads_object
* Make error more explicit and add tests
* Use JsonObjectType in conversation
* Remove quotes
2023-02-07 17:21:55 +01:00
Marc Mueller
b478b4fa16
Add ParamSpec TypeAliases [mypy 1.0] ( #87597 )
2023-02-07 15:56:26 +01:00
J. Nick Koston
0a1d5c85ad
Optimize history.get_last_state_changes query ( #87554 )
...
fixes undefined
2023-02-07 07:50:44 -06:00
J. Nick Koston
3aa744ed19
Fix indent on slow_range_in_select for MySQL/MariaDB ( #87581 )
...
We would calculate this value and throw it away
because it was only used on first_connection
2023-02-06 23:59:54 -05:00
Marc Mueller
342b406dc0
Add Self typing (1) [mypy 1.0] ( #87598 )
2023-02-06 22:29:47 -06:00
Franck Nijhof
939eef3b28
Remove translations from Core ( #87543 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2023-02-06 15:06:43 +01:00
J. Nick Koston
c7980bda82
Fix recorder run history during schema migration and startup ( #87492 )
...
Fix recorder run history during schema migration
RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.
fixes #87112
2023-02-05 22:12:41 -05:00
GitHub Action
f475446ebf
[ci skip] Translation update
2023-02-06 00:23:07 +00:00
J. Nick Koston
c11f3ffa17
Speed up purge time with newer MariaDB versions ( #87409 )
...
* Speed up purge time with newer MariaDB versions
* fix
* document
* document
* document
* rename
* self review
* Update homeassistant/components/recorder/util.py
* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof
3d557b5583
Collection of code styling tweaks ( #87381 )
2023-02-04 12:52:26 -05:00
Franck Nijhof
ca1a12898c
Enable Ruff D212 ( #87347 )
2023-02-03 23:08:48 +01:00
epenet
810367b757
Fix can not typo ( #87254 )
2023-02-03 11:37:16 +01:00
J. Nick Koston
22698b1cc5
Fix statistics graphs not loading with data_rate, electric_current, voltage, information, and unitless units ( #87202 )
...
* Add missing converts to recorder/statistics_during_period API
This was resulting in the stats graphs not loading on the frontend
* its in two places
2023-02-02 14:52:55 -05:00
J. Nick Koston
389fc515a1
Prevent overly large attributes from being stored in the database ( #87105 )
2023-02-02 11:49:33 -06:00
Franck Nijhof
5e81d28116
Update black to 23.1.0 ( #87188 )
2023-02-02 18:35:24 +01:00
epenet
6806e85eab
Add missing converters to recorder statistics ( #87137 )
2023-02-02 09:44:26 +01:00
GitHub Action
0f80932709
[ci skip] Translation update
2023-02-02 00:25:01 +00:00
J. Nick Koston
fb55933b0a
Reduce chance of queue overflow during schema migration ( #87090 )
...
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
GitHub Action
be73e43989
[ci skip] Translation update
2023-02-01 00:27:42 +00:00
J. Nick Koston
5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 ( #87040 )
...
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
J. Nick Koston
00118a6f96
Speed up live history setup if there is no pending data to commit ( #86942 )
2023-01-30 19:00:34 +01:00
J. Nick Koston
f874258e7e
Silence spurious warnings about removing ix_states_entity_id with newer installs ( #86961 )
...
* Silence spurious warnings about removing ix_states_entity_id with newer installs
10696480351069648035
2023-01-30 12:38:33 -05:00
