* Avoid duplicate timestamp conversions for websocket api and recorder
We convert the time from datetime to timestamps one per
open websocket connection and the recorder for every
state update. Only do the conversion once since its
~30% of the cost of building the state diff
* more
* two more
* two more in live history
* Add support for streaming (push) history
Currently we poll for history data, like logbook we can stream
this data to avoid database overhead
* Update homeassistant/components/history/__init__.py
* merge filter
* expose new api
* expose new api
* expose new api
* expose new api
* coverage
* tests
* fixes
* tweak
* tweak
* tweak
* DRY
* leaky
* test for specific entities
* test for specific entities
* test for specific entities
* test for specific entities
* test for specific entities
* cover
* cover
* more cover
* tweak
* make sure it works before history starts
* fix test
* cover
* tweak
* make sure we unsub on overflow
* Update homeassistant/components/history/__init__.py
* Update homeassistant/components/history/__init__.py
* fix race in test
* fix db executor access
* relo above task creation
* Initial orjson support take 2
Still need to work out problem building wheels
--
Redux of #72754 / #32153 Now possible since the following is solved:
ijl/orjson#220 (comment)
This implements orjson where we use our default encoder. This does not implement orjson where `ExtendedJSONEncoder` is used as these areas tend to be called far less frequently. If its desired, this could be done in a followup, but it seemed like a case of diminishing returns (except maybe for large diagnostics files, or traces, but those are not expected to be downloaded frequently).
Areas where this makes a perceptible difference:
- Anything that subscribes to entities (Initial subscribe_entities payload)
- Initial download of registries on first connection / restore
- History queries
- Saving states to the database
- Large logbook queries
- Anything that subscribes to events (appdaemon)
Cavets:
orjson supports serializing dataclasses natively (and much faster) which
eliminates the need to implement `as_dict` in many places
when the data is already in a dataclass. This works
well as long as all the data in the dataclass can also
be serialized. I audited all places where we have an `as_dict`
for a dataclass and found only backups needs to be adjusted (support for `Path` needed to be added for backups). I was a little bit worried about `SensorExtraStoredData` with `Decimal` but it all seems to work out from since it converts it before it gets to the json encoding cc @dgomes
If it turns out to be a problem we can disable this
with option |= [orjson.OPT_PASSTHROUGH_DATACLASS](https://github.com/ijl/orjson#opt_passthrough_dataclass) and it
will fallback to `as_dict`
Its quite impressive for history queries
<img width="1271" alt="Screen_Shot_2022-05-30_at_23_46_30" src="https://user-images.githubusercontent.com/663432/171145699-661ad9db-d91d-4b2d-9c1a-9d7866c03a73.png">
* use for views as well
* handle UnicodeEncodeError
* tweak
* DRY
* DRY
* not needed
* fix tests
* Update tests/components/http/test_view.py
* Update tests/components/http/test_view.py
* black
* templates
* Add WS API for listing available statistic ids
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
