Commit graph

22 commits

Author SHA1 Message Date
J. Nick Koston
20b6c4c48e
Fix recorder hanging at start (#75627) 2022-07-22 12:37:25 -05:00
Erik Montnemery
fd6ffef52f
Support non-live database migration (#72433)
* Support non-live database migration

* Tweak startup order, add test

* Address review comments

* Fix typo

* Clarify comment about promoting dependencies

* Tweak

* Fix merge mistake

* Fix some tests

* Fix additional test

* Fix additional test

* Adjust tests

* Improve test coverage
2022-07-22 15:11:34 +02:00
Thibault Cohen
1d7d2875e1
Add websocket command recorder/import_statistics (#73937)
* Expose ws_add_external_statistics in websocket API

* Refactor

* Add tests

* Improve test coverage

Co-authored-by: Thibault Cohen <titilambert@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2022-07-21 12:36:49 +02:00
Marc Mueller
ca1f0909fb
Fix spelling in recorder integration (#75539) 2022-07-20 19:13:10 -05:00
J. Nick Koston
8b067e83f7
Initial orjson support take 3 (#73849)
* Initial orjson support take 2

Still need to work out problem building wheels

--

Redux of #72754 / #32153 Now possible since the following is solved:
ijl/orjson#220 (comment)

This implements orjson where we use our default encoder.  This does not implement orjson where `ExtendedJSONEncoder` is used as these areas tend to be called far less frequently.  If its desired, this could be done in a followup, but it seemed like a case of diminishing returns (except maybe for large diagnostics files, or traces, but those are not expected to be downloaded frequently).

Areas where this makes a perceptible difference:
- Anything that subscribes to entities (Initial subscribe_entities payload)
- Initial download of registries on first connection / restore
- History queries
- Saving states to the database
- Large logbook queries
- Anything that subscribes to events (appdaemon)

Cavets:
orjson supports serializing dataclasses natively (and much faster) which
eliminates the need to implement `as_dict` in many places
when the data is already in a dataclass. This works
well as long as all the data in the dataclass can also
be serialized. I audited all places where we have an `as_dict`
for a dataclass and found only backups needs to be adjusted (support for `Path` needed to be added for backups).  I was a little bit worried about `SensorExtraStoredData` with `Decimal` but it all seems to work out from since it converts it before it gets to the json encoding cc @dgomes

If it turns out to be a problem we can disable this
with option |= [orjson.OPT_PASSTHROUGH_DATACLASS](https://github.com/ijl/orjson#opt_passthrough_dataclass) and it
will fallback to `as_dict`

Its quite impressive for history queries
<img width="1271" alt="Screen_Shot_2022-05-30_at_23_46_30" src="https://user-images.githubusercontent.com/663432/171145699-661ad9db-d91d-4b2d-9c1a-9d7866c03a73.png">

* use for views as well

* handle UnicodeEncodeError

* tweak

* DRY

* DRY

* not needed

* fix tests

* Update tests/components/http/test_view.py

* Update tests/components/http/test_view.py

* black

* templates
2022-06-22 21:59:51 +02:00
Erik Montnemery
5f2b4001f3
Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston
f79e5e002b
Ensure recorder shuts down when its startup future is canceled out from under it (#72866) 2022-06-01 22:13:09 -07:00
J. Nick Koston
c365454afb
Revert "Initial orjson support (#72754)" (#72789)
This was causing the wheels to fail to build. We need
to workout why when we don't have release pressure

This reverts commit d9d22a9556.
2022-05-31 13:51:55 -07:00
J. Nick Koston
d9d22a9556
Initial orjson support (#72754) 2022-05-31 12:18:11 -07:00
Erik Montnemery
23bd64b7a2
Prevent duplication of statistics metadata (#71637)
* Prevent duplication of statistics metadata

* Add models_schema_28.py

* Handle entity renaming as a recorder job

* Improve tests
2022-05-24 15:34:46 +02:00
J. Nick Koston
4dbc1ed7a7
Add dialect (database engine) and version to recorder system health data (#72339) 2022-05-22 21:29:30 -07:00
J. Nick Koston
9c3f949165
Add live streaming logbook websocket endpoint (#72258)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-22 14:57:54 -05:00
Erik Montnemery
edd7a3427c
Remove support for databases without ROW_NUMBER (#72092) 2022-05-18 21:52:38 -05:00
Erik Montnemery
037f6947d8
Fail recorder setup with unsupported dialect or version (#70888) 2022-05-18 16:52:46 +02:00
J. Nick Koston
8fcee01db0
Remove unnecessary flush from recorder (#71910) 2022-05-16 18:03:02 -05:00
J. Nick Koston
a8f1dda004
Use ciso8601 for parsing MySQLdb datetimes (#71818)
* Use ciso8601 for parsing MySQLDB datetimes

The default parser is this:

5340191feb/MySQLdb/times.py (L66)

* tweak

* tweak

* add coverage for building the MySQLdb connect conv param
2022-05-13 19:26:09 -05:00
J. Nick Koston
a8aa0e1cca
Add Estimated Database Size to the recorder system health (#71463) 2022-05-07 21:02:54 -07:00
J. Nick Koston
d612b9e0b4
Reduce event loop overhead for listeners that already queue (#71364)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-05 23:09:10 -04:00
J. Nick Koston
eb77f8db85
Complete strict typing for recorder (#71274)
* Complete strict typing for recorder

* update tests

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Remove the asserts

* remove ignore comments

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
J. Nick Koston
13ce0a7d6a
Fix history using pre v25 queries during v26 migration (#71295) 2022-05-04 08:56:50 -07:00
J. Nick Koston
e30940ef2a
Move processing of recorder service call arguments into services.py (#71260) 2022-05-03 13:56:22 -07:00
J. Nick Koston
29bda196b5
Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00