Commit graph

243 commits

Author SHA1 Message Date
J. Nick Koston
906de23087
Bump sqlalchemy to 1.4.13 (#50138) 2021-05-05 20:08:48 -07:00
J. Nick Koston
1c8d9ca68b
Check exception causes for matching strings during recorder migration (#49999) 2021-05-02 20:57:42 -07:00
Paulus Schoutsen
1bd9826684
Handle different entity_id formats (#49969) 2021-05-01 20:30:28 -07:00
J. Nick Koston
b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 (#49693) 2021-04-26 19:20:31 -10:00
J. Nick Koston
376b787e4d
Skip recorder commit if there is nothing to do (#49614) 2021-04-25 12:05:49 +02:00
jan iversen
c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick Koston
c10836fcee
Upgrade to sqlalchemy 1.4.11 (#49538) 2021-04-21 20:29:36 -10:00
J. Nick Koston
6d137d2316
Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston
e24f5831a2
Force recorder shutdown at final write event (#49145)
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE

* remove unreachable

* remove unreachable

* simplify

* cancel in async
2021-04-19 11:24:17 +02:00
Franck Nijhof
055cdc64c0
Add support for IoT class in manifest (#46935) 2021-04-15 10:21:38 +02:00
J. Nick Koston
53853f035d
Prevent calling stop or restart services during db upgrade (#49098) 2021-04-12 17:18:38 -07:00
J. Nick Koston
9368891b1b
Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Raman Gupta
3ca69f5568
Raise an exception when event_type exceeds the max length (#48115)
* raise an exception when event_type exceeds the max length that the recorder supports

* add test

* use max length constant in recorder

* update config entry reloaded service name

* remove exception string function because it's not needed

* increase limit to 64 and revert event name change

* fix test

* assert exception args

* fix test

* add comment about migration
2021-04-08 20:46:28 +02:00
Laszlo Magyar
c7e4857d2c
Let recorder deal with event names longer than 32 chars (#47748) 2021-04-08 07:08:49 -10:00
Stefan Agner
815db999da
Use microsecond precision for datetime values on MariaDB/MySQL (#48749)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-04-07 09:13:55 +02:00
Franck Nijhof
3aed84560f
Merge of nested IF-IF cases - O-R (#48371) 2021-03-27 10:38:57 +01:00
J. Nick Koston
c820dd4cb5
Have pylint warn when user visible log messages do not start with capital letter or end with a period (#48064)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-19 09:26:36 -05:00
Marc Mueller
a3cd1854f6
Update typing 12 (#48073) 2021-03-18 14:31:38 +01:00
J. Nick Koston
61a2460c87
Improve error reporting in recorder purge test (#47929) 2021-03-14 19:46:21 -07:00
J. Nick Koston
33c4eb3434
Log the full exception when the recorder fails to setup (#47770) 2021-03-11 21:52:04 -08:00
Marc Mueller
92852b9c10
Add apply_filter attribute to recorder.purge service (#45826) 2021-03-11 17:03:30 -10:00
Marc Mueller
10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
CurrentThread
7c8851264f
Use LONGTEXT column instead of TEXT for MySQL/MariaDB and migrate existing databases (#47026) 2021-03-10 08:12:58 -10:00
J. Nick Koston
a060acc2b1
Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
adrian-vlad
44293a3738
Add enable and disable services for recorder (#45778) 2021-02-24 07:26:05 -06:00
Franck Nijhof
ea4bbd771f
Add service names to previously enriched services (#46929)
Co-authored-by: Tobias Sauerwein <cgtobi@users.noreply.github.com>
2021-02-23 14:10:13 +01:00
J. Nick Koston
9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston
22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
Franck Nijhof
8b69608242
Add selectors to Browser, Recorder, Shopping List service definitions (#46749) 2021-02-18 17:06:25 +01:00
J. Nick Koston
c9df42b69a
Add support for pre-filtering events to the event bus (#46371) 2021-02-14 09:42:55 -10:00
J. Nick Koston
dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql (#46185)
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
Fabian Affolter
60d4dadcb6
Upgrade sqlalchemy to 1.3.23 (#45845) 2021-02-01 16:03:51 -10:00
J. Nick Koston
12af87bc6e
Add index to old_state_id column for postgres and older databases (#44757)
* Add index to old_state_id column for older databases

The schema was updated in #43610 but the index was not
added on migration.

* Handle postgresql missing ondelete

* create index first
2021-01-04 10:51:44 +01:00
Fabian Affolter
12b7b2098d
Upgrade sqlalchemy to 1.3.22 (#44698) 2020-12-31 14:28:15 -10:00
Ville Skyttä
de04a1ed67
Enable more Bandit tests (#44307)
https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing
2020-12-19 12:35:13 +01:00
J. Nick Koston
2cbb93be43
Always keep the current recorder run when purging (#43733) 2020-11-30 09:13:50 +01:00
J. Nick Koston
1162d9a752
Create tables with a charset that can hold all expected data under mysql (#43732)
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.

This will only fix the issue for users who recreate their tables.
2020-11-28 23:54:05 +01:00
moinmoin-sh
337b8d279e
Ensure MariaDB/MySQL can be purged and handle states being deleted out from under the recorder (#43610)
* MariaDB doesn't purge #42402

This addresses  home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.

* Update models.py

Corrected for Black style requirements

* Update homeassistant/components/recorder/models.py

Co-authored-by: J. Nick Koston <nick@koston.org>

* Add the options to foreign key constraints

* purge old states when database gets deleted out from under us

* pylint

Co-authored-by: J. Nick Koston <nick@koston.org>
2020-11-28 08:42:29 -10:00
Franck Nijhof
5dbb5f12eb
Upgrade sqlalchemy to 1.3.20 (#41765) 2020-10-13 08:43:58 -05:00
J. Nick Koston
33b548e247
Reduce ORM overhead when the old state was already written to the database (#41736)
We can avoid processing the relationship when the old state was already
written to the database which will common case with a commit interval
of 1s. Since we already know the value for old_state_id we can use it
instead of asking sqlalchemy to process the relationship at flush/commit
time which can significantly speed up sqlalchemy's _emit_insert_statements
implementation.
2020-10-13 15:34:46 +02:00
Matthias Alphart
93a9a11065
Add config validator helper positive_float (#41640) 2020-10-11 22:04:49 +02:00
J. Nick Koston
1c431aa7bd
Set created field when creating db events and states (#41523)
Avoids thousands of utcnow calls when the session
is commited on busy systems to fill in the field
default.
2020-10-09 09:31:17 +02:00
J. Nick Koston
4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data (#41426) 2020-10-08 09:15:25 +02:00
J. Nick Koston
113d738fa2
Reduce orm overhead by grouping object expiration (#41394) 2020-10-07 15:35:48 +02:00
J. Nick Koston
d35e33610a
Resolve memory leak in recorder (#41349)
Avoids a build up of the InstanceState.
2020-10-06 20:12:47 +02:00
J. Nick Koston
4798f37c6e
Convert States to dicts via as_dict only once (#41208)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-10-05 09:18:57 -05:00
J. Nick Koston
20a136e2a1
Avoid event data serialization during recorder that we throw away (#41217)
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
2020-10-05 15:08:47 +02:00
J. Nick Koston
6b509fd9db
Prevent sqlalchemy from refetching the old_state_id as it will never change (#40982)
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
2020-10-01 19:57:52 +02:00
J. Nick Koston
d9ba32dc3f
Setup recorder model relationships to avoid calling flush (#40467) 2020-09-30 13:11:43 +02:00
J. Nick Koston
aada6a1d88
Fix logbook with empty filter and remove unused code (#40565) 2020-09-30 13:10:11 +02:00