Commit graph

231 commits

Author SHA1 Message Date
Raman Gupta
3ca69f5568
Raise an exception when event_type exceeds the max length (#48115)
* raise an exception when event_type exceeds the max length that the recorder supports

* add test

* use max length constant in recorder

* update config entry reloaded service name

* remove exception string function because it's not needed

* increase limit to 64 and revert event name change

* fix test

* assert exception args

* fix test

* add comment about migration
2021-04-08 20:46:28 +02:00
Laszlo Magyar
c7e4857d2c
Let recorder deal with event names longer than 32 chars (#47748) 2021-04-08 07:08:49 -10:00
Stefan Agner
815db999da
Use microsecond precision for datetime values on MariaDB/MySQL (#48749)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-04-07 09:13:55 +02:00
Franck Nijhof
3aed84560f
Merge of nested IF-IF cases - O-R (#48371) 2021-03-27 10:38:57 +01:00
J. Nick Koston
c820dd4cb5
Have pylint warn when user visible log messages do not start with capital letter or end with a period (#48064)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-19 09:26:36 -05:00
Marc Mueller
a3cd1854f6
Update typing 12 (#48073) 2021-03-18 14:31:38 +01:00
J. Nick Koston
61a2460c87
Improve error reporting in recorder purge test (#47929) 2021-03-14 19:46:21 -07:00
J. Nick Koston
33c4eb3434
Log the full exception when the recorder fails to setup (#47770) 2021-03-11 21:52:04 -08:00
Marc Mueller
92852b9c10
Add apply_filter attribute to recorder.purge service (#45826) 2021-03-11 17:03:30 -10:00
Marc Mueller
10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
CurrentThread
7c8851264f
Use LONGTEXT column instead of TEXT for MySQL/MariaDB and migrate existing databases (#47026) 2021-03-10 08:12:58 -10:00
J. Nick Koston
a060acc2b1
Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
adrian-vlad
44293a3738
Add enable and disable services for recorder (#45778) 2021-02-24 07:26:05 -06:00
Franck Nijhof
ea4bbd771f
Add service names to previously enriched services (#46929)
Co-authored-by: Tobias Sauerwein <cgtobi@users.noreply.github.com>
2021-02-23 14:10:13 +01:00
J. Nick Koston
9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston
22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
Franck Nijhof
8b69608242
Add selectors to Browser, Recorder, Shopping List service definitions (#46749) 2021-02-18 17:06:25 +01:00
J. Nick Koston
c9df42b69a
Add support for pre-filtering events to the event bus (#46371) 2021-02-14 09:42:55 -10:00
J. Nick Koston
dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql (#46185)
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
Fabian Affolter
60d4dadcb6
Upgrade sqlalchemy to 1.3.23 (#45845) 2021-02-01 16:03:51 -10:00
J. Nick Koston
12af87bc6e
Add index to old_state_id column for postgres and older databases (#44757)
* Add index to old_state_id column for older databases

The schema was updated in #43610 but the index was not
added on migration.

* Handle postgresql missing ondelete

* create index first
2021-01-04 10:51:44 +01:00
Fabian Affolter
12b7b2098d
Upgrade sqlalchemy to 1.3.22 (#44698) 2020-12-31 14:28:15 -10:00
Ville Skyttä
de04a1ed67
Enable more Bandit tests (#44307)
https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing
2020-12-19 12:35:13 +01:00
J. Nick Koston
2cbb93be43
Always keep the current recorder run when purging (#43733) 2020-11-30 09:13:50 +01:00
J. Nick Koston
1162d9a752
Create tables with a charset that can hold all expected data under mysql (#43732)
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.

This will only fix the issue for users who recreate their tables.
2020-11-28 23:54:05 +01:00
moinmoin-sh
337b8d279e
Ensure MariaDB/MySQL can be purged and handle states being deleted out from under the recorder (#43610)
* MariaDB doesn't purge #42402

This addresses  home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.

* Update models.py

Corrected for Black style requirements

* Update homeassistant/components/recorder/models.py

Co-authored-by: J. Nick Koston <nick@koston.org>

* Add the options to foreign key constraints

* purge old states when database gets deleted out from under us

* pylint

Co-authored-by: J. Nick Koston <nick@koston.org>
2020-11-28 08:42:29 -10:00
Franck Nijhof
5dbb5f12eb
Upgrade sqlalchemy to 1.3.20 (#41765) 2020-10-13 08:43:58 -05:00
J. Nick Koston
33b548e247
Reduce ORM overhead when the old state was already written to the database (#41736)
We can avoid processing the relationship when the old state was already
written to the database which will common case with a commit interval
of 1s. Since we already know the value for old_state_id we can use it
instead of asking sqlalchemy to process the relationship at flush/commit
time which can significantly speed up sqlalchemy's _emit_insert_statements
implementation.
2020-10-13 15:34:46 +02:00
Matthias Alphart
93a9a11065
Add config validator helper positive_float (#41640) 2020-10-11 22:04:49 +02:00
J. Nick Koston
1c431aa7bd
Set created field when creating db events and states (#41523)
Avoids thousands of utcnow calls when the session
is commited on busy systems to fill in the field
default.
2020-10-09 09:31:17 +02:00
J. Nick Koston
4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data (#41426) 2020-10-08 09:15:25 +02:00
J. Nick Koston
113d738fa2
Reduce orm overhead by grouping object expiration (#41394) 2020-10-07 15:35:48 +02:00
J. Nick Koston
d35e33610a
Resolve memory leak in recorder (#41349)
Avoids a build up of the InstanceState.
2020-10-06 20:12:47 +02:00
J. Nick Koston
4798f37c6e
Convert States to dicts via as_dict only once (#41208)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-10-05 09:18:57 -05:00
J. Nick Koston
20a136e2a1
Avoid event data serialization during recorder that we throw away (#41217)
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
2020-10-05 15:08:47 +02:00
J. Nick Koston
6b509fd9db
Prevent sqlalchemy from refetching the old_state_id as it will never change (#40982)
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
2020-10-01 19:57:52 +02:00
J. Nick Koston
d9ba32dc3f
Setup recorder model relationships to avoid calling flush (#40467) 2020-09-30 13:11:43 +02:00
J. Nick Koston
aada6a1d88
Fix logbook with empty filter and remove unused code (#40565) 2020-09-30 13:10:11 +02:00
J. Nick Koston
c19b5c5ac3
Make recorder block_till_done reliable (#40043) 2020-09-14 08:48:29 +02:00
J. Nick Koston
557684c3ce
Add ability to disable the sqlite3 quick_check (#39479) 2020-09-02 10:12:56 +02:00
Franck Nijhof
1c2ebdf307
Upgrade black to 20.8b1 (#39287) 2020-08-27 13:56:20 +02:00
Fabian Affolter
bd136fa79a
Upgrade sqlalchemy to 1.3.19 (#39167) 2020-08-22 18:09:36 -05:00
J. Nick Koston
7c346c2f7c
Skip the sqlite quick_check on clean restarts (#38972) 2020-08-21 14:20:46 +02:00
J. Nick Koston
7878d97588
Use SimpleQueue for recorder (#38967)
Now that python 3.7 is the minimum supported version, we can
use the more efficient SimpleQueue in the recorder as it does
not have to use threading.Lock
2020-08-21 13:54:13 +02:00
J. Nick Koston
ab2b2f6dd5
Accommodate systems with very large databases and slow disk/cpu (#38947)
On startup we run an sqlite3 quick_check to verify the database
integrity. In the majority of cases, the quick_check takes under
10 seconds.

On systems with very large databases and very slow disk/cpu,
this can take much longer so we freeze the timeout.
2020-08-17 09:47:50 +02:00
Pascal Vizeli
c291d4aa7d
Intelligent timeout handler for setup/bootstrap (#38329)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
2020-08-05 14:58:19 +02:00
J. Nick Koston
1acdb28cdd
Automatically recover when the sqlite3 database is malformed or corrupted (#37949)
* Validate sqlite database on startup and move away if corruption is detected.

* do not switch context in test -- its all sync
2020-07-17 19:07:37 -10:00
Franck Nijhof
53545c984b
Log lines do not end with a full stop (#37527) 2020-07-05 23:04:19 +02:00
J. Nick Koston
4b2ebf5487
Ensure removed entities are not displayed in logbook (#37395) 2020-07-03 23:08:46 -07:00
J. Nick Koston
ccb77ba1e9
Handle index already existing on db migration with MySQLdb backend (#37384)
_create_index needed the same check as _add_columns since
the MySQLdb backend throws OperationalError instead
of InternalError in this case
2020-07-03 15:35:02 -07:00