If a user manually migrated their database to MySQL or PostgresSQL
and incorrectly created the timestamp columns as float we would
fail to correct them to double because when we migrated to use
timestamps for the columns I missed that we needed to change the
columns and types for µs precision
- If the user had previously duplicated data we could end up
picking the next metadata_id and there could be stale rows
in the database that have that metadata_id. This can only happen
from bad manual migrations (which is what this is function
is validating in the first place). To solve this we now insert
data with a future date and look at the latest inserted row
instead of the first.
Example
```
['stored_statistics',
defaultdict(<class 'list'>,
{'recorder.db_test_schema': [{'end': 948589200.0,
'last_reset': None,
'max': None,
'mean': 2021.0,
'min': None,
'start': 948585600.0,
'state': None,
'sum': 394.5068},
{'end': 1601946000.000001,
'last_reset': 1601942400.000001,
'max': 1.000000000000001,
'mean': 1.000000000000001,
'min': 1.000000000000001,
'start': 1601942400.000001,
'state': 1.000000000000001,
'sum': 1.000000000000001}]})]
```
* Fix cpu thrashing during purge after all legacy events were removed
We now remove the the index of of event ids on the states table when its
all NULLs to save space. The purge path needs to avoid checking for legacy
rows to purge if the index has been removed since it will result in a full
table scan each purge cycle that will always find no legacy rows to purge
* one more place
* drop the key constraint as well
* fixes
* more sqlite
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* remove exception catch since the threading.excepthook will actually catch this in production
* fix a few missed ones
* threadsafe
* Update homeassistant/components/recorder/table_managers/statistics_meta.py
* coverage and optimistic caching
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* reduce
* comments
* Split context id migration into states and events tasks
Since events can finish much earlier than states we
would keep looking at the table because states as not
done. Make them seperate tasks
* add retry dec
* fix migration happening twice
* another case
* Deduplicate event_types in the events table
* Deduplicate event_types in the events table
* more fixes
* adjust
* adjust
* fix product
* fix tests
* adjust
* migrate
* migrate
* migrate
* more test fixes
* more test fixes
* fix
* migration test
* adjust
* speed up
* fix index
* fix more tests
* handle db failure
* preload
* tweak
* adjust
* fix stale docs strings, remove dead code
* refactor
* fix slow tests
* coverage
* self join to resolve query performance
* fix typo
* no need for quiet
* no need to drop index already dropped
* remove index that will never be used
* drop index sooner as we no longer use it
* Revert "remove index that will never be used"
This reverts commit 461aad2c52.
* typo
* Make sql subqueries threadsafe
fixes#89224
* fix join outside of lambda
* move statement generation into a seperate function to make it easier to test
* add cache key tests
* no need to mock hass
* Load pending state attributes and event data ids at startup
Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries
* lru
* fix hang
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* cleanup
* Mark PostgreSQL range select as fast
Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ
fixes#83253
* Mark PostgreSQL range select as fast
Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ
fixes#83253
* Adjust size of recorder LRU based on number of entities
If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently
* add a test
* do not actually record 4096 states
* patch target
* patch target
Dropping the database after this test will fail on
MySQL and hang forever because it causes an InnoDB deadlock
```
| 2042 | root | localhost:52698 | NULL | Query | 41 | Waiting for table metadata lock | DROP DATABASE `homeassistant-test` | 0.000 |
```
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753
Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
