You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
mastodon/app/lib
Kaspar V 47f0d7021e
refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487)
* refactor(statuses_vacuum): remove dead code - unused

Method is not called inside class and private.
Clean up dead code.

* refactor(statuses_vacuum): make retention_period present test explicit

This private method only hides functionality.
It is best practice to be as explicit as possible.

* refactor(statuses_vacuum): improve query performance

- fix statuses_scope having sub-select for Account.remote scope by
  `joins(:account).merge(Account.remote)`
- fix statuses_scope unnecessary use of `Status.arel_table[:id].lt`
  because it is inexplicit, bad practice and even slower than normal
  `.where('statuses.id < ?'`
- fix statuses_scope remove select(:id, :visibility) for having reusable
  active record query batches (no re queries)
- fix vacuum_statuses! to use in_batches instead of find_in_batches,
  because in_batches delivers a full blown active record query result,
  in stead of an array - no requeries necessary
- send(:unlink_from_conversations) not to perform another db query, but
  reuse the in_batches result instead.
- remove now obsolete remove_from_account_conversations method
- remove_from_search_index uses array of ids, instead of mapping
  the ids from an array - this should be more efficient
- use the in_batches scope to call delete_all, instead of running
  another db query for this - because it is again more efficient
- add TODO comment for calling models private method with send

* refactor(status): simplify unlink_from_conversations

- add `has_many through:` relation mentioned_accounts
- use model scope local instead of method call `Status#local?`
- more readable add account to inbox_owners when account.local?

* refactor(status): searchable_by way less sub selects

These queries all included a sub-select. Doing the same with a joins
should be more efficient.
Since this method does 5 such queries, this should be significant,
since it technically halves the query count.

This is how it was:

```ruby
[3] pry(main)> Status.first.mentions.where(account: Account.local, silent: false).explain
  Status Load (1.6ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (1.5ms)  SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```

This is how it is with this change:

```ruby
[4] pry(main)> Status.first.mentions.joins(:account).merge(Account.local).active.explain
  Status Load (1.7ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (0.7ms)  SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```
2 years ago
..
activitypub Add featured tag add/remove activity handler (#19408) 2 years ago
admin Change incoming activity processing to happen in `ingress` queue (#20264) 2 years ago
connection_pool Add request pool to improve delivery performance (#10353) 6 years ago
importer Fix ambiguous column names in `tootctl search deploy` (#18993) 3 years ago
nodeinfo Add a nodeinfo endpoint (#12002) 6 years ago
ostatus Remove Atom feeds and old URLs in the form of `GET /:username/updates/:id` (#11247) 6 years ago
rss Fix errors when rendering RSS feeds (#18531) 3 years ago
settings Update Mastodon to Rails 6.1 (#15910) 4 years ago
translation_service Add mention of the translation provider when translating a post (#19433) 2 years ago
vacuum refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487) 2 years ago
access_token_extension.rb Fix single Redis connection being used across all threads (#18135) 3 years ago
account_reach_finder.rb Fix remote reporters not receiving suspend/unsuspend activities (#16050) 4 years ago
activity_tracker.rb Add graphs and retention metrics to admin dashboard (#16829) 4 years ago
application_extension.rb Fix confirmation redirect to app without `Location` header (#18523) 3 years ago
ascii_folding.rb Change how hashtags are normalized (#18795) 3 years ago
cache_buster.rb Add cache buster feature for media files (#15155) 4 years ago
delivery_failure_tracker.rb Fix single Redis connection being used across all threads (#18135) 3 years ago
emoji_formatter.rb Fix emoji substitution not applying only to text nodes in backend code (#20641) 2 years ago
entity_cache.rb Fix cache_collection crashing when given an empty collection (#15921) 4 years ago
extractor.rb Fix being able to post URLs longer than 4096 characters (#17908) 3 years ago
fast_geometry_parser.rb Switch from unmaintained paperclip to kt-paperclip (#16724) 4 years ago
fast_ip_map.rb Add IP-based rules (#14963) 5 years ago
feed_manager.rb Fix improperly checking for blocked domain on followed hashtags (#19472) 2 years ago
hash_object.rb Migrate from ledermann/rails-settings to rails-settings-cached which allows global settings 8 years ago
hashtag_normalizer.rb Improve performance by avoiding regex construction (#20215) 2 years ago
html_aware_formatter.rb Refactor formatter (#17828) 3 years ago
inline_renderer.rb Add caching for payload serialization during fan-out (#19642) 2 years ago
link_details_extractor.rb Add workaround for YouTube Shorts links (#17869) 3 years ago
permalink_redirector.rb Change public accounts pages to mount the web UI (#19319) 3 years ago
plain_text_formatter.rb Refactor formatter (#17828) 3 years ago
potential_friendship_tracker.rb Change auto-following admin-selected accounts, show in recommendations (#16078) 4 years ago
rate_limiter.rb Add rate limit for reporting (#13390) 5 years ago
redis_configuration.rb Add retention policy for cached content and media (#19232) 3 years ago
request.rb fixes ArgumentError when proxy is used (#20420) 2 years ago
request_pool.rb Add request pool to improve delivery performance (#10353) 6 years ago
response_with_limit.rb Fix remote files not using Content-Type header, streaming (#14184) 5 years ago
scope_parser.rb Change authorized applications page (#17656) 3 years ago
scope_transformer.rb Change authorized applications page (#17656) 3 years ago
search_query_parser.rb Fix accounts search by full/partial display name and others (#11580) 6 years ago
search_query_transformer.rb Fix error resposes for `from` search prefix (#17963) 3 years ago
status_cache_hydrator.rb Fix filter handling in status cache hydration (#19963) 2 years ago
status_filter.rb Fix n+1 queries in StatusThreadingConcern (#7321) 7 years ago
status_finder.rb Remove Atom feeds and old URLs in the form of `GET /:username/updates/:id` (#11247) 6 years ago
status_reach_finder.rb Fix reblogs being discarded after the reblogged status (#19731) 2 years ago
suspicious_sign_in_detector.rb Remove sign-in token authentication, instead send e-mail about new sign-in (#17970) 3 years ago
tag_manager.rb Fix TagManager#local_url? erroring out on invalid URL (#18580) 3 years ago
text_formatter.rb Refactor formatter (#17828) 3 years ago
themes.rb Add support for multiple themes (#4959) 8 years ago
toc_generator.rb Add fallback section ID with ToC (#11941) 6 years ago
translation_service.rb Change “Translate” button to only show up when a translation backend is configured (#19434) 2 years ago
user_settings_decorator.rb Change e-mail notifications to only be sent when recipient is offline (#17984) 3 years ago
vacuum.rb Add retention policy for cached content and media (#19232) 3 years ago
validation_error_formatter.rb Add `details` to error response for `POST /api/v1/accounts` in REST API (#15803) 4 years ago
video_metadata_extractor.rb Fix some media attachments being converted with too high framerates (#17619) 3 years ago
webfinger.rb Improve error reporting and logging when processing remote accounts (#15605) 3 years ago
webfinger_resource.rb Fix webfinger returning wrong status code on malformed or missing param (#13759) 5 years ago