118 Commits (main)

Author	SHA1	Message	Date
Claire	32a030dd74	Rewrite import feature (#21054)	2 years ago
Claire	5dc3173ef8	Fix AccountsStatusesCleanupScheduler not spreading deletes across accounts correctly (#24607)	2 years ago
Christian Schmidt	eb38e9df31	Requeue expiration notification (#24311)	2 years ago
Matt Jankowski	688287c59d	Coverage improvement round-out following up previous work (#23987)	2 years ago
Matt Jankowski	39e7525c96	Add basic coverage for some worker jobs (#23943)	2 years ago
Stanislav Dobrovolschii	d9271126ce	Add rspecs for FollowRecommendationsScheduler (#23890)	2 years ago
Claire	20b80c62ff	Change auto-deletion throttling constants to better scale with server size (#23320)	2 years ago
Nick Schonning	84cc805cae	Enable Style/FrozenStringLiteralComment for specs (#23790)	2 years ago
Nick Schonning	0cfdd1a401	Enable Rubocop Style/StringConcatenation defaults (#23792)	2 years ago
Nick Schonning	5116347eb7	Autofix Rubocop RSpec/BeEq (#23740)	2 years ago
Nick Schonning	bf785df9fe	Audofix Rubocop Style/WordArray (#23739)	2 years ago
Nick Schonning	4552685f6b	Autofix Rubocop RSpec/LeadingSubject (#23670)	2 years ago
Nick Schonning	bd1d57c230	Autofix Rubocop RSpec/EmptyLineAfterSubject (#23719)	2 years ago
Nick Schonning	65ba0d92ef	Enable Rubocop RSpec/NotToNot (#23723)	2 years ago
Nick Schonning	08289a38fa	Autofix Rubocop Style/TrailingCommaInArrayLiteral (#23692)	2 years ago
Nick Schonning	669f6d2c0a	Run rubocop formatting except line length (#23632)	2 years ago
Claire	9edefc779f	Fix `UserCleanupScheduler` crash when an unconfirmed account has a moderation note (#23318) ... * Fix `UserCleanupScheduler` crash when an unconfirmed account has a moderation note * Add tests	2 years ago
Claire	5925a31b78	Fix followers count not being updated when migrating follows (#19998) ... Fixes #19900	2 years ago
Eugen Rochko	5c9abdeff1	Add retention policy for cached content and media (#19232)	2 years ago
Eugen Rochko	50948b46aa	Add ability to filter followed accounts' posts by language (#19095)	2 years ago
Eugen Rochko	0b3e4fd5de	Remove digest e-mails (#17985) ... * Remove digest e-mails * Remove digest-related code	2 years ago
Eugen Rochko	3917353645	Fix single Redis connection being used across all threads (#18135) ... * Fix single Redis connection being used across all Sidekiq threads * Fix tests	3 years ago
Claire	2af03164cb	Improve tests involving push_bulk (#17508) ... sidekiq-bulk's push_bulk can either accept arguments directly or run them through a block. Setting expectations on the result of evaluating the blocks allows testing more code (the block itself) and the test is moved closer to the *interface* of the tested code than its precise implementation.	3 years ago
Claire	da91b18a8b	Fix NoMethodError in StatusUpdateDistributionWorker (#17499) ... * Add tests * Fix NoMethodError in StatusUpdateDistributionWorker * Fix tests	3 years ago
Claire	e38fc319dc	Refactor and improve tests (#17386) ... * Change account and user fabricators to simplify and improve tests - `Fabricate(:account)` implicitly fabricates an associated `user` if no `domain` attribute is given (an account with `domain: nil` is considered a local account, but no user record was created), unless `user: nil` is passed - `Fabricate(:account, user: Fabricate(:user))` should still be possible but is discouraged. * Fix and refactor tests - avoid passing unneeded attributes to `Fabricate(:user)` or `Fabricate(:account)` - avoid embedding `Fabricate(:user)` into a `Fabricate(:account)` or the other way around - prefer `Fabricate(:user, account_attributes: …)` to `Fabricate(:user, account: Fabricate(:account, …)` - also, some tests were using remote accounts with local user records, which is not representative of production code.	3 years ago
Eugen Rochko	1060666c58	Add support for editing for published statuses (#16697) ... * Add support for editing for published statuses * Fix references to stripped-out code * Various fixes and improvements * Further fixes and improvements * Fix updates being potentially sent to unauthorized recipients * Various fixes and improvements * Fix wrong words in test * Fix notifying accounts that were tagged but were not in the audience * Fix mistake	3 years ago
Claire	7f803c41e2	Add ability to purge undeliverable domains from admin interface (#16686) ... * Add ability to purge undeliverable domains from admin interface * Add tests	3 years ago
Claire	87085a5152	Fix AccountNote not having a maximum length (#16942)	3 years ago
Claire	989c67d29d	Fix handling announcements with links (#16941) ... Broken since #15827	3 years ago
Claire	5efb1ff337	Fix followers synchronization mechanism not working when URI has empty path (#16510) ... * Fix followers synchronization mechanism not working when URI has empty path To my knowledge, there is no current implementation on the fediverse that can use bare domains (e.g., actor is at https://example.org instead of something like https://example.org/actor) that also plans to support the followers synchronization mechanism. However, Mastodon's current implementation would exclude such accounts from followers list. Also adds tests and rename them to reflect the proper method names. * Move url prefix regexp to its own constant	3 years ago
Claire	4ac78e2a06	Add feature to automatically delete old toots (#16529) ... * Add account statuses cleanup policy model * Record last inspected toot to delete to speed up successive calls to statuses_to_delete * Add service to cleanup a given account's statuses within a budget * Add worker to go through account policies and delete old toots * Fix last inspected status id logic All existing statuses older or equal to last inspected status id must be kept by the current policy. This is an invariant that must be kept so that resuming deletion from the last inspected status remains sound. * Add tests * Refactor scheduler and add tests * Add user interface * Add support for discriminating based on boosts/favs * Add UI support for min_reblogs and min_favs, rework UI * Address first round of review comments * Replace Snowflake#id_at_start with with_random parameter * Add tests * Add tests for StatusesCleanupController * Rework settings page * Adjust load-avoiding mechanisms * Please CodeClimate	3 years ago
Eugen Rochko	120965eb0b	Change Web Push API deliveries to use request pooling (#16014)	4 years ago
ThibG	f1f96ebf02	Fix being able to import more than allowed number of follows (#15384) ... * Fix being able to import more than allowed number of follows Without this commit, if someone tries importing a second list of accounts to follow before the first one has been processed, this will queue imports for the two whole lists, even if they exceed the account's allowed number of outgoing follows. This commit changes it so the individual queued imports aren't exempt from the follow limit check (they remain exempt from the rate-limiting check though). * Catch validation errors to not re-queue failed follows Co-authored-by: Claire <claire.github-309c@sitedethib.com>	4 years ago
ThibG	ca56527140	Add follower synchronization mechanism (#14510) ... * Add support for followers synchronization on the receiving end Check the `collectionSynchronization` attribute on `Create` and `Announce` activities and synchronize followers from provided collection if possible. * Add tests for followers synchronization on the receiving end * Add support for follower synchronization on the sender's end * Add tests for the sending end * Switch from AS attributes to HTTP header Replace the custom `collectionSynchronization` ActivityStreams attribute by an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as the `Signature` header and the following fields: - `collectionId` to specify which collection to synchronize - `digest` for the SHA256 hex-digest of the list of followers known on the receiving instance (where “receiving instance” is determined by accounts sharing the same host name for their ActivityPub actor `id`) - `url` of a collection that should be fetched by the instance actor Internally, move away from the webfinger-based `domain` attribute and use account `uri` prefix to group accounts. * Add environment variable to disable followers synchronization Since the whole mechanism relies on some new preconditions that, in some extremely rare cases, might not be met, add an environment variable (DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and avoid followers being incorrectly removed. The current conditions are: 1. all managed accounts' actor `id` and inbox URL have the same URI scheme and netloc. 2. all accounts whose actor `id` or inbox URL share the same URI scheme and netloc as a managed account must be managed by the same Mastodon instance as well. As far as Mastodon is concerned, breaking those preconditions require extensive configuration changes in the reverse proxy and might also cause other issues. Therefore, this environment variable provides a way out for people with highly unusual configurations, and can be safely ignored for the overwhelming majority of Mastodon administrators. * Only set follower synchronization header on non-public statuses This is to avoid unnecessary computations and allow Follow-related activities to be handled by the usual codepath instead of going through the synchronization mechanism (otherwise, any Follow/Undo/Accept activity would trigger the synchronization mechanism even if processing the activity itself would be enough to re-introduce synchronization) * Change how ActivityPub::SynchronizeFollowersService handles follow requests If the remote lists a local follower which we only know has sent a follow request, consider the follow request as accepted instead of sending an Undo. * Integrate review feeback - rename X-AS-Collection-Synchronization to Collection-Synchronization - various minor refactoring and code style changes * Only select required fields when computing followers_hash * Use actor URI rather than webfinger domain in synchronization endpoint * Change hash computation to be a XOR of individual hashes Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues. * Marginally improve followers_hash computation speed * Further improve hash computation performances by using pluck_each	4 years ago
Eugen Rochko	974b1b79ce	Add option to be notified when a followed user posts (#13546) ... * Add bell button Fix #4890 * Remove duplicate type from post-deployment migration * Fix legacy class type mappings * Improve query performance with better index * Fix validation * Remove redundant index from notifications	4 years ago
Eugen Rochko	68d3b160de	Fix various warnings in rspec (#14729)	4 years ago
ThibG	35cedc922c	Change move handler to carry blocks over (#14144) ... * Change move handler to carry blocks and mutes over When user A blocks user B and B moves to a new account C, make A block C accordingly. Note that it only works if A's instance is aware of the Move, that is, if B is on A's instance or has followers there. * Also notify instances with known people blocking you when moving * Add automatic account notes when blocking/muting an account that had no note	5 years ago
ThibG	65506bac3f	Add user notes on accounts (#14148) ... * Add UserNote model * Add UI for user notes * Put comment in relationships entity * Add API to create user notes * Copy user notes to new account when receiving a Move activity * Address some of the review remarks * Replace modal by inline edition * Please CodeClimate * Button design changes * Change design again * Cancel note edition when pressing Escape * Fixes * Tweak design again * Move “Add note” item, and allow users to add notes to themselves * Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”	5 years ago
ThibG	89f40b6c3e	Make domain block/silence/reject-media code more robust (#13424) ... * Split media cleanup from reject-media domain blocks to its own service * Slightly improve ClearDomainMediaService error handling * Lower DomainClearMediaWorker to lowest-priority queue * Do not catch ActiveRecord::RecordNotFound in domain block workers * Fix DomainBlockWorker spec labels * Add some specs * Change domain blocks to immediately mark accounts as suspended Rather than doing so sequentially, account after account, while cleaning their data. This doesn't change much about the time the block takes to complete, but it immediately prevents interaction with the blocked domain, while up to now, it would only be guaranteed when the process ends.	5 years ago
ThibG	aa138ea350	Fix RefollowWorker not keeping show_reblogs setting (#12707) ... * Fix RefollowWorker not keeping show_reblogs setting * Fix RefollowWorker	5 years ago
ThibG	66c1fe0495	Fix various issues with account migration (#12301) ... * Fix being able to follow oneself by moving to an account that was following the old one * Add specs * Add spec to catch MoveWorker issue with local followers following both accounts * Fix move worker breaking when a local account follows both source and target accounts * Fix migration from remote to local account not sending Undo Follow * Fix show_reblogs not being preserved for moved account's followers	5 years ago
ThibG	bced70469a	Add domain block notes (#11515) ... * Add database columns for adding notes to domain blocks/restrctions * Add admin UI to set private and public comments when blocking a domain * Add text for private and public comments on domain blocks * Show domain block comments in admin UI * Add comments to the domain block undo page * Make UnblockDomainService more robust regarding upgraded domain blocks * Allow editing domain blocks * Rename button from “undo domain block” to “view domain block” in account admin UI * Change test to unsilence silenced users from upgraded blocks	6 years ago
Eugen Rochko	23aeef52cc	Remove Salmon and PubSubHubbub (#11205) ... * Remove Salmon and PubSubHubbub endpoints * Add error when trying to follow OStatus accounts * Fix new accounts not being created in ResolveAccountService	6 years ago
ThibG	9a90ec3b3b	Fix account URI in UpdatePollSerializer (#11194) ... * Fix account URI in UpdatePollSerializer Fixes #11185 * Add specs	6 years ago
ThibG	9d3c6f1849	Improved remote thread fetching (#10106) ... * Fetch up to 5 replies when discovering a new remote status This is used for resolving threads downwards. The originating server must add a “replies” attributes with such replies for it to be useful. * Add some tests for ActivityPub::FetchRepliesWorker * Add specs for ActivityPub::FetchRepliesService * Serialize up to 5 public self-replies for ActivityPub notes * Add specs for ActivityPub::NoteSerializer * Move exponential backoff logic to a worker concern * Fetch first page of paginated collections when fetching thread replies * Add specs for paginated collections in replies * Move Note replies serialization to a first CollectionPage The collection isn't actually paginable yet as it has no id nor a `next` field. This may come in another PR. * Use pluck(:uri) instead of map(&:uri) to improve performances * Fix fetching replies when they are in a CollectionPage	6 years ago
Eugen Rochko	a49d43d112	Add scheduled statuses (#9706) ... Fix #340	6 years ago
Eugen Rochko	24cafd73a2	Lists (#5703) ... * Add structure for lists * Add list timeline streaming API * Add list APIs, bind list-account relation to follow relation * Add API for adding/removing accounts from lists * Add pagination to lists API * Add pagination to list accounts API * Adjust scopes for new APIs - Creating and modifying lists merely requires "write" scope - Fetching information about lists merely requires "read" scope * Add test for wrong user context on list timeline * Clean up tests	7 years ago
aschmitz	554c2fd8af	Clean up reblog tracking keys, related improvements (#5428) ... * Clean up reblog-tracking sets from FeedManager Builds on #5419, with a few minor optimizations and cleanup of sets after they are no longer needed. * Update tests, fix multiply-reblogged case Previously, we would have lost the fact that a given status was reblogged if the displayed reblog of it was removed, now we don't. Also added tests to make sure FeedManager#trim cleans up our reblog tracking keys, fixed up FeedCleanupScheduler to use the right loop, and fixed the test for it.	7 years ago
Eugen Rochko	e528114c53	Follow-up to #4582 and #5027, removing dead code (#5101)	7 years ago
Eugen Rochko	cf615abbf9	Add configuration to disable private status federation over PuSH (#4582)	7 years ago
Eugen Rochko	412ea87306	Improve ActivityPub/OStatus compatibility (#4632) ... *Note: OStatus URIs are invalid for ActivityPub. But we have them for as long as we want to keep old OStatus-sourced content and as long as we remain OStatus-compatible.* - In Announce handling, if object URI is not a URL, fallback to object URL - Do not use specialized ThreadResolveWorker, rely on generalized handling - When serializing notes, if parent's URI is not a URL, use parent's URL	8 years ago
Eugen Rochko	b7370ac8ba	ActivityPub delivery (#4566) ... * Deliver ActivityPub Like * Deliver ActivityPub Undo-Like * Deliver ActivityPub Create/Announce activities * Deliver ActivityPub creates from mentions * Deliver ActivityPub Block/Undo-Block * Deliver ActivityPub Accept/Reject-Follow * Deliver ActivityPub Undo-Follow * Deliver ActivityPub Follow * Deliver ActivityPub Delete activities Incidentally fix #889 * Adjust BatchedRemoveStatusService for ActivityPub * Add tests for ActivityPub workers * Add tests for FollowService * Add tests for FavouriteService, UnfollowService and PostStatusService * Add tests for ReblogService, BlockService, UnblockService, ProcessMentionsService * Add tests for AuthorizeFollowService, RejectFollowService, RemoveStatusService * Add tests for BatchedRemoveStatusService * Deliver updates to a local account to ActivityPub followers * Minor adjustments	8 years ago
Eugen Rochko	a549d1ae6b	Bump to 1.5.0rc1 (#4318)	8 years ago
Eugen Rochko	1fcdaafa6f	Fix webfinger retries (#4275) ... * Do not raise unretryable exceptions in ResolveRemoteAccountService * Removed fatal exceptions from ResolveRemoteAccountService Exceptions that cannot be retried should not be raised. New exception class for those that can be retried (Mastodon::UnexpectedResponseError)	8 years ago
Eugen Rochko	1618b68bfa	HTTP signatures (#4146) ... * Add Request class with HTTP signature generator Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06 * Add HTTP signature verification concern * Add test for SignatureVerification concern * Add basic test for Request class * Make PuSH subscribe/unsubscribe requests use new Request class Accidentally fix lease_seconds not being set and sent properly, and change the new minimum subscription duration to 1 day * Make all PuSH workers use new Request class * Make Salmon sender use new Request class * Make FetchLinkService use new Request class * Make FetchAtomService use the new Request class * Make Remotable use the new Request class * Make ResolveRemoteAccountService use the new Request class * Add more tests * Allow +-30 seconds window for signed request to remain valid * Disable time window validation for signed requests, restore 7 days as PuSH subscription duration (which was previous default due to a bug)	8 years ago
Akihiko Odaki (@fn_aki@pawoo.net)	7d8e3721ae	Overwrite old statuses with reblogs in PrecomputeFeedService (#3984)	8 years ago
Eugen Rochko	e17c2e5da5	Batched remove status service (#3735) ... * Make Pubsubhubbub::DistributionWorker handle both single stream entry arguments, as well as arrays of stream entries * Add BatchedRemoveStatusService, make SuspendAccountService use it * Improve method names * Add test * Add more tests * Use PuSH payloads of 100 to have a clear mapping of 1000 input statuses -> 10 PuSH payloads It was nice while it lasted	8 years ago
Matt Jankowski	ec34ec63b1	Specs for cleanup workers (#3235) ... * Add spec files for feed and media cleanup workers * Add coverage for feed and media cleanup schedulers * Clean up feed and media cleanup workers	8 years ago
Matt Jankowski	d2e0edd721	Add spec coverage for regeneration worker (#3143)	8 years ago
Matt Jankowski	8ec8410651	After remote follow worker specs (#3098) ... * Add specs for the after remote follow workers * Refactor the after remote follow workers	8 years ago
Eugen Rochko	657496b5a9	Do not cancel PuSH subscriptions after encountering "permanent" error… (#3046) ... * Do not cancel PuSH subscriptions after encountering "permanent" error response After talking with MMN about it, turns out some servers/php setups do return 4xx errors while rebooting, so this anti-feature that was meant to take load off of the hub is doing more harm than good in terms of breaking subscriptions * Update delivery_worker.rb	8 years ago
Matt Jankowski	0d70fe2659	Spec coverage and refactor for pubsub/delivery worker (#3021) ... * Framework for delivery worker spec * Refactor of pubsub delivery worker	8 years ago
Matt Jankowski	60ecfb87ae	Coverage for domain block worker (#2997)	8 years ago
Matt Jankowski	08949cca41	Pubsub confirmation worker spec (#2974) ... * Add specs for valid challenge response from pubsub confirmation worker * Refactor the pubsub confirmation worker	8 years ago
Matt Jankowski	cc9a6a710f	Spec feed insert worker (#2965) ... * Spec for feed insert worker when missing records * more specs! * Refactor feed insert worker	8 years ago
Matt Jankowski	d142544159	Spec coverage and refactor of digest mailer worker (#2961)	8 years ago
Eugen Rochko	3a38322a54	Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items (#2954) ... to all subscribers. IDN-normalize callback URLs for subscriptions on insert.	8 years ago
Eugen Rochko	b8e166894b	Fix Scheduler::SubscriptionsScheduler (#2834) ... * Fix Scheduler::SubscriptionsScheduler, add worker test for it * Change production log level of Sidekiq to "warn" instead of "info"	8 years ago