185 Commits (dbb1ee269fa4a6ee097dfea5f77bb2c9428af93b)

Author	SHA1	Message	Date
Eugen Rochko	4a618908e8	Account deletion (#3728) ... * Add form for account deletion * If avatar or header are gone from source, remove them * Add option to have SuspendAccountService remove user record, add tests * Exclude suspended accounts from search	8 years ago
takayamaki	3eedad2737	change sidekiq queueing to bulk push (#3536)	8 years ago
Matt Jankowski	ec34ec63b1	Specs for cleanup workers (#3235) ... * Add spec files for feed and media cleanup workers * Add coverage for feed and media cleanup schedulers * Clean up feed and media cleanup workers	8 years ago
Eugen Rochko	ef900789bc	Replace mastodon:media:clear and mastodon:feeds:clear rake tasks with (#3180) ... sidekiq-scheduler jobs Resolves #2495	8 years ago
Matt Jankowski	d2e0edd721	Add spec coverage for regeneration worker (#3143)	8 years ago
Matt Jankowski	db4119f971	Specs for precompute feed service (#3142) ... * Add spec for precompute feed service * Refactor PrecomputeFeedService * spec wip	8 years ago
Matt Jankowski	8ec8410651	After remote follow worker specs (#3098) ... * Add specs for the after remote follow workers * Refactor the after remote follow workers	8 years ago
Eugen Rochko	657496b5a9	Do not cancel PuSH subscriptions after encountering "permanent" error… (#3046) ... * Do not cancel PuSH subscriptions after encountering "permanent" error response After talking with MMN about it, turns out some servers/php setups do return 4xx errors while rebooting, so this anti-feature that was meant to take load off of the hub is doing more harm than good in terms of breaking subscriptions * Update delivery_worker.rb	8 years ago
Matt Jankowski	0d70fe2659	Spec coverage and refactor for pubsub/delivery worker (#3021) ... * Framework for delivery worker spec * Refactor of pubsub delivery worker	8 years ago
Matt Jankowski	08949cca41	Pubsub confirmation worker spec (#2974) ... * Add specs for valid challenge response from pubsub confirmation worker * Refactor the pubsub confirmation worker	8 years ago
Matt Jankowski	cc9a6a710f	Spec feed insert worker (#2965) ... * Spec for feed insert worker when missing records * more specs! * Refactor feed insert worker	8 years ago
Matt Jankowski	d142544159	Spec coverage and refactor of digest mailer worker (#2961)	8 years ago
Eugen Rochko	3a38322a54	Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items (#2954) ... to all subscribers. IDN-normalize callback URLs for subscriptions on insert.	8 years ago
haoyayoi	2fba94b36e	add content-type in pubsubhubhub request header (#2943) ... * add content-type in pubsubhubhub request header * fix type	8 years ago
Eugen Rochko	b8e166894b	Fix Scheduler::SubscriptionsScheduler (#2834) ... * Fix Scheduler::SubscriptionsScheduler, add worker test for it * Change production log level of Sidekiq to "warn" instead of "info"	8 years ago
Eugen Rochko	81584779cb	More robust PuSH subscription refreshes (#2799) ... * Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron Fix an issue where / in domain would raise exception in TagManager#normalize_domain PuSH subscriptions refresh done in a round-robin way to avoid hammering a single server's hub in sequence. Correct handling of failures/retries through Sidekiq (see also #2613). Optimize Account#with_followers scope. Also, since subscriptions are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire refreshing operation halfway through Fix #2702 - Correct user agent header on outgoing http requests * Add test for SubscribeService * Extract #expiring_accounts into method * Make mastodon:push:refresh no-op * Queues are now defined in sidekiq.yml * Queues are now in sidekiq.yml	8 years ago
Eugen	17c591ffba	Punycode URI normalization (#2370) ... * Fix #2119 - Whenever about to send a HTTP request, normalize the URI * Add test for IDN request in FetchLinkCardService * Perform IDN normalization on domains before they are stored in the DB	8 years ago
Eugen	501514960a	Followers-only post federation (#2111) ... * Make private toots get PuSHed to subscription URLs that belong to domains where you have approved followers * Authorized followers controller, stub for bulk action * Soft block in the background * Add simple test for new controller * Rename Settings::FollowersController to Settings::FollowerDomainsController, paginate results, rename "private" post setting to "followers-only", fix pagination style, improve post privacy preferences style, improve warning style * Extract compose form warnings into own container, show warning when posting to followers-only with unlocked account	8 years ago
Matt Jankowski	c44a700252	Quick best practice cleanup of views/helpers (#1546) ... * Remove trailing whitespace * Use query methods instead of explicit .blank? checks	8 years ago
Matt Jankowski	7f0a865b05	Allow import/export of mutes list (#1541) ... * Allow export of mutes list * Allow importing of mutes list * Refactor to use Settings::Exports::BaseController and DRY up exports code	8 years ago
Matt Jankowski	89e8e110c8	Imports controller errors (#1553) ... * Add spec for settings/imports controller * Add failing spec for settings/imports#create * Fix broken imports * Refactor ImportWorker	8 years ago
Jantso Porali	e30bbb1cb0	Fix for Import issue from URL (#1416)	8 years ago
Eugen	982fef811e	Fix #1141, fix #1126 - Avatar/profile info fetching (#1215) ... * Fix #1141, fix #1126 - Work through UpdateRemoteProfileService for both <feed> and <entry> top-level tags * Improve code quality, remove line unrelated to fix	8 years ago
Eugen	6d6a429af8	Rewrite Atom generation from stream entries to use Ox instead of Nokogiri (#1124) ... * Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder StreamEntry is now limited to only statuses, which allows some optimization. Removed extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer and PubSubHubbub::DistributionWorker PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker instead. All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri * All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account	8 years ago
blackle	540d6efe88	Catch more errors in process_follows so it doesn't fail	8 years ago
Eugen Rochko	5442083b3c	Split SalmonWorker into smaller parts, move profile updating into another job	8 years ago
Eugen Rochko	5b95be1c42	Replace calls to FeedManager#inline_render and #broadcast	8 years ago
Eugen	c9ebd5d19f	Fix wrong variable used in publish channel	8 years ago
Eugen	1b8c244dff	Add proper message to PushUpdateWorker, use redis directly	8 years ago
Eugen	3d8b80e1cc	Merge branch 'master' into broadcast-to-worker	8 years ago
Eugen Rochko	220bc48e8e	Only render public payload once in FanOutOnWrite	8 years ago
Kurtis Rainbolt-Greene	22dcadedb4	We're going to want these nice helper methods, lets share them with a parent class that matches Rails 5 practices (application level abstraction)	8 years ago
Kurtis Rainbolt-Greene	9638894233	Moving in the inline render	8 years ago
Kurtis Rainbolt-Greene	220051b8b2	I don't actually think we need that.	8 years ago
Kurtis Rainbolt-Greene	0069c01285	Moving the queue_at into the worker	8 years ago
Kurtis Rainbolt-Greene	96ef933820	Replacing the broadcast method with the one defined in the feed manager	8 years ago
Kurtis Rainbolt-Greene	1e96ce378e	By pushing this into a worker we can reduce the amount of time the feed manager using workers eat up a connection	8 years ago
Eugen Rochko	bda37489ac	Remove PuSH subscriptions when delivery is answered with a 4xx error	8 years ago
Eugen Rochko	6fd865c000	Spawn FeedInsertWorker to deliver status into personal feed	8 years ago
Eugen Rochko	82aaedec46	Reduce number of items in feeds, optimize regeneration worker slightly, ... make regeneration worker unique, (only schedule/execute once at a time)	8 years ago
Eugen Rochko	b21f7c28f6	Move OStatus processing back into default queue	8 years ago
Eugen Rochko	b510a56c0c	Only call regeneration worker after first login after a 14 day break	8 years ago
Eugen Rochko	f722bd2387	Separate background jobs into different queues. ATTENTION: new queue "pull" ... must be added to the Sidekiq invokation in your systemd file The pull queue will handle link crawling, thread resolving, and OStatus processing. Such tasks are more likely to hang for a longer time (due to network requests) so it is more sensible to not make the "in-house" tasks wait for them.	8 years ago
Eugen Rochko	b7c1b12367	Make default admin UI page reports. Add admin UI for creating a domain block	8 years ago
Eugen Rochko	e8875c6046	Import feature for following/blocking lists (addresses #62, #177, #201, #454)	8 years ago
Eugen Rochko	6b81d10030	Add digest e-mails	8 years ago
Eugen Rochko	40a4053732	Disable PuSH for blocked domains	8 years ago
Eugen Rochko	720ff55262	Adding more unit tests. Fixing Salmon slaps XML	8 years ago
Eugen Rochko	db814543c0	Take out private toot distribution out of PuSH again	8 years ago
Eugen Rochko	446aad4ce2	Make PuSH only distribute to subscriptions that match follower domains ... Allow PuSH to distribute private toots with that condition	8 years ago
Eugen Rochko	00b5731ecb	After FollowService, re-fetch remote account asynchronously, do nothing ... if account lock info was up to date, otherwise re-do the FollowService with now updated information	8 years ago
Eugen Rochko	149887a0ff	Make follow requests federate	8 years ago
Eugen Rochko	19b9e1e2c3	Preheat status cache	8 years ago
Eugen Rochko	3f075c7794	API for apps to register for push notifications	8 years ago
Eugen Rochko	8a880a3d46	Make blocks create entries and unfollows instantly, but do the clean up ... in the background instead. Should fix delay where blocked person can interact with blocker for a short time before background job gets processed	8 years ago
Eugen Rochko	cca82bf0a2	Move merging/unmerging of timelines into background. Move blocking into ... background as well since it's a computationally expensive	8 years ago
Eugen Rochko	f0de621e76	Fix #463 - Fetch and display previews of URLs using OpenGraph tags	8 years ago
Eugen Rochko	6c28886317	Improve background jobs params and error handling	8 years ago
Eugen Rochko	86be6d48c9	Fix accidental distribution of hidden stream entries to PuSH subscribers	8 years ago
Eugen Rochko	49834a6e7f	Add API for retrieving blocked accounts	8 years ago
Eugen Rochko	4c6809f6ab	Adding some rescues to workers, e.g. don't fail removal job when status can't be found in the first place (already removed)	8 years ago
Eugen Rochko	0542773bca	Make unfavouriting async to prevent timeout errors from leaving orphaned records behind	8 years ago
Eugen Rochko	6de079a5af	Removing external hub completely, fix #333 fixing digit-only hashtags, ... removing web app capability from non-webapp pages	8 years ago
Eugen Rochko	668013265c	Restoring old async behaviour of thread resolving as it proved to be more robust	8 years ago
Eugen Rochko	f90133d2ad	Thread resolving no longer needs to be separate from ProcessFeedService, ... since that is only ever called in the background	8 years ago
Eugen Rochko	f406e01fcf	Add filters for suspended accounts	8 years ago
Eugen Rochko	58b3f4fd67	Fix #329 - avatar errors no longer prevent remote accounts from being saved ... (without avatar). Also improved search position of exact matches	8 years ago
Eugen Rochko	4a2347da41	Track successful PuSH deliveries	8 years ago
Eugen Rochko	93a90cd9c3	Delete statuses asynchronously but provide instant feedback in the API	8 years ago
Eugen Rochko	cc70f28f19	Adding rack timeout of 30sec, PuSH jobs moved to push queue so they ... can be processed separately	8 years ago
Eugen Rochko	4e351baf88	Fix URLs in inline-rendered XML	8 years ago
Eugen Rochko	3ba6531611	Fix setting of confirmed=true on successful confirmation	8 years ago
Eugen Rochko	50638174c8	More logging to ConfirmationWorker	8 years ago
Eugen Rochko	4eba76711b	Adding backtracing to Salmon/Processing workers	8 years ago
Eugen Rochko	2d2c81765b	Adding embedded PuSH server	8 years ago
Eugen Rochko	45c7ee39b3	Remove unneeded indices, improve error handling in background workers, don't needlessly reload reblogged status, send Devise e-mails asynchronously	8 years ago
Eugen Rochko	7dea6a23f7	Move Salmon processing to background as well as PuSH	8 years ago
Eugen Rochko	fdc17bea58	Fix rubocop issues, introduce usage of frozen literal to improve performance	8 years ago
Eugen Rochko	a91c3ef6ce	Delegate processing of incoming PuSH data to background workers	8 years ago
Eugen Rochko	bb4d1eb2e8	Improve feed regeneration	8 years ago
Eugen Rochko	3319473b2c	Move PubSubHubbub pinging to a background worker ... It can take as much as 0.5s if not longer to complete	8 years ago
Eugen Rochko	927333f4f8	Improve code style	8 years ago
Eugen Rochko	4bec613897	Fix #24 - Thread resolving for remote statuses ... This is a big one, so let me enumerate: Accounts as well as stream entry pages now contain Link headers that reference the Atom feed and Webfinger URL for the former and Atom entry for the latter. So you only need to HEAD those resources to get that information, no need to download and parse HTML <link>s. ProcessFeedService will now queue ThreadResolveWorker for each remote status that it cannot find otherwise. Furthermore, entries are now processed in reverse order (from bottom to top) in case a newer entry references a chronologically previous one. ThreadResolveWorker uses FetchRemoteStatusService to obtain a status and attach the child status it was queued for to it. FetchRemoteStatusService looks up the URL, first with a HEAD, tests if it's an Atom feed, in which case it processes it directly. Next for Link headers to the Atom feed, in which case that is fetched and processed. Lastly if it's HTML, it is checked for <link>s to the Atom feed, and if such is found, that is fetched and processed. The account for the status is derived from author/name attribute in the XML and the hostname in the URL (domain). FollowRemoteAccountService and ProcessFeedService are used. This means that potentially threads are resolved recursively until a dead-end is encountered, however it is performed asynchronously over background jobs, so it should be ok.	8 years ago
Eugen Rochko	85b00d19b8	Moving Salmon notifications to background processing, fixing mini-profiler ... behaviour with Turbolinks enabled, optimizing Rabl for production	9 years ago
Eugen Rochko	8eeec389c1	Add link to github project to footer, move FanOutOnWriteService calls to ... DistributionWorker. That isn't the heaviest service, yet, but gotta start somewhere	9 years ago