mastodon

Commit Graph

Author	SHA1	Message	Date
Ramūns Usovs	0872f3e3d7	Allow streaming to connect to postgress with self-signed certs (#21431 )	2 years ago
emilweth	7c45145430	fix metrics format (#23520 )	2 years ago
emilweth	c1b823368c	dot is not allowed (#23519 )	2 years ago
Eugen Rochko	c5a4d8c82d	Add information about database pool to metrics endpoint in streaming API (#23469 )	2 years ago
Eugen Rochko	4cb7647516	Add database pool configuration over `DB_POOL` to streaming API (#23470 ) Add connection timeout of 15s	2 years ago
Eugen Rochko	01584f03e8	Add metrics endpoint to streaming API (#23388 )	2 years ago
Nick Schonning	06b68490d1	Enable eslint:recommended ruleset (#22433 ) * Enable ESLint recommended ruleset * Disable failing ESLint recommended rules * Remove rules shadowed by eslint:recommended	2 years ago
Claire	8c3c6dcae9	Fix filters from other users being used in the streaming service (#20719 )	2 years ago
Claire	cd5e98dbdb	Fix public/local timeline posts not being properly filtered (#20567 ) * Fix streaming server using wrong property name for matching filters Late in the PR, the `filter_results` property has been renamed to `filtered`, but the change has not been reflected in the streaming server code. * Fix filter_action attribute being an integer instead of a string	2 years ago
Eugen Rochko	e7aa2be828	Change how hashtags are normalized (#18795 ) * Change how hashtags are normalized * Fix tests	3 years ago
Claire	02851848e9	Revamp post filtering system (#18058 ) * Add model for custom filter keywords * Use CustomFilterKeyword internally Does not change the API * Fix /filters/edit and /filters/new * Add migration tests * Remove whole_word column from custom_filters (covered by custom_filter_keywords) * Redesign /filters Instead of a list, present a card that displays more information and handles multiple keywords per filter. * Redesign /filters/new and /filters/edit to add and remove keywords This adds a new gem dependency: cocoon, as well as a npm dependency: cocoon-js-vanilla. Those are used to easily populate and remove form fields from the user interface when manipulating multiple keyword filters at once. * Add /api/v2/filters to edit filter with multiple keywords Entities: - `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context` `keywords` - `FilterKeyword`: `id`, `keyword`, `whole_word` API endpoits: - `GET /api/v2/filters` to list filters (including keywords) - `POST /api/v2/filters` to create a new filter `keywords_attributes` can also be passed to create keywords in one request - `GET /api/v2/filters/:id` to read a particular filter - `PUT /api/v2/filters/:id` to update a new filter `keywords_attributes` can also be passed to edit, delete or add keywords in one request - `DELETE /api/v2/filters/:id` to delete a particular filter - `GET /api/v2/filters/:id/keywords` to list keywords for a filter - `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a filter - `GET /api/v2/filter_keywords/:id` to read a particular keyword - `PUT /api/v2/filter_keywords/:id` to edit a particular keyword - `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword * Change from `irreversible` boolean to `action` enum * Remove irrelevent `irreversible_must_be_within_context` check * Fix /filters/new and /filters/edit with update for filter_action * Fix Rubocop/Codeclimate complaining about task names * Refactor FeedManager#phrase_filtered? This moves regexp building and filter caching to the `CustomFilter` class. This does not change the functional behavior yet, but this changes how the cache is built, doing per-custom_filter regexps so that filters can be matched independently, while still offering caching. * Perform server-side filtering and output result in REST API * Fix numerous filters_changed events being sent when editing multiple keywords at once * Add some tests * Use the new API in the WebUI - use client-side logic for filters we have fetched rules for. This is so that filter changes can be retroactively applied without reloading the UI. - use server-side logic for filters we haven't fetched rules for yet (e.g. network error, or initial timeline loading) * Minor optimizations and refactoring * Perform server-side filtering on the streaming server * Change the wording of filter action labels * Fix issues pointed out by linter * Change design of “Show anyway” link in accordence to review comments * Drop “irreversible” filtering behavior * Move /api/v2/filter_keywords to /api/v1/filters/keywords * Rename `filter_results` attribute to `filtered` * Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer * Fix systemChannelId value in streaming server * Simplify code by removing client-side filtering code The simplifcation comes at a cost though: filters aren't retroactively applied anymore.	3 years ago
Yamagishi Kazutoshi	5781d1db84	Fix parsing `TRUSTED_PROXY_IP` (#18051 )	3 years ago
Claire	f29458da1d	Fix streaming server sometimes silently dropping subscriptions (#17841 )	3 years ago
Claire	6ea80ba2a2	Change streaming server error messages when failing to parse client input (#17559 ) Fixes #17541 - prefix JSON parsing error message by “Error parsing message from …” - output user id if a user is logged in, IP address otherwise - reduce log level from error to warning when a user is logged in, and to silly otherwise	3 years ago
Claire	bc066d8936	Fix timeline streaming stopping for multiple sessions instead of one (#17259 ) * Fix timeline streaming stopping for multiple sessions instead of one Fixes #17256. In updating the code for a newer version of node-redis, #17183 also broke redis subscription management when multiple streaming clients subscribe to the same channel. This commit restores the redis subscription management code. * Let node-redis actually handle the subscriptions	3 years ago
Lerk	4d1eaf3e6e	Finish update of node-redis (#17183 ) * fix streaming redis client * use console.error instead of console.log * follow node-redis migration guide https://github.com/redis/node-redis/blob/master/docs/v3-to-v4.md * fix config options for node-redis * keep indentation * Update streaming/index.js Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh> Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>	3 years ago
Sasha Sorokin	6c88ebfd4b	fix(streaming): req.scopes can be nullable (#16823 ) When checking for required OAuth scopes, an unexpected error could happen due to missing (null-y) req.scopes. This commit fixes that by checking if req.scopes are present before checking if any required scopes are present, otherwise it skips that straight to rejection.	3 years ago
Eugen Rochko	a0d4129893	Refactor notifications to go through a separate stream in streaming API (#16765 ) Eliminate need to have custom notifications filtering logic in the streaming API code by publishing notifications into a separate stream and then simply using the multi-stream capability to subscribe to that stream when necessary	3 years ago
Eugen Rochko	c5c46dd6ee	Fix "cb is not a function" error in streaming API server (#16134 ) Third argument of `ping` is the callback Regression from #15932	4 years ago
Eugen Rochko	aafe65a142	Change log level of worker start/end to warn in streaming API (#16110 )	4 years ago
Claire	49814d5799	Switch from deprecated ClusterWS/cws to ws package (#15932 ) * Switch from deprecated ClusterWS/cws to ws package Fixes #15184 Co-authored-by: Edho Arief <me@nanaya.pro> * Make bufferutil and utf-8-validate optional dependencies Co-authored-by: Edho Arief <me@nanaya.pro>	4 years ago
abcang	a2da02626e	Fixed ESLint error (#15214 ) * eslint --fix * fix consistent-return * fix promise/catch-or-return * ignore import rule	4 years ago
Eugen Rochko	aa10200e58	Fix streaming API allowing connections to persist after access token invalidation (#15111 ) Fix #14816	4 years ago
fuyu	e39d97f700	Fix crash streaming process when receive invalid json (#14859 )	4 years ago
Eugen Rochko	01647b8acb	Fix destructuring error when unsubscribing without subscribing (#14566 )	4 years ago
Eugen Rochko	ef057584fd	Add support for managing multiple stream subscriptions in a single connection (#14524 )	4 years ago
ThibG	ddcdddd6fc	Change streaming server to treat blank redis password as password-less auth (#14135 ) Fixes #14131 Our `mastodon:setup` task defaults to a blank password rather than the absence of password, but some versions of Redis reject blank password authentication when authentication is possible without a password. The Ruby code only uses the Redis password when it's not blank, so do the same for the node.js part.	5 years ago
Eugen Rochko	5d8398c8b8	Add E2EE API (#13820 )	5 years ago
Takeshi Umeda	26b08a3c54	Add remote only to public timeline (#13504 ) * Add remote only to public timeline * Fix code style	5 years ago
Ben Lubar	0dfba0884e	minor server-sent events fixes (#12945 ) * Send output on the server-sent events stream immediately so the client sees that it was successfully opened even if it doesn't have any messages. Fix transparent SSE streaming for the public:local and hashtag:local stream types. * Tell caches to never store server-sent events.	5 years ago
Eugen Rochko	24552b5160	Add whitelist mode (#11291 )	6 years ago
Eugen Rochko	e7353c47db	Change default interface of web and streaming from 0.0.0.0 to 127.0.0.1 (#11302 )	6 years ago
ThibG	d63c3c0cef	Improve streaming server security (#10818 ) * Check OAuth token scopes in the streaming API * Use Sec-WebSocket-Protocol instead of query string to pass WebSocket token Inspired by https://github.com/kubevirt/kubevirt/issues/1242	6 years ago
abcang	ca6c93a2f5	Migrate from uws to cws (#10805 )	6 years ago
Eugen Rochko	13a7f05030	Fix streaming API always attempting to use SSL with Postgres (#10231 ) Fix #10223	6 years ago
Eugen Rochko	dbeab5a036	Fix SSL configuration regression in streaming API (#10225 ) * Fix SSL configuration regression in streaming API Fix #10223 * Fix code style issues and integrate #10219 * Fix dumb thing	6 years ago
Sascha	42e733681a	config: add DB_SSLMODE for managed/remote PG (#10210 ) * config: add DB_SSLMODE for managed/remote PG * streaming: set PG sslmode, defaults to prefer	6 years ago
Gomasy	8d70d3de38	Fix crash when using UNIX socket (#9036 )	6 years ago
Eugen Rochko	369cc5f555	Check if port/socket is available before forking in Streaming API (#9023 ) Previously, the server would attempt taking port/socket in worker process, and if it was taken, fail, which made the master process create a new worker. This led to really high CPU usage if the streaming API was started when the port or socket were not available. Now, before clustering (forking) into worker processes, a test server is created and then removed to check if it can be done.	6 years ago
Eugen Rochko	18e7ef6eda	Add check for missing tag param in streaming API (#8955 ) * Add check for missing tag param in streaming API Fixes error: ``` TypeError: Cannot read property 'toLowerCase' of undefined at app.get (.../streaming/index.js:493:50) ``` * Fix code style issues	6 years ago
Eugen Rochko	774ac47373	Add conversations API (#8832 ) * Add conversations API * Add web UI for conversations * Add test for conversations API * Add tests for ConversationAccount * Improve web UI * Rename ConversationAccount to AccountConversation * Remove conversations on block and mute * Change last_status_id to be a denormalization of status_ids * Add optimistic locking	6 years ago
Eugen Rochko	f37fafe30b	Add health endpoint to streaming API (#8441 ) GET /api/v1/streaming/health Answers with OK. Fix #8337	6 years ago
MIYAGI Hikaru	a083604d1d	Support UNIX domain socket for streaming service without using PORT (#8217 ) * Support UNIX domain socket for streaming service without using PORT The use of UNIX domain socket for streaming service was not officially supported, but it was made unofficial to use by setting a path to PORT. From now on, SOCKET will be used just like setting for puma. * Hundle relative path	6 years ago
Eugen Rochko	0180037dfb	Fix streaming API still using filtered instead of chosen languages (#8009 )	7 years ago
Yamagishi Kazutoshi	7403e5d306	Add media timeline (#6631 )	7 years ago
Kaito Sinclaire	156b916caf	Direct messages column (#4514 ) * Added a timeline for Direct statuses * Lists all Direct statuses you've sent and received * Displayed in Getting Started * Streaming server support for direct TL * Changes to match other timelines in 2.0	7 years ago
abcang	609bf93029	Perform processing that does not use the database before connecting to the database (#7168 )	7 years ago
nullkal	ccf4f170de	Make sure call `done();` in the listener of public timeline for anonymous connection (#6009 )	7 years ago
nullkal	90e7da16a0	Fix the condition in streaming listener (#6008 )	7 years ago
erin	c986218c3a	Improve error handling in streaming/index.js (#5968 ) On an unhandled worker exception, we should log the exception and exit with nonzero status, instead of letting workers silently fail and restarting them in an endless loop. Note: we previously tried to handle the `'error'` signal. That's not a signal Node fires; my patch traps `'uncaughtException'`, which is what the code was _trying_ to do.	7 years ago
nullkal	cfea28216f	make it possible to stream public timelines without authorization (#5977 ) * make it possible to stream public timelines without authorization * Fix * Make eslint allow `value == null` * Remove redundant line * Improve style and revert .eslintrc.yml * Fix streamWsEnd * Show IP address instead of (anonymous user) * Add missing semicolon	7 years ago
Baptiste Gelez	2864e5e077	Make it possible to bind streming service to 0.0.0.0 (#5744 )	7 years ago
Eugen Rochko	24cafd73a2	Lists (#5703 ) * Add structure for lists * Add list timeline streaming API * Add list APIs, bind list-account relation to follow relation * Add API for adding/removing accounts from lists * Add pagination to lists API * Add pagination to list accounts API * Adjust scopes for new APIs - Creating and modifying lists merely requires "write" scope - Fetching information about lists merely requires "read" scope * Add test for wrong user context on list timeline * Clean up tests	7 years ago
masarakki	0e0c6b1b4b	use-DB_NAME-in-development (#5430 )	7 years ago
Eugen Rochko	909a6d4661	Fix streaming API double-JSON-encoding status IDs for deletes (#5069 )	7 years ago
voidSatisfaction	b6f3869f8d	Fix streaming url to lowercase (#4804 )	7 years ago
Nolan Lawson	3f82d8b979	Gracefully stop streaming server (#4103 )	8 years ago
Eugen Rochko	ebd2dde688	Restore streaming API output format (#4100 ) * Restore streaming API output format Regression from #4090 * Remove whitespace	8 years ago
Yamagishi Kazutoshi	285038972b	Stop using Babel with streaming server (#3950 )	8 years ago
Takuya Yoshida	e5563843a2	Re-fix errorMiddleware (#3922 )	8 years ago
unarist	c972e1ee1f	Ignore DB_NAME for development env on streaming as well as rails side (#3948 )	8 years ago
amazedkoumei	87efa38721	more free pgconfig by .env (#3909 ) * more free pgconfig for streaming by .env * fix wrong default values * database.yml read ENV as same as streaming server	8 years ago
Takuya Yoshida	0c44316b22	Fix errorMiddleware to prevent "TypeError: res.writeHead is not a function" (#3913 ) * Fix errorMiddleware * Add "eslint-disable-line no-unused-vars"	8 years ago
Yamagishi Kazutoshi	eff9416469	Remove unused variables (#3906 )	8 years ago
猫吸血鬼ディフリス / 猫ロキP	d8ec832806	Fix streaming server. Redis connection subscribe for each channel. (#3828 )	8 years ago
Eugen Rochko	0464602978	Fix issue with some Node.js versions not having Array.prototype.includes (#3667 ) by using Array.prototype.indexOf instead	8 years ago
Clworld	dab8fc4584	Execute PushUpdateWorker only for accounts who uses StreamingAPI just now. (#3278 ) * Add redis key "subscribed:timeline:#{account.id}" to indicate active streaming API listeners exists. * Add endpoint for notification only stream. * Run PushUpdateWorker only for users uses Streaming API now. * Move close hander streamTo(Http/Ws) -> stream(Http/Ws)End (Deal with #3370) * Add stream type for stream start log message.	8 years ago
unarist	9a81be0d37	[RFC] Return 401 for an authentication error on WebSockets (#3411 ) * Return 401 for an authentication error on WebSocket * Use upgradeReq instead of a custom object	8 years ago
unarist	91c789ec63	Don't stream toots from users who have blocked the recipient user (#3300 ) * Don't stream toots from users who have blocked the recipient user This filter was already applied on the `/api/v1/timelines/public` API, but not yet for the Streaming API. * Boosted status' account_id doesn't need for filtering in streaming/index.js This filtering is only for public/hashtag timelines, but boosts already filtered on FanOutOnWriteService because those timelines don't show boosts.	8 years ago
Clworld	45837c533e	Re-add stream end log for WebSocket (#3397 )	8 years ago
Eugen Rochko	425d02287a	Improve streaming API cluster logging (#3370 ) * Improve streaming API cluster logging * Less verbose error middleware logging (stack trace useless there) * Fix error logging * Prevent potential issue * Add missing "done()" in catch of Promise.all, websocket heartbeat re-implemented like in example * I actually forgot a done(), the absolute madman	8 years ago
Clworld	2e429c0c25	Reject revoked access_token on Streaming API. (#3367 )	8 years ago
Eugen Rochko	0734e1fe33	Language filtering in streaming API (#3339 )	8 years ago
Matt Jankowski	8e84177305	Use local default for postgres host in node streaming service (#3240 ) This location varies across postgres installations, and it seems like the pg package knows how to guess correctly on each system.	8 years ago
happycoloredbanana	7fba4cb3d1	Allow access token in URI (#3208 )	8 years ago
Eugen Rochko	20c0054460	Adjust REDIS_URL usage in node_redis (#3183 ) Resolves #2780	8 years ago
Yamagishi Kazutoshi	2e112e2406	Improve eslint rules (#3147 ) * Add semi to ESLint rules * Add padded-blocks to ESLint rules * Add comma-dangle to ESLint rules * add config/webpack and storyboard * add streaming/ * yarn test:lint -- --fix	8 years ago
Eugen Rochko	620d0d8029	Account domain blocks (#2381 ) * Add <ostatus:conversation /> tag to Atom input/output Only uses ref attribute (not href) because href would be the alternate link that's always included also. Creates new conversation for every non-reply status. Carries over conversation for every reply. Keeps remote URIs verbatim, generates local URIs on the fly like the rest of them. * Conversation muting - prevents notifications that reference a conversation (including replies, favourites, reblogs) from being created. API endpoints /api/v1/statuses/:id/mute and /api/v1/statuses/:id/unmute Currently no way to tell when a status/conversation is muted, so the web UI only has a "disable notifications" button, doesn't work as a toggle * Display "Dismiss notifications" on all statuses in notifications column, not just own * Add "muted" as a boolean attribute on statuses JSON For now always false on contained reblogs, since it's only relevant for statuses returned from the notifications endpoint, which are not nested Remove "Disable notifications" from detailed status view, since it's only relevant in the notifications column * Up max class length * Remove pending test for conversation mute * Add tests, clean up * Rename to "mute conversation" and "unmute conversation" * Raise validation error when trying to mute/unmute status without conversation * Adding account domain blocks that filter notifications and public timelines * Add tests for domain blocks in notifications, public timelines Filter reblogs of blocked domains from home * Add API for listing and creating account domain blocks * API for creating/deleting domain blocks, tests for Status#ancestors and Status#descendants, filter domain blocks from them * Filter domains in streaming API * Update account_domain_block_spec.rb	8 years ago
Hayato IZUMI	5b3c7572ca	Use REDIS_DB in streaming (#3094 )	8 years ago
Yamagishi Kazutoshi	72c8562cc9	Use babel-preset-env on Streaming Server (#2998 ) * Use babel-preset-env on Streaming Server Change the streaming server to use babel-preset-env as well as asset compilation. This shortens the load time at first boot. * remove babel-plugin-lodash	8 years ago
Ayumu AIZAWA	b913746752	[bug] wrong config setting for dababase. (#2984 )	8 years ago
beatrix	5c1f70b5c5	namespace redis usage (#2869 ) * add redis-namespace gem * namespace redis usage * refactor redis namespace code to be less intrusive previously : would be prepended to keys when the REDIS_NAMESPACE env var was not set now if it is not set the namespacing functions are not used at all, which should prevent disruptions when instances update. * fix redis namespace variable style in streaming js * remove trailing space * final redis namespace style fix	8 years ago
Eshin Kunishima	67b7d3d3b6	Replace ws with uws (#2807 ) * Replace ws with uws * fix indent * Adjust indentation * remove trailing space	8 years ago
abcang	629a4d0fca	fix DB_URL (#2778 )	8 years ago
Eugen Rochko	c997091166	Clean up redis configuration. Allow using REDIS_URL to set advanced (#2732 ) connection options instead of setting REDIS_HOST etc individually Close #1986	8 years ago
Akihiko Odaki	a6788662b0	Revert "Use PostgreSQL inheritance for blocks and mutes (#2520 )" (#2634 ) This reverts commit `5135d609b7`.	8 years ago
Akihiko Odaki	5135d609b7	Use PostgreSQL inheritance for blocks and mutes (#2520 )	8 years ago
YOU	d0665726ca	Update index.js (#2612 )	8 years ago
masarakki	d0ec4fb828	refactor-cores (#2308 )	8 years ago
Akihiko Odaki	16cd648181	Allow to listen Unix socket (#2085 ) * Allow puma to listen Unix socket * streaming: Show the whole listening address instead of the port Port is not always appropriate (e.g. Unix socket) * streaming: Close server before exiting This change especially allows to remove Unix socket before exiting.	8 years ago
178inaba	dd517b9a55	Fix a bug where the hashtag could not be specified with the streaming API (#2258 )	8 years ago
kuro5hin	0790aa9165	Fix streaming server worker count (#2082 ) * Make sure we start at least one streaming worker, even if there's only one CPU and we're in production mode. * Oops, let's make sure we're counting cpus right too.	8 years ago
geta6	64e1d51025	Improve streaming server with cluster (#1970 )	8 years ago
Gavin Mogan	3672a799d4	Dev Tooling fixes (eslint/editorconfig) (#1398 ) * Add eslint to dev dependancies so it gets installed for the repo yarn add --dev eslint babel-eslint eslint-plugin-reac project specific version of eslint, you can globally install eslint-cli if you want the global runtime, or add .bin to your path * fix eslint errors about inconsitent returns * eslint ignore the same as git ignore. allows for eslint . * Add editorconfig file so everyones editor will be setup to follow the same standards	8 years ago
Eugen Rochko	3618cc04ff	Add heartbeat to websockets streaming API connections	8 years ago
Eugen Rochko	c64a1c25c4	Fix #231 - Muting	8 years ago
Eugen Rochko	02e91a96dd	Make streaming API use one pattern-matching redis pubsub connection Refresh timelines when streaming API reconnects in the UI	8 years ago
Eugen Rochko	c8252759df	Add streaming API channels for local-only statuses	8 years ago
Eugen Rochko	4d2be9f432	Add unique request IDs to streaming API to improve logs	8 years ago
Eugen Rochko	fb6aa7ad5c	Add tracking of delay to streaming API	8 years ago

1 2 3 4

159 Commits (9bf2e2eda0ffea6382e161f7ebb43d73aaf658ca)