Twitter Recommendation Algorithm

Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.

home-mixer/README.md

@@ -0,0 +1,102 @@
+Home Mixer
+==========
+
+Home Mixer is the main service used to construct and serve Twitter's Home Timelines. It currently
+powers:
+- For you - best Tweets from people you follow + recommended out-of-network content
+- Following - reverse chronological Tweets from people you follow
+- Lists - reverse chronological Tweets from List members
+
+Home Mixer is built on Product Mixer, our custom Scala framework that facilitates building
+feeds of content.
+
+## Overview
+
+The For You recommendation algorithm in Home Mixer involves the following stages:
+
+- Candidate Generation - fetch Tweets from various Candidate Sources. For example:
+    - Earlybird Search Index
+    - User Tweet Entity Graph
+    - Cr Mixer
+    - Follow Recommendations Service
+- Feature Hydration
+    - Fetch the ~6000 features needed for ranking
+- Scoring and Ranking using ML model
+- Filters and Heuristics. For example:
+    - Author Diversity
+    - Content Balance (In network vs Out of Network)
+    - Feedback fatigue
+    - Deduplication / previously seen Tweets removal
+    - Visibility Filtering (blocked, muted authors/tweets, NSFW settings)
+- Mixing - integrate Tweets with non-Tweet content
+    - Ads
+    - Who-to-follow modules
+    - Prompts
+- Product Features and Serving
+    - Conversation Modules for replies
+    - Social Context
+    - Timeline Navigation
+    - Edited Tweets
+    - Feedback options
+    - Pagination and cursoring
+    - Observability and logging
+    - Client instructions and content marshalling
+
+## Pipeline Structure
+
+### General
+
+Product Mixer services like Home Mixer are structured around Pipelines that split the execution
+into transparent and structured steps.
+
+Requests first go to Product Pipelines, which are used to select which Mixer Pipeline or
+Recommendation Pipeline to run for a given request. Each Mixer or Recommendation
+Pipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.
+
+Mixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together
+(e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines)
+and rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned.
+These pipelines also marshall candidates into a domain object and then into a transport object
+to return to the caller.
+
+Candidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic
+operations on the Candidates, such as filtering out unwanted candidates, applying decorations,
+and hydrating features.
+
+The sections below describe the high level pipeline structure (non-exhaustive) for the main Home
+Timeline tabs powered by Home Mixer.
+
+### For You
+
+- ForYouProductPipelineConfig
+    - ForYouScoredTweetsMixerPipelineConfig (main orchestration layer - mixes Tweets with ads and users)
+        - ForYouScoredTweetsCandidatePipelineConfig (fetch Tweets)
+            - ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)
+                - Fetch Tweet Candidates
+                    - ScoredTweetsInNetworkCandidatePipelineConfig
+                    - ScoredTweetsCrMixerCandidatePipelineConfig
+                    - ScoredTweetsUtegCandidatePipelineConfig
+                    - ScoredTweetsFrsCandidatePipelineConfig
+                - Feature Hydration and Scoring
+                    - ScoredTweetsScoringPipelineConfig
+        - ForYouConversationServiceCandidatePipelineConfig (backup reverse chron pipeline in case Scored Tweets fails)
+        - ForYouAdsCandidatePipelineConfig (fetch ads)
+        - ForYouWhoToFollowCandidatePipelineConfig (fetch users to recommend)
+
+### Following
+
+- FollowingProductPipelineConfig
+    - FollowingMixerPipelineConfig
+        - FollowingEarlybirdCandidatePipelineConfig (fetch tweets from Search Index)
+        - ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
+        - FollowingAdsCandidatePipelineConfig (fetch ads)
+        - FollowingWhoToFollowCandidatePipelineConfig (fetch users to recommend)
+
+### Lists
+
+- ListTweetsProductPipelineConfig
+    - ListTweetsMixerPipelineConfig
+        - ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)
+        - ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
+        - ListTweetsAdsCandidatePipelineConfig (fetch ads)
+