Twitter Recommendation Algorithm

Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
This commit is contained in:
twitter-team
2023-03-31 17:36:31 -05:00
commit ef4c5eb65e
5364 changed files with 460239 additions and 0 deletions

30
home-mixer/BUILD.bazel Normal file
View File

@ -0,0 +1,30 @@
jvm_binary(
name = "bin",
basename = "home-mixer",
main = "com.twitter.home_mixer.HomeMixerServerMain",
runtime_platform = "java11",
tags = ["bazel-compatible"],
dependencies = [
"3rdparty/jvm/ch/qos/logback:logback-classic",
"finagle/finagle-zipkin-scribe/src/main/scala",
"finatra/inject/inject-logback/src/main/scala",
"home-mixer/server/src/main/scala/com/twitter/home_mixer",
"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback",
"twitter-server-internal/src/main/scala",
"twitter-server/logback-classic/src/main/scala",
],
)
# Aurora Workflows build phase convention requires a jvm_app named with home-mixer-app
jvm_app(
name = "home-mixer-app",
archive = "zip",
binary = ":bin",
bundles = [
bundle(
fileset = ["config/**/*"],
owning_target = "home-mixer/config:files",
),
],
tags = ["bazel-compatible"],
)

102
home-mixer/README.md Normal file
View File

@ -0,0 +1,102 @@
Home Mixer
==========
Home Mixer is the main service used to construct and serve Twitter's Home Timelines. It currently
powers:
- For you - best Tweets from people you follow + recommended out-of-network content
- Following - reverse chronological Tweets from people you follow
- Lists - reverse chronological Tweets from List members
Home Mixer is built on Product Mixer, our custom Scala framework that facilitates building
feeds of content.
## Overview
The For You recommendation algorithm in Home Mixer involves the following stages:
- Candidate Generation - fetch Tweets from various Candidate Sources. For example:
- Earlybird Search Index
- User Tweet Entity Graph
- Cr Mixer
- Follow Recommendations Service
- Feature Hydration
- Fetch the ~6000 features needed for ranking
- Scoring and Ranking using ML model
- Filters and Heuristics. For example:
- Author Diversity
- Content Balance (In network vs Out of Network)
- Feedback fatigue
- Deduplication / previously seen Tweets removal
- Visibility Filtering (blocked, muted authors/tweets, NSFW settings)
- Mixing - integrate Tweets with non-Tweet content
- Ads
- Who-to-follow modules
- Prompts
- Product Features and Serving
- Conversation Modules for replies
- Social Context
- Timeline Navigation
- Edited Tweets
- Feedback options
- Pagination and cursoring
- Observability and logging
- Client instructions and content marshalling
## Pipeline Structure
### General
Product Mixer services like Home Mixer are structured around Pipelines that split the execution
into transparent and structured steps.
Requests first go to Product Pipelines, which are used to select which Mixer Pipeline or
Recommendation Pipeline to run for a given request. Each Mixer or Recommendation
Pipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.
Mixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together
(e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines)
and rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned.
These pipelines also marshall candidates into a domain object and then into a transport object
to return to the caller.
Candidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic
operations on the Candidates, such as filtering out unwanted candidates, applying decorations,
and hydrating features.
The sections below describe the high level pipeline structure (non-exhaustive) for the main Home
Timeline tabs powered by Home Mixer.
### For You
- ForYouProductPipelineConfig
- ForYouScoredTweetsMixerPipelineConfig (main orchestration layer - mixes Tweets with ads and users)
- ForYouScoredTweetsCandidatePipelineConfig (fetch Tweets)
- ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)
- Fetch Tweet Candidates
- ScoredTweetsInNetworkCandidatePipelineConfig
- ScoredTweetsCrMixerCandidatePipelineConfig
- ScoredTweetsUtegCandidatePipelineConfig
- ScoredTweetsFrsCandidatePipelineConfig
- Feature Hydration and Scoring
- ScoredTweetsScoringPipelineConfig
- ForYouConversationServiceCandidatePipelineConfig (backup reverse chron pipeline in case Scored Tweets fails)
- ForYouAdsCandidatePipelineConfig (fetch ads)
- ForYouWhoToFollowCandidatePipelineConfig (fetch users to recommend)
### Following
- FollowingProductPipelineConfig
- FollowingMixerPipelineConfig
- FollowingEarlybirdCandidatePipelineConfig (fetch tweets from Search Index)
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
- FollowingAdsCandidatePipelineConfig (fetch ads)
- FollowingWhoToFollowCandidatePipelineConfig (fetch users to recommend)
### Lists
- ListTweetsProductPipelineConfig
- ListTweetsMixerPipelineConfig
- ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
- ListTweetsAdsCandidatePipelineConfig (fetch ads)

View File

@ -0,0 +1,46 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"3rdparty/jvm/com/google/inject:guice",
"3rdparty/jvm/javax/inject:javax.inject",
"3rdparty/jvm/net/codingwell:scala-guice",
"3rdparty/jvm/org/slf4j:slf4j-api",
"finagle/finagle-core/src/main",
"finagle/finagle-http/src/main/scala",
"finagle/finagle-thriftmux/src/main/scala",
"finatra-internal/mtls-http/src/main/scala",
"finatra-internal/mtls-thriftmux/src/main/scala",
"finatra/http-core/src/main/java/com/twitter/finatra/http",
"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations",
"finatra/inject/inject-app/src/main/scala",
"finatra/inject/inject-core/src/main/scala",
"finatra/inject/inject-server/src/main/scala",
"finatra/inject/inject-utils/src/main/scala",
"home-mixer/server/src/main/resources",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/controller",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/module",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product",
"home-mixer/thrift/src/main/thrift:thrift-scala",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter",
"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala",
"src/thrift/com/twitter/timelines/render:thrift-scala",
"stringcenter/client",
"stringcenter/client/src/main/java",
"stringcenter/client/src/main/scala/com/twitter/stringcenter/client",
"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms/view",
"timelines/src/main/scala/com/twitter/timelines/config",
"timelines/src/main/scala/com/twitter/timelines/features/app",
"twitter-server-internal",
"twitter-server/server/src/main/scala",
"util/util-app/src/main/scala",
"util/util-core:scala",
"util/util-slf4j-api/src/main/scala",
],
)

View File

@ -0,0 +1,18 @@
package com.twitter.home_mixer
import com.twitter.finatra.http.routing.HttpWarmup
import com.twitter.finatra.httpclient.RequestBuilder._
import com.twitter.inject.Logging
import com.twitter.inject.utils.Handler
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class HomeMixerHttpServerWarmupHandler @Inject() (warmup: HttpWarmup) extends Handler with Logging {
override def handle(): Unit = {
Try(warmup.send(get("/admin/product-mixer/product-pipelines"), admin = true)())
.onFailure(e => error(e.getMessage, e))
}
}

View File

@ -0,0 +1,118 @@
package com.twitter.home_mixer
import com.google.inject.Module
import com.twitter.finagle.Filter
import com.twitter.finatra.annotations.DarkTrafficFilterType
import com.twitter.finatra.http.HttpServer
import com.twitter.finatra.http.routing.HttpRouter
import com.twitter.finatra.mtls.http.{Mtls => HttpMtls}
import com.twitter.finatra.mtls.thriftmux.Mtls
import com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule
import com.twitter.finatra.thrift.ThriftServer
import com.twitter.finatra.thrift.filters._
import com.twitter.finatra.thrift.routing.ThriftRouter
import com.twitter.home_mixer.controller.HomeThriftController
import com.twitter.home_mixer.module._
import com.twitter.home_mixer.param.GlobalParamConfigModule
import com.twitter.home_mixer.product.HomeMixerProductModule
import com.twitter.home_mixer.{thriftscala => st}
import com.twitter.product_mixer.component_library.module.AccountRecommendationsMixerModule
import com.twitter.product_mixer.component_library.module.CrMixerClientModule
import com.twitter.product_mixer.component_library.module.DarkTrafficFilterModule
import com.twitter.product_mixer.component_library.module.EarlybirdModule
import com.twitter.product_mixer.component_library.module.ExploreRankerClientModule
import com.twitter.product_mixer.component_library.module.GizmoduckClientModule
import com.twitter.product_mixer.component_library.module.OnboardingTaskServiceModule
import com.twitter.product_mixer.component_library.module.SocialGraphServiceModule
import com.twitter.product_mixer.component_library.module.TimelineMixerClientModule
import com.twitter.product_mixer.component_library.module.TimelineRankerClientModule
import com.twitter.product_mixer.component_library.module.TimelineScorerClientModule
import com.twitter.product_mixer.component_library.module.TimelineServiceClientModule
import com.twitter.product_mixer.component_library.module.TweetImpressionStoreModule
import com.twitter.product_mixer.component_library.module.UserSessionStoreModule
import com.twitter.product_mixer.core.controllers.ProductMixerController
import com.twitter.product_mixer.core.module.LoggingThrowableExceptionMapper
import com.twitter.product_mixer.core.module.ProductMixerModule
import com.twitter.product_mixer.core.module.StratoClientModule
import com.twitter.product_mixer.core.module.stringcenter.ProductScopeStringCenterModule
object HomeMixerServerMain extends HomeMixerServer
class HomeMixerServer extends ThriftServer with Mtls with HttpServer with HttpMtls {
override val name = "home-mixer-server"
override val modules: Seq[Module] = Seq(
AccountRecommendationsMixerModule,
AdvertiserBrandSafetySettingsStoreModule,
ClientSentImpressionsPublisherModule,
ConversationServiceModule,
CrMixerClientModule,
EarlybirdModule,
ExploreRankerClientModule,
GizmoduckClientModule,
GlobalParamConfigModule,
HomeAdsCandidateSourceModule,
HomeMixerFlagsModule,
HomeMixerProductModule,
HomeMixerResourcesModule,
HomeNaviModelClientModule,
ImpressionBloomFilterModule,
InjectionHistoryClientModule,
FeedbackHistoryClientModule,
ManhattanClientsModule,
ManhattanFeatureRepositoryModule,
ManhattanTweetImpressionStoreModule,
MemcachedFeatureRepositoryModule,
OnboardingTaskServiceModule,
OptimizedStratoClientModule,
PeopleDiscoveryServiceModule,
ProductMixerModule,
RealGraphInNetworkScoresModule,
RealtimeAggregateFeatureRepositoryModule,
ScoredTweetsMemcacheModule,
ScribeEventPublisherModule,
SimClustersRecentEngagementsClientModule,
SocialGraphServiceModule,
StaleTweetsCacheModule,
StratoClientModule,
ThriftFeatureRepositoryModule,
TimelineMixerClientModule,
TimelineRankerClientModule,
TimelineScorerClientModule,
TimelineServiceClientModule,
TimelinesPersistenceStoreClientModule,
TweetImpressionStoreModule,
TweetyPieClientModule,
TweetypieStaticEntitiesCacheClientModule,
UserMetadataStoreModule,
UserSessionStoreModule,
new DarkTrafficFilterModule[st.HomeMixer.ReqRepServicePerEndpoint](),
new MtlsThriftWebFormsModule[st.HomeMixer.MethodPerEndpoint](this),
new ProductScopeStringCenterModule()
)
def configureThrift(router: ThriftRouter): Unit = {
router
.filter[LoggingMDCFilter]
.filter[TraceIdMDCFilter]
.filter[ThriftMDCFilter]
.filter[StatsFilter]
.filter[AccessLoggingFilter]
.filter[ExceptionMappingFilter]
.filter[Filter.TypeAgnostic, DarkTrafficFilterType]
.exceptionMapper[LoggingThrowableExceptionMapper]
.exceptionMapper[PipelineFailureExceptionMapper]
.add[HomeThriftController]
}
override def configureHttp(router: HttpRouter): Unit =
router.add(
ProductMixerController[st.HomeMixer.MethodPerEndpoint](
this.injector,
st.HomeMixer.ExecutePipeline))
override protected def warmup(): Unit = {
handle[HomeMixerThriftServerWarmupHandler]()
handle[HomeMixerHttpServerWarmupHandler]()
}
}

View File

@ -0,0 +1,73 @@
package com.twitter.home_mixer
import com.twitter.finagle.thrift.ClientId
import com.twitter.finatra.thrift.routing.ThriftWarmup
import com.twitter.home_mixer.{thriftscala => st}
import com.twitter.inject.Logging
import com.twitter.inject.utils.Handler
import com.twitter.product_mixer.core.{thriftscala => pt}
import com.twitter.scrooge.Request
import com.twitter.scrooge.Response
import com.twitter.util.Return
import com.twitter.util.Throw
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class HomeMixerThriftServerWarmupHandler @Inject() (warmup: ThriftWarmup)
extends Handler
with Logging {
private val clientId = ClientId("thrift-warmup-client")
def handle(): Unit = {
val testIds = Seq(1, 2, 3)
try {
clientId.asCurrent {
testIds.foreach { id =>
val warmupReq = warmupQuery(id)
info(s"Sending warm-up request to service with query: $warmupReq")
warmup.sendRequest(
method = st.HomeMixer.GetUrtResponse,
req = Request(st.HomeMixer.GetUrtResponse.Args(warmupReq)))(assertWarmupResponse)
}
}
} catch {
case e: Throwable => error(e.getMessage, e)
}
info("Warm-up done.")
}
private def warmupQuery(userId: Long): st.HomeMixerRequest = {
val clientContext = pt.ClientContext(
userId = Some(userId),
guestId = None,
appId = Some(12345L),
ipAddress = Some("0.0.0.0"),
userAgent = Some("FAKE_USER_AGENT_FOR_WARMUPS"),
countryCode = Some("US"),
languageCode = Some("en"),
isTwoffice = None,
userRoles = None,
deviceId = Some("FAKE_DEVICE_ID_FOR_WARMUPS")
)
st.HomeMixerRequest(
clientContext = clientContext,
product = st.Product.Following,
productContext = Some(st.ProductContext.Following(st.Following())),
maxResults = Some(3)
)
}
private def assertWarmupResponse(
result: Try[Response[st.HomeMixer.GetUrtResponse.SuccessType]]
): Unit = {
result match {
case Return(_) => // ok
case Throw(exception) =>
warn("Error performing warm-up request.")
error(exception.getMessage, exception)
}
}
}

View File

@ -0,0 +1,36 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/query_transformer",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
],
exports = [
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
],
)

View File

@ -0,0 +1,107 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.SocialGraphServiceFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
import com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter
import com.twitter.home_mixer.functional_component.filter.PredicateFeatureFilter
import com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSourceRequest
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
import com.twitter.product_mixer.component_library.filter.FeatureFilter
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator
import com.twitter.product_mixer.core.functional_component.filter.Filter
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
import com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
/**
* Candidate Pipeline Config that fetches tweets from the Conversation Service Candidate Source
*/
class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
conversationServiceCandidateSource: ConversationServiceCandidateSource,
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
socialGraphServiceFeatureHydrator: SocialGraphServiceFeatureHydrator,
namesFeatureHydrator: NamesFeatureHydrator,
override val gates: Seq[BaseGate[Query]],
override val decorator: Option[CandidateDecorator[Query, TweetCandidate]])
extends DependentCandidatePipelineConfig[
Query,
ConversationServiceCandidateSourceRequest,
TweetWithConversationMetadata,
TweetCandidate
] {
override val identifier: CandidatePipelineIdentifier =
CandidatePipelineIdentifier("ConversationService")
private val TweetypieHydratedFilterId = "TweetypieHydrated"
private val QuotedTweetDroppedFilterId = "QuotedTweetDropped"
override val candidateSource: BaseCandidateSource[
ConversationServiceCandidateSourceRequest,
TweetWithConversationMetadata
] = conversationServiceCandidateSource
override val queryTransformer: DependentCandidatePipelineQueryTransformer[
Query,
ConversationServiceCandidateSourceRequest
] = { (_, candidates) =>
val tweetsWithConversationMetadata = candidates.map { candidate =>
TweetWithConversationMetadata(
tweetId = candidate.candidateIdLong,
userId = None,
sourceTweetId = None,
sourceUserId = None,
inReplyToTweetId = None,
conversationId = None,
ancestors = Seq.empty
)
}
ConversationServiceCandidateSourceRequest(tweetsWithConversationMetadata)
}
override val featuresFromCandidateSourceTransformers: Seq[
CandidateFeatureTransformer[TweetWithConversationMetadata]
] = Seq(ConversationServiceResponseFeatureTransformer)
override val resultTransformer: CandidatePipelineResultsTransformer[
TweetWithConversationMetadata,
TweetCandidate
] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }
override val preFilterFeatureHydrationPhase1: Seq[
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
] = Seq(tweetypieFeatureHydrator, socialGraphServiceFeatureHydrator)
override def filters: Seq[Filter[Query, TweetCandidate]] = Seq(
RetweetDeduplicationFilter,
FeatureFilter.fromFeature(FilterIdentifier(TweetypieHydratedFilterId), IsHydratedFeature),
PredicateFeatureFilter.fromPredicate(
FilterIdentifier(QuotedTweetDroppedFilterId),
shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }
),
InvalidConversationModuleFilter
)
override val postFilterFeatureHydration: Seq[
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
] = Seq(namesFeatureHydrator)
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),
HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()
)
}

View File

@ -0,0 +1,34 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.SocialGraphServiceFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class ConversationServiceCandidatePipelineConfigBuilder[Query <: PipelineQuery] @Inject() (
conversationServiceCandidateSource: ConversationServiceCandidateSource,
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
socialGraphServiceFeatureHydrator: SocialGraphServiceFeatureHydrator,
namesFeatureHydrator: NamesFeatureHydrator) {
def build(
gates: Seq[BaseGate[Query]] = Seq.empty,
decorator: Option[CandidateDecorator[Query, TweetCandidate]] = None
): ConversationServiceCandidatePipelineConfig[Query] = {
new ConversationServiceCandidatePipelineConfig(
conversationServiceCandidateSource,
tweetypieFeatureHydrator,
socialGraphServiceFeatureHydrator,
namesFeatureHydrator,
gates,
decorator
)
}
}

View File

@ -0,0 +1,39 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
object ConversationServiceResponseFeatureTransformer
extends CandidateFeatureTransformer[TweetWithConversationMetadata] {
override val identifier: TransformerIdentifier =
TransformerIdentifier("ConversationServiceResponse")
override val features: Set[Feature[_, _]] = Set(
AuthorIdFeature,
InReplyToTweetIdFeature,
IsRetweetFeature,
SourceTweetIdFeature,
SourceUserIdFeature,
ConversationModuleFocalTweetIdFeature,
AncestorsFeature,
SuggestTypeFeature
)
override def transform(candidate: TweetWithConversationMetadata): FeatureMap = FeatureMapBuilder()
.add(AuthorIdFeature, candidate.userId)
.add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)
.add(IsRetweetFeature, candidate.sourceTweetId.isDefined)
.add(SourceTweetIdFeature, candidate.sourceTweetId)
.add(SourceUserIdFeature, candidate.sourceUserId)
.add(ConversationModuleFocalTweetIdFeature, candidate.conversationId)
.add(AncestorsFeature, candidate.ancestors)
.add(SuggestTypeFeature, Some(SuggestType.RankedOrganicTweet))
.build()
}

View File

@ -0,0 +1,84 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.home_mixer.functional_component.candidate_source.StaleTweetsCacheCandidateSource
import com.twitter.home_mixer.functional_component.decorator.HomeFeedbackActionInfoBuilder
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
import com.twitter.home_mixer.functional_component.query_transformer.EditedTweetsCandidatePipelineQueryTransformer
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
import com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.EmptyClientEventInfoBuilder
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
import com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineFocalTweetSafetyLevel
import com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
import javax.inject.Inject
import javax.inject.Singleton
/**
* Candidate Pipeline Config that fetches edited tweets from the Stale Tweets Cache
*/
@Singleton
case class EditedTweetsCandidatePipelineConfig @Inject() (
staleTweetsCacheCandidateSource: StaleTweetsCacheCandidateSource,
namesFeatureHydrator: NamesFeatureHydrator,
homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)
extends DependentCandidatePipelineConfig[
PipelineQuery,
Seq[Long],
Long,
TweetCandidate
] {
override val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier("EditedTweets")
override val candidateSource: BaseCandidateSource[Seq[Long], Long] =
staleTweetsCacheCandidateSource
override val queryTransformer: CandidatePipelineQueryTransformer[
PipelineQuery,
Seq[Long]
] = EditedTweetsCandidatePipelineQueryTransformer
override val resultTransformer: CandidatePipelineResultsTransformer[
Long,
TweetCandidate
] = { candidate => TweetCandidate(id = candidate) }
override val postFilterFeatureHydration: Seq[
BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]
] = Seq(namesFeatureHydrator)
override val decorator: Option[CandidateDecorator[PipelineQuery, TweetCandidate]] = {
val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](
clientEventInfoBuilder = EmptyClientEventInfoBuilder,
entryIdToReplaceBuilder = Some((_, candidate, _) =>
Some(s"${TweetItem.TweetEntryNamespace}-${candidate.id.toString}")),
contextualTweetRefBuilder = Some(
ContextualTweetRefBuilder(
TweetHydrationContext(
// Apply safety level that includes canonical VF treatments that apply regardless of context.
safetyLevelOverride = Some(TimelineFocalTweetSafetyLevel),
outerTweetContext = None
)
)
),
feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)
)
Some(UrtItemCandidateDecorator(tweetItemBuilder))
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.5, 50, 60, 60)
)
}

View File

@ -0,0 +1,123 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.functional_component.gate.RequestContextNotGate
import com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature
import com.twitter.home_mixer.model.request.DeviceContext
import com.twitter.home_mixer.model.request.HasDeviceContext
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.DurationParamBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.ShowAlertCandidateUrtItemBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertColorConfigurationBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertDisplayLocationBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertIconDisplayInfoBuilder
import com.twitter.product_mixer.component_library.gate.FeatureGate
import com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
import com.twitter.product_mixer.core.functional_component.candidate_source.StaticCandidateSource
import com.twitter.product_mixer.core.functional_component.configapi.StaticParam
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseDurationBuilder
import com.twitter.product_mixer.core.functional_component.gate.Gate
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.NewTweets
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Top
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.UpArrow
import com.twitter.product_mixer.core.model.marshalling.response.urt.color.TwitterBlueRosettaColor
import com.twitter.product_mixer.core.model.marshalling.response.urt.color.WhiteRosettaColor
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
import com.twitter.util.Duration
import javax.inject.Inject
import javax.inject.Singleton
/**
* Candidate Pipeline Config that creates the New Tweets Pill
*/
@Singleton
class NewTweetsPillCandidatePipelineConfig[Query <: PipelineQuery with HasDeviceContext] @Inject() (
) extends DependentCandidatePipelineConfig[
Query,
Unit,
ShowAlertCandidate,
ShowAlertCandidate
] {
import NewTweetsPillCandidatePipelineConfig._
override val identifier: CandidatePipelineIdentifier =
CandidatePipelineIdentifier("NewTweetsPill")
override val gates: Seq[Gate[Query]] = Seq(
RequestContextNotGate(Seq(DeviceContext.RequestContext.PullToRefresh)),
FeatureGate.fromFeature(GetNewerFeature)
)
override val candidateSource: CandidateSource[Unit, ShowAlertCandidate] =
StaticCandidateSource(
CandidateSourceIdentifier(identifier.name),
Seq(ShowAlertCandidate(id = identifier.name, userIds = Seq.empty))
)
override val queryTransformer: CandidatePipelineQueryTransformer[Query, Unit] = { _ => Unit }
override val resultTransformer: CandidatePipelineResultsTransformer[
ShowAlertCandidate,
ShowAlertCandidate
] = { candidate => candidate }
override val decorator: Option[CandidateDecorator[Query, ShowAlertCandidate]] = {
val triggerDelayBuilder = new BaseDurationBuilder[Query] {
override def apply(
query: Query,
candidate: ShowAlertCandidate,
features: FeatureMap
): Option[Duration] = {
val delay = query.deviceContext.flatMap(_.requestContextValue) match {
case Some(DeviceContext.RequestContext.TweetSelfThread) => 0.millis
case Some(DeviceContext.RequestContext.ManualRefresh) => 0.millis
case _ => TriggerDelay
}
Some(delay)
}
}
val homeShowAlertCandidateBuilder = ShowAlertCandidateUrtItemBuilder(
alertType = NewTweets,
colorConfigBuilder = StaticShowAlertColorConfigurationBuilder(DefaultColorConfig),
displayLocationBuilder = StaticShowAlertDisplayLocationBuilder(Top),
triggerDelayBuilder = Some(triggerDelayBuilder),
displayDurationBuilder = Some(DurationParamBuilder(StaticParam(DisplayDuration))),
iconDisplayInfoBuilder = Some(StaticShowAlertIconDisplayInfoBuilder(DefaultIconDisplayInfo))
)
Some(UrtItemCandidateDecorator(homeShowAlertCandidateBuilder))
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),
HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()
)
}
object NewTweetsPillCandidatePipelineConfig {
val DefaultColorConfig: ShowAlertColorConfiguration = ShowAlertColorConfiguration(
background = TwitterBlueRosettaColor,
text = WhiteRosettaColor,
border = Some(WhiteRosettaColor)
)
val DefaultIconDisplayInfo: ShowAlertIconDisplayInfo =
ShowAlertIconDisplayInfo(icon = UpArrow, tint = WhiteRosettaColor)
// Unlimited display time (until user takes action)
val DisplayDuration = -1.millisecond
val TriggerDelay = 4.minutes
}

View File

@ -0,0 +1,34 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
import com.twitter.timelineservice.{thriftscala => t}
object TimelineServiceResponseFeatureTransformer extends CandidateFeatureTransformer[t.Tweet] {
override val identifier: TransformerIdentifier = TransformerIdentifier("TimelineServiceResponse")
override val features: Set[Feature[_, _]] = Set(
AuthorIdFeature,
InReplyToTweetIdFeature,
IsRetweetFeature,
SourceTweetIdFeature,
SourceUserIdFeature,
)
override def transform(candidate: t.Tweet): FeatureMap = FeatureMapBuilder()
.add(AuthorIdFeature, candidate.userId)
.add(InReplyToTweetIdFeature, candidate.inReplyToStatusId)
.add(IsRetweetFeature, candidate.sourceStatusId.isDefined)
.add(SourceTweetIdFeature, candidate.sourceStatusId)
.add(SourceUserIdFeature, candidate.sourceUserId)
.build()
}

View File

@ -0,0 +1,20 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
"home-mixer/thrift/src/main/thrift:thrift-scala",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt",
"snowflake/src/main/scala/com/twitter/snowflake/id",
"src/thrift/com/twitter/context:twitter-context-scala",
"src/thrift/com/twitter/timelines/render:thrift-scala",
"twitter-context/src/main/scala",
],
)

View File

@ -0,0 +1,50 @@
package com.twitter.home_mixer.controller
import com.twitter.finatra.thrift.Controller
import com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller
import com.twitter.home_mixer.model.request.HomeMixerRequest
import com.twitter.home_mixer.service.ScoredTweetsService
import com.twitter.home_mixer.{thriftscala => t}
import com.twitter.product_mixer.core.controllers.DebugTwitterContext
import com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder
import com.twitter.product_mixer.core.service.urt.UrtService
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch
import com.twitter.timelines.configapi.Params
import javax.inject.Inject
class HomeThriftController @Inject() (
homeRequestUnmarshaller: HomeMixerRequestUnmarshaller,
urtService: UrtService,
scoredTweetsService: ScoredTweetsService,
paramsBuilder: ParamsBuilder)
extends Controller(t.HomeMixer)
with DebugTwitterContext {
handle(t.HomeMixer.GetUrtResponse) { args: t.HomeMixer.GetUrtResponse.Args =>
val request = homeRequestUnmarshaller(args.request)
val params = buildParams(request)
Stitch.run(urtService.getUrtResponse[HomeMixerRequest](request, params))
}
handle(t.HomeMixer.GetScoredTweetsResponse) { args: t.HomeMixer.GetScoredTweetsResponse.Args =>
val request = homeRequestUnmarshaller(args.request)
val params = buildParams(request)
withDebugTwitterContext(request.clientContext) {
Stitch.run(scoredTweetsService.getScoredTweetsResponse[HomeMixerRequest](request, params))
}
}
private def buildParams(request: HomeMixerRequest): Params = {
val userAgeOpt = request.clientContext.userId.map { userId =>
SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)
}
val fsCustomMapInput = userAgeOpt.map("account_age_in_days" -> _).toMap
paramsBuilder.build(
clientContext = request.clientContext,
product = request.product,
featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),
fsCustomMapInput = fsCustomMapInput
)
}
}

View File

@ -0,0 +1,22 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"3rdparty/jvm/javax/inject:javax.inject",
"finagle/finagle-memcached/src/main/scala",
"finatra/inject/inject-core/src/main/scala",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala",
"src/thrift/com/twitter/search:earlybird-scala",
"stitch/stitch-timelineservice/src/main/scala",
"strato/config/columns/recommendations/similarity:similarity-strato-client",
"strato/src/main/scala/com/twitter/strato/client",
],
exports = [
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
],
)

View File

@ -0,0 +1,44 @@
package com.twitter.home_mixer.functional_component.candidate_source
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
import com.twitter.search.earlybird.{thriftscala => t}
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Singleton
case object EarlybirdResponseTruncatedFeature
extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Boolean] {
override val defaultValue: Boolean = false
}
case object EarlybirdBottomTweetFeature
extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Option[Long]] {
override val defaultValue: Option[Long] = None
}
@Singleton
case class EarlybirdCandidateSource @Inject() (
earlybird: t.EarlybirdService.MethodPerEndpoint)
extends CandidateSourceWithExtractedFeatures[t.EarlybirdRequest, t.ThriftSearchResult] {
override val identifier = CandidateSourceIdentifier("Earlybird")
override def apply(
request: t.EarlybirdRequest
): Stitch[CandidatesWithSourceFeatures[t.ThriftSearchResult]] = {
Stitch.callFuture(earlybird.search(request)).map { response =>
val candidates = response.searchResults.map(_.results).getOrElse(Seq.empty)
val features = FeatureMapBuilder()
.add(EarlybirdResponseTruncatedFeature, candidates.size == request.searchQuery.numResults)
.add(EarlybirdBottomTweetFeature, candidates.lastOption.map(_.id))
.build()
CandidatesWithSourceFeatures(candidates, features)
}
}
}

View File

@ -0,0 +1,34 @@
package com.twitter.home_mixer.functional_component.candidate_source
import com.twitter.hermit.candidate.{thriftscala => t}
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
import com.twitter.stitch.Stitch
import com.twitter.strato.client.Fetcher
import com.twitter.strato.generated.client.recommendations.similarity.SimilarUsersBySimsOnUserClientColumn
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class SimilarityBasedUsersCandidateSource @Inject() (
similarUsersBySimsOnUserClientColumn: SimilarUsersBySimsOnUserClientColumn)
extends CandidateSource[Seq[Long], t.Candidate] {
override val identifier: CandidateSourceIdentifier =
CandidateSourceIdentifier("SimilarityBasedUsers")
private val fetcher: Fetcher[Long, Unit, t.Candidates] =
similarUsersBySimsOnUserClientColumn.fetcher
override def apply(request: Seq[Long]): Stitch[Seq[t.Candidate]] = {
Stitch
.collect {
request.map { userId =>
fetcher.fetch(userId, Unit).map { result =>
result.v.map(_.candidates).getOrElse(Seq.empty)
}
}
}.map(_.flatten)
}
}

View File

@ -0,0 +1,30 @@
package com.twitter.home_mixer.functional_component.candidate_source
import com.google.inject.name.Named
import com.twitter.finagle.memcached.{Client => MemcachedClient}
import com.twitter.home_mixer.param.HomeMixerInjectionNames.StaleTweetsCache
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class StaleTweetsCacheCandidateSource @Inject() (
@Named(StaleTweetsCache) staleTweetsCache: MemcachedClient)
extends CandidateSource[Seq[Long], Long] {
override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier("StaleTweetsCache")
private val StaleTweetsCacheKeyPrefix = "v1_"
override def apply(request: Seq[Long]): Stitch[Seq[Long]] = {
val keys = request.map(StaleTweetsCacheKeyPrefix + _)
Stitch.callFuture(staleTweetsCache.get(keys).map { tweets =>
tweets.map {
case (k, _) => k.replaceFirst(StaleTweetsCacheKeyPrefix, "").toLong
}.toSeq
})
}
}

View File

@ -0,0 +1,34 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.service.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class AuthorChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
CandidatesUtil.getOriginalAuthorId(candidateFeatures).flatMap { authorId =>
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
userId = authorId,
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
promptExternalString = externalStrings.showFewerTweetsString,
confirmationExternalString = externalStrings.showFewerTweetsConfirmationString,
engagementType = t.FeedbackEngagementType.Tweet,
stringCenter = stringCenter,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
}
}
}

View File

@ -0,0 +1,32 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"finagle/finagle-core/src/main",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
"joinkey/src/main/scala/com/twitter/joinkey/context",
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
"src/scala/com/twitter/suggests/controller_data",
"src/thrift/com/twitter/suggests/controller_data:controller_data-scala",
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
"stringcenter/client",
"stringcenter/client/src/main/java",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/translation",
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
],
)

View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class BlockUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val userIdOpt =
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
candidateFeatures.getOrElse(SourceUserIdFeature, None)
else candidateFeatures.getOrElse(AuthorIdFeature, None)
userIdOpt.flatMap { userId =>
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.blockUserString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = Some(BottomSheet),
clientEventInfo = None,
icon = Some(icon.No),
richBehavior = Some(RichFeedbackBehaviorBlockUser(userId)),
subprompt = None
)
}
}
}
}

View File

@ -0,0 +1,88 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DontLike
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.{thriftscala => tls}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class DontLikeFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings,
authorChildFeedbackActionBuilder: AuthorChildFeedbackActionBuilder,
retweeterChildFeedbackActionBuilder: RetweeterChildFeedbackActionBuilder,
notRelevantChildFeedbackActionBuilder: NotRelevantChildFeedbackActionBuilder,
unfollowUserChildFeedbackActionBuilder: UnfollowUserChildFeedbackActionBuilder,
muteUserChildFeedbackActionBuilder: MuteUserChildFeedbackActionBuilder,
blockUserChildFeedbackActionBuilder: BlockUserChildFeedbackActionBuilder,
reportTweetChildFeedbackActionBuilder: ReportTweetChildFeedbackActionBuilder) {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackAction] = {
CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>
val feedbackEntities = Seq(
tlc.FeedbackEntity.TweetId(candidate.id),
tlc.FeedbackEntity.UserId(authorId)
)
val feedbackMetadata = FeedbackMetadata(
engagementType = None,
entityIds = feedbackEntities,
ttl = Some(30.days)
)
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tls.FeedbackType.DontLike,
feedbackMetadata = feedbackMetadata,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
val childFeedbackActions = if (query.params(EnableNahFeedbackInfoParam)) {
Seq(
unfollowUserChildFeedbackActionBuilder(candidateFeatures),
muteUserChildFeedbackActionBuilder(candidateFeatures),
blockUserChildFeedbackActionBuilder(candidateFeatures),
reportTweetChildFeedbackActionBuilder(candidate)
).flatten
} else {
Seq(
authorChildFeedbackActionBuilder(candidateFeatures),
retweeterChildFeedbackActionBuilder(candidateFeatures),
notRelevantChildFeedbackActionBuilder(candidate, candidateFeatures)
).flatten
}
FeedbackAction(
feedbackType = DontLike,
prompt = Some(stringCenter.prepare(externalStrings.dontLikeString)),
confirmation = Some(stringCenter.prepare(externalStrings.dontLikeConfirmationString)),
childFeedbackActions =
if (childFeedbackActions.nonEmpty) Some(childFeedbackActions) else None,
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(Frown),
richBehavior = None,
subprompt = None,
encodedFeedbackRequest = None
)
}
}
}

View File

@ -0,0 +1,119 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stringcenter.client.StringCenter
import com.twitter.stringcenter.client.core.ExternalString
private[decorator] case class SocialContextIdAndScreenName(
socialContextId: Long,
screenName: String)
object EngagerSocialContextBuilder {
private val UserIdRequestParamName = "user_id"
private val DirectInjectionContentSourceRequestParamName = "dis"
private val DirectInjectionIdRequestParamName = "diid"
private val DirectInjectionContentSourceSocialProofUsers = "socialproofusers"
private val SocialProofUrl = ""
}
case class EngagerSocialContextBuilder(
contextType: GeneralContextType,
stringCenter: StringCenter,
oneUserString: ExternalString,
twoUsersString: ExternalString,
moreUsersString: ExternalString,
timelineTitle: ExternalString) {
import EngagerSocialContextBuilder._
def apply(
socialContextIds: Seq[Long],
query: PipelineQuery,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
val validSocialContextIdAndScreenNames = socialContextIds.flatMap { socialContextId =>
realNames
.get(socialContextId).map(screenName =>
SocialContextIdAndScreenName(socialContextId, screenName))
}
validSocialContextIdAndScreenNames match {
case Seq(user) =>
val socialContextString =
stringCenter.prepare(oneUserString, Map("user" -> user.screenName))
Some(mkOneUserSocialContext(socialContextString, user.socialContextId))
case Seq(firstUser, secondUser) =>
val socialContextString =
stringCenter
.prepare(
twoUsersString,
Map("user1" -> firstUser.screenName, "user2" -> secondUser.screenName))
Some(
mkManyUserSocialContext(
socialContextString,
query.getRequiredUserId,
validSocialContextIdAndScreenNames.map(_.socialContextId)))
case firstUser +: otherUsers =>
val otherUsersCount = otherUsers.size
val socialContextString =
stringCenter
.prepare(
moreUsersString,
Map("user" -> firstUser.screenName, "count" -> otherUsersCount))
Some(
mkManyUserSocialContext(
socialContextString,
query.getRequiredUserId,
validSocialContextIdAndScreenNames.map(_.socialContextId)))
case _ => None
}
}
private def mkOneUserSocialContext(socialContextString: String, userId: Long): GeneralContext = {
GeneralContext(
contextType = contextType,
text = socialContextString,
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
)
)
)
}
private def mkManyUserSocialContext(
socialContextString: String,
viewerId: Long,
socialContextIds: Seq[Long]
): GeneralContext = {
GeneralContext(
contextType = contextType,
text = socialContextString,
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = UrtEndpoint,
url = SocialProofUrl,
urtEndpointOptions = Some(UrtEndpointOptions(
requestParams = Some(Map(
UserIdRequestParamName -> viewerId.toString,
DirectInjectionContentSourceRequestParamName -> DirectInjectionContentSourceSocialProofUsers,
DirectInjectionIdRequestParamName -> socialContextIds.mkString(",")
)),
title = Some(stringCenter.prepare(timelineTitle)),
cacheId = None,
subtitle = None
))
))
)
}
}

View File

@ -0,0 +1,78 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Use '@A replied' when the root tweet is out-of-network and the reply is in network.
*
* This function should only be called for the root Tweet of convo modules. This is enforced by
* [[HomeTweetSocialContextBuilder]].
*/
@Singleton
case class ExtendedReplySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val extendedReplyString = externalStrings.socialContextExtendedReply
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// If these values are missing default to not showing an extended reply banner
val inNetworkRoot = candidateFeatures.getOrElse(InNetworkFeature, true)
val inNetworkFocalTweet =
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(false)
if (!inNetworkRoot && inNetworkFocalTweet) {
val focalTweetAuthorIdOpt = candidateFeatures.getOrElse(FocalTweetAuthorIdFeature, None)
val focalTweetRealNames =
candidateFeatures
.getOrElse(FocalTweetRealNamesFeature, None).getOrElse(Map.empty[Long, String])
val focalTweetAuthorNameOpt = focalTweetAuthorIdOpt.flatMap(focalTweetRealNames.get)
(focalTweetAuthorIdOpt, focalTweetAuthorNameOpt) match {
case (Some(focalTweetAuthorId), Some(focalTweetAuthorName)) =>
Some(
GeneralContext(
contextType = ConversationGeneralContextType,
text = stringCenter
.prepare(extendedReplyString, placeholders = Map("user1" -> focalTweetAuthorName)),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
))
))
case _ =>
None
}
} else {
None
}
}
}

View File

@ -0,0 +1,61 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.conversions.DurationOps._
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer
import com.twitter.stringcenter.client.StringCenter
import com.twitter.stringcenter.client.core.ExternalString
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelines.service.{thriftscala => t}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.suggests.{thriftscala => st}
import com.twitter.timelineservice.{thriftscala => tlst}
object FeedbackUtil {
val FeedbackTtl = 30.days
def buildUserSeeFewerChildFeedbackAction(
userId: Long,
namesByUserId: Map[Long, String],
promptExternalString: ExternalString,
confirmationExternalString: ExternalString,
engagementType: t.FeedbackEngagementType,
stringCenter: StringCenter,
injectionType: Option[st.SuggestType]
): Option[ChildFeedbackAction] = {
namesByUserId.get(userId).map { userScreenName =>
val prompt = stringCenter.prepare(
promptExternalString,
Map("user" -> userScreenName)
)
val confirmation = stringCenter.prepare(
confirmationExternalString,
Map("user" -> userScreenName)
)
val feedbackMetadata = FeedbackMetadata(
engagementType = Some(engagementType),
entityIds = Seq(tlc.FeedbackEntity.UserId(userId)),
ttl = Some(FeedbackTtl))
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tlst.FeedbackType.SeeFewer,
feedbackMetadata = feedbackMetadata,
injectionType = injectionType
)
ChildFeedbackAction(
feedbackType = SeeFewer,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior = None,
subprompt = None
)
}
}
}

View File

@ -0,0 +1,53 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class FollowedBySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
contextType = FollowGeneralContextType,
stringCenter = stringCenter,
oneUserString = externalStrings.socialContextOneUserFollowsString,
twoUsersString = externalStrings.socialContextTwoUsersFollowString,
moreUsersString = externalStrings.socialContextMoreUsersFollowString,
timelineTitle = externalStrings.socialContextFollowedByTimelineTitle
)
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// Only apply followed-by social context for OON Tweets
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
if (!inNetwork) {
val validFollowedByUserIds =
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
engagerSocialContextBuilder(
socialContextIds = validFollowedByUserIds,
query = query,
candidateFeatures = candidateFeatures
)
} else {
None
}
}
}

View File

@ -0,0 +1,46 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.finagle.tracing.Trace
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
case class HomeAdsClientEventDetailsBuilder(injectionType: Option[String])
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
override def apply(
query: PipelineQuery,
candidate: UniversalNoun[Any],
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
tweetTypesBitmap = 0L,
traceId = Some(Trace.id.traceId.toLong),
requestJoinId = None)
val serializedControllerData = HomeClientEventDetailsBuilder.ControllerDataSerializer(
ControllerData.V2(
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = injectionType,
controllerData = Some(serializedControllerData),
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

View File

@ -0,0 +1,92 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.bijection.Base64String
import com.twitter.bijection.scrooge.BinaryScalaCodec
import com.twitter.bijection.{Injection => Serializer}
import com.twitter.finagle.tracing.Trace
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
import com.twitter.home_mixer.model.HomeFeatures.PositionFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.joinkey.context.RequestJoinKeyContext
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.suggests.controller_data.Home
import com.twitter.suggests.controller_data.TweetTypeGenerator
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
object HomeClientEventDetailsBuilder {
implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =
BinaryScalaCodec(ControllerData)
val ControllerDataSerializer: Serializer[ControllerData, String] =
Serializer.connect[ControllerData, Array[Byte], Base64String, String]
/**
* define getRequestJoinId as a method(def) rather than a val because each new request
* needs to call the context to update the id.
*/
private def getRequestJoinId(): Option[Long] =
RequestJoinKeyContext.current.flatMap(_.requestJoinId)
}
case class HomeClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](
) extends BaseClientEventDetailsBuilder[Query, Candidate]
with TweetTypeGenerator[FeatureMap] {
import HomeClientEventDetailsBuilder._
override def apply(
query: Query,
candidate: Candidate,
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val tweetTypesBitmaps = mkTweetTypesBitmaps(
Home.TweetTypeIdxMap,
HomeTweetTypePredicates.PredicateMap,
candidateFeatures)
val tweetTypesListBytes = mkItemTypesBitmapsV2(
Home.TweetTypeIdxMap,
HomeTweetTypePredicates.PredicateMap,
candidateFeatures)
val candidateSourceId =
candidateFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toByte)
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
tweetTypesBitmap = tweetTypesBitmaps.getOrElse(0, 0L),
tweetTypesBitmapContinued1 = tweetTypesBitmaps.get(1),
candidateTweetSourceId = candidateSourceId,
traceId = Some(Trace.id.traceId.toLong),
injectedPosition = candidateFeatures.getOrElse(PositionFeature, None),
tweetTypesListBytes = Some(tweetTypesListBytes),
requestJoinId = getRequestJoinId(),
)
val serializedControllerData = ControllerDataSerializer(
ControllerData.V2(
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None).map(_.name),
controllerData = Some(serializedControllerData),
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

View File

@ -0,0 +1,49 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
import com.twitter.timelineservice.suggests.{thriftscala => st}
object HomeConversationServiceCandidateDecorator {
private val ConversationModuleNamespace = EntryNamespace("home-conversation")
def apply(
homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder
): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {
val suggestType = st.SuggestType.RankedOrganicTweet
val component = InjectionScribeUtil.scribeComponent(suggestType).get
val clientEventInfoBuilder = ClientEventInfoBuilder(component)
val tweetItemBuilder = TweetCandidateUrtItemBuilder(
clientEventInfoBuilder = clientEventInfoBuilder,
timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),
feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)
)
val moduleBuilder = TimelineModuleBuilder(
entryNamespace = ConversationModuleNamespace,
clientEventInfoBuilder = clientEventInfoBuilder,
displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),
metadataBuilder = Some(HomeConversationModuleMetadataBuilder())
)
Some(
UrtMultipleModulesDecorator(
urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),
moduleBuilder = moduleBuilder,
groupByKey = (_, _, candidateFeatures) =>
candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)
))
}
}

View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.model.request.ForYouProduct
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.service.{thriftscala => t}
import com.twitter.timelines.util.FeedbackMetadataSerializer
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class HomeFeedbackActionInfoBuilder @Inject() (
notInterestedTopicFeedbackActionBuilder: NotInterestedTopicFeedbackActionBuilder,
dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder)
extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackActionInfo] = {
val supportedProduct = query.product match {
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
case ForYouProduct => true
case _ => false
}
val isAuthoredByViewer = CandidatesUtil.isAuthoredByViewer(query, candidateFeatures)
if (supportedProduct && !isAuthoredByViewer) {
val feedbackActions = Seq(
notInterestedTopicFeedbackActionBuilder(candidateFeatures),
dontLikeFeedbackActionBuilder(query, candidate, candidateFeatures)
).flatten
val feedbackMetadata = FeedbackMetadataSerializer.serialize(
t.FeedbackMetadata(injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)))
Some(
FeedbackActionInfo(
feedbackActions = feedbackActions,
feedbackMetadata = Some(feedbackMetadata),
displayContext = None,
clientEventInfo = None
))
} else None
}
}

View File

@ -0,0 +1,18 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
object HomeQueryTypePredicates {
private[this] val QueryPredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
("request", _ => true),
("get_initial", _.getOrElse(GetInitialFeature, false)),
("get_newer", _.getOrElse(GetNewerFeature, false)),
("get_older", _.getOrElse(GetOlderFeature, false)),
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
("request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
("request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false))
)
val PredicateMap = QueryPredicates.toMap
}

View File

@ -0,0 +1,26 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSendScoresToClient
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
object HomeTimelinesScoreInfoBuilder
extends BaseTimelinesScoreInfoBuilder[PipelineQuery, TweetCandidate] {
private val UndefinedTweetScore = -1.0
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[TimelinesScoreInfo] = {
if (query.params(EnableSendScoresToClient)) {
val score = candidateFeatures.getOrElse(ScoreFeature, None).getOrElse(UndefinedTweetScore)
Some(TimelinesScoreInfo(score))
} else None
}
}

View File

@ -0,0 +1,44 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSocialContextParam
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class HomeTweetSocialContextBuilder @Inject() (
likedBySocialContextBuilder: LikedBySocialContextBuilder,
followedBySocialContextBuilder: FollowedBySocialContextBuilder,
topicSocialContextBuilder: TopicSocialContextBuilder,
extendedReplySocialContextBuilder: ExtendedReplySocialContextBuilder,
receivedReplySocialContextBuilder: ReceivedReplySocialContextBuilder)
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
features: FeatureMap
): Option[SocialContext] = {
if (query.params(EnableSocialContextParam)) {
features.getOrElse(ConversationModuleFocalTweetIdFeature, None) match {
case None =>
likedBySocialContextBuilder(query, candidate, features)
.orElse(followedBySocialContextBuilder(query, candidate, features))
.orElse(topicSocialContextBuilder(query, candidate, features))
case Some(_) =>
val conversationId = features.getOrElse(ConversationModuleIdFeature, None)
// Only hydrate the social context into the root tweet in a conversation module
if (conversationId.contains(candidate.id)) {
extendedReplySocialContextBuilder(query, candidate, features)
.orElse(receivedReplySocialContextBuilder(query, candidate, features))
} else None
}
} else None
}
}

View File

@ -0,0 +1,250 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.timelinemixer.injection.model.candidate.SemanticCoreFeatures
import com.twitter.tweetypie.{thriftscala => tpt}
object HomeTweetTypePredicates {
/**
* IMPORTANT: Please avoid logging tweet types that are tied to sensitive
* internal author information / labels (e.g. blink labels, abuse labels, or geo-location).
*/
private[this] val CandidatePredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
("with_candidate", _ => true),
("retweet", _.getOrElse(IsRetweetFeature, false)),
("reply", _.getOrElse(InReplyToTweetIdFeature, None).nonEmpty),
("image", _.getOrElse(EarlybirdFeature, None).exists(_.hasImage)),
("video", _.getOrElse(EarlybirdFeature, None).exists(_.hasVideo)),
("link", _.getOrElse(EarlybirdFeature, None).exists(_.hasVisibleLink)),
("quote", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuote.contains(true))),
("like_social_context", _.getOrElse(NonSelfFavoritedByUserIdsFeature, Seq.empty).nonEmpty),
("protected", _.getOrElse(EarlybirdFeature, None).exists(_.isProtected)),
(
"has_exclusive_conversation_author_id",
_.getOrElse(ExclusiveConversationAuthorIdFeature, None).nonEmpty),
("is_eligible_for_connect_boost", _.getOrElse(AuthorIsEligibleForConnectBoostFeature, false)),
("hashtag", _.getOrElse(EarlybirdFeature, None).exists(_.numHashtags > 0)),
("has_scheduled_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isScheduled)),
("has_recorded_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isRecorded)),
("is_read_from_cache", _.getOrElse(IsReadFromCacheFeature, false)),
(
"is_self_thread_tweet",
_.getOrElse(ConversationFeature, None).exists(_.isSelfThreadTweet.contains(true))),
("get_initial", _.getOrElse(GetInitialFeature, false)),
("get_newer", _.getOrElse(GetNewerFeature, false)),
("get_middle", _.getOrElse(GetMiddleFeature, false)),
("get_older", _.getOrElse(GetOlderFeature, false)),
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
("polling", _.getOrElse(PollingFeature, false)),
("tls_size_20_plus", _ => false),
("near_empty", _ => false),
("ranked_request", _ => false),
("mutual_follow", _.getOrElse(EarlybirdFeature, None).exists(_.fromMutualFollow)),
("has_ticketed_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasTickets)),
("in_utis_top5", _.getOrElse(PositionFeature, None).exists(_ < 5)),
("is_utis_pos0", _.getOrElse(PositionFeature, None).exists(_ == 0)),
("is_utis_pos1", _.getOrElse(PositionFeature, None).exists(_ == 1)),
("is_utis_pos2", _.getOrElse(PositionFeature, None).exists(_ == 2)),
("is_utis_pos3", _.getOrElse(PositionFeature, None).exists(_ == 3)),
("is_utis_pos4", _.getOrElse(PositionFeature, None).exists(_ == 4)),
(
"is_signup_request",
candidate => candidate.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
("empty_request", _ => false),
("served_size_less_than_5", _.getOrElse(ServedSizeFeature, None).exists(_ < 5)),
("served_size_less_than_10", _.getOrElse(ServedSizeFeature, None).exists(_ < 10)),
("served_size_less_than_20", _.getOrElse(ServedSizeFeature, None).exists(_ < 20)),
("served_size_less_than_50", _.getOrElse(ServedSizeFeature, None).exists(_ < 50)),
(
"served_size_between_50_and_100",
_.getOrElse(ServedSizeFeature, None).exists(size => size >= 50 && size < 100)),
("authored_by_contextual_user", _.getOrElse(AuthoredByContextualUserFeature, false)),
("has_ancestors", _.getOrElse(AncestorsFeature, Seq.empty).nonEmpty),
("full_scoring_succeeded", _.getOrElse(FullScoringSucceededFeature, false)),
(
"account_age_less_than_30_minutes",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
(
"account_age_less_than_1_day",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 1.day)),
(
"account_age_less_than_7_days",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 7.days)),
(
"directed_at_user_is_in_first_degree",
_.getOrElse(EarlybirdFeature, None).exists(_.directedAtUserIdIsInFirstDegree.contains(true))),
("root_user_is_in_first_degree", _ => false),
(
"has_semantic_core_annotation",
_.getOrElse(EarlybirdFeature, None).exists(_.semanticCoreAnnotations.nonEmpty)),
("is_request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false)),
(
"part_of_utt",
_.getOrElse(EarlybirdFeature, None)
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy)))),
("is_random_tweet", _.getOrElse(IsRandomTweetFeature, false)),
("has_random_tweet_in_response", _.getOrElse(HasRandomTweetFeature, false)),
("is_random_tweet_above_in_utis", _.getOrElse(IsRandomTweetAboveFeature, false)),
("is_request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
("viewer_is_employee", _ => false),
("viewer_is_timelines_employee", _ => false),
("viewer_follows_any_topics", _.getOrElse(UserFollowedTopicsCountFeature, None).exists(_ > 0)),
(
"has_ancestor_authored_by_viewer",
candidate =>
candidate
.getOrElse(AncestorsFeature, Seq.empty).exists(ancestor =>
candidate.getOrElse(ViewerIdFeature, 0L) == ancestor.userId)),
("ancestor", _.getOrElse(IsAncestorCandidateFeature, false)),
(
"root_ancestor",
candidate =>
candidate.getOrElse(IsAncestorCandidateFeature, false) && candidate
.getOrElse(InReplyToTweetIdFeature, None).isEmpty),
(
"deep_reply",
candidate =>
candidate.getOrElse(InReplyToTweetIdFeature, None).nonEmpty && candidate
.getOrElse(AncestorsFeature, Seq.empty).size > 2),
(
"has_simcluster_embeddings",
_.getOrElse(
SimclustersTweetTopKClustersWithScoresFeature,
Map.empty[String, Double]).nonEmpty),
(
"tweet_age_less_than_15_seconds",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 15.seconds)),
("is_followed_topic_tweet", _ => false),
("is_recommended_topic_tweet", _ => false),
("is_topic_tweet", _ => false),
("preferred_language_matches_tweet_language", _ => false),
(
"device_language_matches_tweet_language",
candidate =>
candidate.getOrElse(TweetLanguageFeature, None) ==
candidate.getOrElse(DeviceLanguageFeature, None)),
("question", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuestion.contains(true))),
("in_network", _.getOrElse(FromInNetworkSourceFeature, true)),
("viewer_follows_original_author", _ => false),
("has_account_follow_prompt", _ => false),
("has_relevance_prompt", _ => false),
("has_topic_annotation_haug_prompt", _ => false),
("has_topic_annotation_random_precision_prompt", _ => false),
("has_topic_annotation_prompt", _ => false),
(
"has_political_annotation",
_.getOrElse(EarlybirdFeature, None).exists(
_.semanticCoreAnnotations.exists(
_.exists(annotation =>
SemanticCoreFeatures.PoliticalDomains.contains(annotation.domainId) ||
(annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
annotation.entityId == SemanticCoreFeatures.UttPoliticsEntityId))))),
(
"is_dont_at_me_by_invitation",
_.getOrElse(EarlybirdFeature, None).exists(
_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.ByInvitation]))),
(
"is_dont_at_me_community",
_.getOrElse(EarlybirdFeature, None)
.exists(_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.Community]))),
("has_zero_score", _.getOrElse(ScoreFeature, None).exists(_ == 0.0)),
("is_viewer_not_invited_to_reply", _ => false),
("is_viewer_invited_to_reply", _ => false),
("has_gte_10_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10))),
("has_gte_100_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100))),
("has_gte_1k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
(
"has_gte_10k_favs",
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
(
"has_gte_100k_favs",
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100000))),
("above_neighbor_is_topic_tweet", _ => false),
("is_topic_tweet_with_neighbor_below", _ => false),
("has_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasSpace)),
("has_live_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isLive)),
(
"has_gte_10_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 10))),
(
"has_gte_100_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 100))),
(
"has_gte_1k_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 1000))),
(
"has_us_political_annotation",
_.getOrElse(EarlybirdFeature, None)
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
annotation.entityId == SemanticCoreFeatures.usPoliticalTweetEntityId &&
annotation.groupId == SemanticCoreFeatures.UsPoliticalTweetAnnotationGroupIds.BalancedV0)))),
(
"has_toxicity_score_above_threshold",
_.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91))),
(
"text_only",
candidate =>
candidate.getOrElse(HasDisplayedTextFeature, false) &&
!(candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) ||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasVideo) ||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasCard))),
(
"image_only",
candidate =>
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) &&
!candidate.getOrElse(HasDisplayedTextFeature, false)),
("has_1_image", _.getOrElse(NumImagesFeature, None).exists(_ == 1)),
("has_2_images", _.getOrElse(NumImagesFeature, None).exists(_ == 2)),
("has_3_images", _.getOrElse(NumImagesFeature, None).exists(_ == 3)),
("has_4_images", _.getOrElse(NumImagesFeature, None).exists(_ == 4)),
("has_card", _.getOrElse(EarlybirdFeature, None).exists(_.hasCard)),
("3_or_more_consecutive_not_in_network", _ => false),
("2_or_more_consecutive_not_in_network", _ => false),
("5_out_of_7_not_in_network", _ => false),
("7_out_of_7_not_in_network", _ => false),
("5_out_of_5_not_in_network", _ => false),
("user_follow_count_gte_50", _.getOrElse(UserFollowingCountFeature, None).exists(_ > 50)),
("has_liked_by_social_context", _ => false),
("has_followed_by_social_context", _ => false),
("has_topic_social_context", _ => false),
("timeline_entry_has_banner", _ => false),
("served_in_conversation_module", _.getOrElse(ServedInConversationModuleFeature, false)),
(
"conversation_module_has_2_displayed_tweets",
_.getOrElse(ConversationModule2DisplayedTweetsFeature, false)),
("conversation_module_has_gap", _.getOrElse(ConversationModuleHasGapFeature, false)),
("served_in_recap_tweet_candidate_module_injection", _ => false),
("served_in_threaded_conversation_module", _ => false),
(
"author_is_elon",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
(
"author_is_power_user",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)),
(
"author_is_democrat",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsDemocratsFeature, Set.empty[Long]).contains)),
(
"author_is_republican",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsRepublicansFeature, Set.empty[Long]).contains)),
)
val PredicateMap = CandidatePredicates.toMap
}

View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LikeGeneralContextType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class LikedBySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
contextType = LikeGeneralContextType,
stringCenter = stringCenter,
oneUserString = externalStrings.socialContextOneUserLikedString,
twoUsersString = externalStrings.socialContextTwoUsersLikedString,
moreUsersString = externalStrings.socialContextMoreUsersLikedString,
timelineTitle = externalStrings.socialContextLikedByTimelineTitle
)
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// Liked by users are valid only if they pass both the SGS and Perspective filters.
val validLikedByUserIds =
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
.filter(
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
engagerSocialContextBuilder(
socialContextIds = validLikedByUserIds,
query = query,
candidateFeatures = candidateFeatures
)
}
}

View File

@ -0,0 +1,47 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
import com.twitter.home_mixer.functional_component.decorator.builder.ListClientEventDetailsBuilder
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
import com.twitter.timelineservice.suggests.{thriftscala => st}
object ListConversationServiceCandidateDecorator {
private val ConversationModuleNamespace = EntryNamespace("list-conversation")
def apply(): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {
val suggestType = st.SuggestType.OrganicListTweet
val component = InjectionScribeUtil.scribeComponent(suggestType).get
val clientEventInfoBuilder =
ClientEventInfoBuilder(component, Some(ListClientEventDetailsBuilder))
val tweetItemBuilder = TweetCandidateUrtItemBuilder(
clientEventInfoBuilder = clientEventInfoBuilder
)
val moduleBuilder = TimelineModuleBuilder(
entryNamespace = ConversationModuleNamespace,
clientEventInfoBuilder = clientEventInfoBuilder,
displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),
metadataBuilder = Some(HomeConversationModuleMetadataBuilder())
)
Some(
UrtMultipleModulesDecorator(
urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),
moduleBuilder = moduleBuilder,
groupByKey = (_, _, candidateFeatures) =>
candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)
))
}
}

View File

@ -0,0 +1,55 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class MuteUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidateFeatures: FeatureMap
): Option[ChildFeedbackAction] = {
val userIdOpt =
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
candidateFeatures.getOrElse(SourceUserIdFeature, None)
else candidateFeatures.getOrElse(AuthorIdFeature, None)
userIdOpt.flatMap { userId =>
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.muteUserString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.SpeakerOff),
richBehavior = Some(RichFeedbackBehaviorToggleMuteUser(userId)),
subprompt = None
)
}
}
}
}

View File

@ -0,0 +1,71 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class NotInterestedTopicFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidateFeatures: FeatureMap
): Option[FeedbackAction] = {
val isOutOfNetwork = !candidateFeatures.getOrElse(InNetworkFeature, true)
val validFollowedByUserIds =
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
val validLikedByUserIds =
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
.filter(
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
if (isOutOfNetwork && validLikedByUserIds.isEmpty && validFollowedByUserIds.isEmpty) {
val topicIdSocialContext = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
val topicContextFunctionalityType =
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
(topicIdSocialContext, topicContextFunctionalityType) match {
case (Some(topicId), Some(topicContextFunctionalityType))
if topicContextFunctionalityType == RecommendationTopicContextFunctionalityType ||
topicContextFunctionalityType == RecWithEducationTopicContextFunctionalityType =>
Some(
FeedbackAction(
feedbackType = RichBehavior,
prompt = None,
confirmation = None,
childFeedbackActions = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior =
Some(RichFeedbackBehaviorMarkNotInterestedTopic(topicId = topicId.toString)),
subprompt = None,
encodedFeedbackRequest = None
)
)
case _ => None
}
} else {
None
}
}
}

View File

@ -0,0 +1,55 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.{thriftscala => tlst}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class NotRelevantChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[ChildFeedbackAction] = {
val prompt = stringCenter.prepare(externalStrings.notRelevantString)
val confirmation = stringCenter.prepare(externalStrings.notRelevantConfirmationString)
val feedbackMetadata = FeedbackMetadata(
engagementType = None,
entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),
ttl = Some(FeedbackUtil.FeedbackTtl))
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tlst.FeedbackType.NotRelevant,
feedbackMetadata = feedbackMetadata,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
Some(
ChildFeedbackAction(
feedbackType = NotRelevant,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior = None,
subprompt = None
)
)
}
}

View File

@ -0,0 +1,76 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Use '@A received a reply' as social context when the root Tweet is in network and the focal tweet is OON.
*
* This function should only be called for the root Tweet of convo modules. This is enforced by
* [[HomeTweetSocialContextBuilder]].
*/
@Singleton
case class ReceivedReplySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val receivedReplyString = externalStrings.socialContextReceivedReply
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// If these values are missing default to not showing a received a reply banner
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
val inNetworkFocalTweet =
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(true)
if (inNetwork && !inNetworkFocalTweet) {
val authorIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
val authorNameOpt = authorIdOpt.flatMap(realNames.get)
(authorIdOpt, authorNameOpt) match {
case (Some(authorId), Some(authorName)) =>
Some(
GeneralContext(
contextType = ConversationGeneralContextType,
text = stringCenter
.prepare(receivedReplyString, placeholders = Map("user1" -> authorName)),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
)
)
)
)
case _ => None
}
} else {
None
}
}
}

View File

@ -0,0 +1,38 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class ReportTweetChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidate: TweetCandidate
): Option[ChildFeedbackAction] = {
Some(
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(stringCenter.prepare(externalStrings.reportTweetString)),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.Flag),
richBehavior = Some(RichFeedbackBehaviorReportTweet(candidate.id)),
subprompt = None
)
)
}
}

View File

@ -0,0 +1,39 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.service.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class RetweeterChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
if (isRetweet) {
candidateFeatures.getOrElse(AuthorIdFeature, None).flatMap { retweeterId =>
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
userId = retweeterId,
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
promptExternalString = externalStrings.showFewerRetweetsString,
confirmationExternalString = externalStrings.showFewerRetweetsConfirmationString,
engagementType = t.FeedbackEngagementType.Retweet,
stringCenter = stringCenter,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
}
} else None
}
}

View File

@ -0,0 +1,42 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class TopicSocialContextBuilder @Inject() ()
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
if (!inNetwork) {
val topicIdSocialContextOpt = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
val topicContextFunctionalityTypeOpt =
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
(topicIdSocialContextOpt, topicContextFunctionalityTypeOpt) match {
case (Some(topicId), Some(topicContextFunctionalityType)) =>
Some(
TopicContext(
topicId = topicId.toString,
functionalityType = Some(topicContextFunctionalityType)
))
case _ => None
}
} else {
None
}
}
}

View File

@ -0,0 +1,57 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class UnfollowUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
val userIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
if (isInNetwork) {
userIdOpt.flatMap { userId =>
val screenNamesMap =
candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.unfollowUserString,
Map("username" -> userScreenName)
)
val confirmation = stringCenter.prepare(
externalStrings.unfollowUserConfirmationString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.Unfollow),
richBehavior = Some(RichFeedbackBehaviorToggleFollowUser(userId)),
subprompt = None
)
}
}
} else None
}
}

View File

@ -0,0 +1,50 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Renders a fixed 'You Might Like' string above all OON Tweets.
*/
@Singleton
case class YouMightLikeSocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val youMightLikeString = externalStrings.socialContextYouMightLikeString
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
if (!isInNetwork && !isRetweet) {
Some(
GeneralContext(
contextType = SparkleGeneralContextType,
text = stringCenter.prepare(youMightLikeString),
url = None,
contextImageUrls = None,
landingUrl = None
))
} else {
None
}
}
}

View File

@ -0,0 +1,23 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"3rdparty/jvm/com/twitter/bijection:scrooge",
"finagle/finagle-core/src/main",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
"src/scala/com/twitter/suggests/controller_data",
"src/thrift/com/twitter/suggests/controller_data:controller_data-scala",
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
],
)

View File

@ -0,0 +1,44 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.home_mixer.model.HomeFeatures.EntityTokenFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
/**
* Sets the [[ClientEventInfo]] with the `component` field set to the Suggest Type assigned to each candidate
*/
case class HomeClientEventInfoBuilder[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](
detailsBuilder: Option[BaseClientEventDetailsBuilder[Query, Candidate]] = None)
extends BaseClientEventInfoBuilder[Query, Candidate] {
override def apply(
query: Query,
candidate: Candidate,
candidateFeatures: FeatureMap,
element: Option[String]
): Option[ClientEventInfo] = {
val suggestType = candidateFeatures
.getOrElse(SuggestTypeFeature, None)
.getOrElse(throw new UnsupportedOperationException(s"No SuggestType was set"))
Some(
ClientEventInfo(
component = InjectionScribeUtil.scribeComponent(suggestType),
element = element,
details = detailsBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),
action = None,
/**
* A backend entity encoded by the Client Entities Encoding Library.
* Placeholder string for now
*/
entityToken = candidateFeatures.getOrElse(EntityTokenFeature, None)
)
)
}
}

View File

@ -0,0 +1,30 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature
import com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleConversationMetadata
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata
import com.twitter.product_mixer.core.pipeline.PipelineQuery
case class HomeConversationModuleMetadataBuilder[
-Query <: PipelineQuery,
-Candidate <: BaseTweetCandidate
]() extends BaseModuleMetadataBuilder[Query, Candidate] {
override def apply(
query: Query,
candidates: Seq[CandidateWithFeatures[Candidate]]
): ModuleMetadata = ModuleMetadata(
adsMetadata = None,
conversationMetadata = Some(
ModuleConversationMetadata(
allTweetIds = Some((candidates.last.candidate.id +:
candidates.last.features.getOrElse(AncestorsFeature, Seq.empty).map(_.tweetId)).reverse),
socialContext = None,
enableDeduplication = Some(true)
)),
gridCarouselMetadata = None
)
}

View File

@ -0,0 +1,33 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelineservice.suggests.{thriftscala => st}
object ListClientEventDetailsBuilder
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
override def apply(
query: PipelineQuery,
candidate: UniversalNoun[Any],
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = Some(st.SuggestType.OrganicListTweet.name),
controllerData = None,
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

View File

@ -0,0 +1,30 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.AlwaysInclude
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtInstructionBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction
import com.twitter.product_mixer.core.model.marshalling.response.urt.Cover
import com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert
import com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry
import com.twitter.product_mixer.core.pipeline.PipelineQuery
case class AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder[Query <: PipelineQuery](
override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)
extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {
override def build(
query: Query,
entries: Seq[TimelineEntry]
): Seq[AddEntriesTimelineInstruction] = {
if (includeInstruction(query, entries)) {
val entriesToAdd = entries
.filterNot(_.isInstanceOf[ShowAlert])
.filterNot(_.isInstanceOf[Cover])
.filter(_.entryIdToReplace.isEmpty)
if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))
else Seq.empty
} else
Seq.empty
}
}

View File

@ -0,0 +1,12 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder",
"src/thrift/com/twitter/timelines/service:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
],
)

View File

@ -0,0 +1,51 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder
import com.twitter.product_mixer.component_library.model.candidate.UserCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.ExternalStringRegistry
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.service.{thriftscala => tl}
import com.twitter.timelines.util.FeedbackRequestSerializer
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
import com.twitter.timelineservice.thriftscala.FeedbackType
object HomeWhoToFollowFeedbackActionInfoBuilder {
private val FeedbackMetadata = tl.FeedbackMetadata(
injectionType = Some(SuggestType.WhoToFollow),
engagementType = None,
entityIds = Seq.empty,
ttlMs = None
)
private val FeedbackRequest =
tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)
private val EncodedFeedbackRequest =
FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))
}
@Singleton
case class HomeWhoToFollowFeedbackActionInfoBuilder @Inject() (
@ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry],
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {
private val whoToFollowFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(
externalStringRegistry = externalStringRegistryProvider.get(),
stringCenter = stringCenterProvider.get(),
encodedFeedbackRequest = Some(HomeWhoToFollowFeedbackActionInfoBuilder.EncodedFeedbackRequest)
)
override def apply(
query: PipelineQuery,
candidate: UserCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackActionInfo] =
whoToFollowFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)
}

View File

@ -0,0 +1,56 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}
import com.twitter.tweetconvosvc.{thriftscala => tcs}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class AncestorFeatureHydrator @Inject() (
conversationServiceClient: tcs.ConversationService.MethodPerEndpoint)
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Ancestor")
override val features: Set[Feature[_, _]] = Set(AncestorsFeature)
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val ancestorsRequest = tcs.GetAncestorsRequest(Seq(candidate.id))
Stitch.callFuture(conversationServiceClient.getAncestors(ancestorsRequest)).map {
getAncestorsResponse =>
val ancestors = getAncestorsResponse.ancestors.headOption
.collect {
case tcs.TweetAncestorsResult.TweetAncestors(ancestorsResult)
if ancestorsResult.nonEmpty =>
ancestorsResult.head.ancestors ++ getTruncatedRootTweet(ancestorsResult.head)
}.getOrElse(Seq.empty)
FeatureMapBuilder().add(AncestorsFeature, ancestors).build()
}
}
private def getTruncatedRootTweet(
ancestors: ta.TweetAncestors,
): Option[ta.TweetAncestor] = {
ancestors.conversationRootAuthorId.collect {
case rootAuthorId
if ancestors.state == ta.ReplyState.Partial &&
ancestors.ancestors.last.tweetId != ancestors.conversationId =>
ta.TweetAncestor(ancestors.conversationId, rootAuthorId)
}
}
}

View File

@ -0,0 +1,95 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features.AuthorFeaturesAdapter
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.AuthorFeatureRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.timelines.author_features.v1.{thriftjava => af}
import com.twitter.util.Future
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object AuthorFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class AuthorFeatureHydrator @Inject() (
@Named(AuthorFeatureRepository) client: KeyValueRepository[Seq[Long], Long, af.AuthorFeatures],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("AuthorFeature")
override val features: Set[Feature[_, _]] = Set(AuthorFeature)
override val statScope: String = identifier.toString
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val possiblyAuthorIds = extractKeys(candidates)
val authorIds = possiblyAuthorIds.flatten
val response: Future[KeyValueResult[Long, af.AuthorFeatures]] =
if (authorIds.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(authorIds)
}
response.map { result =>
possiblyAuthorIds.map { possiblyAuthorId =>
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
val transformedValue = postTransformer(value)
FeatureMapBuilder()
.add(AuthorFeature, transformedValue)
.build()
}
}
}
}
private def postTransformer(authorFeatures: Try[Option[af.AuthorFeatures]]): Try[DataRecord] = {
authorFeatures.map { features =>
AuthorFeaturesAdapter.adaptToDataRecords(features).asScala.head
}
}
private def extractKeys(
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Seq[Option[Long]] = {
candidates.map { candidate =>
candidate.features
.getTry(AuthorIdFeature)
.toOption
.flatten
}
}
}

View File

@ -0,0 +1,103 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"3rdparty/jvm/com/twitter/storehaus:core",
"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi",
"configapi/configapi-decider",
"finatra/inject/inject-core/src/main/scala",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/author_features",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/earlybird",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/inferred_topic",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content",
"joinkey/src/main/scala/com/twitter/joinkey/context",
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timelines_impression_store",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/topics",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util",
"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common",
"representation-scorer/server/src/main/thrift:thrift-scala",
"servo/repo/src/main/scala",
"snowflake/src/main/scala/com/twitter/snowflake/id",
"src/java/com/twitter/ml/api/constant",
"src/java/com/twitter/search/common/util/lang",
"src/scala/com/twitter/ml/api/util",
"src/scala/com/twitter/timelines/prediction/adapters/real_graph",
"src/scala/com/twitter/timelines/prediction/adapters/realtime_interaction_graph",
"src/scala/com/twitter/timelines/prediction/adapters/twistly",
"src/scala/com/twitter/timelines/prediction/adapters/two_hop_features",
"src/scala/com/twitter/timelines/prediction/common/util",
"src/scala/com/twitter/timelines/prediction/features/common",
"src/scala/com/twitter/timelines/prediction/features/realtime_interaction_graph",
"src/scala/com/twitter/timelines/prediction/features/recap",
"src/scala/com/twitter/timelines/prediction/features/time_features",
"src/thrift/com/twitter/gizmoduck:thrift-scala",
"src/thrift/com/twitter/ml/api:data-java",
"src/thrift/com/twitter/ml/api:embedding-java",
"src/thrift/com/twitter/onboarding/relevance/features:features-java",
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
"src/thrift/com/twitter/search:earlybird-scala",
"src/thrift/com/twitter/search/common:constants-java",
"src/thrift/com/twitter/socialgraph:thrift-scala",
"src/thrift/com/twitter/spam/rtf:safety-result-scala",
"src/thrift/com/twitter/timelineranker:thrift-scala",
"src/thrift/com/twitter/timelines/author_features:thrift-java",
"src/thrift/com/twitter/timelines/conversation_features:conversation_features-scala",
"src/thrift/com/twitter/timelines/impression:thrift-scala",
"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala",
"src/thrift/com/twitter/timelines/real_graph:real_graph-scala",
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
"src/thrift/com/twitter/topic_recos:topic_recos-thrift-java",
"src/thrift/com/twitter/tweetypie:service-scala",
"src/thrift/com/twitter/tweetypie:tweet-scala",
"src/thrift/com/twitter/user_session_store:thrift-java",
"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala",
"src/thrift/com/twitter/wtf/real_time_interaction_graph:wtf-real_time_interaction_graph-thrift-java",
"stitch/stitch-core",
"stitch/stitch-gizmoduck",
"stitch/stitch-socialgraph",
"stitch/stitch-timelineservice",
"stitch/stitch-tweetypie",
"strato/config/columns/topic-signals/tsp",
"strato/config/columns/topic-signals/tsp:tsp-strato-client",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/feedback",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/persistence",
"timelines/src/main/scala/com/twitter/timelines/clients/strato/twistly",
"timelines/src/main/scala/com/twitter/timelines/clients/user_tweet_entity_graph",
"timelines/src/main/scala/com/twitter/timelines/impressionstore/impressionbloomfilter",
"timelines/src/main/scala/com/twitter/timelines/impressionstore/store",
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
"topic-social-proof/server/src/main/thrift:thrift-scala",
"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting",
"tweetconvosvc/thrift/src/main/thrift:thrift-scala",
"twitter-config/yaml",
"user_session_store/src/main/scala/com/twitter/user_session_store",
"util/util-core",
],
exports = [
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
],
)

View File

@ -0,0 +1,45 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.timelinemixer.clients.manhattan.InjectionHistoryClient
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelinemixer.clients.manhattan.DismissInfo
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
import javax.inject.Inject
import javax.inject.Singleton
object DismissInfoQueryFeatureHydrator {
val DismissInfoSuggestTypes = Seq(SuggestType.WhoToFollow)
}
@Singleton
case class DismissInfoQueryFeatureHydrator @Inject() (
dismissInfoClient: InjectionHistoryClient)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("DismissInfo")
override val features: Set[Feature[_, _]] = Set(DismissInfoFeature)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] =
Stitch.callFuture {
dismissInfoClient
.readDismissInfoEntries(
query.getRequiredUserId,
DismissInfoQueryFeatureHydrator.DismissInfoSuggestTypes).map { response =>
val dismissInfoMap = response.mapValues(DismissInfo.fromThrift)
FeatureMapBuilder().add(DismissInfoFeature, dismissInfoMap).build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8, 50, 60, 60)
)
}

View File

@ -0,0 +1,129 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.earlybird.EarlybirdAdapter
import com.twitter.home_mixer.model.HomeFeatures.DeviceLanguageFeature
import com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.home_mixer.util.earlybird.EarlybirdResponseUtil
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.earlybird.{thriftscala => eb}
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Return
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object EarlybirdDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class EarlybirdFeatureHydrator @Inject() (
@Named(EarlybirdRepository) client: KeyValueRepository[
(Seq[Long], Long),
Long,
eb.ThriftSearchResult
],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Earlybird")
override val features: Set[Feature[_, _]] =
Set(EarlybirdDataRecordFeature, EarlybirdFeature, TweetUrlsFeature)
override val statScope: String = identifier.toString
private val scopedStatsReceiver = statsReceiver.scope(statScope)
private val originalKeyFoundCounter = scopedStatsReceiver.counter("originalKey/found")
private val originalKeyLossCounter = scopedStatsReceiver.counter("originalKey/loss")
private val ebFeaturesNotExistPredicate: CandidateWithFeatures[TweetCandidate] => Boolean =
candidate => candidate.features.getOrElse(EarlybirdFeature, None).isEmpty
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val candidatesToHydrate = candidates.filter { candidate =>
val isEmpty = ebFeaturesNotExistPredicate(candidate)
if (isEmpty) originalKeyLossCounter.incr() else originalKeyFoundCounter.incr()
isEmpty
}
Stitch
.callFuture(client((candidatesToHydrate.map(_.candidate.id), query.getRequiredUserId)))
.map(handleResponse(query, candidates, _))
}
private def handleResponse(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]],
results: KeyValueResult[Long, eb.ThriftSearchResult]
): Seq[FeatureMap] = {
val queryFeatureMap = query.features.getOrElse(FeatureMap.empty)
val userLanguages = queryFeatureMap.getOrElse(UserLanguagesFeature, Seq.empty)
val uiLanguageCode = queryFeatureMap.getOrElse(DeviceLanguageFeature, None)
val screenName = queryFeatureMap.getOrElse(UserScreenNameFeature, None)
val searchResults = candidates
.filter(ebFeaturesNotExistPredicate).map { candidate =>
observedGet(Some(candidate.candidate.id), results)
}.collect {
case Return(Some(value)) => value
}
val tweetIdToEbFeatures = EarlybirdResponseUtil.getOONTweetThriftFeaturesByTweetId(
searcherUserId = query.getRequiredUserId,
screenName = screenName,
userLanguages = userLanguages,
uiLanguageCode = uiLanguageCode,
searchResults = searchResults
)
candidates.map { candidate =>
val hydratedEbFeatures = tweetIdToEbFeatures.get(candidate.candidate.id)
val earlybirdFeatures =
if (hydratedEbFeatures.nonEmpty) hydratedEbFeatures
else candidate.features.getOrElse(EarlybirdFeature, None)
val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)
val sourceTweetEbFeatures =
candidate.features.getOrElse(SourceTweetEarlybirdFeature, None)
val originalTweetEbFeatures =
if (candidateIsRetweet && sourceTweetEbFeatures.nonEmpty)
sourceTweetEbFeatures
else earlybirdFeatures
val earlybirdDataRecord =
EarlybirdAdapter.adaptToDataRecords(originalTweetEbFeatures).asScala.head
FeatureMapBuilder()
.add(EarlybirdFeature, earlybirdFeatures)
.add(EarlybirdDataRecordFeature, earlybirdDataRecord)
.add(TweetUrlsFeature, earlybirdFeatures.flatMap(_.urlsList).getOrElse(Seq.empty))
.build()
}
}
}

View File

@ -0,0 +1,38 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableFeedbackFatigueParam
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelinemixer.clients.feedback.FeedbackHistoryManhattanClient
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class FeedbackHistoryQueryFeatureHydrator @Inject() (
feedbackHistoryClient: FeedbackHistoryManhattanClient)
extends QueryFeatureHydrator[PipelineQuery]
with Conditionally[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FeedbackHistory")
override val features: Set[Feature[_, _]] = Set(FeedbackHistoryFeature)
override def onlyIf(query: PipelineQuery): Boolean =
query.params(EnableFeedbackFatigueParam)
override def hydrate(
query: PipelineQuery
): Stitch[FeatureMap] =
Stitch
.callFuture(feedbackHistoryClient.get(query.getRequiredUserId))
.map { feedbackHistory =>
FeatureMapBuilder().add(FeedbackHistoryFeature, feedbackHistory).build()
}
}

View File

@ -0,0 +1,84 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Singleton
/**
* Social context for convo modules is hydrated on the root Tweet but needs info about the focal
* Tweet (e.g. author) to render the banner. This hydrator copies focal Tweet data into the root.
*/
@Singleton
class FocalTweetFeatureHydrator @Inject() ()
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FocalTweet")
override val features: Set[Feature[_, _]] = Set(
FocalTweetAuthorIdFeature,
FocalTweetInNetworkFeature,
FocalTweetRealNamesFeature,
FocalTweetScreenNamesFeature
)
private val DefaultFeatureMap = FeatureMapBuilder()
.add(FocalTweetAuthorIdFeature, None)
.add(FocalTweetInNetworkFeature, None)
.add(FocalTweetRealNamesFeature, None)
.add(FocalTweetScreenNamesFeature, None)
.build()
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
// Build a map of all the focal tweets to their corresponding features
val focalTweetIdToFeatureMap = candidates.flatMap { candidate =>
val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)
if (focalTweetId.contains(candidate.candidate.id)) {
Some(candidate.candidate.id -> candidate.features)
} else None
}.toMap
val updatedFeatureMap = candidates.map { candidate =>
val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)
val conversationId = candidate.features.getOrElse(ConversationModuleIdFeature, None)
// Check if the candidate is a root tweet and ensure its focal tweet's features are available
if (conversationId.contains(candidate.candidate.id)
&& focalTweetId.exists(focalTweetIdToFeatureMap.contains)) {
val featureMap = focalTweetIdToFeatureMap.get(focalTweetId.get).get
FeatureMapBuilder()
.add(FocalTweetAuthorIdFeature, featureMap.getOrElse(AuthorIdFeature, None))
.add(FocalTweetInNetworkFeature, Some(featureMap.getOrElse(InNetworkFeature, true)))
.add(
FocalTweetRealNamesFeature,
Some(featureMap.getOrElse(RealNamesFeature, Map.empty[Long, String])))
.add(
FocalTweetScreenNamesFeature,
Some(featureMap.getOrElse(ScreenNamesFeature, Map.empty[Long, String])))
.build()
} else DefaultFeatureMap
}
Stitch.value(updatedFeatureMap)
}
}

View File

@ -0,0 +1,41 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.UserFollowedTopicsCountFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.component_library.candidate_source.topics.FollowedTopicsCandidateSource
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class FollowedTopicsQueryFeatureHydrator @Inject() (
followedTopicsCandidateSource: FollowedTopicsCandidateSource)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FollowedTopics")
override val features: Set[Feature[_, _]] = Set(UserFollowedTopicsCountFeature)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val request: StratoKeyView[Long, Unit] = StratoKeyView(query.getRequiredUserId, Unit)
followedTopicsCandidateSource(request)
.map { topics =>
FeatureMapBuilder().add(UserFollowedTopicsCountFeature, Some(topics.size)).build()
}.handle {
case _ => FeatureMapBuilder().add(UserFollowedTopicsCountFeature, None).build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9),
HomeMixerAlertConfig.BusinessHours.defaultLatencyAlert(1500.millis)
)
}

View File

@ -0,0 +1,58 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.gizmoduck.{thriftscala => gt}
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.AuthorIsBlueVerifiedFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableGizmoduckAuthorSafetyFeatureHydratorParam
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.stitch.gizmoduck.Gizmoduck
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class GizmoduckAuthorSafetyFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with Conditionally[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("GizmoduckAuthorSafety")
override val features: Set[Feature[_, _]] = Set(AuthorIsBlueVerifiedFeature)
override def onlyIf(query: PipelineQuery): Boolean =
query.params(EnableGizmoduckAuthorSafetyFeatureHydratorParam)
private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Safety)
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val authorIdOption = existingFeatures.getOrElse(AuthorIdFeature, None)
val blueVerifiedStitch = authorIdOption
.map { authorId =>
gizmoduck
.getUserById(
userId = authorId,
queryFields = queryFields
)
.map { _.safety.flatMap(_.isBlueVerified).getOrElse(false) }
}.getOrElse(Stitch.False)
blueVerifiedStitch.map { isBlueVerified =>
FeatureMapBuilder()
.add(AuthorIsBlueVerifiedFeature, isBlueVerified)
.build()
}
}
}

View File

@ -0,0 +1,50 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.gizmoduck.{thriftscala => gt}
import com.twitter.home_mixer.model.HomeFeatures.UserFollowingCountFeature
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
import com.twitter.home_mixer.model.HomeFeatures.UserTypeFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.stitch.gizmoduck.Gizmoduck
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class GizmoduckUserQueryFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("GizmoduckUser")
override val features: Set[Feature[_, _]] =
Set(UserFollowingCountFeature, UserTypeFeature, UserScreenNameFeature)
private val queryFields: Set[gt.QueryFields] =
Set(gt.QueryFields.Counts, gt.QueryFields.Safety, gt.QueryFields.Profile)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
gizmoduck
.getUserById(
userId = userId,
queryFields = queryFields,
context = gt.LookupContext(forUserId = Some(userId), includeSoftUsers = true))
.map { user =>
FeatureMapBuilder()
.add(UserFollowingCountFeature, user.counts.map(_.following.toInt))
.add(UserTypeFeature, Some(user.userType))
.add(UserScreenNameFeature, user.profile.map(_.screenName))
.build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)
)
}

View File

@ -0,0 +1,105 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.graph_feature_service.{thriftscala => gfs}
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.IsExtendedReplyFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.GraphTwoHopRepository
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.home_mixer.util.ReplyRetweetUtil
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.two_hop_features.TwoHopFeaturesAdapter
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object GraphTwoHopFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class GraphTwoHopFeatureHydrator @Inject() (
@Named(GraphTwoHopRepository) client: KeyValueRepository[(Seq[Long], Long), Long, Seq[
gfs.IntersectionValue
]],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("GraphTwoHop")
override val features: Set[Feature[_, _]] = Set(GraphTwoHopFeature, FollowedByUserIdsFeature)
override val statScope: String = identifier.toString
private val twoHopFeaturesAdapter = new TwoHopFeaturesAdapter
private val FollowFeatureType = gfs.FeatureType(gfs.EdgeType.Following, gfs.EdgeType.FollowedBy)
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
// Apply filters to in network candidates for ExtendedReplyAncestors and retweets.
// ExtendedReplyAncestors should also be in candidates. No filter for oon.
val (inNetworkCandidates, oonCandidates) = candidates.partition { candidate =>
candidate.features.getOrElse(InNetworkFeature, false)
}
val inNetworkReplyToAncestorTweet =
ReplyRetweetUtil.replyToAncestorTweetCandidatesMap(inNetworkCandidates)
val inNetworkExtendedReplyAncestors = inNetworkCandidates
.filter(_.features.getOrElse(IsExtendedReplyFeature, false)).flatMap { inNetworkCandidate =>
inNetworkReplyToAncestorTweet.get(inNetworkCandidate.candidate.id)
}.flatten
val inNetworkCandidatesToHydrate = inNetworkExtendedReplyAncestors ++
inNetworkCandidates.filter(_.features.getOrElse(IsRetweetFeature, false))
val candidatesToHydrate = (inNetworkCandidatesToHydrate ++ oonCandidates)
.flatMap(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features)).distinct
val response = Stitch.callFuture(client((candidatesToHydrate, query.getRequiredUserId)))
response.map { result =>
candidates.map { candidate =>
val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)
val value = observedGet(key = originalAuthorId, keyValueResult = result)
val transformedValue = postTransformer(value)
val followedByUserIds = value.toOption.flatMap(getFollowedByUserIds(_)).getOrElse(Seq.empty)
FeatureMapBuilder()
.add(GraphTwoHopFeature, transformedValue)
.add(FollowedByUserIdsFeature, followedByUserIds)
.build()
}
}
}
private def getFollowedByUserIds(input: Option[Seq[gfs.IntersectionValue]]): Option[Seq[Long]] =
input.map(_.filter(_.featureType == FollowFeatureType).flatMap(_.intersectionIds).flatten)
private def postTransformer(input: Try[Option[Seq[gfs.IntersectionValue]]]): Try[DataRecord] =
input.map(twoHopFeaturesAdapter.adaptToDataRecords(_).asScala.head)
}

View File

@ -0,0 +1,57 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature
import com.twitter.home_mixer.model.request.HasSeenTweetIds
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.impressionbloomfilter.{thriftscala => t}
import com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class ImpressionBloomFilterQueryFeatureHydrator[
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
bloomFilter: ImpressionBloomFilter)
extends QueryFeatureHydrator[Query] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
"ImpressionBloomFilter")
private val ImpressionBloomFilterTTL = 7.day
private val ImpressionBloomFilterFalsePositiveRate = 0.002
override val features: Set[Feature[_, _]] = Set(ImpressionBloomFilterFeature)
private val SurfaceArea = t.SurfaceArea.HomeTimeline
override def hydrate(query: Query): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
bloomFilter.getBloomFilterSeq(userId, SurfaceArea).map { bloomFilterSeq =>
val updatedBloomFilterSeq =
if (query.seenTweetIds.forall(_.isEmpty)) bloomFilterSeq
else {
bloomFilter.addElements(
userId = userId,
surfaceArea = SurfaceArea,
tweetIds = query.seenTweetIds.get,
bloomFilterEntrySeq = bloomFilterSeq,
timeToLive = ImpressionBloomFilterTTL,
falsePositiveRate = ImpressionBloomFilterFalsePositiveRate
)
}
FeatureMapBuilder().add(ImpressionBloomFilterFeature, updatedBloomFilterSeq).build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)
)
}

View File

@ -0,0 +1,68 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature
import com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature
import com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.user_session_store.ReadRequest
import com.twitter.user_session_store.ReadWriteUserSessionStore
import com.twitter.user_session_store.UserSessionDataset
import com.twitter.user_session_store.UserSessionDataset.UserSessionDataset
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class LastNonPollingTimeQueryFeatureHydrator @Inject() (
userSessionStore: ReadWriteUserSessionStore)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("LastNonPollingTime")
override val features: Set[Feature[_, _]] = Set(
FollowingLastNonPollingTimeFeature,
LastNonPollingTimeFeature,
NonPollingTimesFeature
)
private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.NonPollingTimes)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
userSessionStore
.read(ReadRequest(query.getRequiredUserId, datasets))
.map { userSession =>
val nonPollingTimestamps = userSession.flatMap(_.nonPollingTimestamps)
val lastNonPollingTime = nonPollingTimestamps
.flatMap(_.nonPollingTimestampsMs.headOption)
.map(Time.fromMilliseconds)
val followingLastNonPollingTime = nonPollingTimestamps
.flatMap(_.mostRecentHomeLatestNonPollingTimestampMs)
.map(Time.fromMilliseconds)
val nonPollingTimes = nonPollingTimestamps
.map(_.nonPollingTimestampsMs)
.getOrElse(Seq.empty)
FeatureMapBuilder()
.add(FollowingLastNonPollingTimeFeature, followingLastNonPollingTime)
.add(LastNonPollingTimeFeature, lastNonPollingTime)
.add(NonPollingTimesFeature, nonPollingTimes)
.build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)
)
}

View File

@ -0,0 +1,42 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.request.HasListId
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.socialgraph.{thriftscala => sg}
import com.twitter.stitch.Stitch
import com.twitter.stitch.socialgraph.SocialGraph
import javax.inject.Inject
import javax.inject.Singleton
case object ListMembersFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {
override val defaultValue: Seq[Long] = Seq.empty
}
@Singleton
class ListMembersQueryFeatureHydrator @Inject() (socialGraph: SocialGraph)
extends QueryFeatureHydrator[PipelineQuery with HasListId] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("ListMembers")
override val features: Set[Feature[_, _]] = Set(ListMembersFeature)
private val MaxRecentMembers = 10
override def hydrate(query: PipelineQuery with HasListId): Stitch[FeatureMap] = {
val request = sg.IdsRequest(
relationships = Seq(sg
.SrcRelationship(query.listId, sg.RelationshipType.ListHasMember, hasRelationship = true)),
pageRequest = Some(sg.PageRequest(selectAll = Some(true), count = Some(MaxRecentMembers)))
)
socialGraph.ids(request).map(_.ids).map { listMembers =>
FeatureMapBuilder().add(ListMembersFeature, listMembers).build()
}
}
}

View File

@ -0,0 +1,81 @@
package com.twitter.home_mixer
package functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.MetricCenterUserCountingFeatureRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.onboarding.relevance.features.{thriftjava => rf}
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Future
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object MetricCenterUserCountingFeature
extends Feature[TweetCandidate, Option[rf.MCUserCountingFeatures]]
@Singleton
class MetricCenterUserCountingFeatureHydrator @Inject() (
@Named(MetricCenterUserCountingFeatureRepository) client: KeyValueRepository[Seq[
Long
], Long, rf.MCUserCountingFeatures],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("MetricCenterUserCounting")
override val features: Set[Feature[_, _]] = Set(MetricCenterUserCountingFeature)
override val statScope: String = identifier.toString
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val possiblyAuthorIds = extractKeys(candidates)
val userIds = possiblyAuthorIds.flatten
val response: Future[KeyValueResult[Long, rf.MCUserCountingFeatures]] = if (userIds.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(userIds)
}
response.map { result =>
possiblyAuthorIds.map { possiblyAuthorId =>
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
FeatureMapBuilder()
.add(MetricCenterUserCountingFeature, value)
.build()
}
}
}
}
private def extractKeys(
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Seq[Option[Long]] = {
candidates.map { candidate =>
candidate.features
.getTry(AuthorIdFeature)
.toOption
.flatten
}
}
}

View File

@ -0,0 +1,97 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.gizmoduck.{thriftscala => gt}
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.stitch.gizmoduck.Gizmoduck
import com.twitter.util.Return
import javax.inject.Inject
import javax.inject.Singleton
protected case class ProfileNames(screenName: String, realName: String)
@Singleton
class NamesFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with Conditionally[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Names")
override val features: Set[Feature[_, _]] = Set(ScreenNamesFeature, RealNamesFeature)
override def onlyIf(query: PipelineQuery): Boolean = query.product match {
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
case _ => true
}
private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Profile)
/**
* The UI currently only ever displays the first 2 names in social context lines
* E.g. "User and 3 others like" or "UserA and UserB liked"
*/
private val MaxCountUsers = 2
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val candidateUserIdsMap = candidates.map { candidate =>
candidate.candidate.id ->
(candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++
candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers) ++
candidate.features.getOrElse(AuthorIdFeature, None) ++
candidate.features.getOrElse(SourceUserIdFeature, None)).distinct
}.toMap
val distinctUserIds = candidateUserIdsMap.values.flatten.toSeq.distinct
Stitch
.collectToTry(distinctUserIds.map(userId => gizmoduck.getUserById(userId, queryFields)))
.map { allUsers =>
val idToProfileNamesMap = allUsers.flatMap {
case Return(allUser) =>
allUser.profile
.map(profile => allUser.id -> ProfileNames(profile.screenName, profile.name))
case _ => None
}.toMap
val validUserIds = idToProfileNamesMap.keySet
candidates.map { candidate =>
val combinedMap = candidateUserIdsMap
.getOrElse(candidate.candidate.id, Nil)
.flatMap {
case userId if validUserIds.contains(userId) =>
idToProfileNamesMap.get(userId).map(profileNames => userId -> profileNames)
case _ => None
}
val perCandidateRealNameMap = combinedMap.map { case (k, v) => k -> v.realName }.toMap
val perCandidateScreenNameMap = combinedMap.map { case (k, v) => k -> v.screenName }.toMap
FeatureMapBuilder()
.add(ScreenNamesFeature, perCandidateScreenNameMap)
.add(RealNamesFeature, perCandidateRealNameMap)
.build()
}
}
}
}

View File

@ -0,0 +1,95 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
import com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.model.request.ForYouProduct
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient
import com.twitter.timelinemixer.clients.persistence.TimelineResponseV3
import com.twitter.timelines.util.client_info.ClientPlatform
import com.twitter.timelineservice.model.TimelineQuery
import com.twitter.timelineservice.model.core.TimelineKind
import com.twitter.timelineservice.model.rich.EntityIdType
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class PersistenceStoreQueryFeatureHydrator @Inject() (
timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3])
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("PersistenceStore")
private val WhoToFollowExcludedUserIdsLimit = 1000
private val ServedTweetIdsDuration = 1.hour
private val ServedTweetIdsLimit = 100
override val features: Set[Feature[_, _]] =
Set(ServedTweetIdsFeature, PersistenceEntriesFeature, WhoToFollowExcludedUserIdsFeature)
private val supportedClients = Seq(
ClientPlatform.IPhone,
ClientPlatform.IPad,
ClientPlatform.Mac,
ClientPlatform.Android,
ClientPlatform.Web,
ClientPlatform.RWeb,
ClientPlatform.TweetDeckGryphon
)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val timelineKind = query.product match {
case FollowingProduct => TimelineKind.homeLatest
case ForYouProduct => TimelineKind.home
case other => throw new UnsupportedOperationException(s"Unknown product: $other")
}
val timelineQuery = TimelineQuery(id = query.getRequiredUserId, kind = timelineKind)
Stitch.callFuture {
timelineResponseBatchesClient
.get(query = timelineQuery, clientPlatforms = supportedClients)
.map { timelineResponses =>
// Note that the WTF entries are not being scoped by ClientPlatform
val whoToFollowUserIds = timelineResponses
.flatMap { timelineResponse =>
timelineResponse.entries
.filter(_.entityIdType == EntityIdType.WhoToFollow)
.flatMap(_.itemIds.toSeq.flatMap(_.flatMap(_.userId)))
}.take(WhoToFollowExcludedUserIdsLimit)
val clientPlatform = ClientPlatform.fromQueryOptions(
clientAppId = query.clientContext.appId,
userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString))
val servedTweetIds = timelineResponses
.filter(_.clientPlatform == clientPlatform)
.filter(_.servedTime >= Time.now - ServedTweetIdsDuration)
.sortBy(-_.servedTime.inMilliseconds)
.flatMap(
_.entries.flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetIdsLimit))
FeatureMapBuilder()
.add(ServedTweetIdsFeature, servedTweetIds)
.add(PersistenceEntriesFeature, timelineResponses)
.add(WhoToFollowExcludedUserIdsFeature, whoToFollowUserIds)
.build()
}
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7, 50, 60, 60)
)
}

View File

@ -0,0 +1,71 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.stitch.timelineservice.TimelineService
import com.twitter.stitch.timelineservice.TimelineService.GetPerspectives
import com.twitter.timelineservice.thriftscala.PerspectiveType
import com.twitter.timelineservice.thriftscala.PerspectiveType.Favorited
import javax.inject.Inject
import javax.inject.Singleton
/**
* Filter out unlike edges from liked-by tweets
* Useful if the likes come from a cache and because UTEG does not fully remove unlike edges.
*/
@Singleton
class PerspectiveFilteredSocialContextFeatureHydrator @Inject() (timelineService: TimelineService)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("PerspectiveFilteredSocialContext")
override val features: Set[Feature[_, _]] = Set(PerspectiveFilteredLikedByUserIdsFeature)
private val MaxCountUsers = 10
private val favoritePerspectiveSet: Set[PerspectiveType] = Set(Favorited)
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val engagingUserIdtoTweetId = candidates.flatMap { candidate =>
candidate.features
.get(FavoritedByUserIdsFeature).take(MaxCountUsers)
.map(favoritedBy => favoritedBy -> candidate.candidate.id)
}
val queries = engagingUserIdtoTweetId.map {
case (userId, tweetId) =>
GetPerspectives.Query(userId = userId, tweetId = tweetId, types = favoritePerspectiveSet)
}
Stitch.collect(queries.map(timelineService.getPerspective)).map { perspectiveResults =>
val validUserIdTweetIds: Set[(Long, Long)] =
queries
.zip(perspectiveResults)
.collect { case (query, perspective) if perspective.favorited => query }
.map(query => (query.userId, query.tweetId))
.toSet
candidates.map { candidate =>
val perspectiveFilteredFavoritedByUserIds: Seq[Long] = candidate.features
.get(FavoritedByUserIdsFeature).take(MaxCountUsers)
.filter { userId => validUserIdTweetIds.contains((userId, candidate.candidate.id)) }
FeatureMapBuilder()
.add(PerspectiveFilteredLikedByUserIdsFeature, perspectiveFilteredFavoritedByUserIds)
.build()
}
}
}
}

View File

@ -0,0 +1,42 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphInNetworkScores
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.storehaus.ReadableStore
import com.twitter.wtf.candidate.{thriftscala => wtf}
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
@Singleton
case class RealGraphInNetworkScoresQueryFeatureHydrator @Inject() (
@Named(RealGraphInNetworkScores) store: ReadableStore[Long, Seq[wtf.Candidate]])
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphInNetworkScores")
override val features: Set[Feature[_, _]] = Set(RealGraphInNetworkScoresFeature)
private val RealGraphCandidateCount = 1000
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
Stitch.callFuture(store.get(query.getRequiredUserId)).map { realGraphFollowedUsers =>
val realGraphScoresFeatures = realGraphFollowedUsers
.getOrElse(Seq.empty)
.sortBy(-_.score)
.map(candidate => candidate.userId -> candidate.score)
.take(RealGraphCandidateCount)
.toMap
FeatureMapBuilder().add(RealGraphInNetworkScoresFeature, realGraphScoresFeatures).build()
}
}
}

View File

@ -0,0 +1,48 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphFeatureRepository
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.Repository
import com.twitter.timelines.real_graph.{thriftscala => rg}
import com.twitter.stitch.Stitch
import com.twitter.timelines.model.UserId
import com.twitter.timelines.real_graph.v1.thriftscala.RealGraphEdgeFeatures
import com.twitter.user_session_store.{thriftscala => uss}
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object RealGraphFeatures extends Feature[PipelineQuery, Option[Map[UserId, RealGraphEdgeFeatures]]]
@Singleton
class RealGraphQueryFeatureHydrator @Inject() (
@Named(RealGraphFeatureRepository) repository: Repository[Long, Option[uss.UserSession]])
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphFeatures")
override val features: Set[Feature[_, _]] = Set(RealGraphFeatures)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
Stitch.callFuture {
repository(query.getRequiredUserId).map { userSession =>
val realGraphFeaturesMap = userSession.flatMap { userSession =>
userSession.realGraphFeatures.collect {
case rg.RealGraphFeatures.V1(realGraphFeatures) =>
val edgeFeatures = realGraphFeatures.edgeFeatures ++ realGraphFeatures.oonEdgeFeatures
edgeFeatures.map { edge => edge.destId -> edge }.toMap
}
}
FeatureMapBuilder().add(RealGraphFeatures, realGraphFeaturesMap).build()
}
}
}
}

View File

@ -0,0 +1,123 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature
import com.twitter.home_mixer.util.MissingKeyException
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphFeaturesAdapter
import com.twitter.timelines.real_graph.v1.{thriftscala => v1}
import com.twitter.timelines.real_graph.{thriftscala => rg}
import com.twitter.util.Throw
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object RealGraphViewerAuthorDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
object RealGraphViewerAuthorsDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class RealGraphViewerAuthorFeatureHydrator @Inject() ()
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphViewerAuthor")
override val features: Set[Feature[_, _]] =
Set(RealGraphViewerAuthorDataRecordFeature, RealGraphViewerAuthorsDataRecordFeature)
private val realGraphEdgeFeaturesAdapter = new RealGraphFeaturesAdapter
private val realGraphEdgeFeaturesCombineAdapter =
new RealGraphEdgeFeaturesCombineAdapter(prefix = "authors.realgraph")
private val MissingKeyFeatureMap = FeatureMapBuilder()
.add(RealGraphViewerAuthorDataRecordFeature, Throw(MissingKeyException))
.add(RealGraphViewerAuthorsDataRecordFeature, Throw(MissingKeyException))
.build()
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val viewerId = query.getRequiredUserId
val realGraphFeatures = query.features
.flatMap(_.getOrElse(RealGraphFeatures, None))
.getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])
val result: FeatureMap = existingFeatures.getOrElse(AuthorIdFeature, None) match {
case Some(authorId) =>
val realGraphAuthorFeatures =
getRealGraphViewerAuthorFeatures(viewerId, authorId, realGraphFeatures)
val realGraphAuthorDataRecord = realGraphEdgeFeaturesAdapter
.adaptToDataRecords(realGraphAuthorFeatures).asScala.headOption.getOrElse(new DataRecord)
val combinedRealGraphFeaturesDataRecord = for {
inReplyToAuthorId <- existingFeatures.getOrElse(InReplyToUserIdFeature, None)
} yield {
val combinedRealGraphFeatures =
getCombinedRealGraphFeatures(Seq(authorId, inReplyToAuthorId), realGraphFeatures)
realGraphEdgeFeaturesCombineAdapter
.adaptToDataRecords(Some(combinedRealGraphFeatures)).asScala.headOption
.getOrElse(new DataRecord)
}
FeatureMapBuilder()
.add(RealGraphViewerAuthorDataRecordFeature, realGraphAuthorDataRecord)
.add(
RealGraphViewerAuthorsDataRecordFeature,
combinedRealGraphFeaturesDataRecord.getOrElse(new DataRecord))
.build()
case _ => MissingKeyFeatureMap
}
Stitch(result)
}
private def getRealGraphViewerAuthorFeatures(
viewerId: Long,
authorId: Long,
realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]
): rg.UserRealGraphFeatures = {
realGraphEdgeFeaturesMap.get(authorId) match {
case Some(realGraphEdgeFeatures) =>
rg.UserRealGraphFeatures(
srcId = viewerId,
features = rg.RealGraphFeatures.V1(
v1.RealGraphFeatures(edgeFeatures = Seq(realGraphEdgeFeatures))))
case _ =>
rg.UserRealGraphFeatures(
srcId = viewerId,
features = rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = Seq.empty)))
}
}
}
object RealGraphViewerAuthorFeatureHydrator {
def getCombinedRealGraphFeatures(
userIds: Seq[Long],
realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]
): rg.RealGraphFeatures = {
val edgeFeatures = userIds.flatMap(realGraphEdgeFeaturesMap.get)
rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = edgeFeatures))
}
}

View File

@ -0,0 +1,74 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter
import com.twitter.timelines.real_graph.v1.{thriftscala => v1}
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object RealGraphViewerRelatedUsersDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class RealGraphViewerRelatedUsersFeatureHydrator @Inject() ()
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphViewerRelatedUsers")
override val features: Set[Feature[_, _]] = Set(RealGraphViewerRelatedUsersDataRecordFeature)
private val RealGraphEdgeFeaturesCombineAdapter = new RealGraphEdgeFeaturesCombineAdapter
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val realGraphQueryFeatures = query.features
.flatMap(_.getOrElse(RealGraphFeatures, None))
.getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])
val allRelatedUserIds = getRelatedUserIds(existingFeatures)
val realGraphFeatures =
RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures(
allRelatedUserIds,
realGraphQueryFeatures)
val realGraphFeaturesDataRecord = RealGraphEdgeFeaturesCombineAdapter
.adaptToDataRecords(Some(realGraphFeatures)).asScala.headOption
.getOrElse(new DataRecord)
Stitch.value {
FeatureMapBuilder()
.add(RealGraphViewerRelatedUsersDataRecordFeature, realGraphFeaturesDataRecord)
.build()
}
}
private def getRelatedUserIds(features: FeatureMap): Seq[Long] = {
(CandidatesUtil.getEngagerUserIds(features) ++
features.getOrElse(AuthorIdFeature, None) ++
features.getOrElse(MentionUserIdFeature, Seq.empty) ++
features.getOrElse(SourceUserIdFeature, None) ++
features.getOrElse(DirectedAtUserIdFeature, None)).distinct
}
}

View File

@ -0,0 +1,64 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.realtime_interaction_graph.RealTimeInteractionGraphFeaturesAdapter
import com.twitter.timelines.prediction.features.realtime_interaction_graph.RealTimeInteractionGraphEdgeFeatures
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object RealTimeInteractionGraphEdgeFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class RealTimeInteractionGraphEdgeFeatureHydrator @Inject() ()
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
"RealTimeInteractionGraphEdge")
override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphEdgeFeature)
private val realTimeInteractionGraphFeaturesAdapter = new RealTimeInteractionGraphFeaturesAdapter
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val userVertex =
query.features.flatMap(_.getOrElse(RealTimeInteractionGraphUserVertexQueryFeature, None))
val realTimeInteractionGraphFeaturesMap =
userVertex.map(RealTimeInteractionGraphEdgeFeatures(_, Time.now))
Stitch.value {
candidates.map { candidate =>
val feature = candidate.features.getOrElse(AuthorIdFeature, None).flatMap { authorId =>
realTimeInteractionGraphFeaturesMap.flatMap(_.get(authorId))
}
FeatureMapBuilder()
.add(
RealTimeInteractionGraphEdgeFeature,
realTimeInteractionGraphFeaturesAdapter.adaptToDataRecords(feature).asScala.head)
.build()
}
}
}
}

View File

@ -0,0 +1,49 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.google.inject.name.Named
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexCache
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.cache.ReadCache
import com.twitter.stitch.Stitch
import com.twitter.wtf.real_time_interaction_graph.{thriftscala => ig}
import javax.inject.Inject
import javax.inject.Singleton
object RealTimeInteractionGraphUserVertexQueryFeature
extends Feature[PipelineQuery, Option[ig.UserVertex]]
@Singleton
class RealTimeInteractionGraphUserVertexQueryFeatureHydrator @Inject() (
@Named(RealTimeInteractionGraphUserVertexCache) client: ReadCache[Long, ig.UserVertex],
override val statsReceiver: StatsReceiver)
extends QueryFeatureHydrator[PipelineQuery]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealTimeInteractionGraphUserVertex")
override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphUserVertexQueryFeature)
override val statScope: String = identifier.toString
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
Stitch.callFuture(
client.get(Seq(userId)).map { results =>
val feature = observedGet(key = Some(userId), keyValueResult = results)
FeatureMapBuilder()
.add(RealTimeInteractionGraphUserVertexQueryFeature, feature)
.build()
}
)
}
}

View File

@ -0,0 +1,196 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.home_mixer.util.ReplyRetweetUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.common.features.thriftscala.ThriftTweetFeatures
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch
import com.twitter.timelines.conversation_features.v1.thriftscala.ConversationFeatures
import com.twitter.util.Duration
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
object InReplyToTweetHydratedEarlybirdFeature
extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]
/**
* The purpose of this hydrator is to
* 1) hydrate simple features into replies and their ancestor tweets
* 2) keep both the normal replies and ancestor source candidates, but hydrate into the candidates
* features useful for predicting the quality of the replies and source ancestor tweets.
*/
@Singleton
class ReplyFeatureHydrator @Inject() (statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("ReplyTweet")
override val features: Set[Feature[_, _]] = Set(
ConversationFeature,
InReplyToTweetHydratedEarlybirdFeature
)
private val DefaultFeatureMap = FeatureMapBuilder()
.add(ConversationFeature, None)
.add(InReplyToTweetHydratedEarlybirdFeature, None)
.build()
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
private val hydratedReplyCounter = scopedStatsReceiver.counter("hydratedReply")
private val hydratedAncestorCounter = scopedStatsReceiver.counter("hydratedAncestor")
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val replyToInReplyToTweetMap =
ReplyRetweetUtil.replyTweetIdToInReplyToTweetMap(candidates)
val candidatesWithRepliesHydrated = candidates.map { candidate =>
replyToInReplyToTweetMap
.get(candidate.candidate.id).map { inReplyToTweet =>
hydratedReplyCounter.incr()
hydratedReplyCandidate(candidate, inReplyToTweet)
}.getOrElse((candidate, None, None))
}
/**
* Update ancestor tweets with descendant replies and hydrate simple features from one of
* the descendants.
*/
val ancestorTweetToDescendantRepliesMap =
ReplyRetweetUtil.ancestorTweetIdToDescendantRepliesMap(candidates)
val candidatesWithRepliesAndAncestorTweetsHydrated = candidatesWithRepliesHydrated.map {
case (
maybeAncestorTweetCandidate,
updatedReplyConversationFeatures,
inReplyToTweetEarlyBirdFeature) =>
ancestorTweetToDescendantRepliesMap
.get(maybeAncestorTweetCandidate.candidate.id)
.map { descendantReplies =>
hydratedAncestorCounter.incr()
val (ancestorTweetCandidate, updatedConversationFeatures): (
CandidateWithFeatures[TweetCandidate],
Option[ConversationFeatures]
) =
hydrateAncestorTweetCandidate(
maybeAncestorTweetCandidate,
descendantReplies,
updatedReplyConversationFeatures)
(ancestorTweetCandidate, inReplyToTweetEarlyBirdFeature, updatedConversationFeatures)
}
.getOrElse(
(
maybeAncestorTweetCandidate,
inReplyToTweetEarlyBirdFeature,
updatedReplyConversationFeatures))
}
Stitch.value(
candidatesWithRepliesAndAncestorTweetsHydrated.map {
case (candidate, inReplyToTweetEarlyBirdFeature, updatedConversationFeatures) =>
FeatureMapBuilder()
.add(ConversationFeature, updatedConversationFeatures)
.add(InReplyToTweetHydratedEarlybirdFeature, inReplyToTweetEarlyBirdFeature)
.build()
case _ => DefaultFeatureMap
}
)
}
private def hydratedReplyCandidate(
replyCandidate: CandidateWithFeatures[TweetCandidate],
inReplyToTweetCandidate: CandidateWithFeatures[TweetCandidate]
): (
CandidateWithFeatures[TweetCandidate],
Option[ConversationFeatures],
Option[ThriftTweetFeatures]
) = {
val tweetedAfterInReplyToTweetInSecs =
(
originalTweetAgeFromSnowflake(inReplyToTweetCandidate),
originalTweetAgeFromSnowflake(replyCandidate)) match {
case (Some(inReplyToTweetAge), Some(replyTweetAge)) =>
Some((inReplyToTweetAge - replyTweetAge).inSeconds.toLong)
case _ => None
}
val existingConversationFeatures = Some(
replyCandidate.features
.getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))
val updatedConversationFeatures = existingConversationFeatures match {
case Some(v1) =>
Some(
v1.copy(
tweetedAfterInReplyToTweetInSecs = tweetedAfterInReplyToTweetInSecs,
isSelfReply = Some(
replyCandidate.features.getOrElse(
AuthorIdFeature,
None) == inReplyToTweetCandidate.features.getOrElse(AuthorIdFeature, None))
)
)
case _ => None
}
// Note: if inReplyToTweet is a retweet, we need to read early bird feature from the merged
// early bird feature field from RetweetSourceTweetFeatureHydrator class.
// But if inReplyToTweet is a reply, we return its early bird feature directly
val inReplyToTweetThriftTweetFeaturesOpt = {
if (inReplyToTweetCandidate.features.getOrElse(IsRetweetFeature, false)) {
inReplyToTweetCandidate.features.getOrElse(SourceTweetEarlybirdFeature, None)
} else {
inReplyToTweetCandidate.features.getOrElse(EarlybirdFeature, None)
}
}
(replyCandidate, updatedConversationFeatures, inReplyToTweetThriftTweetFeaturesOpt)
}
private def hydrateAncestorTweetCandidate(
ancestorTweetCandidate: CandidateWithFeatures[TweetCandidate],
descendantReplies: Seq[CandidateWithFeatures[TweetCandidate]],
updatedReplyConversationFeatures: Option[ConversationFeatures]
): (CandidateWithFeatures[TweetCandidate], Option[ConversationFeatures]) = {
// Ancestor could be a reply. For example, in thread: tweetA -> tweetB -> tweetC,
// tweetB is a reply and ancestor at the same time. Hence, tweetB's conversation feature
// will be updated by hydratedReplyCandidate and hydrateAncestorTweetCandidate functions.
val existingConversationFeatures =
if (updatedReplyConversationFeatures.nonEmpty)
updatedReplyConversationFeatures
else
Some(
ancestorTweetCandidate.features
.getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))
val updatedConversationFeatures = existingConversationFeatures match {
case Some(v1) =>
Some(
v1.copy(
hasDescendantReplyCandidate = Some(true),
hasInNetworkDescendantReply =
Some(descendantReplies.exists(_.features.getOrElse(InNetworkFeature, false)))
))
case _ => None
}
(ancestorTweetCandidate, updatedConversationFeatures)
}
private def originalTweetAgeFromSnowflake(
candidate: CandidateWithFeatures[TweetCandidate]
): Option[Duration] = {
SnowflakeId
.timeFromIdOpt(
candidate.features
.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id))
.map(Time.now - _)
}
}

View File

@ -0,0 +1,128 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.config.yaml.YamlMap
import com.twitter.finagle.tracing.Annotation.BinaryAnnotation
import com.twitter.finagle.tracing.ForwardAnnotation
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.home_mixer.model.request.DeviceContext.RequestContext
import com.twitter.home_mixer.model.request.HasDeviceContext
import com.twitter.home_mixer.param.HomeMixerInjectionNames.DDGStatsAuthors
import com.twitter.joinkey.context.RequestJoinKeyContext
import com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor
import com.twitter.product_mixer.core.pipeline.HasPipelineCursor
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.common.util.lang.ThriftLanguageUtil
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch
import java.util.UUID
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
@Singleton
class RequestQueryFeatureHydrator[
Query <: PipelineQuery with HasPipelineCursor[UrtOrderedCursor] with HasDeviceContext] @Inject() (
@Named(DDGStatsAuthors) ddgStatsAuthors: YamlMap)
extends QueryFeatureHydrator[Query] {
override val features: Set[Feature[_, _]] = Set(
AccountAgeFeature,
ClientIdFeature,
DDGStatsDemocratsFeature,
DDGStatsRepublicansFeature,
DDGStatsElonFeature,
DDGStatsVitsFeature,
DeviceLanguageFeature,
GetInitialFeature,
GetMiddleFeature,
GetNewerFeature,
GetOlderFeature,
GuestIdFeature,
HasDarkRequestFeature,
IsForegroundRequestFeature,
IsLaunchRequestFeature,
PollingFeature,
PullToRefreshFeature,
RequestJoinIdFeature,
ServedRequestIdFeature,
ViewerIdFeature
)
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Request")
private val DarkRequestAnnotation = "clnt/has_dark_request"
private val Democrats = "democrats"
private val Republicans = "republicans"
private val Elon = "elon"
private val Vits = "vits"
// Convert Language code to ISO 639-3 format
private def getLanguageISOFormatByCode(languageCode: String): String =
ThriftLanguageUtil.getLanguageCodeOf(ThriftLanguageUtil.getThriftLanguageOf(languageCode))
private def getRequestJoinId(servedRequestId: Long): Option[Long] =
Some(RequestJoinKeyContext.current.flatMap(_.requestJoinId).getOrElse(servedRequestId))
private def hasDarkRequest: Option[Boolean] = ForwardAnnotation.current
.getOrElse(Seq[BinaryAnnotation]())
.find(_.key == DarkRequestAnnotation)
.map(_.value.asInstanceOf[Boolean])
override def hydrate(query: Query): Stitch[FeatureMap] = {
val requestContext = query.deviceContext.flatMap(_.requestContextValue)
val servedRequestId = UUID.randomUUID.getMostSignificantBits
val featureMap = FeatureMapBuilder()
.add(AccountAgeFeature, query.getOptionalUserId.flatMap(SnowflakeId.timeFromIdOpt))
.add(ClientIdFeature, query.clientContext.appId)
/**
* These author ID lists are used purely for metrics collection. We track how often we are
* serving Tweets from these authors and how often their tweets are being impressed by users.
* This helps us validate in our A/B experimentation platform that we do not ship changes
* that negatively impacts one group over others.
*/
.add(DDGStatsDemocratsFeature, ddgStatsAuthors.longSeq(Democrats).toSet)
.add(DDGStatsRepublicansFeature, ddgStatsAuthors.longSeq(Republicans).toSet)
.add(DDGStatsVitsFeature, ddgStatsAuthors.longSeq(Vits).toSet)
.add(DDGStatsElonFeature, ddgStatsAuthors.longValue(Elon))
.add(DeviceLanguageFeature, query.getLanguageCode.map(getLanguageISOFormatByCode))
.add(
GetInitialFeature,
query.pipelineCursor.forall(cursor => cursor.id.isEmpty && cursor.gapBoundaryId.isEmpty))
.add(
GetMiddleFeature,
query.pipelineCursor.exists(cursor =>
cursor.id.isDefined && cursor.gapBoundaryId.isDefined &&
cursor.cursorType.contains(GapCursor)))
.add(
GetNewerFeature,
query.pipelineCursor.exists(cursor =>
cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&
cursor.cursorType.contains(TopCursor)))
.add(
GetOlderFeature,
query.pipelineCursor.exists(cursor =>
cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&
cursor.cursorType.contains(BottomCursor)))
.add(GuestIdFeature, query.clientContext.guestId)
.add(IsForegroundRequestFeature, requestContext.contains(RequestContext.Foreground))
.add(IsLaunchRequestFeature, requestContext.contains(RequestContext.Launch))
.add(PollingFeature, query.deviceContext.exists(_.isPolling.contains(true)))
.add(PullToRefreshFeature, requestContext.contains(RequestContext.PullToRefresh))
.add(ServedRequestIdFeature, Some(servedRequestId))
.add(RequestJoinIdFeature, getRequestJoinId(servedRequestId))
.add(HasDarkRequestFeature, hasDarkRequest)
.add(ViewerIdFeature, query.getRequiredUserId)
.build()
Stitch.value(featureMap)
}
}

View File

@ -0,0 +1,76 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.product_mixer.component_library.candidate_source.timeline_ranker.TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.common.features.thriftscala.ThriftTweetFeatures
import com.twitter.stitch.Stitch
import com.twitter.timelineranker.thriftscala.CandidateTweet
object SourceTweetEarlybirdFeature extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]
/**
* Feature Hydrator that bulk hydrates source tweets' features to retweet candidates
*/
object RetweetSourceTweetFeatureHydrator
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
"RetweetSourceTweet")
override val features: Set[Feature[_, _]] = Set(
SourceTweetEarlybirdFeature,
)
private val DefaultFeatureMap = FeatureMapBuilder()
.add(SourceTweetEarlybirdFeature, None)
.build()
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val sourceTweetsByTweetId: Option[Map[Long, CandidateTweet]] = {
query.features.map(
_.getOrElse(
TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature,
Map.empty[Long, CandidateTweet]))
}
/**
* Return DefaultFeatureMap (no-op to candidate) when it is unfeasible to hydrate the
* source tweet's feature to the current candidate: early bird does not return source
* tweets info / candidate is not a retweet / sourceTweetId is not found
*/
Stitch.value {
if (sourceTweetsByTweetId.exists(_.nonEmpty)) {
candidates.map { candidate =>
val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)
val sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None)
if (!candidateIsRetweet || sourceTweetId.isEmpty) {
DefaultFeatureMap
} else {
val sourceTweet = sourceTweetsByTweetId.flatMap(_.get(sourceTweetId.get))
if (sourceTweet.nonEmpty) {
val source = sourceTweet.get
FeatureMapBuilder()
.add(SourceTweetEarlybirdFeature, source.features)
.build()
} else {
DefaultFeatureMap
}
}
}
} else {
candidates.map(_ => DefaultFeatureMap)
}
}
}
}

View File

@ -0,0 +1,46 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.socialgraph.{thriftscala => sg}
import com.twitter.stitch.Stitch
import com.twitter.stitch.socialgraph.{SocialGraph => SocialGraphStitchClient}
import javax.inject.Inject
import javax.inject.Singleton
object SGSFollowedUsersFeature extends Feature[PipelineQuery, Seq[Long]]
@Singleton
case class SGSFollowedUsersQueryFeatureHydrator @Inject() (
socialGraphStitchClient: SocialGraphStitchClient)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("SGSFollowedUsers")
override val features: Set[Feature[_, _]] = Set(SGSFollowedUsersFeature)
private val SocialGraphLimit = 14999
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
val request = sg.IdsRequest(
relationships = Seq(
sg.SrcRelationship(userId, sg.RelationshipType.Following, hasRelationship = true),
sg.SrcRelationship(userId, sg.RelationshipType.Muting, hasRelationship = false)
),
pageRequest = Some(sg.PageRequest(count = Some(SocialGraphLimit)))
)
socialGraphStitchClient
.ids(request).map(_.ids)
.map { followedUsers =>
FeatureMapBuilder().add(SGSFollowedUsersFeature, followedUsers).build()
}
}
}

View File

@ -0,0 +1,105 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.socialgraph.{thriftscala => sg}
import com.twitter.stitch.Stitch
import com.twitter.stitch.socialgraph.SocialGraph
import javax.inject.Inject
import javax.inject.Singleton
/**
* This hydrator takes liked-by and followed-by user ids and checks via SGS that the viewer is
* following the engager, that the viewer is not blocking the engager, that the engager is not
* blocking the viewer, and that the viewer has not muted the engager.
*/
@Singleton
class SGSValidSocialContextFeatureHydrator @Inject() (
socialGraph: SocialGraph)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("SGSValidSocialContext")
override val features: Set[Feature[_, _]] = Set(
SGSValidFollowedByUserIdsFeature,
SGSValidLikedByUserIdsFeature
)
private val MaxCountUsers = 10
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val allSocialContextUserIds =
candidates.flatMap { candidate =>
candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++
candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)
}.distinct
getValidUserIds(query.getRequiredUserId, allSocialContextUserIds).map { validUserIds =>
candidates.map { candidate =>
val sgsFilteredLikedByUserIds =
candidate.features
.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers)
.filter(validUserIds.contains)
val sgsFilteredFollowedByUserIds =
candidate.features
.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)
.filter(validUserIds.contains)
FeatureMapBuilder()
.add(SGSValidFollowedByUserIdsFeature, sgsFilteredFollowedByUserIds)
.add(SGSValidLikedByUserIdsFeature, sgsFilteredLikedByUserIds)
.build()
}
}
}
private def getValidUserIds(
viewerId: Long,
socialProofUserIds: Seq[Long]
): Stitch[Seq[Long]] = {
if (socialProofUserIds.nonEmpty) {
val request = sg.IdsRequest(
relationships = Seq(
sg.SrcRelationship(
viewerId,
sg.RelationshipType.Following,
targets = Some(socialProofUserIds),
hasRelationship = true),
sg.SrcRelationship(
viewerId,
sg.RelationshipType.Blocking,
targets = Some(socialProofUserIds),
hasRelationship = false),
sg.SrcRelationship(
viewerId,
sg.RelationshipType.BlockedBy,
targets = Some(socialProofUserIds),
hasRelationship = false),
sg.SrcRelationship(
viewerId,
sg.RelationshipType.Muting,
targets = Some(socialProofUserIds),
hasRelationship = false)
),
pageRequest = Some(sg.PageRequest(selectAll = Some(true)))
)
socialGraph.ids(request).map(_.ids)
} else Stitch.Nil
}
}

View File

@ -0,0 +1,83 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.clients.strato.twistly.SimClustersRecentEngagementSimilarityClient
import com.twitter.timelines.configapi.decider.BooleanDeciderParam
import com.twitter.timelines.prediction.adapters.twistly.SimClustersRecentEngagementSimilarityFeaturesAdapter
import javax.inject.Inject
import javax.inject.Singleton
object SimClustersEngagementSimilarityFeature
extends DataRecordInAFeature[PipelineQuery]
with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class SimClustersEngagementSimilarityFeatureHydrator @Inject() (
simClustersEngagementSimilarityClient: SimClustersRecentEngagementSimilarityClient,
statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with Conditionally[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("SimClustersEngagementSimilarity")
override val features: Set[Feature[_, _]] = Set(SimClustersEngagementSimilarityFeature)
private val scopedStatsReceiver = statsReceiver.scope(identifier.toString)
private val hydratedCandidatesSizeStat = scopedStatsReceiver.stat("hydratedCandidatesSize")
private val simClustersRecentEngagementSimilarityFeaturesAdapter =
new SimClustersRecentEngagementSimilarityFeaturesAdapter
override def onlyIf(query: PipelineQuery): Boolean = {
val param: BooleanDeciderParam =
ScoredTweetsParam.EnableSimClustersSimilarityFeatureHydrationDeciderParam
query.params.apply(param)
}
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val tweetToCandidates = candidates.map(candidate => candidate.candidate.id -> candidate).toMap
val tweetIds = tweetToCandidates.keySet.toSeq
val userId = query.getRequiredUserId
val userTweetEdges = tweetIds.map(tweetId => (userId, tweetId))
val resultFuture = simClustersEngagementSimilarityClient
.getSimClustersRecentEngagementSimilarityScores(userTweetEdges).map {
simClustersRecentEngagementSimilarityScoresMap =>
hydratedCandidatesSizeStat.add(simClustersRecentEngagementSimilarityScoresMap.size)
candidates.map { candidate =>
val similarityFeatureOpt = simClustersRecentEngagementSimilarityScoresMap
.get(userId -> candidate.candidate.id).flatten
val dataRecordOpt = similarityFeatureOpt.map { similarityFeature =>
simClustersRecentEngagementSimilarityFeaturesAdapter
.adaptToDataRecords(similarityFeature)
.get(0)
}
FeatureMapBuilder()
.add(SimClustersEngagementSimilarityFeature, dataRecordOpt.getOrElse(new DataRecord))
.build()
}
}
Stitch.callFuture(resultFuture)
}
}

View File

@ -0,0 +1,67 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.socialgraph.{thriftscala => sg}
import com.twitter.stitch.Stitch
import com.twitter.stitch.socialgraph.{SocialGraph => SocialGraphStitchClient}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class SocialGraphServiceFeatureHydrator @Inject() (socialGraphStitchClient: SocialGraphStitchClient)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("SocialGraphService")
override val features: Set[Feature[_, _]] = Set(InNetworkFeature)
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val viewerId = query.getRequiredUserId
// We use authorId and not sourceAuthorId here so that retweets are defined as in network
val authorIds = candidates.map(_.features.getOrElse(AuthorIdFeature, None).getOrElse(0L))
val distinctNonSelfAuthorIds = authorIds.filter(_ != viewerId).distinct
val idsRequest = createIdsRequest(
userId = viewerId,
relationshipTypes = Set(sg.RelationshipType.Following),
targetIds = Some(distinctNonSelfAuthorIds)
)
socialGraphStitchClient
.ids(request = idsRequest, requestContext = None)
.map { idResult =>
authorIds.map { authorId =>
// Users cannot follow themselves but this is in network by definition
val isSelfTweet = authorId == viewerId
val inNetworkAuthorIds = idResult.ids.toSet
val isInNetwork = isSelfTweet || inNetworkAuthorIds.contains(authorId) || authorId == 0L
FeatureMapBuilder().add(InNetworkFeature, isInNetwork).build()
}
}
}
private def createIdsRequest(
userId: Long,
relationshipTypes: Set[sg.RelationshipType],
targetIds: Option[Seq[Long]] = None
): sg.IdsRequest = sg.IdsRequest(
relationshipTypes.map { relationshipType =>
sg.SrcRelationship(userId, relationshipType, targets = targetIds)
}.toSeq,
Some(sg.PageRequest(selectAll = Some(true)))
)
}

View File

@ -0,0 +1,162 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.contentrecommender.{thriftscala => cr}
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.inferred_topic.InferredTopicAdapter
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
import com.twitter.home_mixer.model.HomeFeatures.TSPMetricTagFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.strato.generated.client.topic_signals.tsp.TopicSocialProofClientColumn
import com.twitter.timelineservice.suggests.logging.candidate_tweet_source_id.{thriftscala => sid}
import com.twitter.topiclisting.TopicListingViewerContext
import com.twitter.tsp.{thriftscala => tsp}
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object TSPInferredTopicFeature extends Feature[TweetCandidate, Map[Long, Double]]
object TSPInferredTopicDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class TSPInferredTopicFeatureHydrator @Inject() (
topicSocialProofClientColumn: TopicSocialProofClientColumn,
statsReceiver: StatsReceiver,
) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TSPInferredTopic")
override val features: Set[Feature[_, _]] =
Set(
TSPInferredTopicFeature,
TSPInferredTopicDataRecordFeature,
TopicIdSocialContextFeature,
TopicContextFunctionalityTypeFeature)
private val topK = 3
private val sourcesToSetSocialProof: Set[sid.CandidateTweetSourceId] = Set(
sid.CandidateTweetSourceId.Simcluster,
sid.CandidateTweetSourceId.CroonTweet
)
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
private val keyFoundCounter = scopedStatsReceiver.counter("key/found")
private val keyLossCounter = scopedStatsReceiver.counter("key/loss")
private val requestFailCounter = scopedStatsReceiver.counter("request/fail")
private val DefaultFeatureMap = FeatureMapBuilder()
.add(TSPInferredTopicFeature, Map.empty[Long, Double])
.add(TSPInferredTopicDataRecordFeature, new DataRecord())
.add(TopicIdSocialContextFeature, None)
.add(TopicContextFunctionalityTypeFeature, None)
.build()
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val tags = candidates.collect {
case candidate if candidate.features.getTry(TSPMetricTagFeature).isReturn =>
candidate.candidate.id -> candidate.features
.getOrElse(TSPMetricTagFeature, Set.empty[tsp.MetricTag])
}.toMap
val topicSocialProofRequest =
tsp.TopicSocialProofRequest(
userId = query.getRequiredUserId,
tweetIds = candidates.map(_.candidate.id).toSet,
displayLocation = cr.DisplayLocation.HomeTimeline,
topicListingSetting = tsp.TopicListingSetting.Followable,
context = TopicListingViewerContext.fromClientContext(query.clientContext).toThrift,
bypassModes = None,
// Only CRMixer source has this data. Convert the CRMixer metric tag to tsp metric tag.
tags = if (tags.isEmpty) None else Some(tags)
)
topicSocialProofClientColumn.fetcher
.fetch(topicSocialProofRequest)
.map(_.v)
.map {
case Some(response) =>
candidates.map { candidate =>
val topicWithScores = response.socialProofs.getOrElse(candidate.candidate.id, Seq.empty)
if (topicWithScores.nonEmpty) {
keyFoundCounter.incr()
val (socialProofId, socialProofFunctionalityType) =
if (candidate.features
.getOrElse(CandidateSourceIdFeature, None)
.exists(sourcesToSetSocialProof.contains)) {
getSocialProof(topicWithScores)
} else {
(None, None)
}
val inferredTopicFeatures = convertTopicWithScores(topicWithScores)
val inferredTopicDataRecord =
InferredTopicAdapter.adaptToDataRecords(inferredTopicFeatures).asScala.head
FeatureMapBuilder()
.add(TSPInferredTopicFeature, inferredTopicFeatures)
.add(TSPInferredTopicDataRecordFeature, inferredTopicDataRecord)
.add(TopicIdSocialContextFeature, socialProofId)
.add(TopicContextFunctionalityTypeFeature, socialProofFunctionalityType)
.build()
} else {
keyLossCounter.incr()
DefaultFeatureMap
}
}
case _ =>
requestFailCounter.incr()
candidates.map { _ =>
DefaultFeatureMap
}
}
}
private def getSocialProof(
topicWithScores: Seq[tsp.TopicWithScore]
): (Option[Long], Option[TopicContextFunctionalityType]) = {
val followingTopicId = topicWithScores
.collectFirst {
case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.Following)) =>
topicId
}
if (followingTopicId.nonEmpty) {
return (followingTopicId, Some(BasicTopicContextFunctionalityType))
}
val implicitFollowingId = topicWithScores.collectFirst {
case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.ImplicitFollow)) =>
topicId
}
if (implicitFollowingId.nonEmpty) {
return (implicitFollowingId, Some(RecommendationTopicContextFunctionalityType))
}
(None, None)
}
private def convertTopicWithScores(
topicWithScores: Seq[tsp.TopicWithScore],
): Map[Long, Double] = {
topicWithScores.sortBy(-_.score).take(topK).map(a => (a.topicId, a.score)).toMap
}
}

View File

@ -0,0 +1,251 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature
import com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.ml.api.DataRecord
import com.twitter.ml.api.RichDataRecord
import com.twitter.ml.api.util.FDsl._
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.common.features.{thriftscala => sc}
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.features.time_features.AccountAgeInterval
import com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures._
import com.twitter.timelines.prediction.features.time_features.TimeFeatures
import com.twitter.util.Duration
import scala.collection.Searching._
object TimeFeaturesDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
object TimeFeaturesHydrator extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TimeFeatures")
override val features: Set[Feature[_, _]] = Set(TimeFeaturesDataRecordFeature)
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
Stitch.value {
val richDataRecord = new RichDataRecord()
setTimeFeatures(richDataRecord, candidate, existingFeatures, query)
FeatureMapBuilder()
.add(TimeFeaturesDataRecordFeature, richDataRecord.getRecord)
.build()
}
}
private def setTimeFeatures(
richDataRecord: RichDataRecord,
candidate: TweetCandidate,
existingFeatures: FeatureMap,
query: PipelineQuery,
): Unit = {
val timeFeaturesOpt = getTimeFeatures(query, candidate, existingFeatures)
timeFeaturesOpt.foreach(timeFeatures => setFeatures(timeFeatures, richDataRecord))
}
private[feature_hydrator] def getTimeFeatures(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap,
): Option[TimeFeatures] = {
for {
requestTimestampMs <- Some(query.queryTime.inMilliseconds)
tweetId <- Some(candidate.id)
viewerId <- query.getOptionalUserId
tweetCreationTimeMs <- timeFromTweetOrUserId(tweetId)
timeSinceTweetCreation = requestTimestampMs - tweetCreationTimeMs
accountAgeDurationOpt = timeFromTweetOrUserId(viewerId).map { viewerAccountCreationTimeMs =>
Duration.fromMilliseconds(requestTimestampMs - viewerAccountCreationTimeMs)
}
timeSinceSourceTweetCreation =
existingFeatures
.getOrElse(SourceTweetIdFeature, None)
.flatMap { sourceTweetId =>
timeFromTweetOrUserId(sourceTweetId).map { sourceTweetCreationTimeMs =>
requestTimestampMs - sourceTweetCreationTimeMs
}
}
.getOrElse(timeSinceTweetCreation)
if (timeSinceTweetCreation > 0 && timeSinceSourceTweetCreation > 0)
} yield {
val timeFeatures = TimeFeatures(
timeSinceTweetCreation = timeSinceTweetCreation,
timeSinceSourceTweetCreation = timeSinceSourceTweetCreation,
timeSinceViewerAccountCreationSecs = accountAgeDurationOpt.map(_.inSeconds),
isDay30NewUser = accountAgeDurationOpt.map(_ < 30.days).getOrElse(false),
isMonth12NewUser = accountAgeDurationOpt.map(_ < 365.days).getOrElse(false),
accountAgeInterval = accountAgeDurationOpt.flatMap(AccountAgeInterval.fromDuration),
isTweetRecycled = false // only set in RecyclableTweetCandidateFilter, but it's not used
)
val timeFeaturesWithLastEngagement = addLastEngagementTimeFeatures(
existingFeatures.getOrElse(EarlybirdFeature, None),
timeFeatures,
timeSinceSourceTweetCreation
).getOrElse(timeFeatures)
val nonPollingTimestampsMs =
query.features.map(_.getOrElse(NonPollingTimesFeature, Seq.empty))
val timeFeaturesWithNonPollingOpt = addNonPollingTimeFeatures(
timeFeaturesWithLastEngagement,
requestTimestampMs,
tweetCreationTimeMs,
nonPollingTimestampsMs
)
timeFeaturesWithNonPollingOpt.getOrElse(timeFeaturesWithLastEngagement)
}
}
private def timeFromTweetOrUserId(tweetOrUserId: Long): Option[Long] = {
if (SnowflakeId.isSnowflakeId(tweetOrUserId))
Some(SnowflakeId(tweetOrUserId).time.inMilliseconds)
else None
}
private def addLastEngagementTimeFeatures(
tweetFeaturesOpt: Option[sc.ThriftTweetFeatures],
timeFeatures: TimeFeatures,
timeSinceSourceTweetCreation: Long
): Option[TimeFeatures] = {
tweetFeaturesOpt.map { tweetFeatures =>
val lastFavSinceCreationHrs = tweetFeatures.lastFavSinceCreationHrs.map(_.toDouble)
val lastRetweetSinceCreationHrs = tweetFeatures.lastRetweetSinceCreationHrs.map(_.toDouble)
val lastReplySinceCreationHrs = tweetFeatures.lastReplySinceCreationHrs.map(_.toDouble)
val lastQuoteSinceCreationHrs = tweetFeatures.lastQuoteSinceCreationHrs.map(_.toDouble)
timeFeatures.copy(
lastFavSinceCreationHrs = lastFavSinceCreationHrs,
lastRetweetSinceCreationHrs = lastRetweetSinceCreationHrs,
lastReplySinceCreationHrs = lastReplySinceCreationHrs,
lastQuoteSinceCreationHrs = lastQuoteSinceCreationHrs,
timeSinceLastFavoriteHrs = getTimeSinceLastEngagementHrs(
lastFavSinceCreationHrs,
timeSinceSourceTweetCreation
),
timeSinceLastRetweetHrs = getTimeSinceLastEngagementHrs(
lastRetweetSinceCreationHrs,
timeSinceSourceTweetCreation
),
timeSinceLastReplyHrs = getTimeSinceLastEngagementHrs(
lastReplySinceCreationHrs,
timeSinceSourceTweetCreation
),
timeSinceLastQuoteHrs = getTimeSinceLastEngagementHrs(
lastQuoteSinceCreationHrs,
timeSinceSourceTweetCreation
)
)
}
}
private def addNonPollingTimeFeatures(
timeFeatures: TimeFeatures,
requestTimestampMs: Long,
creationTimeMs: Long,
nonPollingTimestampsMs: Option[Seq[Long]]
): Option[TimeFeatures] = {
for {
nonPollingTimestampsMs <- nonPollingTimestampsMs
lastNonPollingTimestampMs <- nonPollingTimestampsMs.headOption
earliestNonPollingTimestampMs <- nonPollingTimestampsMs.lastOption
} yield {
val timeSinceLastNonPollingRequest = requestTimestampMs - lastNonPollingTimestampMs
val tweetAgeRatio = timeSinceLastNonPollingRequest / math.max(
1.0,
timeFeatures.timeSinceTweetCreation
)
/*
* Non-polling timestamps are stored in chronological order.
* The latest timestamps occur first, therefore we need to explicitly search in reverse order.
*/
val nonPollingRequestsSinceTweetCreation =
if (nonPollingTimestampsMs.nonEmpty) {
nonPollingTimestampsMs.search(creationTimeMs)(Ordering[Long].reverse).insertionPoint
} else {
0
}
/*
* Calculate the average time between non-polling requests; include
* request time in this calculation as latest timestamp.
*/
val timeBetweenNonPollingRequestsAvg =
(requestTimestampMs - earliestNonPollingTimestampMs) / math
.max(1.0, nonPollingTimestampsMs.size)
val timeFeaturesWithNonPolling = timeFeatures.copy(
timeBetweenNonPollingRequestsAvg = Some(timeBetweenNonPollingRequestsAvg),
timeSinceLastNonPollingRequest = Some(timeSinceLastNonPollingRequest),
nonPollingRequestsSinceTweetCreation = Some(nonPollingRequestsSinceTweetCreation),
tweetAgeRatio = Some(tweetAgeRatio)
)
timeFeaturesWithNonPolling
}
}
private[this] def getTimeSinceLastEngagementHrs(
lastEngagementTimeSinceCreationHrsOpt: Option[Double],
timeSinceTweetCreation: Long
): Option[Double] = {
lastEngagementTimeSinceCreationHrsOpt.map { lastEngagementTimeSinceCreationHrs =>
val timeSinceTweetCreationHrs = (timeSinceTweetCreation / (60 * 60 * 1000)).toInt
timeSinceTweetCreationHrs - lastEngagementTimeSinceCreationHrs
}
}
private def setFeatures(features: TimeFeatures, richDataRecord: RichDataRecord): Unit = {
val record = richDataRecord.getRecord
.setFeatureValue(IS_TWEET_RECYCLED, features.isTweetRecycled)
.setFeatureValue(TIME_SINCE_TWEET_CREATION, features.timeSinceTweetCreation)
.setFeatureValueFromOption(
TIME_SINCE_VIEWER_ACCOUNT_CREATION_SECS,
features.timeSinceViewerAccountCreationSecs)
.setFeatureValue(
USER_ID_IS_SNOWFLAKE_ID,
features.timeSinceViewerAccountCreationSecs.isDefined
)
.setFeatureValueFromOption(ACCOUNT_AGE_INTERVAL, features.accountAgeInterval.map(_.id.toLong))
.setFeatureValue(IS_30_DAY_NEW_USER, features.isDay30NewUser)
.setFeatureValue(IS_12_MONTH_NEW_USER, features.isMonth12NewUser)
.setFeatureValueFromOption(LAST_FAVORITE_SINCE_CREATION_HRS, features.lastFavSinceCreationHrs)
.setFeatureValueFromOption(
LAST_RETWEET_SINCE_CREATION_HRS,
features.lastRetweetSinceCreationHrs
)
.setFeatureValueFromOption(LAST_REPLY_SINCE_CREATION_HRS, features.lastReplySinceCreationHrs)
.setFeatureValueFromOption(LAST_QUOTE_SINCE_CREATION_HRS, features.lastQuoteSinceCreationHrs)
.setFeatureValueFromOption(TIME_SINCE_LAST_FAVORITE_HRS, features.timeSinceLastFavoriteHrs)
.setFeatureValueFromOption(TIME_SINCE_LAST_RETWEET_HRS, features.timeSinceLastRetweetHrs)
.setFeatureValueFromOption(TIME_SINCE_LAST_REPLY_HRS, features.timeSinceLastReplyHrs)
.setFeatureValueFromOption(TIME_SINCE_LAST_QUOTE_HRS, features.timeSinceLastQuoteHrs)
/*
* set features whose values are optional as some users do not have non-polling timestamps
*/
features.timeBetweenNonPollingRequestsAvg.foreach(
record.setFeatureValue(TIME_BETWEEN_NON_POLLING_REQUESTS_AVG, _)
)
features.timeSinceLastNonPollingRequest.foreach(
record.setFeatureValue(TIME_SINCE_LAST_NON_POLLING_REQUEST, _)
)
features.nonPollingRequestsSinceTweetCreation.foreach(
record.setFeatureValue(NON_POLLING_REQUESTS_SINCE_TWEET_CREATION, _)
)
features.tweetAgeRatio.foreach(record.setFeatureValue(TWEET_AGE_RATIO, _))
}
}

View File

@ -0,0 +1,63 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.marshaller.timelines.DeviceContextMarshaller
import com.twitter.home_mixer.model.request.DeviceContext
import com.twitter.home_mixer.model.request.HasDeviceContext
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.stitch.timelineservice.TimelineService
import com.twitter.timelineservice.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
object TimelineServiceTweetsFeature extends Feature[PipelineQuery, Seq[Long]]
@Singleton
case class TimelineServiceTweetsQueryFeatureHydrator @Inject() (
timelineService: TimelineService,
deviceContextMarshaller: DeviceContextMarshaller)
extends QueryFeatureHydrator[PipelineQuery with HasDeviceContext] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("TimelineServiceTweets")
override val features: Set[Feature[_, _]] = Set(TimelineServiceTweetsFeature)
private val MaxTimelineServiceTweets = 200
override def hydrate(query: PipelineQuery with HasDeviceContext): Stitch[FeatureMap] = {
val deviceContext = query.deviceContext.getOrElse(DeviceContext.Empty)
val timelineQueryOptions = t.TimelineQueryOptions(
contextualUserId = query.clientContext.userId,
deviceContext = Some(deviceContextMarshaller(deviceContext, query.clientContext))
)
val timelineServiceQuery = t.TimelineQuery(
timelineType = t.TimelineType.Home,
timelineId = query.getRequiredUserId,
maxCount = MaxTimelineServiceTweets.toShort,
cursor2 = None,
options = Some(timelineQueryOptions),
timelineId2 = query.clientContext.userId.map(t.TimelineId(t.TimelineType.Home, _, None)),
)
timelineService.getTimeline(timelineServiceQuery).map { timeline =>
val tweets = timeline.entries.collect {
case t.TimelineEntry.Tweet(tweet) => tweet.statusId
}
FeatureMapBuilder().add(TimelineServiceTweetsFeature, tweets).build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)
)
}

View File

@ -0,0 +1,87 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.TweetImpressionsFeature
import com.twitter.home_mixer.model.request.HasSeenTweetIds
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.impression.{thriftscala => t}
import com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClient
import com.twitter.util.Duration
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class TweetImpressionsQueryFeatureHydrator[
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
manhattanTweetImpressionStoreClient: ManhattanTweetImpressionStoreClient)
extends QueryFeatureHydrator[Query] {
private val TweetImpressionTTL = 1.day
private val TweetImpressionCap = 3000
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetImpressions")
override val features: Set[Feature[_, _]] = Set(TweetImpressionsFeature)
override def hydrate(query: Query): Stitch[FeatureMap] = {
manhattanTweetImpressionStoreClient.get(query.getRequiredUserId).map { entriesOpt =>
val entries = entriesOpt.map(_.entries).toSeq.flatten
val updatedImpressions =
if (query.seenTweetIds.forall(_.isEmpty)) entries
else updateTweetImpressions(entries, query.seenTweetIds.get)
FeatureMapBuilder().add(TweetImpressionsFeature, updatedImpressions).build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)
)
/**
* 1) Check timestamps and remove expired tweets based on [[TweetImpressionTTL]]
* 2) Filter duplicates between current tweets and those in the impression store (remove older ones)
* 3) Prepend new (Timestamp, Seq[TweetIds]) to the tweets from the impression store
* 4) Truncate older tweets if sum of all tweets across timestamps >= [[TweetImpressionCap]],
*/
private[feature_hydrator] def updateTweetImpressions(
tweetImpressionsFromStore: Seq[t.TweetImpressionsEntry],
seenIdsFromClient: Seq[Long],
currentTime: Long = Time.now.inMilliseconds,
tweetImpressionTTL: Duration = TweetImpressionTTL,
tweetImpressionCap: Int = TweetImpressionCap,
): Seq[t.TweetImpressionsEntry] = {
val seenIdsFromClientSet = seenIdsFromClient.toSet
val dedupedTweetImpressionsFromStore: Seq[t.TweetImpressionsEntry] = tweetImpressionsFromStore
.collect {
case t.TweetImpressionsEntry(ts, tweetIds)
if Time.fromMilliseconds(ts).untilNow < tweetImpressionTTL =>
t.TweetImpressionsEntry(ts, tweetIds.filterNot(seenIdsFromClientSet.contains))
}.filter { _.tweetIds.nonEmpty }
val mergedTweetImpressionsEntries =
t.TweetImpressionsEntry(currentTime, seenIdsFromClient) +: dedupedTweetImpressionsFromStore
val initialTweetImpressionsWithCap = (Seq.empty[t.TweetImpressionsEntry], tweetImpressionCap)
val (truncatedTweetImpressionsEntries: Seq[t.TweetImpressionsEntry], _) =
mergedTweetImpressionsEntries
.foldLeft(initialTweetImpressionsWithCap) {
case (
(tweetImpressions: Seq[t.TweetImpressionsEntry], remainingCap),
t.TweetImpressionsEntry(ts, tweetIds)) if remainingCap > 0 =>
(
t.TweetImpressionsEntry(ts, tweetIds.take(remainingCap)) +: tweetImpressions,
remainingCap - tweetIds.size)
case (tweetImpressionsWithCap, _) => tweetImpressionsWithCap
}
truncatedTweetImpressionsEntries.reverse
}
}

View File

@ -0,0 +1,66 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.ml.api.DataRecord
import com.twitter.ml.api.RichDataRecord
import com.twitter.ml.api.constant.SharedFeatures
import com.twitter.ml.api.util.DataRecordConverters._
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures
import java.lang.{Long => JLong}
object TweetMetaDataDataRecord
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
object TweetMetaDataFeatureHydrator
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetMetaData")
override def features: Set[Feature[_, _]] = Set(TweetMetaDataDataRecord)
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val richDataRecord = new RichDataRecord()
setFeatures(richDataRecord, candidate, existingFeatures)
Stitch.value {
FeatureMapBuilder()
.add(TweetMetaDataDataRecord, richDataRecord.getRecord)
.build()
}
}
private def setFeatures(
richDataRecord: RichDataRecord,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Unit = {
richDataRecord.setFeatureValue[JLong](SharedFeatures.TWEET_ID, candidate.id)
richDataRecord.setFeatureValueFromOption(
TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID,
CandidatesUtil.getOriginalAuthorId(existingFeatures))
richDataRecord.setFeatureValueFromOption(
TimelinesSharedFeatures.CANDIDATE_TWEET_SOURCE_ID,
existingFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toLong))
}
}

View File

@ -0,0 +1,149 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.escherbird.{thriftscala => esb}
import com.twitter.finagle.stats.Stat
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.ContentFeatureAdapter
import com.twitter.home_mixer.model.HomeFeatures.MediaUnderstandingAnnotationIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieContentRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.home_mixer.util.tweetypie.content.FeatureExtractionHelper
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.util.OffloadFuturePools
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.common.util.MediaUnderstandingAnnotations
import com.twitter.tweetypie.{thriftscala => tp}
import com.twitter.util.Future
import com.twitter.util.Return
import com.twitter.util.Throw
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object TweetypieContentDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class TweetypieContentFeatureHydrator @Inject() (
@Named(TweetypieContentRepository) client: KeyValueRepository[Seq[Long], Long, tp.Tweet],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetypieContent")
override val features: Set[Feature[_, _]] = Set(
MediaUnderstandingAnnotationIdsFeature,
TweetypieContentDataRecordFeature
)
override val statScope: String = identifier.toString
private val bulkRequestLatencyStat =
statsReceiver.scope(statScope).scope("bulkRequest").stat("latency_ms")
private val postTransformerLatencyStat =
statsReceiver.scope(statScope).scope("postTransformer").stat("latency_ms")
private val bulkPostTransformerLatencyStat =
statsReceiver.scope(statScope).scope("bulkPostTransformer").stat("latency_ms")
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val tweetIdsToHydrate = candidates.map(getCandidateOriginalTweetId).distinct
val response: Future[KeyValueResult[Long, tp.Tweet]] =
Stat.timeFuture(bulkRequestLatencyStat)(
if (tweetIdsToHydrate.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(tweetIdsToHydrate)
}
)
response.flatMap { result =>
Stat.timeFuture(bulkPostTransformerLatencyStat) {
OffloadFuturePools
.parallelize[CandidateWithFeatures[TweetCandidate], Try[(Seq[Long], DataRecord)]](
candidates,
parTransformer(result, _),
parallelism = 32,
default = Return((Seq.empty, new DataRecord))
).map {
_.map {
case Return(result) =>
FeatureMapBuilder()
.add(MediaUnderstandingAnnotationIdsFeature, result._1)
.add(TweetypieContentDataRecordFeature, result._2)
.build()
case Throw(e) =>
FeatureMapBuilder()
.add(MediaUnderstandingAnnotationIdsFeature, Throw(e))
.add(TweetypieContentDataRecordFeature, Throw(e))
.build()
}
}
}
}
}
}
private def parTransformer(
result: KeyValueResult[Long, tp.Tweet],
candidate: CandidateWithFeatures[TweetCandidate]
): Try[(Seq[Long], DataRecord)] = {
val originalTweetId = Some(getCandidateOriginalTweetId(candidate))
val value = observedGet(key = originalTweetId, keyValueResult = result)
Stat.time(postTransformerLatencyStat)(postTransformer(value))
}
private def postTransformer(
result: Try[Option[tp.Tweet]]
): Try[(Seq[Long], DataRecord)] = {
result.map { tweet =>
val transformedValue = tweet.map(FeatureExtractionHelper.extractFeatures)
val semanticAnnotations = transformedValue
.flatMap { contentFeatures =>
contentFeatures.semanticCoreAnnotations.map {
getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds
}
}.getOrElse(Seq.empty)
val dataRecord = ContentFeatureAdapter.adaptToDataRecords(transformedValue).asScala.head
(semanticAnnotations, dataRecord)
}
}
private def getCandidateOriginalTweetId(
candidate: CandidateWithFeatures[TweetCandidate]
): Long = {
candidate.features
.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id)
}
private def getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds(
semanticCoreAnnotations: Seq[esb.TweetEntityAnnotation]
): Seq[Long] =
semanticCoreAnnotations
.filter(MediaUnderstandingAnnotations.isEligibleNonSensitiveHighRecallMUAnnotation)
.map(_.entityId)
}

View File

@ -0,0 +1,156 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
import com.twitter.home_mixer.model.HomeFeatures.IsNsfwFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.model.request.ForYouProduct
import com.twitter.home_mixer.model.request.ListTweetsProduct
import com.twitter.home_mixer.model.request.ScoredTweetsProduct
import com.twitter.home_mixer.util.tweetypie.RequestFields
import com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw
import com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.spam.rtf.{thriftscala => rtf}
import com.twitter.stitch.Stitch
import com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}
import com.twitter.tweetypie.{thriftscala => tp}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class TweetypieFeatureHydrator @Inject() (tweetypieStitchClient: TweetypieStitchClient)
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Tweetypie")
override val features: Set[Feature[_, _]] = Set(
AuthorIdFeature,
InReplyToTweetIdFeature,
IsHydratedFeature,
IsNsfw,
IsNsfwFeature,
IsRetweetFeature,
QuotedTweetDroppedFeature,
QuotedTweetIdFeature,
QuotedUserIdFeature,
SourceTweetIdFeature,
SourceUserIdFeature,
TweetTextFeature,
VisibilityReason
)
private val DefaultFeatureMap = FeatureMapBuilder()
.add(IsHydratedFeature, false)
.add(IsNsfw, None)
.add(IsNsfwFeature, false)
.add(QuotedTweetDroppedFeature, false)
.add(TweetTextFeature, None)
.add(VisibilityReason, None)
.build()
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val safetyLevel = query.product match {
case FollowingProduct => rtf.SafetyLevel.TimelineHomeLatest
case ForYouProduct => rtf.SafetyLevel.TimelineHome
case ScoredTweetsProduct => rtf.SafetyLevel.TimelineHome
case ListTweetsProduct => rtf.SafetyLevel.TimelineLists
case unknown => throw new UnsupportedOperationException(s"Unknown product: $unknown")
}
val tweetFieldsOptions = tp.GetTweetFieldsOptions(
tweetIncludes = RequestFields.TweetTPHydrationFields,
includeRetweetedTweet = true,
includeQuotedTweet = true,
visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,
safetyLevel = Some(safetyLevel),
forUserId = Some(query.getRequiredUserId)
)
tweetypieStitchClient.getTweetFields(tweetId = candidate.id, options = tweetFieldsOptions).map {
case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), quoteOpt, _) =>
val coreData = found.tweet.coreData
val isNsfwAdmin = coreData.exists(_.nsfwAdmin)
val isNsfwUser = coreData.exists(_.nsfwUser)
val quotedTweetDropped = quoteOpt.exists {
case _: tp.TweetFieldsResultState.Filtered => true
case _: tp.TweetFieldsResultState.NotFound => true
case _ => false
}
val quotedTweetIsNsfw = quoteOpt.exists {
case quoteTweet: tp.TweetFieldsResultState.Found =>
quoteTweet.found.tweet.coreData.exists(data => data.nsfwAdmin || data.nsfwUser)
case _ => false
}
val sourceTweetIsNsfw =
found.retweetedTweet.exists(_.coreData.exists(data => data.nsfwAdmin || data.nsfwUser))
val tweetText = coreData.map(_.text)
val tweetAuthorId = coreData.map(_.userId)
val inReplyToTweetId = coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))
val retweetedTweetId = found.retweetedTweet.map(_.id)
val quotedTweetId = quoteOpt.flatMap {
case quoteTweet: tp.TweetFieldsResultState.Found =>
Some(quoteTweet.found.tweet.id)
case _ => None
}
val retweetedTweetUserId = found.retweetedTweet.flatMap(_.coreData).map(_.userId)
val quotedTweetUserId = quoteOpt.flatMap {
case quoteTweet: tp.TweetFieldsResultState.Found =>
quoteTweet.found.tweet.coreData.map(_.userId)
case _ => None
}
val isNsfw = isNsfwAdmin || isNsfwUser || sourceTweetIsNsfw || quotedTweetIsNsfw
FeatureMapBuilder()
.add(AuthorIdFeature, tweetAuthorId)
.add(InReplyToTweetIdFeature, inReplyToTweetId)
.add(IsHydratedFeature, true)
.add(IsNsfw, Some(isNsfw))
.add(IsNsfwFeature, isNsfw)
.add(IsRetweetFeature, retweetedTweetId.isDefined)
.add(QuotedTweetDroppedFeature, quotedTweetDropped)
.add(QuotedTweetIdFeature, quotedTweetId)
.add(QuotedUserIdFeature, quotedTweetUserId)
.add(SourceTweetIdFeature, retweetedTweetId)
.add(SourceUserIdFeature, retweetedTweetUserId)
.add(TweetTextFeature, tweetText)
.add(VisibilityReason, found.suppressReason)
.build()
// If no tweet result found, return default and pre-existing features
case _ =>
DefaultFeatureMap +
(AuthorIdFeature, existingFeatures.getOrElse(AuthorIdFeature, None)) +
(InReplyToTweetIdFeature, existingFeatures.getOrElse(InReplyToTweetIdFeature, None)) +
(IsRetweetFeature, existingFeatures.getOrElse(IsRetweetFeature, false)) +
(QuotedTweetIdFeature, existingFeatures.getOrElse(QuotedTweetIdFeature, None)) +
(QuotedUserIdFeature, existingFeatures.getOrElse(QuotedUserIdFeature, None)) +
(SourceTweetIdFeature, existingFeatures.getOrElse(SourceTweetIdFeature, None)) +
(SourceUserIdFeature, existingFeatures.getOrElse(SourceUserIdFeature, None))
}
}
}

View File

@ -0,0 +1,161 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.google.inject.name.Named
import com.twitter.conversions.DurationOps.RichDuration
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.HasImageFeature
import com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature
import com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SemanticAnnotationFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieStaticEntitiesCache
import com.twitter.home_mixer.util.tweetypie.RequestFields
import com.twitter.home_mixer.util.tweetypie.content.TweetMediaFeaturesExtractor
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.cache.TtlCache
import com.twitter.spam.rtf.{thriftscala => sp}
import com.twitter.stitch.Stitch
import com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}
import com.twitter.tweetypie.{thriftscala => tp}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class TweetypieStaticEntitiesFeatureHydrator @Inject() (
tweetypieStitchClient: TweetypieStitchClient,
@Named(TweetypieStaticEntitiesCache) cacheClient: TtlCache[Long, tp.Tweet])
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("TweetypieStaticEntities")
override val features: Set[Feature[_, _]] = Set(
AuthorIdFeature,
DirectedAtUserIdFeature,
HasImageFeature,
HasVideoFeature,
InReplyToTweetIdFeature,
InReplyToUserIdFeature,
IsRetweetFeature,
MentionScreenNameFeature,
MentionUserIdFeature,
QuotedTweetIdFeature,
QuotedUserIdFeature,
SemanticAnnotationFeature,
SourceTweetIdFeature,
SourceUserIdFeature
)
private val CacheTTL = 24.hours
private val DefaultFeatureMap = FeatureMapBuilder()
.add(AuthorIdFeature, None)
.add(DirectedAtUserIdFeature, None)
.add(HasImageFeature, false)
.add(HasVideoFeature, false)
.add(InReplyToTweetIdFeature, None)
.add(InReplyToUserIdFeature, None)
.add(IsRetweetFeature, false)
.add(MentionScreenNameFeature, Seq.empty)
.add(MentionUserIdFeature, Seq.empty)
.add(QuotedTweetIdFeature, None)
.add(QuotedUserIdFeature, None)
.add(SemanticAnnotationFeature, Seq.empty)
.add(SourceTweetIdFeature, None)
.add(SourceUserIdFeature, None)
.build()
/**
* Steps:
* 1. query cache with all candidates
* 2. create a cached feature map
* 3. iterate candidates to hydrate features
* 3.a transform cached candidates
* 3.b hydrate non-cached candidates from Tweetypie and write to cache
*/
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val tweetIds = candidates.map(_.candidate.id)
val cachedTweetsMapFu = cacheClient
.get(tweetIds)
.map(_.found)
Stitch.callFuture(cachedTweetsMapFu).flatMap { cachedTweets =>
Stitch.collect {
candidates.map { candidate =>
if (cachedTweets.contains(candidate.candidate.id))
Stitch.value(createFeatureMap(cachedTweets(candidate.candidate.id)))
else readFromTweetypie(query, candidate)
}
}
}
}
private def createFeatureMap(tweet: tp.Tweet): FeatureMap = {
val coreData = tweet.coreData
val quotedTweet = tweet.quotedTweet
val mentions = tweet.mentions.getOrElse(Seq.empty)
val share = coreData.flatMap(_.share)
val reply = coreData.flatMap(_.reply)
val semanticAnnotations =
tweet.escherbirdEntityAnnotations.map(_.entityAnnotations).getOrElse(Seq.empty)
FeatureMapBuilder()
.add(AuthorIdFeature, coreData.map(_.userId))
.add(DirectedAtUserIdFeature, coreData.flatMap(_.directedAtUser.map(_.userId)))
.add(HasImageFeature, TweetMediaFeaturesExtractor.hasImage(tweet))
.add(HasVideoFeature, TweetMediaFeaturesExtractor.hasVideo(tweet))
.add(InReplyToTweetIdFeature, reply.flatMap(_.inReplyToStatusId))
.add(InReplyToUserIdFeature, reply.map(_.inReplyToUserId))
.add(IsRetweetFeature, share.isDefined)
.add(MentionScreenNameFeature, mentions.map(_.screenName))
.add(MentionUserIdFeature, mentions.flatMap(_.userId))
.add(QuotedTweetIdFeature, quotedTweet.map(_.tweetId))
.add(QuotedUserIdFeature, quotedTweet.map(_.userId))
.add(SemanticAnnotationFeature, semanticAnnotations)
.add(SourceTweetIdFeature, share.map(_.sourceStatusId))
.add(SourceUserIdFeature, share.map(_.sourceUserId))
.build()
}
private def readFromTweetypie(
query: PipelineQuery,
candidate: CandidateWithFeatures[TweetCandidate]
): Stitch[FeatureMap] = {
tweetypieStitchClient
.getTweetFields(
tweetId = candidate.candidate.id,
options = tp.GetTweetFieldsOptions(
tweetIncludes = RequestFields.TweetStaticEntitiesFields,
includeRetweetedTweet = false,
includeQuotedTweet = false,
forUserId = query.getOptionalUserId, // Needed to get protected Tweets for certain users
visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,
safetyLevel = Some(sp.SafetyLevel.FilterNone) // VF is handled in the For You product
)
).map {
case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), _, _) =>
cacheClient.set(candidate.candidate.id, found.tweet, CacheTTL)
createFeatureMap(found.tweet)
case _ =>
DefaultFeatureMap + (AuthorIdFeature, candidate.features.getOrElse(AuthorIdFeature, None))
}
}
}

View File

@ -0,0 +1,96 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinAuthorFollowEmbeddingsAdapter
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinAuthorFollow20200101FeatureRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.ml.api.DataRecord
import com.twitter.ml.api.{thriftscala => ml}
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.servo.repository.KeyValueResult
import com.twitter.stitch.Stitch
import com.twitter.util.Future
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object TwhinAuthorFollow20220101Feature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class TwhinAuthorFollow20220101FeatureHydrator @Inject() (
@Named(TwhinAuthorFollow20200101FeatureRepository)
client: KeyValueRepository[Seq[Long], Long, ml.Embedding],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("TwhinAuthorFollow20220101")
override val features: Set[Feature[_, _]] = Set(TwhinAuthorFollow20220101Feature)
override val statScope: String = identifier.toString
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val possiblyAuthorIds = extractKeys(candidates)
val authorIds = possiblyAuthorIds.flatten
val response: Future[KeyValueResult[Long, ml.Embedding]] =
if (authorIds.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(authorIds)
}
response.map { result =>
possiblyAuthorIds.map { possiblyAuthorId =>
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
val transformedValue = postTransformer(value)
FeatureMapBuilder()
.add(TwhinAuthorFollow20220101Feature, transformedValue)
.build()
}
}
}
}
private def postTransformer(embedding: Try[Option[ml.Embedding]]): Try[DataRecord] = {
embedding.map { e =>
TwhinAuthorFollowEmbeddingsAdapter.adaptToDataRecords(e).asScala.head
}
}
private def extractKeys(
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Seq[Option[Long]] = {
candidates.map { candidate =>
candidate.features
.getTry(AuthorIdFeature)
.toOption
.flatten
}
}
}

View File

@ -0,0 +1,80 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserEngagementEmbeddingsAdapter
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserEngagementFeatureRepository
import com.twitter.ml.api.DataRecord
import com.twitter.ml.api.RichDataRecord
import com.twitter.ml.api.util.ScalaToJavaDataRecordConversions
import com.twitter.ml.api.{thriftscala => ml}
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Return
import com.twitter.util.Throw
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object TwhinUserEngagementFeature
extends DataRecordInAFeature[PipelineQuery]
with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class TwhinUserEngagementQueryFeatureHydrator @Inject() (
@Named(TwhinUserEngagementFeatureRepository)
client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],
statsReceiver: StatsReceiver)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("TwhinUserEngagement")
override val features: Set[Feature[_, _]] = Set(TwhinUserEngagementFeature)
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
private val keyFoundCounter = scopedStatsReceiver.counter("key/found")
private val keyLossCounter = scopedStatsReceiver.counter("key/loss")
private val keyFailureCounter = scopedStatsReceiver.counter("key/failure")
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
Stitch.callFuture {
client(Seq(userId)).map { results =>
val embedding: Option[ml.FloatTensor] = results(userId) match {
case Return(value) =>
if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()
else keyLossCounter.incr()
value
case Throw(_) =>
keyFailureCounter.incr()
None
case _ =>
None
}
val dataRecord =
new RichDataRecord(new DataRecord, TwhinUserEngagementEmbeddingsAdapter.getFeatureContext)
embedding.foreach { floatTensor =>
dataRecord.setFeatureValue(
TwhinUserEngagementEmbeddingsAdapter.twhinEmbeddingsFeature,
ScalaToJavaDataRecordConversions.scalaTensor2Java(
ml.GeneralTensor.FloatTensor(floatTensor))
)
}
FeatureMapBuilder()
.add(TwhinUserEngagementFeature, dataRecord.getRecord)
.build()
}
}
}
}

View File

@ -0,0 +1,80 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserFollowEmbeddingsAdapter
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserFollowFeatureRepository
import com.twitter.ml.api.DataRecord
import com.twitter.ml.api.RichDataRecord
import com.twitter.ml.api.util.ScalaToJavaDataRecordConversions
import com.twitter.ml.api.{thriftscala => ml}
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Return
import com.twitter.util.Throw
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object TwhinUserFollowFeature
extends DataRecordInAFeature[PipelineQuery]
with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class TwhinUserFollowQueryFeatureHydrator @Inject() (
@Named(TwhinUserFollowFeatureRepository)
client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],
statsReceiver: StatsReceiver)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("TwhinUserFollow")
override val features: Set[Feature[_, _]] = Set(TwhinUserFollowFeature)
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
private val keyFoundCounter = scopedStatsReceiver.counter("key/found")
private val keyLossCounter = scopedStatsReceiver.counter("key/loss")
private val keyFailureCounter = scopedStatsReceiver.counter("key/failure")
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
Stitch.callFuture(
client(Seq(userId)).map { results =>
val embedding: Option[ml.FloatTensor] = results(userId) match {
case Return(value) =>
if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()
else keyLossCounter.incr()
value
case Throw(_) =>
keyFailureCounter.incr()
None
case _ =>
None
}
val dataRecord =
new RichDataRecord(new DataRecord, TwhinUserFollowEmbeddingsAdapter.getFeatureContext)
embedding.foreach { floatTensor =>
dataRecord.setFeatureValue(
TwhinUserFollowEmbeddingsAdapter.twhinEmbeddingsFeature,
ScalaToJavaDataRecordConversions.scalaTensor2Java(
ml.GeneralTensor
.FloatTensor(floatTensor)))
}
FeatureMapBuilder()
.add(TwhinUserFollowFeature, dataRecord.getRecord)
.build()
}
)
}
}

View File

@ -0,0 +1,84 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.UserFollowedTopicIdsRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Future
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object UserFollowedTopicIdsFeature extends Feature[TweetCandidate, Seq[Long]]
@Singleton
class UserFollowedTopicIdsFeatureHydrator @Inject() (
@Named(UserFollowedTopicIdsRepository)
client: KeyValueRepository[Seq[Long], Long, Seq[Long]],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("UserFollowedTopicIds")
override val features: Set[Feature[_, _]] = Set(UserFollowedTopicIdsFeature)
override val statScope: String = identifier.toString
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val possiblyAuthorIds = extractKeys(candidates)
val authorIds = possiblyAuthorIds.flatten
val response: Future[KeyValueResult[Long, Seq[Long]]] =
if (authorIds.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(authorIds)
}
response.map { result =>
possiblyAuthorIds.map { possiblyAuthorId =>
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
val transformedValue = postTransformer(value)
FeatureMapBuilder()
.add(UserFollowedTopicIdsFeature, transformedValue)
.build()
}
}
}
}
private def postTransformer(input: Try[Option[Seq[Long]]]): Try[Seq[Long]] = {
input.map(_.getOrElse(Seq.empty[Long]))
}
private def extractKeys(
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Seq[Option[Long]] = {
candidates.map { candidate =>
candidate.features
.getTry(AuthorIdFeature)
.toOption
.flatten
}
}
}

View File

@ -0,0 +1,35 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.param.HomeMixerInjectionNames.UserLanguagesStore
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.common.constants.{thriftscala => scc}
import com.twitter.stitch.Stitch
import com.twitter.storehaus.ReadableStore
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object UserLanguagesFeature extends Feature[PipelineQuery, Seq[scc.ThriftLanguage]]
@Singleton
case class UserLanguagesFeatureHydrator @Inject() (
@Named(UserLanguagesStore) store: ReadableStore[Long, Seq[scc.ThriftLanguage]])
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("UserLanguages")
override val features: Set[Feature[_, _]] = Set(UserLanguagesFeature)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
Stitch.callFuture(store.get(query.getRequiredUserId)).map { languages =>
FeatureMapBuilder()
.add(UserLanguagesFeature, languages.getOrElse(Seq.empty))
.build()
}
}
}

View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.UserStateFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.user_health.{thriftscala => uh}
import com.twitter.timelines.user_health.v1.{thriftscala => uhv1}
import com.twitter.user_session_store.ReadOnlyUserSessionStore
import com.twitter.user_session_store.ReadRequest
import com.twitter.user_session_store.UserSessionDataset
import com.twitter.user_session_store.UserSessionDataset.UserSessionDataset
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class UserStateQueryFeatureHydrator @Inject() (
userSessionStore: ReadOnlyUserSessionStore)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("UserState")
override val features: Set[Feature[_, _]] = Set(UserStateFeature)
private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.UserHealth)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
userSessionStore
.read(ReadRequest(query.getRequiredUserId, datasets))
.map { userSession =>
val userState = userSession.flatMap {
_.userHealth match {
case Some(uh.UserHealth.V1(uhv1.UserHealth(userState))) => userState
case _ => None
}
}
FeatureMapBuilder()
.add(UserStateFeature, userState)
.build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)
)
}

View File

@ -0,0 +1,88 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature
import com.twitter.home_mixer.model.HomeFeatures.RepliedByEngagerIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.RetweetedByEngagerIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.UtegSocialProofRepository
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.recos.recos_common.{thriftscala => rc}
import com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
@Singleton
class UtegFeatureHydrator @Inject() (
@Named(UtegSocialProofRepository) client: KeyValueRepository[
(Seq[Long], (Long, Map[Long, Double])),
Long,
uteg.TweetRecommendation
]) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with Conditionally[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Uteg")
override val features: Set[Feature[_, _]] = Set(
FavoritedByUserIdsFeature,
RetweetedByEngagerIdsFeature,
RepliedByEngagerIdsFeature
)
override def onlyIf(query: PipelineQuery): Boolean = query.features
.exists(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double]).nonEmpty)
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val seedUserWeights = query.features.map(_.get(RealGraphInNetworkScoresFeature)).get
val sourceTweetIds = candidates.flatMap(_.features.getOrElse(SourceTweetIdFeature, None))
val inReplyToTweetIds = candidates.flatMap(_.features.getOrElse(InReplyToTweetIdFeature, None))
val tweetIds = candidates.map(_.candidate.id)
val tweetIdsToSend = (tweetIds ++ sourceTweetIds ++ inReplyToTweetIds).distinct
val utegQuery = (tweetIdsToSend, (query.getRequiredUserId, seedUserWeights))
Stitch
.callFuture(client(utegQuery))
.map(handleResponse(candidates, _))
}
private def handleResponse(
candidates: Seq[CandidateWithFeatures[TweetCandidate]],
results: KeyValueResult[Long, uteg.TweetRecommendation],
): Seq[FeatureMap] = {
candidates.map { candidate =>
val candidateProof = results(candidate.candidate.id).toOption.flatten
val sourceProof = candidate.features
.getOrElse(SourceTweetIdFeature, None).flatMap(results(_).toOption.flatten)
val proofs = Seq(candidateProof, sourceProof).flatten.map(_.socialProofByType)
val favoritedBy = proofs.flatMap(_.get(rc.SocialProofType.Favorite)).flatten
val retweetedBy = proofs.flatMap(_.get(rc.SocialProofType.Retweet)).flatten
val repliedBy = proofs.flatMap(_.get(rc.SocialProofType.Reply)).flatten
FeatureMapBuilder()
.add(FavoritedByUserIdsFeature, favoritedBy)
.add(RetweetedByEngagerIdsFeature, retweetedBy)
.add(RepliedByEngagerIdsFeature, repliedBy)
.build()
}
}
}

View File

@ -0,0 +1,59 @@
package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features
import com.twitter.ml.api.DataRecordMerger
import com.twitter.ml.api.Feature
import com.twitter.ml.api.FeatureContext
import com.twitter.ml.api.RichDataRecord
import com.twitter.ml.api.util.CompactDataRecordConverter
import com.twitter.ml.api.util.FDsl._
import com.twitter.timelines.author_features.v1.{thriftjava => af}
import com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase
import com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig
import com.twitter.timelines.prediction.features.user_health.UserHealthFeatures
object AuthorFeaturesAdapter extends TimelinesMutatingAdapterBase[Option[af.AuthorFeatures]] {
private val originalAuthorAggregatesFeatures =
TimelinesAggregationConfig.originalAuthorReciprocalEngagementAggregates
.buildTypedAggregateGroups().flatMap(_.allOutputFeatures)
private val authorFeatures = originalAuthorAggregatesFeatures ++
Seq(
UserHealthFeatures.AuthorState,
UserHealthFeatures.NumAuthorFollowers,
UserHealthFeatures.NumAuthorConnectDays,
UserHealthFeatures.NumAuthorConnect)
private val featureContext = new FeatureContext(authorFeatures: _*)
override def getFeatureContext: FeatureContext = featureContext
override val commonFeatures: Set[Feature[_]] = Set.empty
private val compactDataRecordConverter = new CompactDataRecordConverter()
private val drMerger = new DataRecordMerger()
override def setFeatures(
authorFeaturesOpt: Option[af.AuthorFeatures],
richDataRecord: RichDataRecord
): Unit = {
authorFeaturesOpt.foreach { authorFeatures =>
val dataRecord = richDataRecord.getRecord
dataRecord.setFeatureValue(
UserHealthFeatures.AuthorState,
authorFeatures.user_health.user_state.getValue.toLong)
dataRecord.setFeatureValue(
UserHealthFeatures.NumAuthorFollowers,
authorFeatures.user_health.num_followers.toDouble)
dataRecord.setFeatureValue(
UserHealthFeatures.NumAuthorConnectDays,
authorFeatures.user_health.num_connect_days.toDouble)
dataRecord.setFeatureValue(
UserHealthFeatures.NumAuthorConnect,
authorFeatures.user_health.num_connect.toDouble)
val originalAuthorAggregatesDataRecord =
compactDataRecordConverter.compactDataRecordToDataRecord(authorFeatures.aggregates)
drMerger.merge(dataRecord, originalAuthorAggregatesDataRecord)
}
}
}

Some files were not shown because too many files have changed in this diff Show More