mirror of
https://github.com/twitter/the-algorithm.git
synced 2025-06-10 14:48:16 -05:00
Twitter Recommendation Algorithm
Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
This commit is contained in:
30
home-mixer/BUILD.bazel
Normal file
30
home-mixer/BUILD.bazel
Normal file
@ -0,0 +1,30 @@
|
||||
jvm_binary(
|
||||
name = "bin",
|
||||
basename = "home-mixer",
|
||||
main = "com.twitter.home_mixer.HomeMixerServerMain",
|
||||
runtime_platform = "java11",
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/ch/qos/logback:logback-classic",
|
||||
"finagle/finagle-zipkin-scribe/src/main/scala",
|
||||
"finatra/inject/inject-logback/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer",
|
||||
"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback",
|
||||
"twitter-server-internal/src/main/scala",
|
||||
"twitter-server/logback-classic/src/main/scala",
|
||||
],
|
||||
)
|
||||
|
||||
# Aurora Workflows build phase convention requires a jvm_app named with home-mixer-app
|
||||
jvm_app(
|
||||
name = "home-mixer-app",
|
||||
archive = "zip",
|
||||
binary = ":bin",
|
||||
bundles = [
|
||||
bundle(
|
||||
fileset = ["config/**/*"],
|
||||
owning_target = "home-mixer/config:files",
|
||||
),
|
||||
],
|
||||
tags = ["bazel-compatible"],
|
||||
)
|
102
home-mixer/README.md
Normal file
102
home-mixer/README.md
Normal file
@ -0,0 +1,102 @@
|
||||
Home Mixer
|
||||
==========
|
||||
|
||||
Home Mixer is the main service used to construct and serve Twitter's Home Timelines. It currently
|
||||
powers:
|
||||
- For you - best Tweets from people you follow + recommended out-of-network content
|
||||
- Following - reverse chronological Tweets from people you follow
|
||||
- Lists - reverse chronological Tweets from List members
|
||||
|
||||
Home Mixer is built on Product Mixer, our custom Scala framework that facilitates building
|
||||
feeds of content.
|
||||
|
||||
## Overview
|
||||
|
||||
The For You recommendation algorithm in Home Mixer involves the following stages:
|
||||
|
||||
- Candidate Generation - fetch Tweets from various Candidate Sources. For example:
|
||||
- Earlybird Search Index
|
||||
- User Tweet Entity Graph
|
||||
- Cr Mixer
|
||||
- Follow Recommendations Service
|
||||
- Feature Hydration
|
||||
- Fetch the ~6000 features needed for ranking
|
||||
- Scoring and Ranking using ML model
|
||||
- Filters and Heuristics. For example:
|
||||
- Author Diversity
|
||||
- Content Balance (In network vs Out of Network)
|
||||
- Feedback fatigue
|
||||
- Deduplication / previously seen Tweets removal
|
||||
- Visibility Filtering (blocked, muted authors/tweets, NSFW settings)
|
||||
- Mixing - integrate Tweets with non-Tweet content
|
||||
- Ads
|
||||
- Who-to-follow modules
|
||||
- Prompts
|
||||
- Product Features and Serving
|
||||
- Conversation Modules for replies
|
||||
- Social Context
|
||||
- Timeline Navigation
|
||||
- Edited Tweets
|
||||
- Feedback options
|
||||
- Pagination and cursoring
|
||||
- Observability and logging
|
||||
- Client instructions and content marshalling
|
||||
|
||||
## Pipeline Structure
|
||||
|
||||
### General
|
||||
|
||||
Product Mixer services like Home Mixer are structured around Pipelines that split the execution
|
||||
into transparent and structured steps.
|
||||
|
||||
Requests first go to Product Pipelines, which are used to select which Mixer Pipeline or
|
||||
Recommendation Pipeline to run for a given request. Each Mixer or Recommendation
|
||||
Pipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.
|
||||
|
||||
Mixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together
|
||||
(e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines)
|
||||
and rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned.
|
||||
These pipelines also marshall candidates into a domain object and then into a transport object
|
||||
to return to the caller.
|
||||
|
||||
Candidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic
|
||||
operations on the Candidates, such as filtering out unwanted candidates, applying decorations,
|
||||
and hydrating features.
|
||||
|
||||
The sections below describe the high level pipeline structure (non-exhaustive) for the main Home
|
||||
Timeline tabs powered by Home Mixer.
|
||||
|
||||
### For You
|
||||
|
||||
- ForYouProductPipelineConfig
|
||||
- ForYouScoredTweetsMixerPipelineConfig (main orchestration layer - mixes Tweets with ads and users)
|
||||
- ForYouScoredTweetsCandidatePipelineConfig (fetch Tweets)
|
||||
- ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)
|
||||
- Fetch Tweet Candidates
|
||||
- ScoredTweetsInNetworkCandidatePipelineConfig
|
||||
- ScoredTweetsCrMixerCandidatePipelineConfig
|
||||
- ScoredTweetsUtegCandidatePipelineConfig
|
||||
- ScoredTweetsFrsCandidatePipelineConfig
|
||||
- Feature Hydration and Scoring
|
||||
- ScoredTweetsScoringPipelineConfig
|
||||
- ForYouConversationServiceCandidatePipelineConfig (backup reverse chron pipeline in case Scored Tweets fails)
|
||||
- ForYouAdsCandidatePipelineConfig (fetch ads)
|
||||
- ForYouWhoToFollowCandidatePipelineConfig (fetch users to recommend)
|
||||
|
||||
### Following
|
||||
|
||||
- FollowingProductPipelineConfig
|
||||
- FollowingMixerPipelineConfig
|
||||
- FollowingEarlybirdCandidatePipelineConfig (fetch tweets from Search Index)
|
||||
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
|
||||
- FollowingAdsCandidatePipelineConfig (fetch ads)
|
||||
- FollowingWhoToFollowCandidatePipelineConfig (fetch users to recommend)
|
||||
|
||||
### Lists
|
||||
|
||||
- ListTweetsProductPipelineConfig
|
||||
- ListTweetsMixerPipelineConfig
|
||||
- ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)
|
||||
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
|
||||
- ListTweetsAdsCandidatePipelineConfig (fetch ads)
|
||||
|
@ -0,0 +1,46 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/com/google/inject:guice",
|
||||
"3rdparty/jvm/javax/inject:javax.inject",
|
||||
"3rdparty/jvm/net/codingwell:scala-guice",
|
||||
"3rdparty/jvm/org/slf4j:slf4j-api",
|
||||
"finagle/finagle-core/src/main",
|
||||
"finagle/finagle-http/src/main/scala",
|
||||
"finagle/finagle-thriftmux/src/main/scala",
|
||||
"finatra-internal/mtls-http/src/main/scala",
|
||||
"finatra-internal/mtls-thriftmux/src/main/scala",
|
||||
"finatra/http-core/src/main/java/com/twitter/finatra/http",
|
||||
"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations",
|
||||
"finatra/inject/inject-app/src/main/scala",
|
||||
"finatra/inject/inject-core/src/main/scala",
|
||||
"finatra/inject/inject-server/src/main/scala",
|
||||
"finatra/inject/inject-utils/src/main/scala",
|
||||
"home-mixer/server/src/main/resources",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/controller",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/module",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product",
|
||||
"home-mixer/thrift/src/main/thrift:thrift-scala",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter",
|
||||
"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/render:thrift-scala",
|
||||
"stringcenter/client",
|
||||
"stringcenter/client/src/main/java",
|
||||
"stringcenter/client/src/main/scala/com/twitter/stringcenter/client",
|
||||
"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms/view",
|
||||
"timelines/src/main/scala/com/twitter/timelines/config",
|
||||
"timelines/src/main/scala/com/twitter/timelines/features/app",
|
||||
"twitter-server-internal",
|
||||
"twitter-server/server/src/main/scala",
|
||||
"util/util-app/src/main/scala",
|
||||
"util/util-core:scala",
|
||||
"util/util-slf4j-api/src/main/scala",
|
||||
],
|
||||
)
|
@ -0,0 +1,18 @@
|
||||
package com.twitter.home_mixer
|
||||
|
||||
import com.twitter.finatra.http.routing.HttpWarmup
|
||||
import com.twitter.finatra.httpclient.RequestBuilder._
|
||||
import com.twitter.inject.Logging
|
||||
import com.twitter.inject.utils.Handler
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeMixerHttpServerWarmupHandler @Inject() (warmup: HttpWarmup) extends Handler with Logging {
|
||||
|
||||
override def handle(): Unit = {
|
||||
Try(warmup.send(get("/admin/product-mixer/product-pipelines"), admin = true)())
|
||||
.onFailure(e => error(e.getMessage, e))
|
||||
}
|
||||
}
|
@ -0,0 +1,118 @@
|
||||
package com.twitter.home_mixer
|
||||
|
||||
import com.google.inject.Module
|
||||
import com.twitter.finagle.Filter
|
||||
import com.twitter.finatra.annotations.DarkTrafficFilterType
|
||||
import com.twitter.finatra.http.HttpServer
|
||||
import com.twitter.finatra.http.routing.HttpRouter
|
||||
import com.twitter.finatra.mtls.http.{Mtls => HttpMtls}
|
||||
import com.twitter.finatra.mtls.thriftmux.Mtls
|
||||
import com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule
|
||||
import com.twitter.finatra.thrift.ThriftServer
|
||||
import com.twitter.finatra.thrift.filters._
|
||||
import com.twitter.finatra.thrift.routing.ThriftRouter
|
||||
import com.twitter.home_mixer.controller.HomeThriftController
|
||||
import com.twitter.home_mixer.module._
|
||||
import com.twitter.home_mixer.param.GlobalParamConfigModule
|
||||
import com.twitter.home_mixer.product.HomeMixerProductModule
|
||||
import com.twitter.home_mixer.{thriftscala => st}
|
||||
import com.twitter.product_mixer.component_library.module.AccountRecommendationsMixerModule
|
||||
import com.twitter.product_mixer.component_library.module.CrMixerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.DarkTrafficFilterModule
|
||||
import com.twitter.product_mixer.component_library.module.EarlybirdModule
|
||||
import com.twitter.product_mixer.component_library.module.ExploreRankerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.GizmoduckClientModule
|
||||
import com.twitter.product_mixer.component_library.module.OnboardingTaskServiceModule
|
||||
import com.twitter.product_mixer.component_library.module.SocialGraphServiceModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineMixerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineRankerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineScorerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineServiceClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TweetImpressionStoreModule
|
||||
import com.twitter.product_mixer.component_library.module.UserSessionStoreModule
|
||||
import com.twitter.product_mixer.core.controllers.ProductMixerController
|
||||
import com.twitter.product_mixer.core.module.LoggingThrowableExceptionMapper
|
||||
import com.twitter.product_mixer.core.module.ProductMixerModule
|
||||
import com.twitter.product_mixer.core.module.StratoClientModule
|
||||
import com.twitter.product_mixer.core.module.stringcenter.ProductScopeStringCenterModule
|
||||
|
||||
object HomeMixerServerMain extends HomeMixerServer
|
||||
|
||||
class HomeMixerServer extends ThriftServer with Mtls with HttpServer with HttpMtls {
|
||||
override val name = "home-mixer-server"
|
||||
|
||||
override val modules: Seq[Module] = Seq(
|
||||
AccountRecommendationsMixerModule,
|
||||
AdvertiserBrandSafetySettingsStoreModule,
|
||||
ClientSentImpressionsPublisherModule,
|
||||
ConversationServiceModule,
|
||||
CrMixerClientModule,
|
||||
EarlybirdModule,
|
||||
ExploreRankerClientModule,
|
||||
GizmoduckClientModule,
|
||||
GlobalParamConfigModule,
|
||||
HomeAdsCandidateSourceModule,
|
||||
HomeMixerFlagsModule,
|
||||
HomeMixerProductModule,
|
||||
HomeMixerResourcesModule,
|
||||
HomeNaviModelClientModule,
|
||||
ImpressionBloomFilterModule,
|
||||
InjectionHistoryClientModule,
|
||||
FeedbackHistoryClientModule,
|
||||
ManhattanClientsModule,
|
||||
ManhattanFeatureRepositoryModule,
|
||||
ManhattanTweetImpressionStoreModule,
|
||||
MemcachedFeatureRepositoryModule,
|
||||
OnboardingTaskServiceModule,
|
||||
OptimizedStratoClientModule,
|
||||
PeopleDiscoveryServiceModule,
|
||||
ProductMixerModule,
|
||||
RealGraphInNetworkScoresModule,
|
||||
RealtimeAggregateFeatureRepositoryModule,
|
||||
ScoredTweetsMemcacheModule,
|
||||
ScribeEventPublisherModule,
|
||||
SimClustersRecentEngagementsClientModule,
|
||||
SocialGraphServiceModule,
|
||||
StaleTweetsCacheModule,
|
||||
StratoClientModule,
|
||||
ThriftFeatureRepositoryModule,
|
||||
TimelineMixerClientModule,
|
||||
TimelineRankerClientModule,
|
||||
TimelineScorerClientModule,
|
||||
TimelineServiceClientModule,
|
||||
TimelinesPersistenceStoreClientModule,
|
||||
TweetImpressionStoreModule,
|
||||
TweetyPieClientModule,
|
||||
TweetypieStaticEntitiesCacheClientModule,
|
||||
UserMetadataStoreModule,
|
||||
UserSessionStoreModule,
|
||||
new DarkTrafficFilterModule[st.HomeMixer.ReqRepServicePerEndpoint](),
|
||||
new MtlsThriftWebFormsModule[st.HomeMixer.MethodPerEndpoint](this),
|
||||
new ProductScopeStringCenterModule()
|
||||
)
|
||||
|
||||
def configureThrift(router: ThriftRouter): Unit = {
|
||||
router
|
||||
.filter[LoggingMDCFilter]
|
||||
.filter[TraceIdMDCFilter]
|
||||
.filter[ThriftMDCFilter]
|
||||
.filter[StatsFilter]
|
||||
.filter[AccessLoggingFilter]
|
||||
.filter[ExceptionMappingFilter]
|
||||
.filter[Filter.TypeAgnostic, DarkTrafficFilterType]
|
||||
.exceptionMapper[LoggingThrowableExceptionMapper]
|
||||
.exceptionMapper[PipelineFailureExceptionMapper]
|
||||
.add[HomeThriftController]
|
||||
}
|
||||
|
||||
override def configureHttp(router: HttpRouter): Unit =
|
||||
router.add(
|
||||
ProductMixerController[st.HomeMixer.MethodPerEndpoint](
|
||||
this.injector,
|
||||
st.HomeMixer.ExecutePipeline))
|
||||
|
||||
override protected def warmup(): Unit = {
|
||||
handle[HomeMixerThriftServerWarmupHandler]()
|
||||
handle[HomeMixerHttpServerWarmupHandler]()
|
||||
}
|
||||
}
|
@ -0,0 +1,73 @@
|
||||
package com.twitter.home_mixer
|
||||
|
||||
import com.twitter.finagle.thrift.ClientId
|
||||
import com.twitter.finatra.thrift.routing.ThriftWarmup
|
||||
import com.twitter.home_mixer.{thriftscala => st}
|
||||
import com.twitter.inject.Logging
|
||||
import com.twitter.inject.utils.Handler
|
||||
import com.twitter.product_mixer.core.{thriftscala => pt}
|
||||
import com.twitter.scrooge.Request
|
||||
import com.twitter.scrooge.Response
|
||||
import com.twitter.util.Return
|
||||
import com.twitter.util.Throw
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeMixerThriftServerWarmupHandler @Inject() (warmup: ThriftWarmup)
|
||||
extends Handler
|
||||
with Logging {
|
||||
|
||||
private val clientId = ClientId("thrift-warmup-client")
|
||||
|
||||
def handle(): Unit = {
|
||||
val testIds = Seq(1, 2, 3)
|
||||
try {
|
||||
clientId.asCurrent {
|
||||
testIds.foreach { id =>
|
||||
val warmupReq = warmupQuery(id)
|
||||
info(s"Sending warm-up request to service with query: $warmupReq")
|
||||
warmup.sendRequest(
|
||||
method = st.HomeMixer.GetUrtResponse,
|
||||
req = Request(st.HomeMixer.GetUrtResponse.Args(warmupReq)))(assertWarmupResponse)
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
case e: Throwable => error(e.getMessage, e)
|
||||
}
|
||||
info("Warm-up done.")
|
||||
}
|
||||
|
||||
private def warmupQuery(userId: Long): st.HomeMixerRequest = {
|
||||
val clientContext = pt.ClientContext(
|
||||
userId = Some(userId),
|
||||
guestId = None,
|
||||
appId = Some(12345L),
|
||||
ipAddress = Some("0.0.0.0"),
|
||||
userAgent = Some("FAKE_USER_AGENT_FOR_WARMUPS"),
|
||||
countryCode = Some("US"),
|
||||
languageCode = Some("en"),
|
||||
isTwoffice = None,
|
||||
userRoles = None,
|
||||
deviceId = Some("FAKE_DEVICE_ID_FOR_WARMUPS")
|
||||
)
|
||||
st.HomeMixerRequest(
|
||||
clientContext = clientContext,
|
||||
product = st.Product.Following,
|
||||
productContext = Some(st.ProductContext.Following(st.Following())),
|
||||
maxResults = Some(3)
|
||||
)
|
||||
}
|
||||
|
||||
private def assertWarmupResponse(
|
||||
result: Try[Response[st.HomeMixer.GetUrtResponse.SuccessType]]
|
||||
): Unit = {
|
||||
result match {
|
||||
case Return(_) => // ok
|
||||
case Throw(exception) =>
|
||||
warn("Error performing warm-up request.")
|
||||
error(exception.getMessage, exception)
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,36 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/query_transformer",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
|
||||
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
|
||||
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
|
||||
],
|
||||
exports = [
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
|
||||
],
|
||||
)
|
@ -0,0 +1,107 @@
|
||||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.SocialGraphServiceFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter
|
||||
import com.twitter.home_mixer.functional_component.filter.PredicateFeatureFilter
|
||||
import com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSourceRequest
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
|
||||
import com.twitter.product_mixer.component_library.filter.FeatureFilter
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
|
||||
|
||||
/**
|
||||
* Candidate Pipeline Config that fetches tweets from the Conversation Service Candidate Source
|
||||
*/
|
||||
class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
|
||||
conversationServiceCandidateSource: ConversationServiceCandidateSource,
|
||||
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
|
||||
socialGraphServiceFeatureHydrator: SocialGraphServiceFeatureHydrator,
|
||||
namesFeatureHydrator: NamesFeatureHydrator,
|
||||
override val gates: Seq[BaseGate[Query]],
|
||||
override val decorator: Option[CandidateDecorator[Query, TweetCandidate]])
|
||||
extends DependentCandidatePipelineConfig[
|
||||
Query,
|
||||
ConversationServiceCandidateSourceRequest,
|
||||
TweetWithConversationMetadata,
|
||||
TweetCandidate
|
||||
] {
|
||||
|
||||
override val identifier: CandidatePipelineIdentifier =
|
||||
CandidatePipelineIdentifier("ConversationService")
|
||||
|
||||
private val TweetypieHydratedFilterId = "TweetypieHydrated"
|
||||
private val QuotedTweetDroppedFilterId = "QuotedTweetDropped"
|
||||
|
||||
override val candidateSource: BaseCandidateSource[
|
||||
ConversationServiceCandidateSourceRequest,
|
||||
TweetWithConversationMetadata
|
||||
] = conversationServiceCandidateSource
|
||||
|
||||
override val queryTransformer: DependentCandidatePipelineQueryTransformer[
|
||||
Query,
|
||||
ConversationServiceCandidateSourceRequest
|
||||
] = { (_, candidates) =>
|
||||
val tweetsWithConversationMetadata = candidates.map { candidate =>
|
||||
TweetWithConversationMetadata(
|
||||
tweetId = candidate.candidateIdLong,
|
||||
userId = None,
|
||||
sourceTweetId = None,
|
||||
sourceUserId = None,
|
||||
inReplyToTweetId = None,
|
||||
conversationId = None,
|
||||
ancestors = Seq.empty
|
||||
)
|
||||
}
|
||||
ConversationServiceCandidateSourceRequest(tweetsWithConversationMetadata)
|
||||
}
|
||||
|
||||
override val featuresFromCandidateSourceTransformers: Seq[
|
||||
CandidateFeatureTransformer[TweetWithConversationMetadata]
|
||||
] = Seq(ConversationServiceResponseFeatureTransformer)
|
||||
|
||||
override val resultTransformer: CandidatePipelineResultsTransformer[
|
||||
TweetWithConversationMetadata,
|
||||
TweetCandidate
|
||||
] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }
|
||||
|
||||
override val preFilterFeatureHydrationPhase1: Seq[
|
||||
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
|
||||
] = Seq(tweetypieFeatureHydrator, socialGraphServiceFeatureHydrator)
|
||||
|
||||
override def filters: Seq[Filter[Query, TweetCandidate]] = Seq(
|
||||
RetweetDeduplicationFilter,
|
||||
FeatureFilter.fromFeature(FilterIdentifier(TweetypieHydratedFilterId), IsHydratedFeature),
|
||||
PredicateFeatureFilter.fromPredicate(
|
||||
FilterIdentifier(QuotedTweetDroppedFilterId),
|
||||
shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }
|
||||
),
|
||||
InvalidConversationModuleFilter
|
||||
)
|
||||
|
||||
override val postFilterFeatureHydration: Seq[
|
||||
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
|
||||
] = Seq(namesFeatureHydrator)
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),
|
||||
HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()
|
||||
)
|
||||
}
|
@ -0,0 +1,34 @@
|
||||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.SocialGraphServiceFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class ConversationServiceCandidatePipelineConfigBuilder[Query <: PipelineQuery] @Inject() (
|
||||
conversationServiceCandidateSource: ConversationServiceCandidateSource,
|
||||
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
|
||||
socialGraphServiceFeatureHydrator: SocialGraphServiceFeatureHydrator,
|
||||
namesFeatureHydrator: NamesFeatureHydrator) {
|
||||
|
||||
def build(
|
||||
gates: Seq[BaseGate[Query]] = Seq.empty,
|
||||
decorator: Option[CandidateDecorator[Query, TweetCandidate]] = None
|
||||
): ConversationServiceCandidatePipelineConfig[Query] = {
|
||||
new ConversationServiceCandidatePipelineConfig(
|
||||
conversationServiceCandidateSource,
|
||||
tweetypieFeatureHydrator,
|
||||
socialGraphServiceFeatureHydrator,
|
||||
namesFeatureHydrator,
|
||||
gates,
|
||||
decorator
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,39 @@
|
||||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
|
||||
object ConversationServiceResponseFeatureTransformer
|
||||
extends CandidateFeatureTransformer[TweetWithConversationMetadata] {
|
||||
|
||||
override val identifier: TransformerIdentifier =
|
||||
TransformerIdentifier("ConversationServiceResponse")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
IsRetweetFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature,
|
||||
ConversationModuleFocalTweetIdFeature,
|
||||
AncestorsFeature,
|
||||
SuggestTypeFeature
|
||||
)
|
||||
|
||||
override def transform(candidate: TweetWithConversationMetadata): FeatureMap = FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, candidate.userId)
|
||||
.add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)
|
||||
.add(IsRetweetFeature, candidate.sourceTweetId.isDefined)
|
||||
.add(SourceTweetIdFeature, candidate.sourceTweetId)
|
||||
.add(SourceUserIdFeature, candidate.sourceUserId)
|
||||
.add(ConversationModuleFocalTweetIdFeature, candidate.conversationId)
|
||||
.add(AncestorsFeature, candidate.ancestors)
|
||||
.add(SuggestTypeFeature, Some(SuggestType.RankedOrganicTweet))
|
||||
.build()
|
||||
}
|
@ -0,0 +1,84 @@
|
||||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.functional_component.candidate_source.StaleTweetsCacheCandidateSource
|
||||
import com.twitter.home_mixer.functional_component.decorator.HomeFeedbackActionInfoBuilder
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.query_transformer.EditedTweetsCandidatePipelineQueryTransformer
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.EmptyClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineFocalTweetSafetyLevel
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Candidate Pipeline Config that fetches edited tweets from the Stale Tweets Cache
|
||||
*/
|
||||
@Singleton
|
||||
case class EditedTweetsCandidatePipelineConfig @Inject() (
|
||||
staleTweetsCacheCandidateSource: StaleTweetsCacheCandidateSource,
|
||||
namesFeatureHydrator: NamesFeatureHydrator,
|
||||
homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)
|
||||
extends DependentCandidatePipelineConfig[
|
||||
PipelineQuery,
|
||||
Seq[Long],
|
||||
Long,
|
||||
TweetCandidate
|
||||
] {
|
||||
|
||||
override val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier("EditedTweets")
|
||||
|
||||
override val candidateSource: BaseCandidateSource[Seq[Long], Long] =
|
||||
staleTweetsCacheCandidateSource
|
||||
|
||||
override val queryTransformer: CandidatePipelineQueryTransformer[
|
||||
PipelineQuery,
|
||||
Seq[Long]
|
||||
] = EditedTweetsCandidatePipelineQueryTransformer
|
||||
|
||||
override val resultTransformer: CandidatePipelineResultsTransformer[
|
||||
Long,
|
||||
TweetCandidate
|
||||
] = { candidate => TweetCandidate(id = candidate) }
|
||||
|
||||
override val postFilterFeatureHydration: Seq[
|
||||
BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]
|
||||
] = Seq(namesFeatureHydrator)
|
||||
|
||||
override val decorator: Option[CandidateDecorator[PipelineQuery, TweetCandidate]] = {
|
||||
val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](
|
||||
clientEventInfoBuilder = EmptyClientEventInfoBuilder,
|
||||
entryIdToReplaceBuilder = Some((_, candidate, _) =>
|
||||
Some(s"${TweetItem.TweetEntryNamespace}-${candidate.id.toString}")),
|
||||
contextualTweetRefBuilder = Some(
|
||||
ContextualTweetRefBuilder(
|
||||
TweetHydrationContext(
|
||||
// Apply safety level that includes canonical VF treatments that apply regardless of context.
|
||||
safetyLevelOverride = Some(TimelineFocalTweetSafetyLevel),
|
||||
outerTweetContext = None
|
||||
)
|
||||
)
|
||||
),
|
||||
feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)
|
||||
)
|
||||
|
||||
Some(UrtItemCandidateDecorator(tweetItemBuilder))
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.5, 50, 60, 60)
|
||||
)
|
||||
}
|
@ -0,0 +1,123 @@
|
||||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.functional_component.gate.RequestContextNotGate
|
||||
import com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature
|
||||
import com.twitter.home_mixer.model.request.DeviceContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.DurationParamBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.ShowAlertCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertColorConfigurationBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertDisplayLocationBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertIconDisplayInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.gate.FeatureGate
|
||||
import com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.StaticCandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.configapi.StaticParam
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseDurationBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.NewTweets
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Top
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.UpArrow
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.color.TwitterBlueRosettaColor
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.color.WhiteRosettaColor
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
|
||||
import com.twitter.util.Duration
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Candidate Pipeline Config that creates the New Tweets Pill
|
||||
*/
|
||||
@Singleton
|
||||
class NewTweetsPillCandidatePipelineConfig[Query <: PipelineQuery with HasDeviceContext] @Inject() (
|
||||
) extends DependentCandidatePipelineConfig[
|
||||
Query,
|
||||
Unit,
|
||||
ShowAlertCandidate,
|
||||
ShowAlertCandidate
|
||||
] {
|
||||
import NewTweetsPillCandidatePipelineConfig._
|
||||
|
||||
override val identifier: CandidatePipelineIdentifier =
|
||||
CandidatePipelineIdentifier("NewTweetsPill")
|
||||
|
||||
override val gates: Seq[Gate[Query]] = Seq(
|
||||
RequestContextNotGate(Seq(DeviceContext.RequestContext.PullToRefresh)),
|
||||
FeatureGate.fromFeature(GetNewerFeature)
|
||||
)
|
||||
|
||||
override val candidateSource: CandidateSource[Unit, ShowAlertCandidate] =
|
||||
StaticCandidateSource(
|
||||
CandidateSourceIdentifier(identifier.name),
|
||||
Seq(ShowAlertCandidate(id = identifier.name, userIds = Seq.empty))
|
||||
)
|
||||
|
||||
override val queryTransformer: CandidatePipelineQueryTransformer[Query, Unit] = { _ => Unit }
|
||||
|
||||
override val resultTransformer: CandidatePipelineResultsTransformer[
|
||||
ShowAlertCandidate,
|
||||
ShowAlertCandidate
|
||||
] = { candidate => candidate }
|
||||
|
||||
override val decorator: Option[CandidateDecorator[Query, ShowAlertCandidate]] = {
|
||||
val triggerDelayBuilder = new BaseDurationBuilder[Query] {
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidate: ShowAlertCandidate,
|
||||
features: FeatureMap
|
||||
): Option[Duration] = {
|
||||
val delay = query.deviceContext.flatMap(_.requestContextValue) match {
|
||||
case Some(DeviceContext.RequestContext.TweetSelfThread) => 0.millis
|
||||
case Some(DeviceContext.RequestContext.ManualRefresh) => 0.millis
|
||||
case _ => TriggerDelay
|
||||
}
|
||||
|
||||
Some(delay)
|
||||
}
|
||||
}
|
||||
|
||||
val homeShowAlertCandidateBuilder = ShowAlertCandidateUrtItemBuilder(
|
||||
alertType = NewTweets,
|
||||
colorConfigBuilder = StaticShowAlertColorConfigurationBuilder(DefaultColorConfig),
|
||||
displayLocationBuilder = StaticShowAlertDisplayLocationBuilder(Top),
|
||||
triggerDelayBuilder = Some(triggerDelayBuilder),
|
||||
displayDurationBuilder = Some(DurationParamBuilder(StaticParam(DisplayDuration))),
|
||||
iconDisplayInfoBuilder = Some(StaticShowAlertIconDisplayInfoBuilder(DefaultIconDisplayInfo))
|
||||
)
|
||||
|
||||
Some(UrtItemCandidateDecorator(homeShowAlertCandidateBuilder))
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),
|
||||
HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()
|
||||
)
|
||||
}
|
||||
|
||||
object NewTweetsPillCandidatePipelineConfig {
|
||||
val DefaultColorConfig: ShowAlertColorConfiguration = ShowAlertColorConfiguration(
|
||||
background = TwitterBlueRosettaColor,
|
||||
text = WhiteRosettaColor,
|
||||
border = Some(WhiteRosettaColor)
|
||||
)
|
||||
|
||||
val DefaultIconDisplayInfo: ShowAlertIconDisplayInfo =
|
||||
ShowAlertIconDisplayInfo(icon = UpArrow, tint = WhiteRosettaColor)
|
||||
|
||||
// Unlimited display time (until user takes action)
|
||||
val DisplayDuration = -1.millisecond
|
||||
val TriggerDelay = 4.minutes
|
||||
}
|
@ -0,0 +1,34 @@
|
||||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
|
||||
import com.twitter.timelineservice.{thriftscala => t}
|
||||
|
||||
object TimelineServiceResponseFeatureTransformer extends CandidateFeatureTransformer[t.Tweet] {
|
||||
|
||||
override val identifier: TransformerIdentifier = TransformerIdentifier("TimelineServiceResponse")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
IsRetweetFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature,
|
||||
)
|
||||
|
||||
override def transform(candidate: t.Tweet): FeatureMap = FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, candidate.userId)
|
||||
.add(InReplyToTweetIdFeature, candidate.inReplyToStatusId)
|
||||
.add(IsRetweetFeature, candidate.sourceStatusId.isDefined)
|
||||
.add(SourceTweetIdFeature, candidate.sourceStatusId)
|
||||
.add(SourceUserIdFeature, candidate.sourceUserId)
|
||||
.build()
|
||||
}
|
@ -0,0 +1,20 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
|
||||
"home-mixer/thrift/src/main/thrift:thrift-scala",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt",
|
||||
"snowflake/src/main/scala/com/twitter/snowflake/id",
|
||||
"src/thrift/com/twitter/context:twitter-context-scala",
|
||||
"src/thrift/com/twitter/timelines/render:thrift-scala",
|
||||
"twitter-context/src/main/scala",
|
||||
],
|
||||
)
|
@ -0,0 +1,50 @@
|
||||
package com.twitter.home_mixer.controller
|
||||
|
||||
import com.twitter.finatra.thrift.Controller
|
||||
import com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller
|
||||
import com.twitter.home_mixer.model.request.HomeMixerRequest
|
||||
import com.twitter.home_mixer.service.ScoredTweetsService
|
||||
import com.twitter.home_mixer.{thriftscala => t}
|
||||
import com.twitter.product_mixer.core.controllers.DebugTwitterContext
|
||||
import com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder
|
||||
import com.twitter.product_mixer.core.service.urt.UrtService
|
||||
import com.twitter.snowflake.id.SnowflakeId
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.configapi.Params
|
||||
import javax.inject.Inject
|
||||
|
||||
class HomeThriftController @Inject() (
|
||||
homeRequestUnmarshaller: HomeMixerRequestUnmarshaller,
|
||||
urtService: UrtService,
|
||||
scoredTweetsService: ScoredTweetsService,
|
||||
paramsBuilder: ParamsBuilder)
|
||||
extends Controller(t.HomeMixer)
|
||||
with DebugTwitterContext {
|
||||
|
||||
handle(t.HomeMixer.GetUrtResponse) { args: t.HomeMixer.GetUrtResponse.Args =>
|
||||
val request = homeRequestUnmarshaller(args.request)
|
||||
val params = buildParams(request)
|
||||
Stitch.run(urtService.getUrtResponse[HomeMixerRequest](request, params))
|
||||
}
|
||||
|
||||
handle(t.HomeMixer.GetScoredTweetsResponse) { args: t.HomeMixer.GetScoredTweetsResponse.Args =>
|
||||
val request = homeRequestUnmarshaller(args.request)
|
||||
val params = buildParams(request)
|
||||
withDebugTwitterContext(request.clientContext) {
|
||||
Stitch.run(scoredTweetsService.getScoredTweetsResponse[HomeMixerRequest](request, params))
|
||||
}
|
||||
}
|
||||
|
||||
private def buildParams(request: HomeMixerRequest): Params = {
|
||||
val userAgeOpt = request.clientContext.userId.map { userId =>
|
||||
SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)
|
||||
}
|
||||
val fsCustomMapInput = userAgeOpt.map("account_age_in_days" -> _).toMap
|
||||
paramsBuilder.build(
|
||||
clientContext = request.clientContext,
|
||||
product = request.product,
|
||||
featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),
|
||||
fsCustomMapInput = fsCustomMapInput
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,22 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/javax/inject:javax.inject",
|
||||
"finagle/finagle-memcached/src/main/scala",
|
||||
"finatra/inject/inject-core/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala",
|
||||
"src/thrift/com/twitter/search:earlybird-scala",
|
||||
"stitch/stitch-timelineservice/src/main/scala",
|
||||
"strato/config/columns/recommendations/similarity:similarity-strato-client",
|
||||
"strato/src/main/scala/com/twitter/strato/client",
|
||||
],
|
||||
exports = [
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
|
||||
],
|
||||
)
|
@ -0,0 +1,44 @@
|
||||
package com.twitter.home_mixer.functional_component.candidate_source
|
||||
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.search.earlybird.{thriftscala => t}
|
||||
import com.twitter.stitch.Stitch
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
case object EarlybirdResponseTruncatedFeature
|
||||
extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Boolean] {
|
||||
override val defaultValue: Boolean = false
|
||||
}
|
||||
|
||||
case object EarlybirdBottomTweetFeature
|
||||
extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Option[Long]] {
|
||||
override val defaultValue: Option[Long] = None
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class EarlybirdCandidateSource @Inject() (
|
||||
earlybird: t.EarlybirdService.MethodPerEndpoint)
|
||||
extends CandidateSourceWithExtractedFeatures[t.EarlybirdRequest, t.ThriftSearchResult] {
|
||||
|
||||
override val identifier = CandidateSourceIdentifier("Earlybird")
|
||||
|
||||
override def apply(
|
||||
request: t.EarlybirdRequest
|
||||
): Stitch[CandidatesWithSourceFeatures[t.ThriftSearchResult]] = {
|
||||
Stitch.callFuture(earlybird.search(request)).map { response =>
|
||||
val candidates = response.searchResults.map(_.results).getOrElse(Seq.empty)
|
||||
|
||||
val features = FeatureMapBuilder()
|
||||
.add(EarlybirdResponseTruncatedFeature, candidates.size == request.searchQuery.numResults)
|
||||
.add(EarlybirdBottomTweetFeature, candidates.lastOption.map(_.id))
|
||||
.build()
|
||||
|
||||
CandidatesWithSourceFeatures(candidates, features)
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,34 @@
|
||||
package com.twitter.home_mixer.functional_component.candidate_source
|
||||
|
||||
import com.twitter.hermit.candidate.{thriftscala => t}
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.strato.client.Fetcher
|
||||
import com.twitter.strato.generated.client.recommendations.similarity.SimilarUsersBySimsOnUserClientColumn
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class SimilarityBasedUsersCandidateSource @Inject() (
|
||||
similarUsersBySimsOnUserClientColumn: SimilarUsersBySimsOnUserClientColumn)
|
||||
extends CandidateSource[Seq[Long], t.Candidate] {
|
||||
|
||||
override val identifier: CandidateSourceIdentifier =
|
||||
CandidateSourceIdentifier("SimilarityBasedUsers")
|
||||
|
||||
private val fetcher: Fetcher[Long, Unit, t.Candidates] =
|
||||
similarUsersBySimsOnUserClientColumn.fetcher
|
||||
|
||||
override def apply(request: Seq[Long]): Stitch[Seq[t.Candidate]] = {
|
||||
Stitch
|
||||
.collect {
|
||||
request.map { userId =>
|
||||
fetcher.fetch(userId, Unit).map { result =>
|
||||
result.v.map(_.candidates).getOrElse(Seq.empty)
|
||||
}
|
||||
}
|
||||
}.map(_.flatten)
|
||||
}
|
||||
}
|
@ -0,0 +1,30 @@
|
||||
package com.twitter.home_mixer.functional_component.candidate_source
|
||||
|
||||
import com.google.inject.name.Named
|
||||
import com.twitter.finagle.memcached.{Client => MemcachedClient}
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.StaleTweetsCache
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.stitch.Stitch
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class StaleTweetsCacheCandidateSource @Inject() (
|
||||
@Named(StaleTweetsCache) staleTweetsCache: MemcachedClient)
|
||||
extends CandidateSource[Seq[Long], Long] {
|
||||
|
||||
override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier("StaleTweetsCache")
|
||||
|
||||
private val StaleTweetsCacheKeyPrefix = "v1_"
|
||||
|
||||
override def apply(request: Seq[Long]): Stitch[Seq[Long]] = {
|
||||
val keys = request.map(StaleTweetsCacheKeyPrefix + _)
|
||||
|
||||
Stitch.callFuture(staleTweetsCache.get(keys).map { tweets =>
|
||||
tweets.map {
|
||||
case (k, _) => k.replaceFirst(StaleTweetsCacheKeyPrefix, "").toLong
|
||||
}.toSeq
|
||||
})
|
||||
}
|
||||
}
|
@ -0,0 +1,34 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class AuthorChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
CandidatesUtil.getOriginalAuthorId(candidateFeatures).flatMap { authorId =>
|
||||
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
|
||||
userId = authorId,
|
||||
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
|
||||
promptExternalString = externalStrings.showFewerTweetsString,
|
||||
confirmationExternalString = externalStrings.showFewerTweetsConfirmationString,
|
||||
engagementType = t.FeedbackEngagementType.Tweet,
|
||||
stringCenter = stringCenter,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,32 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"finagle/finagle-core/src/main",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"joinkey/src/main/scala/com/twitter/joinkey/context",
|
||||
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/scala/com/twitter/suggests/controller_data",
|
||||
"src/thrift/com/twitter/suggests/controller_data:controller_data-scala",
|
||||
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
|
||||
"stringcenter/client",
|
||||
"stringcenter/client/src/main/java",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/translation",
|
||||
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
|
||||
],
|
||||
)
|
@ -0,0 +1,54 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class BlockUserChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
val userIdOpt =
|
||||
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
|
||||
candidateFeatures.getOrElse(SourceUserIdFeature, None)
|
||||
else candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
userIdOpt.flatMap { userId =>
|
||||
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
|
||||
val userScreenNameOpt = screenNamesMap.get(userId)
|
||||
userScreenNameOpt.map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
externalStrings.blockUserString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(prompt),
|
||||
confirmation = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = Some(BottomSheet),
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.No),
|
||||
richBehavior = Some(RichFeedbackBehaviorBlockUser(userId)),
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,88 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DontLike
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.common.{thriftscala => tlc}
|
||||
import com.twitter.timelineservice.model.FeedbackInfo
|
||||
import com.twitter.timelineservice.model.FeedbackMetadata
|
||||
import com.twitter.timelineservice.{thriftscala => tls}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class DontLikeFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
authorChildFeedbackActionBuilder: AuthorChildFeedbackActionBuilder,
|
||||
retweeterChildFeedbackActionBuilder: RetweeterChildFeedbackActionBuilder,
|
||||
notRelevantChildFeedbackActionBuilder: NotRelevantChildFeedbackActionBuilder,
|
||||
unfollowUserChildFeedbackActionBuilder: UnfollowUserChildFeedbackActionBuilder,
|
||||
muteUserChildFeedbackActionBuilder: MuteUserChildFeedbackActionBuilder,
|
||||
blockUserChildFeedbackActionBuilder: BlockUserChildFeedbackActionBuilder,
|
||||
reportTweetChildFeedbackActionBuilder: ReportTweetChildFeedbackActionBuilder) {
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackAction] = {
|
||||
CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>
|
||||
val feedbackEntities = Seq(
|
||||
tlc.FeedbackEntity.TweetId(candidate.id),
|
||||
tlc.FeedbackEntity.UserId(authorId)
|
||||
)
|
||||
val feedbackMetadata = FeedbackMetadata(
|
||||
engagementType = None,
|
||||
entityIds = feedbackEntities,
|
||||
ttl = Some(30.days)
|
||||
)
|
||||
val feedbackUrl = FeedbackInfo.feedbackUrl(
|
||||
feedbackType = tls.FeedbackType.DontLike,
|
||||
feedbackMetadata = feedbackMetadata,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
val childFeedbackActions = if (query.params(EnableNahFeedbackInfoParam)) {
|
||||
Seq(
|
||||
unfollowUserChildFeedbackActionBuilder(candidateFeatures),
|
||||
muteUserChildFeedbackActionBuilder(candidateFeatures),
|
||||
blockUserChildFeedbackActionBuilder(candidateFeatures),
|
||||
reportTweetChildFeedbackActionBuilder(candidate)
|
||||
).flatten
|
||||
} else {
|
||||
Seq(
|
||||
authorChildFeedbackActionBuilder(candidateFeatures),
|
||||
retweeterChildFeedbackActionBuilder(candidateFeatures),
|
||||
notRelevantChildFeedbackActionBuilder(candidate, candidateFeatures)
|
||||
).flatten
|
||||
}
|
||||
|
||||
FeedbackAction(
|
||||
feedbackType = DontLike,
|
||||
prompt = Some(stringCenter.prepare(externalStrings.dontLikeString)),
|
||||
confirmation = Some(stringCenter.prepare(externalStrings.dontLikeConfirmationString)),
|
||||
childFeedbackActions =
|
||||
if (childFeedbackActions.nonEmpty) Some(childFeedbackActions) else None,
|
||||
feedbackUrl = Some(feedbackUrl),
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(Frown),
|
||||
richBehavior = None,
|
||||
subprompt = None,
|
||||
encodedFeedbackRequest = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,119 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.stringcenter.client.core.ExternalString
|
||||
|
||||
private[decorator] case class SocialContextIdAndScreenName(
|
||||
socialContextId: Long,
|
||||
screenName: String)
|
||||
|
||||
object EngagerSocialContextBuilder {
|
||||
private val UserIdRequestParamName = "user_id"
|
||||
private val DirectInjectionContentSourceRequestParamName = "dis"
|
||||
private val DirectInjectionIdRequestParamName = "diid"
|
||||
private val DirectInjectionContentSourceSocialProofUsers = "socialproofusers"
|
||||
private val SocialProofUrl = ""
|
||||
}
|
||||
|
||||
case class EngagerSocialContextBuilder(
|
||||
contextType: GeneralContextType,
|
||||
stringCenter: StringCenter,
|
||||
oneUserString: ExternalString,
|
||||
twoUsersString: ExternalString,
|
||||
moreUsersString: ExternalString,
|
||||
timelineTitle: ExternalString) {
|
||||
import EngagerSocialContextBuilder._
|
||||
|
||||
def apply(
|
||||
socialContextIds: Seq[Long],
|
||||
query: PipelineQuery,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
|
||||
val validSocialContextIdAndScreenNames = socialContextIds.flatMap { socialContextId =>
|
||||
realNames
|
||||
.get(socialContextId).map(screenName =>
|
||||
SocialContextIdAndScreenName(socialContextId, screenName))
|
||||
}
|
||||
|
||||
validSocialContextIdAndScreenNames match {
|
||||
case Seq(user) =>
|
||||
val socialContextString =
|
||||
stringCenter.prepare(oneUserString, Map("user" -> user.screenName))
|
||||
Some(mkOneUserSocialContext(socialContextString, user.socialContextId))
|
||||
case Seq(firstUser, secondUser) =>
|
||||
val socialContextString =
|
||||
stringCenter
|
||||
.prepare(
|
||||
twoUsersString,
|
||||
Map("user1" -> firstUser.screenName, "user2" -> secondUser.screenName))
|
||||
Some(
|
||||
mkManyUserSocialContext(
|
||||
socialContextString,
|
||||
query.getRequiredUserId,
|
||||
validSocialContextIdAndScreenNames.map(_.socialContextId)))
|
||||
|
||||
case firstUser +: otherUsers =>
|
||||
val otherUsersCount = otherUsers.size
|
||||
val socialContextString =
|
||||
stringCenter
|
||||
.prepare(
|
||||
moreUsersString,
|
||||
Map("user" -> firstUser.screenName, "count" -> otherUsersCount))
|
||||
Some(
|
||||
mkManyUserSocialContext(
|
||||
socialContextString,
|
||||
query.getRequiredUserId,
|
||||
validSocialContextIdAndScreenNames.map(_.socialContextId)))
|
||||
case _ => None
|
||||
}
|
||||
}
|
||||
|
||||
private def mkOneUserSocialContext(socialContextString: String, userId: Long): GeneralContext = {
|
||||
GeneralContext(
|
||||
contextType = contextType,
|
||||
text = socialContextString,
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = "",
|
||||
urtEndpointOptions = None
|
||||
)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
private def mkManyUserSocialContext(
|
||||
socialContextString: String,
|
||||
viewerId: Long,
|
||||
socialContextIds: Seq[Long]
|
||||
): GeneralContext = {
|
||||
GeneralContext(
|
||||
contextType = contextType,
|
||||
text = socialContextString,
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = UrtEndpoint,
|
||||
url = SocialProofUrl,
|
||||
urtEndpointOptions = Some(UrtEndpointOptions(
|
||||
requestParams = Some(Map(
|
||||
UserIdRequestParamName -> viewerId.toString,
|
||||
DirectInjectionContentSourceRequestParamName -> DirectInjectionContentSourceSocialProofUsers,
|
||||
DirectInjectionIdRequestParamName -> socialContextIds.mkString(",")
|
||||
)),
|
||||
title = Some(stringCenter.prepare(timelineTitle)),
|
||||
cacheId = None,
|
||||
subtitle = None
|
||||
))
|
||||
))
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,78 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Use '@A replied' when the root tweet is out-of-network and the reply is in network.
|
||||
*
|
||||
* This function should only be called for the root Tweet of convo modules. This is enforced by
|
||||
* [[HomeTweetSocialContextBuilder]].
|
||||
*/
|
||||
@Singleton
|
||||
case class ExtendedReplySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val extendedReplyString = externalStrings.socialContextExtendedReply
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
|
||||
// If these values are missing default to not showing an extended reply banner
|
||||
val inNetworkRoot = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
|
||||
val inNetworkFocalTweet =
|
||||
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(false)
|
||||
|
||||
if (!inNetworkRoot && inNetworkFocalTweet) {
|
||||
|
||||
val focalTweetAuthorIdOpt = candidateFeatures.getOrElse(FocalTweetAuthorIdFeature, None)
|
||||
val focalTweetRealNames =
|
||||
candidateFeatures
|
||||
.getOrElse(FocalTweetRealNamesFeature, None).getOrElse(Map.empty[Long, String])
|
||||
val focalTweetAuthorNameOpt = focalTweetAuthorIdOpt.flatMap(focalTweetRealNames.get)
|
||||
|
||||
(focalTweetAuthorIdOpt, focalTweetAuthorNameOpt) match {
|
||||
case (Some(focalTweetAuthorId), Some(focalTweetAuthorName)) =>
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = ConversationGeneralContextType,
|
||||
text = stringCenter
|
||||
.prepare(extendedReplyString, placeholders = Map("user1" -> focalTweetAuthorName)),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = "",
|
||||
urtEndpointOptions = None
|
||||
))
|
||||
))
|
||||
case _ =>
|
||||
None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,61 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.stringcenter.client.core.ExternalString
|
||||
import com.twitter.timelines.common.{thriftscala => tlc}
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
import com.twitter.timelineservice.model.FeedbackInfo
|
||||
import com.twitter.timelineservice.model.FeedbackMetadata
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
import com.twitter.timelineservice.{thriftscala => tlst}
|
||||
|
||||
object FeedbackUtil {
|
||||
|
||||
val FeedbackTtl = 30.days
|
||||
|
||||
def buildUserSeeFewerChildFeedbackAction(
|
||||
userId: Long,
|
||||
namesByUserId: Map[Long, String],
|
||||
promptExternalString: ExternalString,
|
||||
confirmationExternalString: ExternalString,
|
||||
engagementType: t.FeedbackEngagementType,
|
||||
stringCenter: StringCenter,
|
||||
injectionType: Option[st.SuggestType]
|
||||
): Option[ChildFeedbackAction] = {
|
||||
namesByUserId.get(userId).map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
promptExternalString,
|
||||
Map("user" -> userScreenName)
|
||||
)
|
||||
val confirmation = stringCenter.prepare(
|
||||
confirmationExternalString,
|
||||
Map("user" -> userScreenName)
|
||||
)
|
||||
val feedbackMetadata = FeedbackMetadata(
|
||||
engagementType = Some(engagementType),
|
||||
entityIds = Seq(tlc.FeedbackEntity.UserId(userId)),
|
||||
ttl = Some(FeedbackTtl))
|
||||
val feedbackUrl = FeedbackInfo.feedbackUrl(
|
||||
feedbackType = tlst.FeedbackType.SeeFewer,
|
||||
feedbackMetadata = feedbackMetadata,
|
||||
injectionType = injectionType
|
||||
)
|
||||
|
||||
ChildFeedbackAction(
|
||||
feedbackType = SeeFewer,
|
||||
prompt = Some(prompt),
|
||||
confirmation = Some(confirmation),
|
||||
feedbackUrl = Some(feedbackUrl),
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = None,
|
||||
richBehavior = None,
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,53 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class FollowedBySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
|
||||
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
|
||||
contextType = FollowGeneralContextType,
|
||||
stringCenter = stringCenter,
|
||||
oneUserString = externalStrings.socialContextOneUserFollowsString,
|
||||
twoUsersString = externalStrings.socialContextTwoUsersFollowString,
|
||||
moreUsersString = externalStrings.socialContextMoreUsersFollowString,
|
||||
timelineTitle = externalStrings.socialContextFollowedByTimelineTitle
|
||||
)
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
// Only apply followed-by social context for OON Tweets
|
||||
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
if (!inNetwork) {
|
||||
val validFollowedByUserIds =
|
||||
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
|
||||
engagerSocialContextBuilder(
|
||||
socialContextIds = validFollowedByUserIds,
|
||||
query = query,
|
||||
candidateFeatures = candidateFeatures
|
||||
)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,46 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.finagle.tracing.Trace
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
|
||||
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
|
||||
import com.twitter.suggests.controller_data.thriftscala.ControllerData
|
||||
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
|
||||
|
||||
case class HomeAdsClientEventDetailsBuilder(injectionType: Option[String])
|
||||
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UniversalNoun[Any],
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ClientEventDetails] = {
|
||||
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
|
||||
tweetTypesBitmap = 0L,
|
||||
traceId = Some(Trace.id.traceId.toLong),
|
||||
requestJoinId = None)
|
||||
|
||||
val serializedControllerData = HomeClientEventDetailsBuilder.ControllerDataSerializer(
|
||||
ControllerData.V2(
|
||||
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
|
||||
|
||||
val clientEventDetails = ClientEventDetails(
|
||||
conversationDetails = None,
|
||||
timelinesDetails = Some(
|
||||
TimelinesDetails(
|
||||
injectionType = injectionType,
|
||||
controllerData = Some(serializedControllerData),
|
||||
sourceData = None)),
|
||||
articleDetails = None,
|
||||
liveEventDetails = None,
|
||||
commerceDetails = None
|
||||
)
|
||||
|
||||
Some(clientEventDetails)
|
||||
}
|
||||
}
|
@ -0,0 +1,92 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.bijection.Base64String
|
||||
import com.twitter.bijection.scrooge.BinaryScalaCodec
|
||||
import com.twitter.bijection.{Injection => Serializer}
|
||||
import com.twitter.finagle.tracing.Trace
|
||||
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PositionFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.joinkey.context.RequestJoinKeyContext
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.suggests.controller_data.Home
|
||||
import com.twitter.suggests.controller_data.TweetTypeGenerator
|
||||
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
|
||||
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
|
||||
import com.twitter.suggests.controller_data.thriftscala.ControllerData
|
||||
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
|
||||
|
||||
object HomeClientEventDetailsBuilder {
|
||||
implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =
|
||||
BinaryScalaCodec(ControllerData)
|
||||
|
||||
val ControllerDataSerializer: Serializer[ControllerData, String] =
|
||||
Serializer.connect[ControllerData, Array[Byte], Base64String, String]
|
||||
|
||||
/**
|
||||
* define getRequestJoinId as a method(def) rather than a val because each new request
|
||||
* needs to call the context to update the id.
|
||||
*/
|
||||
private def getRequestJoinId(): Option[Long] =
|
||||
RequestJoinKeyContext.current.flatMap(_.requestJoinId)
|
||||
}
|
||||
|
||||
case class HomeClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](
|
||||
) extends BaseClientEventDetailsBuilder[Query, Candidate]
|
||||
with TweetTypeGenerator[FeatureMap] {
|
||||
|
||||
import HomeClientEventDetailsBuilder._
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidate: Candidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ClientEventDetails] = {
|
||||
|
||||
val tweetTypesBitmaps = mkTweetTypesBitmaps(
|
||||
Home.TweetTypeIdxMap,
|
||||
HomeTweetTypePredicates.PredicateMap,
|
||||
candidateFeatures)
|
||||
|
||||
val tweetTypesListBytes = mkItemTypesBitmapsV2(
|
||||
Home.TweetTypeIdxMap,
|
||||
HomeTweetTypePredicates.PredicateMap,
|
||||
candidateFeatures)
|
||||
|
||||
val candidateSourceId =
|
||||
candidateFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toByte)
|
||||
|
||||
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
|
||||
tweetTypesBitmap = tweetTypesBitmaps.getOrElse(0, 0L),
|
||||
tweetTypesBitmapContinued1 = tweetTypesBitmaps.get(1),
|
||||
candidateTweetSourceId = candidateSourceId,
|
||||
traceId = Some(Trace.id.traceId.toLong),
|
||||
injectedPosition = candidateFeatures.getOrElse(PositionFeature, None),
|
||||
tweetTypesListBytes = Some(tweetTypesListBytes),
|
||||
requestJoinId = getRequestJoinId(),
|
||||
)
|
||||
|
||||
val serializedControllerData = ControllerDataSerializer(
|
||||
ControllerData.V2(
|
||||
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
|
||||
|
||||
val clientEventDetails = ClientEventDetails(
|
||||
conversationDetails = None,
|
||||
timelinesDetails = Some(
|
||||
TimelinesDetails(
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None).map(_.name),
|
||||
controllerData = Some(serializedControllerData),
|
||||
sourceData = None)),
|
||||
articleDetails = None,
|
||||
liveEventDetails = None,
|
||||
commerceDetails = None
|
||||
)
|
||||
|
||||
Some(clientEventDetails)
|
||||
}
|
||||
}
|
@ -0,0 +1,49 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
|
||||
object HomeConversationServiceCandidateDecorator {
|
||||
|
||||
private val ConversationModuleNamespace = EntryNamespace("home-conversation")
|
||||
|
||||
def apply(
|
||||
homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder
|
||||
): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {
|
||||
val suggestType = st.SuggestType.RankedOrganicTweet
|
||||
val component = InjectionScribeUtil.scribeComponent(suggestType).get
|
||||
val clientEventInfoBuilder = ClientEventInfoBuilder(component)
|
||||
val tweetItemBuilder = TweetCandidateUrtItemBuilder(
|
||||
clientEventInfoBuilder = clientEventInfoBuilder,
|
||||
timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),
|
||||
feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)
|
||||
)
|
||||
|
||||
val moduleBuilder = TimelineModuleBuilder(
|
||||
entryNamespace = ConversationModuleNamespace,
|
||||
clientEventInfoBuilder = clientEventInfoBuilder,
|
||||
displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),
|
||||
metadataBuilder = Some(HomeConversationModuleMetadataBuilder())
|
||||
)
|
||||
|
||||
Some(
|
||||
UrtMultipleModulesDecorator(
|
||||
urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),
|
||||
moduleBuilder = moduleBuilder,
|
||||
groupByKey = (_, _, candidateFeatures) =>
|
||||
candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)
|
||||
))
|
||||
}
|
||||
}
|
@ -0,0 +1,54 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.model.request.ForYouProduct
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
import com.twitter.timelines.util.FeedbackMetadataSerializer
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeFeedbackActionInfoBuilder @Inject() (
|
||||
notInterestedTopicFeedbackActionBuilder: NotInterestedTopicFeedbackActionBuilder,
|
||||
dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder)
|
||||
extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackActionInfo] = {
|
||||
val supportedProduct = query.product match {
|
||||
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
|
||||
case ForYouProduct => true
|
||||
case _ => false
|
||||
}
|
||||
val isAuthoredByViewer = CandidatesUtil.isAuthoredByViewer(query, candidateFeatures)
|
||||
|
||||
if (supportedProduct && !isAuthoredByViewer) {
|
||||
val feedbackActions = Seq(
|
||||
notInterestedTopicFeedbackActionBuilder(candidateFeatures),
|
||||
dontLikeFeedbackActionBuilder(query, candidate, candidateFeatures)
|
||||
).flatten
|
||||
val feedbackMetadata = FeedbackMetadataSerializer.serialize(
|
||||
t.FeedbackMetadata(injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)))
|
||||
|
||||
Some(
|
||||
FeedbackActionInfo(
|
||||
feedbackActions = feedbackActions,
|
||||
feedbackMetadata = Some(feedbackMetadata),
|
||||
displayContext = None,
|
||||
clientEventInfo = None
|
||||
))
|
||||
} else None
|
||||
}
|
||||
}
|
@ -0,0 +1,18 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
|
||||
object HomeQueryTypePredicates {
|
||||
private[this] val QueryPredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
|
||||
("request", _ => true),
|
||||
("get_initial", _.getOrElse(GetInitialFeature, false)),
|
||||
("get_newer", _.getOrElse(GetNewerFeature, false)),
|
||||
("get_older", _.getOrElse(GetOlderFeature, false)),
|
||||
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
|
||||
("request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
|
||||
("request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false))
|
||||
)
|
||||
|
||||
val PredicateMap = QueryPredicates.toMap
|
||||
}
|
@ -0,0 +1,26 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSendScoresToClient
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
object HomeTimelinesScoreInfoBuilder
|
||||
extends BaseTimelinesScoreInfoBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val UndefinedTweetScore = -1.0
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[TimelinesScoreInfo] = {
|
||||
if (query.params(EnableSendScoresToClient)) {
|
||||
val score = candidateFeatures.getOrElse(ScoreFeature, None).getOrElse(UndefinedTweetScore)
|
||||
Some(TimelinesScoreInfo(score))
|
||||
} else None
|
||||
}
|
||||
}
|
@ -0,0 +1,44 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSocialContextParam
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class HomeTweetSocialContextBuilder @Inject() (
|
||||
likedBySocialContextBuilder: LikedBySocialContextBuilder,
|
||||
followedBySocialContextBuilder: FollowedBySocialContextBuilder,
|
||||
topicSocialContextBuilder: TopicSocialContextBuilder,
|
||||
extendedReplySocialContextBuilder: ExtendedReplySocialContextBuilder,
|
||||
receivedReplySocialContextBuilder: ReceivedReplySocialContextBuilder)
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
features: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
if (query.params(EnableSocialContextParam)) {
|
||||
features.getOrElse(ConversationModuleFocalTweetIdFeature, None) match {
|
||||
case None =>
|
||||
likedBySocialContextBuilder(query, candidate, features)
|
||||
.orElse(followedBySocialContextBuilder(query, candidate, features))
|
||||
.orElse(topicSocialContextBuilder(query, candidate, features))
|
||||
case Some(_) =>
|
||||
val conversationId = features.getOrElse(ConversationModuleIdFeature, None)
|
||||
// Only hydrate the social context into the root tweet in a conversation module
|
||||
if (conversationId.contains(candidate.id)) {
|
||||
extendedReplySocialContextBuilder(query, candidate, features)
|
||||
.orElse(receivedReplySocialContextBuilder(query, candidate, features))
|
||||
} else None
|
||||
}
|
||||
} else None
|
||||
}
|
||||
}
|
@ -0,0 +1,250 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.timelinemixer.injection.model.candidate.SemanticCoreFeatures
|
||||
import com.twitter.tweetypie.{thriftscala => tpt}
|
||||
|
||||
object HomeTweetTypePredicates {
|
||||
|
||||
/**
|
||||
* IMPORTANT: Please avoid logging tweet types that are tied to sensitive
|
||||
* internal author information / labels (e.g. blink labels, abuse labels, or geo-location).
|
||||
*/
|
||||
private[this] val CandidatePredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
|
||||
("with_candidate", _ => true),
|
||||
("retweet", _.getOrElse(IsRetweetFeature, false)),
|
||||
("reply", _.getOrElse(InReplyToTweetIdFeature, None).nonEmpty),
|
||||
("image", _.getOrElse(EarlybirdFeature, None).exists(_.hasImage)),
|
||||
("video", _.getOrElse(EarlybirdFeature, None).exists(_.hasVideo)),
|
||||
("link", _.getOrElse(EarlybirdFeature, None).exists(_.hasVisibleLink)),
|
||||
("quote", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuote.contains(true))),
|
||||
("like_social_context", _.getOrElse(NonSelfFavoritedByUserIdsFeature, Seq.empty).nonEmpty),
|
||||
("protected", _.getOrElse(EarlybirdFeature, None).exists(_.isProtected)),
|
||||
(
|
||||
"has_exclusive_conversation_author_id",
|
||||
_.getOrElse(ExclusiveConversationAuthorIdFeature, None).nonEmpty),
|
||||
("is_eligible_for_connect_boost", _.getOrElse(AuthorIsEligibleForConnectBoostFeature, false)),
|
||||
("hashtag", _.getOrElse(EarlybirdFeature, None).exists(_.numHashtags > 0)),
|
||||
("has_scheduled_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isScheduled)),
|
||||
("has_recorded_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isRecorded)),
|
||||
("is_read_from_cache", _.getOrElse(IsReadFromCacheFeature, false)),
|
||||
(
|
||||
"is_self_thread_tweet",
|
||||
_.getOrElse(ConversationFeature, None).exists(_.isSelfThreadTweet.contains(true))),
|
||||
("get_initial", _.getOrElse(GetInitialFeature, false)),
|
||||
("get_newer", _.getOrElse(GetNewerFeature, false)),
|
||||
("get_middle", _.getOrElse(GetMiddleFeature, false)),
|
||||
("get_older", _.getOrElse(GetOlderFeature, false)),
|
||||
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
|
||||
("polling", _.getOrElse(PollingFeature, false)),
|
||||
("tls_size_20_plus", _ => false),
|
||||
("near_empty", _ => false),
|
||||
("ranked_request", _ => false),
|
||||
("mutual_follow", _.getOrElse(EarlybirdFeature, None).exists(_.fromMutualFollow)),
|
||||
("has_ticketed_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasTickets)),
|
||||
("in_utis_top5", _.getOrElse(PositionFeature, None).exists(_ < 5)),
|
||||
("is_utis_pos0", _.getOrElse(PositionFeature, None).exists(_ == 0)),
|
||||
("is_utis_pos1", _.getOrElse(PositionFeature, None).exists(_ == 1)),
|
||||
("is_utis_pos2", _.getOrElse(PositionFeature, None).exists(_ == 2)),
|
||||
("is_utis_pos3", _.getOrElse(PositionFeature, None).exists(_ == 3)),
|
||||
("is_utis_pos4", _.getOrElse(PositionFeature, None).exists(_ == 4)),
|
||||
(
|
||||
"is_signup_request",
|
||||
candidate => candidate.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
|
||||
("empty_request", _ => false),
|
||||
("served_size_less_than_5", _.getOrElse(ServedSizeFeature, None).exists(_ < 5)),
|
||||
("served_size_less_than_10", _.getOrElse(ServedSizeFeature, None).exists(_ < 10)),
|
||||
("served_size_less_than_20", _.getOrElse(ServedSizeFeature, None).exists(_ < 20)),
|
||||
("served_size_less_than_50", _.getOrElse(ServedSizeFeature, None).exists(_ < 50)),
|
||||
(
|
||||
"served_size_between_50_and_100",
|
||||
_.getOrElse(ServedSizeFeature, None).exists(size => size >= 50 && size < 100)),
|
||||
("authored_by_contextual_user", _.getOrElse(AuthoredByContextualUserFeature, false)),
|
||||
("has_ancestors", _.getOrElse(AncestorsFeature, Seq.empty).nonEmpty),
|
||||
("full_scoring_succeeded", _.getOrElse(FullScoringSucceededFeature, false)),
|
||||
(
|
||||
"account_age_less_than_30_minutes",
|
||||
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
|
||||
(
|
||||
"account_age_less_than_1_day",
|
||||
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 1.day)),
|
||||
(
|
||||
"account_age_less_than_7_days",
|
||||
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 7.days)),
|
||||
(
|
||||
"directed_at_user_is_in_first_degree",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.directedAtUserIdIsInFirstDegree.contains(true))),
|
||||
("root_user_is_in_first_degree", _ => false),
|
||||
(
|
||||
"has_semantic_core_annotation",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.semanticCoreAnnotations.nonEmpty)),
|
||||
("is_request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false)),
|
||||
(
|
||||
"part_of_utt",
|
||||
_.getOrElse(EarlybirdFeature, None)
|
||||
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
|
||||
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy)))),
|
||||
("is_random_tweet", _.getOrElse(IsRandomTweetFeature, false)),
|
||||
("has_random_tweet_in_response", _.getOrElse(HasRandomTweetFeature, false)),
|
||||
("is_random_tweet_above_in_utis", _.getOrElse(IsRandomTweetAboveFeature, false)),
|
||||
("is_request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
|
||||
("viewer_is_employee", _ => false),
|
||||
("viewer_is_timelines_employee", _ => false),
|
||||
("viewer_follows_any_topics", _.getOrElse(UserFollowedTopicsCountFeature, None).exists(_ > 0)),
|
||||
(
|
||||
"has_ancestor_authored_by_viewer",
|
||||
candidate =>
|
||||
candidate
|
||||
.getOrElse(AncestorsFeature, Seq.empty).exists(ancestor =>
|
||||
candidate.getOrElse(ViewerIdFeature, 0L) == ancestor.userId)),
|
||||
("ancestor", _.getOrElse(IsAncestorCandidateFeature, false)),
|
||||
(
|
||||
"root_ancestor",
|
||||
candidate =>
|
||||
candidate.getOrElse(IsAncestorCandidateFeature, false) && candidate
|
||||
.getOrElse(InReplyToTweetIdFeature, None).isEmpty),
|
||||
(
|
||||
"deep_reply",
|
||||
candidate =>
|
||||
candidate.getOrElse(InReplyToTweetIdFeature, None).nonEmpty && candidate
|
||||
.getOrElse(AncestorsFeature, Seq.empty).size > 2),
|
||||
(
|
||||
"has_simcluster_embeddings",
|
||||
_.getOrElse(
|
||||
SimclustersTweetTopKClustersWithScoresFeature,
|
||||
Map.empty[String, Double]).nonEmpty),
|
||||
(
|
||||
"tweet_age_less_than_15_seconds",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow <= 15.seconds)),
|
||||
("is_followed_topic_tweet", _ => false),
|
||||
("is_recommended_topic_tweet", _ => false),
|
||||
("is_topic_tweet", _ => false),
|
||||
("preferred_language_matches_tweet_language", _ => false),
|
||||
(
|
||||
"device_language_matches_tweet_language",
|
||||
candidate =>
|
||||
candidate.getOrElse(TweetLanguageFeature, None) ==
|
||||
candidate.getOrElse(DeviceLanguageFeature, None)),
|
||||
("question", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuestion.contains(true))),
|
||||
("in_network", _.getOrElse(FromInNetworkSourceFeature, true)),
|
||||
("viewer_follows_original_author", _ => false),
|
||||
("has_account_follow_prompt", _ => false),
|
||||
("has_relevance_prompt", _ => false),
|
||||
("has_topic_annotation_haug_prompt", _ => false),
|
||||
("has_topic_annotation_random_precision_prompt", _ => false),
|
||||
("has_topic_annotation_prompt", _ => false),
|
||||
(
|
||||
"has_political_annotation",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(
|
||||
_.semanticCoreAnnotations.exists(
|
||||
_.exists(annotation =>
|
||||
SemanticCoreFeatures.PoliticalDomains.contains(annotation.domainId) ||
|
||||
(annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
|
||||
annotation.entityId == SemanticCoreFeatures.UttPoliticsEntityId))))),
|
||||
(
|
||||
"is_dont_at_me_by_invitation",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(
|
||||
_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.ByInvitation]))),
|
||||
(
|
||||
"is_dont_at_me_community",
|
||||
_.getOrElse(EarlybirdFeature, None)
|
||||
.exists(_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.Community]))),
|
||||
("has_zero_score", _.getOrElse(ScoreFeature, None).exists(_ == 0.0)),
|
||||
("is_viewer_not_invited_to_reply", _ => false),
|
||||
("is_viewer_invited_to_reply", _ => false),
|
||||
("has_gte_10_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10))),
|
||||
("has_gte_100_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100))),
|
||||
("has_gte_1k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
|
||||
(
|
||||
"has_gte_10k_favs",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
|
||||
(
|
||||
"has_gte_100k_favs",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100000))),
|
||||
("above_neighbor_is_topic_tweet", _ => false),
|
||||
("is_topic_tweet_with_neighbor_below", _ => false),
|
||||
("has_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasSpace)),
|
||||
("has_live_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isLive)),
|
||||
(
|
||||
"has_gte_10_retweets",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 10))),
|
||||
(
|
||||
"has_gte_100_retweets",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 100))),
|
||||
(
|
||||
"has_gte_1k_retweets",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 1000))),
|
||||
(
|
||||
"has_us_political_annotation",
|
||||
_.getOrElse(EarlybirdFeature, None)
|
||||
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
|
||||
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
|
||||
annotation.entityId == SemanticCoreFeatures.usPoliticalTweetEntityId &&
|
||||
annotation.groupId == SemanticCoreFeatures.UsPoliticalTweetAnnotationGroupIds.BalancedV0)))),
|
||||
(
|
||||
"has_toxicity_score_above_threshold",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91))),
|
||||
(
|
||||
"text_only",
|
||||
candidate =>
|
||||
candidate.getOrElse(HasDisplayedTextFeature, false) &&
|
||||
!(candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) ||
|
||||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasVideo) ||
|
||||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasCard))),
|
||||
(
|
||||
"image_only",
|
||||
candidate =>
|
||||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) &&
|
||||
!candidate.getOrElse(HasDisplayedTextFeature, false)),
|
||||
("has_1_image", _.getOrElse(NumImagesFeature, None).exists(_ == 1)),
|
||||
("has_2_images", _.getOrElse(NumImagesFeature, None).exists(_ == 2)),
|
||||
("has_3_images", _.getOrElse(NumImagesFeature, None).exists(_ == 3)),
|
||||
("has_4_images", _.getOrElse(NumImagesFeature, None).exists(_ == 4)),
|
||||
("has_card", _.getOrElse(EarlybirdFeature, None).exists(_.hasCard)),
|
||||
("3_or_more_consecutive_not_in_network", _ => false),
|
||||
("2_or_more_consecutive_not_in_network", _ => false),
|
||||
("5_out_of_7_not_in_network", _ => false),
|
||||
("7_out_of_7_not_in_network", _ => false),
|
||||
("5_out_of_5_not_in_network", _ => false),
|
||||
("user_follow_count_gte_50", _.getOrElse(UserFollowingCountFeature, None).exists(_ > 50)),
|
||||
("has_liked_by_social_context", _ => false),
|
||||
("has_followed_by_social_context", _ => false),
|
||||
("has_topic_social_context", _ => false),
|
||||
("timeline_entry_has_banner", _ => false),
|
||||
("served_in_conversation_module", _.getOrElse(ServedInConversationModuleFeature, false)),
|
||||
(
|
||||
"conversation_module_has_2_displayed_tweets",
|
||||
_.getOrElse(ConversationModule2DisplayedTweetsFeature, false)),
|
||||
("conversation_module_has_gap", _.getOrElse(ConversationModuleHasGapFeature, false)),
|
||||
("served_in_recap_tweet_candidate_module_injection", _ => false),
|
||||
("served_in_threaded_conversation_module", _ => false),
|
||||
(
|
||||
"author_is_elon",
|
||||
candidate =>
|
||||
candidate
|
||||
.getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
|
||||
(
|
||||
"author_is_power_user",
|
||||
candidate =>
|
||||
candidate
|
||||
.getOrElse(AuthorIdFeature, None)
|
||||
.exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)),
|
||||
(
|
||||
"author_is_democrat",
|
||||
candidate =>
|
||||
candidate
|
||||
.getOrElse(AuthorIdFeature, None)
|
||||
.exists(candidate.getOrElse(DDGStatsDemocratsFeature, Set.empty[Long]).contains)),
|
||||
(
|
||||
"author_is_republican",
|
||||
candidate =>
|
||||
candidate
|
||||
.getOrElse(AuthorIdFeature, None)
|
||||
.exists(candidate.getOrElse(DDGStatsRepublicansFeature, Set.empty[Long]).contains)),
|
||||
)
|
||||
|
||||
val PredicateMap = CandidatePredicates.toMap
|
||||
}
|
@ -0,0 +1,54 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LikeGeneralContextType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class LikedBySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
|
||||
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
|
||||
contextType = LikeGeneralContextType,
|
||||
stringCenter = stringCenter,
|
||||
oneUserString = externalStrings.socialContextOneUserLikedString,
|
||||
twoUsersString = externalStrings.socialContextTwoUsersLikedString,
|
||||
moreUsersString = externalStrings.socialContextMoreUsersLikedString,
|
||||
timelineTitle = externalStrings.socialContextLikedByTimelineTitle
|
||||
)
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
|
||||
// Liked by users are valid only if they pass both the SGS and Perspective filters.
|
||||
val validLikedByUserIds =
|
||||
candidateFeatures
|
||||
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
|
||||
.filter(
|
||||
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
|
||||
|
||||
engagerSocialContextBuilder(
|
||||
socialContextIds = validLikedByUserIds,
|
||||
query = query,
|
||||
candidateFeatures = candidateFeatures
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,47 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
|
||||
import com.twitter.home_mixer.functional_component.decorator.builder.ListClientEventDetailsBuilder
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
|
||||
object ListConversationServiceCandidateDecorator {
|
||||
|
||||
private val ConversationModuleNamespace = EntryNamespace("list-conversation")
|
||||
|
||||
def apply(): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {
|
||||
val suggestType = st.SuggestType.OrganicListTweet
|
||||
val component = InjectionScribeUtil.scribeComponent(suggestType).get
|
||||
val clientEventInfoBuilder =
|
||||
ClientEventInfoBuilder(component, Some(ListClientEventDetailsBuilder))
|
||||
val tweetItemBuilder = TweetCandidateUrtItemBuilder(
|
||||
clientEventInfoBuilder = clientEventInfoBuilder
|
||||
)
|
||||
|
||||
val moduleBuilder = TimelineModuleBuilder(
|
||||
entryNamespace = ConversationModuleNamespace,
|
||||
clientEventInfoBuilder = clientEventInfoBuilder,
|
||||
displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),
|
||||
metadataBuilder = Some(HomeConversationModuleMetadataBuilder())
|
||||
)
|
||||
|
||||
Some(
|
||||
UrtMultipleModulesDecorator(
|
||||
urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),
|
||||
moduleBuilder = moduleBuilder,
|
||||
groupByKey = (_, _, candidateFeatures) =>
|
||||
candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)
|
||||
))
|
||||
}
|
||||
}
|
@ -0,0 +1,55 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class MuteUserChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ChildFeedbackAction] = {
|
||||
val userIdOpt =
|
||||
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
|
||||
candidateFeatures.getOrElse(SourceUserIdFeature, None)
|
||||
else candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
userIdOpt.flatMap { userId =>
|
||||
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
|
||||
val userScreenNameOpt = screenNamesMap.get(userId)
|
||||
userScreenNameOpt.map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
externalStrings.muteUserString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(prompt),
|
||||
confirmation = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.SpeakerOff),
|
||||
richBehavior = Some(RichFeedbackBehaviorToggleMuteUser(userId)),
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,71 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class NotInterestedTopicFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackAction] = {
|
||||
val isOutOfNetwork = !candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
val validFollowedByUserIds =
|
||||
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
|
||||
val validLikedByUserIds =
|
||||
candidateFeatures
|
||||
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
|
||||
.filter(
|
||||
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
|
||||
|
||||
if (isOutOfNetwork && validLikedByUserIds.isEmpty && validFollowedByUserIds.isEmpty) {
|
||||
val topicIdSocialContext = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
|
||||
val topicContextFunctionalityType =
|
||||
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
|
||||
|
||||
(topicIdSocialContext, topicContextFunctionalityType) match {
|
||||
case (Some(topicId), Some(topicContextFunctionalityType))
|
||||
if topicContextFunctionalityType == RecommendationTopicContextFunctionalityType ||
|
||||
topicContextFunctionalityType == RecWithEducationTopicContextFunctionalityType =>
|
||||
Some(
|
||||
FeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = None,
|
||||
confirmation = None,
|
||||
childFeedbackActions = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = None,
|
||||
richBehavior =
|
||||
Some(RichFeedbackBehaviorMarkNotInterestedTopic(topicId = topicId.toString)),
|
||||
subprompt = None,
|
||||
encodedFeedbackRequest = None
|
||||
)
|
||||
)
|
||||
case _ => None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,55 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.common.{thriftscala => tlc}
|
||||
import com.twitter.timelineservice.model.FeedbackInfo
|
||||
import com.twitter.timelineservice.model.FeedbackMetadata
|
||||
import com.twitter.timelineservice.{thriftscala => tlst}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class NotRelevantChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ChildFeedbackAction] = {
|
||||
val prompt = stringCenter.prepare(externalStrings.notRelevantString)
|
||||
val confirmation = stringCenter.prepare(externalStrings.notRelevantConfirmationString)
|
||||
val feedbackMetadata = FeedbackMetadata(
|
||||
engagementType = None,
|
||||
entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),
|
||||
ttl = Some(FeedbackUtil.FeedbackTtl))
|
||||
val feedbackUrl = FeedbackInfo.feedbackUrl(
|
||||
feedbackType = tlst.FeedbackType.NotRelevant,
|
||||
feedbackMetadata = feedbackMetadata,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
|
||||
Some(
|
||||
ChildFeedbackAction(
|
||||
feedbackType = NotRelevant,
|
||||
prompt = Some(prompt),
|
||||
confirmation = Some(confirmation),
|
||||
feedbackUrl = Some(feedbackUrl),
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = None,
|
||||
richBehavior = None,
|
||||
subprompt = None
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,76 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Use '@A received a reply' as social context when the root Tweet is in network and the focal tweet is OON.
|
||||
*
|
||||
* This function should only be called for the root Tweet of convo modules. This is enforced by
|
||||
* [[HomeTweetSocialContextBuilder]].
|
||||
*/
|
||||
@Singleton
|
||||
case class ReceivedReplySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val receivedReplyString = externalStrings.socialContextReceivedReply
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
|
||||
// If these values are missing default to not showing a received a reply banner
|
||||
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
|
||||
val inNetworkFocalTweet =
|
||||
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(true)
|
||||
|
||||
if (inNetwork && !inNetworkFocalTweet) {
|
||||
|
||||
val authorIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
|
||||
val authorNameOpt = authorIdOpt.flatMap(realNames.get)
|
||||
|
||||
(authorIdOpt, authorNameOpt) match {
|
||||
case (Some(authorId), Some(authorName)) =>
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = ConversationGeneralContextType,
|
||||
text = stringCenter
|
||||
.prepare(receivedReplyString, placeholders = Map("user1" -> authorName)),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = "",
|
||||
urtEndpointOptions = None
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
case _ => None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,38 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class ReportTweetChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidate: TweetCandidate
|
||||
): Option[ChildFeedbackAction] = {
|
||||
Some(
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(stringCenter.prepare(externalStrings.reportTweetString)),
|
||||
confirmation = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.Flag),
|
||||
richBehavior = Some(RichFeedbackBehaviorReportTweet(candidate.id)),
|
||||
subprompt = None
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,39 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class RetweeterChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
|
||||
|
||||
if (isRetweet) {
|
||||
candidateFeatures.getOrElse(AuthorIdFeature, None).flatMap { retweeterId =>
|
||||
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
|
||||
userId = retweeterId,
|
||||
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
|
||||
promptExternalString = externalStrings.showFewerRetweetsString,
|
||||
confirmationExternalString = externalStrings.showFewerRetweetsConfirmationString,
|
||||
engagementType = t.FeedbackEngagementType.Retweet,
|
||||
stringCenter = stringCenter,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
}
|
||||
} else None
|
||||
}
|
||||
}
|
@ -0,0 +1,42 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class TopicSocialContextBuilder @Inject() ()
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
if (!inNetwork) {
|
||||
val topicIdSocialContextOpt = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
|
||||
val topicContextFunctionalityTypeOpt =
|
||||
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
|
||||
(topicIdSocialContextOpt, topicContextFunctionalityTypeOpt) match {
|
||||
case (Some(topicId), Some(topicContextFunctionalityType)) =>
|
||||
Some(
|
||||
TopicContext(
|
||||
topicId = topicId.toString,
|
||||
functionalityType = Some(topicContextFunctionalityType)
|
||||
))
|
||||
case _ => None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,57 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class UnfollowUserChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
|
||||
val userIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
if (isInNetwork) {
|
||||
userIdOpt.flatMap { userId =>
|
||||
val screenNamesMap =
|
||||
candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
|
||||
val userScreenNameOpt = screenNamesMap.get(userId)
|
||||
userScreenNameOpt.map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
externalStrings.unfollowUserString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
val confirmation = stringCenter.prepare(
|
||||
externalStrings.unfollowUserConfirmationString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(prompt),
|
||||
confirmation = Some(confirmation),
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.Unfollow),
|
||||
richBehavior = Some(RichFeedbackBehaviorToggleFollowUser(userId)),
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
} else None
|
||||
}
|
||||
}
|
@ -0,0 +1,50 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Renders a fixed 'You Might Like' string above all OON Tweets.
|
||||
*/
|
||||
@Singleton
|
||||
case class YouMightLikeSocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val youMightLikeString = externalStrings.socialContextYouMightLikeString
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
|
||||
if (!isInNetwork && !isRetweet) {
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = SparkleGeneralContextType,
|
||||
text = stringCenter.prepare(youMightLikeString),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = None
|
||||
))
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,23 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/com/twitter/bijection:scrooge",
|
||||
"finagle/finagle-core/src/main",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/scala/com/twitter/suggests/controller_data",
|
||||
"src/thrift/com/twitter/suggests/controller_data:controller_data-scala",
|
||||
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
|
||||
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
|
||||
],
|
||||
)
|
@ -0,0 +1,44 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.EntityTokenFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
|
||||
|
||||
/**
|
||||
* Sets the [[ClientEventInfo]] with the `component` field set to the Suggest Type assigned to each candidate
|
||||
*/
|
||||
case class HomeClientEventInfoBuilder[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](
|
||||
detailsBuilder: Option[BaseClientEventDetailsBuilder[Query, Candidate]] = None)
|
||||
extends BaseClientEventInfoBuilder[Query, Candidate] {
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidate: Candidate,
|
||||
candidateFeatures: FeatureMap,
|
||||
element: Option[String]
|
||||
): Option[ClientEventInfo] = {
|
||||
val suggestType = candidateFeatures
|
||||
.getOrElse(SuggestTypeFeature, None)
|
||||
.getOrElse(throw new UnsupportedOperationException(s"No SuggestType was set"))
|
||||
|
||||
Some(
|
||||
ClientEventInfo(
|
||||
component = InjectionScribeUtil.scribeComponent(suggestType),
|
||||
element = element,
|
||||
details = detailsBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),
|
||||
action = None,
|
||||
/**
|
||||
* A backend entity encoded by the Client Entities Encoding Library.
|
||||
* Placeholder string for now
|
||||
*/
|
||||
entityToken = candidateFeatures.getOrElse(EntityTokenFeature, None)
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,30 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleConversationMetadata
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
case class HomeConversationModuleMetadataBuilder[
|
||||
-Query <: PipelineQuery,
|
||||
-Candidate <: BaseTweetCandidate
|
||||
]() extends BaseModuleMetadataBuilder[Query, Candidate] {
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidates: Seq[CandidateWithFeatures[Candidate]]
|
||||
): ModuleMetadata = ModuleMetadata(
|
||||
adsMetadata = None,
|
||||
conversationMetadata = Some(
|
||||
ModuleConversationMetadata(
|
||||
allTweetIds = Some((candidates.last.candidate.id +:
|
||||
candidates.last.features.getOrElse(AncestorsFeature, Seq.empty).map(_.tweetId)).reverse),
|
||||
socialContext = None,
|
||||
enableDeduplication = Some(true)
|
||||
)),
|
||||
gridCarouselMetadata = None
|
||||
)
|
||||
}
|
@ -0,0 +1,33 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
|
||||
object ListClientEventDetailsBuilder
|
||||
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UniversalNoun[Any],
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ClientEventDetails] = {
|
||||
val clientEventDetails = ClientEventDetails(
|
||||
conversationDetails = None,
|
||||
timelinesDetails = Some(
|
||||
TimelinesDetails(
|
||||
injectionType = Some(st.SuggestType.OrganicListTweet.name),
|
||||
controllerData = None,
|
||||
sourceData = None)),
|
||||
articleDetails = None,
|
||||
liveEventDetails = None,
|
||||
commerceDetails = None
|
||||
)
|
||||
|
||||
Some(clientEventDetails)
|
||||
}
|
||||
}
|
@ -0,0 +1,30 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.AlwaysInclude
|
||||
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction
|
||||
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtInstructionBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.Cover
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
case class AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder[Query <: PipelineQuery](
|
||||
override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)
|
||||
extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {
|
||||
|
||||
override def build(
|
||||
query: Query,
|
||||
entries: Seq[TimelineEntry]
|
||||
): Seq[AddEntriesTimelineInstruction] = {
|
||||
if (includeInstruction(query, entries)) {
|
||||
val entriesToAdd = entries
|
||||
.filterNot(_.isInstanceOf[ShowAlert])
|
||||
.filterNot(_.isInstanceOf[Cover])
|
||||
.filter(_.entryIdToReplace.isEmpty)
|
||||
if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))
|
||||
else Seq.empty
|
||||
} else
|
||||
Seq.empty
|
||||
}
|
||||
}
|
@ -0,0 +1,12 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder",
|
||||
"src/thrift/com/twitter/timelines/service:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
|
||||
],
|
||||
)
|
@ -0,0 +1,51 @@
|
||||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.UserCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.ExternalStringRegistry
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.service.{thriftscala => tl}
|
||||
import com.twitter.timelines.util.FeedbackRequestSerializer
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
import com.twitter.timelineservice.thriftscala.FeedbackType
|
||||
|
||||
object HomeWhoToFollowFeedbackActionInfoBuilder {
|
||||
private val FeedbackMetadata = tl.FeedbackMetadata(
|
||||
injectionType = Some(SuggestType.WhoToFollow),
|
||||
engagementType = None,
|
||||
entityIds = Seq.empty,
|
||||
ttlMs = None
|
||||
)
|
||||
private val FeedbackRequest =
|
||||
tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)
|
||||
private val EncodedFeedbackRequest =
|
||||
FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class HomeWhoToFollowFeedbackActionInfoBuilder @Inject() (
|
||||
@ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry],
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {
|
||||
|
||||
private val whoToFollowFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(
|
||||
externalStringRegistry = externalStringRegistryProvider.get(),
|
||||
stringCenter = stringCenterProvider.get(),
|
||||
encodedFeedbackRequest = Some(HomeWhoToFollowFeedbackActionInfoBuilder.EncodedFeedbackRequest)
|
||||
)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UserCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackActionInfo] =
|
||||
whoToFollowFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)
|
||||
}
|
@ -0,0 +1,56 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}
|
||||
import com.twitter.tweetconvosvc.{thriftscala => tcs}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class AncestorFeatureHydrator @Inject() (
|
||||
conversationServiceClient: tcs.ConversationService.MethodPerEndpoint)
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Ancestor")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(AncestorsFeature)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val ancestorsRequest = tcs.GetAncestorsRequest(Seq(candidate.id))
|
||||
|
||||
Stitch.callFuture(conversationServiceClient.getAncestors(ancestorsRequest)).map {
|
||||
getAncestorsResponse =>
|
||||
val ancestors = getAncestorsResponse.ancestors.headOption
|
||||
.collect {
|
||||
case tcs.TweetAncestorsResult.TweetAncestors(ancestorsResult)
|
||||
if ancestorsResult.nonEmpty =>
|
||||
ancestorsResult.head.ancestors ++ getTruncatedRootTweet(ancestorsResult.head)
|
||||
}.getOrElse(Seq.empty)
|
||||
|
||||
FeatureMapBuilder().add(AncestorsFeature, ancestors).build()
|
||||
}
|
||||
}
|
||||
|
||||
private def getTruncatedRootTweet(
|
||||
ancestors: ta.TweetAncestors,
|
||||
): Option[ta.TweetAncestor] = {
|
||||
ancestors.conversationRootAuthorId.collect {
|
||||
case rootAuthorId
|
||||
if ancestors.state == ta.ReplyState.Partial &&
|
||||
ancestors.ancestors.last.tweetId != ancestors.conversationId =>
|
||||
ta.TweetAncestor(ancestors.conversationId, rootAuthorId)
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,95 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features.AuthorFeaturesAdapter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.AuthorFeatureRepository
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.repository.KeyValueResult
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.author_features.v1.{thriftjava => af}
|
||||
import com.twitter.util.Future
|
||||
import com.twitter.util.Try
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object AuthorFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class AuthorFeatureHydrator @Inject() (
|
||||
@Named(AuthorFeatureRepository) client: KeyValueRepository[Seq[Long], Long, af.AuthorFeatures],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("AuthorFeature")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(AuthorFeature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
Stitch.callFuture {
|
||||
val possiblyAuthorIds = extractKeys(candidates)
|
||||
val authorIds = possiblyAuthorIds.flatten
|
||||
|
||||
val response: Future[KeyValueResult[Long, af.AuthorFeatures]] =
|
||||
if (authorIds.isEmpty) {
|
||||
Future.value(KeyValueResult.empty)
|
||||
} else {
|
||||
client(authorIds)
|
||||
}
|
||||
|
||||
response.map { result =>
|
||||
possiblyAuthorIds.map { possiblyAuthorId =>
|
||||
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
|
||||
val transformedValue = postTransformer(value)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(AuthorFeature, transformedValue)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def postTransformer(authorFeatures: Try[Option[af.AuthorFeatures]]): Try[DataRecord] = {
|
||||
authorFeatures.map { features =>
|
||||
AuthorFeaturesAdapter.adaptToDataRecords(features).asScala.head
|
||||
}
|
||||
}
|
||||
|
||||
private def extractKeys(
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Seq[Option[Long]] = {
|
||||
candidates.map { candidate =>
|
||||
candidate.features
|
||||
.getTry(AuthorIdFeature)
|
||||
.toOption
|
||||
.flatten
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,103 @@
|
||||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/com/twitter/storehaus:core",
|
||||
"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi",
|
||||
"configapi/configapi-decider",
|
||||
"finatra/inject/inject-core/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/author_features",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/earlybird",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/inferred_topic",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content",
|
||||
"joinkey/src/main/scala/com/twitter/joinkey/context",
|
||||
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timelines_impression_store",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/topics",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util",
|
||||
"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common",
|
||||
"representation-scorer/server/src/main/thrift:thrift-scala",
|
||||
"servo/repo/src/main/scala",
|
||||
"snowflake/src/main/scala/com/twitter/snowflake/id",
|
||||
"src/java/com/twitter/ml/api/constant",
|
||||
"src/java/com/twitter/search/common/util/lang",
|
||||
"src/scala/com/twitter/ml/api/util",
|
||||
"src/scala/com/twitter/timelines/prediction/adapters/real_graph",
|
||||
"src/scala/com/twitter/timelines/prediction/adapters/realtime_interaction_graph",
|
||||
"src/scala/com/twitter/timelines/prediction/adapters/twistly",
|
||||
"src/scala/com/twitter/timelines/prediction/adapters/two_hop_features",
|
||||
"src/scala/com/twitter/timelines/prediction/common/util",
|
||||
"src/scala/com/twitter/timelines/prediction/features/common",
|
||||
"src/scala/com/twitter/timelines/prediction/features/realtime_interaction_graph",
|
||||
"src/scala/com/twitter/timelines/prediction/features/recap",
|
||||
"src/scala/com/twitter/timelines/prediction/features/time_features",
|
||||
"src/thrift/com/twitter/gizmoduck:thrift-scala",
|
||||
"src/thrift/com/twitter/ml/api:data-java",
|
||||
"src/thrift/com/twitter/ml/api:embedding-java",
|
||||
"src/thrift/com/twitter/onboarding/relevance/features:features-java",
|
||||
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
|
||||
"src/thrift/com/twitter/search:earlybird-scala",
|
||||
"src/thrift/com/twitter/search/common:constants-java",
|
||||
"src/thrift/com/twitter/socialgraph:thrift-scala",
|
||||
"src/thrift/com/twitter/spam/rtf:safety-result-scala",
|
||||
"src/thrift/com/twitter/timelineranker:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/author_features:thrift-java",
|
||||
"src/thrift/com/twitter/timelines/conversation_features:conversation_features-scala",
|
||||
"src/thrift/com/twitter/timelines/impression:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/real_graph:real_graph-scala",
|
||||
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
|
||||
"src/thrift/com/twitter/topic_recos:topic_recos-thrift-java",
|
||||
"src/thrift/com/twitter/tweetypie:service-scala",
|
||||
"src/thrift/com/twitter/tweetypie:tweet-scala",
|
||||
"src/thrift/com/twitter/user_session_store:thrift-java",
|
||||
"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala",
|
||||
"src/thrift/com/twitter/wtf/real_time_interaction_graph:wtf-real_time_interaction_graph-thrift-java",
|
||||
"stitch/stitch-core",
|
||||
"stitch/stitch-gizmoduck",
|
||||
"stitch/stitch-socialgraph",
|
||||
"stitch/stitch-timelineservice",
|
||||
"stitch/stitch-tweetypie",
|
||||
"strato/config/columns/topic-signals/tsp",
|
||||
"strato/config/columns/topic-signals/tsp:tsp-strato-client",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/feedback",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/persistence",
|
||||
"timelines/src/main/scala/com/twitter/timelines/clients/strato/twistly",
|
||||
"timelines/src/main/scala/com/twitter/timelines/clients/user_tweet_entity_graph",
|
||||
"timelines/src/main/scala/com/twitter/timelines/impressionstore/impressionbloomfilter",
|
||||
"timelines/src/main/scala/com/twitter/timelines/impressionstore/store",
|
||||
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
|
||||
"topic-social-proof/server/src/main/thrift:thrift-scala",
|
||||
"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting",
|
||||
"tweetconvosvc/thrift/src/main/thrift:thrift-scala",
|
||||
"twitter-config/yaml",
|
||||
"user_session_store/src/main/scala/com/twitter/user_session_store",
|
||||
"util/util-core",
|
||||
],
|
||||
exports = [
|
||||
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
|
||||
],
|
||||
)
|
@ -0,0 +1,45 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.timelinemixer.clients.manhattan.InjectionHistoryClient
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.manhattan.DismissInfo
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object DismissInfoQueryFeatureHydrator {
|
||||
val DismissInfoSuggestTypes = Seq(SuggestType.WhoToFollow)
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class DismissInfoQueryFeatureHydrator @Inject() (
|
||||
dismissInfoClient: InjectionHistoryClient)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("DismissInfo")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(DismissInfoFeature)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] =
|
||||
Stitch.callFuture {
|
||||
dismissInfoClient
|
||||
.readDismissInfoEntries(
|
||||
query.getRequiredUserId,
|
||||
DismissInfoQueryFeatureHydrator.DismissInfoSuggestTypes).map { response =>
|
||||
val dismissInfoMap = response.mapValues(DismissInfo.fromThrift)
|
||||
FeatureMapBuilder().add(DismissInfoFeature, dismissInfoMap).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8, 50, 60, 60)
|
||||
)
|
||||
}
|
@ -0,0 +1,129 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.earlybird.EarlybirdAdapter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.DeviceLanguageFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRepository
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.home_mixer.util.earlybird.EarlybirdResponseUtil
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.search.earlybird.{thriftscala => eb}
|
||||
import com.twitter.servo.keyvalue.KeyValueResult
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.util.Return
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object EarlybirdDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class EarlybirdFeatureHydrator @Inject() (
|
||||
@Named(EarlybirdRepository) client: KeyValueRepository[
|
||||
(Seq[Long], Long),
|
||||
Long,
|
||||
eb.ThriftSearchResult
|
||||
],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Earlybird")
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(EarlybirdDataRecordFeature, EarlybirdFeature, TweetUrlsFeature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(statScope)
|
||||
private val originalKeyFoundCounter = scopedStatsReceiver.counter("originalKey/found")
|
||||
private val originalKeyLossCounter = scopedStatsReceiver.counter("originalKey/loss")
|
||||
|
||||
private val ebFeaturesNotExistPredicate: CandidateWithFeatures[TweetCandidate] => Boolean =
|
||||
candidate => candidate.features.getOrElse(EarlybirdFeature, None).isEmpty
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val candidatesToHydrate = candidates.filter { candidate =>
|
||||
val isEmpty = ebFeaturesNotExistPredicate(candidate)
|
||||
if (isEmpty) originalKeyLossCounter.incr() else originalKeyFoundCounter.incr()
|
||||
isEmpty
|
||||
}
|
||||
Stitch
|
||||
.callFuture(client((candidatesToHydrate.map(_.candidate.id), query.getRequiredUserId)))
|
||||
.map(handleResponse(query, candidates, _))
|
||||
}
|
||||
|
||||
private def handleResponse(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]],
|
||||
results: KeyValueResult[Long, eb.ThriftSearchResult]
|
||||
): Seq[FeatureMap] = {
|
||||
val queryFeatureMap = query.features.getOrElse(FeatureMap.empty)
|
||||
val userLanguages = queryFeatureMap.getOrElse(UserLanguagesFeature, Seq.empty)
|
||||
val uiLanguageCode = queryFeatureMap.getOrElse(DeviceLanguageFeature, None)
|
||||
val screenName = queryFeatureMap.getOrElse(UserScreenNameFeature, None)
|
||||
|
||||
val searchResults = candidates
|
||||
.filter(ebFeaturesNotExistPredicate).map { candidate =>
|
||||
observedGet(Some(candidate.candidate.id), results)
|
||||
}.collect {
|
||||
case Return(Some(value)) => value
|
||||
}
|
||||
|
||||
val tweetIdToEbFeatures = EarlybirdResponseUtil.getOONTweetThriftFeaturesByTweetId(
|
||||
searcherUserId = query.getRequiredUserId,
|
||||
screenName = screenName,
|
||||
userLanguages = userLanguages,
|
||||
uiLanguageCode = uiLanguageCode,
|
||||
searchResults = searchResults
|
||||
)
|
||||
|
||||
candidates.map { candidate =>
|
||||
val hydratedEbFeatures = tweetIdToEbFeatures.get(candidate.candidate.id)
|
||||
val earlybirdFeatures =
|
||||
if (hydratedEbFeatures.nonEmpty) hydratedEbFeatures
|
||||
else candidate.features.getOrElse(EarlybirdFeature, None)
|
||||
|
||||
val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)
|
||||
val sourceTweetEbFeatures =
|
||||
candidate.features.getOrElse(SourceTweetEarlybirdFeature, None)
|
||||
|
||||
val originalTweetEbFeatures =
|
||||
if (candidateIsRetweet && sourceTweetEbFeatures.nonEmpty)
|
||||
sourceTweetEbFeatures
|
||||
else earlybirdFeatures
|
||||
|
||||
val earlybirdDataRecord =
|
||||
EarlybirdAdapter.adaptToDataRecords(originalTweetEbFeatures).asScala.head
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(EarlybirdFeature, earlybirdFeatures)
|
||||
.add(EarlybirdDataRecordFeature, earlybirdDataRecord)
|
||||
.add(TweetUrlsFeature, earlybirdFeatures.flatMap(_.urlsList).getOrElse(Seq.empty))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,38 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableFeedbackFatigueParam
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.feedback.FeedbackHistoryManhattanClient
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class FeedbackHistoryQueryFeatureHydrator @Inject() (
|
||||
feedbackHistoryClient: FeedbackHistoryManhattanClient)
|
||||
extends QueryFeatureHydrator[PipelineQuery]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FeedbackHistory")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(FeedbackHistoryFeature)
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean =
|
||||
query.params(EnableFeedbackFatigueParam)
|
||||
|
||||
override def hydrate(
|
||||
query: PipelineQuery
|
||||
): Stitch[FeatureMap] =
|
||||
Stitch
|
||||
.callFuture(feedbackHistoryClient.get(query.getRequiredUserId))
|
||||
.map { feedbackHistory =>
|
||||
FeatureMapBuilder().add(FeedbackHistoryFeature, feedbackHistory).build()
|
||||
}
|
||||
}
|
@ -0,0 +1,84 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Social context for convo modules is hydrated on the root Tweet but needs info about the focal
|
||||
* Tweet (e.g. author) to render the banner. This hydrator copies focal Tweet data into the root.
|
||||
*/
|
||||
@Singleton
|
||||
class FocalTweetFeatureHydrator @Inject() ()
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FocalTweet")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
FocalTweetAuthorIdFeature,
|
||||
FocalTweetInNetworkFeature,
|
||||
FocalTweetRealNamesFeature,
|
||||
FocalTweetScreenNamesFeature
|
||||
)
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(FocalTweetAuthorIdFeature, None)
|
||||
.add(FocalTweetInNetworkFeature, None)
|
||||
.add(FocalTweetRealNamesFeature, None)
|
||||
.add(FocalTweetScreenNamesFeature, None)
|
||||
.build()
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
// Build a map of all the focal tweets to their corresponding features
|
||||
val focalTweetIdToFeatureMap = candidates.flatMap { candidate =>
|
||||
val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)
|
||||
if (focalTweetId.contains(candidate.candidate.id)) {
|
||||
Some(candidate.candidate.id -> candidate.features)
|
||||
} else None
|
||||
}.toMap
|
||||
|
||||
val updatedFeatureMap = candidates.map { candidate =>
|
||||
val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)
|
||||
val conversationId = candidate.features.getOrElse(ConversationModuleIdFeature, None)
|
||||
|
||||
// Check if the candidate is a root tweet and ensure its focal tweet's features are available
|
||||
if (conversationId.contains(candidate.candidate.id)
|
||||
&& focalTweetId.exists(focalTweetIdToFeatureMap.contains)) {
|
||||
val featureMap = focalTweetIdToFeatureMap.get(focalTweetId.get).get
|
||||
FeatureMapBuilder()
|
||||
.add(FocalTweetAuthorIdFeature, featureMap.getOrElse(AuthorIdFeature, None))
|
||||
.add(FocalTweetInNetworkFeature, Some(featureMap.getOrElse(InNetworkFeature, true)))
|
||||
.add(
|
||||
FocalTweetRealNamesFeature,
|
||||
Some(featureMap.getOrElse(RealNamesFeature, Map.empty[Long, String])))
|
||||
.add(
|
||||
FocalTweetScreenNamesFeature,
|
||||
Some(featureMap.getOrElse(ScreenNamesFeature, Map.empty[Long, String])))
|
||||
.build()
|
||||
} else DefaultFeatureMap
|
||||
}
|
||||
|
||||
Stitch.value(updatedFeatureMap)
|
||||
}
|
||||
}
|
@ -0,0 +1,41 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserFollowedTopicsCountFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.candidate_source.topics.FollowedTopicsCandidateSource
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class FollowedTopicsQueryFeatureHydrator @Inject() (
|
||||
followedTopicsCandidateSource: FollowedTopicsCandidateSource)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FollowedTopics")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(UserFollowedTopicsCountFeature)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val request: StratoKeyView[Long, Unit] = StratoKeyView(query.getRequiredUserId, Unit)
|
||||
followedTopicsCandidateSource(request)
|
||||
.map { topics =>
|
||||
FeatureMapBuilder().add(UserFollowedTopicsCountFeature, Some(topics.size)).build()
|
||||
}.handle {
|
||||
case _ => FeatureMapBuilder().add(UserFollowedTopicsCountFeature, None).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9),
|
||||
HomeMixerAlertConfig.BusinessHours.defaultLatencyAlert(1500.millis)
|
||||
)
|
||||
}
|
@ -0,0 +1,58 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => gt}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIsBlueVerifiedFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableGizmoduckAuthorSafetyFeatureHydratorParam
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.gizmoduck.Gizmoduck
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class GizmoduckAuthorSafetyFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("GizmoduckAuthorSafety")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(AuthorIsBlueVerifiedFeature)
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean =
|
||||
query.params(EnableGizmoduckAuthorSafetyFeatureHydratorParam)
|
||||
|
||||
private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Safety)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val authorIdOption = existingFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
val blueVerifiedStitch = authorIdOption
|
||||
.map { authorId =>
|
||||
gizmoduck
|
||||
.getUserById(
|
||||
userId = authorId,
|
||||
queryFields = queryFields
|
||||
)
|
||||
.map { _.safety.flatMap(_.isBlueVerified).getOrElse(false) }
|
||||
}.getOrElse(Stitch.False)
|
||||
|
||||
blueVerifiedStitch.map { isBlueVerified =>
|
||||
FeatureMapBuilder()
|
||||
.add(AuthorIsBlueVerifiedFeature, isBlueVerified)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,50 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => gt}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserFollowingCountFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserTypeFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.gizmoduck.Gizmoduck
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class GizmoduckUserQueryFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("GizmoduckUser")
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(UserFollowingCountFeature, UserTypeFeature, UserScreenNameFeature)
|
||||
|
||||
private val queryFields: Set[gt.QueryFields] =
|
||||
Set(gt.QueryFields.Counts, gt.QueryFields.Safety, gt.QueryFields.Profile)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
gizmoduck
|
||||
.getUserById(
|
||||
userId = userId,
|
||||
queryFields = queryFields,
|
||||
context = gt.LookupContext(forUserId = Some(userId), includeSoftUsers = true))
|
||||
.map { user =>
|
||||
FeatureMapBuilder()
|
||||
.add(UserFollowingCountFeature, user.counts.map(_.following.toInt))
|
||||
.add(UserTypeFeature, Some(user.userType))
|
||||
.add(UserScreenNameFeature, user.profile.map(_.screenName))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)
|
||||
)
|
||||
}
|
@ -0,0 +1,105 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.graph_feature_service.{thriftscala => gfs}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsExtendedReplyFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.GraphTwoHopRepository
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.home_mixer.util.ReplyRetweetUtil
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.adapters.two_hop_features.TwoHopFeaturesAdapter
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object GraphTwoHopFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class GraphTwoHopFeatureHydrator @Inject() (
|
||||
@Named(GraphTwoHopRepository) client: KeyValueRepository[(Seq[Long], Long), Long, Seq[
|
||||
gfs.IntersectionValue
|
||||
]],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("GraphTwoHop")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(GraphTwoHopFeature, FollowedByUserIdsFeature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
private val twoHopFeaturesAdapter = new TwoHopFeaturesAdapter
|
||||
|
||||
private val FollowFeatureType = gfs.FeatureType(gfs.EdgeType.Following, gfs.EdgeType.FollowedBy)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
// Apply filters to in network candidates for ExtendedReplyAncestors and retweets.
|
||||
// ExtendedReplyAncestors should also be in candidates. No filter for oon.
|
||||
val (inNetworkCandidates, oonCandidates) = candidates.partition { candidate =>
|
||||
candidate.features.getOrElse(InNetworkFeature, false)
|
||||
}
|
||||
|
||||
val inNetworkReplyToAncestorTweet =
|
||||
ReplyRetweetUtil.replyToAncestorTweetCandidatesMap(inNetworkCandidates)
|
||||
|
||||
val inNetworkExtendedReplyAncestors = inNetworkCandidates
|
||||
.filter(_.features.getOrElse(IsExtendedReplyFeature, false)).flatMap { inNetworkCandidate =>
|
||||
inNetworkReplyToAncestorTweet.get(inNetworkCandidate.candidate.id)
|
||||
}.flatten
|
||||
|
||||
val inNetworkCandidatesToHydrate = inNetworkExtendedReplyAncestors ++
|
||||
inNetworkCandidates.filter(_.features.getOrElse(IsRetweetFeature, false))
|
||||
|
||||
val candidatesToHydrate = (inNetworkCandidatesToHydrate ++ oonCandidates)
|
||||
.flatMap(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features)).distinct
|
||||
|
||||
val response = Stitch.callFuture(client((candidatesToHydrate, query.getRequiredUserId)))
|
||||
|
||||
response.map { result =>
|
||||
candidates.map { candidate =>
|
||||
val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)
|
||||
|
||||
val value = observedGet(key = originalAuthorId, keyValueResult = result)
|
||||
val transformedValue = postTransformer(value)
|
||||
val followedByUserIds = value.toOption.flatMap(getFollowedByUserIds(_)).getOrElse(Seq.empty)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(GraphTwoHopFeature, transformedValue)
|
||||
.add(FollowedByUserIdsFeature, followedByUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def getFollowedByUserIds(input: Option[Seq[gfs.IntersectionValue]]): Option[Seq[Long]] =
|
||||
input.map(_.filter(_.featureType == FollowFeatureType).flatMap(_.intersectionIds).flatten)
|
||||
|
||||
private def postTransformer(input: Try[Option[Seq[gfs.IntersectionValue]]]): Try[DataRecord] =
|
||||
input.map(twoHopFeaturesAdapter.adaptToDataRecords(_).asScala.head)
|
||||
}
|
@ -0,0 +1,57 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature
|
||||
import com.twitter.home_mixer.model.request.HasSeenTweetIds
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.impressionbloomfilter.{thriftscala => t}
|
||||
import com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class ImpressionBloomFilterQueryFeatureHydrator[
|
||||
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
|
||||
bloomFilter: ImpressionBloomFilter)
|
||||
extends QueryFeatureHydrator[Query] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
|
||||
"ImpressionBloomFilter")
|
||||
|
||||
private val ImpressionBloomFilterTTL = 7.day
|
||||
private val ImpressionBloomFilterFalsePositiveRate = 0.002
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(ImpressionBloomFilterFeature)
|
||||
|
||||
private val SurfaceArea = t.SurfaceArea.HomeTimeline
|
||||
|
||||
override def hydrate(query: Query): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
bloomFilter.getBloomFilterSeq(userId, SurfaceArea).map { bloomFilterSeq =>
|
||||
val updatedBloomFilterSeq =
|
||||
if (query.seenTweetIds.forall(_.isEmpty)) bloomFilterSeq
|
||||
else {
|
||||
bloomFilter.addElements(
|
||||
userId = userId,
|
||||
surfaceArea = SurfaceArea,
|
||||
tweetIds = query.seenTweetIds.get,
|
||||
bloomFilterEntrySeq = bloomFilterSeq,
|
||||
timeToLive = ImpressionBloomFilterTTL,
|
||||
falsePositiveRate = ImpressionBloomFilterFalsePositiveRate
|
||||
)
|
||||
}
|
||||
FeatureMapBuilder().add(ImpressionBloomFilterFeature, updatedBloomFilterSeq).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)
|
||||
)
|
||||
}
|
@ -0,0 +1,68 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.user_session_store.ReadRequest
|
||||
import com.twitter.user_session_store.ReadWriteUserSessionStore
|
||||
import com.twitter.user_session_store.UserSessionDataset
|
||||
import com.twitter.user_session_store.UserSessionDataset.UserSessionDataset
|
||||
import com.twitter.util.Time
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class LastNonPollingTimeQueryFeatureHydrator @Inject() (
|
||||
userSessionStore: ReadWriteUserSessionStore)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("LastNonPollingTime")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
FollowingLastNonPollingTimeFeature,
|
||||
LastNonPollingTimeFeature,
|
||||
NonPollingTimesFeature
|
||||
)
|
||||
|
||||
private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.NonPollingTimes)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
userSessionStore
|
||||
.read(ReadRequest(query.getRequiredUserId, datasets))
|
||||
.map { userSession =>
|
||||
val nonPollingTimestamps = userSession.flatMap(_.nonPollingTimestamps)
|
||||
|
||||
val lastNonPollingTime = nonPollingTimestamps
|
||||
.flatMap(_.nonPollingTimestampsMs.headOption)
|
||||
.map(Time.fromMilliseconds)
|
||||
|
||||
val followingLastNonPollingTime = nonPollingTimestamps
|
||||
.flatMap(_.mostRecentHomeLatestNonPollingTimestampMs)
|
||||
.map(Time.fromMilliseconds)
|
||||
|
||||
val nonPollingTimes = nonPollingTimestamps
|
||||
.map(_.nonPollingTimestampsMs)
|
||||
.getOrElse(Seq.empty)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(FollowingLastNonPollingTimeFeature, followingLastNonPollingTime)
|
||||
.add(LastNonPollingTimeFeature, lastNonPollingTime)
|
||||
.add(NonPollingTimesFeature, nonPollingTimes)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)
|
||||
)
|
||||
}
|
@ -0,0 +1,42 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.request.HasListId
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.socialgraph.{thriftscala => sg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.socialgraph.SocialGraph
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
case object ListMembersFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {
|
||||
override val defaultValue: Seq[Long] = Seq.empty
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class ListMembersQueryFeatureHydrator @Inject() (socialGraph: SocialGraph)
|
||||
extends QueryFeatureHydrator[PipelineQuery with HasListId] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("ListMembers")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(ListMembersFeature)
|
||||
|
||||
private val MaxRecentMembers = 10
|
||||
|
||||
override def hydrate(query: PipelineQuery with HasListId): Stitch[FeatureMap] = {
|
||||
val request = sg.IdsRequest(
|
||||
relationships = Seq(sg
|
||||
.SrcRelationship(query.listId, sg.RelationshipType.ListHasMember, hasRelationship = true)),
|
||||
pageRequest = Some(sg.PageRequest(selectAll = Some(true), count = Some(MaxRecentMembers)))
|
||||
)
|
||||
socialGraph.ids(request).map(_.ids).map { listMembers =>
|
||||
FeatureMapBuilder().add(ListMembersFeature, listMembers).build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,81 @@
|
||||
package com.twitter.home_mixer
|
||||
package functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.MetricCenterUserCountingFeatureRepository
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.onboarding.relevance.features.{thriftjava => rf}
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.keyvalue.KeyValueResult
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.util.Future
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
object MetricCenterUserCountingFeature
|
||||
extends Feature[TweetCandidate, Option[rf.MCUserCountingFeatures]]
|
||||
|
||||
@Singleton
|
||||
class MetricCenterUserCountingFeatureHydrator @Inject() (
|
||||
@Named(MetricCenterUserCountingFeatureRepository) client: KeyValueRepository[Seq[
|
||||
Long
|
||||
], Long, rf.MCUserCountingFeatures],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("MetricCenterUserCounting")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(MetricCenterUserCountingFeature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
Stitch.callFuture {
|
||||
val possiblyAuthorIds = extractKeys(candidates)
|
||||
val userIds = possiblyAuthorIds.flatten
|
||||
|
||||
val response: Future[KeyValueResult[Long, rf.MCUserCountingFeatures]] = if (userIds.isEmpty) {
|
||||
Future.value(KeyValueResult.empty)
|
||||
} else {
|
||||
client(userIds)
|
||||
}
|
||||
|
||||
response.map { result =>
|
||||
possiblyAuthorIds.map { possiblyAuthorId =>
|
||||
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(MetricCenterUserCountingFeature, value)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def extractKeys(
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Seq[Option[Long]] = {
|
||||
candidates.map { candidate =>
|
||||
candidate.features
|
||||
.getTry(AuthorIdFeature)
|
||||
.toOption
|
||||
.flatten
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,97 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => gt}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.gizmoduck.Gizmoduck
|
||||
import com.twitter.util.Return
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
protected case class ProfileNames(screenName: String, realName: String)
|
||||
|
||||
@Singleton
|
||||
class NamesFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Names")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(ScreenNamesFeature, RealNamesFeature)
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean = query.product match {
|
||||
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
|
||||
case _ => true
|
||||
}
|
||||
|
||||
private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Profile)
|
||||
|
||||
/**
|
||||
* The UI currently only ever displays the first 2 names in social context lines
|
||||
* E.g. "User and 3 others like" or "UserA and UserB liked"
|
||||
*/
|
||||
private val MaxCountUsers = 2
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
|
||||
val candidateUserIdsMap = candidates.map { candidate =>
|
||||
candidate.candidate.id ->
|
||||
(candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++
|
||||
candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers) ++
|
||||
candidate.features.getOrElse(AuthorIdFeature, None) ++
|
||||
candidate.features.getOrElse(SourceUserIdFeature, None)).distinct
|
||||
}.toMap
|
||||
|
||||
val distinctUserIds = candidateUserIdsMap.values.flatten.toSeq.distinct
|
||||
|
||||
Stitch
|
||||
.collectToTry(distinctUserIds.map(userId => gizmoduck.getUserById(userId, queryFields)))
|
||||
.map { allUsers =>
|
||||
val idToProfileNamesMap = allUsers.flatMap {
|
||||
case Return(allUser) =>
|
||||
allUser.profile
|
||||
.map(profile => allUser.id -> ProfileNames(profile.screenName, profile.name))
|
||||
case _ => None
|
||||
}.toMap
|
||||
|
||||
val validUserIds = idToProfileNamesMap.keySet
|
||||
|
||||
candidates.map { candidate =>
|
||||
val combinedMap = candidateUserIdsMap
|
||||
.getOrElse(candidate.candidate.id, Nil)
|
||||
.flatMap {
|
||||
case userId if validUserIds.contains(userId) =>
|
||||
idToProfileNamesMap.get(userId).map(profileNames => userId -> profileNames)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val perCandidateRealNameMap = combinedMap.map { case (k, v) => k -> v.realName }.toMap
|
||||
val perCandidateScreenNameMap = combinedMap.map { case (k, v) => k -> v.screenName }.toMap
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(ScreenNamesFeature, perCandidateScreenNameMap)
|
||||
.add(RealNamesFeature, perCandidateRealNameMap)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,95 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.model.request.ForYouProduct
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient
|
||||
import com.twitter.timelinemixer.clients.persistence.TimelineResponseV3
|
||||
import com.twitter.timelines.util.client_info.ClientPlatform
|
||||
import com.twitter.timelineservice.model.TimelineQuery
|
||||
import com.twitter.timelineservice.model.core.TimelineKind
|
||||
import com.twitter.timelineservice.model.rich.EntityIdType
|
||||
import com.twitter.util.Time
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class PersistenceStoreQueryFeatureHydrator @Inject() (
|
||||
timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3])
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("PersistenceStore")
|
||||
|
||||
private val WhoToFollowExcludedUserIdsLimit = 1000
|
||||
private val ServedTweetIdsDuration = 1.hour
|
||||
private val ServedTweetIdsLimit = 100
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(ServedTweetIdsFeature, PersistenceEntriesFeature, WhoToFollowExcludedUserIdsFeature)
|
||||
|
||||
private val supportedClients = Seq(
|
||||
ClientPlatform.IPhone,
|
||||
ClientPlatform.IPad,
|
||||
ClientPlatform.Mac,
|
||||
ClientPlatform.Android,
|
||||
ClientPlatform.Web,
|
||||
ClientPlatform.RWeb,
|
||||
ClientPlatform.TweetDeckGryphon
|
||||
)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val timelineKind = query.product match {
|
||||
case FollowingProduct => TimelineKind.homeLatest
|
||||
case ForYouProduct => TimelineKind.home
|
||||
case other => throw new UnsupportedOperationException(s"Unknown product: $other")
|
||||
}
|
||||
val timelineQuery = TimelineQuery(id = query.getRequiredUserId, kind = timelineKind)
|
||||
|
||||
Stitch.callFuture {
|
||||
timelineResponseBatchesClient
|
||||
.get(query = timelineQuery, clientPlatforms = supportedClients)
|
||||
.map { timelineResponses =>
|
||||
// Note that the WTF entries are not being scoped by ClientPlatform
|
||||
val whoToFollowUserIds = timelineResponses
|
||||
.flatMap { timelineResponse =>
|
||||
timelineResponse.entries
|
||||
.filter(_.entityIdType == EntityIdType.WhoToFollow)
|
||||
.flatMap(_.itemIds.toSeq.flatMap(_.flatMap(_.userId)))
|
||||
}.take(WhoToFollowExcludedUserIdsLimit)
|
||||
|
||||
val clientPlatform = ClientPlatform.fromQueryOptions(
|
||||
clientAppId = query.clientContext.appId,
|
||||
userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString))
|
||||
|
||||
val servedTweetIds = timelineResponses
|
||||
.filter(_.clientPlatform == clientPlatform)
|
||||
.filter(_.servedTime >= Time.now - ServedTweetIdsDuration)
|
||||
.sortBy(-_.servedTime.inMilliseconds)
|
||||
.flatMap(
|
||||
_.entries.flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetIdsLimit))
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(ServedTweetIdsFeature, servedTweetIds)
|
||||
.add(PersistenceEntriesFeature, timelineResponses)
|
||||
.add(WhoToFollowExcludedUserIdsFeature, whoToFollowUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7, 50, 60, 60)
|
||||
)
|
||||
}
|
@ -0,0 +1,71 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.timelineservice.TimelineService
|
||||
import com.twitter.stitch.timelineservice.TimelineService.GetPerspectives
|
||||
import com.twitter.timelineservice.thriftscala.PerspectiveType
|
||||
import com.twitter.timelineservice.thriftscala.PerspectiveType.Favorited
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Filter out unlike edges from liked-by tweets
|
||||
* Useful if the likes come from a cache and because UTEG does not fully remove unlike edges.
|
||||
*/
|
||||
@Singleton
|
||||
class PerspectiveFilteredSocialContextFeatureHydrator @Inject() (timelineService: TimelineService)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("PerspectiveFilteredSocialContext")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(PerspectiveFilteredLikedByUserIdsFeature)
|
||||
|
||||
private val MaxCountUsers = 10
|
||||
private val favoritePerspectiveSet: Set[PerspectiveType] = Set(Favorited)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val engagingUserIdtoTweetId = candidates.flatMap { candidate =>
|
||||
candidate.features
|
||||
.get(FavoritedByUserIdsFeature).take(MaxCountUsers)
|
||||
.map(favoritedBy => favoritedBy -> candidate.candidate.id)
|
||||
}
|
||||
|
||||
val queries = engagingUserIdtoTweetId.map {
|
||||
case (userId, tweetId) =>
|
||||
GetPerspectives.Query(userId = userId, tweetId = tweetId, types = favoritePerspectiveSet)
|
||||
}
|
||||
|
||||
Stitch.collect(queries.map(timelineService.getPerspective)).map { perspectiveResults =>
|
||||
val validUserIdTweetIds: Set[(Long, Long)] =
|
||||
queries
|
||||
.zip(perspectiveResults)
|
||||
.collect { case (query, perspective) if perspective.favorited => query }
|
||||
.map(query => (query.userId, query.tweetId))
|
||||
.toSet
|
||||
|
||||
candidates.map { candidate =>
|
||||
val perspectiveFilteredFavoritedByUserIds: Seq[Long] = candidate.features
|
||||
.get(FavoritedByUserIdsFeature).take(MaxCountUsers)
|
||||
.filter { userId => validUserIdTweetIds.contains((userId, candidate.candidate.id)) }
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(PerspectiveFilteredLikedByUserIdsFeature, perspectiveFilteredFavoritedByUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,42 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphInNetworkScores
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.storehaus.ReadableStore
|
||||
import com.twitter.wtf.candidate.{thriftscala => wtf}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class RealGraphInNetworkScoresQueryFeatureHydrator @Inject() (
|
||||
@Named(RealGraphInNetworkScores) store: ReadableStore[Long, Seq[wtf.Candidate]])
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("RealGraphInNetworkScores")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(RealGraphInNetworkScoresFeature)
|
||||
|
||||
private val RealGraphCandidateCount = 1000
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
Stitch.callFuture(store.get(query.getRequiredUserId)).map { realGraphFollowedUsers =>
|
||||
val realGraphScoresFeatures = realGraphFollowedUsers
|
||||
.getOrElse(Seq.empty)
|
||||
.sortBy(-_.score)
|
||||
.map(candidate => candidate.userId -> candidate.score)
|
||||
.take(RealGraphCandidateCount)
|
||||
.toMap
|
||||
|
||||
FeatureMapBuilder().add(RealGraphInNetworkScoresFeature, realGraphScoresFeatures).build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,48 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphFeatureRepository
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.repository.Repository
|
||||
import com.twitter.timelines.real_graph.{thriftscala => rg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.model.UserId
|
||||
import com.twitter.timelines.real_graph.v1.thriftscala.RealGraphEdgeFeatures
|
||||
import com.twitter.user_session_store.{thriftscala => uss}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
object RealGraphFeatures extends Feature[PipelineQuery, Option[Map[UserId, RealGraphEdgeFeatures]]]
|
||||
|
||||
@Singleton
|
||||
class RealGraphQueryFeatureHydrator @Inject() (
|
||||
@Named(RealGraphFeatureRepository) repository: Repository[Long, Option[uss.UserSession]])
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("RealGraphFeatures")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(RealGraphFeatures)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
Stitch.callFuture {
|
||||
repository(query.getRequiredUserId).map { userSession =>
|
||||
val realGraphFeaturesMap = userSession.flatMap { userSession =>
|
||||
userSession.realGraphFeatures.collect {
|
||||
case rg.RealGraphFeatures.V1(realGraphFeatures) =>
|
||||
val edgeFeatures = realGraphFeatures.edgeFeatures ++ realGraphFeatures.oonEdgeFeatures
|
||||
edgeFeatures.map { edge => edge.destId -> edge }.toMap
|
||||
}
|
||||
}
|
||||
|
||||
FeatureMapBuilder().add(RealGraphFeatures, realGraphFeaturesMap).build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,123 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature
|
||||
import com.twitter.home_mixer.util.MissingKeyException
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter
|
||||
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphFeaturesAdapter
|
||||
import com.twitter.timelines.real_graph.v1.{thriftscala => v1}
|
||||
import com.twitter.timelines.real_graph.{thriftscala => rg}
|
||||
import com.twitter.util.Throw
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object RealGraphViewerAuthorDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
object RealGraphViewerAuthorsDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class RealGraphViewerAuthorFeatureHydrator @Inject() ()
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("RealGraphViewerAuthor")
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(RealGraphViewerAuthorDataRecordFeature, RealGraphViewerAuthorsDataRecordFeature)
|
||||
|
||||
private val realGraphEdgeFeaturesAdapter = new RealGraphFeaturesAdapter
|
||||
private val realGraphEdgeFeaturesCombineAdapter =
|
||||
new RealGraphEdgeFeaturesCombineAdapter(prefix = "authors.realgraph")
|
||||
|
||||
private val MissingKeyFeatureMap = FeatureMapBuilder()
|
||||
.add(RealGraphViewerAuthorDataRecordFeature, Throw(MissingKeyException))
|
||||
.add(RealGraphViewerAuthorsDataRecordFeature, Throw(MissingKeyException))
|
||||
.build()
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val viewerId = query.getRequiredUserId
|
||||
val realGraphFeatures = query.features
|
||||
.flatMap(_.getOrElse(RealGraphFeatures, None))
|
||||
.getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])
|
||||
|
||||
val result: FeatureMap = existingFeatures.getOrElse(AuthorIdFeature, None) match {
|
||||
case Some(authorId) =>
|
||||
val realGraphAuthorFeatures =
|
||||
getRealGraphViewerAuthorFeatures(viewerId, authorId, realGraphFeatures)
|
||||
val realGraphAuthorDataRecord = realGraphEdgeFeaturesAdapter
|
||||
.adaptToDataRecords(realGraphAuthorFeatures).asScala.headOption.getOrElse(new DataRecord)
|
||||
|
||||
val combinedRealGraphFeaturesDataRecord = for {
|
||||
inReplyToAuthorId <- existingFeatures.getOrElse(InReplyToUserIdFeature, None)
|
||||
} yield {
|
||||
val combinedRealGraphFeatures =
|
||||
getCombinedRealGraphFeatures(Seq(authorId, inReplyToAuthorId), realGraphFeatures)
|
||||
realGraphEdgeFeaturesCombineAdapter
|
||||
.adaptToDataRecords(Some(combinedRealGraphFeatures)).asScala.headOption
|
||||
.getOrElse(new DataRecord)
|
||||
}
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(RealGraphViewerAuthorDataRecordFeature, realGraphAuthorDataRecord)
|
||||
.add(
|
||||
RealGraphViewerAuthorsDataRecordFeature,
|
||||
combinedRealGraphFeaturesDataRecord.getOrElse(new DataRecord))
|
||||
.build()
|
||||
case _ => MissingKeyFeatureMap
|
||||
}
|
||||
Stitch(result)
|
||||
}
|
||||
|
||||
private def getRealGraphViewerAuthorFeatures(
|
||||
viewerId: Long,
|
||||
authorId: Long,
|
||||
realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]
|
||||
): rg.UserRealGraphFeatures = {
|
||||
realGraphEdgeFeaturesMap.get(authorId) match {
|
||||
case Some(realGraphEdgeFeatures) =>
|
||||
rg.UserRealGraphFeatures(
|
||||
srcId = viewerId,
|
||||
features = rg.RealGraphFeatures.V1(
|
||||
v1.RealGraphFeatures(edgeFeatures = Seq(realGraphEdgeFeatures))))
|
||||
case _ =>
|
||||
rg.UserRealGraphFeatures(
|
||||
srcId = viewerId,
|
||||
features = rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = Seq.empty)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
object RealGraphViewerAuthorFeatureHydrator {
|
||||
def getCombinedRealGraphFeatures(
|
||||
userIds: Seq[Long],
|
||||
realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]
|
||||
): rg.RealGraphFeatures = {
|
||||
val edgeFeatures = userIds.flatMap(realGraphEdgeFeaturesMap.get)
|
||||
rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = edgeFeatures))
|
||||
}
|
||||
}
|
@ -0,0 +1,74 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter
|
||||
import com.twitter.timelines.real_graph.v1.{thriftscala => v1}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object RealGraphViewerRelatedUsersDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class RealGraphViewerRelatedUsersFeatureHydrator @Inject() ()
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("RealGraphViewerRelatedUsers")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(RealGraphViewerRelatedUsersDataRecordFeature)
|
||||
|
||||
private val RealGraphEdgeFeaturesCombineAdapter = new RealGraphEdgeFeaturesCombineAdapter
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val realGraphQueryFeatures = query.features
|
||||
.flatMap(_.getOrElse(RealGraphFeatures, None))
|
||||
.getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])
|
||||
|
||||
val allRelatedUserIds = getRelatedUserIds(existingFeatures)
|
||||
val realGraphFeatures =
|
||||
RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures(
|
||||
allRelatedUserIds,
|
||||
realGraphQueryFeatures)
|
||||
val realGraphFeaturesDataRecord = RealGraphEdgeFeaturesCombineAdapter
|
||||
.adaptToDataRecords(Some(realGraphFeatures)).asScala.headOption
|
||||
.getOrElse(new DataRecord)
|
||||
|
||||
Stitch.value {
|
||||
FeatureMapBuilder()
|
||||
.add(RealGraphViewerRelatedUsersDataRecordFeature, realGraphFeaturesDataRecord)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
private def getRelatedUserIds(features: FeatureMap): Seq[Long] = {
|
||||
(CandidatesUtil.getEngagerUserIds(features) ++
|
||||
features.getOrElse(AuthorIdFeature, None) ++
|
||||
features.getOrElse(MentionUserIdFeature, Seq.empty) ++
|
||||
features.getOrElse(SourceUserIdFeature, None) ++
|
||||
features.getOrElse(DirectedAtUserIdFeature, None)).distinct
|
||||
}
|
||||
}
|
@ -0,0 +1,64 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.adapters.realtime_interaction_graph.RealTimeInteractionGraphFeaturesAdapter
|
||||
import com.twitter.timelines.prediction.features.realtime_interaction_graph.RealTimeInteractionGraphEdgeFeatures
|
||||
import com.twitter.util.Time
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object RealTimeInteractionGraphEdgeFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class RealTimeInteractionGraphEdgeFeatureHydrator @Inject() ()
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
|
||||
"RealTimeInteractionGraphEdge")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphEdgeFeature)
|
||||
|
||||
private val realTimeInteractionGraphFeaturesAdapter = new RealTimeInteractionGraphFeaturesAdapter
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val userVertex =
|
||||
query.features.flatMap(_.getOrElse(RealTimeInteractionGraphUserVertexQueryFeature, None))
|
||||
val realTimeInteractionGraphFeaturesMap =
|
||||
userVertex.map(RealTimeInteractionGraphEdgeFeatures(_, Time.now))
|
||||
|
||||
Stitch.value {
|
||||
candidates.map { candidate =>
|
||||
val feature = candidate.features.getOrElse(AuthorIdFeature, None).flatMap { authorId =>
|
||||
realTimeInteractionGraphFeaturesMap.flatMap(_.get(authorId))
|
||||
}
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(
|
||||
RealTimeInteractionGraphEdgeFeature,
|
||||
realTimeInteractionGraphFeaturesAdapter.adaptToDataRecords(feature).asScala.head)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,49 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.google.inject.name.Named
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexCache
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.cache.ReadCache
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.wtf.real_time_interaction_graph.{thriftscala => ig}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object RealTimeInteractionGraphUserVertexQueryFeature
|
||||
extends Feature[PipelineQuery, Option[ig.UserVertex]]
|
||||
|
||||
@Singleton
|
||||
class RealTimeInteractionGraphUserVertexQueryFeatureHydrator @Inject() (
|
||||
@Named(RealTimeInteractionGraphUserVertexCache) client: ReadCache[Long, ig.UserVertex],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends QueryFeatureHydrator[PipelineQuery]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("RealTimeInteractionGraphUserVertex")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphUserVertexQueryFeature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
|
||||
Stitch.callFuture(
|
||||
client.get(Seq(userId)).map { results =>
|
||||
val feature = observedGet(key = Some(userId), keyValueResult = results)
|
||||
FeatureMapBuilder()
|
||||
.add(RealTimeInteractionGraphUserVertexQueryFeature, feature)
|
||||
.build()
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,196 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.home_mixer.util.ReplyRetweetUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.search.common.features.thriftscala.ThriftTweetFeatures
|
||||
import com.twitter.snowflake.id.SnowflakeId
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.conversation_features.v1.thriftscala.ConversationFeatures
|
||||
import com.twitter.util.Duration
|
||||
import com.twitter.util.Time
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object InReplyToTweetHydratedEarlybirdFeature
|
||||
extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]
|
||||
|
||||
/**
|
||||
* The purpose of this hydrator is to
|
||||
* 1) hydrate simple features into replies and their ancestor tweets
|
||||
* 2) keep both the normal replies and ancestor source candidates, but hydrate into the candidates
|
||||
* features useful for predicting the quality of the replies and source ancestor tweets.
|
||||
*/
|
||||
@Singleton
|
||||
class ReplyFeatureHydrator @Inject() (statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("ReplyTweet")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
ConversationFeature,
|
||||
InReplyToTweetHydratedEarlybirdFeature
|
||||
)
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(ConversationFeature, None)
|
||||
.add(InReplyToTweetHydratedEarlybirdFeature, None)
|
||||
.build()
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
|
||||
private val hydratedReplyCounter = scopedStatsReceiver.counter("hydratedReply")
|
||||
private val hydratedAncestorCounter = scopedStatsReceiver.counter("hydratedAncestor")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val replyToInReplyToTweetMap =
|
||||
ReplyRetweetUtil.replyTweetIdToInReplyToTweetMap(candidates)
|
||||
val candidatesWithRepliesHydrated = candidates.map { candidate =>
|
||||
replyToInReplyToTweetMap
|
||||
.get(candidate.candidate.id).map { inReplyToTweet =>
|
||||
hydratedReplyCounter.incr()
|
||||
hydratedReplyCandidate(candidate, inReplyToTweet)
|
||||
}.getOrElse((candidate, None, None))
|
||||
}
|
||||
|
||||
/**
|
||||
* Update ancestor tweets with descendant replies and hydrate simple features from one of
|
||||
* the descendants.
|
||||
*/
|
||||
val ancestorTweetToDescendantRepliesMap =
|
||||
ReplyRetweetUtil.ancestorTweetIdToDescendantRepliesMap(candidates)
|
||||
val candidatesWithRepliesAndAncestorTweetsHydrated = candidatesWithRepliesHydrated.map {
|
||||
case (
|
||||
maybeAncestorTweetCandidate,
|
||||
updatedReplyConversationFeatures,
|
||||
inReplyToTweetEarlyBirdFeature) =>
|
||||
ancestorTweetToDescendantRepliesMap
|
||||
.get(maybeAncestorTweetCandidate.candidate.id)
|
||||
.map { descendantReplies =>
|
||||
hydratedAncestorCounter.incr()
|
||||
val (ancestorTweetCandidate, updatedConversationFeatures): (
|
||||
CandidateWithFeatures[TweetCandidate],
|
||||
Option[ConversationFeatures]
|
||||
) =
|
||||
hydrateAncestorTweetCandidate(
|
||||
maybeAncestorTweetCandidate,
|
||||
descendantReplies,
|
||||
updatedReplyConversationFeatures)
|
||||
(ancestorTweetCandidate, inReplyToTweetEarlyBirdFeature, updatedConversationFeatures)
|
||||
}
|
||||
.getOrElse(
|
||||
(
|
||||
maybeAncestorTweetCandidate,
|
||||
inReplyToTweetEarlyBirdFeature,
|
||||
updatedReplyConversationFeatures))
|
||||
}
|
||||
Stitch.value(
|
||||
candidatesWithRepliesAndAncestorTweetsHydrated.map {
|
||||
case (candidate, inReplyToTweetEarlyBirdFeature, updatedConversationFeatures) =>
|
||||
FeatureMapBuilder()
|
||||
.add(ConversationFeature, updatedConversationFeatures)
|
||||
.add(InReplyToTweetHydratedEarlybirdFeature, inReplyToTweetEarlyBirdFeature)
|
||||
.build()
|
||||
case _ => DefaultFeatureMap
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
private def hydratedReplyCandidate(
|
||||
replyCandidate: CandidateWithFeatures[TweetCandidate],
|
||||
inReplyToTweetCandidate: CandidateWithFeatures[TweetCandidate]
|
||||
): (
|
||||
CandidateWithFeatures[TweetCandidate],
|
||||
Option[ConversationFeatures],
|
||||
Option[ThriftTweetFeatures]
|
||||
) = {
|
||||
val tweetedAfterInReplyToTweetInSecs =
|
||||
(
|
||||
originalTweetAgeFromSnowflake(inReplyToTweetCandidate),
|
||||
originalTweetAgeFromSnowflake(replyCandidate)) match {
|
||||
case (Some(inReplyToTweetAge), Some(replyTweetAge)) =>
|
||||
Some((inReplyToTweetAge - replyTweetAge).inSeconds.toLong)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val existingConversationFeatures = Some(
|
||||
replyCandidate.features
|
||||
.getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))
|
||||
|
||||
val updatedConversationFeatures = existingConversationFeatures match {
|
||||
case Some(v1) =>
|
||||
Some(
|
||||
v1.copy(
|
||||
tweetedAfterInReplyToTweetInSecs = tweetedAfterInReplyToTweetInSecs,
|
||||
isSelfReply = Some(
|
||||
replyCandidate.features.getOrElse(
|
||||
AuthorIdFeature,
|
||||
None) == inReplyToTweetCandidate.features.getOrElse(AuthorIdFeature, None))
|
||||
)
|
||||
)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
// Note: if inReplyToTweet is a retweet, we need to read early bird feature from the merged
|
||||
// early bird feature field from RetweetSourceTweetFeatureHydrator class.
|
||||
// But if inReplyToTweet is a reply, we return its early bird feature directly
|
||||
val inReplyToTweetThriftTweetFeaturesOpt = {
|
||||
if (inReplyToTweetCandidate.features.getOrElse(IsRetweetFeature, false)) {
|
||||
inReplyToTweetCandidate.features.getOrElse(SourceTweetEarlybirdFeature, None)
|
||||
} else {
|
||||
inReplyToTweetCandidate.features.getOrElse(EarlybirdFeature, None)
|
||||
}
|
||||
}
|
||||
|
||||
(replyCandidate, updatedConversationFeatures, inReplyToTweetThriftTweetFeaturesOpt)
|
||||
}
|
||||
|
||||
private def hydrateAncestorTweetCandidate(
|
||||
ancestorTweetCandidate: CandidateWithFeatures[TweetCandidate],
|
||||
descendantReplies: Seq[CandidateWithFeatures[TweetCandidate]],
|
||||
updatedReplyConversationFeatures: Option[ConversationFeatures]
|
||||
): (CandidateWithFeatures[TweetCandidate], Option[ConversationFeatures]) = {
|
||||
// Ancestor could be a reply. For example, in thread: tweetA -> tweetB -> tweetC,
|
||||
// tweetB is a reply and ancestor at the same time. Hence, tweetB's conversation feature
|
||||
// will be updated by hydratedReplyCandidate and hydrateAncestorTweetCandidate functions.
|
||||
val existingConversationFeatures =
|
||||
if (updatedReplyConversationFeatures.nonEmpty)
|
||||
updatedReplyConversationFeatures
|
||||
else
|
||||
Some(
|
||||
ancestorTweetCandidate.features
|
||||
.getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))
|
||||
|
||||
val updatedConversationFeatures = existingConversationFeatures match {
|
||||
case Some(v1) =>
|
||||
Some(
|
||||
v1.copy(
|
||||
hasDescendantReplyCandidate = Some(true),
|
||||
hasInNetworkDescendantReply =
|
||||
Some(descendantReplies.exists(_.features.getOrElse(InNetworkFeature, false)))
|
||||
))
|
||||
case _ => None
|
||||
}
|
||||
(ancestorTweetCandidate, updatedConversationFeatures)
|
||||
}
|
||||
|
||||
private def originalTweetAgeFromSnowflake(
|
||||
candidate: CandidateWithFeatures[TweetCandidate]
|
||||
): Option[Duration] = {
|
||||
SnowflakeId
|
||||
.timeFromIdOpt(
|
||||
candidate.features
|
||||
.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id))
|
||||
.map(Time.now - _)
|
||||
}
|
||||
}
|
@ -0,0 +1,128 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.config.yaml.YamlMap
|
||||
import com.twitter.finagle.tracing.Annotation.BinaryAnnotation
|
||||
import com.twitter.finagle.tracing.ForwardAnnotation
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.home_mixer.model.request.DeviceContext.RequestContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.DDGStatsAuthors
|
||||
import com.twitter.joinkey.context.RequestJoinKeyContext
|
||||
import com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor
|
||||
import com.twitter.product_mixer.core.pipeline.HasPipelineCursor
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.search.common.util.lang.ThriftLanguageUtil
|
||||
import com.twitter.snowflake.id.SnowflakeId
|
||||
import com.twitter.stitch.Stitch
|
||||
import java.util.UUID
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class RequestQueryFeatureHydrator[
|
||||
Query <: PipelineQuery with HasPipelineCursor[UrtOrderedCursor] with HasDeviceContext] @Inject() (
|
||||
@Named(DDGStatsAuthors) ddgStatsAuthors: YamlMap)
|
||||
extends QueryFeatureHydrator[Query] {
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AccountAgeFeature,
|
||||
ClientIdFeature,
|
||||
DDGStatsDemocratsFeature,
|
||||
DDGStatsRepublicansFeature,
|
||||
DDGStatsElonFeature,
|
||||
DDGStatsVitsFeature,
|
||||
DeviceLanguageFeature,
|
||||
GetInitialFeature,
|
||||
GetMiddleFeature,
|
||||
GetNewerFeature,
|
||||
GetOlderFeature,
|
||||
GuestIdFeature,
|
||||
HasDarkRequestFeature,
|
||||
IsForegroundRequestFeature,
|
||||
IsLaunchRequestFeature,
|
||||
PollingFeature,
|
||||
PullToRefreshFeature,
|
||||
RequestJoinIdFeature,
|
||||
ServedRequestIdFeature,
|
||||
ViewerIdFeature
|
||||
)
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Request")
|
||||
|
||||
private val DarkRequestAnnotation = "clnt/has_dark_request"
|
||||
private val Democrats = "democrats"
|
||||
private val Republicans = "republicans"
|
||||
private val Elon = "elon"
|
||||
private val Vits = "vits"
|
||||
|
||||
// Convert Language code to ISO 639-3 format
|
||||
private def getLanguageISOFormatByCode(languageCode: String): String =
|
||||
ThriftLanguageUtil.getLanguageCodeOf(ThriftLanguageUtil.getThriftLanguageOf(languageCode))
|
||||
|
||||
private def getRequestJoinId(servedRequestId: Long): Option[Long] =
|
||||
Some(RequestJoinKeyContext.current.flatMap(_.requestJoinId).getOrElse(servedRequestId))
|
||||
|
||||
private def hasDarkRequest: Option[Boolean] = ForwardAnnotation.current
|
||||
.getOrElse(Seq[BinaryAnnotation]())
|
||||
.find(_.key == DarkRequestAnnotation)
|
||||
.map(_.value.asInstanceOf[Boolean])
|
||||
|
||||
override def hydrate(query: Query): Stitch[FeatureMap] = {
|
||||
val requestContext = query.deviceContext.flatMap(_.requestContextValue)
|
||||
val servedRequestId = UUID.randomUUID.getMostSignificantBits
|
||||
|
||||
val featureMap = FeatureMapBuilder()
|
||||
.add(AccountAgeFeature, query.getOptionalUserId.flatMap(SnowflakeId.timeFromIdOpt))
|
||||
.add(ClientIdFeature, query.clientContext.appId)
|
||||
/**
|
||||
* These author ID lists are used purely for metrics collection. We track how often we are
|
||||
* serving Tweets from these authors and how often their tweets are being impressed by users.
|
||||
* This helps us validate in our A/B experimentation platform that we do not ship changes
|
||||
* that negatively impacts one group over others.
|
||||
*/
|
||||
.add(DDGStatsDemocratsFeature, ddgStatsAuthors.longSeq(Democrats).toSet)
|
||||
.add(DDGStatsRepublicansFeature, ddgStatsAuthors.longSeq(Republicans).toSet)
|
||||
.add(DDGStatsVitsFeature, ddgStatsAuthors.longSeq(Vits).toSet)
|
||||
.add(DDGStatsElonFeature, ddgStatsAuthors.longValue(Elon))
|
||||
.add(DeviceLanguageFeature, query.getLanguageCode.map(getLanguageISOFormatByCode))
|
||||
.add(
|
||||
GetInitialFeature,
|
||||
query.pipelineCursor.forall(cursor => cursor.id.isEmpty && cursor.gapBoundaryId.isEmpty))
|
||||
.add(
|
||||
GetMiddleFeature,
|
||||
query.pipelineCursor.exists(cursor =>
|
||||
cursor.id.isDefined && cursor.gapBoundaryId.isDefined &&
|
||||
cursor.cursorType.contains(GapCursor)))
|
||||
.add(
|
||||
GetNewerFeature,
|
||||
query.pipelineCursor.exists(cursor =>
|
||||
cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&
|
||||
cursor.cursorType.contains(TopCursor)))
|
||||
.add(
|
||||
GetOlderFeature,
|
||||
query.pipelineCursor.exists(cursor =>
|
||||
cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&
|
||||
cursor.cursorType.contains(BottomCursor)))
|
||||
.add(GuestIdFeature, query.clientContext.guestId)
|
||||
.add(IsForegroundRequestFeature, requestContext.contains(RequestContext.Foreground))
|
||||
.add(IsLaunchRequestFeature, requestContext.contains(RequestContext.Launch))
|
||||
.add(PollingFeature, query.deviceContext.exists(_.isPolling.contains(true)))
|
||||
.add(PullToRefreshFeature, requestContext.contains(RequestContext.PullToRefresh))
|
||||
.add(ServedRequestIdFeature, Some(servedRequestId))
|
||||
.add(RequestJoinIdFeature, getRequestJoinId(servedRequestId))
|
||||
.add(HasDarkRequestFeature, hasDarkRequest)
|
||||
.add(ViewerIdFeature, query.getRequiredUserId)
|
||||
.build()
|
||||
|
||||
Stitch.value(featureMap)
|
||||
}
|
||||
}
|
@ -0,0 +1,76 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.component_library.candidate_source.timeline_ranker.TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.search.common.features.thriftscala.ThriftTweetFeatures
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelineranker.thriftscala.CandidateTweet
|
||||
|
||||
object SourceTweetEarlybirdFeature extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]
|
||||
|
||||
/**
|
||||
* Feature Hydrator that bulk hydrates source tweets' features to retweet candidates
|
||||
*/
|
||||
object RetweetSourceTweetFeatureHydrator
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
|
||||
"RetweetSourceTweet")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
SourceTweetEarlybirdFeature,
|
||||
)
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(SourceTweetEarlybirdFeature, None)
|
||||
.build()
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val sourceTweetsByTweetId: Option[Map[Long, CandidateTweet]] = {
|
||||
query.features.map(
|
||||
_.getOrElse(
|
||||
TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature,
|
||||
Map.empty[Long, CandidateTweet]))
|
||||
}
|
||||
|
||||
/**
|
||||
* Return DefaultFeatureMap (no-op to candidate) when it is unfeasible to hydrate the
|
||||
* source tweet's feature to the current candidate: early bird does not return source
|
||||
* tweets info / candidate is not a retweet / sourceTweetId is not found
|
||||
*/
|
||||
Stitch.value {
|
||||
if (sourceTweetsByTweetId.exists(_.nonEmpty)) {
|
||||
candidates.map { candidate =>
|
||||
val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)
|
||||
val sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None)
|
||||
if (!candidateIsRetweet || sourceTweetId.isEmpty) {
|
||||
DefaultFeatureMap
|
||||
} else {
|
||||
val sourceTweet = sourceTweetsByTweetId.flatMap(_.get(sourceTweetId.get))
|
||||
if (sourceTweet.nonEmpty) {
|
||||
val source = sourceTweet.get
|
||||
FeatureMapBuilder()
|
||||
.add(SourceTweetEarlybirdFeature, source.features)
|
||||
.build()
|
||||
} else {
|
||||
DefaultFeatureMap
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
candidates.map(_ => DefaultFeatureMap)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,46 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.socialgraph.{thriftscala => sg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.socialgraph.{SocialGraph => SocialGraphStitchClient}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object SGSFollowedUsersFeature extends Feature[PipelineQuery, Seq[Long]]
|
||||
|
||||
@Singleton
|
||||
case class SGSFollowedUsersQueryFeatureHydrator @Inject() (
|
||||
socialGraphStitchClient: SocialGraphStitchClient)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("SGSFollowedUsers")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(SGSFollowedUsersFeature)
|
||||
|
||||
private val SocialGraphLimit = 14999
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
|
||||
val request = sg.IdsRequest(
|
||||
relationships = Seq(
|
||||
sg.SrcRelationship(userId, sg.RelationshipType.Following, hasRelationship = true),
|
||||
sg.SrcRelationship(userId, sg.RelationshipType.Muting, hasRelationship = false)
|
||||
),
|
||||
pageRequest = Some(sg.PageRequest(count = Some(SocialGraphLimit)))
|
||||
)
|
||||
|
||||
socialGraphStitchClient
|
||||
.ids(request).map(_.ids)
|
||||
.map { followedUsers =>
|
||||
FeatureMapBuilder().add(SGSFollowedUsersFeature, followedUsers).build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,105 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.socialgraph.{thriftscala => sg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.socialgraph.SocialGraph
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* This hydrator takes liked-by and followed-by user ids and checks via SGS that the viewer is
|
||||
* following the engager, that the viewer is not blocking the engager, that the engager is not
|
||||
* blocking the viewer, and that the viewer has not muted the engager.
|
||||
*/
|
||||
@Singleton
|
||||
class SGSValidSocialContextFeatureHydrator @Inject() (
|
||||
socialGraph: SocialGraph)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("SGSValidSocialContext")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
SGSValidFollowedByUserIdsFeature,
|
||||
SGSValidLikedByUserIdsFeature
|
||||
)
|
||||
|
||||
private val MaxCountUsers = 10
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
|
||||
val allSocialContextUserIds =
|
||||
candidates.flatMap { candidate =>
|
||||
candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++
|
||||
candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)
|
||||
}.distinct
|
||||
|
||||
getValidUserIds(query.getRequiredUserId, allSocialContextUserIds).map { validUserIds =>
|
||||
candidates.map { candidate =>
|
||||
val sgsFilteredLikedByUserIds =
|
||||
candidate.features
|
||||
.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers)
|
||||
.filter(validUserIds.contains)
|
||||
|
||||
val sgsFilteredFollowedByUserIds =
|
||||
candidate.features
|
||||
.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)
|
||||
.filter(validUserIds.contains)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(SGSValidFollowedByUserIdsFeature, sgsFilteredFollowedByUserIds)
|
||||
.add(SGSValidLikedByUserIdsFeature, sgsFilteredLikedByUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def getValidUserIds(
|
||||
viewerId: Long,
|
||||
socialProofUserIds: Seq[Long]
|
||||
): Stitch[Seq[Long]] = {
|
||||
if (socialProofUserIds.nonEmpty) {
|
||||
val request = sg.IdsRequest(
|
||||
relationships = Seq(
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.Following,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = true),
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.Blocking,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = false),
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.BlockedBy,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = false),
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.Muting,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = false)
|
||||
),
|
||||
pageRequest = Some(sg.PageRequest(selectAll = Some(true)))
|
||||
)
|
||||
socialGraph.ids(request).map(_.ids)
|
||||
} else Stitch.Nil
|
||||
}
|
||||
}
|
@ -0,0 +1,83 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.clients.strato.twistly.SimClustersRecentEngagementSimilarityClient
|
||||
import com.twitter.timelines.configapi.decider.BooleanDeciderParam
|
||||
import com.twitter.timelines.prediction.adapters.twistly.SimClustersRecentEngagementSimilarityFeaturesAdapter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object SimClustersEngagementSimilarityFeature
|
||||
extends DataRecordInAFeature[PipelineQuery]
|
||||
with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class SimClustersEngagementSimilarityFeatureHydrator @Inject() (
|
||||
simClustersEngagementSimilarityClient: SimClustersRecentEngagementSimilarityClient,
|
||||
statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("SimClustersEngagementSimilarity")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(SimClustersEngagementSimilarityFeature)
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(identifier.toString)
|
||||
|
||||
private val hydratedCandidatesSizeStat = scopedStatsReceiver.stat("hydratedCandidatesSize")
|
||||
|
||||
private val simClustersRecentEngagementSimilarityFeaturesAdapter =
|
||||
new SimClustersRecentEngagementSimilarityFeaturesAdapter
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean = {
|
||||
val param: BooleanDeciderParam =
|
||||
ScoredTweetsParam.EnableSimClustersSimilarityFeatureHydrationDeciderParam
|
||||
query.params.apply(param)
|
||||
}
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val tweetToCandidates = candidates.map(candidate => candidate.candidate.id -> candidate).toMap
|
||||
val tweetIds = tweetToCandidates.keySet.toSeq
|
||||
val userId = query.getRequiredUserId
|
||||
val userTweetEdges = tweetIds.map(tweetId => (userId, tweetId))
|
||||
val resultFuture = simClustersEngagementSimilarityClient
|
||||
.getSimClustersRecentEngagementSimilarityScores(userTweetEdges).map {
|
||||
simClustersRecentEngagementSimilarityScoresMap =>
|
||||
hydratedCandidatesSizeStat.add(simClustersRecentEngagementSimilarityScoresMap.size)
|
||||
candidates.map { candidate =>
|
||||
val similarityFeatureOpt = simClustersRecentEngagementSimilarityScoresMap
|
||||
.get(userId -> candidate.candidate.id).flatten
|
||||
val dataRecordOpt = similarityFeatureOpt.map { similarityFeature =>
|
||||
simClustersRecentEngagementSimilarityFeaturesAdapter
|
||||
.adaptToDataRecords(similarityFeature)
|
||||
.get(0)
|
||||
}
|
||||
FeatureMapBuilder()
|
||||
.add(SimClustersEngagementSimilarityFeature, dataRecordOpt.getOrElse(new DataRecord))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
Stitch.callFuture(resultFuture)
|
||||
}
|
||||
|
||||
}
|
@ -0,0 +1,67 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.socialgraph.{thriftscala => sg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.socialgraph.{SocialGraph => SocialGraphStitchClient}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class SocialGraphServiceFeatureHydrator @Inject() (socialGraphStitchClient: SocialGraphStitchClient)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("SocialGraphService")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(InNetworkFeature)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val viewerId = query.getRequiredUserId
|
||||
|
||||
// We use authorId and not sourceAuthorId here so that retweets are defined as in network
|
||||
val authorIds = candidates.map(_.features.getOrElse(AuthorIdFeature, None).getOrElse(0L))
|
||||
val distinctNonSelfAuthorIds = authorIds.filter(_ != viewerId).distinct
|
||||
|
||||
val idsRequest = createIdsRequest(
|
||||
userId = viewerId,
|
||||
relationshipTypes = Set(sg.RelationshipType.Following),
|
||||
targetIds = Some(distinctNonSelfAuthorIds)
|
||||
)
|
||||
|
||||
socialGraphStitchClient
|
||||
.ids(request = idsRequest, requestContext = None)
|
||||
.map { idResult =>
|
||||
authorIds.map { authorId =>
|
||||
// Users cannot follow themselves but this is in network by definition
|
||||
val isSelfTweet = authorId == viewerId
|
||||
val inNetworkAuthorIds = idResult.ids.toSet
|
||||
val isInNetwork = isSelfTweet || inNetworkAuthorIds.contains(authorId) || authorId == 0L
|
||||
FeatureMapBuilder().add(InNetworkFeature, isInNetwork).build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def createIdsRequest(
|
||||
userId: Long,
|
||||
relationshipTypes: Set[sg.RelationshipType],
|
||||
targetIds: Option[Seq[Long]] = None
|
||||
): sg.IdsRequest = sg.IdsRequest(
|
||||
relationshipTypes.map { relationshipType =>
|
||||
sg.SrcRelationship(userId, relationshipType, targets = targetIds)
|
||||
}.toSeq,
|
||||
Some(sg.PageRequest(selectAll = Some(true)))
|
||||
)
|
||||
}
|
@ -0,0 +1,162 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.contentrecommender.{thriftscala => cr}
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.inferred_topic.InferredTopicAdapter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TSPMetricTagFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.strato.generated.client.topic_signals.tsp.TopicSocialProofClientColumn
|
||||
import com.twitter.timelineservice.suggests.logging.candidate_tweet_source_id.{thriftscala => sid}
|
||||
import com.twitter.topiclisting.TopicListingViewerContext
|
||||
import com.twitter.tsp.{thriftscala => tsp}
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object TSPInferredTopicFeature extends Feature[TweetCandidate, Map[Long, Double]]
|
||||
object TSPInferredTopicDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class TSPInferredTopicFeatureHydrator @Inject() (
|
||||
topicSocialProofClientColumn: TopicSocialProofClientColumn,
|
||||
statsReceiver: StatsReceiver,
|
||||
) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TSPInferredTopic")
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(
|
||||
TSPInferredTopicFeature,
|
||||
TSPInferredTopicDataRecordFeature,
|
||||
TopicIdSocialContextFeature,
|
||||
TopicContextFunctionalityTypeFeature)
|
||||
|
||||
private val topK = 3
|
||||
|
||||
private val sourcesToSetSocialProof: Set[sid.CandidateTweetSourceId] = Set(
|
||||
sid.CandidateTweetSourceId.Simcluster,
|
||||
sid.CandidateTweetSourceId.CroonTweet
|
||||
)
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
|
||||
private val keyFoundCounter = scopedStatsReceiver.counter("key/found")
|
||||
private val keyLossCounter = scopedStatsReceiver.counter("key/loss")
|
||||
private val requestFailCounter = scopedStatsReceiver.counter("request/fail")
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(TSPInferredTopicFeature, Map.empty[Long, Double])
|
||||
.add(TSPInferredTopicDataRecordFeature, new DataRecord())
|
||||
.add(TopicIdSocialContextFeature, None)
|
||||
.add(TopicContextFunctionalityTypeFeature, None)
|
||||
.build()
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val tags = candidates.collect {
|
||||
case candidate if candidate.features.getTry(TSPMetricTagFeature).isReturn =>
|
||||
candidate.candidate.id -> candidate.features
|
||||
.getOrElse(TSPMetricTagFeature, Set.empty[tsp.MetricTag])
|
||||
}.toMap
|
||||
|
||||
val topicSocialProofRequest =
|
||||
tsp.TopicSocialProofRequest(
|
||||
userId = query.getRequiredUserId,
|
||||
tweetIds = candidates.map(_.candidate.id).toSet,
|
||||
displayLocation = cr.DisplayLocation.HomeTimeline,
|
||||
topicListingSetting = tsp.TopicListingSetting.Followable,
|
||||
context = TopicListingViewerContext.fromClientContext(query.clientContext).toThrift,
|
||||
bypassModes = None,
|
||||
// Only CRMixer source has this data. Convert the CRMixer metric tag to tsp metric tag.
|
||||
tags = if (tags.isEmpty) None else Some(tags)
|
||||
)
|
||||
|
||||
topicSocialProofClientColumn.fetcher
|
||||
.fetch(topicSocialProofRequest)
|
||||
.map(_.v)
|
||||
.map {
|
||||
case Some(response) =>
|
||||
candidates.map { candidate =>
|
||||
val topicWithScores = response.socialProofs.getOrElse(candidate.candidate.id, Seq.empty)
|
||||
if (topicWithScores.nonEmpty) {
|
||||
keyFoundCounter.incr()
|
||||
val (socialProofId, socialProofFunctionalityType) =
|
||||
if (candidate.features
|
||||
.getOrElse(CandidateSourceIdFeature, None)
|
||||
.exists(sourcesToSetSocialProof.contains)) {
|
||||
getSocialProof(topicWithScores)
|
||||
} else {
|
||||
(None, None)
|
||||
}
|
||||
val inferredTopicFeatures = convertTopicWithScores(topicWithScores)
|
||||
val inferredTopicDataRecord =
|
||||
InferredTopicAdapter.adaptToDataRecords(inferredTopicFeatures).asScala.head
|
||||
FeatureMapBuilder()
|
||||
.add(TSPInferredTopicFeature, inferredTopicFeatures)
|
||||
.add(TSPInferredTopicDataRecordFeature, inferredTopicDataRecord)
|
||||
.add(TopicIdSocialContextFeature, socialProofId)
|
||||
.add(TopicContextFunctionalityTypeFeature, socialProofFunctionalityType)
|
||||
.build()
|
||||
} else {
|
||||
keyLossCounter.incr()
|
||||
DefaultFeatureMap
|
||||
}
|
||||
}
|
||||
case _ =>
|
||||
requestFailCounter.incr()
|
||||
candidates.map { _ =>
|
||||
DefaultFeatureMap
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def getSocialProof(
|
||||
topicWithScores: Seq[tsp.TopicWithScore]
|
||||
): (Option[Long], Option[TopicContextFunctionalityType]) = {
|
||||
val followingTopicId = topicWithScores
|
||||
.collectFirst {
|
||||
case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.Following)) =>
|
||||
topicId
|
||||
}
|
||||
if (followingTopicId.nonEmpty) {
|
||||
return (followingTopicId, Some(BasicTopicContextFunctionalityType))
|
||||
}
|
||||
val implicitFollowingId = topicWithScores.collectFirst {
|
||||
case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.ImplicitFollow)) =>
|
||||
topicId
|
||||
}
|
||||
if (implicitFollowingId.nonEmpty) {
|
||||
return (implicitFollowingId, Some(RecommendationTopicContextFunctionalityType))
|
||||
}
|
||||
(None, None)
|
||||
}
|
||||
|
||||
private def convertTopicWithScores(
|
||||
topicWithScores: Seq[tsp.TopicWithScore],
|
||||
): Map[Long, Double] = {
|
||||
topicWithScores.sortBy(-_.score).take(topK).map(a => (a.topicId, a.score)).toMap
|
||||
}
|
||||
}
|
@ -0,0 +1,251 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.ml.api.RichDataRecord
|
||||
import com.twitter.ml.api.util.FDsl._
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.search.common.features.{thriftscala => sc}
|
||||
import com.twitter.snowflake.id.SnowflakeId
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.features.time_features.AccountAgeInterval
|
||||
import com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures._
|
||||
import com.twitter.timelines.prediction.features.time_features.TimeFeatures
|
||||
import com.twitter.util.Duration
|
||||
import scala.collection.Searching._
|
||||
|
||||
object TimeFeaturesDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
object TimeFeaturesHydrator extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TimeFeatures")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TimeFeaturesDataRecordFeature)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
Stitch.value {
|
||||
val richDataRecord = new RichDataRecord()
|
||||
setTimeFeatures(richDataRecord, candidate, existingFeatures, query)
|
||||
FeatureMapBuilder()
|
||||
.add(TimeFeaturesDataRecordFeature, richDataRecord.getRecord)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
private def setTimeFeatures(
|
||||
richDataRecord: RichDataRecord,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap,
|
||||
query: PipelineQuery,
|
||||
): Unit = {
|
||||
val timeFeaturesOpt = getTimeFeatures(query, candidate, existingFeatures)
|
||||
timeFeaturesOpt.foreach(timeFeatures => setFeatures(timeFeatures, richDataRecord))
|
||||
}
|
||||
|
||||
private[feature_hydrator] def getTimeFeatures(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap,
|
||||
): Option[TimeFeatures] = {
|
||||
for {
|
||||
requestTimestampMs <- Some(query.queryTime.inMilliseconds)
|
||||
tweetId <- Some(candidate.id)
|
||||
viewerId <- query.getOptionalUserId
|
||||
tweetCreationTimeMs <- timeFromTweetOrUserId(tweetId)
|
||||
timeSinceTweetCreation = requestTimestampMs - tweetCreationTimeMs
|
||||
accountAgeDurationOpt = timeFromTweetOrUserId(viewerId).map { viewerAccountCreationTimeMs =>
|
||||
Duration.fromMilliseconds(requestTimestampMs - viewerAccountCreationTimeMs)
|
||||
}
|
||||
timeSinceSourceTweetCreation =
|
||||
existingFeatures
|
||||
.getOrElse(SourceTweetIdFeature, None)
|
||||
.flatMap { sourceTweetId =>
|
||||
timeFromTweetOrUserId(sourceTweetId).map { sourceTweetCreationTimeMs =>
|
||||
requestTimestampMs - sourceTweetCreationTimeMs
|
||||
}
|
||||
}
|
||||
.getOrElse(timeSinceTweetCreation)
|
||||
if (timeSinceTweetCreation > 0 && timeSinceSourceTweetCreation > 0)
|
||||
} yield {
|
||||
val timeFeatures = TimeFeatures(
|
||||
timeSinceTweetCreation = timeSinceTweetCreation,
|
||||
timeSinceSourceTweetCreation = timeSinceSourceTweetCreation,
|
||||
timeSinceViewerAccountCreationSecs = accountAgeDurationOpt.map(_.inSeconds),
|
||||
isDay30NewUser = accountAgeDurationOpt.map(_ < 30.days).getOrElse(false),
|
||||
isMonth12NewUser = accountAgeDurationOpt.map(_ < 365.days).getOrElse(false),
|
||||
accountAgeInterval = accountAgeDurationOpt.flatMap(AccountAgeInterval.fromDuration),
|
||||
isTweetRecycled = false // only set in RecyclableTweetCandidateFilter, but it's not used
|
||||
)
|
||||
|
||||
val timeFeaturesWithLastEngagement = addLastEngagementTimeFeatures(
|
||||
existingFeatures.getOrElse(EarlybirdFeature, None),
|
||||
timeFeatures,
|
||||
timeSinceSourceTweetCreation
|
||||
).getOrElse(timeFeatures)
|
||||
|
||||
val nonPollingTimestampsMs =
|
||||
query.features.map(_.getOrElse(NonPollingTimesFeature, Seq.empty))
|
||||
val timeFeaturesWithNonPollingOpt = addNonPollingTimeFeatures(
|
||||
timeFeaturesWithLastEngagement,
|
||||
requestTimestampMs,
|
||||
tweetCreationTimeMs,
|
||||
nonPollingTimestampsMs
|
||||
)
|
||||
timeFeaturesWithNonPollingOpt.getOrElse(timeFeaturesWithLastEngagement)
|
||||
}
|
||||
}
|
||||
|
||||
private def timeFromTweetOrUserId(tweetOrUserId: Long): Option[Long] = {
|
||||
if (SnowflakeId.isSnowflakeId(tweetOrUserId))
|
||||
Some(SnowflakeId(tweetOrUserId).time.inMilliseconds)
|
||||
else None
|
||||
}
|
||||
|
||||
private def addLastEngagementTimeFeatures(
|
||||
tweetFeaturesOpt: Option[sc.ThriftTweetFeatures],
|
||||
timeFeatures: TimeFeatures,
|
||||
timeSinceSourceTweetCreation: Long
|
||||
): Option[TimeFeatures] = {
|
||||
tweetFeaturesOpt.map { tweetFeatures =>
|
||||
val lastFavSinceCreationHrs = tweetFeatures.lastFavSinceCreationHrs.map(_.toDouble)
|
||||
val lastRetweetSinceCreationHrs = tweetFeatures.lastRetweetSinceCreationHrs.map(_.toDouble)
|
||||
val lastReplySinceCreationHrs = tweetFeatures.lastReplySinceCreationHrs.map(_.toDouble)
|
||||
val lastQuoteSinceCreationHrs = tweetFeatures.lastQuoteSinceCreationHrs.map(_.toDouble)
|
||||
|
||||
timeFeatures.copy(
|
||||
lastFavSinceCreationHrs = lastFavSinceCreationHrs,
|
||||
lastRetweetSinceCreationHrs = lastRetweetSinceCreationHrs,
|
||||
lastReplySinceCreationHrs = lastReplySinceCreationHrs,
|
||||
lastQuoteSinceCreationHrs = lastQuoteSinceCreationHrs,
|
||||
timeSinceLastFavoriteHrs = getTimeSinceLastEngagementHrs(
|
||||
lastFavSinceCreationHrs,
|
||||
timeSinceSourceTweetCreation
|
||||
),
|
||||
timeSinceLastRetweetHrs = getTimeSinceLastEngagementHrs(
|
||||
lastRetweetSinceCreationHrs,
|
||||
timeSinceSourceTweetCreation
|
||||
),
|
||||
timeSinceLastReplyHrs = getTimeSinceLastEngagementHrs(
|
||||
lastReplySinceCreationHrs,
|
||||
timeSinceSourceTweetCreation
|
||||
),
|
||||
timeSinceLastQuoteHrs = getTimeSinceLastEngagementHrs(
|
||||
lastQuoteSinceCreationHrs,
|
||||
timeSinceSourceTweetCreation
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private def addNonPollingTimeFeatures(
|
||||
timeFeatures: TimeFeatures,
|
||||
requestTimestampMs: Long,
|
||||
creationTimeMs: Long,
|
||||
nonPollingTimestampsMs: Option[Seq[Long]]
|
||||
): Option[TimeFeatures] = {
|
||||
for {
|
||||
nonPollingTimestampsMs <- nonPollingTimestampsMs
|
||||
lastNonPollingTimestampMs <- nonPollingTimestampsMs.headOption
|
||||
earliestNonPollingTimestampMs <- nonPollingTimestampsMs.lastOption
|
||||
} yield {
|
||||
val timeSinceLastNonPollingRequest = requestTimestampMs - lastNonPollingTimestampMs
|
||||
val tweetAgeRatio = timeSinceLastNonPollingRequest / math.max(
|
||||
1.0,
|
||||
timeFeatures.timeSinceTweetCreation
|
||||
)
|
||||
/*
|
||||
* Non-polling timestamps are stored in chronological order.
|
||||
* The latest timestamps occur first, therefore we need to explicitly search in reverse order.
|
||||
*/
|
||||
val nonPollingRequestsSinceTweetCreation =
|
||||
if (nonPollingTimestampsMs.nonEmpty) {
|
||||
nonPollingTimestampsMs.search(creationTimeMs)(Ordering[Long].reverse).insertionPoint
|
||||
} else {
|
||||
0
|
||||
}
|
||||
/*
|
||||
* Calculate the average time between non-polling requests; include
|
||||
* request time in this calculation as latest timestamp.
|
||||
*/
|
||||
val timeBetweenNonPollingRequestsAvg =
|
||||
(requestTimestampMs - earliestNonPollingTimestampMs) / math
|
||||
.max(1.0, nonPollingTimestampsMs.size)
|
||||
val timeFeaturesWithNonPolling = timeFeatures.copy(
|
||||
timeBetweenNonPollingRequestsAvg = Some(timeBetweenNonPollingRequestsAvg),
|
||||
timeSinceLastNonPollingRequest = Some(timeSinceLastNonPollingRequest),
|
||||
nonPollingRequestsSinceTweetCreation = Some(nonPollingRequestsSinceTweetCreation),
|
||||
tweetAgeRatio = Some(tweetAgeRatio)
|
||||
)
|
||||
timeFeaturesWithNonPolling
|
||||
}
|
||||
}
|
||||
|
||||
private[this] def getTimeSinceLastEngagementHrs(
|
||||
lastEngagementTimeSinceCreationHrsOpt: Option[Double],
|
||||
timeSinceTweetCreation: Long
|
||||
): Option[Double] = {
|
||||
lastEngagementTimeSinceCreationHrsOpt.map { lastEngagementTimeSinceCreationHrs =>
|
||||
val timeSinceTweetCreationHrs = (timeSinceTweetCreation / (60 * 60 * 1000)).toInt
|
||||
timeSinceTweetCreationHrs - lastEngagementTimeSinceCreationHrs
|
||||
}
|
||||
}
|
||||
|
||||
private def setFeatures(features: TimeFeatures, richDataRecord: RichDataRecord): Unit = {
|
||||
val record = richDataRecord.getRecord
|
||||
.setFeatureValue(IS_TWEET_RECYCLED, features.isTweetRecycled)
|
||||
.setFeatureValue(TIME_SINCE_TWEET_CREATION, features.timeSinceTweetCreation)
|
||||
.setFeatureValueFromOption(
|
||||
TIME_SINCE_VIEWER_ACCOUNT_CREATION_SECS,
|
||||
features.timeSinceViewerAccountCreationSecs)
|
||||
.setFeatureValue(
|
||||
USER_ID_IS_SNOWFLAKE_ID,
|
||||
features.timeSinceViewerAccountCreationSecs.isDefined
|
||||
)
|
||||
.setFeatureValueFromOption(ACCOUNT_AGE_INTERVAL, features.accountAgeInterval.map(_.id.toLong))
|
||||
.setFeatureValue(IS_30_DAY_NEW_USER, features.isDay30NewUser)
|
||||
.setFeatureValue(IS_12_MONTH_NEW_USER, features.isMonth12NewUser)
|
||||
.setFeatureValueFromOption(LAST_FAVORITE_SINCE_CREATION_HRS, features.lastFavSinceCreationHrs)
|
||||
.setFeatureValueFromOption(
|
||||
LAST_RETWEET_SINCE_CREATION_HRS,
|
||||
features.lastRetweetSinceCreationHrs
|
||||
)
|
||||
.setFeatureValueFromOption(LAST_REPLY_SINCE_CREATION_HRS, features.lastReplySinceCreationHrs)
|
||||
.setFeatureValueFromOption(LAST_QUOTE_SINCE_CREATION_HRS, features.lastQuoteSinceCreationHrs)
|
||||
.setFeatureValueFromOption(TIME_SINCE_LAST_FAVORITE_HRS, features.timeSinceLastFavoriteHrs)
|
||||
.setFeatureValueFromOption(TIME_SINCE_LAST_RETWEET_HRS, features.timeSinceLastRetweetHrs)
|
||||
.setFeatureValueFromOption(TIME_SINCE_LAST_REPLY_HRS, features.timeSinceLastReplyHrs)
|
||||
.setFeatureValueFromOption(TIME_SINCE_LAST_QUOTE_HRS, features.timeSinceLastQuoteHrs)
|
||||
/*
|
||||
* set features whose values are optional as some users do not have non-polling timestamps
|
||||
*/
|
||||
features.timeBetweenNonPollingRequestsAvg.foreach(
|
||||
record.setFeatureValue(TIME_BETWEEN_NON_POLLING_REQUESTS_AVG, _)
|
||||
)
|
||||
features.timeSinceLastNonPollingRequest.foreach(
|
||||
record.setFeatureValue(TIME_SINCE_LAST_NON_POLLING_REQUEST, _)
|
||||
)
|
||||
features.nonPollingRequestsSinceTweetCreation.foreach(
|
||||
record.setFeatureValue(NON_POLLING_REQUESTS_SINCE_TWEET_CREATION, _)
|
||||
)
|
||||
features.tweetAgeRatio.foreach(record.setFeatureValue(TWEET_AGE_RATIO, _))
|
||||
}
|
||||
}
|
@ -0,0 +1,63 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.marshaller.timelines.DeviceContextMarshaller
|
||||
import com.twitter.home_mixer.model.request.DeviceContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.timelineservice.TimelineService
|
||||
import com.twitter.timelineservice.{thriftscala => t}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object TimelineServiceTweetsFeature extends Feature[PipelineQuery, Seq[Long]]
|
||||
|
||||
@Singleton
|
||||
case class TimelineServiceTweetsQueryFeatureHydrator @Inject() (
|
||||
timelineService: TimelineService,
|
||||
deviceContextMarshaller: DeviceContextMarshaller)
|
||||
extends QueryFeatureHydrator[PipelineQuery with HasDeviceContext] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("TimelineServiceTweets")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TimelineServiceTweetsFeature)
|
||||
|
||||
private val MaxTimelineServiceTweets = 200
|
||||
|
||||
override def hydrate(query: PipelineQuery with HasDeviceContext): Stitch[FeatureMap] = {
|
||||
val deviceContext = query.deviceContext.getOrElse(DeviceContext.Empty)
|
||||
|
||||
val timelineQueryOptions = t.TimelineQueryOptions(
|
||||
contextualUserId = query.clientContext.userId,
|
||||
deviceContext = Some(deviceContextMarshaller(deviceContext, query.clientContext))
|
||||
)
|
||||
|
||||
val timelineServiceQuery = t.TimelineQuery(
|
||||
timelineType = t.TimelineType.Home,
|
||||
timelineId = query.getRequiredUserId,
|
||||
maxCount = MaxTimelineServiceTweets.toShort,
|
||||
cursor2 = None,
|
||||
options = Some(timelineQueryOptions),
|
||||
timelineId2 = query.clientContext.userId.map(t.TimelineId(t.TimelineType.Home, _, None)),
|
||||
)
|
||||
|
||||
timelineService.getTimeline(timelineServiceQuery).map { timeline =>
|
||||
val tweets = timeline.entries.collect {
|
||||
case t.TimelineEntry.Tweet(tweet) => tweet.statusId
|
||||
}
|
||||
|
||||
FeatureMapBuilder().add(TimelineServiceTweetsFeature, tweets).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)
|
||||
)
|
||||
}
|
@ -0,0 +1,87 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TweetImpressionsFeature
|
||||
import com.twitter.home_mixer.model.request.HasSeenTweetIds
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.impression.{thriftscala => t}
|
||||
import com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClient
|
||||
import com.twitter.util.Duration
|
||||
import com.twitter.util.Time
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class TweetImpressionsQueryFeatureHydrator[
|
||||
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
|
||||
manhattanTweetImpressionStoreClient: ManhattanTweetImpressionStoreClient)
|
||||
extends QueryFeatureHydrator[Query] {
|
||||
|
||||
private val TweetImpressionTTL = 1.day
|
||||
private val TweetImpressionCap = 3000
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetImpressions")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TweetImpressionsFeature)
|
||||
|
||||
override def hydrate(query: Query): Stitch[FeatureMap] = {
|
||||
manhattanTweetImpressionStoreClient.get(query.getRequiredUserId).map { entriesOpt =>
|
||||
val entries = entriesOpt.map(_.entries).toSeq.flatten
|
||||
val updatedImpressions =
|
||||
if (query.seenTweetIds.forall(_.isEmpty)) entries
|
||||
else updateTweetImpressions(entries, query.seenTweetIds.get)
|
||||
|
||||
FeatureMapBuilder().add(TweetImpressionsFeature, updatedImpressions).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)
|
||||
)
|
||||
|
||||
/**
|
||||
* 1) Check timestamps and remove expired tweets based on [[TweetImpressionTTL]]
|
||||
* 2) Filter duplicates between current tweets and those in the impression store (remove older ones)
|
||||
* 3) Prepend new (Timestamp, Seq[TweetIds]) to the tweets from the impression store
|
||||
* 4) Truncate older tweets if sum of all tweets across timestamps >= [[TweetImpressionCap]],
|
||||
*/
|
||||
private[feature_hydrator] def updateTweetImpressions(
|
||||
tweetImpressionsFromStore: Seq[t.TweetImpressionsEntry],
|
||||
seenIdsFromClient: Seq[Long],
|
||||
currentTime: Long = Time.now.inMilliseconds,
|
||||
tweetImpressionTTL: Duration = TweetImpressionTTL,
|
||||
tweetImpressionCap: Int = TweetImpressionCap,
|
||||
): Seq[t.TweetImpressionsEntry] = {
|
||||
val seenIdsFromClientSet = seenIdsFromClient.toSet
|
||||
val dedupedTweetImpressionsFromStore: Seq[t.TweetImpressionsEntry] = tweetImpressionsFromStore
|
||||
.collect {
|
||||
case t.TweetImpressionsEntry(ts, tweetIds)
|
||||
if Time.fromMilliseconds(ts).untilNow < tweetImpressionTTL =>
|
||||
t.TweetImpressionsEntry(ts, tweetIds.filterNot(seenIdsFromClientSet.contains))
|
||||
}.filter { _.tweetIds.nonEmpty }
|
||||
|
||||
val mergedTweetImpressionsEntries =
|
||||
t.TweetImpressionsEntry(currentTime, seenIdsFromClient) +: dedupedTweetImpressionsFromStore
|
||||
val initialTweetImpressionsWithCap = (Seq.empty[t.TweetImpressionsEntry], tweetImpressionCap)
|
||||
|
||||
val (truncatedTweetImpressionsEntries: Seq[t.TweetImpressionsEntry], _) =
|
||||
mergedTweetImpressionsEntries
|
||||
.foldLeft(initialTweetImpressionsWithCap) {
|
||||
case (
|
||||
(tweetImpressions: Seq[t.TweetImpressionsEntry], remainingCap),
|
||||
t.TweetImpressionsEntry(ts, tweetIds)) if remainingCap > 0 =>
|
||||
(
|
||||
t.TweetImpressionsEntry(ts, tweetIds.take(remainingCap)) +: tweetImpressions,
|
||||
remainingCap - tweetIds.size)
|
||||
case (tweetImpressionsWithCap, _) => tweetImpressionsWithCap
|
||||
}
|
||||
truncatedTweetImpressionsEntries.reverse
|
||||
}
|
||||
}
|
@ -0,0 +1,66 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.ml.api.RichDataRecord
|
||||
import com.twitter.ml.api.constant.SharedFeatures
|
||||
import com.twitter.ml.api.util.DataRecordConverters._
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures
|
||||
import java.lang.{Long => JLong}
|
||||
|
||||
object TweetMetaDataDataRecord
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
object TweetMetaDataFeatureHydrator
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetMetaData")
|
||||
|
||||
override def features: Set[Feature[_, _]] = Set(TweetMetaDataDataRecord)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val richDataRecord = new RichDataRecord()
|
||||
|
||||
setFeatures(richDataRecord, candidate, existingFeatures)
|
||||
|
||||
Stitch.value {
|
||||
FeatureMapBuilder()
|
||||
.add(TweetMetaDataDataRecord, richDataRecord.getRecord)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
private def setFeatures(
|
||||
richDataRecord: RichDataRecord,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Unit = {
|
||||
richDataRecord.setFeatureValue[JLong](SharedFeatures.TWEET_ID, candidate.id)
|
||||
|
||||
richDataRecord.setFeatureValueFromOption(
|
||||
TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID,
|
||||
CandidatesUtil.getOriginalAuthorId(existingFeatures))
|
||||
|
||||
richDataRecord.setFeatureValueFromOption(
|
||||
TimelinesSharedFeatures.CANDIDATE_TWEET_SOURCE_ID,
|
||||
existingFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toLong))
|
||||
}
|
||||
}
|
@ -0,0 +1,149 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.escherbird.{thriftscala => esb}
|
||||
import com.twitter.finagle.stats.Stat
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.ContentFeatureAdapter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.MediaUnderstandingAnnotationIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieContentRepository
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.home_mixer.util.tweetypie.content.FeatureExtractionHelper
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.util.OffloadFuturePools
|
||||
import com.twitter.servo.keyvalue.KeyValueResult
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.common.util.MediaUnderstandingAnnotations
|
||||
import com.twitter.tweetypie.{thriftscala => tp}
|
||||
import com.twitter.util.Future
|
||||
import com.twitter.util.Return
|
||||
import com.twitter.util.Throw
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object TweetypieContentDataRecordFeature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class TweetypieContentFeatureHydrator @Inject() (
|
||||
@Named(TweetypieContentRepository) client: KeyValueRepository[Seq[Long], Long, tp.Tweet],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetypieContent")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
MediaUnderstandingAnnotationIdsFeature,
|
||||
TweetypieContentDataRecordFeature
|
||||
)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
private val bulkRequestLatencyStat =
|
||||
statsReceiver.scope(statScope).scope("bulkRequest").stat("latency_ms")
|
||||
private val postTransformerLatencyStat =
|
||||
statsReceiver.scope(statScope).scope("postTransformer").stat("latency_ms")
|
||||
private val bulkPostTransformerLatencyStat =
|
||||
statsReceiver.scope(statScope).scope("bulkPostTransformer").stat("latency_ms")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
Stitch.callFuture {
|
||||
val tweetIdsToHydrate = candidates.map(getCandidateOriginalTweetId).distinct
|
||||
|
||||
val response: Future[KeyValueResult[Long, tp.Tweet]] =
|
||||
Stat.timeFuture(bulkRequestLatencyStat)(
|
||||
if (tweetIdsToHydrate.isEmpty) {
|
||||
Future.value(KeyValueResult.empty)
|
||||
} else {
|
||||
client(tweetIdsToHydrate)
|
||||
}
|
||||
)
|
||||
|
||||
response.flatMap { result =>
|
||||
Stat.timeFuture(bulkPostTransformerLatencyStat) {
|
||||
OffloadFuturePools
|
||||
.parallelize[CandidateWithFeatures[TweetCandidate], Try[(Seq[Long], DataRecord)]](
|
||||
candidates,
|
||||
parTransformer(result, _),
|
||||
parallelism = 32,
|
||||
default = Return((Seq.empty, new DataRecord))
|
||||
).map {
|
||||
_.map {
|
||||
case Return(result) =>
|
||||
FeatureMapBuilder()
|
||||
.add(MediaUnderstandingAnnotationIdsFeature, result._1)
|
||||
.add(TweetypieContentDataRecordFeature, result._2)
|
||||
.build()
|
||||
case Throw(e) =>
|
||||
FeatureMapBuilder()
|
||||
.add(MediaUnderstandingAnnotationIdsFeature, Throw(e))
|
||||
.add(TweetypieContentDataRecordFeature, Throw(e))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def parTransformer(
|
||||
result: KeyValueResult[Long, tp.Tweet],
|
||||
candidate: CandidateWithFeatures[TweetCandidate]
|
||||
): Try[(Seq[Long], DataRecord)] = {
|
||||
val originalTweetId = Some(getCandidateOriginalTweetId(candidate))
|
||||
|
||||
val value = observedGet(key = originalTweetId, keyValueResult = result)
|
||||
Stat.time(postTransformerLatencyStat)(postTransformer(value))
|
||||
}
|
||||
|
||||
private def postTransformer(
|
||||
result: Try[Option[tp.Tweet]]
|
||||
): Try[(Seq[Long], DataRecord)] = {
|
||||
result.map { tweet =>
|
||||
val transformedValue = tweet.map(FeatureExtractionHelper.extractFeatures)
|
||||
val semanticAnnotations = transformedValue
|
||||
.flatMap { contentFeatures =>
|
||||
contentFeatures.semanticCoreAnnotations.map {
|
||||
getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds
|
||||
}
|
||||
}.getOrElse(Seq.empty)
|
||||
val dataRecord = ContentFeatureAdapter.adaptToDataRecords(transformedValue).asScala.head
|
||||
(semanticAnnotations, dataRecord)
|
||||
}
|
||||
}
|
||||
|
||||
private def getCandidateOriginalTweetId(
|
||||
candidate: CandidateWithFeatures[TweetCandidate]
|
||||
): Long = {
|
||||
candidate.features
|
||||
.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id)
|
||||
}
|
||||
|
||||
private def getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds(
|
||||
semanticCoreAnnotations: Seq[esb.TweetEntityAnnotation]
|
||||
): Seq[Long] =
|
||||
semanticCoreAnnotations
|
||||
.filter(MediaUnderstandingAnnotations.isEligibleNonSensitiveHighRecallMUAnnotation)
|
||||
.map(_.entityId)
|
||||
}
|
@ -0,0 +1,156 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsNsfwFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.model.request.ForYouProduct
|
||||
import com.twitter.home_mixer.model.request.ListTweetsProduct
|
||||
import com.twitter.home_mixer.model.request.ScoredTweetsProduct
|
||||
import com.twitter.home_mixer.util.tweetypie.RequestFields
|
||||
import com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw
|
||||
import com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.spam.rtf.{thriftscala => rtf}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}
|
||||
import com.twitter.tweetypie.{thriftscala => tp}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class TweetypieFeatureHydrator @Inject() (tweetypieStitchClient: TweetypieStitchClient)
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Tweetypie")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
IsHydratedFeature,
|
||||
IsNsfw,
|
||||
IsNsfwFeature,
|
||||
IsRetweetFeature,
|
||||
QuotedTweetDroppedFeature,
|
||||
QuotedTweetIdFeature,
|
||||
QuotedUserIdFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature,
|
||||
TweetTextFeature,
|
||||
VisibilityReason
|
||||
)
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(IsHydratedFeature, false)
|
||||
.add(IsNsfw, None)
|
||||
.add(IsNsfwFeature, false)
|
||||
.add(QuotedTweetDroppedFeature, false)
|
||||
.add(TweetTextFeature, None)
|
||||
.add(VisibilityReason, None)
|
||||
.build()
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val safetyLevel = query.product match {
|
||||
case FollowingProduct => rtf.SafetyLevel.TimelineHomeLatest
|
||||
case ForYouProduct => rtf.SafetyLevel.TimelineHome
|
||||
case ScoredTweetsProduct => rtf.SafetyLevel.TimelineHome
|
||||
case ListTweetsProduct => rtf.SafetyLevel.TimelineLists
|
||||
case unknown => throw new UnsupportedOperationException(s"Unknown product: $unknown")
|
||||
}
|
||||
|
||||
val tweetFieldsOptions = tp.GetTweetFieldsOptions(
|
||||
tweetIncludes = RequestFields.TweetTPHydrationFields,
|
||||
includeRetweetedTweet = true,
|
||||
includeQuotedTweet = true,
|
||||
visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,
|
||||
safetyLevel = Some(safetyLevel),
|
||||
forUserId = Some(query.getRequiredUserId)
|
||||
)
|
||||
|
||||
tweetypieStitchClient.getTweetFields(tweetId = candidate.id, options = tweetFieldsOptions).map {
|
||||
case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), quoteOpt, _) =>
|
||||
val coreData = found.tweet.coreData
|
||||
val isNsfwAdmin = coreData.exists(_.nsfwAdmin)
|
||||
val isNsfwUser = coreData.exists(_.nsfwUser)
|
||||
|
||||
val quotedTweetDropped = quoteOpt.exists {
|
||||
case _: tp.TweetFieldsResultState.Filtered => true
|
||||
case _: tp.TweetFieldsResultState.NotFound => true
|
||||
case _ => false
|
||||
}
|
||||
val quotedTweetIsNsfw = quoteOpt.exists {
|
||||
case quoteTweet: tp.TweetFieldsResultState.Found =>
|
||||
quoteTweet.found.tweet.coreData.exists(data => data.nsfwAdmin || data.nsfwUser)
|
||||
case _ => false
|
||||
}
|
||||
|
||||
val sourceTweetIsNsfw =
|
||||
found.retweetedTweet.exists(_.coreData.exists(data => data.nsfwAdmin || data.nsfwUser))
|
||||
|
||||
val tweetText = coreData.map(_.text)
|
||||
|
||||
val tweetAuthorId = coreData.map(_.userId)
|
||||
val inReplyToTweetId = coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))
|
||||
val retweetedTweetId = found.retweetedTweet.map(_.id)
|
||||
val quotedTweetId = quoteOpt.flatMap {
|
||||
case quoteTweet: tp.TweetFieldsResultState.Found =>
|
||||
Some(quoteTweet.found.tweet.id)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val retweetedTweetUserId = found.retweetedTweet.flatMap(_.coreData).map(_.userId)
|
||||
val quotedTweetUserId = quoteOpt.flatMap {
|
||||
case quoteTweet: tp.TweetFieldsResultState.Found =>
|
||||
quoteTweet.found.tweet.coreData.map(_.userId)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val isNsfw = isNsfwAdmin || isNsfwUser || sourceTweetIsNsfw || quotedTweetIsNsfw
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, tweetAuthorId)
|
||||
.add(InReplyToTweetIdFeature, inReplyToTweetId)
|
||||
.add(IsHydratedFeature, true)
|
||||
.add(IsNsfw, Some(isNsfw))
|
||||
.add(IsNsfwFeature, isNsfw)
|
||||
.add(IsRetweetFeature, retweetedTweetId.isDefined)
|
||||
.add(QuotedTweetDroppedFeature, quotedTweetDropped)
|
||||
.add(QuotedTweetIdFeature, quotedTweetId)
|
||||
.add(QuotedUserIdFeature, quotedTweetUserId)
|
||||
.add(SourceTweetIdFeature, retweetedTweetId)
|
||||
.add(SourceUserIdFeature, retweetedTweetUserId)
|
||||
.add(TweetTextFeature, tweetText)
|
||||
.add(VisibilityReason, found.suppressReason)
|
||||
.build()
|
||||
|
||||
// If no tweet result found, return default and pre-existing features
|
||||
case _ =>
|
||||
DefaultFeatureMap +
|
||||
(AuthorIdFeature, existingFeatures.getOrElse(AuthorIdFeature, None)) +
|
||||
(InReplyToTweetIdFeature, existingFeatures.getOrElse(InReplyToTweetIdFeature, None)) +
|
||||
(IsRetweetFeature, existingFeatures.getOrElse(IsRetweetFeature, false)) +
|
||||
(QuotedTweetIdFeature, existingFeatures.getOrElse(QuotedTweetIdFeature, None)) +
|
||||
(QuotedUserIdFeature, existingFeatures.getOrElse(QuotedUserIdFeature, None)) +
|
||||
(SourceTweetIdFeature, existingFeatures.getOrElse(SourceTweetIdFeature, None)) +
|
||||
(SourceUserIdFeature, existingFeatures.getOrElse(SourceUserIdFeature, None))
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,161 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.google.inject.name.Named
|
||||
import com.twitter.conversions.DurationOps.RichDuration
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.HasImageFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SemanticAnnotationFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieStaticEntitiesCache
|
||||
import com.twitter.home_mixer.util.tweetypie.RequestFields
|
||||
import com.twitter.home_mixer.util.tweetypie.content.TweetMediaFeaturesExtractor
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.cache.TtlCache
|
||||
import com.twitter.spam.rtf.{thriftscala => sp}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}
|
||||
import com.twitter.tweetypie.{thriftscala => tp}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class TweetypieStaticEntitiesFeatureHydrator @Inject() (
|
||||
tweetypieStitchClient: TweetypieStitchClient,
|
||||
@Named(TweetypieStaticEntitiesCache) cacheClient: TtlCache[Long, tp.Tweet])
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("TweetypieStaticEntities")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
DirectedAtUserIdFeature,
|
||||
HasImageFeature,
|
||||
HasVideoFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
InReplyToUserIdFeature,
|
||||
IsRetweetFeature,
|
||||
MentionScreenNameFeature,
|
||||
MentionUserIdFeature,
|
||||
QuotedTweetIdFeature,
|
||||
QuotedUserIdFeature,
|
||||
SemanticAnnotationFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature
|
||||
)
|
||||
|
||||
private val CacheTTL = 24.hours
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, None)
|
||||
.add(DirectedAtUserIdFeature, None)
|
||||
.add(HasImageFeature, false)
|
||||
.add(HasVideoFeature, false)
|
||||
.add(InReplyToTweetIdFeature, None)
|
||||
.add(InReplyToUserIdFeature, None)
|
||||
.add(IsRetweetFeature, false)
|
||||
.add(MentionScreenNameFeature, Seq.empty)
|
||||
.add(MentionUserIdFeature, Seq.empty)
|
||||
.add(QuotedTweetIdFeature, None)
|
||||
.add(QuotedUserIdFeature, None)
|
||||
.add(SemanticAnnotationFeature, Seq.empty)
|
||||
.add(SourceTweetIdFeature, None)
|
||||
.add(SourceUserIdFeature, None)
|
||||
.build()
|
||||
|
||||
/**
|
||||
* Steps:
|
||||
* 1. query cache with all candidates
|
||||
* 2. create a cached feature map
|
||||
* 3. iterate candidates to hydrate features
|
||||
* 3.a transform cached candidates
|
||||
* 3.b hydrate non-cached candidates from Tweetypie and write to cache
|
||||
*/
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val tweetIds = candidates.map(_.candidate.id)
|
||||
val cachedTweetsMapFu = cacheClient
|
||||
.get(tweetIds)
|
||||
.map(_.found)
|
||||
|
||||
Stitch.callFuture(cachedTweetsMapFu).flatMap { cachedTweets =>
|
||||
Stitch.collect {
|
||||
candidates.map { candidate =>
|
||||
if (cachedTweets.contains(candidate.candidate.id))
|
||||
Stitch.value(createFeatureMap(cachedTweets(candidate.candidate.id)))
|
||||
else readFromTweetypie(query, candidate)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def createFeatureMap(tweet: tp.Tweet): FeatureMap = {
|
||||
val coreData = tweet.coreData
|
||||
val quotedTweet = tweet.quotedTweet
|
||||
val mentions = tweet.mentions.getOrElse(Seq.empty)
|
||||
val share = coreData.flatMap(_.share)
|
||||
val reply = coreData.flatMap(_.reply)
|
||||
val semanticAnnotations =
|
||||
tweet.escherbirdEntityAnnotations.map(_.entityAnnotations).getOrElse(Seq.empty)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, coreData.map(_.userId))
|
||||
.add(DirectedAtUserIdFeature, coreData.flatMap(_.directedAtUser.map(_.userId)))
|
||||
.add(HasImageFeature, TweetMediaFeaturesExtractor.hasImage(tweet))
|
||||
.add(HasVideoFeature, TweetMediaFeaturesExtractor.hasVideo(tweet))
|
||||
.add(InReplyToTweetIdFeature, reply.flatMap(_.inReplyToStatusId))
|
||||
.add(InReplyToUserIdFeature, reply.map(_.inReplyToUserId))
|
||||
.add(IsRetweetFeature, share.isDefined)
|
||||
.add(MentionScreenNameFeature, mentions.map(_.screenName))
|
||||
.add(MentionUserIdFeature, mentions.flatMap(_.userId))
|
||||
.add(QuotedTweetIdFeature, quotedTweet.map(_.tweetId))
|
||||
.add(QuotedUserIdFeature, quotedTweet.map(_.userId))
|
||||
.add(SemanticAnnotationFeature, semanticAnnotations)
|
||||
.add(SourceTweetIdFeature, share.map(_.sourceStatusId))
|
||||
.add(SourceUserIdFeature, share.map(_.sourceUserId))
|
||||
.build()
|
||||
}
|
||||
|
||||
private def readFromTweetypie(
|
||||
query: PipelineQuery,
|
||||
candidate: CandidateWithFeatures[TweetCandidate]
|
||||
): Stitch[FeatureMap] = {
|
||||
tweetypieStitchClient
|
||||
.getTweetFields(
|
||||
tweetId = candidate.candidate.id,
|
||||
options = tp.GetTweetFieldsOptions(
|
||||
tweetIncludes = RequestFields.TweetStaticEntitiesFields,
|
||||
includeRetweetedTweet = false,
|
||||
includeQuotedTweet = false,
|
||||
forUserId = query.getOptionalUserId, // Needed to get protected Tweets for certain users
|
||||
visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,
|
||||
safetyLevel = Some(sp.SafetyLevel.FilterNone) // VF is handled in the For You product
|
||||
)
|
||||
).map {
|
||||
case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), _, _) =>
|
||||
cacheClient.set(candidate.candidate.id, found.tweet, CacheTTL)
|
||||
createFeatureMap(found.tweet)
|
||||
case _ =>
|
||||
DefaultFeatureMap + (AuthorIdFeature, candidate.features.getOrElse(AuthorIdFeature, None))
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,96 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinAuthorFollowEmbeddingsAdapter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinAuthorFollow20200101FeatureRepository
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.ml.api.{thriftscala => ml}
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.servo.repository.KeyValueResult
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.util.Future
|
||||
import com.twitter.util.Try
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
import scala.collection.JavaConverters._
|
||||
|
||||
object TwhinAuthorFollow20220101Feature
|
||||
extends DataRecordInAFeature[TweetCandidate]
|
||||
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class TwhinAuthorFollow20220101FeatureHydrator @Inject() (
|
||||
@Named(TwhinAuthorFollow20200101FeatureRepository)
|
||||
client: KeyValueRepository[Seq[Long], Long, ml.Embedding],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("TwhinAuthorFollow20220101")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TwhinAuthorFollow20220101Feature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
Stitch.callFuture {
|
||||
val possiblyAuthorIds = extractKeys(candidates)
|
||||
val authorIds = possiblyAuthorIds.flatten
|
||||
|
||||
val response: Future[KeyValueResult[Long, ml.Embedding]] =
|
||||
if (authorIds.isEmpty) {
|
||||
Future.value(KeyValueResult.empty)
|
||||
} else {
|
||||
client(authorIds)
|
||||
}
|
||||
|
||||
response.map { result =>
|
||||
possiblyAuthorIds.map { possiblyAuthorId =>
|
||||
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
|
||||
val transformedValue = postTransformer(value)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(TwhinAuthorFollow20220101Feature, transformedValue)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def postTransformer(embedding: Try[Option[ml.Embedding]]): Try[DataRecord] = {
|
||||
embedding.map { e =>
|
||||
TwhinAuthorFollowEmbeddingsAdapter.adaptToDataRecords(e).asScala.head
|
||||
}
|
||||
}
|
||||
|
||||
private def extractKeys(
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Seq[Option[Long]] = {
|
||||
candidates.map { candidate =>
|
||||
candidate.features
|
||||
.getTry(AuthorIdFeature)
|
||||
.toOption
|
||||
.flatten
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,80 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserEngagementEmbeddingsAdapter
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserEngagementFeatureRepository
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.ml.api.RichDataRecord
|
||||
import com.twitter.ml.api.util.ScalaToJavaDataRecordConversions
|
||||
import com.twitter.ml.api.{thriftscala => ml}
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.util.Return
|
||||
import com.twitter.util.Throw
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
object TwhinUserEngagementFeature
|
||||
extends DataRecordInAFeature[PipelineQuery]
|
||||
with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class TwhinUserEngagementQueryFeatureHydrator @Inject() (
|
||||
@Named(TwhinUserEngagementFeatureRepository)
|
||||
client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],
|
||||
statsReceiver: StatsReceiver)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("TwhinUserEngagement")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TwhinUserEngagementFeature)
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
|
||||
private val keyFoundCounter = scopedStatsReceiver.counter("key/found")
|
||||
private val keyLossCounter = scopedStatsReceiver.counter("key/loss")
|
||||
private val keyFailureCounter = scopedStatsReceiver.counter("key/failure")
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
Stitch.callFuture {
|
||||
client(Seq(userId)).map { results =>
|
||||
val embedding: Option[ml.FloatTensor] = results(userId) match {
|
||||
case Return(value) =>
|
||||
if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()
|
||||
else keyLossCounter.incr()
|
||||
value
|
||||
case Throw(_) =>
|
||||
keyFailureCounter.incr()
|
||||
None
|
||||
case _ =>
|
||||
None
|
||||
}
|
||||
val dataRecord =
|
||||
new RichDataRecord(new DataRecord, TwhinUserEngagementEmbeddingsAdapter.getFeatureContext)
|
||||
embedding.foreach { floatTensor =>
|
||||
dataRecord.setFeatureValue(
|
||||
TwhinUserEngagementEmbeddingsAdapter.twhinEmbeddingsFeature,
|
||||
ScalaToJavaDataRecordConversions.scalaTensor2Java(
|
||||
ml.GeneralTensor.FloatTensor(floatTensor))
|
||||
)
|
||||
}
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(TwhinUserEngagementFeature, dataRecord.getRecord)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,80 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserFollowEmbeddingsAdapter
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserFollowFeatureRepository
|
||||
import com.twitter.ml.api.DataRecord
|
||||
import com.twitter.ml.api.RichDataRecord
|
||||
|
||||
import com.twitter.ml.api.util.ScalaToJavaDataRecordConversions
|
||||
import com.twitter.ml.api.{thriftscala => ml}
|
||||
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.util.Return
|
||||
import com.twitter.util.Throw
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
object TwhinUserFollowFeature
|
||||
extends DataRecordInAFeature[PipelineQuery]
|
||||
with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {
|
||||
override def defaultValue: DataRecord = new DataRecord()
|
||||
}
|
||||
|
||||
@Singleton
|
||||
class TwhinUserFollowQueryFeatureHydrator @Inject() (
|
||||
@Named(TwhinUserFollowFeatureRepository)
|
||||
client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],
|
||||
statsReceiver: StatsReceiver)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("TwhinUserFollow")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TwhinUserFollowFeature)
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
|
||||
private val keyFoundCounter = scopedStatsReceiver.counter("key/found")
|
||||
private val keyLossCounter = scopedStatsReceiver.counter("key/loss")
|
||||
private val keyFailureCounter = scopedStatsReceiver.counter("key/failure")
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
Stitch.callFuture(
|
||||
client(Seq(userId)).map { results =>
|
||||
val embedding: Option[ml.FloatTensor] = results(userId) match {
|
||||
case Return(value) =>
|
||||
if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()
|
||||
else keyLossCounter.incr()
|
||||
value
|
||||
case Throw(_) =>
|
||||
keyFailureCounter.incr()
|
||||
None
|
||||
case _ =>
|
||||
None
|
||||
}
|
||||
val dataRecord =
|
||||
new RichDataRecord(new DataRecord, TwhinUserFollowEmbeddingsAdapter.getFeatureContext)
|
||||
embedding.foreach { floatTensor =>
|
||||
dataRecord.setFeatureValue(
|
||||
TwhinUserFollowEmbeddingsAdapter.twhinEmbeddingsFeature,
|
||||
ScalaToJavaDataRecordConversions.scalaTensor2Java(
|
||||
ml.GeneralTensor
|
||||
.FloatTensor(floatTensor)))
|
||||
}
|
||||
FeatureMapBuilder()
|
||||
.add(TwhinUserFollowFeature, dataRecord.getRecord)
|
||||
.build()
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
@ -0,0 +1,84 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.UserFollowedTopicIdsRepository
|
||||
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.servo.keyvalue.KeyValueResult
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.util.Future
|
||||
import com.twitter.util.Try
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
object UserFollowedTopicIdsFeature extends Feature[TweetCandidate, Seq[Long]]
|
||||
|
||||
@Singleton
|
||||
class UserFollowedTopicIdsFeatureHydrator @Inject() (
|
||||
@Named(UserFollowedTopicIdsRepository)
|
||||
client: KeyValueRepository[Seq[Long], Long, Seq[Long]],
|
||||
override val statsReceiver: StatsReceiver)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with ObservedKeyValueResultHandler {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("UserFollowedTopicIds")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(UserFollowedTopicIdsFeature)
|
||||
|
||||
override val statScope: String = identifier.toString
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
Stitch.callFuture {
|
||||
val possiblyAuthorIds = extractKeys(candidates)
|
||||
val authorIds = possiblyAuthorIds.flatten
|
||||
|
||||
val response: Future[KeyValueResult[Long, Seq[Long]]] =
|
||||
if (authorIds.isEmpty) {
|
||||
Future.value(KeyValueResult.empty)
|
||||
} else {
|
||||
client(authorIds)
|
||||
}
|
||||
|
||||
response.map { result =>
|
||||
possiblyAuthorIds.map { possiblyAuthorId =>
|
||||
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
|
||||
val transformedValue = postTransformer(value)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(UserFollowedTopicIdsFeature, transformedValue)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def postTransformer(input: Try[Option[Seq[Long]]]): Try[Seq[Long]] = {
|
||||
input.map(_.getOrElse(Seq.empty[Long]))
|
||||
}
|
||||
|
||||
private def extractKeys(
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Seq[Option[Long]] = {
|
||||
candidates.map { candidate =>
|
||||
candidate.features
|
||||
.getTry(AuthorIdFeature)
|
||||
.toOption
|
||||
.flatten
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,35 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.UserLanguagesStore
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.search.common.constants.{thriftscala => scc}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.storehaus.ReadableStore
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
object UserLanguagesFeature extends Feature[PipelineQuery, Seq[scc.ThriftLanguage]]
|
||||
|
||||
@Singleton
|
||||
case class UserLanguagesFeatureHydrator @Inject() (
|
||||
@Named(UserLanguagesStore) store: ReadableStore[Long, Seq[scc.ThriftLanguage]])
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("UserLanguages")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(UserLanguagesFeature)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
Stitch.callFuture(store.get(query.getRequiredUserId)).map { languages =>
|
||||
FeatureMapBuilder()
|
||||
.add(UserLanguagesFeature, languages.getOrElse(Seq.empty))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,54 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserStateFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.user_health.{thriftscala => uh}
|
||||
import com.twitter.timelines.user_health.v1.{thriftscala => uhv1}
|
||||
import com.twitter.user_session_store.ReadOnlyUserSessionStore
|
||||
import com.twitter.user_session_store.ReadRequest
|
||||
import com.twitter.user_session_store.UserSessionDataset
|
||||
import com.twitter.user_session_store.UserSessionDataset.UserSessionDataset
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class UserStateQueryFeatureHydrator @Inject() (
|
||||
userSessionStore: ReadOnlyUserSessionStore)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("UserState")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(UserStateFeature)
|
||||
|
||||
private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.UserHealth)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
userSessionStore
|
||||
.read(ReadRequest(query.getRequiredUserId, datasets))
|
||||
.map { userSession =>
|
||||
val userState = userSession.flatMap {
|
||||
_.userHealth match {
|
||||
case Some(uh.UserHealth.V1(uhv1.UserHealth(userState))) => userState
|
||||
case _ => None
|
||||
}
|
||||
}
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(UserStateFeature, userState)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)
|
||||
)
|
||||
}
|
@ -0,0 +1,88 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RepliedByEngagerIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RetweetedByEngagerIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.UtegSocialProofRepository
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.recos.recos_common.{thriftscala => rc}
|
||||
import com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}
|
||||
import com.twitter.servo.keyvalue.KeyValueResult
|
||||
import com.twitter.servo.repository.KeyValueRepository
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class UtegFeatureHydrator @Inject() (
|
||||
@Named(UtegSocialProofRepository) client: KeyValueRepository[
|
||||
(Seq[Long], (Long, Map[Long, Double])),
|
||||
Long,
|
||||
uteg.TweetRecommendation
|
||||
]) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Uteg")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
FavoritedByUserIdsFeature,
|
||||
RetweetedByEngagerIdsFeature,
|
||||
RepliedByEngagerIdsFeature
|
||||
)
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean = query.features
|
||||
.exists(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double]).nonEmpty)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val seedUserWeights = query.features.map(_.get(RealGraphInNetworkScoresFeature)).get
|
||||
|
||||
val sourceTweetIds = candidates.flatMap(_.features.getOrElse(SourceTweetIdFeature, None))
|
||||
val inReplyToTweetIds = candidates.flatMap(_.features.getOrElse(InReplyToTweetIdFeature, None))
|
||||
val tweetIds = candidates.map(_.candidate.id)
|
||||
val tweetIdsToSend = (tweetIds ++ sourceTweetIds ++ inReplyToTweetIds).distinct
|
||||
|
||||
val utegQuery = (tweetIdsToSend, (query.getRequiredUserId, seedUserWeights))
|
||||
|
||||
Stitch
|
||||
.callFuture(client(utegQuery))
|
||||
.map(handleResponse(candidates, _))
|
||||
}
|
||||
|
||||
private def handleResponse(
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]],
|
||||
results: KeyValueResult[Long, uteg.TweetRecommendation],
|
||||
): Seq[FeatureMap] = {
|
||||
candidates.map { candidate =>
|
||||
val candidateProof = results(candidate.candidate.id).toOption.flatten
|
||||
val sourceProof = candidate.features
|
||||
.getOrElse(SourceTweetIdFeature, None).flatMap(results(_).toOption.flatten)
|
||||
val proofs = Seq(candidateProof, sourceProof).flatten.map(_.socialProofByType)
|
||||
|
||||
val favoritedBy = proofs.flatMap(_.get(rc.SocialProofType.Favorite)).flatten
|
||||
val retweetedBy = proofs.flatMap(_.get(rc.SocialProofType.Retweet)).flatten
|
||||
val repliedBy = proofs.flatMap(_.get(rc.SocialProofType.Reply)).flatten
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(FavoritedByUserIdsFeature, favoritedBy)
|
||||
.add(RetweetedByEngagerIdsFeature, retweetedBy)
|
||||
.add(RepliedByEngagerIdsFeature, repliedBy)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
@ -0,0 +1,59 @@
|
||||
package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features
|
||||
|
||||
import com.twitter.ml.api.DataRecordMerger
|
||||
import com.twitter.ml.api.Feature
|
||||
import com.twitter.ml.api.FeatureContext
|
||||
import com.twitter.ml.api.RichDataRecord
|
||||
import com.twitter.ml.api.util.CompactDataRecordConverter
|
||||
import com.twitter.ml.api.util.FDsl._
|
||||
import com.twitter.timelines.author_features.v1.{thriftjava => af}
|
||||
import com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase
|
||||
import com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig
|
||||
import com.twitter.timelines.prediction.features.user_health.UserHealthFeatures
|
||||
|
||||
object AuthorFeaturesAdapter extends TimelinesMutatingAdapterBase[Option[af.AuthorFeatures]] {
|
||||
|
||||
private val originalAuthorAggregatesFeatures =
|
||||
TimelinesAggregationConfig.originalAuthorReciprocalEngagementAggregates
|
||||
.buildTypedAggregateGroups().flatMap(_.allOutputFeatures)
|
||||
private val authorFeatures = originalAuthorAggregatesFeatures ++
|
||||
Seq(
|
||||
UserHealthFeatures.AuthorState,
|
||||
UserHealthFeatures.NumAuthorFollowers,
|
||||
UserHealthFeatures.NumAuthorConnectDays,
|
||||
UserHealthFeatures.NumAuthorConnect)
|
||||
private val featureContext = new FeatureContext(authorFeatures: _*)
|
||||
|
||||
override def getFeatureContext: FeatureContext = featureContext
|
||||
|
||||
override val commonFeatures: Set[Feature[_]] = Set.empty
|
||||
|
||||
private val compactDataRecordConverter = new CompactDataRecordConverter()
|
||||
private val drMerger = new DataRecordMerger()
|
||||
|
||||
override def setFeatures(
|
||||
authorFeaturesOpt: Option[af.AuthorFeatures],
|
||||
richDataRecord: RichDataRecord
|
||||
): Unit = {
|
||||
authorFeaturesOpt.foreach { authorFeatures =>
|
||||
val dataRecord = richDataRecord.getRecord
|
||||
|
||||
dataRecord.setFeatureValue(
|
||||
UserHealthFeatures.AuthorState,
|
||||
authorFeatures.user_health.user_state.getValue.toLong)
|
||||
dataRecord.setFeatureValue(
|
||||
UserHealthFeatures.NumAuthorFollowers,
|
||||
authorFeatures.user_health.num_followers.toDouble)
|
||||
dataRecord.setFeatureValue(
|
||||
UserHealthFeatures.NumAuthorConnectDays,
|
||||
authorFeatures.user_health.num_connect_days.toDouble)
|
||||
dataRecord.setFeatureValue(
|
||||
UserHealthFeatures.NumAuthorConnect,
|
||||
authorFeatures.user_health.num_connect.toDouble)
|
||||
|
||||
val originalAuthorAggregatesDataRecord =
|
||||
compactDataRecordConverter.compactDataRecordToDataRecord(authorFeatures.aggregates)
|
||||
drMerger.merge(dataRecord, originalAuthorAggregatesDataRecord)
|
||||
}
|
||||
}
|
||||
}
|
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user