1 of 24

Slide Notes

DownloadGo Live

Influence

Published on Nov 18, 2015

No Description

View Outline

MORE DECKS TO EXPLORE

PRESENTATION OUTLINE

1.

INFLUENCE

TRACKING THE ZEITGEIST
Photo by ep_jhu
2.

OLD IMPLEMENTATION

Photo by AMANITO
3.

PIPELINE

  • Poll for tweets
  • Extract URLs + scrape pages
  • Extract main content + topics
  • Calculate TF-IDF scores
  • Save 99th percentile
Photo by wili_hybrid
4.

IMPLEMENTATION

  • Used green threads to parallelize
  • Used celery to scale
  • Could process ~7-12 Tweets / s
Photo by Entropyer
5.

WHY NOT PRODUCTIONIZE?

Photo by TunnelBug
6.

INSUFFICIENCIES

  • Too slow for complex analysis
  • Twitter specific
  • Resource intensive
  • Requires custom IO primitives
Photo by VinothChandar
7.

INCOMPATIBILITIES

  • Poor Celery support for gevent
  • No muitiprocessing within worker
  • No processed once guarantee
Photo by timtom.ch
8.

SCALA TO THE RESCUE

Photo by Mary-and-Tobit
9.

NATIVE CONCURRENCY

  • Futures
  • Lightweight threading
  • Lazy async evaluation
Photo by yewenyi
10.

AKKA ACTORS

  • Message passing
  • Green threading
  • 50M messages / s
  • 2.5M actors / GB of heap
Photo by geezaweezer
11.

FASTER PROTOTYPING

  • Types + powerful DSL
  • More concise code
  • Most time spent learning
Photo by Nick Chill Photography
12.

NOT ALL GOOD

  • Version fragmentation
  • Only recently popular
  • Centrally controlled
Photo by ndmpsy
13.

WHERE WE'RE AT

Photo by Telstar Logistics
14.

Untitled Slide

  • Re-implemented RecSys Core
  • Connected to Twitter Streaming API
  • Live update of TF-IDF rankings
Photo by mrtopp
15.

ARCHITECTURE

Photo by marcp_dmoz
16.

KEY PARTS

  • API written with Play
  • Akka worker pool
  • Spark cluster for TF-IDF
Photo by .-. Handle with care .-.
17.

BENEFITS

  • Horizontally scalable on all levels
  • No duplicated data
  • Highly re-usable
Photo by kevin dooley
18.

IMPROVEMENTS

  • Kafka queue
  • Stronger decoupling
Photo by Daniele Zanni
19.

WHAT'S LEFT?

Photo by Leonard John Matthews
20.

FOR PRODUCTION

  • Calculating metrics per campaign
  • Writing microservice API
  • IDF calculated from global baselines
  • Configuration management
Photo by · · · — — — · · ·
21.

FOR QUALITY

  • Windowed ranking calculations
  • Store raw tweets for reprocessing
  • Handling >5000 Twitter handles
  • Better keyword recommendations
  • Allow users to get trends from searches
Photo by nataliej
22.

THE FUTURE!

Photo by danielfoster437
23.

Untitled Slide

  • Tracking brand audience
  • Track global trend activity
Photo by Brad Higham
24.

DEMO

class SimpleCoordinator extends SubscribableActor {
val feeder = context.system.actorOf(TwitterFeederActor.props())
feeder ! AddUser("Newscred_Devs_P")
feeder ! SubscribeReceiver(self)
val scraper = context.system.actorOf(Props[ScraperActor])

receiveBuilder += {
case SendStatus(status: Status) =>
scraper ! ExtractTopics(status.getURLEntities()(0).getURL(), scraper)
}
}
Photo by jurvetson

Andrew FigPope