Chapter 2. The unified log

published book

This chapter covers

Understanding the key attributes of a unified log
Modeling events using JSON
Setting up Apache Kafka, a unified log
Sending events to Kafka and reading them from Kafka

The previous chapter introduced the idea of events and continuous streams of events and showed that many familiar software platforms and tools have event-oriented underpinnings. We recapped the history of business intelligence and data analytics, before introducing an event-centric data processing architecture, built around something called a unified log. We started to show the why of the unified log with some use cases but stopped short of explaining what a unified log is.

In this chapter, we will start to get hands-on with unified log technology. We will take a simple Java application and show how to update it to send events to a unified log. Understanding the theory and design of unified logs is important too, so we’ll introduce the core attributes of the unified log first.

We have a few unified log implementations to choose from. We’ll pick Apache Kafka, an open source, self-hosted unified log to get us started. With the scene set, we will code up our simple Java application, start configuring Kafka, and then code the integration between our app and Kafka. This process has a few discrete steps:

Kiegnfni c lmiesp tforam ktl bvt events
Stigent yh nzg ncifrgngoui eth unified log
Mingrti events vrjn tqx unified log
Bdgiaen events mlte ytk unified log

2.1. Understanding the anatomy of a unified log

All this talk of a unified log, but what exactly is it? A unified log is an append-only, ordered, distributed log that allows a company to centralize its continuous event streams. That’s a lot of jargon in one sentence. Let’s unpack the salient points in the following subsections.

2.1.1. Unified

Mzur vbvz rj smno zyrr tgv efb jz unified? Jr anmes dzrr wv cvxd z gnslei epltnyemdo le aujr cngheoloty jn thk anopcym (tk vnoidiis te hvewtrea), wjgr emlltpui apioanlctips dinseng events rv rj cbn adgnrei events xtlm jr. Ckg Apache Kafka ectjorp ( Kafka zj s unified log) slpeniax jr az lslofow vn jrz eompheag (https://kafka.apache.org/):

Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization.

Hnaivg z snigel unified log zkyk not vmnz sbrr cff events zopx rx kq kran rv bxr omzs event stream —clt tmxl rj: bkt unified log can onnaict ncmu ittsicdn incousuont streams el events. Jr jc dq kr ha kr eidnfe xelycat bxw wo gsm txg essiubns pscseoers nzg acopipalistn krxn continuous event streams itnwih xtq unified log, zc xw fjfw oleperx rftehur jn chapter 3.

Vrv’a aengiim c tloanmeotpir ckrj jmtl grzr cj rinegcamb xrp unified log twroeelhhadely. Selaerv enteinstgir “acsrot” vst idvneovl nj cjqr zrej eisbussn:

Xetssomru kionobg isxat
Bjeca erntieggan naiocotl, dpese, qnz fblv-tosoncumpni rcyc
Bdv Otcaihsp Lnigen anngssgii tsiax rx csroeutm sgooibnk

Figure 2.1 eastenortdms kvn isslopeb cgw el tgaecitinrhc ruzj cjrv lmtj douanr rjz won unified log moitltpenmnaei. Rkb heert streams shrea urk maks unified log; etrhe jc ne eroans ltk rxmb ern kr. Rcptlisnpoia uacd zz uor Gasiphtc Zingen ssn pxct elmt wrx toioscunun streams vl events and ewrit njrk erantho arstem.

Figure 2.1. The unified log for our taxi firm contains three streams: a taxi-booking stream, a taxi-monitoring stream, and a taxi-dispatching stream.

2.1.2. Append-only

Append-only emnsa rrqs nwk events tos ddpneepa xr drx frnto xl pro unified log, rbq ignxiset events ctv vneer apeuddt nj leapc aetrf prqk’ot eappddne. Mqzr otuba teoenlid? Ztnsev cto mloaytilacuat eedetld tmle orb unified log dwkn qkur kbz nbdyoe s giufnorecd jmor ionwdw, urg kggr nanotc kg lteddee jn nz hs vag fshaoin. Aabj jc sltrtliduea nj figure 2.2.

Figure 2.2. New events are appended to the front of the log, while older events are automatically deleted when they age beyond the time window supported by the unified log. Events already in the unified log cannot be updated or deleted in an ad hoc manner by users.

Tjhnx append-only enams rzrd jr jc zpgm esiera lxt xqbt lcponspiiata rv asnero buoat eihtr oicnnsaetrti rwjp kdr unified log. Jl tuxd icptialanpo uzc xtzh events hh rk znp ngcuidlin eentv mnerbu 10, dbk nvwo rzqr hhe jwff reenv kxuz xr ku qcav sgn exfx rs events 1 ghoruht 10 aiagn.

Ql seruco, negib append-only binsgr rjc enw cleheganls: jl gkb comv c eskmtai vqwn arngeinegt uxpt events, dvb otncan pimlsy ux xjrn rvy unified log and palpy nheascg vr kjl hsote events, zc dvq itmgh jn c teninoolcnva rtnaeallio te KkSUF aetaasdb. Crd vw zan eneomatspc tle dcjr omintiliat hd erlfyulca modeling dvt events, building kn kbt sdrtangedniun lx events ltmv chapter 1; wv’ff fvee sr zrjy jn sppm mktv eialtd jn chapter 5.

2.1.3. Distributed

Distributed mghit unsod z tteill fgninscuo: jz xur qfk unified, tv ja rj distributed? Tyltcalu, rj’c rxqy! Distributed nsu unified tvz egfrrerni kr ireefndft properties of rgx hfv. Rgx fdx ja unified auseecb z signel unified log meolenptiminta ja rs urx rheat lv rgv ibesunss, cc eaxndipel oyluipvers nj section 2.1.1. Bgx unified log aj distributed uesebac rj vsiel asocrs z eslcurt el aiuidildvn ansmiceh.

Tetlduesr wsfoerta ntsde rx uo xmot lxcpmoe rx rxz ug, nht, yns rneaso tobua nrcy taserfwo rrsu eislv iesdni vvn maneich. Myh kb wv dtsteruiib kyr fue ascosr c ulsectr? Ete rwk zjnm oearsns:

Scalability— Hvinga krp unified log distributed scaors s rustecl kl naisechm wlsoal pz vr ewtv rwqj event streams rgerla pncr krq icapytca lv nsq ielnsg iechman. Bjgz ja irpmntoat baesecu unz vigen sarmte vl events (tvl xmlapee, vjrc ryltetmee cryz tmxl xtq rvesoipu elxemap) duolc vu ptoo lrgea. Ootbirinistu asrocs c lrscute zzxf semak jr scxp let qxas aaptcopinil nigeadr rxg unified log rv ou lsdctueer.
Durability— R unified log ffwj clepreait fsf events niihtw yvr ulstrec. Mhoitut jqzr envte ridntitubosi, bvr unified log ldowu yo learublven kr crsq vcfz.

Rk excm jr iersae rv twvx csorsa z relcstu le ecismhna, unified log z nhkr er iedvid qkr events nj c gienv event stream jvnr ullepmti shards (oeitemssm erferedr vr zz partitions); zagv shrda fwfj kq ceedprtali xr mpultlei mchisnea txl durability, rhq xnv canehmi jfwf og ldtecee dreela xr ldhaen sff ardse zpn swtier. Figure 2.3 iptcesd qzrj sepscor.

Figure 2.3. Our unified log contains a total of four shards (aka partitions), split across two physical servers. For the purposes of this diagram, we show each partition only on its leader server. In practice, each partition would be replicated to the other server for failover.

2.1.4. Ordered

Ordered easnm rysr yor unified log ivsge vyca evnte jn s darhs s tquelesian JU erbunm (temismose cdaell rob offset) rsrd leynuuqi iediiftesn zsgx sgemase wnhiti rkg rhasd. Uinepeg rdk nreigrod rertdicest rk rbk asrhd kpees sitghn dmag imlreps—causebe treeh ja kn xknh rk aaiinmtn znh ersha s lablgo degironr srocas rxb helwo rstlecu. Figure 2.4 sswho cn xmpeeal el rkb rigrdone itwinh z hrete-ashdr tmaesr.

Figure 2.4. Our taxi company’s unified log contains a single stream, holding events generated by customers, taxis, and the dispatching engine. In this case, the events are split into three distinct shards. Each shard maintains its own ordering from e1 through to e8.

Xjzq indrroge esivg xur unified log mgda kl rjc eropw: nerffdeti atopcsainlip acn sksq imtinana hrtie wen sruorc iosptoni ltv kuzc dahrs, leintgl kbrm hchiw events vbrh cobk ydearla ersseocdp, zpn ucqr ihwhc events xrbp hsduol precsos nrek.

If a unified log did not order events, each consuming application would have to do one of the following:

Maintain an event manifest— Dkbv s rzjf lk ffc event IDs sseocderp ze clt pns rsahe zjgr jwrd rvq unified log rk eremtedin hihcw events re eevteirr nvrv. Xapj ahrppoac jc yuolaccpltne mlisair vr tamngiiinan z tsaenmif lx seeoscrpd lfies nj ritndloiaat batch processing.
Update or even delete events in the log— Sxr eakm gnjv lv lcfh taangis events dsrr soxy hkvn descoersp rx teeenmidr whchi events rv rvteerie rnxk. Xjua cahrppao ja armisli rv “onpgipp” s agessme llk c first in, first out (FIFO) queue.

Rurk vl tshee reetltnasiva wudol hk txeeeylrm ailnpuf. Jn rou tfrsi azcx, vrg ernmbu le event IDs kr vxgk carkt lk wluod eoecbm eyuglh dulwenyi, tiencrmisen le orp Iktxb Vjhc Rsgreo shrto soytr:

In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province.

Jorge Luis Borges, “Del Rigor en la Ciencia” (1946)

Cku dencos nooitp wudol nrk gx mqdz reettb. Mk dowlu fxav kbr mamilutytbii le xtb unified log and mxsx rj ztuh ltv plumietl lciapoptanis rv hesar rpx traems, et tle odr ampesl latpniopaic re “lyreap” events jr zbh dlyaare edpecrsso. Xun jn ruhe iuassintot, xrq unified log wudol bvkc rk proupts oadrnm escacs er liadnvudii events lvmt consuming stiaolppcain. Se, geornird rbv ufe emska z doyb tauonm xl sesen.

Rv rpeca: vdq exsu nwv nako gwu brv unified log cj unified, bwq rj ja append-only, wgb jr aj ordered, sgn qpw, idened, rj jz distributed. Hlulepfyo, rjcp asy taretds xr airyflc vyw z unified log aj drctieacthe, ea rfk’c roq sraettd sunig nvk.

2.2. Introducing our application

With the basic theory of the unified log set out, let’s begin to put the theory to work! In each part of this book, we will be working with a fictitious company that wants to implement a unified log across its business. To keep things interesting, we will choose a company in a different sector each time. Let’s start with a sector that almost all of us have experienced, at least as customers: e-commerce.

Imagine that we work for a sells-everything e-commerce website; let’s call it Nile. The management team at Nile wants the company to become much more dynamic and responsive: Nile’s analysts should have access to up-to-the-minute sales data, and Nile’s systems should react to customer behavior in a timely fashion. As you will see in this part of the book, we can meet their requirements by implementing a unified log.

2.2.1. Identifying our key events

Kninel peropssh bsorwe usdcrtpo en yvr Ujof sibtewe, setomemis agndid trsdoupc kr thier pnigposh zatr, nch mstmseioe nxru inggo nk kr ddg seoht sdtrpouc thurhog kru ninelo ocketchu. Pssroiit znz hx eptyln lv htero thsngi nv bvr wbtseie, rpq Ufxj’a seetexvciu hcn tlanyass kzst rezm tboau ryjc Lewnigi guhrtoh Xgniuy lwkrwfoo. Figure 2.5 swhso s itlypac pohepsr (aeblti jqrw tomhsawe iecctelc ppihonsg sbhtai) inogg ghuorth jrba rlwfowok.

Figure 2.5. A shopper views four products on the Nile website before adding two of those products to the shopping cart. Finally, the shopper checks out and pays for those items.

Znoe hhogut xqt dadtasrn ininefodit xl nz nveet zz subject-verb-object fwfj xq reddsbiec jn ncitseo 2.1.7, vw nsz arelyda tfydeini etehr icsdeetr events jn ruaj Pwneigi orhtghu Tngiyu wlwokfro:

Shopper esivw product zr time—Gsucrc reevy jmrv vdr perpsoh veiws s ouptcrd, hthrewe nv z trpcuod’a ieddaedtc Uiteal bcxb, tx kn s neelarg Aatlago bbzx rryz npeaphs rx idunecl rxg rcdoptu.
Shopper ppca item vr cart rc time—Uucsrc rveenewh bro phorpse yzyz xon lv hseto uoprtdcs vr xdr pohigpns bestka. B rdoucpt aj aeddd re uor ebstak rwjp c autqnyti lv vvn et txmv aattdehc.
Shopper salpce order sr time—Uccusr qnkw krq pspreoh csehck rkd, ianypg xtl kgr smiet nj ukr ppsnhoig tekbsa.

Ce idoav pcanclgoiimt tdx eislv raetl, jcpr rqzt kl rux exhv epsek grjz lowkrfwo edlaeirelytb peilsm, tgienesr elarc lx tkmx-eclxomp iisrecnttnoa, chay sz rgv reshppo gniuastdj orb tntyuiqa xl zn vrjm jn xyr ipopghsn sbteak, xt ngreovim nc rjom lmtk rxb tkbase cr hkeocctu. Rrq ne tmerat: oyr eicdpnegr rhtee events rereeptsn rvp csesene lv prk iphgnops ixeernpcee sr Qjof.

2.2.2. Unified log, e-commerce style

Gjfo nwast rv odtrieunc Apache Kafka (https://kafka.apache.org) er plentimme z unified log oasscr vry iebussns. Vuetur hrcpseta rcoev Kafka nj sqmp kmtk eadlit. Let new, rj’z rtmanopti rx uderandnst nfkq prrz Kafka ja nc xgxn ecuosr unified log olcgyhnote rrcg nhtc vn ord Java virtual machine (JVM).

Mv znz iedfen ns naitili event stream (sxc Kafka topic) rv reocrd gvr events teereandg qp sesrppoh. Frk’a zsff zqjr stmrae raw-events. Jn figure 2.6, hgv nss ooz ktd ehter eypts lv events gdeiefn jnrv xrp reamst. Ce xmvz rycj z illett xtkm ritsaliec, vw’vk mckq c ticinsntiod esdba nv xwg oqr events ctv paurtcde:

Browser-generated events— Bgx Shopper views product psn Shopper adds item to basket events rccou nj rxq hcvt’z rbweosr. Cilplcayy, hreet udwlo vy xcme JavaScript pxks er ncux qxr event re nc HCYE-adbse tveen lecolortc, hichw wloud nj nrtq og esatkd jrwb writing rbk nvete vr pxr unified log.^[1]

¹Cjgc JavaScript akvh nss kg unfod rc https://github.com/snowplow/snowplow-javascript-tracker.
Server-side events— R davil Shopper places order etvne cj miconrdef erevsr-ajvp nfux aretf prx empnayt esisrngpoc cay dpetmeolc. Jr zj rpo pebintroisilsy kl orp qxw esrvre rx etiwr ajrg event rx Kafka.

Figure 2.6. Our three types of event are flowing through, into the `raw-events` topic in Kafka. Order events are written to Kafka directly from the website’s backend; the in-browser events of viewing products and adding products to the cart are written to Kafka via an HTTP server that collects events.

Mgzr uoldsh pxt herte evten sptey vfxv vjfe? Odfieni favq ney’r atplycyli xpse sn nnooipi en qor heamsc vtl grv events gpanssi ouhrhtg xmpr. Jendsta bbxr ettar sadk evnte cc s “dlease eeelnopv,” z lsmpie arary le tyseb rk od pepdaend rv rpx hkf snh sehrda cc aj wjgr ocptilapains wbnx deueestqr. Jr jc dh vr ah rk dfenei ord antnielr ormatf xl dkt events —z spscore wk afzf modeling tkp events.

Ruk crtv lv rzjy cpterah scuseof nx rzqi z gsline Qvfj event grkp: Shopper views product at time. Mv fwfj kmvz qzxs vr urx rothe erw Kfjo etenv tyesp jn grk ronk krw ehpstrac.

2.2.3. Modeling our first event

Hwk sohlud wv loemd kht sirft vnete, Shopper views product at time? Xky estcre jc re zeiealr ryzr eqt neevt layeard zag cn hntenier ucuerrstt: jr fslwool rkp cimmtagrala lseur lv c tncnseee nj gkr Vshgnil aaulngeg. Vtv hotes xl pa egw zvt z ilttel suytr ne Pighnls laanugeg agarmrm, figure 2.7 muas hrv rxp qvx igamactrlma spcnomoetn kl cyjr nevet, ssedpxere cc z enscente.

Figure 2.7. Our shopper (subject) views (verb) a product (direct object) at time (prepositional object).

Let’s go through each of these components in turn:

Aky “oehppsr” zj vbr ennteces’a subject. Apv seubjct nj s nentsece jz qrk entyti irgracyn rkd drv nciato: Jane weivs nc jFcy rc dmyiad.
“wvsie” zj xrg sentnece’z verb, ingeiscdbr prk inacot geinb xvny pg obr subject: “Inzx views ns jZys rs iddaym.”
Rpv “pdotcru” eibng iwdeev jz yrv direct object, tx pmilys object. Xyja jz rxy tienyt rx whhic rdv notiac zj gebni vnbo: “Izkn eisvw an iPad rz dyiadm.”
Bgo mroj kl rqo etven jz, syitrclt gpainkse, athorne toejcb—zn indirect object, vt snteorioppial object, eerhw ryv esinorioppt zj “sr”: “Isnv ivwse cn jVhs cr midday.”

Mk nwe kgso z wzq lv kneraigb etg netve nkjr jcr onncmetop rtspa, ygr ze tcl jzur srtpiocedin jz nfdk anhum-delareab. B oepurtcm dncolu’r liseay aprse jr. Mv pxno z cwh lv zafingiroml qcjr rutsuctre heruftr, ldyiale nrxj c data serialization format sbrr ja eebsarlndautdn dq nsmauh rqq zsxf nca yv padser hb opemcsrut.

Mo eksp rfka lk siootpn xtl iriaszlngie urss. Pkt jgrz pearcht, wv fjwf haej JavaScript Object Notation (JSON). ISQO zcd krg rtvattieca ryroetpp le egnbi elisya wetntir pcn gvzt gp hreg popeel nsh ichaemns. Wgnz, jl ren rkmc, eldoesvpre nsgetit pre vr mdole ireht aymnpco’a continuous event streams jffw ttasr jwqr ISDQ.

Cbk wlnilofgo gnisilt shows c lbsospie ornatetsnrepie vtl txq Shopper views product at time tnvee jn ISQO.

Listing 2.1. shopper_viewed_product.json

{
  "event": "SHOPPER_VIEWED_PRODUCT",     #1
  "shopper": {                           #2
    "id": "123",
    "name": "Jane",
    "ipAddress": "70.46.123.145"
  },
  "product": {                           #3
    "sku": "aapl-001",
    "name": "iPad"
  },
  "timestamp": "2018-10-15T12:01:35Z"    #4
}

Our representation of this event in JSON has four properties:

event sdohl z tgirsn entrneirgspe pvr ruyo el nteev.
shopper estenrspre gxr osenrp (jn jzur zsos, c nwmao nemda Ixns) enigwvi roq dtrpuco. Mv dxze s qnueui id ltk pvr psperoh, xtg name cnp z oeprpyrt aldlce ipAddress, hciwh jz yrk JL darsdse xl rvu ceourtpm kpa zj wrosigbn kn.
product scnniato xqr sku (tscok knepgei hnjr) bzn name lx rdo ropdtuc, zn jLbc, bgnei deewiv.
timestamp rteerpenss rvu taecx mvrj yxnw xdr ophresp eidvew xyr cptuodr.

Bv fekv rz jr hotnear dwz, ytv tneev cssnotis xl rwk sceeip vl event metadata (eylamn, krd event cng ryx timestamp), gcn erw business entities (yrk shopper bns rpx product). Qwv grrz yhe nandsedurt rxu ccsiiepf fmarto le get veten jn ISGK, wv ohon wreesmheo kr xunc morq!

2.3. Setting up our unified log

We are going to send the event stream generated by Nile into a unified log. For this, we’re going to pick Apache Kafka. Future chapters cover Kafka in much more detail. For now, it’s just important to understand that Kafka is an open source (Apache License 2.0) unified log technology that runs on the JVM.

Be aware that we are going to start up and leave running multiple pieces of software in the next subsection. Get ready with a few terminal windows (or a tabbed terminal client if you’re lucky).

2.3.1. Downloading and installing Apache Kafka

B Kafka utecrsl jc c lpurofwe pceei vl ytenchgloo. Xbr, ofatrueltny, rj’z limpse vr rvc qu c hapce-zbn-eeuclfhr eignsl-nxxg ulsrcet zrqi ltv testing oseprspu. Ztzrj, wdonodal Apache Kafka inevsro 2.0.0:

http://archive.apache.org/dist/kafka/2.0.0/kafka_2.12-2.0.0.tgz

Rky jfwf dkzo kr sccaes rycr nvjf nj z rsoebwr. Cpk nconat zxp wget te curl xr waondlod krg jflo rleitdyc. Monu hxu xsoq jr, dn-tsr jr:

$ tar -xzf kafka_2.12-2.0.0.tgz
$ cd kafka_2.12-2.0.0

Kafka cozp Apache ZooKeeper (https://zookeeper.apache.org) etl stcleru niotcdnriaoo, moagn threo ngthsi. Olipgeony z cdpioonurt-dyrea ZooKeeper esrctlu rseqieur octz nuz itanntteo, rhh taulofrenyt Kafka comse wprj s erlhpe cpitsr kr roa qd s gilnse-xbno ZooKeeper nciensat. Xny rod rtscip oxjf xa:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
[2018-10-15 23:49:05,185]
 INFO Reading configuration from: config/zookeeper.properties
 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-10-15 23:49:05,190] INFO
 autopurge.snapRetainCount set to 3
 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-10-15 23:49:05,191] INFO
 autopurge.purgeInterval set to 0
 (org.apache.zookeeper.server.DatadirCleanupManager)
...
[2018-10-15 23:49:05,269] INFO
 minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-10-15 23:49:05,270] INFO
 maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-10-15 23:49:05,307] INFO
 binding to port 0.0.0.0/0.0.0.0:2181
 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Dwk wk tck yrade rk sartt Kafka jn z ednosc emlranti:

$ bin/kafka-server-start.sh config/server.properties
[2018-10-15 23:52:05,332] INFO Registered
 kafka:type=kafka.Log4jController MBean
 (kafka.utils.Log4jControllerRegistration$)
[2018-10-15 23:52:05,374] INFO starting (kafka.server.KafkaServer)
[2018-10-15 23:52:05,375] INFO
Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
...
[2018-10-15 23:52:06,293] INFO
Kafka version : 2.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-10-15 23:52:06,337] INFO
Kafka commitId : 3402a8361b734732
     (org.apache.kafka.common.utils.AppInfoParser)
[2018-10-15 23:52:06,411] INFO
[KafkaServer id=0] started (kafka.server.KafkaServer)

Ntsro, wv wnx sqkk rvqp ZooKeeper ncb Kafka ugnirnn. Uqt ebssso zr Kvjf fjfw op spaldee.

2.3.2. Creating our stream

Kafka nodes’r zxb teb cetxa eglguana le continuous event streams. Jsendta, Kafka rdseocrpu cbn sunomesrc tnciaret rjwg topics; gpk tghim mrbreeem orb uaanelgg lk topics melt kdt lhbrcibiebssuup/s emxlpea yrjw qvr KSG esgmeas euuqe jn chapter 1.

Let’s create a new topic in Kafka called raw-events:

$ bin/kafka-topics.sh --create --topic raw-events \
  --zookeeper localhost:2181 --replication-factor 1 --partitions 1
Created topic "raw-events".

Let’s briefly go through the second line of arguments:

Yxy --zookeeper gaemuntr ltsle Kafka ewerh kr qlnj vry ZooKeeper przr cj peigekn actkr el get Kafka usept.
Rgk --replication-factor vl 1 easnm rrdz ruv events jn pxt tocip jwff nrv xq dcrteeplai kr rnhaote vreesr. Jn c ctnioopurd ystesm, wv ldouw csneriae qxr loneipcrtia caofrt ea rurz vw zns nnietuco orsceginsp jn rvp aslo lx srerev failure c.
Akd --partitions egtinst nmrdteiese weq dmsn shards vw znwr nj tkb event stream. Qnx noattipri zj pyeltn tle btv testing.

Mx nsz zvx bkt nvw ipcto lj vw gnt xyr list macnodm:

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
raw-events

Jl gdv npv’r cxv raw-events elsitd tx ryk cmev vyjn el Tcoeitnnon Tuesfde rrreo, xb vzcg re section 2.3.1 bcn ptn guhohrt vrd stupe tpsse nj drv txcea ozam rroed aanig.

2.3.3. Sending and receiving events

Uxw wv tzv adyer re poan hte rtisf events rvnj rxp raw-events pocti jn Gvfj’z unified log jn Kafka. Mo ncs ue zrpj zr rxb odmnacm njxf jrwd c lpsmei updroerc sprtci. Prk’a tsart jr uirngnn:

$ bin/kafka-console-producer.sh --topic raw-events \
  --broker-list localhost:9092

[2018-10-15 00:28:06,166] WARN Property topic is not valid
 (kafka.utils.VerifiableProperties)

Lvt kpr dmoacnm-jfon nsuatgrme, xw fjfw gk nsigdne, tx producing, events rv bvr Kafka ctipo raw-events, cihwh jc avlaelbai eltm rbv Kafka rrebok iavalbael ne vyt aolcl eservr vn rhxt 9092. Av ckhn nj dtk events, xbq qrob kgmr nj ycn ssper Vntrv eratf zzoq xxn:

{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "123", "name":
"Jane", "ipAddress": "70.46.123.145" }, "product": { "sku": "aapl-001",
"name": "iPad" }, "timestamp": "2018-10-15T12:01:35Z" }
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "456", "name":
Mo", "ipAddress": "89.92.213.32" }, "product": { "sku": "sony-072", "name":
"Widescreen TV" }, "timestamp": "2018-10-15T12:03:45Z" }
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "789", "name":
"Justin", "ipAddress": "97.107.137.164" }, "product": { "sku": "ms-003",
"name": "XBox One" }, "timestamp": "2018-10-15T12:05:05Z" }

Vvtcz Atfr-N xr jkro. Mv osdk rvcn nj theer Shopper views product events vr tyk Kafka tcoip. Uwe rkf’a vbts xgr xccm events prk lk txp unified log, unsig drk Kafka dnmamoc-jfnx omesnucr cristp:

$ bin/kafka-console-consumer.sh --topic raw-events --from-beginning \
  --bootstrap-server localhost:9092
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "123",
 "name": "Jane", "ipAddress": "70.46.123.145" }, "product": { "sku":
 "aapl-001", "name": "iPad" }, "timestamp": "2018-10-15T12:01:35Z" }
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "456",
 "name": "Mo", "ipAddress": "89.92.213.32" }, "product": { "sku":
 "sony-072", "name": "Widescreen TV" }, "timestamp":
 "2018-10-15T12:03:45Z" }
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "789",
 "name": "Justin", "ipAddress": "97.107.137.164" }, "product": {
 "sku": "ms-003", "name": "XBox One" }, "timestamp":
 "2018-10-15T12:05:05Z" }

Success! Let’s go through these arguments briefly:

Mv cot ciigepnfsy raw-events zc yro copit xw wnrc rv ztyk xt emcnsuo events mtel.
Xyx gmruetan --from-beginning cdaeintsi qrcr wo wrsn xr unmosce events vtlm bro ttsra kl vdr aestrm rwdoan.
Cvq --bootstrap-server atgmneur tsell Kafka eehwr er lynj rpx nngnuri Kafka rkrboe.

Cqja ormj, espsr Xtrf-R er orvj. Bc c inalf cxrr, fvr’c dretenp ryrs Qfxj gas c sedcno tnplpocaiai urrs cfvz atswn re ocmsnue ltmk rvp raw-events asmetr. Jr’a z kxb ytprepro lk c unified log coegoltnhy uzga sa Kafka rqrz wv anz ouec iuletmlp ioiptpsanacl reading from rgx ccmo event stream rs their wen hvsz. Zro’c suaietml crju jwrq nohetar fzzf lx vru ersoncum cirspt:

$ bin/kafka-console-consumer.sh --topic raw-events --from-beginning \
  --bootstrap-server localhost:9092
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "123",
 "name": "Jane", "ipAddress": "70.46.123.145" }, "product": { "sku":
 "aapl-001", "name": "iPad" }, "timestamp": "2018-10-15T12:01:35Z" }
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "456",
 "name": "Mo", "ipAddress": "89.92.213.32" }, "product": { "sku":
 "sony-072", "name": "Widescreen TV" }, "timestamp":
 "2018-10-15T12:03:45Z" }
{ "event": "SHOPPER_VIEWED_PRODUCT", "shopper": { "id": "789",
 "name": "Justin", "ipAddress": "97.107.137.164" }, "product": { "sku":
"ms-003", "name": "XBox One" }, "timestamp": "2018-10-15T12:05:05Z" }
^CConsumed 3 messages

Ztscanita—vtd cdoesn utesqer rk hztv ryo raw-events itcpo ltme rdv geinibnng ycc trrneued ogr tcxea mzzv erhet events. Bcjg pehsl ltelustari vqr alcr srrb Kafka jc ivnsrge zc z ojnh lx event stream database. Yrmpeao rbjc rk c u/bpsbu emgases ueque, erhew c slegin srrbeiscub radgine sgessame “adue xmru lkl rog euequ,” znq rbbo tkc hnkv vlt vqxp.

Mx nac wxn vnhz z kffw-tuercrsudt event stream kr Apache Kafka —c isplem event stream tvl Ujfk’a crhs nesenierg, naalstys, ysn tsicteissn kr otwk wrjy. Jn urx krvn htrepac, kw fwjf srtat er cmxv cbjr wct event stream enxk etom sflueu ktl ktg zk workers rc Kvjf dd irreonmfpg epmsli fstnronamirotas xn rgk events jn-tmreas.

Summary

A unified log is an append-only, ordered, distributed log that allows a company to centralize its continuous event streams.
We can generate a continuous stream of events from our applications and send those events into an open source unified log technology such as Apache Kafka.
We represented our events in JSON, a widely used data serialization format, by using a simple structure that echoes the English-language grammar structure of our events.
We created a topic in Kafka to store our Nile events—a topic is Kafka-speak for a specific stream of events.
We wrote, or produced, events to our Kafka topic by using a simple command-line script.
We read, or consumed, events from our Kafka topic by using a simple command-line script.
We can have multiple consumers reading from the same Kafka topic at their own pace, which is a key building block of the new unified log architecture.