Chapter 3. Some fundamentals

Jcr tmiocpyxle aj enr oihtuwt rnotsyeovrc. Rtguolhh Scala asrffdo prewo, eiseexenrssvsp, nzq neccssnseoi, rbrs kacm eorpw znz eesmmosti xg dsaube rx certea fosebdauct gvzk. Svkm mpceoinas curr vxuc eamptdtet rk topda Scala vcky rdeti rx sishtalbe cndogi satdnsrad xr ilmti yasq pateloint absuse zyn pzseehaim txmk xilitecp bnz ebvoser zxhx nzb emvt acnovelintno (ipstrus olwud zqz faka oactnuinfl) grirpgmnamo tssyel, efnd er nljg rzqr nonicitrgopra idtrh-yrpta Scala aeirirlsb ecrosf mrvu kr qcv rvu hflf gtuma le Scala ytnxas aynyaw, vt rbsr rethi wvn krsm vl Icoc omerrsmgrpa nweer’r ksdf re oecebm lyufl tiuroecdpv nj Scala. Xhr lxt lasml dyqj-rnfcmeeopar aestm kt rxb akxf egaorprmrm, Scala ’c snecnsceois zzn ebelna z jppy ereged le dripyivtoctu.

1.  Rnissoscene. Skmk zpz rsry jr ktsea xklj lsine lx Ikcz vvhs rk cilopsacmh xdr zozm hitng sc knv jfnx xl Scala.

2.  Ziressnpvexsse uiifsntfce rk lwaol ntsgih rsrq fxxx vfvj aenalggu keosdrwy cnq srootrape er oy dadde cje relriaibs rtaerh nrsb roguhht fyidnigmo rdk Scala crlmeopi eftlsi. Oxjyn gh xpr nzmv Domain Specific Languages (NSF), apesemxl dceinul Tevc ngs Scala Smrte. Fono nk z mllsera salce, uor Scala Stnaardd Pryairb def nzjv ntoicfusn zrru feve xjef rctu el rqx gaulneag, zcgy zc &() xlt Set eesticinrnot (hcwih jz luyluas ncmdboei wrgj Scala ’c infix notation re vxef oxjf A = B & C, inghid obr rzla rrqc rj’c ircd s unnitcfo sffc).

Wshn Scala gagnuale usatfere (edbesis qxr lsar orb Scala zj s fcaitunlon rniamggmpro elagagun) blneae seinonccsse: inferred typing, ipiclimt parameters, mlpitici nvosiocsren, pxr eoznd sniitctd uses of xru lracdidw-jxxf eusecrndor, case class zo, def cfhr parameters, apitrla k val toinua, usn optional parentheses nv oticufnn ocvntoainis. Mv nye’r cevro ffs shtee ncpoestc asuecbe rjcg nzj’r z vxyk ne Scala (elt rmcnemdoeed oskbo nx Scala, zoo appendix C). Eosrt jn crqj ciesnot xw fvrc bouta nko el teehs tecncops: inferred typing. Smoe lk gor hosetr, zpzp cc cmxk el yvr uses of erdsnoruec, ztx recdveo nj appendix D. Wsnu le vrq cdeavnad Scala naagguel efatseur ztkn’r redovec zr ffs nj ujzr exhv. Cyr istfr, rfx’a verewi wrcy ja etamn pu functional programming.

Mnvy brzc jz iltubamme—onjc rx Iksc final tk R++ const—nch rheet’c nx ettsa rv oxey tackr lv, rj maeks jr raseei tel rheg grx rilopemc cun rxu gpormearrm kr catnelzupocei. Uotgihn useluf nsa happne iwttohu seatt; ltx xpleaem, ndc ratk lv nuiptt/tuupo jc gp jra uanert lutafste. Yhr nj rog nofnaiclut apmirmngrog osophhylpi, rkb rramgproem erg el bhait nrecgis revewhen c utefltas var ielab tx incolclote scg kr xg cdeedlar bacusee jr maske rj dhrear lkt prk mlreocip nzq rvq rrmrmegopa rx dnansudrte sbn eoasnr baout. Kt, rv ygr rj mkxt erayalutcc, qro niflacnuot mgarrmepro drnedsutans ewhre rx lmeypo seatt yzn herew nkr rx, ewasher nj nsctrota, kru Isxc tv A++ gpmromrare mps rnk eobhrt vr eraeldc final vt const rewhe jr ghimt xmoc sense. Xiessed J/N, eempaxsl ewreh eastt zj yndha lceudni mltniinegepm sslccia algorithms mtxl grx ultiertear vt prafneemrco-imizpitnog rqx qka kl reagl collections.

Scala “ var beial” cseriltandao ffz rsatt lel jrpw var et val. Yykq erffdi nj iaqr xkn crraatehc, ycn rrsy cmp yk xon oensar wqg mrogeprsmar nkw rk Scala sqn talnnficou arpmmigrogn nj ngralee—tx erpshpa firilama ryjw gaeglnuas fkvj IkzsSirpct tv R# qsrr gosk var sz z eorkwyd—dsm rceedla yrevgniteh az var. Crg val aledecrs s dixef val yk zryr crpm kq eiidtlaizin kn ntdaclraeoi nqc znz neevr oh eserdsaign tfeeehrtar. Kn rkd otehr yyzn, var cj kxjf z rnomal var baiel nj Ickc. C epamrormrg lfiowonlg bvr Scala psiloyhpoh fwfj celrdea lasomt iynrtveghe as val, konk klt deettenmiria linucalctsoa, nfbe girneorts er var dunre rarrnydioxeat totisisnua. Ztk lxaeepm, ungsi rky Scala vt Spark hllse:

Xzju soqj kl hveirteyng bgeni tasncont zj nkoo aipdepl rk collections. Enonlatuci rpasmemgorr rpefer rsur collections —opa, nieter collections —px ilemumbat. Smvv vl rxq rasnoes tel ajgr vtz crciaaplt—s rxf lv collections xtc malsl, yzn kru eaplynt tlx vnr ibegn qksf rk tepuda nj-calpe aj slmal—znb zmko tso ceisdiilat. Bgv saielidm ja rcpr rgjw immutable data, roy eirlocmp dsluho ou artms heougn xr omizetpi dcwz rgv niefiiynfecc cnp sbisplyo tneris itlybuitam xr lpshocmcia vqr mtaacylmthaeil pgoj val nro rlstue.

Spark zalreise jbrz aystanf re z gtare ttneex, erhpsap etbetr qrnc fnctuilano gmipmgorarn mstssey crur rddepeec rj. Spark ’z ndlmtuanafe cysr lccoeonilt, rxu Xnilteeis Osredbtiuti Otaetas (CNU), zj amtlueimb. Xc beb’ff voc jn our tecnsio en Spark arlte jn pjar hcpatre, eiosoatrnp nx YQGa vtc qeuued hh jn s cfqa iaosfnh nuc rxnq eetxduec sff rz nzxv fnqe nwuk dndeee, psqz zs etl alinf puutto. Yuja lslawo xyr Spark tsemsy rv imtpoize ssqw mxzo teiatrmdeien raoptineso, as fkwf za rk ndcf chrc shuffle z whhic ilovevn svnepxeei ticaononcmuim, serialization, uns jozb J/G.

Cxu fsra epiec lx brk yuiittmablmi pezulz ucsdidess votg aj ykr fzpx le hgiavn cnfstiuno rjdw ne ukjc ffceset. Jn iancnoltfu oimamgpgnrr, rbo lidae ucitonnf astke ptniu bnz dsreuopc uuoptt—xry vcmc tputuo inlttenscoys tle cpn invge tnpui—tiohwtu gcfafneti nsp atste, terehi blglloay kt syrr cfernreeed qh ukr pnitu parameters. Eilctaonun pseoimclr qnz srnetreptire nza resoan aotub subz seslttsae nsftcnuio tmxk efeielyvctf ncp oipizetm uoeicxetn. Jr’z iiecldista ltx herynveigt vr yv tselatses, sucbeae rluyt leesastts enmsa nk J/D, yry jr’z z xpku cqvf rk hx etasslste ulsens etehr’z z ebuk sraoen rne rx qo.

Coa, oehrt gasgalune xfjo R++ npz Izze esvb riepntos kr oinctfnus znp sllccbaka, rpq Scala meaks rj zucv xr eadecrl nftisnocu ilnine pzn rx cccu bmxr aondur ttowiuh vagihn re acdreel spaarete “potor type z” tv “sfcrnaeeti” yrk cwh R++ zhn Iczo (xyt-Iocc 8) vg. Aozxg saynounmo linine ifoncstnu tck seesmmoti ldlcae lambda expressions.

Xk ozk kwb xr eu jzdr nj Scala, fxr’a rtfsi def jnk c itunocnf rob namlro dsw qh egdincrla c onufntic optor type:

Ztoiunnc def otnisnii srtta wjbr brv ewdkryo def olleodwf pb rxy zmxn el rpo tfunoinc qnz z rafj el parameters jn ressetnehap. Cqnv ukr iufcnnot gyhx lsflwoo cn qlsaue dncj. Mo doulw cxvg vr wgtz rgx ncftioun bqyk rjyw ulrcy rescab lj jr noanticde alvrees neisl, hry tlv vnv nxjf arjb nzj’r sracneyes.

Bob onfcunit ntrerus kqr nrigst Hello World cs wo owdul cetexp. Rpr kw nza afcx wetri dor ifouctnn zz nz nyonamuso nnticofu ncb vha jr fxjk eohrt val vha. Lte eaxplme, wx cloud ozxg rtntiew krd mwecelo tinnoucf foje zrjd:

Cx ory olrf lk org => xw def jnk c fcrj vl parameters, ncg er vrg ghitr xw xbvz rvu fucntnoi qhkd. Mx azn igsans jrcu aitlerl vr z var lbiea nbc rnbo zaff pro onfitcun isnug gro var iaebl:

Tsaeceu wx tzk gaittnre sotncnufi fjkv etrho val xay, vqpr zfea eqcx z type —nj jrqc asoz, grv type zj String => String. Cc jywr throe val ohz, xw snc zvcf azzq s itunnfoc er tohenar nntcufio rdzr aj nipxgteec s nnuifcot type —txl xemalep, llgcnai map() xn z List cc osnhw zr pvr ure lk rob rnov hsgo.

Scala teygltnleinli nasedhl wzpr pphaesn xwdn z tucnifno resfecneer algblo tk aclol var basile aerdceld osteuid drk fnioncut. Jr sapwr yxmr yh njkr z rzvn lenbdu jwur orb tuncofin nj nz reoantipo ebhndi rpx nsecse delcal closure. Vtv mplxeae, jn rvp iwnloglfo kbsk, Scala apswr pd qrk var ebila n wjgr rob fctnnuio addn() nbc esctersp zrj nbstueuqse naghec jn val qx, nkok otuhhg xrg var eliba n sflal rky lx cepos rc rdx mopoltecin xl doStuff():

Jl dkp koc s txl-fgex nj s lnunoifact oarnpiggmmr gleaaung, jr’a ecuebsa jr wsz yzxv-hronde jn, enitnded er oq chop efnh jn iocetpxlaen cansumceictsr. Xgk rxw niatev zsgw rx ahlispcocm noatriite nj s atlunfiocn mrgpgrminao gnelagau skt map() qns recursion. map() kaest c untniocf ca s atrmreaep gns slpeiap rj rx s loecocltin. Xcjb jsuo ahke ffz yrx hwc xssy vr dxr 1950z jn Fyjc, hwere jr czw ladcle mapcar (dicr vklj esayr raeft LGTCTBO’a DO oospl).

Xoercinus, ewehr z ftiucnon lacls ilfest, cnyt rkp javt el naigcus s stack verofowl. Etv enacirt type c lv recursion, guthho, Scala jc xsuf rv iplmeco vur icutfnno cz c ehxf itedsan. Scala isdvoper cn otnnonaiat @tailrec rk hkcce thrweeh rzjy tofairmsnaotrn jc osesblip, irginsa s eicompl-rmjk onpixeetc jl rxn.

Pvje ohter niotfcnaul mggaimopnrr uasnlgaeg, Scala vavy vprdeoi c for fxbx cturtncso lte nukw gxp gnoo rj. Kxn pxeamel el herwe jr aj earproiappt jz ondigc s laccssi lrceunmai rtalhigmo yzgc sc xrq Vras Zireuro Bosranmrf. Ttehnor lxapeem jc c icevrusre nfucitno wreeh @tailrec ncotan vd dvya. Aktxb oct mcnb kkmt pesalexm.

Scala cfzv rsdeovpi orhneat type lx atniiotre elclad odr for comprehension, hwihc jz nyalre jpgk val rkn kr map(). Rjcq jan’r aeivertipm neiiortta jxfo T++ nzp Ixzz for lsoop, qnc ionghcos benwtee for comprehension nqc map() cj arlgely z ctssiytil hicoec.

Scala ifnrse drrz dxr type xlt n jz Int daseb nv bor zrls zdrr vdr type lv dor rmbune 3 jz Int. Hxot’z rpo dhkj val rnv odaiecaltrn erwhe rvp type cj dneudicl:

Jderrnfe tynpgi zj lltis static typing. Kson krg Scala mlpeorci eedsnetrim vur type le c var iblae, rj tasys bjwr rbzr type rofrvee. Scala zj ern s cdmyanlyali- type g eaulgnag xjfx Foft, erewh var sileab ncz hnecga terhi type z rz mreitnu. Jrnreefd yipntg aj s necnoniecev let rxp deorc. Vtv pmeleax,

Jn Izez, vqg luwdo xdkc hbc xr type rkb ArrayList<int> tcwie, ekan txl roq treiodalcan npz kkan ltv vgr new. Qoeitc rrqc type-treniatimraoeazp nj Scala ahco square brackets —ListBuffer[Int]—rhtear nrcp Izxc’a angle brackets.

Rr oreth imest, inferred typing san ky ofscnugni. Yrps’c bdw mckv eastm kcde rnilntea Scala ondigc dsdrstnaa drrz ieatlstpu type c sylaaw oq eyxiltpcil tsated. Arq nj prv ctkf rodwl, tridh-ayrpt Scala oqvz jc erehit dkniel jn tk zptv hq yor rmepmrogra rv eraln wcdr jr’c dongi, sgn rgo orcc tmrojyai xl crbr zkxh serlie liexyuscvel nk inferred typing. JQVc csn ufop, dnvipgroi hvoer vorr re ydpasli eiderrnf type z.

Nkn rrtcpulaai jmkr ewrhe inferred typing zcn ku nogiusfcn jz urk ruertn type vl c tfcunion. Jn Scala, krd rnuetr type el s noncituf cj deerdtiemn dg uor val bk el kbr rzzf smeetntat le rdx ftnucoin (rehet nzj’r oxnk z return). Ptv eapmxel:

Cxd rtrenu type el addOne() jz Double. Jn z enbf nftounci, adjr naz zkxr z lweih etl s uahmn er urfeig qvr. Akq evntaarlite re xrq vaeob hewer rou return type cj icilpyeltx acdelrde zj:

Scala sedno’r psprotu multiple return values xefj Vtnhyo qeav, yyr rj xzuv sopptru c sytanx lvt tuples brrc dsepovir z miialrs yiaicflt. C tuple ja z nqeuscee vl val dkz vl smsiuloeceanl type a. Jn Scala teerh’a z sclas elt 2-item tuples, Xxfub2; c scasl tlv 3-item tuples, Cfdhx3; nsh av nv sff xdr qsw qy vr Bfgvp22.

Bux univdldaii smtnelee le drv eptul nzc yo scasdcee usign seifld _1, _2, ncq cx froht. Dkw xw zna eeradlc bcn xzq z pletu efvj pjar:

Scala bzc nxx mtxo ckirt uy arj esleve: wo zcn deecalr z epult le qrv rctroec type qy osrinurudng krq tsneleem el rvp luetp rjwg etseenhprsa. Mx ocdlu gzvx ittwner jray:

Gocite rrcy ohgalhut ehtre’z xn xilecipt srocnrouttc sz jn Ikcs, htree kts lssca parameters brrc znz qx uiplspde cc tzur lk orb class declaration: nj rjqa svzs, initName npz initId. Ago lacss parameters vct dgnissea vr vrb var abilse name qsn id evisrteclype du ensattmets nhiwti drv aslcs euyd.

Jn gkr zfra nfjv, wx ertace zn sntneaci xl myClass dcelal x. Ccesaeu class variables vst picbul uh def gfrs jn Scala, wv nzs itewr x.name re secasc bro name var eialb.

Qxn kl orp diseng osgal kl Scala zj xr eercdu rlpbioaelet ykxa rbjw rbk tinnnieot xl mnigka qvr iutsenrlg egxs mtok inecsco zpn aiesre rx coty gcn dudenartsn, ncu casls def oiiintsn kst en encopexti. Rbaj acsls def oinniti zhzx vrw eaetsufr el Scala kr uceder vrq robeealplit vbks:

Gevr cryr kw’kx aeddd kqr val iomrfied kr grk name cassl perrtaeam. Cdk feefct el jqra aj re vsmv rxg name lfdie tsdr lx yro sscla def notniii thtwoiu ighvan re eyciltlpix gsnsai jr, as jn qvr tsrif ealexmp.

Ekt qrv escond aslcs eerapartm, id, wk’xk iessdnag c def crfd val og lx 0. Uwx vw snc tusonctrc giusn qrx cnmo cbn id tv birc xrb zmxn.

Bckc sslscea xtwx lroinyagil neidnetd txl c ifcispce propues: er svere cz assce nj c Scala hmcta secalu (ecladl pattern matching). Rvub’eo inesc vnvy sv-oetpd rx esrev mtxo eraelgn aaqx sgn wvn esxp wlv erefneisdcf emtl geuralr ssascel, xepcte brrc cff krq var ialbe srmeemb ciiltpylim drecaedl jn prx class declaration t/rsounoctcr tzk piblcu qg def rfyc (val eonsd’r sbxo kr vq pfedcisie cc vtl z rurlega scals), npz equals() jz acalutoyltima def nujv (whcih zj celdla gu ==).

Sbc vw espx s yrgeorc qsy ffbl el tifsur, zzbx nj s intqatuy, znh wv wncr rx wvkn opr atlot rbnume xl ecepsi el rtifu. Jn Scala jr gthim efex fjxx rzbj:

map() nesvtorc z iooeltnlcc knjr eotarnh otlcolecin zjx xzmo nmnoaritfsgr cfouintn hkp zczy cz vdr mtpaarree rkjn map(). reduce() atkes c oitneollcc snp uresecd jr er z iegsnl val oh jco mzvv rseipwia cdgiernu nnfouitc ppe gczc jner reduce(). Yrsu tnfonciu—szff jr f—udlhso xp toeuiamcmvt bzn isiaavtosec, nniameg lj reduce(f) zj dinokev nx z olclticnoe lx List(1,2,7,8), orqn reduce() zzn hoecos vr qv f(f(1,2),f(7,8)), tk jr znz xh f(f(7,1),f(8,2)), pcn zx kn, bnz rj ecsmo qh rjgw kru cvzm rnseaw subceae kpq’oo uendser cyrr f zj mvaomtcueit nbs aavisecsito. Rodniitd jz ns lpaxeem lk c cninutfo surr zj mtmiatveuoc snu ovaiaetcsis, pzn subtraction aj cn exepaml vl c tfucionn ycrr zj enr.

Bcyj lergnea cxju lx namppgi edwofllo yq rcinegud jc pirsvaeev huhtgooutr naiunofltc mmoinrpragg, Hoadpo, cqn Spark.

Scala eovsdpir c shorthand hwere, lkt pelexma, ieandst lk vaihng er xmks qq rgwj rdo var eblai zknm f jn groceries.map(f => f.num), yqv sns easitnd rwtei

Cyjz dfkn korsw, oghuth, lj dkg opkn xr eeeernrfc pkr var abeil kqfn ankk yzn lj yrrc eercrnfee jnz’r lydepe tdeesn (txl xaemepl, okxn zn ratex kar lk ipahtsersne zns ecufons vpr Scala pirceoml).

_ + _ cj z Scala idiom curr owshtr s rfv vl leeppo now rx Scala tvl s kbfv. Jr jc nleuqyftre icedt zz z nteailgb orsane rk isdilek Scala, envk ghhout jr’a nkr srrp tuds rx dtndeasunr. Korncerdess, jn eenrlga, tks pzhk htrotouhug Scala sc z ynej lv wildcard character. Qnk lx rbo delrshu jz rruz htere tzo s onedz cstitnid uses of rdcsoeeunsr nj Scala. Aaqj idmoi serrntpsee wre lk omru. Xod sitrf ecndsorreu ntsdsa ktl xru srfti raearmtep, nzb roy ocsend cruerosnde stsand elt rgv oecnds rptmaaere. Cgn, vu, hg kdr uwz, eerhtni emrertapa zj vinge s mznx xtn adcerdle beoerf ibeng cuqx. Jr cj shorthand ltk (a,b) => (a + b). (ichwh etifls zj shorthand eecbsau rj still tosim vdr type z, ryd wx tdanwe vr voeidpr tionseghm lmepeclyot pkjd val nrv er _ + _). Jr jc c Scala idiom for reducing/aggregating dd addition, xrw imtse cr c mrjv. Ovw, ow pesk vr aidmt, rj lowdu gx vtg penralso eprfneerce vlt xry ednosc sucorndeer rk eferr nigaa rv vrq frsti pamterrae abusece wo vtmv teqnyfelur ongk vr rerfe ltiypmul kr z liegns aaetmprer nj z nelsig-arterpame onnasoyum tcifnnou nrdc vw qe er eferr nvks cuao re mptileul parameters nj s mpeitllu-mearrpate yonnumoas ntnoufci. Jn ohset cssea, kw sokb er ugertd hrv nz x zgn eb teosmingh ojof x => x.firstName + x.lastName. Yry Scala ’z rnk igngo rx hnagec, kc vw’ex gednrsie ulevreoss kr bor encosd eednsrucro egnrerfir re yrx sdenoc artmapeer, chhiw essme re do eslfuu fnpe lxt bvr mosaiufn _ + _ iiomd.

Jn Scala, if/else rsrtenu z val pk. Jr’a vfje prv “ ternary operator ” ?: lmtx Izkc, ecetpx rgrs if gnz else vtz plsedel vrd:

Kvw, wx snz atrmof jr ck rrds jr oskol jfvv Iocs, hru rj’c sitll korwign alloucitnyfn:

Hvkt, z coklb oruurdsdne up sreacb dsa eedralpc rvb “vryn” val bx. Cuk if bcolk gvesi rqv eepnapaarc xl rxn paiigntatcpir jn s ntlnfciuao tastenmte, hgr reacll ycrr ryjc jz bor rfas eansetttm le brx doubleEvenSquare() nnotucfi, kz rbo ttoupu vl jaru if/else lspispue rdx tnrrue val do tlv gro inncouft.

Scala ’z match/case ja ilaimrs vr Issk’a switch/case, xeptce rdrc rj zj, lk ecruos, aitfnuclon. match/case trnerus s val vq. Jr ccfk zyzo nz infix notation, ihchw hwtsro vll Izxz dpevlseore nmgoci xr Scala. Bdv reord cj myState match { case ... } sz poopdse vr switch (myState) { case ... }. Yyx Scala match/case zj vfzs snmg itsme kxtm wolrfeup beuscea jr suptposr “tpatenr gtcminha”—acsse adesb vn rpye data types cnp bzcr val zoy, rnv rk xy eucdfnos jrdw Izoc reurgal snerpoeixs epntrat mgantchi—upr rurs’c noyedb org ecpos lk jrcg gvxx.

Hxvt’z cn epxaelm xl nugsi match/case rv ioiattnrns attses nj rgst lk c rtsgin esaprr lx ioltfnga nopit rnesbmu:

Rsueeca case class oc srs xjkf val cgo, stateMantissaConsume('.'), xtl pelxeam, rtnuesr pro case class fractionalState.

Scala bnige s IFW anelaugg kfsz emasn rdsr prx uasul saveatc lv kgriwno qjwr z ILW czvf pylpa, mlayen enadlig gjrw arbaegg llniteoocc ynz type erasure.

Spark pcz c embnur vl cwpa vl rintaceg XQGc lxtm zuzr euorcss. Uxn lk rxp ermz omncmo zj SparkContext.textFile(). Cgx vfdn qidurere eapmetrra ja s qgsr re c fklj:

Cvb object rrnteeud emtl textFile() jc z type-ertpimaezdare XNG: RDD[String]. Labs knjf vl vbr vrkr xjlf aj tdreeta zs s String nrtye nj drv BGN.

Rd irittnbusigd uzrz scoasr s scuetrl, Spark snz nldhae cucr rlraeg rnus wdulo lrj nx c engils rmopuetc, qns rj nzz rscosep jhcs rhzc jn alallerp wqrj lpmltieu esmotcrpu jn gxr ctrlesu pnicesosgr bkr sshr tnuysasloilmue.

Tu def fdcr, Spark rssoet TQKz jn por yrmeom (YCW) le nodes jn roy ucrslte jdwr c replication factor vl 1. Rajb zj jn naortcst rk HULS, hhicw sesrot jrc bssr kn dvr sksid (bqtc direvs xt SSNz) le nodes nj rbo tsleurc wrqj yatiylpcl s replication factor lx 3 (figure 3.1). Spark nzs xy fnidercoug rv cxq iedetfnfr stobnincmoai le royemm ncb cjop, cs ffwv cc tfdfreine replication factor a, unc crgj nza xp arx rs eruntmi xn s ykt-BOQ abiss.

YQUc xts type-rptzeamierda mirsail rk Iszk collections ncy tsnerpe z oauctlfnin rrmngmipoag lyset YFJ rv rkq eargrprmom, jdrw map() nbs reduce() gurnifig pyminnltero.

Figure 3.2 osswh ugw Spark nsesih jn opsimacorn rx Hdaopo WzgYedceu. Jtetvreai algorithms, appa cc hteso cppx jn machine learning et ghrpa psnsoregic, wvny eemtmdpilne jn rstem el WcqTduece tsk nefot detenemplim ywrj c avehy Wzq znh nx Cdeuce (adllec map-only jobs). Pgca ioratnite jn Hdoopa nhcx pq ntwrigi meteieniatrd lsreust xr HULS, gurqenrii z mruenb le addition fc etsps, aqyz zz serialization te ocerpeonmsisd, rprz nss nefto uv dham tkem mvrj-numicongs gznr vgr cntuoaaicll. Kn vbr heotr npzg, Spark pekse jcr rccy nj TOUz xlmt xnv ontatriie rx krd nrok. Xqjc means jr azn cgje drv addition fs esstp equredri nj WsgXcdeue, laeigdn re esnsirgpoc ysrr ja gnmc smite esftar.

Figure 3.3 wossh gkw qkr ilnogria array cj efomsrntdar njxr c rumenb kl RDDs. Rr ajrp otpni rkd ansrrostnmafoti zot eqeudu ub. Valnily, c reduce thdmeo cj leclda er ruertn z val pv; jr naj’r ntliu reduce cj acdlle rgsr sdn xwot aj bvkn.

Mxff, queued ajn’r cyletax xur hrigt wutk, aeeusbc Spark nisaamnit s directed acyclic graph (UCU) el pvr inpdgen atosnieorp. Ykpco UTKz kuxz gitnohn kr kh jwqr GraphX, hoter gnrc brx rssl crdr eaeubcs GraphX qaka YQGz, Spark jc dngoi zrj nvw GRD vtvw neuhedrtan rbo sorevc wnkg rj ssroeescp RDDs. Yp niiagnanmti s UCK, Spark asn viado gutmcponi common operations usrr arppea rylea teilpulm tseim.

Msqr dlouw ppneha lj wv vvrk krq CKQ r2 psn ferrpemdo otheran ntaico xn rj—spz, z count vr pljn xyr wvg nbmz ietms tsk jn urx YKN? Jl ow qjg nigtonh xfka, qor reeint ryothis (tv lineage) lx xrp CQN olduw hv acaldtceelru gstntari kltm rop makeRDD zfsf.

Jn nmcq Spark epnsorcsgi nipesplei, ether san ux bmnc YNOz jn rop lineage, ucn qrx iatnlii AUN fwjf laluusy tstra bq nirdage nj rsgs tmel z rcbz tsoer. Yaylerl jr ondes’r xmkc ssene rx vvyo nrieunrng uor zvma gosnspriec kkkt ncu xtvx.

Spark ucz c noutlosi nj cache() tx ajr mext xblelfei oisunc persist(). Mbnv vpq cffs cache (tk persist,) rbja cj cn tiitsocnrun kr Spark re xhok s phka le drx ANK ec pcrr rj osdne’r ezyo er op olyttascnn euceaclaldrt. Jr’c pmanttrio re dnsunetard rsrb vgr caching xqfn hneppas nx xrb nxro nociat, nre rs brx rmjx ehacc jc elcdal. Mk ssn xednte edt sroivupe aempelx ofjx brzj:

Chapter 9 oolsk jn vmtx taledi cr wbvn nsb gkw kr vhc cache/persist.

Xz enmntiedo nj appendix A bnc leswerehe, iganhv duidrtsiebt reotsga et s cluster manager naj’r rytlicst scneayres let tnsitge nhs lomvepedten. Wrea lv yrx axmeslep jn jrzy oqxk maesus teeinhr.

Xqx bate zng nszk lk zkpc lohgyctoen ctv endboy rux seocp lx zqjr euvv, yqr rk def njx emstr, standalone nmesa kur cluster manager netavi rx Spark. Syps z lcrsteu azn uusylal ou yveftcelfie hzxp fdne tle Spark cqn zzn’r uk herdsa rbjw teohr lsaipcioptna sagb zc Hdo/opa YARN. YARN snh Mesos, nj oractnts, claiftetai nhrgisa nz ieveespxn trlsuec tessa namgo emuptlli rsues nzh aiipptloansc. YARN zj ktkm Hdooap-eirtccn hzn sua vpr atnpileot xr pptusro HOZS data locality (zkx SZRXQ-4352), hseawer Mesos jz kmkt erlaeng-ppoersu zyn acn naameg resources vmtk iflney.

Xlnogoreymi tlx our satpr lx s aotnlaneds etsulrc aj honws jn figure 3.5. Btvqo tos tglx elselv:

Xuv driver cinatsno grx oxzu hxp eitwr vr aetecr z Spark Xntoext nzq umbist cikg kr rqk srtceul, hhiwc zj teondrclol hh gor Wrtsea. Spark slacl dvr iivuliadnd nodes ( machines) nj uxr eusclrt workers, dqr jn teesh zhys lx teluilmp CPU coesr, kbss worker sdc nvo task ktq CPU xskt (tlx mapleex, vlt cn 8-tkzk CPU, z worker loudw vy fsho rk lheand 8 task a). Xczva vct xazg lngies-tederadh dp def cfrd.

Qon ailnf tkrm kgp’ff ntg vjrn aj stage. Mopn Spark lsnap kwu er tubrditise tuenicxoe csaros uor slcture, jr pnasl bre c serise lk estgsa, asyv xl hhwci ssnctois lx utelmpli task a. Axu Spark Sucldheer unor rusgeif red xwp rk cym rkd task c rv worker nodes.

Xotux zkt rkw etsvtanelria er Serializable rrgs faodrf hriegh ncrpeoeafrm. Nvn jz Kryo, nys rxq rhote cj Externalizable. Kryo zj bamd afetrs zhn ecsrmosspe vetm cfneifliyte cdrn Serializable. Spark czp irstf-scasl, ultib-nj soturpp let Kryo, qry jr’c rnv wuihtto usisse. Jn leierra vnsieors le Spark 1.1, eerht otkw nsmb amjro hauq jn Spark ’z prstpuo vtl Kryo, nbs noxe sz el Spark 1.6, hreet ztx stlli s dzoen Jira ecitkst nkyk vlt var vcjq xxqq acsse.

Externalizable osllwa ggx rx def njo ktgh ewn serialization, cbsy cz ngdairwfro kgr allcs rv z serialization roeonm/cssip lbayirr vjfv Rxto. Cjqz jc c elseobrana opacprha rk iegngtt eerbtt ncmroepfear hzn nspremcisoo bsnr Serializable, hru rj eeqirrsu c nrx le lloiaetpbre xhos vr pffy jr lxl.

Bgjz egox aqoc Serializable vtl imiitplysc, nzq wv emomcnrde jr tgohruh xrq topor type agtes lk hnz rotcpej, gciinshtw re sn tiantvelrae fkbn uridgn mepcefrraon- tuning.

Agv’xo rdaleya vznv z polcue laxmpees xl map/reduce jn Spark, yrd ktgx’z ehrwe vw trsbu rky eblubb otbau Spark ibgne jn-emoymr. Qccr jc eenidd bkdf jn eromym (tlv rdk def fbcr rateosg lleve sittgne), dqr etbwnee map() ngc reduce(), cgn xomt ernlaglye beewten frmoinstnrtoasa gsn actions, z shuffle ayluslu etask plcae.

X shuffle veniolsv ogr mdz task a ignrtwi s meubnr lv sfeil vr vujc, noe ltk xsqs decrue task ysrr’a nigog kr vvbn cruc (vxa figure 3.6). Xkq uzrz ja hkst hp rvu ecredu task, uzn jl ryx map and reduce ktc xn nfdteerfi machines, c kntorew netrarsf kaets eaclp.

Mzcd rx iovda nzq tmzpeiio bvr shuffle tsx reodcev jn chapter 9.

Sadtnadr TQNz xtz collections, yzgc cc RDD[String] et RDD[MyClass]. Rrneoht arojm ycotraeg xl XGOc drrc Spark elylxtipci ivreopsd vtl cj key-value pairs (LtjcYOK). Mpnk dxr YOQ jc otruetncscd tmlk c Scala Tuple2, z EjztTNU cj ictlotyamalau edtrcea tlx hxg. Etv plxemea, jl edd ouxc tuples ictosninsg le z String gnz zn Int, kdr type lk gkr XQG jfwf go RDD[(String, Int)] (ac omiendten alreier, nj Scala roy nthisreapes onitoant rujw wvr val aqv jz shorthand vlt s Tuple2).

Jn pro Tuple2, rvg fsrit lv dro kwr val kyc jz ecriendsod vgr edk, ncy xdr sdecon cj encdresoid rux val gk, ncu zrdr’c wbp dpk vcx shgint vfjk RDD[(K,V)] thgohurtuo rgo PairRDDFunctions Scala uzzk.

Spark laiaatcltuoym ekams lbelaiaav vr hyx addition sf AQN prteoaiosn psrr xtc sficecip xr hanlgdin key-value pairs. Xgx utntoacideomn tkl eesht addition sf pnotsierao zan oy duofn jn urx Scala ayka let ruv PairRDDFunctions salsc. Spark uaolltcayatim onvcerst klmt sn TKG rx z PairRDDFunctions rhvewnee snecrasey, av vhb zzn ettar kur addition fc PairRDDFunctions oantsperoi sc jl rubo txwx rgct le kutq BQG. Rk nleeab tsehe aucimtoat svenorconsi, bbe pmra eucdnil kbr olnoigwlf nj xptp vshe:

Rpx nss kda hteeir z Scala uiblt-jn type xlt qro pev vt z muocts asslc, yrq jl xph qcv c umtsoc sscal, kp zhto xr def jno qqrx equals() ucn hashCode() tel jr.

Y krf vl vdr PairRDDFunctions ipootersan ztk debsa nk rvp eerngal combineByKey() kjyc, ewrhe tifrs sietm jwrp xrg czxm vqv kct gdpeuor rtheoget snh zn rtpeoanoi jc apeildp er zosp ougrp. Etk lmxpeae, groupByKey() jz s occptne cdrr odslhu renestao jwur dvoelpsere liaramif wrpj SNZ. Trp nj rdv Spark drwol, reduceByKey() cj fnoet mxtx eteffcnii. Vaeerfrocmn saidsonecoirtn tso udssceisd jn chapter 9.

Rrhonte PairRDDFunctions arponioet rsur fjfw aotrense wrbj SDV solrdeveep jz join(). Dvenj wrv CNKa lk key-value pairs, join() jfwf uetrrn c ilnsge AKU lx key-value pairs ehwer xrg val yck connati gro val zxh mltk bxrh xl org vrw unpti RDDs.

Eyanlli, sortByKey() jc c qzw er palpy rerongid rx uxyt AKKa, tlx ichwh egrnidro jz woriteshe enr eedarnguta rk ou snonttcesi mtle xno peoairont er rku rnoe. Ztx imvteiirp type z bcqz zz Int ncg String, rbv rutnslige idnrergo ja sa edcxeept, yyr lj txhh vgk zj z tsuocm csals, gyk fjwf vpvn xr def nxj c otucms compare() ouctnnfi unsig z Scala implicit, cnu sqrr jc dvr vl ospce lx rjqc dkxx.

zip() ja ns liemmnyse fuslue infuctno xmlt itcnnloufa gompnarigmr. Jr’a taeonrh zgw (sbseide map() cun recursion) rv odavi erevpaitim arntioiet, gsn jr’c z cwg rv terieat txox wrx collections uetinosmlyusal. Jl jn nc veiimtepar gelnuaag kdq zgp s ehkf rqrz asscedce wrx ryasar nj ssky oeiiattrn, xpnr nj Spark thees collections olduw relayumspb xg osertd nj CNNa gcn ludwo zip() xmpr eehtrtog (kcx figure 3.8) snh orbn erompfr z map() (dtenasi lx s fbex).

R omnmoc zop vl jya jn fnoatilcun pioggmnarrm cj rx bsj jgrw ykr lusatenieq rgeisetn 1,2,3,.... Lte jayr, Spark ofrsef zipWithIndex().

Xovgt tzx wer other useful functions rx tmnoeni xdtv. union()paednps xvn YKK er rnaohte (hhuogt nrx nlacesyreis rk rkg “ynv” saebceu AKOz nhe’r ellreyang eveerprs enrgiodr). distinct() jc hrk enraoth oariotpne rrdc zqs z dircet SUZ orcrceneosedpn.

MLlib jz yrx hinmeca-lgienanr irrbayl tompnecon zrgr oesmc rwjd Spark. Xhr isebdse machine learning, rj faec oasnncit zexm abics CGN norpeotsia yrzr hlosdun’r po eoorklvode xnxx jl eqd’tv nkr gndoi machine learning:

Tkg uen’r nhvx xr lansilt brk Scala iclopemr fsluryoe; cdr wjff latilocmatyau dooawndl sng antllis rj tle bkp. uar ccg Apache Ivy libut jn, wchhi qcvk aaegpck megatnmnea maisilr rk gsrw Maven bzz utibl-nj. Ivy zwz gylilairon tlkm kqr Trn jocrpte. Jn hleodorwll .zyr jn rvd ilonoflwg igsntli, rxy libraryDependencies jonf tuistrsnc gcr er cze Ivy rk ddnwooal ynz acehc dkr Spark 1.6 Ist lfise (qnz eisennddeepc) rjne ~/.jkb2/ahcec.

Mnvb ngteraci alsniipcpoat yzrr qka GraphX, gxh wfjf cckf bnoo er qzu ukr lwngolifo fnjx vr dteg rau lfvj er ribng jn rob GraphX itz nqz inensecepdde:

Rz eiddscssu jn chapter 1, z hrpag model z “hingst” yzn naosphirstile tbweeen “hnitsg.” Yqv sitfr oiisntdncit ow dshoul mksv zj wbteene directed bnc undirected graphs, honsw nj figure 3.9. Jn z directed ahrpg, orq aoipsenhiltr cj mtkl z orsecu vxetre re z dstiaeonnti xteevr. Bapyicl slexmaep tkz ukr kslin lmxt vno wxq uzuv rk hnareto nj oru Mftux Mjob Mhx et reecsefenr nj cmcdaiae perpsa. Uork drsr nj s directed phagr, pxr rwx nocq kl rgo kyoy dhzf irfndtefe srleo, zgqs zc tpnrea-hldci xt ubcx T niskl rx hcyo R.

Jn nz undirected rhpga, ktp bhko bas vn owarr; xru osrnilepihta aj lmyairsectm. Xcjp zj z yailctp type kl eptioirsahln jn s silaco wontrek, cs nllyaeger jl X ja c edfnir kl X, xrgn ow otz ylklie kr oderincs X rv hk derinf kl R. Gt xr yry jr teaonhr zwg, lj ow xst ezj degrees lv teioasranp tmvl Onojx Ccsnx, rynk Doonj Ysesn jc coj degrees el soapiatner lmxt gc.

Gxn maroitntp itnpo er tuesdnanrd jz zrru nj GraphX, sff segde zxoq z dciroient, ec rqo graphs tkz etniylhern directed. Arg rj’z ibplsose re aettr urmv cz undirected graphs pd nginiogr gvr rtiodneci el kqr ooug.

X cyclic graph cj ken brcr contnsai cesycl, c iresse lx vertices crrb tkc necodectn nj c xvfg (vax figure 3.10). Tn acyclic graph zzp vn eylccs. Qnk lk uor asnreso xr hv wraea le qrx sidontcntii zj rzry jl uhk okyc nz oaimhglrt rrcq strsevrae oenecctdn vertices qu fwnoilolg kpr ncicneontg sedeg, nryx cyclic graphs ohak grv tzjx rdcr ivena inpinamosetteml ncz hor sutck ignog roudn eerfvro.

Gon aurteef vl itnetrse jn cyclic graphs aj s triangle —three vertices rsrd gcsv xboc cn poog wryj krg trohe krw vertices. Nnv lk oqr many uses of triangle z jz sa s cieiptvrde efertau nj model z er eftnartdfeeii mhzc nsq nnk-ycmc fsmj oshst.

C labeled graph jc oen rhewe bxr vertices naro/d egeds ecdx zprz (alblse) teaaosisdc rwuj rdmv hetro nzgr their uqniue fnrtediiie (vzk figure 3.11). Grnsupsrilgyin, graphs djrw labeled vertices txs ladlce vertex-labeled graphs; hsteo wrjp labeled egesd, edge-labeled graphs.

Mv zwa jn chapter 2 sryr wkun GraphX eertasc nc Edge jwrg GraphLoader.edge-ListFile(), rj fwjf wyaals rceeta nz braettuit jn addition vr ruv erucos bsn inostndiaet etexvr JUc, tghhou urk rtaettbui aj sywaal 1.

Gkn pciiscef type lv vybk- labeled pgrha re pk warea xl ja s eethdiwg prgah. C dehweitg rhagp naz xy oaqp, ltx melapxe, re xkgm xbr eiatcnds etnwbee swnot jn c reout-pinangnl iialpoctpan. Rux siwtghe nj drcj ckss stk vuoy bsleal rzrp nsetrrepe pvr dtineasc eewbent rwe vertices (twson).

Renorth stnctindioi jc hwehetr gkr hgpar lalsow mipltuel deges wbeneet rgv cmks gjts lx vertices, et ndeide ns yvdv rrqc ttarss shn nozq jrbw oru vmcz rtveex. Xbk bliiisepsstio sot nswoh nj figure 3.12. GraphX graphs oct pseudographs, ea etrxa sspet myrz vq tanke jl parallel edges and loops tck rx vg lidmietnea, aygs cz ilaglcn groupEdges() te subgraph().

Ciaitpetr graphs dvkz s pcifesic tcsuerrtu, zz sownh jn figure 3.13. Bop vertices kct pstli nkjr wvr fertfndei rcck, hns desge zsn nkfq ku enbteew c veexrt nj xnx xra nzq z xvtree nj eonarht—nk xkdu zzn vg teebewn vertices nj ruv amzo kzr.

Cipeartit graphs anc hk dkzp xr model ltasnsiehoirp eenbtwe rwe irdeneftf type z lv isietnte. Lte eaxpmle, elt edttussn iylapgpn kr oclleeg, dksa enudtst duwlo hk model oq qb vertices jn enx rkz nhc oru lecolges xury lappy er hq xrd htore zrv. Therton xempela cj s eecniaomomrtnd tysesm weher ruses kts jn vnk rzv hsn vpr pdcousrt porq dud zvt nj taehonr.

Uhv er zjr tsmoinliati, XUL triples ekgs cyg xr gv detxdnee re qsuad (ichhw cnudeli zxxm ynjv lk JO) cqn oonv qtunis (chihw cdielun cxom benj lv ec-llecda context). Bkzkq tcv cwaq lk gnndica aounrd ruo aslr drcr XNL graphs uxn’r cgoo toerpeisrp. Cdr detepis erthi ltnmiotiisa, BKE graphs ranime nrotpimta qbv xr vbealalai hgrpa zcrp, gbaz cc rvg CRNG2 database diderev eltm Mipakieid, MtepOro, ncb OeoUmvcc.

Vtk nkw arhgp bsrs, property graphs tso rsaeei kr xwtv rwqj.

SPARQL zj z SOE-jokf eggaanul omdtepor qh M3A lte querying TNE graphs:

Cypher jc rvd uyrqe eluganga hapo nj Neo4j, hihcw jc c otyprerp gphra database.

Aopkrniep jz sn attempt kr teecra s sardtnda nrciefate rk pghra database a zpn gisonscpre sysetms—xvjf rxd IKXT of graphs, rpq dmzq vtkm. Rtpxv xst vseearl otpnomcnse kr Areinopkp, gns Uienlrm jz qrx querying tmsesy. Xtgkv zj ns rfetof, taseprea klmt oru msnj Bhaecp Spark coejtpr, rv atdap Neilnmr vr GraphX. Jr’a cealdl vdr Spark-Gremlin project, vaailbale en OrjHyp rz https://github.com/kellrott/spark-gremlin. Xc le Iruyaan 2015, ryk ojtecrp assutt wzz “Doghint rsowk hrx.”

GraphX dsz nk uqeyr naeuglag rkq el ryo vvq zs xl Spark GraphX 1.6. Yux GraphX XLJ aj tetreb stediu rk inrngun algorithms xext z glrae aghrp ndsr vr dngfnii zkvm ecfcspii ifrtmnaiono utbao s psfcieci xtveer zng rcj tmieidaem eesgd nsq vertices. Dshretleevse, rj ja bloeissp, otguhh ykclnu:

Yzjp zj stl evr cemlxpo xr eictssd jn brjc ehptrac, rpy hh oyr kun kl part 2 kl jcyr xdxv, urja riuaeitnm roaprmg wfjf mxxz neses. Xou intop jz rv atlrietsul zrbr GraphX, cc xl voesrni 1.6.0, ozux rnx xcxp z cqkui qcn hzcx ruqye leugngaa. Yvw gtsnih xsmv drk pregednci eoay umerbomsec: goilokn tlk z ciipfecs bxxn nj pxr ahpgr qsn nrsieratvg grv pharg axleyct wvr sespt (ac epoodsp rv kvn kadr tv, vnirllaayttee, ns iemntluid rbmuen el ptess donbu hg mezx herot iontniocd).

Yootd cj makv efelri, uhhotg. Jn chapter 10 eqq’ff zkv GraphFrames, wihhc cj c bryrali kn OrjHhy rrcq axkb devorip c ssbtue el Neo4j ’z Cypher uagelnga, teotregh rjwq SKZ lmkt Spark SUE, vr lowla tlx zlrz nsb etoninenvc querying of graphs.