Spark实验报告.docx
Spark报金航1510122526SPark试验报告-、环境搭建1、卜载版本卜载地址为：2、解压和安装：解压：tar-xvfscala-2.11.4.tgz安装：mvscala-2.11.4opV3、m-.bashprofile文件增加SCA1.AHoME环境变量配置,exportexportC1.ASSPATH=.：SJAVA_HOME/jre),lib:$JAVA_HOME/lib:$JAVA_HOME.'lib.aools.jarPATH=SPATHSHOMEBn:$JAVAjHoME，bin:$SCA1.A_HOMEybin马上生效source/.bash_profile4、验证SCala：scala-version5、CoPy到SIaVe机器scp/.bash_PrOfiIe:Ybashjxofile6. bspark.wget7、在master主机配置SPark：格下校的SPark-I2.0-bin-hadoop2.4.tgz解压到TOPt/即，配置环境变量SPARKJHOME# setjavaenvexportC1.ASSPATH-.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lih'tTOls.jarPATH=$PATH:$HOME.'，bin:$JAVA_HOME.'bin:$SCA1.A_HOME/bin：SSPARK_HOMEbinSHADOOPJHOMEKbin配置完成后运用source吩咐使配置生效进入SParkCOnf书目：sparkSlPA11opt$cd(sparkSlPA11spark-1.2.0-bin-hadoop2,4SIsbinconfdataec2exampleslib1.ICENSElogsNOTICEpythonREADME,mdRE1.EASESbinworksparkS1PA11spark-1.2.0-bin-hadoop2.4Scdnfspark(三)SlPA11conf)$Issaves.templatespark-env.shIog4j.properties.templateslavesspark-spark-first：修改SlaVeS文件,增加两个SlaVe节点SIPA11、S1PA222sparkS1PA11conf$vislavesS1PA11S1PA222second：配置spark-env.sh首先把sark-env.sh.temlatecopyspark-env.shvispark-env.sh文件在最下面增加：exportexportSPARKWORKERMEMORY=2ghadpHADoOP_CONF_DiR是Hadoop配置文件书目，SPARK_MASTEFUP主机IP地址,SPARKWORKER.MEMORY是WOrker运用的最大内存完成配世后，将sparkHIlcopyslave机器scp-r# oplpark-l.2.0-bin-hadoop2.4:沁PV8、启动spark分布式集群并查看信息ISparkSIPA11sbin$./start-all.sh查看:(spark<g>SlPA11sbnSjps31233ResourceManager27201Jps30498NameNode30733SecondaryNameNode5648Worker5399Master15888JobHistoryServer假如HDFS没有启动，启动起来.# slave节点：(sparkSlPA222scala$jps20352Bootstrap30737NodeManager7219Jps30482DataNode29500Bootstrap757Worker9、页面查看集群状况：进去spark集群的Web管理页面，访问×WSpariiKUiteratpaZ10.M.4447:707741.ra0Mi(MM4ZVn*f*v?CC*-OUMCM*r40-GgOtMnBeCyMa。C*v*WCCMw9ICu9M*AM因为我们看到两个worker节点，因为master和slave都是WOrker节点我们进入spark的bin书Il，肩动SPark-shell限制台1SOVM14:17:30XXTOMtt.Btt2cckTcMfr*mo:Srrrcxfloa3“01IV01i(B112O口2M«)f*.Sl<MUteMHHiMUctT111.mWBlMknr9(1SM¼14:0:20XlTS>tot*.al9Utaa9*MM*cXctt:V*qi>t<siagfelockaat49tlocal三t:$?!itliS.,>DRAM.BlMttai9c4<dti*vs>.Isc*lb3t,S72)120l4<14:17:20XMTO¾r9.ftl>eUtaMrtMr:叼1tZU0ctMt9r1WO14<14P(2OWWrt*l.f<rtU<SZE9filM<4t.ec*txt*il4tla*»c.访问,我们可以看到SParkWEBUl页面WorRv*>OW¼QOf4PVUIqoredsTV1.temtat,64Uto，M”，4t*u<MA*MAMaMT>UUIU0JAU3(O)<ISP0MNWWWUM三X*-»*3ft*MI.M4*Wv一»，OMtoMMA)I»«*«»OA4wPi，awz»vv4aMt(qo111»JoJ)88msV®dsJobsStagesStorageEnvironmentExecutorsExecutors(1)Memory:OOBUsed(2650MBTotal)Disk:OOBUsedExecutorIDAddressRDDBlocksMemoryUsedDiskUsedActiveTasksFailedTasksCompleteTasksTotalTasks<driver>localhost5740100.0B/2650MBOOB0000Spark120SPark集群环境搭建胜利了10.isfjspark-shell测试之前我们在tmp书目上住了一个README文件，我们现在就用spark读取hdfs中README.txt文件tmpPermissionOwnerGroup-W-sparksupergroupdrwxsparksupergrouphttp:drwxr-xr-xsparksupergroupHadoop.2014.取得hdfs文件:15/01/06I5IOCIS/0X/04l<ctoril»c.textrilIebdf*:/SlVAll:50Wtw>*,BXt>f)Ki«f3214U913214:29:32X«：25:32H29MXMFOtoc.loryflVoctXNFO3"9.MMMr°"r,XMFOat0ra9.Maaor4tor:XKFOatocagv.M3gyflt0t:eour<Fx<e(17(124)ZlXithCmmPBlockferM4al.0一“valwUMeo(l*atdla172.0Rrr344.1MD)m4uFxmJf*o(23U4>call*dVldCUrMsl"X".¾>三M-2772<>Blockbxt>4c>t-0-pcOaor*da»byt«»i»M*9xystina%dais22.».txw2,4.0MBIIMFOVt7.>lorhMr>>y<fInfot*Mvdb>t>*t,inonloolholf49211(it22.8few(2“9MBIOftI412t22INFOcor9.BloclMn*r>Ur<Up4tdIntoOfb：o«ii"o4caj0jSx0OC14:29:32XNrO«p«rk.9paxk3c%xt:CratdbrAdcatOfxc三txt11leat<cool>x12Fil!oxg.Ach.n>rkr4d.RDD(Striae)Mf:/SlVAll1.txtNpf>>020111ttt11la<cv*ol>l2count下READM.txt文件中文字总数,FI：三三鲁WW,:HI我们过渡README.txt包括The单词有多个scala>vaxtheCount=readmeFile.filter(line=>line.contains("The*)theCount:org.apache.spark.rdd.RODString)5三：rilteredRDD(3tfilterat<conaole>我们算出来一共有4个The单词(sparkSlPA222hadoop-2.6.0$IsNOTICE.txtREADMspark81PA222hadoop-2.6.0)$grepTheREADME.txtwc437269_h1.1.pi.h;nAoh1.nC1-IWkSUI1UHOspark0S!PA222hadoop-2.6.0$我们通过WC也罚出来有4个The单词我们再实现下HadOoPWordCoUnt功能：首先对读取的readmeFile执行以卜吩咐:cala>valWordCOUnt=readmeFile.fIatMap(line=>line.split("").map(word=wordcount:org.apache.spark.rdd.RDD(StringrInt)=ShuffledRDD(6atred',hlip:>!(>.cs<ln.notst<rks11m<'i3cala>I其次运用llect吩咐提交并执行job:0"0UtMo<is：m010CISiM/¢1/0(IStMoci»：mocis：M104ismoeIStMi0C13:N,0"06ISiM12IWTO*rrh.VF4r*Co4vtt0ri*gjobrC91Xett<conol>tl?1V010<IStMl»/01/0C1S:M1VO)OCISiM<2ZMFOvtorg.NM>ry0tor*s
