无状态与有状态

简单来说，无状态就是每个采集周期分别采集，并不会把前面的采集周期的数据一起计算

有状态就是：把前面采集周期的也算进来，

比如wordcount，无状态统计的就是每个采集周期内的个数，有状态的话是统计所有采集周期内的个数。

有状态就是把前面的采集周期采集的数据存到缓存中，想要安全一些就设置检查点存储到磁盘，然后当前的DStream去和磁盘交互，一起统计出来。

package date_10_17_SparkStreamingimport org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Duration, Seconds, StreamingContext}object upState {def main(args: Array[String]): Unit = {val conf = new SparkConf().setAppName("wordCount").setMaster("local[*]")val streamingContext = new StreamingContext(conf,Seconds(5))streamingContext.checkpoint("cp")//连接kafkaval kafkaStream = KafkaUtils.createStream(streamingContext,"chun1:2181","chun",Map("chun"->3))//wordcount运算val mapDStream = kafkaStream.flatMap(_._2.split(" ")).map((_,1))//有状态val resultDStream:DStream[(String,Int)] = mapDStream.updateStateByKey {case (seq, buffer) => {val sum = buffer.getOrElse(0) + seq.sumOption(sum)}}resultDStream.print()//启动采集器streamingContext.start()//等待采集器关闭才关闭DriverstreamingContext.awaitTermination()}
}