Spark2.1.0——SparkContext初始化之Spark环境的创建

阅读指导:在《Spark2.1.0——SparkContext概述》一文中,曾经简单介绍了SparkEnv。本节内容将详细介绍SparkEnv的创建过程。

         在Spark中,凡是需要执行任务的地方就需要SparkEnv。在生产环境中,SparkEnv往往运行于不同节点的Executor中。但是由于local模式在本地执行的需要,因此在Driver本地的Executor也需要SparkEnv。SparkContext中创建SparkEnv的实现见代码清单1。

代码清单1  创建SparkEnv

    private[spark] val listenerBus = new LiveListenerBus(this)
    _jobProgressListener = new JobProgressListener(_conf)
    listenerBus.addListener(jobProgressListener)

    _env = createSparkEnv(_conf, isLocal, listenerBus)
    SparkEnv.set(_env)

因为SparkEnv内的很多组件都将向LiveListenerBus的事件队列中投递事件,所以在代码清单1中首先创建LiveListenerBus和JobProgressListener,然后将JobProgressListener添加到LiveListenerBus的监听器列表中,最后将LiveListenerBus通过SparkEnv的构造器传递给SparkEnv以及SparkEnv内部的组件。JobProgressListener继承自SparkListener,LiveListenerBus和SparkListener的详细内容已在《Spark2.1.0之源码分析——事件总线》及其系列文章中介绍,此处不再赘述。

         createDriverEnv方法用于创建SparkEnv,根据createDriverEnv这个方法名,我们知道此方法将为Driver实例创建SparkEnv。调用createSparkEnv方法创建完SparkEnv后,SparkEnv实例的引用将通过SparkEnv的set方法设置到SparkEnv伴生对象[1]的env属性中,类似于设置为Java类的静态属性,这将便于在任何需要SparkEnv的地方,通过伴生对象的get方法获取SparkEnv。

         createSparkEnv方法创建SparkEnv的代码如下:

  private[spark] def createSparkEnv(
      conf: SparkConf,
      isLocal: Boolean,
      listenerBus: LiveListenerBus): SparkEnv = {
    SparkEnv.createDriverEnv(conf, isLocal, listenerBus, SparkContext.numDriverCores(master))
  }

可以看到实际调用了SparkEnv的createDriverEnv方法来创建SparkEnv。SparkEnv的createDriverEnv方法的实现如下:

  private[spark] def createDriverEnv(
      conf: SparkConf,
      isLocal: Boolean,
      listenerBus: LiveListenerBus,
      numCores: Int,
      mockOutputCommitCoordinator: Option[OutputCommitCoordinator] = None): SparkEnv = {
    assert(conf.contains(DRIVER_HOST_ADDRESS),
      s"${DRIVER_HOST_ADDRESS.key} is not set on the driver!")
    assert(conf.contains("spark.driver.port"), "spark.driver.port is not set on the driver!")
    val bindAddress = conf.get(DRIVER_BIND_ADDRESS)
    val advertiseAddress = conf.get(DRIVER_HOST_ADDRESS)
    val port = conf.get("spark.driver.port").toInt
    val ioEncryptionKey = if (conf.get(IO_ENCRYPTION_ENABLED)) {
      Some(CryptoStreamUtils.createKey(conf))
    } else {
      None
    }
    create(
      conf,
      SparkContext.DRIVER_IDENTIFIER,
      bindAddress,
      advertiseAddress,
      port,
      isLocal,
      numCores,
      ioEncryptionKey,
      listenerBus = listenerBus,
      mockOutputCommitCoordinator = mockOutputCommitCoordinator
    )
  }

createDriverEnv方法首先从SparkConf中获取4个属性。

  • bindAddress:Driver实例的host。此属性通过从SparkConf中获取DRIVER_BIND_ADDRESS指定的属性值。DRIVER_BIND_ADDRESS的定义如下:
  private[spark] val DRIVER_HOST_ADDRESS = ConfigBuilder("spark.driver.host")
    .doc("Address of driver endpoints.")
    .stringConf
    .createWithDefault(Utils.localHostName())

  private[spark] val DRIVER_BIND_ADDRESS = ConfigBuilder("spark.driver.bindAddress")
    .doc("Address where to bind network listen sockets on the driver.")
    .fallbackConf(DRIVER_HOST_ADDRESS)

根据DRIVER_BIND_ADDRESS的定义,说明按照优先级从高到低,可以通过spark.driver.bindAddress属性、spark.driver.host属性以及调用Utils的localHostName方法获得bindAddress。Utils的localHostName方法的实现请参阅附录A

  • advertiseAddress:Driver实例对外宣称的host。可以通过spark.driver.host属性或者Utils的localHostName方法获得。
  • port:Driver实例的端口,可以通过spark.driver.port属性指定。
  • ioEncryptionKey:I/O加密的密钥。当spark.io.encryption.enabled属性为true时,调用CryptoStreamUtils的createKey方法创建密钥。

真正创建SparkEnv的实现都在create方法中,由于SparkEnv是Driver和Executor实例中都有的组件,本书将在第5章对SparkEnv作详细介绍。


[1] 什么是伴生对象?这是Scala语言的语法特性。Scala中的伴生对象可以提供类似于Java中的静态方法、静态属性等的定义,也可以支持类似于Java中的main方法。更多内容请读者阅读Scala语言相关资料。

发布了99 篇原创文章 · 获赞 293 · 访问量 87万+
展开阅读全文

spark sparkcontext 初始化失败

05-22

环境 Ubuntu 16.04 hadoop 2.7.3 scala 2.11.8 spark 2.1.0 已经安装好了hadoop scala,之后配置了下 spark 运行 spark-shell 就爆出来下面的错误 ``` 18/05/22 15:43:30 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: For input string: "true #是否记录Spark事件,用于应用程序在完成后重构webUI" at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290) at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260) at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:29) at org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:407) at org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:407) at scala.Option.map(Option.scala:146) at org.apache.spark.SparkConf.getBoolean(SparkConf.scala:407) at org.apache.spark.SparkContext.isEventLogEnabled(SparkContext.scala:238) at org.apache.spark.SparkContext.<init>(SparkContext.scala:407) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95) at $line3.$read$$iw$$iw.<init>(<console>:15) at $line3.$read$$iw.<init>(<console>:42) at $line3.$read.<init>(<console>:44) at $line3.$read$.<init>(<console>:48) at $line3.$read$.<clinit>(<console>) at $line3.$eval$.$print$lzycompute(<console>:7) at $line3.$eval$.$print(<console>:6) at $line3.$eval.$print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638) at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637) at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19) at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38) at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214) at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37) at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) at org.apache.spark.repl.Main$.doMain(Main.scala:68) at org.apache.spark.repl.Main$.main(Main.scala:51) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) java.lang.IllegalArgumentException: For input string: "true #是否记录Spark事件,用于应用程序在完成后重构webUI" at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290) at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260) at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:29) at org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:407) at org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:407) at scala.Option.map(Option.scala:146) at org.apache.spark.SparkConf.getBoolean(SparkConf.scala:407) at org.apache.spark.SparkContext.isEventLogEnabled(SparkContext.scala:238) at org.apache.spark.SparkContext.<init>(SparkContext.scala:407) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95) ... 47 elided <console>:14: error: not found: value spark import spark.implicits._ ^ <console>:14: error: not found: value spark import spark.sql ``` 问答

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 点我我会动 设计师: 上身试试

分享到微信朋友圈

×

扫一扫,手机浏览