java - Unable to broadcast a large ConcurrentHashMap in Spark 1.5.0 -
i using spark 1.5.0 cloudera distribution , in java code trying broadcast concurrent hashmap. in map() function, when try read broadcast variable, nullpointer exception in resource manager logs. can please me out? unable find resolution this. following code snippet:
// broadcasting before calling mapper final broadcast<concurrenthashmap<constantkeys, object>> constantmapfinal = context.broadcast(constantmap);
.......
// in map function javardd<string> outputrdd = temprdd.map(new org.apache.spark.api.java.function.function() { private static final long serialversionuid = 6104325309455195113l; public object call(final object arg0) throws **exception { concurrenthashmap<constantkeys, object> constantmap = constantmapfinal.value(); // line 428 } });
the exception resource manager logs:
016-11-17 10:40:10 error applicationmaster:96 - user class threw exception: org.apache.spark.sparkexception: job aborted due stage failure: task 1 in stage 2.0 failed 4 times, recent failure: lost task 1.3 in stage 2.0 (tid 20, ******(server name)): java.io.ioexception: java.lang.nullpointerexception @ org.apache.spark.util.utils$.tryorioexception(utils.scala:1177) @ org.apache.spark.broadcast.torrentbroadcast.readbroadcastblock(torrentbroadcast.scala:165) @ org.apache.spark.broadcast.torrentbroadcast._value$lzycompute(torrentbroadcast.scala:64) @ org.apache.spark.broadcast.torrentbroadcast._value(torrentbroadcast.scala:64) @ org.apache.spark.broadcast.torrentbroadcast.getvalue(torrentbroadcast.scala:88) @ org.apache.spark.broadcast.broadcast.value(broadcast.scala:70) @ com.***.text.fuzzymatch.execute.fuzzymatchwrapper$2.call(fuzzymatchwrapper.java:428) @ org.apache.spark.api.java.javapairrdd$$anonfun$toscalafunction$1.apply(javapairrdd.scala:1027) @ scala.collection.iterator$$anon$11.next(iterator.scala:328) @ scala.collection.iterator$$anon$11.next(iterator.scala:328) @ org.apache.spark.rdd.pairrddfunctions$$anonfun$saveashadoopdataset$1$$anonfun$13$$anonfun$apply$6.apply$mcv$sp(pairrddfunctions.scala:1109) @ org.apache.spark.rdd.pairrddfunctions$$anonfun$saveashadoopdataset$1$$anonfun$13$$anonfun$apply$6.apply(pairrddfunctions.scala:1108) @ org.apache.spark.rdd.pairrddfunctions$$anonfun$saveashadoopdataset$1$$anonfun$13$$anonfun$apply$6.apply(pairrddfunctions.scala:1108) @ org.apache.spark.util.utils$.trywithsafefinally(utils.scala:1205) @ org.apache.spark.rdd.pairrddfunctions$$anonfun$saveashadoopdataset$1$$anonfun$13.apply(pairrddfunctions.scala:1116) @ org.apache.spark.rdd.pairrddfunctions$$anonfun$saveashadoopdataset$1$$anonfun$13.apply(pairrddfunctions.scala:1095) @ org.apache.spark.scheduler.resulttask.runtask(resulttask.scala:66) @ org.apache.spark.scheduler.task.run(task.scala:88) @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:214) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:745) caused by: java.lang.nullpointerexception @ java.util.hashmap.put(hashmap.java:493) @ com.esotericsoftware.kryo.serializers.mapserializer.read(mapserializer.java:135) @ com.esotericsoftware.kryo.serializers.mapserializer.read(mapserializer.java:17) @ com.esotericsoftware.kryo.kryo.readclassandobject(kryo.java:729) @ com.esotericsoftware.kryo.serializers.mapserializer.read(mapserializer.java:134) @ com.esotericsoftware.kryo.serializers.mapserializer.read(mapserializer.java:17) @ com.esotericsoftware.kryo.kryo.readclassandobject(kryo.java:729) @ org.apache.spark.serializer.kryodeserializationstream.readobject(kryoserializer.scala:192) @ org.apache.spark.broadcast.torrentbroadcast$.unblockifyobject(torrentbroadcast.scala:217) @ org.apache.spark.broadcast.torrentbroadcast$$anonfun$readbroadcastblock$1.apply(torrentbroadcast.scala:178) @ org.apache.spark.util.utils$.tryorioexception(utils.scala:1174) ... 21 more
this working smaller sizes of map. map can contain numerous key value pairs based on input request. can please me out?
Comments
Post a Comment