Hello,
I have been trying to read hive tables by invoking HiveContext within SparkContext.
The data is being read from hive tables,However when I try to connect it to Solr external zookeeper server from Hadoop edge node,I am facing an issue.
The same code works when I try to push sample json from SparkContext to Solr using same zookeeper server.I am using lucidworks spark-solr:2.1.0 package to do this. End goal is to push parquet files to Solr directly.
Please find the error below.
17/02/22 20:14:17 ERROR ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.
17/02/22 20:14:17 WARN ConnectionManager: zkClient received AuthFailed
17/02/22 20:14:17 WARN SolrQuerySupport: Can't get uniqueKey for testspark due to: com.google.common.util.concurrent.UncheckedExecutionException: org.apache.solr.common.cloud.ZooKeeperException:
17/02/22 20:14:17 ERROR ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.
17/02/22 20:14:17 WARN ConnectionManager: zkClient received AuthFailed
Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: org.apache.solr.common.cloud.ZooKeeperException:
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:93)
at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:97)
at com.lucidworks.spark.util.SolrRelationUtil$.getBaseSchema(SolrRelationUtil.scala:34)
at com.lucidworks.spark.SolrRelation.<init>(SolrRelation.scala:83)
at solr.DefaultSource.createRelation(DefaultSource.scala:26)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at com.cisco.SparkSolr.main(SparkSolr.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:742)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.solr.common.cloud.ZooKeeperException:
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:475)
at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:83)
at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:89)
at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:38)
at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:36)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
... 20 more
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /clusterstate.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:289)
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:467)
... 28 more
hdsvmg@hddev-c01-edge-01:/users/hdsvmg>
I have been trying to read hive tables by invoking HiveContext within SparkContext.
The data is being read from hive tables,However when I try to connect it to Solr external zookeeper server from Hadoop edge node,I am facing an issue.
The same code works when I try to push sample json from SparkContext to Solr using same zookeeper server.I am using lucidworks spark-solr:2.1.0 package to do this. End goal is to push parquet files to Solr directly.
Please find the error below.
17/02/22 20:14:17 ERROR ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.
17/02/22 20:14:17 WARN ConnectionManager: zkClient received AuthFailed
17/02/22 20:14:17 WARN SolrQuerySupport: Can't get uniqueKey for testspark due to: com.google.common.util.concurrent.UncheckedExecutionException: org.apache.solr.common.cloud.ZooKeeperException:
17/02/22 20:14:17 ERROR ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.
17/02/22 20:14:17 WARN ConnectionManager: zkClient received AuthFailed
Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: org.apache.solr.common.cloud.ZooKeeperException:
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:93)
at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:97)
at com.lucidworks.spark.util.SolrRelationUtil$.getBaseSchema(SolrRelationUtil.scala:34)
at com.lucidworks.spark.SolrRelation.<init>(SolrRelation.scala:83)
at solr.DefaultSource.createRelation(DefaultSource.scala:26)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at com.cisco.SparkSolr.main(SparkSolr.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:742)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.solr.common.cloud.ZooKeeperException:
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:475)
at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:83)
at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:89)
at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:38)
at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:36)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
... 20 more
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /clusterstate.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:289)
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:467)
... 28 more
hdsvmg@hddev-c01-edge-01:/users/hdsvmg>