Trending questions in Apache Spark

0 votes
1 answer

Is it better to have one large parquet file or lots of smaller parquet files?

Ideally, you would use snappy compression (default) ...READ MORE

May 23, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 14,595 views
0 votes
1 answer

How to prevent executor from self-destructing?

I think there is a timeout set ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 1,928 views
0 votes
1 answer

How to enable write ahead logs for receivers?

To enable write-ahead logs, run the following ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,587 views
0 votes
1 answer

Set Library to launch Yarn master

You can make use of Special Library path to ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,205 views
0 votes
1 answer

Installing Spark on Ubuntu

Hey. Follow these steps to install Spark ...READ MORE

Feb 20, 2019 in Apache Spark by Omkar
 • 69,180 points • 2,737 views
0 votes
1 answer

How to set stage abort in Spark?

You can do it as follows. Use ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 1,843 views
0 votes
1 answer

Need to disable unpersist in Spark

You can dynamically change this function by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,517 views
0 votes
1 answer

Getting "buffer limit exceeded" exception inside Kryo.

Seems like the object being sent for ...READ MORE

Mar 7, 2019 in Apache Spark by Pavitra
 • 2,026 views
0 votes
1 answer

How to automatically kill executors on blacklisting?

You can set the property to directly ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 1,816 views
0 votes
1 answer

Can I set different protocol for SSL in Spark?

There is no protocol set by default. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,678 views
0 votes
1 answer

How enable XSS protection for Spark application?

Spark has a built-in prevention system against XSS. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,662 views
0 votes
1 answer

Spark context (sc) not found

Maybe the hadoop service didn't start properly. Try ...READ MORE

Feb 14, 2019 in Apache Spark by John
 • 2,938 views
0 votes
1 answer

Get Spark SQL configuration in Java

You will need to use Spark session ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 1,481 views
+1 vote
1 answer

Facing out-of-memory errors in Spark driver

I am guessing that the configuration set ...READ MORE

Feb 23, 2019 in Apache Spark by Rishab
 • 2,447 views
0 votes
1 answer

Sliding function in spark

The sliding function is used when you ...READ MORE

Jan 29, 2019 in Apache Spark by Omkar
 • 69,180 points • 3,511 views
0 votes
1 answer

Using R to display configuration of Spark SQL

Try the below-mentioned code. sparkR.session() properties <- sql("SET -v") showDF(properties, ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 1,433 views
0 votes
1 answer

Network encryption for Spark

You can enable encryption for the Spark ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,640 views
0 votes
1 answer

How to spread applications to less nodes?

You can limit the spread out by ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,108 views
0 votes
1 answer

How to change block manager port for driver only?

Yes, it is possible to change the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 1,772 views
0 votes
1 answer

How can I run a blacklisted task?

When a task results in too many ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 1,691 views
0 votes
1 answer

Increasing retry before blacklisting executor

You can do it like this: val sc ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 1,670 views
0 votes
1 answer

How to check if user has permission in Web UI?

You can implement this as follows: First, add ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 1,533 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

Mar 25, 2019 in Apache Spark by hari
 • 1,059 views
0 votes
1 answer

How to make Spark skip node locality?

If you set the node wait time ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 1,654 views
0 votes
1 answer

Change number of completed drivers displayed

You can change the number of completed ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,049 views
0 votes
1 answer

Change heart beat time to R process

The heartbeat interval is assigned to the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,290 views
0 votes
1 answer

Not able to clone Hadoop configuration.

Run the following command in Spark shell ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 1,704 views
0 votes
1 answer

Retaining the batch by status APIs before Garbage Collection

By default, 1000 batches are retained by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,284 views
0 votes
1 answer

Configuring location of credential provider

To configure the location of the credential ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,451 views
0 votes
1 answer

How to increase Spark memory for execution?

Probably the spill is because you have ...READ MORE

Mar 7, 2019 in Apache Spark by Pavitra
edited Mar 8, 2019 • 1,791 views
0 votes
1 answer

Password for keystore in Spark

You can dynamically set a password to ...READ MORE

Mar 15, 2019 in Apache Spark by karan
 • 1,427 views
0 votes
1 answer

How to enable SASL encryption?

You can use dynamic configuration setting to ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 1,461 views
0 votes
1 answer

Spark logs not overwriting

Spark does not allow you to overwrite ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 1,800 views
0 votes
1 answer

Error reading avro dataset in spark

For avro, you need to download and ...READ MORE

Feb 4, 2019 in Apache Spark by Omkar
 • 69,180 points • 3,069 views
0 votes
1 answer

Change number of threads use by R back end

Refer to the below commands to know ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,198 views
0 votes
1 answer

How many executors run when dynamic allocation is enabled?

The number of executors running by default ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,447 views
0 votes
1 answer

Set maximum reading rate from Kafka partition

You can do it by using the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,170 views
0 votes
1 answer

How to monitor interrupted Spark tasks?

To enable monitoring interrupted tasks, run the following ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 1,467 views
0 votes
1 answer

How to limit the cores being used by a cluster?

You can set the maximum number of ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 1,499 views
0 votes
1 answer

How to set maximum receiving rate for backpressure mechanism?

You can set the maximum receiving rate ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 1,195 views
0 votes
1 answer

Not able to preserve shuffle files in Spark

You lose the files because by default, ...READ MORE

Feb 24, 2019 in Apache Spark by Rana
 • 2,153 views
0 votes
1 answer

How to change encryption key value for local I/O?

There another property where you can set ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 1,320 views
0 votes
1 answer

Authenticate internal connections in Spark

To make Spark authenticate internal connections, you ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,347 views
0 votes
1 answer

Loading Spark properties dynamically

First, create an empty conf using this ...READ MORE

Feb 22, 2019 in Apache Spark by Mansoor
 • 2,160 views
0 votes
1 answer

Multidimensional Array in Scala

Multidimensional array is an array which store ...READ MORE

Feb 11, 2019 in Apache Spark by Omkar
 • 69,180 points • 2,605 views
0 votes
1 answer

How to give modify access to group?

You can do this using the following ...READ MORE

Mar 15, 2019 in Apache Spark by Vikas
 • 1,216 views
0 votes
1 answer

Dynamic allocation: Set minimum number of executor

You cans et it dynamically like this:  val ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,285 views
0 votes
1 answer

Log every block update in Spark

By default, Spark does not log all ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 1,586 views
0 votes
1 answer

Setting port for SSL for Spark application

You can set the port in the ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,195 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

Mar 15, 2019 in Apache Spark by Raj
 • 1,194 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.