Security [SOLVED]: Authentication for Spark standalone cluster

Security [SOLVED]: Authentication for Spark standalone cluster

Home Forums Security Security [SOLVED]: Authentication for Spark standalone cluster

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
  • #36732



    I have a standalone Spark cluster running on a remote server and I’m new to Spark. It appears that there’s no authentication scheme protecting the cluster master’s (7077) port by default. Anyone can just simply submit their own code to the cluster without any restrictions.

    The Spark documentation states that authentication is possible in stand-alone deploy mode using the spark.authenticate.secret parameter, but doesn’t really elaborate how exactly this should be used.

    Is it possible to use some sort of shared secret that would prevent any potential attacker from submitting tasks to the cluster? Can anyone explain how exactly that can be configured?



    Accepted AnswerAnswer

    there are 2 parts to enable support of authentication:

    1. setting the secret on the master an all the slaves
    2. using the same secret when submitting jobs to the cluster

    master and slaves

    on each server in your cluster, add the following config to conf/spark-defaults.conf:

    spark.authenticate.secret      SomeSecretKey

    submitting jobs

    when you initialize the spark context, you should add the same config to it as well, ie:

    val conf = new SparkConf()
          .set("spark.authenticate.secret", "SomeSecretKey")
    val sc = new SparkContext(conf)

    or if you are using SparkSession:

    val spark = SparkSession.builder()
        .conf("spark.authenticate.secret", "SomeSecretKey")

    Author: lev
    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.