I have a standalone Spark cluster running on a remote server and I’m new to Spark. It appears that there’s no authentication scheme protecting the cluster master’s (7077) port by default. Anyone can just simply submit their own code to the cluster without any restrictions.
The Spark documentation states that authentication is possible in stand-alone deploy mode using the spark.authenticate.secret parameter, but doesn’t really elaborate how exactly this should be used.
Is it possible to use some sort of shared secret that would prevent any potential attacker from submitting tasks to the cluster? Can anyone explain how exactly that can be configured?