Unable to load elasticsearch class

I am trying to run this code:

            var es_reader = spark.Read()
                            .Format("org.elasticsearch.spark.sql")
                            .Option("inferSchema", "true")
                            .Option("es.read.field.as.array.include", "tags")
                            .Option("es.nodes", "myelasticsearch:9200")
                            .Option("es.net.http.auth.user", "elastic")
                            .Option("es.net.http.auth.pass", "mypass");

                           var security_df = es_reader.Load("my-index/");

[JvmBridge] java.lang.ClassNotFoundException: Failed to find data source: org.elasticsearch.spark.sql. Please find packages at http://spark.apache.org/third-party-projects.html

This same code is working in Jupyter. I have no clue what’s going on… Can someone help me pls

[GoEddie]

resource must be specified for Elasticsearch resources

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#_required_settings

You must add the option es.resource ??

[mohanwd]

Hi, Folks I am also facing the similar issue. But one small difference is if i add the dependency in pyspark --packages org.elasticsearch:elasticsearch-hadoop:7.7.1 I am able to establish a connection / its trying to connect to the es cluster. Whereas with spark-submit i could not solve the class not found issue tried with hadoop / spark jars both