{"id":150,"date":"2022-06-01T19:23:10","date_gmt":"2022-06-01T19:23:10","guid":{"rendered":"https:\/\/kindsonthegenius.com\/apache-spark\/?p=150"},"modified":"2022-06-01T19:23:10","modified_gmt":"2022-06-01T19:23:10","slug":"spark-setup-in-intellij-with-scala","status":"publish","type":"post","link":"https:\/\/www.kindsonthegenius.com\/apache-spark\/spark-setup-in-intellij-with-scala\/","title":{"rendered":"Spark – Setup in IntelliJ With Scala"},"content":{"rendered":"

In this tutorial, you will learn how to setup Spark to run in IntelliJ with Scala. Then we would create a simple HelloWorld application in IntelliJ.<\/p>\n

 <\/p>\n

Create a Spark Project and Add Dependencies<\/strong><\/h4>\n

Step 1<\/strong> – First you need to setup Scala with IntelliJ. See the steps here<\/a>.<\/p>\n

Step 2<\/strong> – Create a new Scala Project in IntelliJ<\/p>\n

Step 3<\/strong> – Add Spark dependencies:<\/p>\n

Open the build.sbt file and add the Spark Core and Spark SQL and Streaming dependencies.<\/p>\n

the complete content of my build.sbt file is shown below<\/p>\n

name := \"FirstProg\"\r\n\r\nversion := \"0.1\"\r\n\r\nscalaVersion := \"2.13.8\"\r\n\r\nlibraryDependencies ++= Seq(\r\n  \"org.apache.spark\" %% \"spark-core\" % \"3.2.1\",\r\n  \"org.apache.spark\" %% \"spark-sql\" % \"3.2.1\",\r\n  \"org.apache.spark\" %% \"spark-streaming\" % \"3.2.1\",\r\n)<\/pre>\n
You’ll need to refresh the sbt to download the dependencies.<\/p>\n
 <\/p>\n
Create an Application<\/strong><\/h4>\n
Create a new Scala Class in IntelliJ. Choose the type to be Object.<\/p>\n
The content of this file is as follows
\n<\/p>\n
package<\/span> org.kindsonthegenius<\/span>\r\n\r\nimport<\/span> org.apache.spark.sql.SparkSession<\/span>\r\n\r\nobject<\/span> SparkSessionTest<\/span> extends<\/span> App<\/span>{<\/span>\r\n  val<\/span> spark =<\/span> SparkSession<\/span>.<\/span>builder()<\/span>\r\n    .<\/span>master(<\/span>\"local[1]\"<\/span>)<\/span>\r\n    .<\/span>appName(<\/span>\"Spark Tutorial by Kindson\"<\/span>)<\/span>\r\n    .<\/span>getOrCreate()<\/span>\r\n\r\n  println(<\/span>\"************** Your First SparkContext ***************\"<\/span>)<\/span>\r\n  println(<\/span>\"App Name - \"<\/span> +<\/span> spark.<\/span>sparkContext.<\/span>appName)<\/span>\r\n  println(<\/span>\"Deploy Mode - \"<\/span> +<\/span> spark.<\/span>sparkContext.<\/span>deployMode)<\/span>\r\n  println(<\/span>\"Master - \"<\/span> +<\/span>spark.<\/span>sparkContext.<\/span>master)<\/span>\r\n\r\n  val<\/span> sparkSession2 =<\/span> SparkSession<\/span>\r\n    .<\/span>builder.<\/span>master(<\/span>\"local[1]\"<\/span>)<\/span>\r\n    .<\/span>appName(<\/span>\"Spark Tutorial by Kindson\"<\/span>)<\/span>\r\n    .<\/span>getOrCreate\r\n\r\n  println(<\/span>\"************* Your Second Spark Context ***************\"<\/span>)<\/span>\r\n  println(<\/span>\"APP Name - \"<\/span> +<\/span> sparkSession2.<\/span>sparkContext.<\/span>appName)<\/span>\r\n  println(<\/span>\"Deploy Mode - \"<\/span> +<\/span> sparkSession2.<\/span>sparkContext.<\/span>deployMode)<\/span>\r\n  println(<\/span>\"Master - \"<\/span> +<\/span> sparkSession2.<\/span>sparkContext.<\/span>master)<\/span>\r\n}<\/span>\r\n<\/pre>\n
 <\/p>\n
You can now run the program and you will have the output shown below<\/p>\n
\"Output
Output of Spark Program<\/figcaption><\/figure>\n
 <\/p>\n
I recommend you watch the video if you have any challenges. You can also leave me a comment below.<\/p>\n","protected":false},"excerpt":{"rendered":"
In this tutorial, you will learn how to setup Spark to run in IntelliJ with Scala. Then we would create a simple HelloWorld application in … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,3],"tags":[36],"class_list":["post-150","post","type-post","status-publish","format-standard","hentry","category-pyspark","category-spark","tag-sparkcontext"],"_links":{"self":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts\/150","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/comments?post=150"}],"version-history":[{"count":2,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts\/150\/revisions"}],"predecessor-version":[{"id":158,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts\/150\/revisions\/158"}],"wp:attachment":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/media?parent=150"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/categories?post=150"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/tags?post=150"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}