{"id":146,"date":"2022-05-31T21:48:44","date_gmt":"2022-05-31T21:48:44","guid":{"rendered":"https:\/\/kindsonthegenius.com\/apache-spark\/?p=146"},"modified":"2022-06-01T17:03:53","modified_gmt":"2022-06-01T17:03:53","slug":"spark-installation-and-setup","status":"publish","type":"post","link":"https:\/\/www.kindsonthegenius.com\/apache-spark\/spark-installation-and-setup\/","title":{"rendered":"Spark – Installation and Setup"},"content":{"rendered":"

In this tutorial, you will learn how to install Apache Spark on Mac and Windows.<\/p>\n

We would take to following steps<\/p>\n

    \n
  1. Install Homebrew<\/a><\/li>\n
  2. Install Java<\/a><\/li>\n
  3. Install Scala<\/a><\/li>\n
  4. Install Apache Spark<\/a><\/li>\n
  5. Verify Apache Spark Installation<\/a><\/li>\n
  6. Install Spark on Windows<\/a><\/li>\n<\/ol>\n

     <\/p>\n

    We’ll first go through Spark installation on MacOS. Next we go through the Windows installation as well.<\/p>\n

     <\/p>\n

    1. Install Homebrew<\/strong><\/h4>\n

    Homebrew is a package manager for MacOS.\u00a0 You use it to install packages. So go ahead to install it using the command<\/p>\n

    \/<\/span>bin\/<\/span>bash -<\/span>c \"$(curl -fsSL https:\/\/raw.githubusercontent.com\/Homebrew\/install\/master\/install.sh)\"<\/span>\r\n<\/pre>\n
    This command would install XCode command line tools and Homebrew<\/p>\n
     <\/p>\n
    2. Install Java<\/strong><\/h4>\n
    Spark using Java tools for performing various operations. Therefore, you need to have Java installed in your system.<\/p>\n
    Install Java using the command:<\/p>\n
    brew install openjdk\r\n<\/pre>\n
     <\/p>\n
    Check Java installation using the command:<\/p>\n
    java --version<\/span><\/pre>\n
     <\/p>\n
     <\/p>\n
    3. Install Scala<\/strong><\/h4>\n
    Spark was actually written in Scala. So we would need Scala installed as well.<\/p>\n
    brew install Scala<\/span>\r\n<\/pre>\n
     <\/p>\n
    You can find Scala tutorials here<\/a><\/p>\n
    4. Install Apache Spark<\/strong><\/h4>\n
    You now need to install Spark using this command<\/p>\n
    brew install apache-spark<\/span><\/pre>\n
     <\/p>\n
    This command would install Apache Spark. You can then launch the Spark shell using the command below:<\/p>\n
    spark-shell<\/span><\/pre>\n
     <\/p>\n
    5. Verify Apache Spark Installation<\/strong><\/h4>\n
    Let’s now validate Spark installation by writing a simple command that creates a Spark DataFrame<\/p>\n
    import<\/span> spark.implicits._<\/span>\r\nval<\/span> data =<\/span> Seq<\/span>((<\/span>\"Java\"<\/span>,<\/span> \"Programmin\"<\/span>,<\/span> \"60800\"<\/span>),<\/span> (<\/span>\"Python\"<\/span>,<\/span> \"Analysis\"<\/span>,<\/span> \"150000\"<\/span>),<\/span> (<\/span>\"Scala\"<\/span>,<\/span> \"Coding\"<\/span>,<\/span> \"3500\"<\/span>))<\/span>\r\nval<\/span> df =<\/span> data.<\/span>toDF()<\/span> \r\ndf.<\/span>show()<\/span>\r\n<\/pre>\n
    Note: You can type the codes on line at a time<\/p>\n
    The output screen is as shown below:<\/p>\n
    \"Creating
    Creating a DataFrame in Spark<\/figcaption><\/figure>\n
     <\/p>\n
    6. Spark Installation on Window<\/strong><\/h4>\n
    Follow the steps below to install Apache Spark on Window.<\/p>\n
    Step 1<\/strong> – Navigate the Apache Spark download page using this link – https:\/\/spark.apache.org\/downloads.html<\/a><\/p>\n
    Step 2<\/strong> – Download the tgz file<\/p>\n
    Step 3<\/strong> – Unzip the file into a local directory<\/p>\n
    Step 4<\/strong> – Create a directory path in your C drive and copy the Spark folder content into the directory<\/p>\n
    Step 5<\/strong> – Set the JAVA_HOME, SPARK_HOME, HADOOP_HOME an PATH environment variables. Use the data below:<\/p>\n
    JAVA_HOME = C:\\Program Files\\Java\\jdk18.0.1.1\r\nPATH = %PATH%;%JAVA_HOME%\r\n\r\nSPARK_HOME  = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nHADOOP_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nPATH=%PATH%;%SPARK_HOME%\r\n<\/pre>\n
    Note<\/strong>: The directory path should be exactly the location on you local drive<\/p>\n
    Step 6<\/strong> – Download winutils.exe<\/strong> for Hadoop and copy it to the bin folder in the spark path. Get it from here https:\/\/github.com\/cdarlint\/winutils<\/a><\/p>\n
    Step 7<\/strong> – Open command prompt. Navigate into the bin directory of the Spark installation.<\/p>\n
    Step 8<\/strong> – Start spark-shell just like we did in Mac<\/p>\n
     <\/p>\n
    7. Access the Spark Web UI<\/strong><\/h4>\n
    Apache Spark comes with a suite of Web User Interface(UI) to help you monitor\u00a0 your spark application. The Spark Web UI can be access via the link: http:\/\/localhost:4040\/<\/strong><\/p>\n
    This is shown below<\/p>\n
    \"Spark<\/a>
    Spark Web UI<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"
    In this tutorial, you will learn how to install Apache Spark on Mac and Windows. We would take to following steps Install Homebrew Install Java … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,3],"tags":[32,33,35,34],"class_list":["post-146","post","type-post","status-publish","format-standard","hentry","category-pyspark","category-spark","tag-hadoop","tag-java","tag-scala","tag-winutils"],"_links":{"self":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts\/146","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/comments?post=146"}],"version-history":[{"count":5,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts\/146\/revisions"}],"predecessor-version":[{"id":155,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/posts\/146\/revisions\/155"}],"wp:attachment":[{"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/media?parent=146"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/categories?post=146"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-json\/wp\/v2\/tags?post=146"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}