- \n
- Start Spark Shell and Create a Text File<\/a><\/li>\n
- Create an RDD<\/a><\/li>\n
- Execute Work Count Transformation<\/a><\/li>\n
- Cache and Save the RDD<\/a><\/li>\n
- View the Output<\/a><\/li>\n<\/ol>\n
<\/p>\n
1. Start Spark Shell and Create a Text File<\/strong><\/h4>\n
By now you should have installed the Spark and can start the Spark Shell.<\/p>\n
Step 1<\/strong> – Start the Spark Shell<\/p>\n
Step 2<\/strong> – Create a text file (I name it inputfile.txt<\/em>) in the home directory where the spark shell started from<\/p>\n
Have this content in the text file<\/p>\n
People can be prejudiced not only toward those of another nationality,<\/span> \r\nrace,<\/span> tribe,<\/span> or language but also toward those of a different religion,<\/span> \r\ngender,<\/span> or social class.<\/span> Some judge people negatively based on their age,<\/span> \r\neducation,<\/span> disabilities,<\/span> or physical appearance.<\/span> \r\nYet,<\/span> they still feel that they are not prejudiced.<\/span>\r\n<\/pre>\n<\/p>\n
2. Create an RDD<\/strong><\/h4>\n
So we would read this file using Spark-Scala API and create an RDD from it.<\/p>\n
Execute the command below:<\/p>\n
val<\/span> fileRDD =<\/span> sc.<\/span>textFile(<\/span>\"inputfile.txt\"<\/span>)<\/span>\r\n<\/pre>\n<\/p>\n
This command reads a text file from the given path and creates an RDD named fileRDD. In this case, the file is expected to be in the current location.<\/p>\n
<\/p>\n
3. Execute Word Count Transformation<\/strong><\/h4>\n
We would like to have a word count of each word in the file. To do this we would use three functions:<\/p>\n
flatmap()<\/strong> – split the content of the file by space<\/p>\n
map()<\/strong> – get the word count<\/p>\n
reduceByKey(_+_)<\/strong> – add values of similar keys<\/p>\n
The complete command is given below:<\/p>\n
val<\/span> wordCounts =<\/span> fileRDD.<\/span>flatMap(<\/span>line =><\/span> line.<\/span>split(<\/span>\" \"<\/span>)).<\/span>map(<\/span>word =><\/span> (<\/span>word,<\/span> 1<\/span>)).<\/span>reduceByKey(<\/span>_<\/span>+<\/span>_<\/span>);<\/span>\r\n<\/pre>\n<\/p>\n
This command is a transformation applied to the fileRDD and creates a new RDD. If you want to see this new RDD (though it would not make much sense to you!), you can used the command below:<\/p>\n
wordCounts.<\/span>toDebugString\r\n<\/pre>\n<\/p>\n
4. Cache and Save the RDD<\/strong><\/h4>\n
Now we want to cache our transformed RDD and also persist it to storage.<\/p>\n
The command below uses the cache()<\/em> function(persist can also be used) to persist the RDD in memory<\/p>\n
wordCounts.<\/span>cache\r\n<\/pre>\n<\/p>\n
Use the command below to apply an action<\/strong> to the RDD. Here, we want to save the output of the transformation to a text file. This is saved in a folder named outputDir<\/em>.<\/p>\n
wordCounts.<\/span>saveAsTextFile(<\/span>\"outputDir\"<\/span>)<\/span>\r\n<\/pre>\n<\/p>\n
5. View the Output<\/strong><\/h4>\n
Now, you want to view the ouptutfile. We also want to see the status of the Spark job in the GUI.<\/p>\n
Step 1<\/strong> – Open a new terminal and navigate into the outputDir folder<\/p>\n
Step 2<\/strong> – Use the the command below to view the content<\/p>\n
ls -<\/span>1<\/span>\r\n<\/pre>\n<\/p>\n
You will see the output<\/p>\n
part-00000 \r\npart-00001 \r\n_<\/span>SUCCESS\r\n<\/pre>\n<\/p>\n
Step 3<\/strong> – You can then use the cat command to view the content of each file<\/p>\n
Finally, you can view the status on the Web GUI using the the link:<\/p>\n
http:\/\/localhost:4040\r\n<\/pre>\n
<\/p>\n
- Create an RDD<\/a><\/li>\n
![\"Apache](\"https:\/\/www.kindsonthegenius.com\/apache-spark\/wp-content\/uploads\/sites\/13\/2022\/11\/Screenshot-2022-11-22-at-22.34.25.png\")
<\/a>