Apache SparkTM is known as popular big data framework which is faster than Hadoop MapReduce, easy-to-use, and fault-tolerant. Most of the Spark tutorials require readers to understand Scala, Java, or Python as base programming language. But, in my opinion, SQL is enough to write a spark batch script.
In this article, I will show that you can write Spark batches only in SQL if your input data is ready as structured dataset. This means that you don’t need to learn Scala or Python, RDD, DataFrame if your job can be expressed in SQL. Moreover, the expression power of SparkSQL may be stronger than you think.
The SQL scripts as follows can be found at https://github.com/sanori/spark-sql-example. You can test them yourself.
Read More