Bright Java Tutorial

is brought to you by DoITBright LLC

Apache Spark

Writing Your First Spark Application

I am using a Windows 8 operating system so the instructions will be for that kind of environment. Open your Windows Explorer and choose a directory where you wish to create a folder to put your HTML files. I choose C:\SparkStuff. Then, I will create a directory here and name it as SparkDev. Inside the SparkDev directory, create a directory and name it as bin. Download the executable winutils and place it in your c:\SparkStuff\SparkDev\bin directory. Download and extract the Eclipse IDE inside your c:\SparkStuff directory. Launch your Eclipse and point the workspace to c:\SparkStuff directory.

Eclipse workspace for Apache Spark development.

Edit the pom.xml file generated by your Eclipse IDE when you created the Maven project. This should be the content of your pom.xml after you made the changes.

  <project xmlns="http://maven.apache.org/POM/4.0.0" 
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
	  http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.brightjavatutorial.spark</groupId>
	<artifactId>SparkDev</artifactId>
	<version>0.0.1-SNAPSHOT</version>

	<properties>
	  <java.version>1.8</java.version>
	</properties>

	<dependencies>

	  <dependency>
	    <groupId>org.apache.spark</groupId>
		<artifactId>spark-core_2.10</artifactId>
		<version>2.0.0</version>
	  </dependency>

	</dependencies>

	<build>
	  <pluginManagement>
	    <plugins>
		  <plugin>
		    <groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.1</version>
			<configuration>
			  <source>${java.version}</source>
			  <target>${java.version}</target>
			</configuration>
		  </plugin>
		</plugins>
	  </pluginManagement>
	</build>
  </project>
                           
Create a Java class. Here is the content of the class.

  package com.brightjavatutorial.spark;

  import org.apache.spark.SparkConf;
  import org.apache.spark.api.java.JavaSparkContext;
 
  public class SparkSample {
 
    public static void main(String[] args) {
 
      SparkConf sparkConf = new SparkConf();
 
      sparkConf.setAppName("Hello Spark");
      sparkConf.setMaster("local");
 
      JavaSparkContext context 
        = new JavaSparkContext(sparkConf);
 
      System.out.println(">>> hi there!");
 
      context.close();
 
    }
 
  }
                           
To run the application, click on the class file and on your Eclipse, look for the 'Run' then 'Run Configurations'. Here is how it would look like...

Eclipse workspace for Apache Spark development.


Eclipse workspace for Apache Spark development.