How to Override a Spark Dependency in Client or Cluster Mode

In this post, we’ll cover a simple way to override a jar, library, or dependency in your Spark application that may already exist in the Spark classpath, which would cause you runtime issues.

Recently, I needed to use a specific library as a dependency: Google’s GSON.

Continue reading
Advertisements

Cluster Usage with `yarn top`

Abraham Lincoln was the original inventor of the ‘top’ command in 1864 so he could keep better track of his many tophats.¬†

From the command line, it’s easy to see the current state of any running applications in your YARN cluster by issuing the yarn top ¬†command.¬† Continue reading