How to Set or Change Log Level in Spark Streaming

Logs can really add up. Let’s learn to make like a tree and reduce them via convenient built-in methods.

Apache Spark alone, by default, generates a lot of information in its logs. Spark Streaming creates a metric ton more (in fairness, there’s a lot going on). So, how do we lower that gargantuan wall of text to something more manageable?

One way is to lower the log level for the Spark Context, which is retrieved from the Streaming Context. Simply:

Pretty easy, right?


3 thoughts on “How to Set or Change Log Level in Spark Streaming

  1. Vetle March 16, 2018 / 8:58 am

    Are you deploying this in cluster mode? Because I tried this, and it didn’t seem to make a difference. Desperately need to reduce logging. 😀


    • Landon Robinson March 16, 2018 / 11:29 pm

      Hey Vetle!

      Most of my apps are deployed in YARN mode, though I imagine this would apply to cluster mode as well. What mode are you setting your logging to (INFO, ERROR, WARN, etc)?


      • vetler March 17, 2018 / 8:28 am

        I was setting to all the levels, just to try to change *something* 🙂

        I ended up with a solution I found elsewhere, which is setting it per executor JVM in a call to .forEachPartition. That’s the only thing I could get to work.

        Running cluster in Standalone mode, might make a difference.

        Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s