Custom metrics source in apache spark

Apache Spark

Setting up a simple metric source

Here are the step to set up a simple spark metric source:

  1. The first step is to write a class that extends the Source trait. org.apache.spark.metrics.source.Source is the top-level class for the
    metric registries in Spark. Sources expose their internal status.
  2. Next, enable the sink. Here, the output is printed to console.
  3. Finally, instantiate the source and register it with SparkEnv.
  4. Now we are ready to collect the metrics.
-- Counters --------------------------------------------------------------------
...
local-1813126621247.driver.CustomMetricSource.metricCounter
count = 10
-- Histograms ------------------------------------------------------------------
...
local-1813126621247.driver.CustomMetricSource.metricHistory
count = 10
min = 1
max = 1
mean = 1.00
stddev = 0.00
median = 1.00
75% <= 1.00
95% <= 1.00
98% <= 1.00
99% <= 1.00
99.9% <= 1.00

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store