Monitoring suave.io app with Datadog

03 September 2016 – Karelia

At work, we have a lot of infrastructure and we have to know what is going on with our servers and apps. There are different options to do that and one of them is Datadog service. Datadog can handle all stats from your servers and show you nice graphs and send alerts in case of a problem. To use it, you have to create an account and install datadog agent service on your server. Today I decided to write an suave.io web site and connect it to our datadog account to show some info about request rates and so on. There are different options how to connect your app to your datadog account. For example you could write datadog custom check, but it is easier to use existing one. We use golang expvars checks for our golang servers, so I decided to emulate them in suave.

What is expvars?

Expvars is a golang module which allows you to create some variables and expose them via web interface in json format. You could find more info here. This is an example of expvars output. There are some predefined expvars in golang, so you could check memory stats and so on.

 
{
"cmdline": ["/xxx"],
"err_checks-rps": 20,
"memstats": {"Alloc":408478640,"a lot of other mem stats" : "tons of stats"},
"succ_checks-rps": 3471,
}

By convention address of this json is “IP:8000/debug/vars”.

Let’s implement one expvar in a suave app. First of all, you have to create a datadog account and after that install datadog agent as described here. In our case this is an ubuntu server. After installation we have to create “/etc/dd-agent/conf.d/go_expvar.yaml” file.

 
init_config:

instances:
  - expvar_url: http://localhost:8000/debug/vars
    metrics:
       - path: hellorps

Actually we also need to comment this line in “/etc/dd-agent/checks.d/go_expvar.py”:

 
#self.get_gc_collection_histogram(data, tags, url, namespace)

This line will throw in case of missing memstats data. Now restart the datadog agent “/etc/init.d/datadog-agent restart”) More about how to setup datadog with expvars you could find here Last step is to create suave.io app with “hellorps” expvar and run it. We will use suave script template

 
open System
open System.IO

Environment.CurrentDirectory <- __SOURCE_DIRECTORY__
 
if (File.Exists "paket.exe") then
    File.Delete "paket.exe"
let url = "https://github.com/fsprojects/Paket/releases/download/3.18.2/paket.exe"
use wc = new Net.WebClient()
let tmp = Path.GetTempFileName()
wc.DownloadFile(url, tmp)
File.Move(tmp,Path.GetFileName url);;
 
// Step 1. Resolve and install the packages
 
#r "paket.exe"
 
Paket.Dependencies.Install """
source https://nuget.org/api/v2
nuget Suave
""";;
 
// Step 2. Use the packages
 
#r "packages/Suave/lib/net40/Suave.dll"
#r "System.Runtime.Serialization"
open Suave // always open suave
open Suave.Filters
open Suave.Operators
open Suave.Successful
open Suave.Json
open System.Runtime.Serialization
open System.Net

Now we have to add a thread safe rate counter.

 
type RateCounter() = 
    let mutable lastTick = System.Environment.TickCount
    let mutable lastFrameRate = 0
    let mutable frameRate = 0
    let lockObj = new obj()
    let synchronized f v = lock lockObj (fun () -> f v)
    let inc v = 
            if System.Environment.TickCount - lastTick >= 1000 
            then
                lastFrameRate <- frameRate
                frameRate <- v
                lastTick <- System.Environment.TickCount
            else
                frameRate <- frameRate + v
            lastFrameRate
    

    member x.Inc v = synchronized inc v |> ignore
    member x.Get () = synchronized inc 0

let helloCounter = RateCounter()

Create hello webpart and start server.

 
let hello (x : HttpContext) =
      helloCounter.Inc 1 
      OK "Hello World!" x

let _, server = startWebServerAsync defaultConfig hello

And do the same for expvars.

 
let expvarsConfig = {defaultConfig with bindings = 
            [ HttpBinding.mkSimple  Protocol.HTTP "0.0.0.0" 8000]}

let _, expvarsServer = startWebServerAsync expvarsConfig (path "/debug/vars" 
                        >=> fun ctx -> (OK (sprintf "{\"hellorps\":%d}" 
                        <| helloCounter.Get() ) ctx))

create load generator.

 
let client = new WebClient ()
let rec gen_load sleepMs = async{
    let! _ = client.AsyncDownloadString(new Uri("http://127.0.0.1:8083/"))
    do! Async.Sleep(sleepMs)
    return! gen_load(sleepMs) 
}

And finally run everything

 
Async.Parallel [server; expvarsServer; gen_load(0)] 
    |> Async.RunSynchronously 
    |> ignore

Done. Now we can go to datadog and see our metric.

Datadog Metric

Full code

Monitoring suave.io app with Datadog

What is expvars?

Related Posts