Fluent-Bit, HTTP Output, and JSON Structure

Background

I recently went deep into tweaking and deploying the fluent bit helm chart into my Kubernetes Cluster. Most importantly for this post, I was using the HTTP Output Plugin to send logs to Logz.io (They have a free tier). But every time fluent-bit would push logs, I’d get a HTTP 400 response code.

And so it begins…

Warning

In its current state, the Fluent-Bit Helm Chart must be customized (not just values.yaml), to interpret the additional HTTP output plugin configuration properties.

JSON

When using backend.type.http with Format json set in config, fluent-bit will group multiple messages into a JSON array, and that array composes the body of the HTTP POST request. In this case, Logz.io can’t parse that JSON structure correctly (Their support person confirmed). I tested different structures with cURL:

‘Malformed’ JSON Array

Notice the brackets encapsulating multiple JSON objects, denoting an array.

$ echo "[{\"iamkey\":\"iamvalue\"},{\"who\":\"you\"}]" | curl -X POST "https://listener.logz.io:8071" -v --data-binary @-
...
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=UTF-8
<
* Connection #0 to host example.com left intact
{"malformedLines":1,"oversizedLines":0,"successfulLines":0}

Acceptable JSON Format

$ echo "{\"iamkey\":\"iamvalue\"},{\"who\":\"you\"}" | curl -X POST "https://listener.logz.io:8071" -v --data-binary @-
...
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8

The way to get this working with fluent-bit is using Format json_stream, like so:

[OUTPUT]
        Name  http
        Match *
        Host listener.logz.io
        Port 8071
        URI "?token=PUT_TOKEN_HERE&type=json"
        # Proxy http://proxy-server.internal.com
        tls on
        Format json_stream

References

  • https://github.com/fluent/fluent-bit/issues/680

This is a preview of Clap Button, a new feedback and analytics tools for Hydejack, built by yours truly. You can try it out on localhost for free, but it will be removed (together with this message) when building with JEKYLL_ENV=production. To use Clap Button on your site, get a subscription
and set clap_button: true in your config file.


© 2021. All rights reserved.