Fluent-Bit, HTTP Output, and JSON Structure
Fluent-Bit, HTTP Output, and JSON Structure
Background
I recently went deep into tweaking and deploying the fluent bit helm chart into my Kubernetes Cluster.
Most importantly for this post, I was using the HTTP Output Plugin to send logs to Logz.io (They have a free tier). But every time fluent-bit would push logs, I’d get a HTTP 400 response code.
And so it begins…
Warning
In its current state, the Fluent-Bit Helm Chart must be customized (not just values.yaml), to interpret the additional HTTP output plugin configuration properties.
JSON
When using backend.type.http with Format json set in config, fluent-bit will group multiple messages into a JSON array, and that array composes the body of the HTTP POST request. In this case, Logz.io can’t parse that JSON structure correctly (Their support person confirmed). I tested different structures with cURL:
‘Malformed’ JSON Array
Notice the brackets encapsulating multiple JSON objects, denoting an array.
$ echo "[{\"iamkey\":\"iamvalue\"},{\"who\":\"you\"}]" | curl -X POST "https://listener.logz.io:8071" -v --data-binary @-
...
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=UTF-8
<
* Connection #0 to host example.com left intact
{"malformedLines":1,"oversizedLines":0,"successfulLines":0}
Acceptable JSON Format
$ echo "{\"iamkey\":\"iamvalue\"},{\"who\":\"you\"}" | curl -X POST "https://listener.logz.io:8071" -v --data-binary @-
...
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
The way to get this working with fluent-bit is using Format json_stream, like so:
[OUTPUT]
Name http
Match *
Host listener.logz.io
Port 8071
URI "?token=PUT_TOKEN_HERE&type=json"
# Proxy http://proxy-server.internal.com
tls on
Format json_stream