Docker Swarm / FluentD - Removing a Key

If you are setting up FluentD within a Docker Swarm, with the docker metadata filter plugin you may come across this error when there’s a swarm label with an empty value. In my case, the problem was with this key: docker.labels.com.docker.swarm.task.

Error Message

{
	"type": "illegal_argument_exception",
	"reason": "Can't merge a non object mapping [docker.labels.com.docker.swarm.task] with an object mapping [docker.labels.com.docker.swarm.task]"
}

Example of an event that has this empty label, triggering the error

{
	"message": "2018-06-23T18:18:05+0000 INFO takes the value and converts it to string.\n",
	"type": "http-bulk",
	"fluentd_tags": "final",
	"docker": {
		"id": "611e37729095550c20c8d464ccf6a5fcbdae3db7859441d5b45fdfbe26b74168",
		"name": "testing_random-logger.1.cwx24otbnqhgmszjyhonsp8wo",
		"container_hostname": "611e37729095",
		"image": "chentex/random-logger:[email protected]:f287d3ff193fc99211943ddfcf4fcfcff6acbe2ac7dd8385acb07bcaaec6a41f",
		"image_id": "sha256:759976efd70726fd508dfe7795ebd55cea9b96585f615c558e4ccc28e5d33f12",
		"labels": {
			"com.docker.stack.namespace": "testing",
			"com.docker.swarm.node.id": "gz4fb30qwy2lhq4htazaeyypw",
			"com.docker.swarm.service.id": "qa72pgmzc26akxclz99j170d3",
			"com.docker.swarm.service.name": "testing_random-logger",
			"com.docker.swarm.task": "",
			"com.docker.swarm.task.id": "cwx24otbnqhgmszjyhonsp8wo",
			"com.docker.swarm.task.name": "testing_random-logger.1.cwx24otbnqhgmszjyhonsp8wo"
		}
	},
	"@timestamp": "2018-06-23T18:18:05.275+00:00",
	"stream": "stdout"
}

Config block that removes the JSON key that has an empty value and corrects the error

<filter docker.**>
  @type record_transformer
  remove_keys $['docker']['labels']['com.docker.swarm.task']
</filter>

Victory.

References

  • https://github.com/moby/moby/issues/28806
  • https://discuss.elastic.co/t/using-autodiscover-feature-for-docker-does-not-work-when-running-in-swarm-mode/120342/2

This is a preview of Clap Button, a new feedback and analytics tools for Hydejack, built by yours truly. You can try it out on localhost for free, but it will be removed (together with this message) when building with JEKYLL_ENV=production. To use Clap Button on your site, get a subscription
and set clap_button: true in your config file.


© 2021. All rights reserved.