Why Won’t The Logstash API reply to cURL? Logstash and Docker

Background

I spent some time trying to figure out why Logstash would not reply to cURL on the API port (By default :9600/tcp) after it started up. There’s seemingly a discrepancy with Elasticsearch’s docs on their Docker image (more on that later). There’s also an implicit default for the API endpoint that is confusing.

The Evidence

Logstash logs never fail to say they successfully start the API endpoint:

{
  "level": "INFO",
  "loggerName": "logstash.agent",
  "timeMillis": 1537215513885,
  "thread": "Api Webserver",
  "logEvent": {
    "message": "Successfully started Logstash API endpoint",
    "port": 9600
  }
}

Note: log.format: "json" is set in logstash.yml

Don’t take that earlier INFO message to mean what it says. Maybe it did successfully start it, but that doesn’t mean it will behave as you’d expect for a running service without errors.

And so I do a basic query to the API:

curl -XGET 'localhost:9600/?pretty'

But I get a mixture of these errors:

curl: (56) Recv failure: Connection reset by peer

or

curl: (52) Empty reply from server

The Fix

Ensure that your logstash.yml has this property:

http.host: "0.0.0.0"

The documentation insisted this was set by default, but it only worked when I set it explicitly.

After startup you should be able to get a good reply:

curl -XGET 'localhost:9600/?pretty'
{
  "host" : "3f1ad9dc77c6",
  "version" : "6.4.0",
  "http_address" : "0.0.0.0:9600",
  "id" : "cc9ef93c-23bd-4039-9a35-c8cebfdf019d",
  "name" : "3f1ad9dc77c6",
  "build_date" : "2018-08-18T00:25:22Z",
  "build_sha" : "f8014ac54e6c8ff6c071c0960ca1b00e9735f43a",
  "build_snapshot" : false
}

A Gotchya

Make sure there are no ERROR’s in the logs pertaining to a plugin. For example, while I was testing to see if I could cURL the API, I was using the Lumberjack Output Plugin, where I had Firewall ACLs blocking the outbound port (accidentally), so it could never establish a connection with the endpoint. But the app continues to run, just repeating those errors over and over again:

curl -XGET 'localhost:9600/?pretty'
{
  "level": "ERROR",
  "loggerName": "logstash.outputs.lumberjack",
  "timeMillis": 1537215336189,
  "thread": "[main]-pipeline-manager",
  "logEvent": {
    "message": "All hosts unavailable, sleeping",
    "hosts": [
      "55.21.41.256"
    ],
  }
}

References

This is a preview of Clap Button, a new feedback and analytics tools for Hydejack, built by yours truly. You can try it out on localhost for free, but it will be removed (together with this message) when building with JEKYLL_ENV=production. To use Clap Button on your site, get a subscription
and set clap_button: true in your config file.


© 2021. All rights reserved.