Why Won't The Logstash API reply to cURL? | Logstash and Docker

Why Won’t The Logstash API reply to cURL? | Logstash and Docker

Background

I spent some time trying to figure out why Logstash would not reply to cURL on the API port (By default :9600/tcp) after it started up. There’s seemingly a discrepancy with Elasticsearch’s docs on their Docker image (more on that later). There’s also an implicit default for the API endpoint that is confusing.

The Evidence

Logstash logs never fail to say they successfully start the API endpoint:

{
  "level": "INFO",
  "loggerName": "logstash.agent",
  "timeMillis": 1537215513885,
  "thread": "Api Webserver",
  "logEvent": {
    "message": "Successfully started Logstash API endpoint",
    "port": 9600
  }
}

Note: log.format: "json" is set in logstash.yml

Don’t take that earlier INFO message to mean what it says. Maybe it did successfully start it, but that doesn’t mean it will behave as you’d expect for a running service without errors.

And so I do a basic query to the API:

curl -XGET 'localhost:9600/?pretty'

But I get a mixture of these errors:

curl: (56) Recv failure: Connection reset by peer

or

curl: (52) Empty reply from server

The Fix

Ensure that your logstash.yml has this property:

http.host: "0.0.0.0"

The documentation insisted this was set by default, but it only worked when I set it explicitly.

After startup you should be able to get a good reply:

curl -XGET 'localhost:9600/?pretty'

{
  "host" : "3f1ad9dc77c6",
  "version" : "6.4.0",
  "http_address" : "0.0.0.0:9600",
  "id" : "cc9ef93c-23bd-4039-9a35-c8cebfdf019d",
  "name" : "3f1ad9dc77c6",
  "build_date" : "2018-08-18T00:25:22Z",
  "build_sha" : "f8014ac54e6c8ff6c071c0960ca1b00e9735f43a",
  "build_snapshot" : false
}

A Gotchya

Make sure there are no ERROR’s in the logs pertaining to a plugin. For example, while I was testing to see if I could cURL the API, I was using the Lumberjack Output Plugin, where I had Firewall ACLs blocking the outbound port (accidentally), so it could never establish a connection with the endpoint. But the app continues to run, just repeating those errors over and over again:

curl -XGET 'localhost:9600/?pretty'

{
  "level": "ERROR",
  "loggerName": "logstash.outputs.lumberjack",
  "timeMillis": 1537215336189,
  "thread": "[main]-pipeline-manager",
  "logEvent": {
    "message": "All hosts unavailable, sleeping",
    "hosts": [
      "55.21.41.256"
    ],
  }
}

References