‘Permission Denied’ with Nomad and Consul

I’ve spent days banging my head on this issue. In my sandbox, Nomad is integrated with Consul. For whatever reason, Nomad’s default of registering itself with Consul’s service discovery feature just wasn’t working.

Error after error about the ACL that Nomad was using being insufficient. But how, with this?

key "nomad" { policy = "write" }
service "" { policy = "read" }
service "nomad" {  policy = "write" }

…Especially given this clear documentation showing it needs service:write permissions: https://www.consul.io/api/agent/service.html

Errors from Nomad:

Sep 02 18:02:11 test-1 nomad[18203]:     2018/09/02 18:02:11.501506 [ERR] consul.sync: still unable to update services in Consul after 1330 failures; latest error: Unexpected response code: 403 (Permission denied)

Errors from Consul:

Sep 02 17:56:11 test-1 consul[24482]:     2018/09/02 17:56:11 [ERR] http: Request PUT /v1/agent/service/register, error: Permission denied from=127.0.0.1:57368

Example of the Consul block in my nomad.json file (With TLS enabled):

  "consul": {
    "address": "127.0.0.1:8501",
    "token": "NOMAD_ACL_TOKEN_FROM_CONSUL",
    "key_file": "tls/nomad-key.pem",
    "cert_file": "tls/nomad.pem",
    "ca_file": "nomad-ca.pem",
    "ssl": true,
    "server_service_name": "nomad",
    "client_service_name": "nomad-client",
    "auto_advertise": true,
    "server_auto_join": true,
    "client_auto_join": true
  }

Ah, but what exactly needs service:write?

The Nomad ACL?… or the ACL of the Consul Agent which is doing the bidding of Nomad when it comes to communicating with Consul. That’s it!

The ACL I had configured for Consul ACL token had service:read permissions. After staring at Consul’s logs long enough, the path in the URL (/v1/agent/) is what gave it away for me.

'Permission Denied' with Nomad and Consul

‘Permission Denied’ with Nomad and Consul

References