The error details
[root@mynode ~]# /opt/puppetlabs/bin/puppet agent -t --server puppet_master.mycompany.com --environment production
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Find /puppet/v3/node/?environment=production&configured_environment=produc... resulted in 404 with the message: Not Found: Could not find node
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find node 'mynode.company.com'; cannot compile
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Further investigation:
This was the only server affected so the fault didn't appear to be across the whole puppet master. Any error which staes that a node cannot be found would indicate that either the site.pp nodes list doesn't contain the node definition, or in this case, the external node classifier (Cobbler) is unable to find the given node.
When browsing Cobbler, I can definitely see the node listed. More specifically,
- The Cobbler client name is the short name (e.g. mynode)
- The cobbler network name is the FQDN (mynode.mycompany.com)
When performing the API call for two nodes (directly to Cobbler) I found that it was indeed unable to retrieve the correct data for the node where the error was occurring. The correct values were however returned for the other node, proving that the ENC relationship between Cobbler and Puppet was working correctly.
Query (lookup data for the node with the fault):
http://192.168.20.44/cblr/svc/op/puppet/hostname/mynode.company.com
Result:
classes: {}
parameters: {}
Query (lookup data for a node whose puppet client environment is operating normally):
http://192.168.20.44/cblr/svc/op/puppet/hostname/webserver.company.com
Result:
classes:
roles::operations::puppetmaster: {}
parameters:
from_cobbler: 1
my_motd: 'Nginx web server 4'
preserve_data: true
tree: http://@@http_server@@/cblr/links/centos7.2-x86_64
As it turns out, the puppet API call relies on the “per-interface” DNS name, not the global hostname, i.e. (below JSON is abbreviated)
{
"hostname": "",
"interfaces": {
"em1": {
"dns_name": "undefined",
}
}
}
Is incorrect. When using koan to define our nodes, we don't yet set the hostname (until later in the workflow). In this situation, the hostname had been set correctly the global network settings but not on a per-interface basis.
The Solution
The interfaces à interface_name à dns_name must be "mynode.company.com" for this to translate correctly for the puppet integration to work.
It looks like any hostname can match, as most systems only have this dns_name parameter set correctly for the primary interface em1. On this occasion, another interface had snuck in as a string appearing earlier in the sort, therefore networking had been applied to the alternate interface instead of em1
In screenshots,
Global Hostname
· This sets the system hostname (/etc/sysconfig/network --> HOSTNAME={blah} )
Per-Interface hostname
· At least one of these needs to match, for puppet ENC lookup to work correctly
· Duplicates are OK (i.e. multiple interfaces on the same node with the same dns-name record)