Chef-Client 15.8.23 - AIX - Cannot find chef-client

[Firenice03]

Description

When bootstrapping new AIX LPAR nodes, 15.8.23 is the defaulted client to bootstrap (I have a CB pinned @15.6) appears to install normally but bootstrap fails when starting Chef-Client.

If trying to verify client the below is received:
chef-client
ksh: chef-client: not found
chef
ksh: chef: cannot execute

Bootstrap Command..
knife bootstrap 10.xx.xx.xx -N hostname --sudo -U userid --policy-name cookbook_base --policy-group prd

Chef Version

15.8.23 is the boostrapped Chef-Client version
We use (pinned) Chef-Client 15.6
Chef-Infra: chef-server 13.1.13
Chef Wrkstn: 0.15.18

Platform Version

This client is bootstrapping AIX 7.1 or 7.2 nodes
Chef Infra runs on RHEL 7.5
CW is running from Win10

Replication Case

Client Output

Installing Chef Omnibus (stable/15)

downloading https://omnitruck.chef.io/chef/install.sh
to file /tmp/install.sh.31982016/install.sh
[10.xx.xx.xx] trying perl...

[10.xx.xx.xx] aix 7.2 powerpc
Getting information for chef stable 15 for aix...
downloading https://omnitruck.chef.io/stable/chef/metadata?v=15&p=aix&pv=7.2&m=powerpc
to file /tmp/install.sh.6684970/metadata.txt
[10.xx.xx.xx] trying perl...
[10.xx.xx.xx] sha1 6645e99a3cca27c920fb8e1e2c50326df733b08f
sha256 b22c1415d8f79150009af9782f11f575d49df8d0aad2857f6bdc1f349a2f3ef2
url https://packages.chef.io/files/stable/chef/15.8.23/aix/7.2/chef-15.8.23-1.powerpc.bff
version 15.8.23
[10.xx.xx.xx]
[10.xx.xx.xx] downloaded metadata file looks valid...
[10.xx.xx.xx] downloading https://packages.chef.io/files/stable/chef/15.8.23/aix/7.2/chef-15.8.23-1.powerpc.bff
to file /tmp/install.sh.6684970/chef-15.8.23-1.powerpc.bff
[10.xx.xx.xx] trying perl...

[10.xx.xx.xx] Comparing checksum with shasum...

[10.xx.xx.xx] Installing chef 15
installing with installp...
[10.xx.xx.xx] +-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
[10.xx.xx.xx] WARNINGS

Problems described in this section are not likely to be the source of any
immediate or serious failures, but further actions may be necessary or
desired.

Already Installed

The number of selected filesets that are either already installed
or effectively installed through superseding filesets is 1. See
the summaries at the end of this installation for details.

NOTE: Base level filesets may be reinstalled using the "Force"
option (-F flag), or they may be removed, using the deinstall or
"Remove Software Products" facility (-u flag), and then reinstalled.

<< End of Warning Section >>

[10.xx.xx.xx] Verifying req
uisites...done
Results...

+-----------------------------------------------------------------------------+
BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
Pre-installation Failure/Warning Summary

Name Level Pre-installation Failure/Warning

chef 15.8.23.1 Already installed

[10.xx.xx.xx] Starting the first Chef Infra Client Client run...
[10.xx.xx.xx] /tmp/chef_Z6GhSS/bootstrap.sh[265]: chef-client: not found
ERROR: The following error occurred on 10.xx.xx.xx:
ERROR: +--------------------------------------------------
---------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

   +-----------------------------------------------------------------------------+
               BUILDDATE Verification ...
   +-----------------------------------------------------------------------------+
   Verifying build dates...done
   FILESET STATISTICS
   ------------------
       1  Selected to be installed, of which:
           1  Already installed (directly or via superseding filesets)
     ----
       0  Total to be installed


   Pre-installation Failure/Warning Summary
   ----------------------------------------
   Name                      Level           Pre-installation Failure/Warning
   ------------------------------------------------------------------

---

   chef                      15.8.23.1       Already installed

   /tmp/chef_Z6GhSS/bootstrap.sh[265]: chef-client:  not found

## Stacktrace
<!--- Please include the stacktrace.out output or link to a gist of it, if there is one. -->

[Firenice03]

Want to update - as not sure if clear.. the 15.6.10 client works/worked fine. The 15.8 does not - not sure if something changed between the versions. But other nodes 15.6 runs fine.

[tas50]

Moving this over to mixlib-install where the install.sh code exists. We should figure out if passing the force option is what we want to do on AIX.

[lamont-granquist]

I think this isn't that 15.8 doesn't work but that the update doesn't work.

It looks like the failure is really coming from installp installing the bff where it is finding the prior package and aborting:

chef                      15.8.23.1       Already installed

Somehow it then winds up running the scripts that nuke the /usr/bin/chef-client symlink anyway I guess, which then results in:

   /tmp/chef_Z6GhSS/bootstrap.sh[265]: chef-client:  not found

But the root cause here is probably mixlib-install related to the upgrade process, not anything in the chef/chef repo.

[Firenice03]

If it helps... Either an update or initial install of 15.8 (at time of bootstrap). If 15.8 is installed ... and either command is ran..

Running as root or with sudo the output..
chef-client
ksh: chef-client: not found
chef
ksh: chef: cannot execute

[Firenice03]

FYI - this still occurs...
Newly built AIX LPAR - No previous client; Bootstrapped node, it installs 15.11.8 client all cookbooks sync. I have the Chef_Client_Updater Supermarket cookbook pinned to version 15.6.10, everything appears to work accordingly...

Recipe: chef_client_updater::default

• chef_client_updater[update chef-client] action update
  • chef_gem[mixlib-install] action install
    • install version ~> 3.11 of package mixlib-install
  • move all files under /opt/chef to /opt/chef.upgrade
  • upgrade chef 15.11.8 to 15.6.10[2020-06-09T12:46:43-04:00] WARN: New chef-client installed and exit is allowed. Forcing chef exit!

Then
[2020-06-09T12:46:43-04:00] ERROR: Running exception handlers
Running handlers complete
[2020-06-09T12:46:43-04:00] ERROR: Exception handlers complete
Chef Infra Client failed. 11 resources updated in 46 seconds
[2020-06-09T12:46:43-04:00] FATAL: LoadError: cannot load such file -- enc/trans/single_byte

When trying a subsequent chef run as root...
chef-client
ksh: chef-client: not found.