KVM VM install failure with virt-install

I've been working quite a bit lately with CentOS and KVM. This includes Fedora and RedHat releases. Most of my VM creation has been done with virt-manager. It works very well, but I wanted to do the same via CLI and virt-install.

However, I ran into a problem trying to create and VM with virt-install. Every time it would fail and drop into emergency mode.  Sample screen output below. The result was the same for both CentOS and Fedora. I did not try this with RedHat, but suspect the results will be the same.

The creation CLI:

# virt-install -n testvm --vcpus 1 --memory 1024 \
--disk path=/var/lib/libvirt/images/testvm.img,size=8,format=raw \
--location=http://192.168.1.177/centos --noreboot \
--noautoconsole -x 'console=ttyS0 ks=http://192.168.1.177/ks.cfg'

# virsh console testvm

...
[   34.668127] dracut-initqueue[947]: curl: (23) Failed writing body (4350 != 16384)
[   35.538991] loop: module loaded
[   35.670688] dracut-initqueue[947]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
[   35.674472] dracut-initqueue[947]: missing codepage or helper program, or other error
[   35.677472] dracut-initqueue[947]: In some cases useful info is found in syslog - try
[   35.678199] dracut-initqueue[947]: dmesg | tail or so.
[   35.710336] dracut-initqueue[947]: umount: /run/initramfs/squashfs: not mounted
[   35.774186] dracut-initqueue[947]: /sbin/dmsquash-live-root: line 286: printf: write error: No space left on device
[   36.734669] dracut-initqueue[947]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   36.742602] dracut-initqueue[947]: Dload  Upload   Total   Spent    Left  Speed
100  1033  100  1033    0     0  54302      0 --:--:-- --:--:-- --:--:-- 57388  0 --:--:-- --:--:-- --:--:--     0
[   37.930641] dracut-initqueue[947]: Traceback (most recent call last):
[   37.935047] dracut-initqueue[947]: File "/sbin/parse-kickstart", line 688, in
[   37.938548] dracut-initqueue[947]: outfile, output = process_kickstart(path)
[   37.941385] dracut-initqueue[947]: File "/sbin/parse-kickstart", line 675, in process_kickstart
[   37.945262] dracut-initqueue[947]: processed_file = preprocessKickstart(ksfile)
[   37.946152] dracut-initqueue[947]: File "/usr/lib/python2.7/site-packages/pykickstart/parser.py", line 136, in preprocessKickstart
[   37.947140] dracut-initqueue[947]: rc = _preprocessStateMachine (iter(fh.readlines()))
[   37.947751] dracut-initqueue[947]: File "/usr/lib/python2.7/site-packages/pykickstart/parser.py", line 81, in _preprocessStateMachine
[   37.949683] dracut-initqueue[947]: os.write(outF, l)
[   37.951503] dracut-initqueue[947]: OSError: [Errno 28] No space left on device
...
[   32.216242] dracut-initqueue[947]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   32.222391] dracut-initqueue[947]: Dload  Upload   Total   Spent    Left  Speed
 76  432M   76  330M    0     0   135M      0  0:00:03  0:00:02  0:00:01  135M  0 --:--:-- --:--:-- --:--:--     0
[   34.668127] dracut-initqueue[947]: curl: (23) Failed writing body (4350 != 16384)
[   35.670688] dracut-initqueue[947]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
[   35.674472] dracut-initqueue[947]: missing codepage or helper program, or other error
[   35.677472] dracut-initqueue[947]: In some cases useful info is found in syslog - try
[   35.678199] dracut-initqueue[947]: dmesg | tail or so.
[   35.710336] dracut-initqueue[947]: umount: /run/initramfs/squashfs: not mounted
[   35.774186] dracut-initqueue[947]: /sbin/dmsquash-live-root: line 286: printf: write error: No space left on device
[   36.734669] dracut-initqueue[947]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   36.742602] dracut-initqueue[947]: Dload  Upload   Total   Spent    Left  Speed
100  1033  100  1033    0     0  54302      0 --:--:-- --:--:-- --:--:-- 57388  0 --:--:-- --:--:-- --:--:--     0
[   37.930641] dracut-initqueue[947]: Traceback (most recent call last):
[   37.935047] dracut-initqueue[947]: File "/sbin/parse-kickstart", line 688, in
[   37.938548] dracut-initqueue[947]: outfile, output = process_kickstart(path)
[   37.941385] dracut-initqueue[947]: File "/sbin/parse-kickstart", line 675, in process_kickstart
[   37.945262] dracut-initqueue[947]: processed_file = preprocessKickstart(ksfile)
[   37.946152] dracut-initqueue[947]: File "/usr/lib/python2.7/site-packages/pykickstart/parser.py", line 136, in preprocessKickstart
[   37.947140] dracut-initqueue[947]: rc = _preprocessStateMachine (iter(fh.readlines()))
[   37.947751] dracut-initqueue[947]: File "/usr/lib/python2.7/site-packages/pykickstart/parser.py", line 81, in _preprocessStateMachine
[   37.949683] dracut-initqueue[947]: os.write(outF, l)
[   37.951503] dracut-initqueue[947]: OSError: [Errno 28] No space left on device
[   38.000148] dracut-initqueue[947]: /lib/anaconda-lib.sh: line 217: /tmp/ks.info: No such file or directory
[  OK  ] Started dracut initqueue hook.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
         Starting dracut pre-mount hook...
[  OK  ] Started dracut pre-mount hook.
[  OK  ] Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[  OK  ] Started Reload Configuration from the Real Root.
[  OK  ] Reached target Initrd File Systems.
         Starting dracut mount hook...
[   39.194205] dracut-mount[2230]: Warning: Can't mount root filesystem
[   39.301710] dracut-mount[2230]: Warning: /dev/root does not exist
[   39.343224] dracut-mount[2230]: /lib/dracut-lib.sh: line 1049: echo: write error: No space left on device
         Starting Dracut Emergency Shell...
Warning: /dev/root does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


:/#


After several attempts and research, I found an older issue with creating VMs with 1GB of ram or less. It appears this is still a limitation. With some testing, I found 1024MB fails, but 1280MB works fine. This appears to be bare minimum to net install. Your mileage may vary. Might be good to use 1536MB or even more.

# virt-install -n testvm --vcpus 1 --memory 1280 \
--disk path=/var/lib/libvirt/images/testvm.img,size=8,format=raw \
--location=http://192.168.1.177/centos --noreboot \
--noautoconsole -x 'console=ttyS0 ks=http://192.168.1.177/ks.cfg' 

Console output:

...
100  432M  100  432M    0     0   122M      0  0:00:03  0:00:03 --:--:--  122M  0 --:--:-- --:--:-- --:--:--     0
[   32.583223] loop: module loaded
[   34.261656] dracut-initqueue[967]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   34.267978] dracut-initqueue[967]: Dload  Upload   Total   Spent    Left  Speed
100  1033  100  1033    0     0  67622      0 --:--:-- --:--:-- --:--:-- 73785  0 --:--:-- --:--:-- --:--:--     0
[  OK  ] Started dracut initqueue hook.
...
[  OK  ] Reached target Switch Root.
         Starting Switch Root...
[   41.032244] systemd-journald[78]: Received SIGTERM from PID 1 (systemd).
[   41.875854] SELinux:  Class bpf not defined in policy.
[   41.877189] SELinux: the above unknown classes and permissions will be allowed
[   41.886986] type=1403 audit(1550976715.194:2): policy loaded auid=4294967295 ses=4294967295
[   41.905389] systemd[1]: Successfully loaded SELinux policy in 466.129ms.
[   42.122265] systemd[1]: Relabelled /dev, /run and /sys/fs/cgroup in 69.149ms.

Welcome to CentOS Linux 7 (Core)!
...

The process continues on to do a successful KickStart installation. If you see similar failures, it might be a worth a try to bump up VM memory. You can always adjust VM down to the desired size (in my case 512MB) after installation.

EMC Secure Remote Service CLI

This is a re-post.

EMC Secure Remote Service (ESRS) is a secure, two way remote connection between EMC and your EMC products. It heartbeats routinely to EMC for checking connectivity as well as product status. It also has nice integration into MyService360 and CloudIQ to provide even more support data. It has a nice web based interface, but I like doing as much as possible via command line when I have the option. We have more than one ESRS GW server so saving any time reviewing each multiplies across the enterprise.

In this case, I'm not looking to do any changes or additions. I want to be able to basically 'check the checker'. Are all ESRS services running? What devices do I have configured? Do I have them all? Status of each? Sometimes we can have network changes or problems impact connectivity to EMC via the internet so I want to monitor and report on that automatically if possible.

After doing some looking on the ESRS GW, I found a command line utility folder as noted below. To my knowledge there is no documentation on this tool, at least I've not found any. Running the script and providing credentials, I was able to successfully run a --service-status command.

esrshost:/opt/esrs/utilities/commandlineutil # ls -ltr
total 2456
-rwxr-xr-x 1 root root     138 Nov 18  2016 ESRS-VE-CLI-Util.sh
-rwxr-xr-x 1 root root 2501636 Nov 18  2016 CommandLineUtil.jar
drwxr-xr-x 2 root root    4096 Sep 20  2017 logs

esrshost:/opt/esrs/utilities/commandlineutil # ./ESRS-VE-CLI-Util.sh --service-status

Please enter ESRS VE WebUI credentials to use this utility.
Enter User Name: esrsuser
Enter Password:

---------------------------------------------------------------------------------------
RESTFul Services
------------------
        Service            Status                    Description
---------------------------------------------------------------------------------------
esrsalarm                 Running         Service to create alarms
esrsauditlogging          Running         Service to audit the ESRS VE activities
esrsauth                  Running         Authorization service
esrsusermanagement        Running         Manage Users service
esrsconfigtool            Running         Service for ESRS VE configuration setup
esrsconnectivityreport    Running         Service to run the Connectivity checks
esrsdataitems             Running         Service to manage data items
esrsdevicemanagement      Running         Device management service for ESRS VE
esrsjcemc                 Running         ConnectEMC Restful Service
esrskeepalive             Running         Keepalive service for managed devices
esrsmftauth               Running         MFT Service
esrsupdate                Running         Service to check for  ESRS VE updates
esrsremotescripts         Running         Service to update remote scripts status
esrsrsc                   Running         Service for Secure Credentials
esrsvesp                  Running         ESRS Secure Service Provider

Core Services
--------------
        Service            Status                    Description
---------------------------------------------------------------------------------------
esrsclient                Running         ESRS VE core agent
esrshttpd                 Running         ESRS Apache for inbound traffic
esrshttpdR                Running         ESRS Apache for outbound traffic
esrsconnectemc            Running         ConnectEMC service for Connect Home files
esrsclientproxy           Running         ESRSVE as a proxy service
esrswatchdog              Running         Watchdog service to monitor the ESRS VE core services
esrshttpdftp              Running         FTP service for Connect Home files
esrshttpdlistener         Running         HTTPS listener service for Connect Home files
postfix                   Running         SMTP service for Connect Home files
shibd                     Running         Shibboleth interface for ESRS SP
apache2                   Running         ESRS Apache for RSC traffic
--------------------END OF SERVICES STATAUS--------------------------------------------

Nice, but I want to run this remotely. Assuming root account for the ESRS GW, I tried the following:

user@remhost:/home/user> ssh root@esrshost /opt/esrs/utilities/commandlineutil/ESRS-VE-CLI-Util.sh --service-status
Password:
Error: Cannot find jar file

I looked at the sh script and see that the jar file is not fully pathed, causing the error above.

esrshost:/opt/esrs/utilities/commandlineutil # cat ESRS-VE-CLI-Util.sh
#!/bin/bash

JARFILE="CommandLineUtil.jar"

if [ -f $JARFILE ]; then
        java -jar $JARFILE "$@"
else
        echo "Error: Cannot find jar file"
fi

Based on the simple script design, I'm simply going to by-pass the script and call the java directly as noted in the script.

user@remhost:/home/user> ssh root@esrshost "java -jar /opt/esrs/utilities/commandlineutil/CommandLineUtil.jar --service-status"
Password:
Couldn't get Console instance

Well, making some progress. Now getting a complaint from the jar program. Based on the console instance error, we'll need to utilize the -t option of ssh.

-t      Force pseudo-tty allocation.  This can be used to execute arbitrary screen-based programs on a remote machine, which can be
        very useful, e.g. when implementing menu services.  Multiple -t options force tty allocation, even if ssh has no local tty.

Trying the remote call again...

user@remhost:/home/user> ssh -t root@esrshost "java -jar /opt/esrs/utilities/commandlineutil/CommandLineUtil.jar --service-status"
Password:

Please enter ESRS VE WebUI credentials to use this utility.
Enter User Name: esrsuser
Enter Password:

---------------------------------------------------------------------------------------
RESTFul Services
------------------
        Service            Status                    Description
---------------------------------------------------------------------------------------
esrsalarm                 Running         Service to create alarms
esrsauditlogging          Running         Service to audit the ESRS VE activities
esrsauth                  Running         Authorization service
esrsusermanagement        Running         Manage Users service
esrsconfigtool            Running         Service for ESRS VE configuration setup
esrsconnectivityreport    Running         Service to run the Connectivity checks
esrsdataitems             Running         Service to manage data items
esrsdevicemanagement      Running         Device management service for ESRS VE
esrsjcemc                 Running         ConnectEMC Restful Service
esrskeepalive             Running         Keepalive service for managed devices
esrsmftauth               Running         MFT Service
esrsupdate                Running         Service to check for  ESRS VE updates
esrsremotescripts         Running         Service to update remote scripts status
esrsrsc                   Running         Service for Secure Credentials
esrsvesp                  Running         ESRS Secure Service Provider

Core Services
--------------
        Service            Status                    Description
---------------------------------------------------------------------------------------
esrsclient                Running         ESRS VE core agent
esrshttpd                 Running         ESRS Apache for inbound traffic
esrshttpdR                Running         ESRS Apache for outbound traffic
esrsconnectemc            Running         ConnectEMC service for Connect Home files
esrsclientproxy           Running         ESRSVE as a proxy service
esrswatchdog              Running         Watchdog service to monitor the ESRS VE core services
esrshttpdftp              Running         FTP service for Connect Home files
esrshttpdlistener         Running         HTTPS listener service for Connect Home files
postfix                   Running         SMTP service for Connect Home files
shibd                     Running         Shibboleth interface for ESRS SP
apache2                   Running         ESRS Apache for RSC traffic
--------------------END OF SERVICES STATAUS--------------------------------------------

Connection to esrshost closed.

Perfect. Now I have a functioning remote call to the ESRS CLI. Using some Expect I can automate these commands to retrieve the data. These are the commands I'll probably be using most.

--agent-status
--service-status
--device-list

The complete list of supported commands and description:
To Display Agent Status Information :  --agent-status
To Display Active Remote Sessions   :  --remote-session
To Display ESRS Related Service     :  --service-status
To List the Managed Devices         :  --device-list
To View Policy Manager              :  --view-policymanager
To Add Policy Manager               :  --add-policymanager
To Remove Policy Manager            :  --remove-policymanager
To Enable Proxy Configuration       :  --add-proxy
To View Proxy Configuration         :  --view-proxy
To Remove Proxy Configuration       :  --remove-proxy
To Display Help Details             :  --help

Happy ESRS monitoring.

CentOS7 Root Password Recovery

Highly condensed set of steps to recover root password. Tested in RHEL7, CentOS7 and Fedora29.

1. At grub boot menu, press 'e' key on desired menu item.
2. Scroll down to line with linux[16] vmlinuz
3. Append the following options to the end of this line.
   rd.break enforcing=0 systemd.unit=emergency.target
4. Ctrl-x
5. mount -o remount,rw /sysroot
6. chroot /sysroot
7. passwd #enter new root password as prompted
8. exit  #chroot
9. exit  #continue original boot
   Allow system to boot emergency target. Do not issue a reboot here.
10. Login with new root password
11. ls -lZ /etc|grep unlabeled
    Likely the following two files will be unlabeled:
    - /etc/shadow
    - /etc/fstab
12. restorecon /etc/shadow
13. restorecon /etc/fstab
14. reboot  #allow normal boot