Accessing the Anaconda Installer Remotely
So you're stuck in the Anaconda installer and can't copy files to and from to troubleshoot an issue?
Could not load host key /etc/ssh/ssh_host_rsa_key
or you're getting something like this after some FS XFS corruption due to storage failure:
[ 358.344254] XFS (dm-0): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x3c44aa0
[ 358.349741] XFS (dm-0): Unmount and run xfs_repair
[ 358.352350] XFS (dm-0): First 64 bytes of corrupted metadata buffer:
[ 358.355398] ffff92237a9e2000: 35 64 64 39 38 38 37 36 35 34 36 65 39 30 65 66 5dd98876546e90ef
[ 358.359107] ffff92237a9e2010: 66 65 36 32 64 38 34 30 37 37 62 65 36 66 33 37 fe62d84077be6f37
[ 358.362794] ffff92237a9e2020: 62 62 37 65 62 63 31 66 61 37 31 38 65 35 36 64 bb7ebc1fa718e56d
[ 358.366488] ffff92237a9e2030: 34 30 62 32 64 33 63 36 35 30 34 66 33 62 38 66 40b2d3c6504f3b8f
[ 358.370229] XFS (dm-0): metadata I/O error: block 0x3c44aa0 ("xfs_trans_read_buf_map") error 117 numblks 16
[ 358.375987] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[ 358.379058] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
Run the following commands to enable networking and the SSHD daemon:
- ssh-keygen
- cp /root/.ssh/id_rsa /etc/ssh/ssh_host_rsa_key
- cp /etc/ssh/sshd_config.anaconda /etc/ssh/sshd_config
- /sbin/sshd -p 22
- ip addr add 10.0.0.100/24 dev ens3
- (THIS) ip route add 10.0.0.0/24 via 10.0.0.1 dev ens3
- (OR) ip route add default via 10.0.0.1 dev ens3
- ip link set ens3 up
SSH in to troubleshoot:
[root@remote01 ~]# ssh 10.0.0.100
Warning: Permanently added '10.0.0.100' (RSA) to the list of known hosts.
[anaconda root@localhost ~]#
Particularly handy when troubleshooting boot issues. Running tail -f /tmp/*.log will now be possible remotely while troubleshooting boot problems.
Use the same procedure to access corrupted filesystems on environments.
BONUS: Repair XFS Filesystem in Maintenance
Here's how to recover a corrupted XFS filesystem by logging into the system using the above steps.
vgscan -v –mknodes
vgchange -a y
ls -altri /dev/centos/*
Verify LV's exist. Repair. Example:
xfs_repair /dev/centos/home
xfs_repair /dev/centos/root
If you run into this:
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption — please attempt a mount
of the filesystem before doing this.
Simply follow instructions and mount it:
[anaconda root@localhost ~]# mkdir lv-root
[anaconda root@localhost ~]# mount /dev/centos/root lv-root
[anaconda root@localhost ~]# cd lv-root/
[anaconda root@localhost lv-root]# ls
bin dev home lib64 mds.xyz misc n nix.mds.xyz proc run srv sys usr
boot etc lib lost+found media mnt net opt root sbin sssd-updates.txt tmp var
[anaconda root@localhost lv-root]#
This replays the journal log. After this, you should be able to boot into the system.
BONUS: Install RPM in recovery.
Once you complete the above, mount the failed systems folders over a directory of your choice. For example:
mkdir lv-root; mount /dev/mapper/centos-root lv-root
mkdir lv-home mount /dev/mapper/centos-home lv-home
chroot lv-root
mknod -m 666 /dev/random c 1 8
mknod -m 666 /dev/urandom c 1 9
Install and update a package in an attempt to fix the environment.
[anaconda root@localhost /]# yum search kernel
================== N/S matched: kernel ===========================
<SOME PACKAGE>.x86_64 : The Linux kernel
Then install:
[anaconda root@localhost /]# yum update <SOME PACKAGE>.x86_64
Installed:
<SOME PACKAGE.x86_64 0:3.10.0-1062.18.1.el7
Dependency Updated:
<SOME PACKAGE DEPENDENCY>.noarch 0:20190429-72.gitddde598.el7
Complete!
[anaconda root@localhost /]#
BONUS: Install RPM in recovery. ( METHOD 2 )
- Mount the ISO or DVD. Ideally the Everything DVD of CentOS.
- Select Troubleshooting -> Rescue a CentOS Linux System -> Continue.
- Once on the command prompt, type cd /mnt/install/repo/Packages
- mkdir /mnt/sysimage/root/kernel; cp -ip kernel* /mnt/sysimage/root/kernel
- chroot /mnt/sysimage
- cd /root/kernel
- yum install kernel*
- ( Optional ) Edit /etc/default/grub in case custom parameters are needed.
- grub2-mkconfig -o /boot/grub2/grub.cfg
- exit
- sync;sync; reboot
- Login and Enjoy!
The above steps are a great replacement for single user mode such as using single or init=/bin/bash. The filesystems on the original OS remain unmounted allowing a full range of troubleshooting options.
Hope this helps.
Thx,