Header Shadow Image

Accessing the Anaconda Installer Remotely

So you're stuck in the Anaconda installer and can't copy files to and from to troubleshoot an issue?  

Could not load host key /etc/ssh/ssh_host_rsa_key

or you're getting something like this after some FS XFS corruption due to storage failure:  

[  358.344254] XFS (dm-0): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x3c44aa0
[  358.349741] XFS (dm-0): Unmount and run xfs_repair
[  358.352350] XFS (dm-0): First 64 bytes of corrupted metadata buffer:
[  358.355398] ffff92237a9e2000: 35 64 64 39 38 38 37 36 35 34 36 65 39 30 65 66  5dd98876546e90ef
[  358.359107] ffff92237a9e2010: 66 65 36 32 64 38 34 30 37 37 62 65 36 66 33 37  fe62d84077be6f37
[  358.362794] ffff92237a9e2020: 62 62 37 65 62 63 31 66 61 37 31 38 65 35 36 64  bb7ebc1fa718e56d
[  358.366488] ffff92237a9e2030: 34 30 62 32 64 33 63 36 35 30 34 66 33 62 38 66  40b2d3c6504f3b8f
[  358.370229] XFS (dm-0): metadata I/O error: block 0x3c44aa0 ("xfs_trans_read_buf_map") error 117 numblks 16
[  358.375987] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[  358.379058] XFS (dm-0): Please umount the filesystem and rectify the problem(s)

Run the following commands to enable networking and the SSHD daemon:  

  1. ssh-keygen
  2. cp /root/.ssh/id_rsa /etc/ssh/ssh_host_rsa_key
  3. cp /etc/ssh/sshd_config.anaconda /etc/ssh/sshd_config
  4. /sbin/sshd -p 22
  5. ip addr add dev ens3
  6. (THIS) ip route add via dev ens3
  7. (OR) ip route add default via dev ens3
  8. ip link set ens3 up

SSH in to troubleshoot:  

[root@remote01 ~]# ssh
Warning: Permanently added '' (RSA) to the list of known hosts.
[anaconda root@localhost ~]#

Particularly handy when troubleshooting boot issues. Running tail -f /tmp/*.log will now be possible remotely while troubleshooting boot problems.  

Use the same procedure to access corrupted filesystems on environments.  

BONUS: Repair XFS Filesystem in Maintenance

Here's how to recover a corrupted XFS filesystem by logging into the system using the above steps.

vgscan -v –mknodes
vgchange -a y
ls -altri /dev/centos/*

Verify LV's exist.  Repair.  Example:

xfs_repair /dev/centos/home
xfs_repair /dev/centos/root

If you run into this:

ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption — please attempt a mount
of the filesystem before doing this.

Simply follow instructions and mount it:

[anaconda root@localhost ~]# mkdir lv-root
[anaconda root@localhost ~]# mount /dev/centos/root lv-root
[anaconda root@localhost ~]# cd lv-root/
[anaconda root@localhost lv-root]# ls
bin   dev  home  lib64       mds.xyz  misc  n    nix.mds.xyz  proc  run   srv               sys  usr
boot  etc  lib   lost+found  media    mnt   net  opt          root  sbin  sssd-updates.txt  tmp  var
[anaconda root@localhost lv-root]#

This replays the journal log.  After this, you should be able to boot into the system. 

BONUS: Install RPM in recovery.

Once you complete the above, mount the failed systems folders over a directory of your choice.  For example: 

mkdir lv-root; mount /dev/mapper/centos-root lv-root
mkdir lv-home mount /dev/mapper/centos-home lv-home

chroot lv-root

mknod -m 666 /dev/random c 1 8
mknod -m 666 /dev/urandom c 1 9

Install and update a package in an attempt to fix the environment.

[anaconda root@localhost /]# yum search kernel
================== N/S matched: kernel ===========================
<SOME PACKAGE>.x86_64 : The Linux kernel

Then install:

[anaconda root@localhost /]# yum update <SOME PACKAGE>.x86_64
  <SOME PACKAGE.x86_64 0:3.10.0-1062.18.1.el7

Dependency Updated:
  <SOME PACKAGE DEPENDENCY>.noarch 0:20190429-72.gitddde598.el7

[anaconda root@localhost /]#

BONUS: Install RPM in recovery. ( METHOD 2 )

  1. Mount the ISO or DVD.  Ideally the Everything DVD of CentOS.
  2. Select Troubleshooting -> Rescue a CentOS Linux System -> Continue.
  3. Once on the command prompt, type cd /mnt/install/repo/Packages
  4. mkdir /mnt/sysimage/root/kernel; cp -ip kernel* /mnt/sysimage/root/kernel
  5. chroot /mnt/sysimage
  6. cd /root/kernel
  7. yum install kernel*
  8. ( Optional )  Edit /etc/default/grub in case custom parameters are needed.
  9. grub2-mkconfig -o /boot/grub2/grub.cfg
  10. exit
  11. sync;sync; reboot
  12. Login and Enjoy!

The above steps are a great replacement for single user mode such as using single or init=/bin/bash.  The filesystems on the original OS remain unmounted allowing a full range of troubleshooting options. 

Hope this helps.


Leave a Reply

You must be logged in to post a comment.

  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License