Header Shadow Image


Setup a USB Null Modem for Kernel Dump Captures

We will setup a serial null modem cable for administering and connecting to a physical machine via another in the event that:  

1) We want to capture kernel crashes and dumps.  
2) Login to the machine machine remotely via another linux box to do things like restart the network.  

For this we will need:  

1) One of DB9 RS232 Serial Null Modem Cable F/F
2) Two of USB to RS232 Serial Port DB9 9 Pin Male

Connect the USB to Serial Adapter to both systems.  There is only one possible way to connect these between systems:

https://www.microdevsys.comWordPressImages/USB-To-Serial-RS232-Null-Model-Console-Connection.png

Following the above, set the tty specifc settings on ttyUSB0 on PC1:

6889  stty -F /dev/ttyUSB0 115200 cs8 -cstopb -parenb
6890  stty -F /dev/ttyUSB0 -a

 

Test the serial connection by running the following:

6894  /sbin/agetty -L 115200 ttyUSB0
 

Use minicom from the connecting linux host.  When test running /sbin/agetty -L 115200 ttyUSB0, you should see a prompt:

[root@rfc1178-01 ~]# minicom

Welcome to minicom 2.6.2

OPTIONS: I18n
Compiled on Jun 25 2013, 10:33:48.
Port /dev/ttyUSB0, 11:30:08

Press CTRL-A Z for help on special keys

Scientific Linux release 6.10 (Carbon)
Kernel 4.18.19 on an x86_64

mbpc-pc login: root
Password:
Last login: Fri Apr 19 12:51:19 from 192.168.0.76
0;root@mbpc-pc:~[root@mbpc-pc ~]#
0;root@mbpc-pc:~[root@mbpc-pc ~]#
0;root@mbpc-pc:~[root@mbpc-pc ~]#
0;root@mbpc-pc:~[root@mbpc-pc ~]# uptime
 13:03:19 up 14 min,  1 user,  load average: 0.06, 0.13, 0.18
0;root@mbpc-pc:~[root@mbpc-pc ~]#

 

You should be able to login as above confirming the physical layer (USB to Serial -> Null Modem Female-to-Female -> Serial to USB) functions correctly and root is allowed to login.  Configure the kernel to send messages on the tty:

title Scientific Linux (4.18.19)
        root (hd0,0)
        kernel /vmlinuz-4.18.19 ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-:768M pci=nomsi nohpet clocksource=rtc console=ttyUSB0,115200n8 console=tty0

Configure:

[root@mbpc-pc ~]# cat /etc/securetty |grep USB
ttyUSB0
[root@mbpc-pc ~]# cat /etc/init/ttyUSB0.conf
# ttyUSB0 – agetty
#
# This service maintains a agetty on ttyUSB0.

stop on runlevel [S06]
start on runlevel [12435]

respawn
exec agetty -L /dev/ttyUSB0 115200
[root@mbpc-pc ~]#

 

Configure the minicom settings on the external host (CTRL – A, followed by Z.  Look for option cOnfigure Minicom..O or directly using CTRL – A followed by O):

+—–[configuration]——+
| Filenames and paths      |
| File transfer protocols  |
| Serial port setup        |
| Modem and dialing        |
| Screen and keyboard      |
| Save setup as dfl        |
| Save setup as..          |
| Exit                     |
+————————–+

Followed by the settings below:

+———————————————————————–+
| A –    Serial Device      : /dev/ttyUSB0                              |
|                                                                       |
| C –   Callin Program      :                                           |
| D –  Callout Program      :                                           |
| E –    Bps/Par/Bits       : 115200 8N1                                |
| F – Hardware Flow Control : No                                        |
| G – Software Flow Control : Yes                                       |
|                                                                       |
|    Change which setting?                                              |
+———————————————————————–+

Hit ESC when done and save the configuration:

| Save setup as dfl        |

Restart the server to ensure changes take effect.  You should now see messages from the minicom terminal on the secondary system:

Welcome to minicom 2.6.2

OPTIONS: I18n
Compiled on Jun 25 2013, 10:33:48.
Port /dev/ttyUSB0, 12:03:52

Press CTRL-A Z for help on special keys


Scientific Linux release 6.10 (Carbon)
Kernel 4.18.19 on an x86_64

mbpc-pc login:

Next, test restart with the console connected to see restart messages being printed:

Linux version 4.18.19 (root@mbpc-pc) (gcc version 4.4.7 201209
Command line: ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_8
x86/fpu: x87 FPU will use FXSAVE
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x0000000000093fff] usable
BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000dfceffff] usable
BIOS-e820: [mem 0x00000000dfcf0000-0x00000000dfcf0fff] ACPI NVS
BIOS-e820: [mem 0x00000000dfcf1000-0x00000000dfcfffff] ACPI data
BIOS-e820: [mem 0x00000000dfd00000-0x00000000dfdfffff] reserved
BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000011fffffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.4 present.
DMI: Gigabyte Technology Co., Ltd. GA-890XA-UD3/GA-890XA-UD3, BIOS FC 08/02/2010
AGP: No AGP bridge found

 

Testing can be done using this:

[root@mbpc-pc cores]# echo "This is a ttyUSB0 test from mbpc-pc." > /dev/ttyUSB0
[root@mbpc-pc cores]#

 

Result on the console is:

[root@mbpc-pc ~]# This is a ttyUSB0 test from mbpc-pc.
CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.6.2  | VT102 | Online 08:12

 

If you get a prompt but no kernel messages, ensure you compile the following options into the kernel:

CONFIG_USB_SERIAL=y
CONFIG_USB_SERIAL_CONSOLE=y
CONFIG_USB_SERIAL_EDGEPORT_TI=y
CONFIG_USB_SERIAL_MOS7840=y

You can find the above in the make menuconfig driver sections.  You can find the above by pressing forward slash ( / ) followed by the search string CONFIG_USB_SERIAL which will give you the path of the option:


  |   Location:                                               |
  |     -> Device Drivers                                     |
  |       -> USB support (USB_SUPPORT [=y])                   |
  |         -> USB Serial Converter support (USB_SERIAL [=y]) |

 

If you get kernel messages but no prompt (after enabling additional kernel parameters above) then try adding the following additional parameters:

[root@mbpc-pc linux-4.18.19]# cat /etc/init/ttyUSB0.conf
# ttyUSB0 – agetty
#
# This service maintains a agetty on ttyUSB0.

stop on runlevel [S06] and (
            not-container or
            container CONTAINER=lxc or
            container CONTAINER=lxc-libvirt)

start on runlevel [12435]

respawn
exec agetty -L /dev/ttyUSB0 115200 vt100
[root@mbpc-pc linux-4.18.19]#

 

However for us it was just a matter of restarting against since agetty didn't come up the first time.  If with the addition of the above items in green you now get a console, all is good and you should be all set to capture the kernel messages when crashes happen!

April 10 2020

Testing a new connection using the same equipment, to capture kernel panic messages.

https://i0.wp.com/www.microdevsys.com/WordPressImages/USB-To-Serial-RS232-Null-Model-Console-Connection-Test-Session.png?ssl=1

NOTE: Cable connection is also important.  If other devices particularly RS232's are already connected to some of the ports, the messages will travel to those RS232 devices. meaning you'll either need to plug the USB RS232 to that correct USB port or swap devices on ports.

With the correct cabling, the following should be visible.

https://i1.wp.com/www.microdevsys.com/WordPressImages/USB-To-Serial-RS232-Null-Model-Console-Connection-Test-Session-Correct-Cabling.png?ssl=1

REF: https://wiki.freepbx.org/display/PC/Capturing+Kernel+Panic+via+Serial+Port

Cheers,
TK


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License