After a recent upgrade, system wouldn’t boot to a UI. Blank screen is shown. To solve this, hit SHIFT + F2 at the blank screen, to switch to a terminal login screen. Once at the login screen, change directory to /lib/modules/:
cd /lib/modules/
and type:
ls -altri
You may see that the kernel version has been updated, potentially unsuccessfully so no network drivers can be loaded. Try to boot with an older kernel, which was the case for us, and that allowed us to get into the system since the previous kernel apparently has the proper network drivers to load at boot time, to bring Network connectivity back.
However, how did the system error out like that in the first place? Turns out that the WineHQ for Ubuntu had a broken repository list, erroring out any apt update. Not 100% sure or if WineHQ updated anything in the last few days, but an:
apt update
appears to have been ran and did not properly complete. Appears this resulted in an incomplete kernel being updated or installed. Again not 100% sure however, there were more package definitions in:
/etc/apt/source.list.d/
for WineHQ that had to be removed. Once this was done, running:
apt update
or
apt upgrade
completed successfully and the updated packages for the kernel were downloading, fixing the latest Kernel as well. This fixed the issue permanently, and the need to use a previous kernel to get around the boot issue above, was resolved.
To get the latest version of OpenVPN, which will be 2.6 since files for 3.0 are still not yet available at the time of this article, we first need to import the OpenVPN repository key:
Verify the repository is working correctly by running an apt-get update:
# apt-get update
or
apt update
Followed by:
# apt install openvpn
Verify with:
# dpkg -l | grep -Ei vpn
Following this, configure your OpenVPN Client as follows. For the purpose of this article, assumption will be made that all the necessary files to connect to an OpenVPN Servers have been generated and sent over in a file such as user-vpn-config.zip or equivalent ( Replace user-vpn-cnofig.zip with the actual zip name sent to you. ):
Copy the zip then extract the files to a folder such as:
# mkdir ~/openvpn # cd ~/openvpn # wget https://SOURCE-URL.com/SOURCE-PATH/user-vpn=config.zip # Replace the uppercase text with the actual path provided by your OpenVPN Server provider. # unzip user-vpn-config.zip
The ‘~’ indicates the home folder of the currently logged in user. To find the current user, type:
# who am i
at the terminal prompt. Use a line such as this to verify the contents of the .ovpn config file:
# grep -Eiv “#|;” openvpn.domain.com-user.ovpn | sed ‘/^[[:space:]]*$/d’ client dev tun2 proto tcp remote openvpn.domain.com 12345 resolv-retry infinite nobind persist-key persist-tun ca ca.crt cert user.crt key user.key remote-cert-tls server tls-auth ta.key 1 data-ciphers AES-256-GCM:AES-128-GCM data-ciphers-fallback AES-256-CBC auth SHA256 comp-lzo no verb 3 auth-nocache tun-mtu 1500 #
Time to connect. Issue the following to connect to the remote site:
# openvpn openvpn.domain.com-user.ovpn
Important to note that when running the above command, it will not exit in that terminal window. This is expected. To access the remote host resources, either use another terminal tab or just another plain terminal or simply minimize the terminal running the OpenVPN command above, and use the rest of your workstation as if it was connected directly to your ISP.
Try to access some of the resources on the remote site. See your OpenVPN Server or ISP emails for further details.
(Sun Mar 10 10:09:21 2024) [[sssd[ldap_child[2407]]]] [ldap_child_get_tgt_sync] (0x0010): Failed to init credentials: Cannot contact any KDC for realm ‘NIX.MDS.XYZ’
Mar 10 11:46:32 idmipa01.nix.mds.xyz ns-slapd[1587]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_993))
# vi /var/log/krb5kdc.log Mar 09 23:40:37 idmipa01.nix.mds.xyz krb5kdc[1114](info): Didn’t connect to LDAP on startup: 110 krb5kdc: Server error – while fetching master key K/M for realm NIX.MDS.XYZ
mdscisco07#sh mac address-table interface gigabitEthernet 1/19 Unicast Entries vlan mac address type protocols port ——-+—————+——–+———————+——————– 5 abcd.1234.wxyz dynamic ip,ipx GigabitEthernet1/19
Building an Ubuntu Workstation for development and general use, one that can be accessed remotely with central authentication. Most of the commands below will be ran as the root user, hence
# sudo su –
to root will be needed. Let’s get going:
ALIAS SETUP
Personally, the following alias just makes it a tad easier to get around:
$ grep -Ei altri ~/.bashrc alias lt=’ls -altri’ $
Some of the other commands that can come in handy is the bash search CTRL + r, allowing history searches and recalling previous commands.
Setup VNC for a number of users. This file is global and hence, owned by root:
# cat /etc/tigervnc/vncserver.users # TigerVNC User assignment # # This file assigns users to specific VNC display numbers. # The syntax is <display>=<username>. E.g.: # # :2=andrew # :3=lisa
This will work but won’t allow root direct login via VNC. Next, configure multiple instances to start on bootup, one for a non-privileged user. To do so, the following line can be used to start sessions for each user:
The systemd startup script calls yet another, this time, a bash script. The below copy is highly modified for the purpose of this article:
# cat -n /usr/libexec/tigervncsession-start
1 #!/bin/bash
2 #
3 # Copyright 2019 Pierre Ossman for Cendio AB
4 #
5 # This is free software; you can redistribute it and/or modify
6 # it under the terms of the GNU General Public License as published by
7 # the Free Software Foundation; either version 2 of the License, or
8 # (at your option) any later version.
9 #
10 # This software is distributed in the hope that it will be useful,
11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13 # GNU General Public License for more details.
14 #
15 # You should have received a copy of the GNU General Public License
16 # along with this software; if not, write to the Free Software
17 # Foundation, Inc., 59 Temple Place – Suite 330, Boston, MA 02111-1307,
18 # USA.
19 #
20
21 USERSFILE=”/etc/tigervnc/vncserver.users”
22 echo “Using configuration file $USERSFILE … ”
23
24 if [[ $# -ne 1 ]]; then
25 echo “Syntax:” >&2
26 echo ” $0 <display>” >&2
27 echo “Example:”
28 echo ” $0 \”:1\”” >&2
29 exit 1
30 fi
31
32 if [[ ! -f “${USERSFILE}” ]]; then
33 echo “Users file ${USERSFILE} is missing. Exiting!” >&2
34 exit 1
35 fi
36
37 DISPLAY=”$1″
38
39 # Check if that server is already running.
40 for VNCENTRY in $(/usr/bin/tigervncserver –list|awk ‘{ if ( $1 ~ /[0-9]+/ ) print $1 }’); do
41 echo “Checking if $VNCENTRY is already running …”;
42 if [[ $VNCENTRY -eq $DISPLAY ]]; then
43 echo “ERROR: Server $VNCENTRY is already running. If intent is to restart, issue a stop and start on that server:”;
44 echo ” systemctl stop tigervncserver@1″;
45 echo ” systemctl start tigervncserver@1″;
46 echo “OR issue the following”;
47 echo ” /usr/bin/tigervncserver –kill :$DISPLAY”;
48 exit 1;
49 fi
50 done
51
52 USER=$(grep “^[ ]*:${DISPLAY}=” “${USERSFILE}” 2>/dev/null | head -1 | cut -d = -f 2- | sed ‘s/ *$//g’)
53
54 echo “User for display $DISPLAY identified as \”$USER\” from file $USERSFILE …”;
55 echo “Users defined in config file are:”
56 echo “———————————”
57 grep -Eiv “#” $USERSFILE;
58 echo “———————————”
59
60 if [[ -z “${USER}” ]]; then
61 echo “No user configured for display ${DISPLAY}” >&2
62 exit 1
63 else
64 echo “Starting VNC Server as user ${USER} … ”
65 fi
66
67 # exec “/usr/sbin/tigervncsession” “${USER}” “:${DISPLAY}”
68 /usr/sbin/tigervncsession “${USER}” “:${DISPLAY}”;
69 echo “Return code from running /usr/sbin/tigervncsession is: $?”;
70
71 echo “Checking if PID file has been created.”
72 vnccount=0;
73 while [[ true ]]; do
74 vnccount=$((vnccount + 1))
75 if [[ -r /run/tigervncsession-:$DISPLAY.pid ]]; then
76 echo “PID File created by /usr/sbin/tigervncsession exists in /run/tigervncsession-:$DISPLAY.pid …”;
77 echo “Successfully started the TigerVNC Server on display port $DISPLAY. “;
78 break;
79 else
80 echo “PID Process file /run/tigervncsession-:$DISPLAY.pid not yet created. Waiting …”;
81 sleep 1;
82 fi
83
84 if [[ $vnccount -gt 10 ]]; then
85 echo “ERROR: TigerVNC process startup failed to create a PID file in /run/tigervncsession-:$DISPLAY.pid …”;
86 echo “ERROR: This indicares issues in startup. Exiting!”;
87 exit 1;
88 fi
89 done
At this point, looks like everything has a decent chance of working when started up. Before plowing ahead, check and kill any previous VNC sessions that might still be hanging around started manually. Need to do this to prevent errors particularly with port conflicts.
X DISPLAY # RFB PORT # RFB UNIX PATH PROCESS ID # SERVER root@fr-ubuntu-01:~/.vnc# root@fr-ubuntu-01:~/.vnc# root@fr-ubuntu-01:~/.vnc# /usr/bin/tigervncserver –kill :* tigervncserver: No VNC server running for this user! root@fr-ubuntu-01:~/.vnc#
And connect using a VNC Viewer application installed on your Windows 10/11 or installed on a Linux Desktop. A few options exist including RealVNC and TightVNC:
PROBLEMS and FIXES
When encountering the following:
Jan 20 20:14:35 fr-ubuntu-01 systemd[40475]: tigervncserver@1.service: Failed to determine user credentials: No such process Jan 20 20:14:35 fr-ubuntu-01 systemd[40475]: tigervncserver@1.service: Failed at step USER spawning /usr/libexec/tigervncsession-start: No such process
Fix the above by adding a user to the service:
# cat /lib/systemd/system/tigervncserver@.service [Unit] Description=Remote desktop service (VNC) After=network.target
Jan 20 20:16:49 fr-ubuntu-01 tigervncsession-start[40631]: No user configured for display 1
Fix by adding a user to this file:
# cat /etc/tigervnc/vncserver.users # TigerVNC User assignment # # This file assigns users to specific VNC display numbers. # The syntax is <display>=<username>. E.g.: # # :2=andrew # :3=lisa
:1=root :2=vncuser
Next issue:
# tail -f /var/log/syslog /var/log/auth.log Jan 21 10:09:04 fr-ubuntu-01 gdm-password]: pam_succeed_if(gdm-password:auth): requirement “user != root” not met by user “root”
Means that the system is not configured to allow direct root login. This is a security feature, worth having. There are ways to enable this but it’s not recommended. Best to configure access to a non-privileged user. Hence, let’s configure for that above.
Another issue that can be seen is:
Jan 21 13:03:24 fr-ubuntu-01 systemd[1]: tigervncserver@1.service: Can’t open PID file /run/tigervncsession-1.pid (yet?) after start: Operation not permitted
this required a small update to the startup script:
/usr/libexec/tigervncsession-start
by adding in the following lines and removing the exec:
52 # exec "/usr/sbin/tigervncsession" "${USER}" ":${DISPLAY}"
53 /usr/sbin/tigervncsession "${USER}" ":${DISPLAY}"
54 echo "Return code from running /usr/sbin/tigervncsession is: $?"
55 echo "Checking if PID file has been created."56 vnccount=057 while [[ true ]]; do58 vnccount=$((vnccount + 1))59 if [[ -r /run/tigervncsession-:$DISPLAY.pid ]]; then60 echo "PID File created by /usr/sbin/tigervncsession exists in /run/tigervncsession-:$DISPLAY.pid ...";61 echo "Successfully started the TigerVNC Server on display port $DISPLAY. ";62 else63 echo "PID Process file /run/tigervncsession-:$DISPLAY.pid not yet created. Waiting ...";64 sleep 1;65 fi6667 if [[ $vnccount -gt 10 ]]; then68 echo "ERROR: TigerVNC process startup failed to create a PID file in /run/tigervncsession-:$DISPLAY.pid ...";69 echo "ERROR: This indicares issues in startup. Exiting!";70 exit 1;71 fi72 done
and modify the startup script as follows since the services append a ‘:’ to the file name:
# vi /lib/systemd/system/tigervncserver@.service PIDFile=/run/tigervncsession-:%i.pid
then retry to start / restart the service.
Connectivity issue such as this:
# telnet fr-ubuntu-01.nix.mds.xyz 5902 Connecting To fr-ubuntu-01.nix.mds.xyz…Could not open connection to the host, on port 5902: Connect failed
is not due to the F/W, however, it’s good to check:
but it has a lot to do with missing startup files in the USER/.vnc/ folder, such as these, with the correct permissions:
root@fr-ubuntu-01:/home/francesco/.vnc# lt total 40 500 -rw——- 1 francesco francesco 8 Jan 14 23:57 passwd 1462 -rwx—— 1 francesco francesco 121 Jan 21 00:31 xstartup 1464 -rw——- 1 francesco francesco 177 Jan 21 12:20 config 34 drwxr-x— 15 francesco francesco 23 Jan 21 16:24 .. 1814 -rw-rw-r– 1 francesco francesco 6 Jan 21 16:24 fr-ubuntu-01.nix.mds.xyz:5902.pid 16 -rw-r–r– 1 francesco francesco 205 Jan 21 16:24 fr-ubuntu-01:2.log 504 -rw-rw-r– 1 francesco francesco 6083 Jan 21 16:27 fr-ubuntu-01.nix.mds.xyz:5902.log 496 drwxrwxr-x 2 francesco francesco 8 Jan 21 16:41 . root@fr-ubuntu-01:/home/francesco/.vnc# cat xstartup #!/bin/bash -x
Installing through the software center, bring in a version that’s dated and may have issues on newer OS versions like Ubuntu 22.04. Therefore, install directly from the latest package version from the following site:
Quick configuration to proxy AD requests through another server. Set the following on the proxy server. Note the frontend server is on port 443 whereas the backend server is on port 389:
Connection to https://idmipa01.nix.mds.xyz/ipa/json failed with [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failename mismatch, certificate is not valid for ‘idmipa01.nix.mds.xyz’. (_ssl.c:1007) Connection to https://idmipa02.nix.mds.xyz/ipa/json failed with [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failename mismatch, certificate is not valid for ‘idmipa02.nix.mds.xyz’. (_ssl.c:1007)
On the surface this message doesn’t make much sense. The certificate definitely matches the hostname:
Even from a locally installed Ubuntu server, resolution returns external IP addresses:
root@g73sw01:~# ping atlas-c01 PING atlas-c01.mds.xyz (3.64.163.50) 56(84) bytes of data. From _gateway (192.168.0.1) icmp_seq=1 Redirect Network(New nexthop: _gateway (192.168.0.6))
The Linux box is surprising since normally, it should get the IP and DNS from the netplan:
root@g73sw01:~# cat /etc/netplan/01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
version: 2
renderer: NetworkManager
ethernets:
enp5s0:
dhcp4: no
addresses: [192.168.0.15/24]
gateway4: 192.168.0.1
nameservers:
addresses: [192.168.0.224,192.168.0.46,192.168.0.51]
root@g73sw01:~#
Still, /etc/resolv.conf has the localhost IP as the nameserver, regardless what netplan has:
Turns out that the local DNS resolver service is running on Ubuntu resolving local domain’s with external IP’s. Turning this off:
root@g73sw01:~# systemctl status systemd-resolved ? systemd-resolved.service - Network Name Resolution Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2023-11-19 18:30:48 EST; 2s ago Docs: man:systemd-resolved.service(8) man:org.freedesktop.resolve1(5) https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients Process: 1750 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=0/SUCCESS) Main PID: 1750 (code=exited, status=0/SUCCESS) Status: "Shutting down..." CPU: 8.759s
Nov 19 17:23:24 g73sw01.nix.mds.xyz systemd-resolved[1750]: Grace period over, resuming full feature set (UDP> Nov 19 17:24:05 g73sw01.nix.mds.xyz systemd-resolved[1750]: Using degraded feature set TCP instead of UDP for> Nov 19 17:27:25 g73sw01.nix.mds.xyz systemd-resolved[1750]: Using degraded feature set UDP instead of TCP for> Nov 19 17:33:04 g73sw01.nix.mds.xyz systemd-resolved[1750]: Using degraded feature set UDP instead of UDP+EDN> Nov 19 17:38:19 g73sw01.nix.mds.xyz systemd-resolved[1750]: Using degraded feature set TCP instead of UDP for> Nov 19 18:03:09 g73sw01.nix.mds.xyz systemd-resolved[1750]: Using degraded feature set TCP instead of UDP for> Nov 19 18:30:48 g73sw01.nix.mds.xyz systemd[1]: Stopping Network Name Resolution... Nov 19 18:30:48 g73sw01.nix.mds.xyz systemd[1]: systemd-resolved.service: Deactivated successfully. Nov 19 18:30:48 g73sw01.nix.mds.xyz systemd[1]: Stopped Network Name Resolution. Nov 19 18:30:48 g73sw01.nix.mds.xyz systemd[1]: systemd-resolved.service: Consumed 8.759s CPU time. root@g73sw01:~#
resolves the issue:
C:\Windows\system32>ping atlas-c01.nix.mds.xyz
Pinging atlas-c01.nix.mds.xyz [10.0.0.77] with 32 bytes of data:
Control-C
^C
C:\Windows\system32>ping atlas-c01.nix.mds.xyz
digging in further to find out how this is configured. In this case we want to disable the resolution entirely, so the Ubuntu server doesn’t act as a DNS for the rest of the network:
In this case we want to only enable the following service:
systemctl restart resolvconf
and do not wish to have our Ubuntu server resolve for the rest of the network. So the following service will remain disabled:
systemctl disabled systemd-resolved
However, in other environments, as needed, the service can now be reenabled. But that was not the case here it appears. Had to enable both services:
root@g73sw01:~# vi /etc/resolv.conf root@g73sw01:~# vi /etc/resolvconf/resolv.conf.d/head root@g73sw01:~# root@g73sw01:~# root@g73sw01:~# root@g73sw01:~# systemctl restart systemd-resolved resolvconf root@g73sw01:~# grep -v "#" /etc/resolvconf/resolv.conf.d/head nameserver 192.168.0.224 nameserver 192.168.0.46 nameserver 192.168.0.51 domain nix.mds.xyz search mds.xyz nix.mds.xyz mws.mds.xyz root@g73sw01:~# cat /etc/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN # 127.0.0.53 is the systemd-resolved stub resolver. # run "systemd-resolve --status" to see details about the actual nameservers. nameserver 192.168.0.224 nameserver 192.168.0.46 nameserver 192.168.0.51 nameserver 127.0.0.53 search nix.mds.xyz mds.xyz mws.mds.xyz root@g73sw01:~#
Yet this didn’t work either. Finally, disabling the Ubuntu service altogether resolved it:
root@g73sw01:~# grep -v "#" /etc/resolv.conf nameserver 192.168.0.224 nameserver 192.168.0.46 nameserver 192.168.0.51 search nix.mds.xyz mds.xyz mws.mds.xyz root@g73sw01:~# systemctl disable systemd-resolved resolvconf Synchronizing state of resolvconf.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable resolvconf root@g73sw01:~#
Summary
What did work, is removing the resolvconf package and simply updating the DNS entries with nmcli (NetworkManager) commands like this:
root@tom-G73Sw:~# cd /etc/netplan/ root@tom-G73Sw:/etc/netplan# cat 01-network-manager-all.yaml # Let NetworkManager manage all devices on this system network: version: 2 renderer: NetworkManager ethernets: enp5s0: dhcp4: no addresses: [192.168.0.15/24] gateway4: 192.168.0.1 nameservers: addresses: [192.168.0.224,192.168.0.46,192.168.0.51] root@tom-G73Sw:/etc/netplan#
Then issue:
netplan try
so changes take effect. Via UI, go to Activities then Settings, select Network or WiFi, based on which interface you wish to set the IP on. Typically leaving the WiFi as DHCP might make some sense for redundancy. Your choice:
