Azure VM is not accessible /proc/net/route contains no routes

Problem

Oversudden Azure VMs running Ubuntu 18LTS became unreachable (WestUS region)
/proc/net/route contains no routes


Solution

Connected to Azure VM via Console and got following running lines of error messages

2019/05/30 18:24:20.747552 ERROR ExtHandler /proc/net/route contains no routes

Restart waagent service produced the following:

INFO Daemon Azure Linux Agent Version:2.2.32.2
2019/05/30 18:24:18.242022 INFO Daemon OS: ubuntu 18.04
2019/05/30 18:24:18.246632 INFO Daemon Python: 3.6.7
2019/05/30 18:24:18.251476 INFO Daemon Add daemon process pid 2330 to walinuxagent systemd cgroup
2019/05/30 18:24:18.258645 INFO Daemon CGroups: ok
2019/05/30 18:24:18.263173 INFO Daemon Run daemon
2019/05/30 18:24:18.268142 INFO Daemon Clean protocol
2019/05/30 18:24:18.272685 INFO Daemon Provisioning already completed, skipping.
2019/05/30 18:24:18.277567 INFO Daemon RDMA capabilities are not enabled, skipping
2019/05/30 18:24:18.285862 INFO Daemon Determined Agent WALinuxAgent-2.2.40 to be the latest agent
2019/05/30 18:24:18.613217 INFO ExtHandler Agent WALinuxAgent-2.2.40 is running as the goal state agent
2019/05/30 18:24:18.635339 INFO ExtHandler Detect protocol endpoints
2019/05/30 18:24:18.642625 INFO ExtHandler Clean protocol
2019/05/30 18:24:18.651472 INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2019/05/30 18:24:18.658883 INFO ExtHandler Test for route to 168.63.129.16
2019/05/30 18:24:18.665334 WARNING ExtHandler No route exists to 168.63.129.16
2019/05/30 18:24:18.670932 INFO ExtHandler Checking for dhcp lease cache
2019/05/30 18:24:18.675843 INFO ExtHandler looking for leases in path [/var/lib/dhcp/dhclient.*.leases]
2019/05/30 18:24:18.684096 INFO ExtHandler cached endpoint not found
2019/05/30 18:24:18.690504 INFO ExtHandler Cache exists [False]
2019/05/30 18:24:18.695994 INFO ExtHandler Send dhcp request
2019/05/30 18:24:18.700569 INFO ExtHandler Examine /proc/net/route for primary interface
2019/05/30 18:24:18.705882 ERROR ExtHandler /proc/net/route contains no routes
2019/05/30 18:24:18.710763 WARNING ExtHandler Could not determine primary interface, please ensure /proc/net/route is correct
2019/05/30 18:24:18.717827 WARNING ExtHandler Contents of /proc/net/route:
Iface   Destination     Gateway         Flags   RefCnt  Use     Metric  Mask            MTU     Window  IRTT

2019/05/30 18:24:18.732775 WARNING ExtHandler Primary interface examination will retry silently
2019/05/30 18:24:20.747552 ERROR ExtHandler /proc/net/route contains no routes
2019/05/30 18:24:22.755975 ERROR ExtHandler /proc/net/route contains no routes

VM reboot didn't help, but VM instance stop/start helped.

References
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133
https://github.com/Azure/WALinuxAgent/issues/980
https://github.com/Azure/WALinuxAgent/issues/1439

Tibco Jaspersoft throws error java.lang.NoClassDefFoundError: Could not initialize class java.awt.Color

Problem

Tibco Jaspersoft Report server throws java.lang.NoClassDefFoundError: Could not initialize class java.awt.Color when attempting to render report containing graphical elements.

Solution

Check missing dependencies using following command:

ldd /opt/jasperreports-server-7.1.1/java/lib/amd64/libawt_xawt.so

Results:

        linux-vdso.so.1 (0x00007ffff17f4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f346bc2f000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f346b891000)
        libawt.so => /opt/jasperreports-server-7.1.1/java/lib/amd64/libawt.so (0x00007f346b5bf000)
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f346b3ad000)
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f346b075000)
        libXrender.so.1 => not found
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f346ae71000)
        libXtst.so.6 => not found
        libXi.so.6 => not found
        libjava.so => /opt/jasperreports-server-7.1.1/java/lib/amd64/libjava.so (0x00007f346ac45000)
        libjvm.so => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f346a854000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f346c0a7000)
        libjvm.so => not found
        libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f346a62c000)
        libjvm.so => not found
        libverify.so => /opt/jasperreports-server-7.1.1/java/lib/amd64/libverify.so (0x00007f346a41d000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f346a219000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f346a013000)
        libjvm.so => not found
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f3469dfe000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3469bf6000)

Install missing packages using following command, libjvm.so can be ignored.

sudo apt-get install libfontconfig1 libxrender1 libxi6 libxtst6

Restart Jasper Report server using commands

sudo /opt/jasperreports-server-7.1.1/ctlscript.sh stop
sudo /opt/jasperreports-server-7.1.1/ctlscript.sh start

Create GitLab backup without repositories

Problem

Need to create GitLab backup skipping git repositories themselves. GitLab installed using Omnibus package.

Solution

According GitLab Omnibus Documentation and Backing up and restoring GitLab, required backup can be created using command below:

gitlab-rake gitlab:backup:create BACKUP=gitlab_20190420 GZIP_RSYNCABLE=yes SKIP=repositories

Please note, that following files are not included to backup and must be backed up separately.

/etc/gitlab/gitlab.rb
/etc/gitlab/gitlab-secrets.json

Reference

https://docs.gitlab.com/omnibus/settings/backups.html

https://docs.gitlab.com/ee/raketasks/backup_restore.html

https://docs.gitlab.com/omnibus/

Add user to sudoers

Problem

Need to add existing user to sudoers.

Solution

Ubuntu

usermod -aG sudo <username>

Install and configure Domain Controller in Microsoft Azure

Problem

Need to install and configure domain controller in MS Azure.

Solution

There is excellent article how to setup DC in Azure.
https://www.assistanz.com/steps-to-create-new-active-directory-forest-in-azure-portal/

Configure Azure Firewall ports required to join Domain Controller

Problem

There are two virtual networks, Network 1 (contains Domain Controller, Windows Server 2016) and Network 2 (contains workload VMs, Windows Server 2016). Traffic between them goes through Azure Firewall and User Defined Routes.

Solution

After some experiments came up with following categories of rules:

Name	Proto	Src	Dest	Ports
tcp-to-dc	tcp	*	<dc server>	53,88,135,139,389,445,464,3268,3269,49152-64000
udp-to-dc	udp	*	<dc server>	53,88,123,135,137,138,464,389,49152-64000

References
https://support.microsoft.com/en-us/help/179442/how-to-configure-a-firewall-for-domains-and-trusts
http://powershell365.com/2016/01/19/firewall-ports-required-to-join-ad-domain/

Sync files using rsync with sudo privileges

Problem

Need to copy files from one Linux VM to another (Ubuntu 18). Root account is not available, have accounts with sudo privileges.

Solution

Sample command listed below:

sudo rsync -e "ssh" --rsync-path="sudo rsync" -Pav remoteuser@<remote server>:/mnt/disk01/folder_to_sync/ /mnt/disk01/folder_to_sync/

When executing, the system will ask you for password to existing user and then password for remoteuser.

Note, that remoteuser should be able to execute sudo command without prompting to type password.

Refer to https://sk.solutionmentors.com/2019/05/run-commands-with-sudo-without-having.html for instructions.

Run commands with sudo without having to enter a password

Problem

Need to temporarily allow access to user to run commands with sudo without typing password (for batch jobs)

Solution

Add following entry to /etc/sudoers to allow remoteuser to execute commands with sudo without entering password.

remoteuser ALL=(ALL) NOPASSWD:ALL

References
https://askubuntu.com/questions/334318/sudoers-file-enable-nopasswd-for-user-all-commands/340669