When changing the ulimit settings in /etc/security/limits.conf or in /etc/security/limits.d/filename for a specific user, you will only see the correct change in /proc/[pid]/limits if you are looking as the user the limit was set for. Looking at this setting in /proc as root or su will display the settings for root not the user you set within the file.

The command:
ulimit -a username
is the best way to tell if the limits config file change is active.

To start, it will be more convenient if you have ssh keys setup on the servers. Otherwise you will be entering a password for each server you need to connect to.


To break this out create three files:
1. iplist.txt
2. checkservers
3. serverscript

* If your command is small enough you do not need to have a seperate serverscript file.  I find it easier to put more complex scripts in their own file which is what the following example reflects.

Add your server IPs to iplist.txt:
192.168.1.10
192.168.1.22
192.168.1.45
192.168.1.100

Add you loop connection syntax in checkservers:
#!/bin/bash
for server in $(cat iplist.txt); do

ssh username@$server -i usernameprivatekeyfile 'bash -s'  < serverscript  >> server-results.txt

done

Add your script to serverscript:
#!/bin/bash

sudo -i

hostname

your
script
here

Final steps:
chmod 744 checkservers
chmod 744 serverscript

Finally run ./checkservers

If you are tired of reading long blog posts of using reverse ssh you came to the right place.

Scenario:
You want to connect from your workstation to a server.
Both are on their own networks and are behind NAT.

Target: Server behind firewall
Jump: Server on the internet that you have control of the firewall
Source: Your Workstation behind firewall

Steps (Target connects to Jump, Source connects to Jump, From Jump you connect to Target):

1. Target:
   ssh -fN -R 45000:localhost:22 username@Jumpserverip
2. Source:
   ssh username@jumpserverip
3. Jump:
   ssh useronTarget@localhost -p 45000

tcpdump tcp and host 172.22.1.111 and port 22 -s0 -xx -X -c 1000

Example:
Drives:
/dev/sda
/dev/sdb

Raid 1 with boot, swap, and root - created with Ubuntu installer

Problem:
Lost Drive 0 : /dev/sda

Note:  When using grub and if the drives are not hot swappable the server will need rebooted into the drive that has grub and the raid1 data.  In the case of this example on reboot the BIOS drive boot order needs changed from drive0 being first to drive1 being above drive0 to make it first.  After the repair has taken place this can be put back in the BIOS.

The servers stats pre drive loss:
root@ubuntumdtest:~# fdisk -l /dev/sda
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ECE8771-DEC1-444A-B370-880C670ABE8B

Device       Start      End  Sectors Size Type
/dev/sda1     2048     4095     2048   1M BIOS boot
/dev/sda2     4096  2101247  2097152   1G Linux filesystem
/dev/sda3  2101248  6295551  4194304   2G Linux filesystem
/dev/sda4  6295552 41940991 35645440  17G Linux filesystem

root@ubuntumdtest:~# fdisk -l /dev/sdb
Disk /dev/sdb: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ECE8771-DEC1-444A-B370-880C670ABE8B

Device       Start      End  Sectors Size Type
/dev/sdb1     2048     4095     2048   1M BIOS boot
/dev/sdb2     4096  2101247  2097152   1G Linux filesystem
/dev/sdb3  2101248  6295551  4194304   2G Linux filesystem
/dev/sdb4  6295552 41940991 35645440  17G Linux filesystem


root@ubuntumdtest:~# lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
loop0       7:0    0 55.5M  1 loop  /snap/core18/2284
loop1       7:1    0 55.4M  1 loop  /snap/core18/2128
loop2       7:2    0 43.6M  1 loop  /snap/snapd/14978
loop3       7:3    0 61.9M  1 loop  /snap/core20/1328
loop4       7:4    0 70.3M  1 loop  /snap/lxd/21029
loop5       7:5    0 67.2M  1 loop  /snap/lxd/21835
sda         8:0    0   20G  0 disk
├─sda1      8:1    0    1M  0 part
├─sda2      8:2    0    1G  0 part
│ └─md126   9:126  0 1022M  0 raid1 /boot
├─sda3      8:3    0    2G  0 part
│ └─md127   9:127  0    2G  0 raid1 [SWAP]
└─sda4      8:4    0   17G  0 part
  └─md125   9:125  0   17G  0 raid1 /
sdb         8:16   0   20G  0 disk
├─sdb1      8:17   0    1M  0 part
├─sdb2      8:18   0    1G  0 part
│ └─md126   9:126  0 1022M  0 raid1 /boot
├─sdb3      8:19   0    2G  0 part
│ └─md127   9:127  0    2G  0 raid1 [SWAP]
└─sdb4      8:20   0   17G  0 part
  └─md125   9:125  0   17G  0 raid1 /
sr0        11:0    1 1024M  0 rom


cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md125 : active raid1 sda4[2] sdb4[1]
      17805312 blocks super 1.2 [2/2] [UU]

md126 : active raid1 sda2[2] sdb2[1]
      1046528 blocks super 1.2 [2/2] [UU]

md127 : active raid1 sda3[2] sdb3[1]
      2094080 blocks super 1.2 [2/2] [UU]

unused devices: <none>


The Repair: Process:
Replace Drive0 which equates to /dev/sda

Boot into Drive1 by reordering the Bios boot order:

Login and sudo up:

cat /proc/mdstat
fdisk -l
fdisk -l /dev/sda
sfdisk -d /dev/sdb | sed 's/,\s*uuid=[^,]\+//gi' | sfdisk /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/sda
cat /proc/mdstat
mdadm --add /dev/md125 /dev/sda4
mdadm --add /dev/md126 /dev/sda3
mdadm --add /dev/md127 /dev/sda2
cat /proc/mdstat
grub-install /dev/sda
reboot

Change the Bios back to Drive0 being first in the boot order if desired.

By: tconrad