VMware - ... and vSphere

mail

vSphere snapshots

  • making a new snapshot creates a delta disk
  • the guest OS writes to this delta disk to preserve the base disk (or the previous snapshot, if any)
    This is why snapshots shouldn't live for too long, or they can grow LARGE !
  • the Delete snapshot command actually merges the delta disk into the base disk (or the previous snapshot, if any )
  • to bring a VM back to an earlier state, you'll have to :

Snapshots from a file perspective (source) :

A snapshot consists of the following files :
  • delta disk files :
    • a myVirtualMachine.vmdk file to which the guest operating system can write
    • this represents the difference between
      • the state that existed when the snapshot was taken
      • and the current state of the virtual disk
      When making a snapshot, the state of the base disk (or the previous snapshot, if any ) disk is preserved by preventing the guest operating system from writing to it.
  • a myVirtualMachine-flat.vmdk file that represents the base disk
    This is over-simplified to give you the "big picture", there are more files and more details. Docs say that not all files are listed in the Datastore Browser and appear only after SSHing into the corresponding ESXi server.
  • other files :
    • snapshot metadata
    • memory dump

Best practices (source) :

  • for best performance, limit to 2-3 snapshots per VM
  • avoid using a snapshot for more than 72 hours since the files constituting the snapshot keep growing and may impact system performance

Deleting a snapshot (source) :

  • does not change the virtual machine or other snapshots
  • consolidates the changes between snapshots and previous disk states and writes all the data from the delta disk that contains the information about the deleted snapshot to the parent disk (i.e. the "delta" is merged into the "base")
  • removes the snapshot from the Snapshot Manager
mail

How to re-scan / detect an enlarged disk in a running vSphere virtual machine ?

Situation

  1. before :
    lsblk
    NAME					MAJ:MIN	RM	SIZE	RO	TYPE	MOUNTPOINT
sr0					11:0	1	1024M	0	rom
sda					8:0	0	50G	0	disk
├─sda1					8:1	0	500M	0	part	/boot
└─sda2					8:2	0	49.5G	0	part
  ├─VolGroup-lv_root (dm-0)		253:0	0	45.6G	0	lvm	/
  └─VolGroup-lv_swap (dm-1)		253:1	0	4G	0	lvm	[SWAP]
sdb					8:16	0	50G	0	disk
└─sdb1					8:17	0	50G	0	part
  └─VolGroup-lv_data (dm-3)		253:3	0	50G	0	lvm	/data
sdc					8:32	0	50G	0	disk
└─sdc1					8:33	0	50G	0	part
  2. in vSphere :
    1. Rclick VM name | Edit settings
    2. change size of the disk #3 from 50GB to 55GB
  3. back to the shell, I still see :
    sdc						50G

Solution

  1. re-scan the sdc storage device :
    echo 1 > /sys/block/sdc/device/rescan
  2. check :
    lsblk
    NAME					MAJ:MIN	RM	SIZE	RO	TYPE	MOUNTPOINT

sdc					8:32	0	55G	0	disk
└─sdc1					8:33	0	50G	0	part
mail

How to detect a new disk added to a vSphere virtual machine ?

for i in /sys/class/scsi_host/host*; do grep -q 'mptspi' "$i/proc_name" && echo '- - -' > "$i/scan"; done; lsblk

How it works

  • In the /sys pseudo filesystem, we have : 
    /sys/class/scsi_host
├── host0
│    └── proc_name
├── host1
│    └── proc_name
└── host2
     └── proc_name
  • Each proc_name contains the name of the module used by the SCSI host it belongs to, generally one of :
    • ata_piix
    • mptspi
  • The SCSI host we're trying to identify (i.e. the one holding the virtual disks) is the one using the mptspi module (looks specific to how VMware handles this)
  • Once we've found the SCSI host, we just have to "force rescan" by writing - - - into /sys/class/scsi_host/host2/scan
mail

How to boot a VM on a recovery media ?

Situation

Still in the context of resetting the root password of an old Linux VM, we know how to :

...but things are never simple (life sucks ) and you _may_ find that the security of this old VM has been hardened and prevents from entering boot options without a password .

The alternate solution is now to (try to) boot on any recovery media and see if we can change the root password.

Solution

  1. browse the VMware data storage : it's very likely you'll find a directory containing ISO images. Any bootable media giving you a shell will be fine
  2. otherwise, get a Debian "netinst" image and upload it there
  3. edit the VM settings to insert this ISO image (as Datastore ISO file) into the virtual CD-ROM (don't forget to check the Connect At Power On option)
  4. alter boot options to boot on the CDROM
  5. boot the VM and open the Web Console. You should see a BIOS setup UI
  6. change the boot device order so that you'll boot on the CDROM. Save and exit. You'll now boot on the Debian "netinst" CD.
  7. enter the Rescue mode (you'll be asked a few questions : hostname, time settings, ...)
  8. at the root file system, I asked to mount /dev/rootvg/lv_root as my / filesystem (maybe not the easiest solution...)
  9. let the boot process continue until you're given a shell prompt.
    • vSphere's Web Console is pretty buggy when it comes to the keyboard layout. It advertises a US keyboard layout, but it's actually something intermediate / weird.
    • I found no way to change the keyboard layout (quick search, though). So for the french AZERTY keyboard'ed ones (leave the Enforce US Keyboard Layout button untouched) :
      To get ... Type ... i.e. ...
      / > Shift-<
      : . Shift-;
      ! / Shift-:
      - 6 Shift--
  10. at that time, there are several approaches :
    • simply change the root password with passwd
      Didn't work for me : no /usr/bin/passwd at that time . Maybe I shouldn't have mounted the VM root filesystem as /
    • or be gory and edit /etc/shadow with Vi (I chose that option in the steps below)
  11. just in case : cp /etc/shadow /etc/shadow_OLD (good ol' "cpold" method)
  12. then vi /etc/shadow
  13. you should see (/etc/shadow format) : 
    root:$1$12345678$abcdefghijklmnopqrstuv:12345:0:99999:7:::
  14. erase the full encrypted password (2nd field of the line, :-separated)
  15. save and exit : Esc : x ! becomes Esc Shift-; x Shift-:
  16. change the VM settings back :
    • remove the CDROM
    • restore boot options
  17. you can now log as root with a blank password
    Don't forget to set a new strong password and to store it in a safe place (such as a password vault like KeePass).
mail

How to access the bootloader and set some boot options ?

Situation

There's a virtual Linux machine for which the root password is lost (). I have a method to (try to) recover it, but this requires altering Grub settings. How may I even reach those settings ?

Solution

In vSphere's web UI :
  1. pick your VM in the left column
  2. in the main part of the screen, in the title line, you'll see :
    1. the VM name
    2. a series of buttons : Play | Stop | Console | Settings | Snapshot
    3. the Actions drop-down list
  3. click the Settings icon. You'll get a Edit Settings | myVm pop-up window
  4. select the VM Options tab
  5. expand the Boot Options entry
  6. now you can either :
    • set a long Boot Delay, like 10000ms
    • or tick the Force BIOS Setup checkbox
  7. Ok | Ok | Ok ...
  8. (re)boot the VM and launch the web console
  9. hit SPACE, you should see the Grub menu
  10. then continue with this procedure