Introduction

Proxmox Virtual Environment (Proxmox VE) is an open-source server virtualization environment which provides a management layer on top of KVM. It is a Debian-based Linux distribution with a modified Ubuntu LTS kernel, and allows deployment and management of virtual machines and containers. Proxmox VE includes a Web console and command-line tools, and provides a REST API for third-party tools. Two types of virtualization are supported: container-based with LXC, and full virtualization with KVM. It comes with a bare-metal installer and includes a Web-based management interface.

Subscriptions

In order to get updates without a paid subscription follow this:

1. add “deb http://download.proxmox.com/debian/pve stretch pve-no-subscription” to the sources.list file
2. remove /etc/apt/sources.list.d/pve-enterprise.list
3. https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_no_subscription_repository
4. https://www.svennd.be/proxmox-ve-5-0-fix-updates-upgrades/

*Note: internet access is required for the server to ‘call home’

Email Config

Proxmox uses postfix, so settings for etcpostfixmain.cf should look like this:

Importing Existing VM’s

Existing VMs in many formats can be imported into proxmox.

Example of importing VM

Firstly shutdown your VM, logon to the proxmox host shell then:

Importing existing Virtual Machine Disks over the network

Can use sshfs to access the remote images from the proxmox host easily, this will allow us to import the vm’s quickly across the network

http://www.linuxjournal.com/content/accessing-remote-files-easy-and-secure

Example

From the proxmox host machine:

To import a VM from another proxmox host, simply backup the VM to a shared storage area then restore to the new host. Using this method there is no need to clean up the hardware, as the hardware config & associated drives will all be backed up & then restored.

Importing UEFI Boot Enabled Machines

Machines configured with UEFI boot must have the correct Bios enabled OVMF (EFI) and an EFI boot disk created (from hardware, click Add/EFI Disk and select a storage location for the tiny disk)

Points to note after conversion – Linux machines

libvirt startup/shutdown scripts will need to be removed:

rm /etc/init.d/libvirt*

The network bridge interface will probably need renaming, often called br0 will need to be renamed to vmbr0 in /etc/network/interfaces.

Proxmox VM Configuration Files

Each VM has an associated config file which can be found in /etc/pve/qemu-server/

Example of a config file:

#created for testing. #applied latest updates #disabled auto updates by setting the wuauserv service to ‘disabled’ LibUSB-Win32-1.2.6.0 bootdisk: ide0 cores: 2 ide0: local-lvm:vm-111-disk-1,size=100G ide2: none,media=cdrom memory: 2024 name: windows10-64 net0: e1000=1E:BA:6D:37:68:7E,bridge=vmbr0 numa: 0 ostype: win10 scsihw: virtio-scsi-pci smbios1: uuid=810c2f26-37cc-403b-99c4-baf41c44ae60 sockets: 2

Exporting VM’s

Virtual Machines stored on LVM or similar volumes, can be exported to raw format if required by firstly backing up the VM to lzo format. Then unzipping the lzo and running the vma extract command.

E.g

vma extract vzdump-qemu-141-2019_05_09-08_53_19.vma -v /tmp/restorels

Using ‘pass through’ disks

fdisk -l
ls -l /dev/disk/by-label
ls -l /dev/disk/by-uuid
ls -l /dev/disk/by-id

Note: that using /dev/sdX is not a great option since the letter attribution can change, whereas a disk’s uuid will not.

Then, you’ll want to copy whatever label/uuid/id that is relevant, and to allow passthrough there are now two options:

1. Through the Proxmox console

qm set {vmid} -{ide|sata|scsi}# /dev/disk/by-{label|uuid|id}/{reference} with vmid the ID of your VM, ide or sata the type of disk you want to add.

Note: that ide value can be 0-3, sata can be 0-5 and scsi 0-13. Also, ide0 is generally the boot disk and ide2 will be the CD drive; Adjust depending on your configuration.

example:

qm set 101 -sata0 /dev/disk/by-label/data01 This will modify the {vmid}.conf file, which bring us to option 2.

2. By directly modifying the {vmid}. conf file in /etc/pve/qemu-server/

Add a line as follows:

{ide|sata|scsi}#: /dev/disk/by-{label|uuid|id}/{reference}

example:

sata0: /dev/disk/by-label/data01

You’ll want to shutdown then boot the VM (not just a regular restart) for the changes to take effect, after which the disk should be accessible in your VM.

P2V Conversion

Physical to Virtual conversions can be done by creating an image of the disk to be cloned, converting that raw image into a KVM compatible format, creating a VM and finally importing the KVM disk.

Example:

dd if=/dev/sdb3 bs=2048 conv=sync,noerror of=/mnt/pve/RESDATA/resbootdd.img

qemu-img convert -O qcow2 resbootdd.img /var/lib/vz/resbootdd.qcow2

qm create 120 --net0 virtio,bridge=vmbr0 --name resinosbuild --serial0 socket --bootdisk ide0

qm importdisk 120 /var/lib/vz/resbootdd.qcow2 local-lvm

Qemu Guest Agent – installation

https://pve.proxmox.com/wiki/Qemu-guest-agent

Moving VM’s and Disks using Non-Shared Storage

It is possible from the command line, to migrate live VM’s even though the storage is not configured as shared in proxmox or at the SAN level, but the nodes must be part of the cluster.

Sample:

Installing SSL Certificates single node servers (wildcard certs)

SSH to the Proxmox node and execute the following commands:

1. pvecm updatecerts -f
2. cp certname.pem /etc/pve/nodes//pveproxy-ssl.pem
3. cp private-key.key /etc/pve/nodes//pveproxy-ssl.key
4. systemctl restart pveproxy

That should restart the admin interface for Proxmox with the new wildcard cert. To check if there were any errors you can execute:journalctl -b -u pveproxy.service

Storage Configuration

This section provides an overview of the current storage configuration.

Extending the default local-lvm Storage with a Physical Disk

In order to extend the LVM pve/data storage area by adding a new physical drive, identify the drive (in this case sdb) and then run something like the following commands:

1. pvs show current info on the phsical volume
2. vgs show current info on the volume groups
3. lsblk show new drive
4. pvdisplay show the physical drives
5. pvcreate /dev/sdb create a new physical volume from sdb ready to be added to a volume group
6. lvs show the current logical volumes (note the size of the data LV)
7. vgextend pve /dev/sdb will extend the pve volume group to include the physical volume sdb
8. pvscan will now show the newly added physical volume along with the others
9. lvextend -l +100%FREE /dev/pve/data will now increase the data logical volume to include 100% of the available space on the volume group
10. lvs will now show the new logical volumes (note the size of the data LV)

Extending hosts LVM storage by increasing disk space in an array

If you have an array which has been extended (e.g a hardware raid 5 config where you have added an additional disk and extended the array):

After you have added the physical drive, and extended the array in the Dell Perc controller:

1. identify the drive you need to extend using lsblk, fdisk -l etc
2. use something like fdisk /dev/sdb and delete the partition
3. recreate the partition and stick to fdisks default, you will be warned that there is an existing LVM2_member signature, do NOT remove this!
4. finally resize the physical volume used by the volume group – something like pvresize /dev/sdb1 – this will resize the lvm physical disk and use the new size of the disk
5. done

Using SAN Disks as local host Mounted Proxmox Storage

If you need to mount a SAN disk for use by proxmox, you should:

• define the LUN in the SAN config tool
• create folder you want mapping on the proxmox server
• fdisk (or parted if very large disk) to create your partition on the SAN disk e.g  fdisk /dev/mapper/3600a098000584f5c00002cdb5c2ce043
• ensure fdisk is using GPT and note that the new partition will use the unique ID and append it with -part1
• note – the machine will need to be rebooted at this point!
• format the partition e.g mkfs.ext4 /dev/mapper/3600a098000584f5c00002cdb5c2ce043-part1
• mount the partition to the folder using the unique device ID, eg   /dev/mapper/3600a098000584f5c00002cdb5c2ce043-part1 (you can ensure you have the right ID by looking at the  SAN tool under Storage & Copy Services tab e.g mount /dev/mapper/3600a098000584f5c00002cdb5c2ce043-part1 /staging
• In order to put an entry in fstab you must remember that the network needs to have started before mapping, otherwise it will fail. You can use the _netdev option as shown in the example below:/dev/mapper/3600a098000585271000023c65c59e11b-part1 /staging ext4 _netdev 0 0

iSCSI SAN Connectivity

Useful information on managing and adding SAN storage:

Proxmox uses Open-iSCSI more information here: https://github.com/open-iscsi/open-iscsi

Extending hosts SAN Storage

First extend storage using the SAN management application. This will take some time, but once complete the node wont see the extra physical storage.

Secondly on the corresponding node locate the physical volume to be extended: using pvs command:

root@pvhost2:~# pvs
 PV VG Fmt Attr PSize PFree 
 /dev/mapper/3600a0980005852710000219b5c2ce4c0 SAN lvm2 a-- 1.5t 1.05t
 /dev/sda3 pve lvm2 a-- 2.62t 15.83g

(note in this example we increased the drive from 1.5t to 2.32t but its not showing the extra capacity)

then extend using the pvresize command. eg:

pvresize /dev/mapper/3600a0980005852710000219b5c2ce4c0

Now the extra storage is available for use.

root@pvhost2:~# pvs
 PV VG Fmt Attr PSize PFree 
 /dev/mapper/3600a0980005852710000219b5c2ce4c0 SAN lvm2 a-- 2.32t 1.05t
 /dev/sda3 pve lvm2 a-- 2.62t 15.83g

Storage Layout Examples

NFS Storage

Two NFS storage mounts are used to hold backups & ISO files.

local-lvm Storage

*LVM-Thin volumes cannot be shared

*Local disk volumes cannot be shared, if you want to share a volume use the SAN or NFS

SharedStore

LVM

By default Proxmox creates VM’s using LVM, which is a thin software layer on top of hard disks and partitions. It can be used to split available disk space into smaller logical volumes. LVM is widely used on Linux and makes managing hard drives easier.

http://landoflinux.com/linux_lvm_command_examples.html

LVM-Thin

LVM-Thin supports snapshots which is a very important requirement for the daily administration of a VM estate.sdavg1lv2lv1sdapv1vg1thin-vol2thin-vol1sdbpv2pv1thin-vol3tpool

Steps to setup lvm-thin for use in Proxmox:

Now you can use the new storage you have defined to store virtual disks.

ZFS Storage Notes

ZFS file systems can be used to facilitate replication of VM’s between nodes.

ZFS over iscsi does not currently support multipath! proxmox v5.2-2

ZFS has advantages when you need software features provided by LVM along with RAID, however we mainly use hardware RAID so do not use ZFS.

PVE-zync

This downloadable proxmox utility allows you to synchronize your virtual machine (virtual disks and VM configuration) or directory stored on ZFS between two servers (either inside cluster or not!). By synchronizing, you have a full copy of your virtual machine on the second host and you can start your virtual machines on the second server (in case of data loss on the first server).

Main features:

• Speed limiter
• Syncing interval can be set by cron
• Syncing VM (disks and config) but also ZFS Datasets
• Can keep multiple backups
• Can be used in both directions
• Can send on local host
• Traffic is encrypted

PVE Storage Replication vs PVE-zsync:

https://pve.proxmox.com/wiki/PVE-zsync

Miscellaneous

Nested Virtualization

Running a Hypervisor within a Hypervisor (nested virtualization) can be enabled on a host via:

(Intel CPUS)

1. cat /sys/module/kvm_intel/parameters/nested
2. shows the current status of nested virtualisation
3. echo “options kvm-intel nested=Y” > /etc/modprobe.d/kvm-intel.conf
4. activates nested virtualisation
5. modprobe -r kvm_intel
6. modprobe kvm_intel
7. restarts KVM kernel module (or reboot machine)

Installing VirtualBox alongside Proxmox

This can be achieved via the following steps:

1. Create a file /etc/apt/sources.list.d/virtualbox.list and add: deb http://download.virtualbox.org/virtualbox/debian stretch contrib
2. Add the virtualbox key: wget https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | apt-key add –
3. Update sources: apt update
4. Install Virtualbox: install VirtualBox-5.2
5. During installation VirtualBox will inform you it needs some more bits: This system is currently not set up to build kernel modules. Please install the Linux kernel “header” files matching the current kernel for adding new hardware support to the system. The distribution packages containing the headers are probably: linux-headers-amd64 linux-headers-4.9.0-3-amd64 This system is currently not set up to build kernel modules. Please install the Linux kernel “header” files matching the current kernel for adding new hardware support to the system. The distribution packages containing the headers are probably: linux-headers-amd64 linux-headers-4.9.0-3-amd64
6. So we need to install the proxmox kernel header, first find out which kernel is installed: uname -r
7. Then install the headers: apt install pve-headers-4.15.17-3-pve
8. Finally finish the installation: /sbin/vboxconfig
9. Download the guest additions (ensure you select the correct version from here): wget https://download.virtualbox.org/virtualbox/5.2.6/Oracle_VM_VirtualBox_Extension_Pack-5.2.6-120293.vbox-extpack
10. Install the guest additions: VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.2.6-120293.vbox-extpack
11. Set up an account for VirtualBox so it doesn’t run as root useradd -d /home/vbox -m -g vboxusers -s /bin/bash vbox passwd vbox (use vbox)

Installing ESXi inside Proxmox

VirtualBox Web Service

https://github.com/phpvirtualbox/phpvirtualbox/wiki

1. Tell the system which user the web service will run as. In /etc/default/virtualbox add: VBOXWEB_USER=vbox VBOXWEB_HOST=127.0.0.1 sudo -u vbox vboxwebsrv –background
2. Start the vboxweb service as the dedicated user:
3. Install webserver and dependencies
4. aptitude install apache2 php php-soap php-xml unzip
5. Download phpVirtualBox:cd /var/www/html
6. unzip phpvirtualbox.zip
7. wget http://sourceforge.net/projects/phpvirtualbox/files/latest/download -O phpvirtualbox.zip
8. Rename the folder for easier typing:
9. mv phpvirtualbox-5.0-5/ phpvirtualbox
10. Change into the phpVirtualBox folder and copy the initial config file:
11. cd /var/www/html/phpvirtualbox
12. cp config.php-example config.php
13. Open up the config file and add the vboxweb service credentials. In /var/www/html/phpvirtualbox/config.php change:
14. var $username = ‘vbox’;
15. var $password = ‘vbox’;

Point your browser to http://yourserver/phpvirtualbox (use your own server name or IP address). The default credentials are admin/admin.

Disaster Recovery is important and disasters come in all shapes & sizes. Fire, flood, explosion, virus attack, human error, etc.

Having a realistic plan is vital, the plan needs to be fully tested and detailed with all the information someone might possibly need to deal with whatever disaster they are faced with – all presented in easy to understand language.

In 2011 I wrote a DR document for a large college I was working for, the document was 20 pages long and here I’m going to summarise the essentials.

Hopefully this will help provide a framework document for anyone looking to implement something similar, or at least help you think about what needs to be covered.

INTRODUCTION AND DEFINITIONS

The documetn should start by stating that whilst a major incident or disaster is always extremely unlikely to occur – and difficult to prevent – an organisation needs to put in place procedures to cover any eventuality which might arise.  By having a carefully prepared and flexible Disaster Recovery Plan, the damage and disruption can be minimised and the IT services returned to a normal as quickly as possible. 

The document should describe the manner in which the organisation would respond to a variety of disasters, and provide simple to read, accurate and up to date information.

It should be clearyly stated that an ‘Incident Manager’ will be called upon to form a ‘Recovery Team’ who will be responsible for carrying out the technical work required to establish operational systems. 

Definition of a Disaster 

For the purposes of the document a disaster is seen as the loss of IT service provision for a large part of the organisation which adversely affects operational running. 

Additional Information 

The document should be reviewed and distributed to all staff who may be expected to play a part in recovery. The document should be stored at various safe locations electronically and in printed form. Anything incorrect should be remedied immediately and the appropriate changes made. If a procedure or issue is identified that will save time and may often be suspected as occurring it must be included in the documentation. 

All amendments to the document should be logged in an appendum. 

The document provides an overview of the processes involved, but more detailed information for each system/server etc should be provided in sperate DR Manuals. These documents would contain enough technical information for anyone with a high enough level of technical skill (though not necesarily any experience) to work on restoration.

Estimated time to restore critical services would depend upon requirements and scope., but you should try to provide estimates based on identified scenarios.

STAFFING

An ‘Incident Manager’ will be appointed and be expected to: 

1. Ascertain the extent of damage to computer hardware, software, wiring etc 

2. Identify requirements and prioritisation for essential services 

3. Establish a ‘recovery team’ and act as a central communications point 

4. Prioritize order in which services are to be restored 

5. Act as a central communications point for recovery team 

6. Liase with 3rd parties as required 

Recovery Team 

A large pool of staff should be identified as having sufficient skills to be called upon to form part of the ‘recovery team’, this team of technical experts, managers etc. will be instrumental in ensuring services are restored and effective communications are in place.

Here you should list the names, job titles, mobile phone numbers, email addresses of each member of staff.

DISASTER RECOVERY KIT

Disaster Recovery Kits to be used by staff in the event of a DR include the USB storage devices, applications, OS and other essential tools which may be required. 

The kits should be stored in safe locations, along with the required technical DR Manuals. 

SERVICE DEPENDENCY

In order to recover a service, it may be necessary to recovery many components. It is vital to understand dependencies, such as Exchange upon Active Directory, and AD upon DNS; in order to fully restore a pre-defined service.

Here you should list each service along with the dependencies, for example you might have an on premise email server which is dependent upon a working DNS server.

BACKUP PROCESS OVERVIEW

Here you should provide information on how your systems are backed up, what the schedules are, where the data is stored etc.

NETWORK

Everything depends upon a working network, so the network and its dependencies should be listed along with appropriate network diagrams. Here you should also list any telecoms requirements and diagrams.

CHANGE LOG

Ensure all changes to the document are captured and recorded here.

If you or your company use an old NAS box, you might be tempted to move it into the cloud and keep the way in which its acceessed by users. When working with NAS boxes people are used to saving documents to network drives, so things like the Y drive are comfortable and familier. But if you move data into the cloud, how can you retain those access methods?

This blog provides an example of moving your NAS into AWS, there is however one issue I couldnt resolve. When copying or moving the network file shares to the S3 bucket, the files lost the creation and last modified dates. Bit of a dealbreaker I think, but maybe you can find a fix?

Bucket Configuration

Setup a storage bucket called “Cloud_NAS” with no public access, and a folder structure like:

team1
team2
team3

Create policies restricting access to appropriate folders inside the bucket, then create user groups in IAM with correct policy assigned. Finally create users (use access keys and record these as you’ll need them when installing the client software) as required and assign to groups.

Sample access policy (this one should be assigned to users in team1, you can create two more access policies for the other teams):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowUserToSeeBucketListInTheConsole",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::Cloud_NAS.myaws"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "",
                        "team1/*"
                    ],
                    "s3:delimiter": [
                        "/"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::Cloud_NAS.myaws"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "/team1/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*Object"
            ],
            "Resource": [
                "arn:aws:s3:::Cloud_NAS.myaws/accounts/*"
            ]
        }
    ]
}

Client Software

Windows cannot natively access S3 storage, so we use something like tntDrive or S3 Drive.

Install your client software, you will need the to provide the REST endpoint (Cloud.NAS.myaws in my example), the users access key and secret key.

https://tntdrive.com/

https://tntdrive.com/

My experience of DNS servers was largely based around Windows Servers until a few years ago when I had to work on a linux bind9 box. I have to admit that until this point I never fully appreciated just how complex these little things are. They are however a fundamental part of the internet – without them, the internet wouldnt be half as much fun as it is!

Hopefully this short guide will help others build a simple DNS server in Linux.

Steps

Download and install ubuntu 18.04

Ensure DNS Server is selected during installation (if you are using a different OS version then once the OS is installed and you have ran apt update, then run apt install dnsutils

Configure your network, hostname (also if this is running as a VM then disk size, CPU count, etc – doesnt need anything powerful)

Once the server is installed and up to date with the latest patches so run apt update followed by apt upgrade

You need to work on some settings files, in my example the files are:

Now lets look at the contents of each file in turn, starting with named.conf.local

named.conf.local

zone "domain.com" IN {
  type master;
  file "domain.com.zone";
};
zone "10.1.in-addr.arpa" {
  type master;
  file "db.domain.rev";
};

In this file 2 zone files are defined, we can see they are both master files and we can see the network is 10.1.x.x

named.conf.options

options {
        directory "/var/cache/bind";
        dnssec-validation auto;
        auth-nxdomain no;    # conform to RFC1035
        listen-on port 53 { localhost; 10.1.0.0/16; 10.55.55.0/24; };
        allow-query { localhost; 10.1.0.0/16; 10.55.55.0/24; };
        forwarders { 8.8.8.8; };
        recursion yes;
};

Here we see the directory containing the zone files defined as /var/cache/bind, the security level is set to auto, the server will only respond to requests from the DNS server, 10.1.0.0/16 and 10.55.55.0/24 networks and we are forwarding DNS requests out to Googles 8.8.8.8 server.

/var/cache/bind/domain.com.zone

$TTL 86400
@ IN SOA domain.com root.domain.com (
  2022180203
  3600
  900
  604800
  86400
)
@               IN NS dns
dns             IN A 10.1.0.2
files           IN A 10.1.5.176
win7            IN A 10.1.5.61
tester          IN A 10.10.10.10
poopot          IN A 9.9.9.9
cbs2            IN A 10.1.5.211

/var/cache/bind/db.domain.rev

$TTL    86400
@       IN      SOA     dns.domain.com. hostmaster.domain.com. (
                        2022180203
                        10800
                        3600
                        604800
                        86400   )
        IN      NS      dns.domain.com.
$ORIGIN 0.1.10.in-addr.arpa.
2       IN      PTR     dns.domain.com.
4       IN      PTR     pc123.domain.com.
$ORIGIN 5.1.10.in-addr.arpa.
211     IN      PTR     cbs2.domain.com.

Set your dns servers dns server to use the same IP by editing your /etc/network/interfaces file so it includes something like:

dns-nameservers 10.1.0.2 8.8.8.8
dns-search domain.com

hints

systemctl status bind9 provides some pretty useful information on the DNS service.