Wednesday, June 6, 2018

Linux Package Managers (RPM and Debian)


Introduction
============

In this post, we will discuss the 2 common package managers in Linux.
They are the following:

1. RPM-based - rpm,yum
2. Debian-based - dpkg,apt-get,dselect,aptitude,apt-cache

RPM
===

Examples of systems that uses RPM-based package managers are Red Hat, Fedora,
and Centos. The "rpm" command is to install packages directly. It doesn't handle
dependency requirements so you need to install the needed packages (if there is
any) before installing the package you want. For example, if you want to install
packageA but it needs packageB, you need to install packageB first before
proceeding with packageA. Here are the common uses of "rpm" command.

Installing/
Upgrading
# installs a package (install only if it doesn't exist)
rpm -i

# same as the above, but increases verbosity (-v) and hashes as progress
# indicator (-h)
rpm -ivh

# installs a package even if it exists (package reinstall)
rpm -ivh --force

# freshens a package (upgrade a package if ONLY an older version exists)
rpm -Fvh

# installs a package if that doesn't exist or upgrade a package if it
# exists
rpm -Uvh

# installs a package ignoring dependency checking (use this with caution
# because the program you will install might not work)
rpm -ivh --nodeps

# do a dry-run instead of installing actual package
rpm -i --test
Removing
# removes a package
rpm -e
Querying
# queries a package (-q) by printing information (-i); takes only
# package name and not filename (file.rpm)
rpm -qi

# lists files associated in the package (-l)
rpm -ql

# shows you on what package the file/directory/script came from; example
# is rpm -qf $(which cp)
rpm -qf

# prints change log of the given package
rpm -qi --change-log

# prints scripts that ran when the package was installed
rpm -qi --scripts

# queries a package filename instead (-p) of package name; example is
# rpm -qip my-program.rpm
rpm -q[other query options]p file_name.rpm 

YUM
===

There is also an RPM-based tool that handles dependency conflicts automatically
and that is YUM (Yellowdog Updater Modified). "yum" command is the one we used
which can also do what "rpm" command can. This is the best package management
tool to use if you are using the Linux distributions mentioned above.

The main disadvantage of this is you need a repository before you can do any
transactions whereas RPM don't. Repositories are the ones inside
/etc/yum.repos.d directory and will be discussed on a separate post in this
blog. YUM commands that are commonly used are:

Installing/
Upgrading
# installs a package or multiple packages
yum install

# same as above but answers "y" to all installation questions
yum install -y

# updates all installed packages on the system
yum update

# updates a single or multiple packages
yum update

# installs a local rpm file using your yum repository to resolve
# dependencies
yum localinstall
Removing
# removes packages and all packages where the package depends on
yum remove

# same as above
yum erase
Querying
# lists all installed packages
yum list

# searches all package names that contains the string; prints both
# installed and available packages
yum search *pattern*

# deeper search compare to the above command (searches also contents not
# only package names)
yum search all *pattern*

# shows what package owns the file/directory
yum provides

# use this when you don't get a result from the above command
yum whatprovides */

# shows info on a package (similar to rpm -qi)
yum info

# checks whether there are updates available
yum check-update
  
DPKG
====

Aside from RPM-based systems, we also have Debian-based systems. Examples of
these are Ubuntu and of course Debian. The package management tools they use are
apt-get, aptitude, dselect, and dpkg. "apt-get" is similar to yum where you need
a repository which is defined under /etc/apt/sources.lst, "dpkg" is similar to
rpm, "dselect" is a menu based tool, and aptitude is a combination of dselect's
menu based and apt-get's cli features. Debian packages by convention ends with
.deb. Let's start with dpkg. Here are the common commands for dpkg.

Installing/
Upgrading/
Configuring
# installs a package
dpkg -i

# ignore dependency conflict (similar to --nodeps in rpm)
dpgk -i --ignore-depends=

# tests for dependency only (similar to --test in rpm)
dpkg -i --no-act

# doesn't install a package if a newer version of same package is
# already installed
dpkg -iG

# doesn't install a package if same version of package is already
# installed
dpkg -iE
Removing
# removes a package while retaining its configuration files
dpkg -r

# removes a package and configuration files
dpkg -P
Querying
# displays information about an installed package
dpkg -P

# displays information about an uninstalled package
dpkg -I

# displays current installed packages
dpkg --get-selections

# list installed packages matching the pattern string
dpkg -l

# lists files associated with the package (similar to rpm -ql)
dpkg -L
   
One command that is very useful if you want to return the package to its
original state (fresh install with default settings) is "dpkg-reconfigure".

For example, the command below will reconfigure the samba package, asking the
packages initial installation questions and restarting its daemons.

dpkg-reconfigure samba

APT-GET
=======

Similar to yum , Debian-based systems have the tool called "apt-get" which
automatically handle dependency conflicts. Before using it, make sure you have
the appropriate sources inside /etc/apt/sources.lst. Here are the common usage
of that command:

Installing/
Upgrading
# installs a package
apt-get install

# installs a package assuming "y" to all questions
apt-get install -y

# don't upgrade a package if an older version exists
apt-get install --no-upgrade

# downloads a package but doesn't install it
apt-get install -d

# performs a simulation/dry-run without installing any package or
# configuring any file
apt-get install -s

# obtains information on available packages inside /etc/apt/sources.lst
apt-get update

# upgrades all installed packages to newer versions
apt-get upgrade

# similar to above command but performs "smart" conflict resolution
apt-get dist-upgrade
Removing
# removes a package
apt-get remove
Configuring
# performs any changes in package status left undone after running
# deselect
apt-get dselect-upgrade

# performs housekeeping (like yum clean)
apt-get clean

# similar to the above command but removes about packages that can no
# longer be downloaded
apt-get autoclean
Querying
# checks package database for consistency and broken package
# installations
apt-get check
  
APT-CACHE
=========

We also have apt-cache. Its only purpose is to provide information about Debian
package database. Here are sample commands:

Listing
# displays information about a package
apt-cache showpkg

# displays package statistics (how many installed, dependencies recorded
# etc..)
apt-cache stats

# displays information about unmet dependencies
apt-cache unmet

#  shows on what packages the one you specified depend on
apt-cache depends

# displays all installed packages
apt-cache pkgnames

# displays list of installed packages matching the string specified
apt-cache pkgnames

DSELECT
=======

A menu-based package manager also exists for Debian-based systems. That is
"dselect". When invoked, it will display you the following menu.

0. [A]ccess Choose the access method to use.
1. [U]pdate Update list of available packages, if possible.
2. [S]elect Request which packages you want on your system.
3. [I]nstall Install and upgrade wanted packages.
4. [C]onfig Configure any packages that are unconfigured.
5. [R]emove Remove unwanted software.
6. [Q]uit Quit dselect.

APTITUDE
========

A tool that combines dselect's menu based gui and apt-get's cli is "aptitude".
Here are the common usage:

Listing/
Querying
# searches packages related to the one you
# NOTE: (apt-get seems not to have this feature
aptitude search specified

# update package list from APT repository
aptitude update

# shows complete options
aptitude help
Installing/
Upgrading
# installs a package
aptitude install

# upgrades all installed packages
aptitude full-upgrade

# conservative version of the above command
aptitude safe-upgrade

# removes already-downloaded packages that are no longer available
aptitude autoclean
Removing
# removes a package (w/ leading dash)
aptitude install -

# same as above
aptitude remove

Tuesday, June 5, 2018

NTP (Network Time Protocol)

Basics
------

- user space daemon is "ntpd" running on udp port 123
- ntpd updates the system clock
- system clock is a software clock running in the kernel w/c is more accurate
  compared to RTC
- RTC is the hardware clock whose value varies depending on the temperature
  (real time clock)

What is Drift File?
-------------------

- stores the frequency offset between the system clock running at its nominal
  frequency and the frequency required to remain in synchronization with UTC
- used to adjust system clock every time ntpd start
- located in: /var/lib/ntp/drift (directory must be writable by NTPD)
- entry on config file: driftfile /var/lib/ntp/drift

Time on Virtual Machines
------------------------

- they have no hardware clock w/c is a more accurate time source
- load of hosts (ESX) where they resides affects the timekeeping (e.g higher
  load slows down clock ticks)

NTP Strata
----------

stratum 0
atomic clocks, GPS, mobile phone systems
stratum 1
computer with radio clock, GPSclock, or atomic clock attached
stratum 2
Reads from stratum 1; Serves to lower strata
stratum 3
Reads from stratum 2; Serves to lower strata
stratum n+1
Reads from stratum n; Serves to lower strata
stratum 15
Reads from stratum 14; This is the lowest stratum
stratum 16
unsynchronized state

How ntpd chooses a time source?
-------------------------------

1. it will check which is the closest (e.g least delay, offset, jitter)
2. it will check the stratum
3. it will check "prefer" directive in /etc/ntp.conf

NTP configuration file: /etc/ntp.conf
-------------------------------------

assigning a time source
## basic
server [options]

## assigning from an ntp pool
server 0.rhel.pool.ntp.org iburst
restricting access
## don't allow IPs on these list to modify your config file and query
## your NTP server
restrict 192.168.1.0 mask 255.255.255.255 nomodify noquery

## makes sure localhost has full permissions w/o any restricting keyword
restrict 127.0.0.1
misc
## fails back to local clock if no time source is available
server 127.127.1.0
fudge 127.127.1.0 stratum 10

Some facts about leap seconds
-----------------------------

- affected systems are only those who use NTP
- by default, systems not running NTP are not affected
- leap second might make the system (or the application running inside) crash
- when a leap second is inserted, kernel prints it to /var/log/messages
- Cron jobs scheduled on 00:00 UTC won't be launched twice during leap second

Action Plans to do for Leap Second (from Redhat support)
--------------------------------------------------------

1) You can stop ntpd before June 30th, and then start it again when it is more
   convenient. Once it is started, it will sync with the NTP servers.
2) Use ntpd in slew mode (-x option). With this configuration the time won't be
   stepped, instead it will be gradually adjusted.

Commands
--------

Displaying
## checks time servers (detailed status)
ntpq -p

COLUMNS:
remote and refid = remote NT P server, and its NT P server
st = stratum of server
t = type of server (local, unicast, multicast, or broadcast)
poll = how frequently to query server (in seconds)
when = how long since last poll (in seconds)
reach = octal bitmask of success or failure of last 8 queries
        (left-shifted); 377 = 11111111 = all recent queries were
        successful; 257 = 10101111 = 4 most recent were successful,
             5 and 7 failed
delay = network round trip time (in milliseconds)
offset = difference between local clock and remote clock
        (in milliseconds)
jitter = difference of successive time values from server (high
              jitter could be due to an unstable clock

NOTE:
- unreliable time source are those with high delays, offset, and jitters
- its current time source is the one with *

## checks for brief ntp status
ntpstat

## syncs with ntp server manually
ntpdate –u

## queries only; doesn't set the clock
ntpdate -q

Monday, June 4, 2018

NIC Bonding in RHEL/CentOS 6


1. create bond0 config file
# vi /etc/sysconfig/network-scripts/ifcfg-bond0
--- START EDIT ---
DEVICE=bond0
IPADDR=10.10.10.11
NETWORK=10.10.10.0
NETMASK=255.255.255.0
USRCTL=no
BOOTPROTO=none
ONBOOT=yes
--- END EDIT ---

2. edit the following files:
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
--- START EDIT ---
DEVICE=eth0
MASTER=bond0
SLAVE=yes
HWADDR=00:0C:29:8D:FB:EF
TYPE=Ethernet
UUID=5606b424-2186-410d-9dbc-dcb65b330bd9
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
--- END EDIT ---

# vi /etc/sysconfig/network-scripts/ifcfg-eth1
--- START EDIT ---
DEVICE=eth1
MASTER=bond0
SLAVE=yes
HWADDR=00:25:B5:0B:B1:00
TYPE=Ethernet
UUID=6353d563-0a85-47d2-b566-37f6c91c8973
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
--- END EDIT ---

3. create bond module file
# vi /etc/modprobe.d/modprobe.conf
--- START EDIT ---
alias bond0 bonding
options bond0 mode=balance-alb miimon=100
--- END EDIT ---

4. restart network
# service network restart

Sunday, June 3, 2018

Installing Data Domain Plugin in Netbackup media server


Installing
==========

1. download the plugin from the linke below:
(Data Domain Software > Data Domain Boost - Open Storage)

2. copy the file to the media server that needs access to the DD

3. stop nbrmms
# nbrmms -terminate

4. extract the installation files and install
# gunzip -c OST_2.4.1.0-289644_OST_solaris_sparc.tar.gz | tar xvf -
# ./install.sh

5. start nbrmms
# nbrmms

6. verify installation
# bpstsinfo -pi -stype DataDomain
# ls -lt /usr/openv/lib/ost-plugins

plugins are located in these paths:

/usr/openv/lib/ost-plugins --> Unix/Linux
C:\Program Files\Veritas\bin\ost-plugins --> Windows
C:\Program Files\VERITAS\NetBackup\bin\ost-plugins --> Windows

sample ost-plugin in windows:
libstspiDataDomain.dll

sample ost-plugin installer in windows:
ibstspiDataDomainSetup.exe

Uninstalling
============

1. there is no way of uninstalling it (e.g ./uninstall.sh)

2. you just need to install a different version and the previous version
will be overwritten


Saturday, June 2, 2018

Renaming a volume group in RHEL/CentOS 6


1. Verify old name
# vgs

2. Rename VG
# vgrename old_vg_name new_vg_name

3. Backup and replace references in fstab
# cp -p /etc/fstab /etc/fstab.$(date +%F)
# grep old_vg_name /etc/fstab
/dev/mapper/old_vg_name-lv_root    /                       ext4    defaults        1 1
/dev/mapper/old_vg_name-lv_swap    swap                    swap    defaults        0 0

4. Backup and replace referencs in grub.conf
# cp -p /boot/grub/grub.conf /boot/grub/grub.conf.$(date +%F)
# grep old_vg_name /boot/grub/grub.conf
kernel /vmlinuz-2.6.32-642.1.1.el6.x86_64 ro root=/dev/mapper/old_vg_name-lv_root ... rd_LVM_LV=old_vg_name/lv_root ... rd_LVM_LV=old_vg_name/lv_swap ...

5. Backup and rebuild initramfs
# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
# dracut -f

6. Restart server
# reboot

Friday, June 1, 2018

Online data migration w/ LVM


If you want to migrate your data to a new disc - let's say you decided to
purchase another storage appliance and you want your MYSQL datadir to move to
that appliance without any downtime, LVM mirroring is the way to go.

1. Prepare your LUN and assign/zone it to your server

2. Once the LUN is available, add it to the current VG that holds the MYSQL
datadir
# vgextend myql_vg /dev/sdc

3. Create a mirror of the MYSQL logical volume pointing to the newly added LUN
(this is the RIGHT side of the mirror)
# lvconvert -m1 /dev/mapper/mysql_vg-mysql_lv /dev/sdc

4. Wait until the sync is complete (must be 100% under Cpy%Sync field)
# lvs

5. Once the sync is complete, remove/detach the LEFT side of the mirror (that is
the old LUN) from the MYSQL logical volume
# lvconvert -m0 /dev/mapper/mysql_vg-mysql_lv /dev/sdb

6. Remove the old LUN from the MYSQL VG
# vgreduce mysql_vg /dev/sdb

7. Remove PV label
# pvremove /dev/sdb

8. Detach it from your server

9. Hola! You were able to move your MYSQL database to a new and faster disk
without a  downtime!

Thursday, May 31, 2018

Disks & Raid Groups


How RAID parity works?
----------------------

Parity data is used by some RAID levels to achieve redundancy. If a drive in the
array fails, remaining data on the other drives can be combined with the parity
data (using the Boolean XOR function) to reconstruct the missing data. For
example, suppose two drives in a three-drive RAID 5 array contained the
following data:

Drive 1: 01101101
Drive 2: 11010100

To calculate parity data for the two drives, an XOR is performed on their data:

    01101101
XOR 11010100
------------
    10111001

The resulting parity data, 10111001, is then stored on Drive 3.

Should any of the three drives fail, the contents of the failed drive can be
reconstructed on a replacement drive by subjecting the data from the remaining
drives to the same XOR operation. If Drive 2 were to fail, its data could be
rebuilt using the XOR results of the contents of the two remaining drives,
Drive 1 and Drive 3:

Drive 1: 01101101
Drive 3: 10111001

as follows:

    10111001
XOR 01101101
------------
    11010100

The result of that XOR calculation yields Drive 2's contents. 11010100 is then
stored on Drive 2, fully repairing the array. This same XOR concept applies
similarly to larger arrays, using any number of disks. In the case of a RAID 3
array of 12 drives, 11 drives participate in the XOR calculation shown above and
yield a value that is then stored on the dedicated parity drive.

what is data striping?
----------------------

In computer data storage, data striping is the technique of segmenting logically
sequential data, such as a file, so that consecutive segments are stored on
different physical storage devices.

Striping is useful when a processing device requests data more quickly than a
single storage device can provide it. By spreading segments across multiple
devices which can be accessed concurrently, total data throughput is increased.
It is also a useful method for balancing I/O load across an array of disks.
Striping is used across disk drives in redundant array of independent disks
(RAID) storage, network interface controllers, different computers inclustered
file systems and grid-oriented storage, and RAM in some systems.

RAID Disk types
---------------

Data disk
  - holds data stored on behalf of clients within RAID groups
  - and any data generated about the state of the storage system
    as a result of a malfunction

Spare disk
  - does not hold usable data, but is available to be added to a
    RAID group in an aggregate. Any functioning disk that is not
    assigned to an aggregate but is assigned to a system
  - functions as a hot spare disk

Parity disk
  - stores row parity information that is used for data reconstruction
    when a single disk drive fails within the RAID group

dParity disk
  - stores diagonal parity information that is used for data
    reconstruction when two disk drives fail within the RAID group,
    if RAID-DP is enabled.

how are RAID groups formed?
--------------------------

- they are automatically formed when you add disks to an aggregate
- Data ONTAP adds new drives to the most recently created RAID group until
  it reaches its maximum size (default behavior)

What size of RAID group should I choose?
----------------------------------------

- recommended range of RAID group size is between 12 and 20

Data ONTAP Storage units
------------------------

1. disks
  - physical device that you put into the shelves
  - e.g, SATA,BSAS,or SAS

2. raid groups
  - a collection of one or more disks providing a RAID level
  - if its a RAID-DP, there is atleast one data disk and 2 parity
  - disks (parity disk + dparity disk)

2. plexes
  - a collection of one or more raid groups

3. aggregates
  - consists of disks within the raid groups
  - collection of one or two plexes
  - if unmirrored, it contains a single plex
  - if mirrored, it contains 2 plexes

3. volumes
  - 2 types: traditional and FlexVol
  - tradional -> it inherit properties its containing aggregate (directly tied
                 to aggr)
  - FlexVol -> loosely coupled to its containing aggregate; you can alter
               properties on the fly (RECOMMENDED!!!)

4. qtrees
  - subdirectory of the root directory of a volume
  - you can use qtrees to subdivide a volume in order to group LUNs

5. LUNS
  - logical unit of storage under a volume
  - you can create LUNs in the root of a volume (traditional or flexible) or in
    the root of a qtree
  - NOTE: don't create LUNs under Data ONTAP's root volume (/vol/vol0)

Disk types based on speed
-------------------------

SAS - Serial Attached SCSI (faster)
BSAS - Bridged SAS / SATA drives (slower); this
       means you have SATA disk in a SAS enclosure (it is a DS4243 shelf)

How disks are named?
--------------------

{slot}{port}.{shelfID}.{bay}

example: 3c.10.21

How are disk firmwares updated?
-------------------------------

- when you assign it to storage system
- you can do it (ask Netapp technical support what is your target disk firmware)

Commands
--------

Displaying
# lists all disk on the cluster
storage show disk
storage show disk -T --> adds TYPE of disk column at the end (SAS,BSAS,etc.)

# shows info for a particular disk
disk show 1b.29

# display unowned disks
disk show -n

# shows busy disk
stats show disk:*:disk_busy
example output: disk:39CF5F4E:715905F5:E3AEEB55:DCADA33F:0[...]000:disk_busy:100%

# this command shows more of the hardware side of the disk
filer01> storage show disk 4b.01.13
Disk: 3a.01.13
Shelf: 1
Bay: 13
Serial: 6SL3LM8B0000N2367MFW
Vendor: NETAPP
Model: X412_S15K7560A15
Rev: NA00
RPM: 15000
WWN: 5:000:c5004b:e8877c
UID: 5000C500:4BE8877F:00000000:00000000:00[...]
Downrev: yes
Pri Port: A
Sec Name: 4b.01.13
Sec Port: B
Power-on Hours: N/A
Blocks read:      0
Blocks written:   0
Time interval: 00:00:00
Glist count: 0
Scrub last done: 00:00:00
Scrub count: 0
LIP count: 0
Dynamically qualified: No
Current owner: 4294967295
Home owner: 4294967295
Reservation owner: 0
filer01>
       
# to see firmware version of netapp disk (it is the column with NAxx)
sysconfig -a
storage show disk

# how to see raid size of an aggregate?
aggr status -v
  * raid size = data disks + parity disks
  * under Options, example: raidsize=16

Assigning
# unowning a disk
disk assign 0c.51 -s unowned -f

# assigning a spare disk to another system
c
# assigning multiple disks to local node
disk assign disk_1 disk_2 … disk_N

Tutorials
---------

Manual Update of Disk Firmware
1. remove all files under /etc/disk_fw (make a backup just to make sure)
2. download the target disk firmwares (here are examples)
3. within 2 minutes it should start updating by its own
4. if disk firmwares are upgraded, verify by issuing this
   command: storage show disk -x

Debugging/Troubleshooting
-------------------------

data is copied to a spare disk
Sun Sep  6 04:13:56 EDT [filer01:raid.rg.diskcopy.start:notice]: /aggr1/plex0/rg2: starting disk copy from 3a.01.11 to 4b.02.0
Sun Sep  6 04:14:03 EDT [filer01:raid.disk.predictiveFailure:warning]: Disk /aggr1/plex0/rg2/3a.01.11 Shelf 1 Bay 11 [NETAPP   X412_S15K7560A15 NA06] S/N [6SL3KS740000N23610JD] reported a predictive failure and it is prefailed; it will be copied to a spare and failed
...
Sun Sep  6 06:31:28 EDT [filer01:raid.rg.diskcopy.done:notice]: /aggr1/plex0/rg2: disk copy from 3a.01.11 to 4b.02.0 completed in 2:17:32.30