Replicated Volume via GlusterFS

In this post, we will demonstrate on how to create a replicated glusterFS
volume. We will use 3 hosts, 2 gluster servers and 1 client all running
Centos 7.3.

vm01 = gluster server 1
vm02 = gluster server 2
vm03 = client

As an introduction, a replicated gluster FS volume ensures that data is present
on all bricks (the building blocks of a volume). This is useful in situations
where you want high-availability and redundancy. There are other types of
gluster FS volumes which you can see at Gluster Volumes.

Configuring the servers

Please note that most of the commands below needs to be ran on both servers
unless the step explicitly tells to run on 1 node only.

1. Install glusterFS server packages on the servers. You have 2 options on
how to install the server packages, one is to install from SIG repository and
the other is from gluster.org. Let's try the first one by searching for the
latest stable repository and installing it.

[root@vm01 ~]# yum search centos-release-gluster
=================================================================================== N/S matched: centos-release-gluster ====================================================================================
centos-release-gluster310.noarch : Gluster 3.10 (Long Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster312.noarch : Gluster 3.12 (Long Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster313.noarch : Gluster 3.13 (Short Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster36.noarch : GlusterFS 3.6 packages from the CentOS Storage SIG repository
centos-release-gluster37.noarch : GlusterFS 3.7 packages from the CentOS Storage SIG repository
centos-release-gluster38.noarch : GlusterFS 3.8 packages from the CentOS Storage SIG repository
centos-release-gluster39.noarch : Gluster 3.9 (Short Term Stable) packages from the CentOS Storage SIG repository

  Name and summary matches only, use "search all" for everything.

[root@vm01 ~]#
[root@vm01 ~]# yum install -y centos-release-gluster310
<output truncated>
root@vm01 ~]#

2. Now that we have a repository to get the packages, let's install the actual
server packages.

[root@vm01 ~]# yum install -y glusterfs glusterfs-cli glusterfs-libs glusterfs-server
<output truncated>
root@vm01 ~]#

3. Start and enable gluster service.

[root@vm01 ~]# systemctl enable --now glusterd
Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
[root@vm01 ~]#

4. Add a separate disk on your nodes to hold our gluster partitions. This make
sure that it is separated from the OS partition to mimic real world setup. Once
added, make sure it can be seen by your hosts. In my case, I added a 5 GB disk
to each gluster server and can be seen as /dev/sdb inside the hosts.

[root@vm01 ~]# fdisk -l | grep sd
Disk /dev/sda: 8589 MB, 8589934592 bytes, 16777216 sectors
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200    16777215     7339008   8e  Linux LVM
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
[root@vm01 ~]#

5. Let's now create the underlying filesystem.

[root@vm01 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.
[root@vm01 ~]# vgcreate gluster_vg /dev/sdb
  Volume group "gluster_vg" successfully created
[root@vm01 ~]# lvcreate -n gluster_lv -l 100%FREE gluster_vg
  Logical volume "gluster_lv" created.
[root@vm01 ~]# mkfs.xfs /dev/mapper/gluster_vg-gluster_lv
meta-data=/dev/mapper/gluster_vg-gluster_lv isize=512    agcount=4, agsize=327424 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1309696, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@vm01 ~]#

6. Mount the filesystem and make sure its persistent across reboots.

[root@vm01 ~]# mkdir /brick
[root@vm01 ~]# echo '/dev/mapper/gluster_vg-gluster_lv /brick xfs defaults 0 0' >> /etc/fstab
[root@vm01 ~]# mount -a
[root@vm01 ~]# df -h /brick/
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/gluster_vg-gluster_lv  5.0G   33M  5.0G   1% /brick
[root@vm01 ~]#

7. Open up needed ports - for bricks and pool communication. In versions below
3.4, each brick needs a port starting at 24009/tcp. For versions higher, each
brick needs a port starting at 49152/tcp. Opening this brick ports allows client
to access your volumes. Since we will use 1 brick only, we will open up port
49152/tcp. In addition to that, pool communication on any versions needs to
ports 24007/tcp and 24008/tcp on all servers.

[root@vm01 ~]# firewall-cmd --add-port={24007-24008/tcp,49152/tcp} --permanent
success
[root@vm01 ~]# firewall-cmd --reload
success
[root@vm01 ~]#

8. (Do this on 1 node only). Gluster servers work via Trusted Storage Pool or
"TSP". This pool contains the members sharing the bricks. In order to add a
member, let's execute the command below on the 1st server. Once executed, you
don't need to run it again on the other node.

[root@vm01 ~]# gluster peer probe vm02       
peer probe: success.                                             
[root@vm01 ~]#

9. After being probing for members, let's verify if the pool status.

* from vm01 *

[root@vm01 ~]# gluster peer status
Number of Peers: 1

Hostname: vm02
Uuid: 06525146-da28-4081-b6e3-35372ebf269c
State: Peer in Cluster (Connected)
[root@vm01 ~]#

* from vm02 *

[root@vm02 ~]# gluster peer status
Number of Peers: 1

Hostname: vm01
Uuid: 0c77f8b8-5706-413e-9a25-6e3951d738f5
State: Peer in Cluster (Connected)
[root@vm02 ~]#

10. Create the directory that we will export via a gluster volume

[root@vm01 ~]# mkdir /brick/export1

11. (Do this on 1 node only) Let's create the actual gluster volume and start
it.

[root@vm01 ~]# gluster volume create glustervol01 replica 2 transport tcp vm01:/brick/export1 vm02:/brick/export1
volume create: glustervol01: success: please start the volume to access data
[root@vm01 ~]# gluster volume start glustervol01
volume start: glustervol01: success
[root@vm01 ~]#

12. Let's verify the volume status. Same output should appear on both servers.

[root@vm01 ~]# gluster volume info

Volume Name: glustervol01
Type: Replicate
Volume ID: bf0b27da-ff4b-4bfd-aaa5-1708c16237e1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vm01:/brick/export1
Brick2: vm02:/brick/export1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@vm01 ~]#

13. Now that our gluster volume "glustervol01" has been setup, let's configure
our client.

Configuring the client

1. Install needed packages to mount the gluster volume

[root@vm03 ~]# yum install -y glusterfs glusterfs-fuse attr
<output truncated>
[root@vm03 ~]#

2. Mount the volume. You may mount the volume from vm01 or vm02.
It doesn't make any difference since this is a shared storage. Take note that
we mount the volume by its volume name "glustervol01" and not by the path of
the brick (/brick/export1).

[root@vm03 ~]# mkdir /mnt/glustervol01
[root@vm03 ~]# mount -t glusterfs vm01:/glustervol01 /mnt/glustervol01
[root@vm03 ~]# touch /mnt/glustervol01/file-from-vm03
[root@vm03 ~]# ll /mnt/glustervol01/file-from-vm03
-rw-r--r--. 1 root root 0 Jan  8 21:50 /mnt/glustervol01/file-from-vm03
[root@vm03 ~]#

3. Client has successfully mounted the volume and can immediately start writing
files to it. Once a file has been created, you will notice that it is present on
the 2 bricks (1 on vm01 and the other on vm02) which signifies a
replicated storage.

* from vm01 *

[root@vm01 ~]# ls -l /brick/export1/
total 0
-rw-r--r--. 2 root root 0 Jan  8 21:50 file-from-vm03
[root@vm01 ~]#

* from vm02 *

[root@vm02 ~]# ls -l /brick/export1/
total 0
-rw-r--r--. 2 root root 0 Jan  8 21:50 file-from-vm03
[root@vm02 ~]#

That concludes our mini tutorial. Hope you enjoy and learn a lot from this.

Managing GlusterFS Volumes

Increasing Replicas on Replicated GlusterFS

Let's say existing volume is backed by 2 bricks having a size of 5 GB each

To add 2 more bricks, perform this on 1 node only

gluster volume add-brick my_volume replica 4 node1:/bricks/brick2 node2:/bricks/brick2

"replica 4" means we are increasing the replica count of a file
If we used "replica 2", we will end up in a "distributed-replicated" type of
volume which is not correct if we still want to maintain the volume type as
replicated.

Also, if the sizes of the new bricks are greater then the existing bricks, the
volume size will be the size of the original bricks. The lower size are always
being used so it is better to add bricks with same size of the existing ones.

Expanding a Distributed GlusterFS Volume

1. Perform this on 1 node

   gluster volume add-brick my_volume node1:/brick/brick2

2. The total size of brick(s) will be added on top of the existing size of
   the volume.

Expanding a Replicated GlusterFS Volume

1. Expand the underlying storage on all nodes.

   lvextend -rL +5G /dev/mapper/gluster_vg-gluster_lv

2. The gluster volume size should now be increased by 5G

Tips in recovering Jenkins after unsuccessful upgrade

1. Backup the following

jobs/
plugins/
config.xml
credentials.xml

2. Take note of global tools configuration setup
3. Install new jenkins version
4. Restore the backups above
5. Ensure plugins are updated
6. Ensure global tool configuration is updated

Running Zenity in Crontab

What is Zenity?

  It is a GNOME application that produces pop-up windows that can be used to send message to user. It also have file selection capabilities by using --file-selection option.

  Example usage are:

  zenity --info # sends a simple pop up message
  zenity --info --text="Backup is running, press OK to close this window"  # with user-defined message

How to run this in CRON?

  Since CRON doesn't process X applications by default, you need to add the following option:

  zenity --info --text="Hi, I was launched via CRON" --display=:0

Sources

http://promberger.info/linux/2009/01/02/running-x-apps-like-zenity-from-crontab-solving-cannot-open-display-problem/

zenity(1)

Virtual Console Actions on HP Hardwares

1. Momentary Press

  - If server is ON, this action will power OFF the server (simulates graceful shutdown from the OS)

  - If server is OFF, this action will power ON the server

2. Press and Hold

  - When triggered while server is ON, this will force shutdown the server

  - Simulates a physical "press and hold for 6 seconds" on the power switch

  - Useful if OS is not responding to Momentary Press

3. Cold Boot

  - This action will power OFF a running server, waits for 6 seconds and powers it ON

  - You can't use this action when the server is OFF

4. Reset

  - Simulates a reboot from the OS (shutdown -r now)

  - Quickest way to close a frozen program so system may try to recover data and repair software

  - In contrast to "Cold boot", this is referred as "Warm boot"

Useful Ansible Ad-Hoc Commands

FILE/DIRECTORY

# copies a file to multiple severs in parallel
ansible all -m copy -a "src=/tmp/file dest=/tmp/file"

# changes file permission
ansible lab -m file -a "dest=/tmp/file mode=600 owner=bob group=wheel"

# creates a directory
ansible webservers -m file -a "dest=/path/to/c mode=755 owner=root group=mdehaan state=directory"

# deletes a directory
ansible lab -m file -a "dest=/tmp/dir state=absent"

Managing Packages:

# ensures a package is installed (don't update)
ansible lab -m yum -a "name=openssh state=present"

# ensure a package is installed to a specific version
ansible webservers -m yum -a "name=acme-1.5 state=present"

# ensure a package is on the latest version (update if lower version was found)
ansible lab -m yum -a "name=openssh state=latest"

# ensure a package is not installed (uninstalls it if needed)
ansible lab -m yum -a "name=bind state=absent"

Users and Groups

# ensures a this particular user exist
ansible webservers -m user -a "name=apache state=present"

# ensures a this particular user doesn't exist
ansible webservers -m user -a "name=hacker state=absent"

DEPLOYING FROM SOURCE CONTROL

# gets configuration from git
ansible dbservers -m git -a "repo=git://my.db.uk/repo.git dest=/var/lib/db version=HEAD"

Managing Services:

# ensures service is started
ansible webservers -m service -a "name=httpd state=started"

# restarts a service
ansible webservers -m service -a "name=httpd state=restarted"

# ensures a service is stopped
ansible webservers -m service -a "name=httpd state=stopped"

TIME LIMITED OPERATIONS

# executes a long running operation
ansible all -B 3600 -P 0 -a "/usr/bin/long_running_operation --do-stuff"
  * -B 3600 = timeout of 3600 seconds
  * -P 0 = no polling

# checks the status of the long running job
ansible web1.example.com -m async_status -a "jid=488359678239.2844"
  * jid = job id returned when command was ran in the background

# similar to the above but with polling
ansible all -B 1800 -P 60 -a "/usr/bin/long_running_operation --do-stuff"
  * -P 60 = polls every 60 seconds

GATHERING FACTS

# returns the settings of all target hosts
ansible all -m setup

SOURCES

ansible modules:
http://docs.ansible.com/ansible/list_of_all_modules.html

Linux Command Line Basics

When you first login to a Linux terminal, you are now using bash. Bourne-Again SHell (or BASH) is the default shell. A shell is the interpreter of anything you type on the terminal. In this post, I will briefly discuss the basics of linux command line.

Kinds of Shells

bash

  - Bourne Again SHell

  - based on earlier version of sh

  - commonly default shell in Linux

sh

  - Bourne shell

  - this is were BASH was based

  - not often used in Linux but usually a pointer to /bin/bash

tcsh

  - based on earlier C shell (csh)

  - not popular in Linux

csh

  - the original C Shell

  - not commonly used in Linux

  - very similar to tcsh

ksh

  - combination of features of csh and sh

zsh

  - Z Shell

  - evolution of ksh

How to start a shell?

There are several ways and here are the most common ways.

1. Once you logged in to a linux server (either by SSH or via console), you automatically starts your own shell

2. In xterm or GUI, you can right click on the desktop and open a terminal. In that way, you are also starting a shell

Parts of a shell

Understanding every part of the shell is very important specially for linux administrators because it is the one you interact with everytime you deal with Linux. As an example, when you logged as a regular, the shell will provide you a prompt where you can type your commands.

$ uname

Linux

$

The "$" represents something. In Linux by default, it means that you are logged in as regular user. You may customized it by changing the PS1 environment variable. A root's prompt starts with "#" so its important to clearly distinguished a regular user's prompt from a root's prompt. That can avoid accidentally running destructive command like "rm -fr *" on /. The command used in the example above is "uname" and its output is "Linux". Depending on the command you type, it may return an output or nor. Or it may even return an error.

Types of commands you can use

2 types of commands are "Internal" and "External" commands. Internal are the ones built-in in linux while External are the  ones came from outside sources (e.g came from installing a package). To see what type of command, execute the following:

$ type cd

cd is a shell builtin

$ type date

date is /bin/date

$

$ type -a pwd

pwd is a shell builtin

pwd is /bin/pwd

$

The first example tells us that "cd" is an Internal command while the second one indicates that "date" is an external command. There are some commands you may encounter that has both Internal and External versions. So to see it, add "-a" on type command as shown in the 3rd example above.

Getting help

Since we are dealing with command line, we often need more information on a particular command we are using. There are several ways on how to display command information.

1. Using "man" - perhaps the most important help utility in Linux

  man <command>  # displays man pages of a command

  man -k <string>  # searches the man database for commands (or description) matching the specified string

  man -K <string>  # similar as above but do a full search (takes significantly longer to complete)

  man <section number> <command/config file>  # prints information on a specific manual section (see discussion below)

Man contains sections which groups the type of information you can see on a particular command. Not all command have the same manual sections. Here are the example of manual sections:

Section Number

  1 -- user commands (contains traditional command usage and options)

  2 -- system calls provided by the kernel (not commonly used by sys ads)

  3 -- library calls

  4 -- device files

  5 -- file formats (these are config files like "man 5 fstab")

  6 -- games

  7 -- miscellaneous

  8 -- system administration commands (usually ones that must be run as root)

  9 -- kernel routines

On the 9 manual sections, only 1, 5, and 8 are the most commonly used by system administrators.

2. Using pinfo/info - not all commands support this

  info coreutils 'ls invocation'  # as an example, you can see this command under "SEE ALSO" of ls (1)

3. Exploring /usr/share/doc

  - some packages includes README files under this directory

  - you can see examples here

  - sometimes there are also sample configurations

Environment Variables

These are global values read by the shell and inherited by the user (which can also be overriden) during startup. To see all environment variables, type "env". Here are the common environment variables in Linux.

PS1 - this is your prompt (e.g [user@host] $)

HOME - contains path to your home directory

HOSTNAME - the hostname of current machine where you are logged in

LOGNAME - the username used to login to the machine

To display the values, use the following command:

echo $HOME