MRepo tutorial

WHAT IS MREPO?

  It is an open-source tool that creates a repository out of an ISO file (usually your installation disc) or from
public URLs like http://mirror.centos.org/. Those are some of the possible sources, explore the docs (see
sources below this post) for more choices.

  Aside from creating a repository, mrepo has the capability of making sure that both sides are in sync. When
a package from a http://mirror.centos.org/centos-7/7/os/x86_64/Packages/ has been remove, your local
repository will be udpated when you launch the mrepo command with the appropriate options (see tutorials
below on how to use mrepo command). Or if a newer package is available, mrepo will download it for you
and add it to your local repo.

INSTALLATION

1. Download the tarball from the site below. On other distribution, this is available via RPM on EPEL.
http://dag.wieers.com/home-made/mrepo/mrepo-0.8.7.tar.bz2

2. Untar it to any place you want.
tar xvf mrepo-0.8.7.tar.bz2

3. Go to the extracted directory and install it
cd mrepo-0.8.7
make install
# the installation will also produce a sysV script since this tool was prior to systemd. You may convert
# it to a unit file.

TIP: You might want to keep the extracted directory because it contains useful information like sample
    configurations and tutorials

CREATING A REPOSITORY FROM AN ISO FILE

In this part, we will create a repository out of an ISO file which typically is your distribution's installation disc.

1. Make sure the required directory exists

mkdir /var/mrepo

2. Create a subdirectory for your distribution

mkdir /var/mrepo/centos7-x86_64
# I chose the subdirectory name centos7-x86_64 because it is a valid format which is $dist-$arch.
# $dist and $arch is part of the config file as you will see in step 4. You may also use a format of $dist
# only but the former has an advantage in terms of containing the packages. If you use $dist only and
# in the future you provided URLs as source of your repository together with an existing ISO source,
# mrepo will create another directory which is /var/mrepo/$dist-$arch and put RPMs there instead on
# /var/mrepo/$dist. So in result, you will have 2 directories under /var/mrepo, subdir $dist which contain
# the actual ISO file and $dist-$arch which contains the RPMs from the URLs. So to make things cleaner,
# just create /var/mrepo/$dist-$arch at the start which will contain both the ISO and the URL RPMs w/o
# forcing mrepo to create separate directories.

3. Copy your distribution's ISO file (installation disc) to that directory

cp CentOS-7-x86_64-DVD-1611.iso /var/mrepo/centos7-x86_64/

4. Update the main config file, mrepo.conf, by appending the configurations below

cat << EOF >> /etc/mrepo.conf
[centos7]  # $dist or tag; part of subdirectory name in step 2
name = Centos 7 (64-bit) # mrepo doesn't care any description you put here
release = 7
arch = x86_64  # self explanatory? :) this is also part of subdirectory name in step 2
metadata = repomd  # we'll pick this as we are creating an RPM repository (another possible value is apt)
iso = CentOS-7-x86_64-DVD-1611.iso  # must match the filename copied in step 3
# it is important that you specify the filename correctly because mrepo will look for that file
EOF

TIP: You may also create a separate .conf file under /etc/mrepo.conf.d/ and mrepo will read it as if it is
included in the main config

5. Create the repository

mrepo -ugv  # see mrepo -h for flag uses
# As a summary, that command will perform the following:
#  1. Create /var/www/mrepo/$disr-$arch directory tree
#  2. Look for ISO files under the specified directories in order
#      /var/mrepo/$dist-$arch/$iso
#      /var/mrepo/$tag/$iso
#      /var/mrepo/iso/$iso
#      /var/mrepo/$iso
#  3. Mount ISO file to /var/www/mrepo/$dist-$arch/disc1 as a loop device
#  4. Create the links under /var/www/mrepo/$dist-$arch/

TIP: To see the detailed steps on how it create the repository, you may increase verbosity by -vvvvv (5 Vs!)

6. Once the above command completes, you will expect a directory structure similar to mine:

[root@home ~]# ll /var/www/mrepo/centos7-x86_64/  # this is the /var/www/mrepo/$dist-$arch
total 266
drwxr-xr-x. 8 root root   2048 Dec  5 21:20 disc1  # mountpoint of iso under /var/mrepo/$dist/$iso
lrwxrwxrwx. 1 root root     50 Mar 29 20:24 HEADER.shtml -> ../../../../usr/share/mrepo/html/HEADER.repo.shtml
drwxr-xr-x. 2 root root     42 Mar 29 20:24 iso  # contains a symlink to /var/mrepo/$dist/$iso
lrwxrwxrwx. 1 root root     50 Mar 29 20:24 README.shtml -> ../../../../usr/share/mrepo/html/README.repo.shtml
drwxr-xr-x. 2 root root      6 Mar 29 20:24 RPMS.all  # empty
drwxr-xr-x. 3 root root 212992 Mar 29 20:24 RPMS.os  # contains the actual RPMs as well as the repodata that came from the iso file
[root@home ~]#

7. You have now created a repository out from your ISO. But that repository is not yet available to clients
    until you share it via any methods like http, ftp, or nfs. Please see the other part of this post on where
    I demonstrate to you on how to share the repository via http method.

CREATING REPOSITORIES FROM URLS

In this part, we are going to create and sync a local repository from a public URL repository. As an example, http://mirror.centos.org/centos/7/extras/x86_64/, contains extra packages for Centos 7 so we will use that for this demonstration.

1. Update the mrepo.conf and add the following lines:

cat << EOF >> /etc/mrepo.conf
[centos7]
name = Centos 7 (64-bit)
release = 7
arch = x86_64
metadata = repomd
base = http://mirror.centos.org/centos/$release/os/$arch/
updates = http://mirror.centos.org/centos/$release/updates/$arch/
extras = http://mirror.centos.org/centos/$release/extras/$arch/
epel = https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$arch
EOF
# base, updates, extras, and epel are strings to describe the corresponding URL repository
# the following directory trees will be created:
#  /var/mrepo/$dist-$arch/{base,updates,extras,epel}
#  /var/www/mrepo/$dist-$arch/RPMS.{base,updates,extras,epel}

2. Install lftp because mrepo will use that to get packages from the URLs

yum install -y lftp

3. Sync the repositories. Since this is the first time, this will download al packages. All succeeding sync using
mrepo command will just download new packages or remove packages in your local that are no longer existing from the URLs.

mrepo -ugv

4. When you are finish downloading all packages, you can now make the repositories accessible to clients.
    See next section for the steps.

NOTE: If you can use "mirrorlists" in traditional yum repo, here in mrepo you cannot.

Making your repositories accessible via HTTP

There are many ways on how you can make your repositories accessible. Some of this are via
NFS and FTP. We will use HTTP for this demonstration since this is by far the most common way.

* On server *

1. Install http

yum install -y httpd

2. Start and enable at boot

RHEL 6.X/sysV systems:
service httpd start
chkconfig httpd on
RHEL 7.X/systemd systems:
systemctl start httpd
systemctl enable httpd

3. Open up port 80 on firewall

IPTABLES:
iptables -A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
service iptables restart
FIREWALLD:
firewall-cmd --add-service=http --permanent
firewall-cmd --reload

4. Add link of each repo to DocumentRoot. You may use another approach but this is the easiest for me. :)

ln -s /var/www/mrepo/ /var/www/html/mrepo

5. If selinux is enabled and in enforcing mode, change selinux context of /var/mrepo to allow httpd access

semanage fcontext -at httpd_sys_content_t '/var/mrepo(./*)?'
restorecon -Rv /var/mrepo

6. Make sure your repository is accessible by testing it on your local browser.
http://localhost/mrepo/centos7-x86_64/RPMS.base

* On client *

1. Create .repo file

cat << EOF >> /etc/yum.repos.d/server.repo
[base]
name=Centos 7 - Base Packages (64-bit)
baseurl=http://server/mrepo/centos7-x86_64/RPMS.base
EOF

2. See if you can see the repository

yum repolist all

3. Validate by installing a package

yum install -y zsh

SOURCES

Official site:
http://dag.wiee.rs/home-made/mrepo/

mrepo command usage (no manual page but this is sufficient):
mrepo -h

Documentations:
mrepo-0.8.7/docs

Sample configurations:
mrepo-0.8.7/configs

Other web turorial:
https://asenjo.nl/wiki/index.php/Mrepo_centos7

Basic RAID Levels

1. RAID 0 (striping)

  - minimum of 1 disks

  - striping

  - no mirroring

  - excellent performance

  - no fault tolerance

2. RAID 1 (mirroring)

  - minimum of 2 disks

  - mirroring

  - no striping

  - good performance

  - excellent fault tolerance

3. RAID 5 (striping with + distributed parity)

  - minimum of 3 disks

  - striping with distributed parity

  - good performance and fault tolerance

4. RAID 10 (mirroring + striping)

  - minimum of 4 disk

  - striping with mirroring

  - excellent performance and fault tolerance

  - expensive

Setup NIC Bonding in Linux in Centos 6

1. create bond0 config file

# vi /etc/sysconfig/network-scripts/ifcfg-bond0

--- START EDIT ---

DEVICE=bond0

IPADDR=10.10.10.11

NETWORK=10.10.10.0

NETMASK=255.255.255.0

USRCTL=no

BOOTPROTO=none

ONBOOT=yes
--- END EDIT ---

2. edit the following files:

# vi /etc/sysconfig/network-scripts/ifcfg-eth0

--- START EDIT ---

DEVICE=eth0

MASTER=bond0

SLAVE=yes

HWADDR=00:0C:29:8D:FB:EF

TYPE=Ethernet

UUID=5606b424-2186-410d-9dbc-dcb65b330bd9

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

--- END EDIT ---

# vi /etc/sysconfig/network-scripts/ifcfg-eth1

--- START EDIT ---

DEVICE=eth1

MASTER=bond0

SLAVE=yes

HWADDR=00:25:B5:0B:B1:00

TYPE=Ethernet

UUID=6353d563-0a85-47d2-b566-37f6c91c8973

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

--- END EDIT ---

3. create bond module file

# vi /etc/modprobe.d/modprobe.conf

--- START EDIT ---

alias bond0 bonding

options bond0 mode=balance-alb miimon=100

--- END EDIT ---

4. restart network

# service network restart

RPM and Debian Package Managers

In this post, we will discuss the 2 common package managers in Linux. They are the following:

1. RPM-based - rpm,yum

2. Debian-based - dpkg,apt-get,dselect,aptitude,apt-cache

RPM

Examples of systems that uses RPM-based package managers are Red Hat, Fedora, and Centos. The "rpm" command is to install packages directly. It doesn't handle dependency requirements so you need to install the needed packages (if there is any) before installing the package you want. For example, if you want to install packageA but it needs packageB, you need to install packageB first before proceeding with packageA. Here are the common uses of "rpm" command.

* Installing/Upgrading *

rpm -i <filename.rpm>  # installs a package (install only if it doesn't exist)

rpm -ivh <filename.rpm>  # same as the above, but increases verbosity (-v) and hashes as progress indicator (-h)

rpm -ivh <filename.rpm> --force  # installs a package even if it exists (package reinstall)

rpm -Fvh <filename.rpm>  # freshens a package (upgrade a package if ONLY an older version exists)

rpm -Uvh <filename.rpm>  # installs a package if that doesn't exist or upgrade a package if it exists

rpm -ivh <filename.rpm> --nodeps  # installs a package ignoring dependency checking (use this with caution because the program you will install might not work)

rpm -i <package name> --test  # do a dry-run instead of installing actual package

* Removing *

rpm -e <filename> -- removes a package

* Querying *

rpm -qi <package name>  # queries a package (-q) by printing information (-i); takes only package name and not filename (file.rpm)

rpm -ql <package name>  # lists files associated in the package (-l)

rpm -qf <package name>  # shows you on what package the file/directory/script came from; example is rpm -qf $(which cp)

rpm -qi <package name> --change-log  # prints change log of the given package

rpm -qi <package name> --scripts  # prints scripts that ran when the package was installed

rpm -q[other query options]p file_name.rpm  # queries a package filename instead (-p) of package name; example is rpm -qip my-program.rpm

YUM

There is also an RPM-based tool that handles dependency conflicts automatically and that is YUM (Yellowdog Updater Modified). "yum" command is the one we used which can also do what "rpm" command can. This is the best package management tool to use if you are using the Linux distributions mentioned above. The main disadvantage of this is you need a repository before you can do any transactions whereas RPM don't. Repositories are the ones inside /etc/yum.repos.d directory and will be discussed on a separate post in this blog. YUM commands that are commonly used are:

* Installing/Upgrading *

yum install # installs a package or multiple packages

yum install <package name(s)> -y  # same as above but answers "y" to all installation questions

yum update  # updates all installed packages on the system

yum update <package name(s)>  # updates a single or multiple packages

yum localinstall <filename.rpm>  # installs a local rpm file using your yum repository to resolve dependencies

* Removing *

yum remove <package name(s)>  # removes packages and all packages where the package depends on

yum erase <package name(s)>  # same as above

* Querying *

yum list  # lists ALL installed packages and shows you if there is a newer version for each of those

yum list <package name> # lists installed packages and shows you if there is a newer version for that
yum list available <package name> # lists all available packages for update

yum search *pattern*  # searches all package names that contains the string; prints both installed and available packages

yum search all *pattern*  # deeper search compare to the above command (searches also contents not only package names)

yum provides <file/directory>  # shows what package owns the file/directory

yum whatprovides */<file/directory>  # use this when you don't get a result from the above command

yum info <package name>  # shows info on a package (similar to rpm -qi)

yum check-update  # checks whether there are updates available
yum --disablerepo=* --eablerepo=<repo ID> list <pacakge name>  # list package on a particular repo

DPKG

Aside from RPM-based systems, we also have Debian-based systems. Examples of these are Ubuntu and of course Debian. The package management tools they use are apt-get, aptitude, dselect, and dpkg. "apt-get" is similar to yum where you need a repository which is defined under /etc/apt/sources.lst, "dpkg" is similar to rpm, "dselect" is a menu based tool, and aptitude is a combination of dselect's menu based and apt-get's cli features. Debian packages by convention ends with .deb. Let's start with dpkg. Here are the common commands for dpkg.

* Installing/Upgrading/Configuring*

dpkg -i <package name>  # installs a package

dpgk -i <package name> --ignore-depends=<package>  # ignore dependency conflict (similar to --nodeps in rpm)

dpkg -i <package name> --no-act  # tests for dependency only (similar to --test in rpm)

dpkg -iG <package name>  # doesn't install a package if a newer version of same package is already installed

dpkg -iE <package name>  # doesn't install a package if same version of package is already installed

* Removing *

dpkg -r <package name>  # removes a package while retaining its configuration files

dpkg -P <package name>  # removes a package and configuration files

* Querying *

dpkg -P <package name>  # displays information about an installed package

dpkg -I <uninstalled package name>  # displays information about an uninstalled package

dpkg --get-selections  # displays current installed packages

dpkg -l <pattern>  # list installed packages matching the pattern string

dpkg -L <package name>  # lists files associated with the package (similar to rpm -ql)

One command that is very useful if you want to return the package to its original state (fresh install with default settings) is "dpkg-reconfigure". For example, the command below will reconfigure the samba package, asking the packages initial installation questions and restarting its daemons.

dpkg-reconfigure samba

APT-GET

Similar to yum , Debian-based systems have the tool called "apt-get" which automatically handle dependency conflicts. Before using it, make sure you have the appropriate sources inside /etc/apt/sources.lst. Here are the common usage of that command:

* Installing/Upgrading *

apt-get install <package name>  # installs a package

apt-get install <package name> -y  # installs a package assuming "y" to all questions
apt-get install <package name> -s  # doesn't install, simulates a dry-run

apt-get install <package name> --no-upgrade  # don't upgrade a package if an older version exists

apt-get install <package name> -d  # downloads a package but doesn't install it

apt-get install <package name> -s  # performs a simulation/dry-run without installing any package or configuring any file

apt-get install /path/to/deb/package -f  # installs a .deb package and resolves dependencies through APT repositories

apt-get update  # obtains information on available packages inside /etc/apt/sources.lst

apt-get upgrade  # upgrades all installed packages to newer versions

apt-get dist-upgrade  # similar to above command but performs "smart" conflict resolution

* Removing *

apt-get remove <package name>  # removes a package

* Configuring *

apt-get dselect-upgrade  # performs any changes in package status left undone after running deselect

apt-get clean  # performs housekeeping (like yum clean)

apt-get autoclean  # similar to the above command but removes about packages that can no longer be downloaded

* Querying *

apt-get check  # checks package database for consistency and broken package installations

apt-file search /path/to/file  # searches for package containing a file

APT-CACHE

We also have apt-cache. Its only purpose is to provide information about Debian package database. Here are sample commands:

apt-cache show <package name>  # displays descripion of package

apt-cache showpkg <package name>  # same as above but displays dependency information instead

apt-cache show <package name>  # displays descripion of package

apt-cache stats  # displays package statistics (how many installed, dependencies recorded, etc..)

apt-cache unmet  # displays information about unmet dependencies

apt-cache depends <package name>  #  shows on what packages the one you specified depend on

apt-cache pkgnames  # displays all installed packages

apt-cache pkgnames <string>  # displays list of installed packages matching the string specified

DSELECT

A menu-based package manager also exists for Debian-based systems. That is "dselect". When invoked, it will display you the following menu.

0. [A]ccess Choose the access method to use.

1. [U]pdate Update list of available packages, if possible.

2. [S]elect Request which packages you want on your system.

3. [I]nstall Install and upgrade wanted packages.

4. [C]onfig Configure any packages that are unconfigured.

5. [R]emove Remove unwanted software.

6. [Q]uit Quit dselect.

APTITUDE

A tool that combines dselect's menu based gui and apt-get's cli is "aptitude". Here are the common usage:

aptitude search <package name>  # searches packages related to the one you specified (apt-get seems not to have this feature)

aptitude update  # update package list from APT repository

aptitude install <package name>  # installs a package

aptitude install <package name>-  # removes a package (w/ leading dash)

aptitude remove <package name>  # same as above

aptitude full-upgrade  # upgrades all installed packages

aptitude safe-upgrade  # conservative version of the above command

aptitude autoclean  # removes already-downloaded packages that are no longer available

aptitude help  # shows complete options

Setup NGINX Proxy

Once in a while there are applications that needs to run in unprivileged port
(ports above 1024). What can you do to protect its identity from attacks? Aha!
Use HTTPS.. But how? We can setup another host in front of it (proxy server) to
accept incoming requests via encrypted channel (HTTPS, port 443/tcp) and
redirect that to the backend server (or proxied host) via an uncrypted channel.
In this post, we will use Nginx to have that setup.

1. First, let's say we have VM01 that runs an application on port 8081.

2. Now, let's spin up another host, VM02 (Centos 7.3 w/ SELinux in enforcing
mode), and install Nginx. First, be sure that the nginx repo is enabled.

[root@vm02 ~]# cat /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/x86_64/
gpgcheck=0
enabled=1
[root@vm02 ~]#

3. Install nginx package.

[root@vm02 ~]# yum install -y nginx

4. Start and enable nginx at boot

[root@vm02 ~]# systemctl start nginx
[root@vm02 ~]# systemctl enable nginx
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
[root@vm02 ~]#

5. Create selfsigned certificates. In this part, you may use `genkey` or
`openssl`. I always wanted the openssl way because its faster. BTW, don't
memorize the command below. Just be familiar with it because you can always see
it inside `/etc/pki/tls/certs/make-dummy-cert`

[root@vm02 ~]# /usr/bin/openssl req -newkey rsa:2048 -keyout vm02.key -nodes -x509 -days 365 -out vm02.crt
Generating a 2048 bit RSA private key
.........+++
..........................................................................................+++
writing new private key to 'vm02.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:California
Locality Name (eg, city) [Default City]:Los Angeles
Organization Name (eg, company) [Default Company Ltd]:dummy
Organizational Unit Name (eg, section) []:dummy
Common Name (eg, your name or your server's hostname) []:vm02
Email Address []:dummy@nxdomain.com
[root@vm02 ~]#

6. Move the certificates to the correct paths and run `restorecon` to make sure
SELinux contexts are correct.

[root@vm02 ~]# mv *crt /etc/pki/tls/certs
[root@vm02 ~]# mv *key /etc/pki/tls/private/                                                                                                                                                                [root@vm02 ~]# restorecon -Rv /etc/pki/tls/
restorecon reset /etc/pki/tls/certs/vm02.crt context unconfined_u:object_r:admin_home_t:s0->unconfined_u:object_r:cert_t:s0
restorecon reset /etc/pki/tls/private/vm02.key context unconfined_u:object_r:admin_home_t:s0->unconfined_u:object_r:cert_t:s0
[root@vm02 ~]#

7. Update nginx configuration to use SSL, point to the correct certificates
(ssl_certificate_*), and activate reverse proxy (proxy_pass). We will just use
the default config and use the minimal directives needed for simplicity.

[root@vm02 ~]# cat /etc/nginx/conf.d/default.conf
server {
    listen       443;
    server_name  vm02;
    ssl on;
    ssl_certificate /etc/pki/tls/certs/vm02.crt;
    ssl_certificate_key /etc/pki/tls/private/vm02.key;

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    location / {
        proxy_pass   http://vm01:8081;
    }
}
[root@vm02 ~]#

8. Open up port 443/tcp on the firewall to allow incoming connections. You may
use firewall-cmd's "--add-service" or "--add-port". Let's use "--add-service"
since there is already an existing service defined for HTTPS.

[root@vm02 ~]# firewall-cmd --add-service=https --permanent
success
[root@vm02 ~]# firewall-cmd --reload
success
[root@vm02 ~]#

9. Activate this SELinux boolean to allow HTTP to forward requests to our
upstream server (VM01).

[root@vm02 ~]# setsebool -P httpd_can_network_connect on
[root@vm02 ~]#

10. Now, our proxy server is ready. Let's try connecting. It must display the
data from the upstream like in #1.

So that are the basics steps in hiding your application via a proxy server.
This is very important if your application accepts user details like username
and passwords. You never want your credentials to be sent in cleartext!

Hope you learned something from this post :)

Building an RPM package

INTRODUCTION

  In this post, I will teach you how to create an RPM package from a source code. This tutorial assumes that
you already have the tarballed source code ready to be unpacked.

  As an example in our previous post about mrepo, we have installed it from source and not by using RPM
so that is a perfect way to demomstrate the RPM creation.

1. Install the needed packages

yum install -y rpm-build rpmdevtools
# rpm-build is required which contains "rpmbuild" command
# rpmdevtools is optional which is helpful in creating the directory tree

2. Create the directory tree

rpmdev-setuptree
# that command will create /root/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}

3. Copy the tarballed source file to SOURCES

cp mrepo-0.8.7.tar.bz2 /root/rpmbuild/SOURCES/

4. Copy the spec file to SPECS. It is usually included inside the tarball so unpack the tarball to /tmp first then
get the spec file from there.

tar xvf mrepo-0.8.7.tar.bz2 -C /tmp
cp /tmp/mrepo-0.8.7/mrepo.spec /root/rpmbuild/SPECS

INFO: A spec file describes the software and contains instructions on how to install the software. Creation
and detailed discussion of a spec file is not covered in this post but I might create one soon.

5. Now create the RPMs (src and binary)

rpmbuild -ba /root/rpmbuild/SPECS/mrepo.spec
# that command will create the following files
#  /root/rpmbuilds/RPMS/noarch/mrepo-0.8.7-1.noarch.rpm -> the actual rpm you can install
#  /root/rombuild/SRPMS/mrepo-0.8.7-1.src.rpm -> contains the original source code and the spec file

6. Validate by installing the rpm

rpm -ivh /root/rpmbuild/noarch/mrepo-0.8.7-1.noarch.rpm

SOURCES

Other tutorials:
http://www.tldp.org/HOWTO/RPM-HOWTO/build.html
http://www.thegeekstuff.com/2015/02/rpm-build-package-example/

Understanding Linux load average

We often hear the term “load average” but how much do we know about it? In this post, I will try my best to explain in the fewest letters everything we need to know about load average.

First, how do we determine the load average? We can use commands like uptime and top. Both will present you 3 values.

bash-3.2$ uptime
 03:46:39 up 703 days, 11:00,  2 users,  load average: 3.04, 3.10, 3.08
bash-3.2$

From the output above, 3.04, 3.10, and 3.08 is the load average for the past 1, 5, and 15 minutes respectively. You may also use “cat procloadavg” and will get same result for the 1st 3 columns.

Now, what is LOAD AVERAGE? It is the average of all cpu loads on your system.

What is a CPU LOAD? It is the NUMBER of processes using + NUMBER of processes in queue for a single cpu core. It is NOT the CPU usage.

What is CPU USAGE? It tells us how active is your cpu cores.

To determine CPU LOAD on each cpu core, we follow these 2 formulas:

load per cpu core = (load average) / (# of cpu cores)

load per cpu core = (processes using the cpu core) + (processes inqueue for that cpu core)

where:

load average = the one reported in procloadavg, uptime, top, etc ..

# of cores = grep -c ^proc proccpuinfo

To better illustrate, look at the scenarios below for a 1-core and 2-core machine.

1-core machine (CPU A only):

0 process using CPU A = load average of 0 (under capacity)

1 process using CPU A = load average of 1 (at max capacity)

1 process using CPU A, 1 process waiting in line = load average of 2 (overcapacity)

1 process using CPU A, 2 process waiting in line = load average of 3 (overcapacity)

… and so on ..

2-core machine (CPU A and CPU B):

0 process using CPU A, 0 process using CPU B = load average of 0 (under capacity)

1 process using CPU A, 0 process using CPU B = load average of 0.5 (under capacity)

1 process using CPU A, 1 process using CPU B = load average of 1 (max capacity)

1 process using CPU A, 1 process using CPU B, 1 processes waiting in line = load average of 1.5 (slightly overcapacity)

1 process using CPU A, 1 process using CPU B, 2 processes waiting in line = load average of 2 (overcapacity)

… and so on ..

Based from the examples above, a load average of 1 means all of your cpu cores are at max capacity. Load average of 2 means each cpu core has 1 process running and 1 process waiting in line.

There are instances where your load average is high but the CPU usage is low. An example is when you have several hunged processes occupying all your cpu cores. Sinced hunged, those processes doesn’t generate CPU activity but they still hold the CPU cores. When all CPU cores are being held, no other processes can use them. So if processes can’t get hold of the CPU, this can slow down your system.

In short, a system with LOW CPU USAGE but with HIGH LOAD AVERAGE can still slow down your system.