Tuesday, June 19, 2018

Nginx Reverse Proxy


Introduction
------------

- proxy server servers content in behalf of another server
- proxy servers are used to hide the identity of other servers
- a proxied request means that the client request was redirected to a backend
  server
- when nginx proxies a request, it sends it to the proxied server, fetches the
  response, and sends the response back to the client
- examples of proxied servers are:
    a. another Nginx server
    b. any kind of HTTP server
    c. a non-HTTP server w/c can run applications built on python, php, etc..
- nginx proxies request using any specified protocol like:
    a. FastCGI
    b. uwsgi
    c. SCGI
    d. memcached
- a proxied server is also known as backend/upstream server
- common use of reverse proxy is to provide load balancing
- a proxied request has a "Via" header field

Ways of proxying requests
-------------------------

Simple way of passing a request
to an HTTP proxied server
server {
  listen 80;
  server_name proxy.example.com;

  location /backend/ {
    proxy_pass http://backend.example.com/;
      # this will proxy the request by re-writing the url
      # from:
      # to:
    ...
  }

  location /proxy2 {
    proxy_pass http://proxy2.example.com/;
      # this version is valid
  }


  location ~ /proxy3/? {
    proxy_pass http://proxy3.example.com/;
      # this version is also valid
  }


  location /proxy4 {
    proxy_pass http://proxy4.example.com;
      # this doesn't work correctly
      # in my testing (nginx/1.10.2)
  }
}
If using a regex to specify
location, appending `/` or
`/any/path` at the end of the
proxied host is not allowed

related error:
"proxy_pass" cannot have URI part
in location given by regular
expression
location ~ /notes/? {
    proxy_pass http://www.example.com;
}
A proxied server can also be set
as an IP:PORT combination
location ~ \.php {
    proxy_pass http://127.0.0.1:8000;
      # I'm encoutering "failed (13: Permission denied)
      # while connecting to upstream" on this (nginx/1.10.2)
}

location /notes/ {
      # I'm encoutering "failed (13: Permission denied)
      # while connecting to upstream" on this (nginx/1.10.2)
}

Protocols for proxying a request
--------------------------------

fastcgi_pass   -- passes a request to a FastCGI server
uwsgi_pass     -- passes a request to a uwsgi server
scgi_pass      -- passes a request to an SCGI server
memcached_pass -- passes a request to a memcached server

Passing request Headers
-----------------------

- by default, nginx redefines 2 headers: "Host" and "Connection"
- nginx eliminate empty header fields
- "Host" is set to $proxy_host
- "Connection" is set to "close"

Example of HTTP headers:


To redefine value for a header
location /some/path/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://localhost:8000;
}
To prevent a header from being
passed to the proxied server,
set it to empty string
location /some/path/ {
    proxy_set_header Accept-Encoding "";
    proxy_pass http://localhost:8000;
}

Buffering
---------

- by default, a response from proxied servers is put into internal buffers and
  is not sent to client until the whole response is received
- this allows the proxied server to process response quickly
- `proxy_buffering` is set to on by default
- `proxy_buffers` sets the size and number of buffers allocated per request
- `proxy_buffer_size` controls the size of the 1st part of the response from
   the proxied server which usually contains smaller response header compared
   to the rest of the response
- if buffering is disabled, the response from the proxied server is sent
  synchronously to client while being received by the proxy server (this
  behaviour is desirable for clients who needs to get response as soon as
  possible)

Size of the buffer for the first
part is set smaller (2k) compared
to the size of buffers for the
rest of the response (4k)
location /some/path/ {
    proxy_buffers 16 4k;
    proxy_buffer_size 2k;
    proxy_pass http://localhost:8000;
}
Disables buffering on a specific
location
location /some/path/ {
    proxy_buffering off;
    proxy_pass http://localhost:8000;
}

Choosing an outgoing address
----------------------------

- if a proxy server has multiple IPs, you can choose an IP for connecting to the
  proxied server by using `proxy_bind` directive
- useful if the proxied server can only accept connections from an IP range

Sets different IPs in connecting
to proxied server
location /app1/ {
    proxy_bind 127.0.0.1;
    proxy_pass http://example.com/app1/;
}

location /app2/ {
    proxy_bind 127.0.0.2;
    proxy_pass http://example.com/app2/;
}
IP address can also be a variable
location /app3/ {
    proxy_bind $server_addr;
    proxy_pass http://example.com/app3/;
}

Monday, June 18, 2018

Qtrees


Overview
--------

Basic Details:

- partition volumes into smaller segments
- properties are: quotas, backups, security style, CIFS oplocks settings
- can be used in creating CIFS shares
- you can create several qtrees inside a volume and each of them can have
  different quotas

Properties:

Opportunistic and Lease oplocks

  Traditional oplocks (opportunistic locks) and lease oplocks enable a CIFS
client in certain file-sharing scenarios to perform client-side caching of
read-ahead, write-behind, and lock information

  A client can then read from or write to a file without regularly reminding the
server that it needs access to the file in question. This improves performance
by reducing network traffic.

Commands
--------

Displaying
# quick view of status
qtree status [-i|-v]

# displays statistics
qtree stats
** stats are reset upon system reboot
** stats are reset when volume containing it is brought online
** stats are reset when you trigger this command: qtree stats -z

Creating
# creates a qtree under a specified volume
qtree create /vol// -m
  ** mode is permission
  ** you can see default mode under wafl.default_qtree_mode option

# creates a qtree under the root volume (/vol/vol0)
qtree create

Modifying
# Enabling/Disabling for entire storage
cifs.oplocks.enable on
cifs.oplocks.enable off

# Enabling/Disabling for qtrees qtree oplocks /vol/vol2/proj enable
qtree oplocks /vol/vol1/qtree enable
qtree oplocks /vol/vol1/qtree disable

Renaming
1. volume that contains the qtree must be available
  on a UNIX client: mount filer:/vol/my_vol/ /mnt
  on a Windows client: map the qtree into windows explorer

2. find the qtree and rename it
  on a UNIX client: mv /mnt/qtree_old /mnt/qtree_new
  on a Windows client: rename using windows explorer
** you can rename qtree depending on the qtree permissions

Deleting
# basic
qtree delete /path/to/qtree

# force (if for some reason directory is not emtpy)
qtree -f delete /path/to/qtree
  * you might need to go to "priv set advanced"

Tutorials
---------

Actual commands on CIFS creation
qtree create /vol/data_share_vol/data_share_qtree
qtree security /vol/data_share_vol/data_share_qtree ntfs
qtree oplocks /vol/data_share_vol/data_share_qtree enable
cifs shares -add data$ /vol/data_share_vol/data_share_qtree


Sunday, June 17, 2018

Igroups


What are igroups?
-----------------

--> contains initiators that can access your storage
--> tables of WWPNs (for FCP) or iqn host node names (for iSCSI) that are
    allowed to access a LUN
--> igroups can have multiple initiators
--> multiple igroups can have same initiator
--> a LUN cannot be mapped to multiple igroups having same initiator
--> an initiator can be a member of igroups of different ostypes

Commands
--------

Displaying
# prints igroups
igroup show
igroup show -v

Creating
# creates an ISCSI igroup
igroup create -i -t
igroup create -i -t windows_2008 win_host5_group2 iqn.1991-05.com.microsoft:host5.domain.com
** os types: solaris, Solaris_efi, windows, windows_gpt,
             windows_2008, hpux, aix, linux, netware, vmware, xen, hyper_v

# creates an FCP igroup
igroup create -f -t …
igroup create -f -t aix aix-igroup3 10:00:00:00:0c:2b:cc:92
** NOTE: you must use WWPN (portname) and not WWN

Managing
# adding an initiator
igroup add
igroup add INTDB01 10:00:00:00:c9:5e:ca:5e

# removing an initiator
iggroup remove
igroup remove INTDB01 10:00:00:00:c9:5e:ca:5e

Deleting
# LUNS must be unmapped first
igroup destroy

# this will remove all LUN maps and destroy the igroup
igroup destroy -f

Modifying
# renaming
igroup rename
z
** this will not impact access to LUNS that are inside the igroup

changing the WWNN (World Wide Node Name) of a system
----------------------------------------------------

- both HA pairs must have same WWNN
- in order to change WWN, use this command: fcp nodename 50:0a:09:80:82:02:8d:ff
- for that to take effect, both HA pairs must be rebooted

Troubleshooting
---------------

different WWNN on HA pairs
message:

from autosupport:
HA Group Notification from filer01 (FILER SCSI TARGET MISCONFIGURED) ERROR

from /etc/messages:
Tue Feb 17 17:12:39 EST [filer01:scsitarget.cluster.misconfigured:notice]: Filer SCSI Target Misconfigured. Run 'lun config_check'.

resolution / fix:
1. match WWNs of both nodes (they must be the same)
2. reboot both nodes

Saturday, June 16, 2018

Python Dictionaries

Introduction
------------

- another data type in Python
- consists of `key: value` pair inside `{}`
- sequences are indexed via element number (fruit[0]) while dictionaries are indexed via keys (telephone_number['John'])
- keys are immutable
- keys can be any of the following:
    a. strings
    b. numbers
    c. tuples (if contains only immutable object)
- you cannot use lists as keys because they can be modified using append(), insert(), etc..
- adding value on an existing key replaces the old value
- extracting a value from a non-existing key will result to `KeyError`

How to use dictionaries?
------------------------

Simple demo on how to use dictionaries
>>> tel = {'jack': 4098, 'sape': 4139}  # creating a dictionary
>>> tel['guido'] = 4127  # adds an entry
>>> tel
{'sape': 4139, 'guido': 4127, 'jack': 4098}
>>> tel['jack']
4098
>>> del tel['sape']  # deleting an entry
>>> tel['irv'] = 4127
>>> tel
{'guido': 4127, 'irv': 4127, 'jack': 4098}
>>> list(tel.keys())
['irv', 'guido', 'jack']
>>> sorted(tel.keys())
['guido', 'irv', 'jack']
>>> 'guido' in tel
True
>>> 'jack' not in tel
False
Creating an empty dictionary and adding
an entry
>>> empty = {}
>>> empty
{}
>>>
>>> empty['first entry'] = 1
>>> empty
{'first entry': 1}
>>>
Listing all keys of a dictionary
>>> tel.keys()
dict_keys(['jack', 'sape'])
>>>
Sorting keys
>>> sorted(dict.keys())
You can use dic() constructor to build a
dictionary directly from a sequence of
key-value pairs
>>> dict([('sape', 4139), ('guido', 4127), ('jack', 4098)])
{'sape': 4139, 'jack': 4098, 'guido': 4127}
>>>
>>> dict(sape=4139, guido=4127, jack=4098)  # alternative way, useful if keys are
{'sape': 4139, 'jack': 4098, 'guido': 4127} # simple strings
>>>
Creating a dict comprehension
>>> {x: x**2 for x in (2, 4, 6)}
{2: 4, 4: 16, 6: 36}
>>>
Other operations that can be done on
dictionaries
dict.clear()
dict.copy()
dict.fromkeys(seq[, value])
iter(dict)

Operations on dictionaries
--------------------------

Getting keys values
using get()
Format:
dict.get(KEY, VALUE)
 
Usage:
- a non-existent key returns None
- you may provide a value on non-existing keys


>>> dict = { 'name': 'john', 'tel': '123456' }
>>> dict.get('name')
'john'
>>> dict.get('occupation')
>>> print(dict.get('occupation'))
None
>>> print(dict.get('occupation', 'taxi driver'))
taxi driver
>>>
Merging
# How to merge two dictionaries
# in Python 3.5+

>>> x = {'a': 1, 'b': 2}
>>> y = {'b': 3, 'c': 4}
>>> z = {**x, **y}
>>> z
{'c': 4, 'a': 1, 'b': 3}

# In Python 2.x you could
# use this:
>>> z = dict(x, **y)
>>> z
{'a': 1, 'c': 4, 'b': 3}

# In these examples, Python merges dictionary keys
# in the order listed in the expression, overwriting
# duplicates from left to right.
#
Sorting
Use dict.items() to generate a list then feed that into
lambda.

>>> d
{'a': 500, 'b': 1000, 'c': 1}
>>>
>>>
>>> sorted(d.items(), key=lambda x: x[1])
[('c', 1), ('a', 500), ('b', 1000)]
>>>
>>> sorted(d.items(), key=lambda x: x[0])
[('a', 500), ('b', 1000), ('c', 1)]
>>>
>>>

Or you may use `operator` module.

>>> import operator
>>> sorted(d.items(), key=operator.itemgetter(1))
[('c', 1), ('a', 500), ('b', 1000)]
>>>
Ways of unpacking
Using a wrapper function:

 
>>> d
{'a': 1, 'b': 2, 'c': 3}
>>> def unpack_me(x, y, z):
...   print(x, y, z)
...
>>> unpack_me(*d)
a b c
>>>

Friday, June 15, 2018

Netapp Volumes (7-Mode)


Root Volume: /vol/vol0
----------------------

--> this is were DataONTAP is installed and booted
--> minimum size depends on hardware model (consult hwu)
--> fractional reserve must be 100%
--> can be traditional or FlexVol
--> default RAID type is RAID-DP (starting from Data ONTAP 7.3)
--> you can change RAID type by: vol options vol0 raidtype raid4
--> you can designate another root volume: vol options root

Commands
--------

Displaying        
# quick view of size and snapshots
df [-g|-h]

# displays block size
vol status -b

# detailed breakdown of space consumed inside a volume
vol status -S

# amount of space a volume is using within the aggregate (footprint)
vol status -F

# displays language used on each volumes
vol status -l

Creating
# basic
vol create [k|m|g|t]
  -> if -l is not specified, language will be same as the root volume's language

# thick provisioned (w/ space reservation, enabled by default)
vol create [k|m|g|t]
vol create myvolume myaggregate 1g
** you can also add "-s volume" option

# thin provisioned (w/o space reservation)
vol create -s none [k|m|g|t]
vol create -s none myvolume myaggregate 1g

# w/ language is specified --> not sure if this is required in creating ESX LUNS???
vol create -l en_US [k|m|g|t]
vol create devsql13_vol -l en_US aggr2 4t

TIPS:
- it is better to specify values in lower units to increase percentage of getting the desired size: e.g use 15360 GB instead of 15 TB

Modifying
# turns off snap reserve
snap reserve 0

# disables snapshots (any of the 2 commands is applicable)
vol options nosnap on
vol options nosnap 1

# renames a volume (non-disruptive)
vol rename

Deleting
# do steps in order
filer> vol offline /vol/
filer> vol destroy /vol/

NOTES:
  - deleting large volumes will not reclaim aggregate space rightaway
    it will take some time to reclaim all space
  - As an example, a 34 TB can be reclaimed in 24 hours

Resizing
# increase
vol size +[k|m|g|t]

## reduce
vol size -[k|m|g|t]

Tutorials
---------

Actual commands on Volume Creation
vol create data_share_vol -l en_US -s volume aggr2 30720g
vol options data_share_vol nosnap 1
snap reserve data_share_vol 0
Netapp share creation
1. Locate aggregate with enough space for new share(s)

aggr show_space -g
df -g -A

2. Create volumes - create volume, turn off automatic snapshots
   & remove snapshot reserve space

vol create -l en_US -s volume g
vol options nosnap 1
snap reserve 0

note: is usually images[0-9][0-9] & is 250. As of this
      writing filer01 has 2 available: aggr0 & aggr1

2. Create QTrees - create QTree, set security mode to NTFS and
   enable "opportunistic locks"

qtree create /vol//
qtree security /vol// ntfs
qtree oplocks /vol// enable

note: is usually the same as

3. Create CIFS shares
cifs shares -add /vol//

note: is usually "$"


Troubleshooting
---------------

running out of inodes?
Related message:

Fri Nov 20 14:52:54 EST [filer01:wafl.vol.outOfInodes:notice]: file system on Volume data_share_vol is out of inodes

Solution:

1. Check the current inode value of the volume
maxfiles data_share_vol
df -i data_share_vol

2. Increase the max inodes
maxfiles data_share_vol 35000000
  -> 35000000 is a value greater than the current max inodes

3. Verify
maxfiles data_share_vol
df -i data_share_vol
creare_ucode config error
Log message:
Fri Apr 15 14:25:26 GMT [filer02:cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services volumes vol0, backups_vol, backups2_vol, and vdi_vol should have the volume option create_ucode set to On. . 
Fri Apr 15 14:25:26 GMT [filer02:callhome.sys.config:error]: Call home for SYSTEM CONFIGURATION WARNING

Solution:
For each volume, run the following command:
filer> vol options [volname] create_ucode on

Root cause:
Clustered filers in a NetApp Storage Area Network (SAN) environment require the following options to be enabled to guarantee that failover and giveback occur quickly enough to not interfere with host requests to the LUNs. These options are automatically enabled when FCP/iSCSI service is turned on:
  •        volume option create_ucode to on
  •        coredump.timeout.enable to on
  •        coredump.timout.seconds set to 60 or less

Thursday, June 14, 2018

Useful AWK Commands


prints column 1
awk '{print $1}'
filters lines with "abc" (same as grep abc)
awk '/'abc'/'
filters lines with "abc" and prints column 1
awk '/'abc'/ {print $1}'
prints column 2 and 3
awk '{print $2,$3}'
prints all columns
awk '{print $0}'
gets instructions from a file
awk -f script.awk in putfile
executes a program using data from input file
awk 'program' inputfile
classic hello world
awk "BEGIN {print \"Hello World\"}"
Print what's entered on the command line until EOF (^D)
awk '{print}'
awk script for the classic "Hello, world!"
(make it executable with chmod and run it as-is)
#! /bin/awk -f
BEGIN { print "Hello, world!" }
Comments in awk scripts
# This is a program that prints \
"Hello, world!"
# and exits
prints first column on a : delimited file
awk -F: '{print $1}' /etc/passwd
prints length of each line
cat /etc/passwd | awk '{print length($0)}'
prints the length of the longest line
cat /etc/passwd | awk '{if (length($0)>max) max=length($0)} END {print max}'
shows all lines longer than 80
cat passwd.sample | awk 'length($0)>80'
same as above but instead of printing the lines,
this prints the lengths of those lines
cat passwd.sample | awk 'length($0)>80 {print length($0)}'
prints line w/ more than 2 fields (NF stands for Number
of Fields)
awk 'NF>2' file.txt
Print seven random numbers from 0 to 100
awk 'BEGIN { for (i = 1; i <= 7; i++)
print int(101 * rand()) }'
prints the sum of file sizes (in bytes) in the current
directory
ls -l | awk '{sum=sum+$5} END {print sum}'
Print the total number of kilobytes used by files in
the current directory
ls -l . | awk '{ x += $5 } ; END { print "total kilobytes: " (x + 1023)/1024 }'
print number of lines (NR stands for Number of Rows)
awk 'END {print NR}' passwd.sample
prints lines whose field 3 is equal to "root"
ls -l | awk '$3=="root"'
prints bcd
echo aaaabcd | awk '{ sub(/a+/, ""); print }'
prints even-numbered lines
awk 'NR%2==0' file.txt
weird stuff
ls -lh | awk '{ owner = $3 ; $3 = $3 " 0wnz"; print $3 }' | uniq
prints number of lines
awk 'END {print NR}' passwd.sample
extract username and uid from inputfile
awk -F: '{print "Username: " $1"\t\tUID: "$3}' passwd.sample
prints only those lines containing "abc"
awk '/abc/ {print}' myfile.txt
uses a more complicated regular expression
awk '/[0-9]+\.[0-9]*/ {print}' inputfile
prints lines whose 2nd field is equal to "2"
ls -l | awk '$2==2 {print}'
  * In this example, $2==2 is the condition. If that is not
    satisfied, it will not execute the action which is {print}
uses another regular expression
ls -l | awk '$3 ~ /abc/ {print}'
prints all lines that doesn't have "abc"
ls -l | awk '! /abc/ {print}'
uses && operator for a more precise selection
awk '$3=="abc" && $5==4096 {print}' ls.sample