Useful AWK Commands

prints column 1	awk '{print $1}'
filters lines with "abc" (same as grep abc)	awk '/'abc'/'
filters lines with "abc" and prints column 1	awk '/'abc'/ {print $1}'
prints column 2 and 3	awk '{print $2,$3}'
prints all columns	awk '{print $0}'
gets instructions from a file	awk -f script.awk in putfile
executes a program using data from input file	awk 'program' inputfile
classic hello world	awk "BEGIN {print \"Hello World\"}"
Print what's entered on the command line until EOF (^D)	awk '{print}'
awk script for the classic "Hello, world!" (make it executable with chmod and run it as-is)	#! /bin/awk -f BEGIN { print "Hello, world!" }
Comments in awk scripts	# This is a program that prints \ "Hello, world!" # and exits
prints first column on a : delimited file	awk -F: '{print $1}' /etc/passwd
prints length of each line	cat /etc/passwd | awk '{print length($0)}'
prints the length of the longest line	cat /etc/passwd | awk '{if (length($0)>max) max=length($0)} END {print max}'
shows all lines longer than 80	cat passwd.sample | awk 'length($0)>80'
same as above but instead of printing the lines, this prints the lengths of those lines	cat passwd.sample | awk 'length($0)>80 {print length($0)}'
prints line w/ more than 2 fields (NF stands for Number of Fields)	awk 'NF>2' file.txt
Print seven random numbers from 0 to 100	awk 'BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand()) }'
prints the sum of file sizes (in bytes) in the current directory	ls -l | awk '{sum=sum+$5} END {print sum}'
Print the total number of kilobytes used by files in the current directory	ls -l . | awk '{ x += $5 } ; END { print "total kilobytes: " (x + 1023)/1024 }'
print number of lines (NR stands for Number of Rows)	awk 'END {print NR}' passwd.sample
prints lines whose field 3 is equal to "root"	ls -l | awk '$3=="root"'
prints bcd	echo aaaabcd | awk '{ sub(/a+/, ""); print }'
prints even-numbered lines	awk 'NR%2==0' file.txt
weird stuff	ls -lh | awk '{ owner = $3 ; $3 = $3 " 0wnz"; print $3 }' | uniq
prints number of lines	awk 'END {print NR}' passwd.sample
extract username and uid from inputfile	awk -F: '{print "Username: " $1"\t\tUID: "$3}' passwd.sample
prints only those lines containing "abc"	awk '/abc/ {print}' myfile.txt
uses a more complicated regular expression	awk '/[0-9]+\.[0-9]*/ {print}' inputfile
prints lines whose 2nd field is equal to "2"	ls -l | awk '$2==2 {print}'   * In this example, $2==2 is the condition. If that is not     satisfied, it will not execute the action which is {print}
uses another regular expression	ls -l | awk '$3 ~ /abc/ {print}'
prints all lines that doesn't have "abc"	ls -l | awk '! /abc/ {print}'
uses && operator for a more precise selection	awk '$3=="abc" && $5==4096 {print}' ls.sample

Netapp CDot Administration

Cluster vs SVM Admins

---------------------

SVM Admins

 - can only administer their own SVM

 - SVMs are short for Storage Virtual Machines (formerly called vservers)

 - manages resources to that SVM (volumes, protocols, lifs, etc..)

Cluster Admins

 - can administer both the cluster and all SVMs underneath

 - can setup SVMs and delegate roles to SVM admins

How to manage DataONTAP?

------------------------

Ways:

1. command line (tcsh shell)

  cluser admins:

    a. serial port (default admin account: admin)

    b. ssh

        - enabled by default

        - account must be permitted via ssh login (`security login -application`)

        - if using AD, domain ssh access methos must be "domain"

        - if using ipv6, ipv6 must be configure on the cluster

    c. rsh/telnet

        - disabled by default since they are insecure protocols

        - to enable, see tutorial below

2. url

Some notes on SSH

-----------------

 - SSHv1 is not supported, only SSHv2 (cDOT 8.3)

 - DOT supports 64 concurrent SSH connections per node

 - if rate of incoming connections is higher than 10 per second,

   service is temporarily disabled for 60 seconds

 - if using AD, use same username and domain create from DOT

Privileged Levels

-----------------

Levels:

admin	cluster_name::> - most commands and parameters are available - used for common routine tasks
advanced	cluster_name::*> - commands here are infrequently used - requires advanced knowledge
diagnostics	what does the prompt looks like?? - commands here are potentially disruptve - used by support personnel to diagnose and fix problems

note:

  - command preceded by `*` can only be executed under advanced

    privileged level or higher

Different Shells

----------------

* for cluster admins only *

1. clustershell

    - default shell when you log in

    - used to manage the cluster

2. nodeshell

    - shell for a specific node

    - many commands from nodeshell can be accessed from clustershell

3. systemshell

    - used for diagnostics/troubleshooting purposes

    - requires diag priveleged level

    - intended for technical support use

Display Preferences

-------------------

What preferences can I set?

- privilege level of the command session

- whether confirmations are issued for potentially disruptive commands

- whether show commands display all fields

- the character or characters to use as the field separator

- the default unit when reporting data sizes

- the number of rows the screen displays in the current cli session

  before the interface pauses output (if the preferred number of rows

  is not specified, it is automatically adjusted based on the actual

  height of the terminal. if the actual height is undefined, the default

  number of rows is 24)

- the default storage virtual machine (svm) or node

- whether a continuing command should stop if it encounters an error

Ways of executing Commands

--------------------------

1. full path

cluster1::> storage disk show

2. per directory

cluster1::> storage

cluster1::storage> disk

cluster1::storage disk> show

  * use `top` to go to top level

  * use `up` or `..` to go one level higher

3. abbreviating commands

cluster1::> st d sh

Rules for specifying values in CLI

----------------------------------

- a value can be a number, string, boolean specifier

- some accepts a comma-separated list (doesn't need "")

- enclose values with spaces inside ""

- `?` is interpreted as help

- example of case-insensitive command is `vserver cifs `

- nodenames, volumes, aggregates, LIFs, etc .. are case-sensitive

- to clear a value, use "" or -

- lines starting with # are comments

some examples:

# sets a comment then deletes it

cluster1::> vserver create -vserver vs0 -subtype default -rootvolume root_vs0

-aggregate aggr1 -rootvolume-security-style unix -language C.UTF-8 -is-repository

false -ipspace ipspaceA -comment "My SVM"

cluster1::> vserver modify -vserver vs0 -comment ""

# tells that the command do

cluster1::> security login create -vserver vs0 -user-or-group-name new-admin

-application ssh -authmethod password #This command creates a new user account

Query operators

---------------

*	match all entries # list all volumes with "tmp" on their name volume show -volume *tmp*
!	NOT operator # indicates not to match vs0 !vs0
|	OR operator # vs0 or vs1 vs0 | vs1 # matches a, anything that starts with b, or those with c a | b* | *c*
..	range operator # any value from 5 to 10 5..10
<	less than operator
>	greater than operator
<=	less then or equal to
>=	greater than or equal to
{query}	extended query   - must be specified as the 1st argument after     the command name before any other parameters   - can only be used in `modify` and `delete`     commands   - not applicable on `create` or `show` commands   - example of confusing exended query: p.22 of     "ONTAP 9 System Administration Reference" # offlines all volumes whose names contain "tmp" volume modify {-volume *tmp*} -state offline
"string literal"	you may also query any characters as literals by enclosing them in "" e.g "^" "*"
using multiple query operators	# displays all volumes whose size is greater than 1GB, # percent used is less than 50% and not in SVM vs1 volume show -size >1GB -percent-used <50 -vserver="" p="" vs1="">

Commands

--------

nodeshell/clustershell

# querying clustershell cli vserver options -vserver -option-name ? # accessing vserver man page man vserver options # cluster shell help help [?|help] # node shell help help [?|help] # accessing node shell system node run -node   * local - node you use to access the cluster   * `system node run` alias is `run` # exits/return to previous shell (if there is any) exit CTRL+D

ssh	# connecting using local ssh account ssh joe@cluster.ip # connecting using AD account ssh DOMAIN\\joe@cluster.ip ssh "DOMAIN\joe"@cluster.ip # executing remote command via ssh ssh joe@cluster.ip cluster show

history/redo/reissue

# prints history history # redo nth command in history redo # redo command executed Nth number ago redo -

privelege levels

# changes privilege level set -privilege

setting display preferences

# key command set # sets number of rows on the current session rows # changes separator and units used set -showseparator "," -units GB

displaying

# displays full details cluster1::> volume show -instance Vserver Name: cluster1-1 Volume Name: vol0 Aggregate Name: aggr0 ... Space Guarantee Style: volume Space Guarantee in Effect: true ... Press to page down, for next line, or 'q' to quit... ... cluster1::> # displays only the fields you specify cluster1::> volume show -fields space-guarantee,space-guarantee-enabled vserver volume space-guarantee space-guarantee-enabled -------- ------ --------------- ----------------------- cluster1-1 vol0 volume true cluster1-2 vol0 volume true ... cluster1::> # show valid fields show -fields ?

Command Shortcuts

-----------------

- DataOntap shell is based on unix tcsh

- below are copy pasted from the pdf

If you want to ..	Use the ff keyboard shortcut
move the cursor back by one character	Ctrl-B / Back arrow
move the cursor forward by one character	Ctrl-F / Forward arrow
move the cursor back by one word	Esc-B
move the cursor forward by one word	Esc-F
move the cursor to the beginning of the line	Ctrl-A
move the cursor to the end of the line	Ctrl-E
Remove the content of the command line from the beginning of the line to the cursor, and save it in the cut buffer The cut buffer acts like temporary memory, similar to what is called a clipboard in some programs.	Ctrl-U
Remove the content of the command line from the cursor to the end of the line, and save it in the cut buffer	Ctrl-K
Remove the content of the command line from the cursor to the end of the following word, and save it in the cut buffer	Esc-D
Remove the word before the cursor, and save it in the cut buffer	Ctrl-W
Yank the content of the cut buffer, and push it into the command line at the cursor	Ctrl-Y
Delete the character before the cursor	Ctrl-H / Backspace
Delete the character where the cursor is	Ctrl-D
Clear the line	Ctrl-C
Clear the screen	Ctrl-L
Replace the current content of the command line with the previous entry on the history list With each repetition of the keyboard shortcut, the history cursor moves to the previous entry.	Ctrl-P / Esc-P / Up arrow
Replace the current content of the command line with the next entry on the history list With each repetition of the keyboard shortcut, the history cursor moves to the next entry	Ctrl-N / Esc-N / Down arrow
Expand a partially entered command or list valid input from the current editing position	Tab / Ctrl-I
Display context-sensitive help	?
Escape the special mapping for the question mark (“?”) character For instance, to enter a question mark into a command's argument, press Esc and then the “?” character.	Esc-?
Start TTY output	Ctrl-Q
Stop TTY output	Ctrl-S

Tutorials

---------

Enabling rsh/telnet

1. Use `system services firewall policy clone` command to create    a new management firewall policy based from the default which    is "mgmt" firewall policy 2. Use `system services firewall policy create` command to enable    telnet or rsh on the new firewall policy 3. Use `network interfaces modify` command to associate the new    policy with the cluster management LIF 4. Then to access your cluster:      telnet cluster.ip      rsh cluster.ip -l username:password

HTTP Authentication in Nginx

Introduction

------------

- available on both Nginx opensource and Nginx PLUS

- requires password file creation file creation tool (e.g apache-2 utils)

- can enforce restriction based on:

    a. IP address

    b. geographical location

Setting up authentication

-------------------------

1. Install apache2-utils

     # yum install httpd-tools

2. Create a password file containing the 1st user

     # htpasswd -c /etc/nginx/.htpasswd_users user1

3. Add another user if needed

     # htpasswd /etc/nginx/.htpasswd_users user2

4. Make sure selinux (if selinux is enabled) and permission is correct on the password file

     # chown nginx /etc/nginx/.htpasswd_users

     # restorecon -Rv /etc/nginx

5. Add the following directives on the location you wish to protect

  location ~ \.(pdf|PDF) {

    root /payroll;

    auth_basic "restricted area";

    auth_basic_user_file /etc/nginx/.htpasswd_users;

  }

6. Restart nginx

     # systemctl restart nginx

7. Try downloading a file from that location

     # wget http://server.home.net/01-01-1970.pdf --user=user1 --password=pass123

Common configurations:

limiting access to the whole website	server {     ...     # Restrict access to all location below     auth_basic  "My personal files";     auth_basic_user_file /etc/nginx/.htpasswd_users;     location ~ \.(mp3|mp4) {       root /music;     }     location ~ \.(jpg|png) {       root /pictures;     } }
bypassing `server` level authentication	server {     ...     # Restrict access to all location below except for 1 location     auth_basic  "My personal files";     auth_basic_user_file /etc/nginx/.htpasswd_users;     location ~ \.(mp3|mp4) {  # this will require credentials       root /music;     }     location ~ \.(jpg|png) {  # this will require credentials       root /pictures;     }     location /public {       root /downloads;       auth_basic off;  # this wouldn't require credentials     } }
Restricting by either source IP or credentials	location ~ \.txt {   satisfy any;         # this will honor source IP or credentials   allow 192.168.1.11;  # Nginx will allow from this source IP only   deny all;            # others are deny   root /files/public;   auth_basic "my personal files";   auth_basic_user_file /etc/nginx/.htpasswd_users; }
Restricting by both source IP and credentials	location ~ \.txt {   satisfy all;         # source IP and credentials must be correct   allow 192.168.1.11;  # Nginx will allow from this source IP only   deny all;            # others are deny   root /files/public;   auth_basic "my personal files";   auth_basic_user_file /etc/nginx/.htpasswd_users; }

Nimble Storage

Introduction

------------

- there are no LUNs in nimble, just volumes (volume and LUN is interchangeable)

- Nimble can only be used as LUNs; If you want to create  a CIFS share, you must

  create a LUN in Nimble and mount that to a host then do the sharing there

- it uses its SSDs to cache frequently used data

- has RAID 6 protection for dual-disk failures

- Adaptive Flash is utilizing SSDs instead of DRAM (DRAM is more expensive than

  SSDs) to cache frequently used data

- it uses thin provisioning

- it uses inline compression w/ no performance impact (compresses data as it is

  written to the storage array)

- It uses snapshots so that backup and restore time takes only seconds because

  backup is located within the array

Cabling

-------

Rules:

1. interface pair (e.g eth1 on controller A and eth2 on controller B) must be on

   the same switch and subnet (see sample topology below)

2. IP address are only assigned on the ports on controller A; controller B ports

   will automatically inherit those IP addresse during failover

3. the iSCSI discovery address must be the data IP

4. mgt IP and data IP must be on different subnet

5. data links are those interfaces w/ higher bandwith like tg1/2 (10 GBps) while

   mgt links are those with lower like eth1/2 (1 Gbps)

6. only 1 default gateway is needed which is for the MGT IP (Nimble OS 1.4)

Disks

-----

- if two disks are reconstructing simultaneously, failure of a third disk,

  though unlikely, could cause data loss

- you can view disk status by either going on the web gui or on cli

- on web gui, here are the disk states

Displaying

# list all disks together w/ their physical info like slot #, Serial #, Size, etc disk --list # displays detailed info on a particualr disk disk --info

Ways of Accessing Nimble Storage

--------------------------------

1. via ssh

2. via web url

Note: default credential is admin/admin

Working w/ Multiple Tables in MySQL

Using JOIN

----------

# this joins 2 table via "name" fields

#

# table 1

+----------+-----------+---------+------+------------+------------+

| name     | owner     | species | sex  | birth      | death      |

+----------+-----------+---------+------+------------+------------+

| Whistler | Gwen      | bird    | NULL | 1997-12-09 | NULL       |

| Gunner   | Mark      | dog     | m    | 1992-01-14 | NULL       |

| Heath    | Stephanie | penguin | m    | 2001-12-12 | NULL       |

| Winnona  | Stephanie | dog     | m    | 2009-03-18 | NULL       |

| Hunter   | Stephanie | fish    | f    | 1997-09-21 | NULL       |

| Splitter | Gwen      | fish    | m    | 1996-01-12 | 1999-03-04 |

| Styder   | Gwen      | fish    | f    | 2018-05-17 | NULL       |

+----------+-----------+---------+------+------------+------------+

# table 2

+----------+------------+------------+------------+

| name     | date       | type       | remark     |

+----------+------------+------------+------------+

| Whistler | 0000-00-00 | litter     | arghhh     |

| Gunner   | 2001-01-13 | litter     | not again  |

| Heath    | 1991-09-24 | broken rib | huhu       |

| Winnona  | 2003-09-09 | pregnant   | yeyy       |

| Hunter   | 2011-12-12 | sick       | waaa       |

| Splitter | 1993-01-01 | sick       | waaa       |

| Styder   | 1994-05-06 | pregnant   | again yeyy |

+----------+------------+------------+------------+

# query

mysql>

mysql> SELECT pet.name,

    -> TIMESTAMPDIFF(YEAR,birth,date) AS age,

    -> remark

    -> FROM pet INNER JOIN event

    ->   ON pet.name = event.name

    -> WHERE event.type = 'litter';

Using UNION

-----------

This is alternative to using OR:

mysql> select * from pet where owner='Gwen' union select * from pet where owner='Stephanie';

+----------+-----------+---------+------+------------+------------+

| name     | owner     | species | sex  | birth      | death      |

+----------+-----------+---------+------+------------+------------+

| Whistler | Gwen      | bird    | NULL | 1997-12-09 | NULL       |

| Splitter | Gwen      | fish    | m    | 1996-01-12 | 1999-03-04 |

| Styder   | Gwen      | fish    | f    | 2018-05-17 | NULL       |

| Heath    | Stephanie | penguin | m    | 2001-12-12 | NULL       |

| Winnona  | Stephanie | dog     | m    | 2009-03-18 | NULL       |

| Hunter   | Stephanie | fish    | f    | 1997-09-21 | NULL       |

+----------+-----------+---------+------+------------+------------+

6 rows in set (0.00 sec)

mysql>