Archive for July 28th, 2007

Jul/07

Simplify data extraction using Linux text utilities

No comments · Posted by nguyen in Linux Docs

Table 1. Sample regular expressions Example Description
[abc] Matches one of “a”, “b”, or “c”
[a-z] Matches any one lowercase letter from “a” to “z”
[A-Z] Matches any one uppercase letter from “A” to “Z”
[0-9] Matches any one number from 0 to 9
[^0-9] Matches any character other than the numbers from 0 to 9
[-0-9] Matches any number from 0 to 9, or a dash (“-”)
[0-9-] Matches any number from 0 to 9, or a dash (“-”)
[^-0-9] Matches any character other than the numbers from 0 to 9, or a dash (“-”)
[a-zA-Z0-9] Matches any alphabetic or numeric character

grep

The grep utility works by searching through each line of a file (or files) for the first occurrence of a given string. If that string is found, the line is printed; otherwise, the line is not printed. The following file, which I’ll name “memo,” illustrates grep’s usage and results.

To: All Employees

From: Human Resources

In order to better serve the needs of our mass market customers, ABC Publishing is integrating the groups selling to this channel for ABC General Reference and ABC Computer Publishing. This change will allow us to better coordinate our selling and marketing efforts, as well as simplify ABC’s relationships with these customers in the areas of customer service, co-op management, and credit and collection. Two national account managers, Ricky Ponting and Greeme Smith, have joined the sales team as a result of these changes.

To achieve this goal, we have also organized the new mass sales group into three distinct teams reporting to our current sales directors, Stephen Fleming and Boris Baker. I have outlined below the national account managers and their respective accounts in each of the teams. We have also hired two new national account managers and a new sales administrator to complete our account coverage. They include:

Sachin Tendulkar, who joins us from XYZ Consumer Electronics as a national account manager covering traditional mass merchants.

Brian Lara, who comes to us via PQR Company and will be responsible for managing our West Coast territory.

Shane Warne, who will become an account administrator for our warehouse clubs business and joins us from DEF division.

Effectively, we have seven new faces on board:

1. RICKY PONTING
2. GREEME SMITH
3. STEPHEN FLEMING
4. BORIS BAKER
5. SACHIN TENDULKAR
6. BRIAN LARA
7. SHANE WARNE

Please join me in welcoming each of our new team members.

As a simple example, to find the lines that have the word “welcoming”, the best approach would be to use the following command line:
# grep welcoming memo Please join me in welcoming each of our new team members.

If you look for the word “market”, the results are slightly different, as shown below.
# grep market memo In order to better serve the needs of our mass market customers, ABC Publishing is integrating the groups selling to this channel for ABC General Reference and ABC Computer Publishing. This change will allow us to better coordinate our selling and marketing efforts, as well as simplify ABC's relationships with these customers in the areas of customer service, co-op management, and credit and collection. Two national account managers, Ricky Ponting and Greeme Smith, have joined the sales team as a result of these changes.

Note that two matches are found: the requested “market”, and “marketing”. If the words “marketable” or “marketed” had occurred in the file, the utility would have displayed the lines containing those words as well.

Wildcards and meta-characters can be used with grep, and I strongly recommend that you place them inside quotation marks so that the shell doesn’t interpret them as commands.

To find all lines that contain a number, use the following:

# grep "[0-9]" memo 1. RICKY PONTING 2. GREEME SMITH 3. STEPHEN FLEMING 4. BORIS BAKER 5. SACHIN TENDULKAR 6. BRIAN LARA 7. SHANE WARNE

To find all lines that contain “the”, use this:

# grep the memo In order to better serve the needs of our mass market customers, ABC Publishing is integrating the groups selling to this channel for ABC General Reference and ABC Computer Publishing. This change will allow us to better coordinate our selling and marketing efforts, as well as simplify ABC's relationships with these customers in the areas of customer service, co-op management, and credit and collection. Two national account managers, Ricky Ponting and Greeme Smith, have joined the sales team as a result of these changes.

To achieve this goal, we have also organized
the new mass sales group into three distinct
teams reporting to our current sales
directors, Stephen Flemming and Boris Baker. I
have outlined below the national account
managers and their respective accounts in each
of the teams. We have also hired two new
national account managers and a new sales
administrator to complete our account
coverage. They include:

As you might have noticed, the output contains the word “these”, along with exact matches of the word “the”.

The grep utility, like almost every other UNIX/Linux utility, is case-sensitive, which means that a completely different result comes from looking for “The” instead of “the”.

# grep The memo To achieve this goal, we have also organized the new mass sales group into three distinct teams reporting to our current sales directors, Stephen Flemming and Boris Baker. I have outlined below the national account managers and their respective accounts in each of the teams. We have also hired two new national account managers and a new sales administrator to complete our account coverage. They include:

If you are seeking a particular word or phrase and don’t care about the case, there are two ways to proceed. The first is to look for both “The” and “the” by using square brackets, as shown below:

# grep "[T, t]he" memo In order to better serve the needs of our mass market customers, ABC Publishing is integrating the groups selling to this channel for ABC General Reference and ABC Computer Publishing. This change will allow us to better coordinate our selling and marketing efforts, as well as simplify ABC's relationships with these customers in the areas of customer service, co-op management, and credit and collection. Two national account managers, Ricky Ponting and Greeme Smith, have joined the sales team as a result of these changes.

The second method is to use the -i option, which tells grep to ignore case sensitivity.

# grep -i the memo In order to better serve the needs of our mass market customers, ABC Publishing is integrating the groups selling to this channel for ABC General Reference and ABC Computer Publishing. This change will allow us to better coordinate our selling and marketing efforts, as well as simplify ABC's relationships with these customers in the areas of customer service, co-op management, and credit and collection. Two national account managers, Ricky Ponting and Greeme Smith, have joined the sales team as a result of these changes.

In addition to -i, there are several other command-line options to change grep’s output. The most relevant are the following:

* -c — Suppress normal output; instead, print a count of matching lines for each input file.
* -l — Suppress normal output; instead, print the name of each input file from which output would have normally been printed.
* -n — Prefix each line of output with the line number within its input file.
* -v — Invert the sense of matching — that is, select lines that don’t match the search criteria.

fgrep

fgrep searches files for a string and prints all lines that contain that string. Unlike grep, fgrep searches for a string instead of searching for a pattern that matches an expression. The fgrep utility can be thought of as grep with a few enhancements:

* You can search for more than one object at a time.
* The fgrep utility is always much faster than grep.
* You can’t use fgrep to search for regular expressions with patterns.

Suppose you want to pull uppercase names from your earlier memo file. In order to find “STEPHEN” and “BRIAN”, you would have to issue two separate grep commands, as shown below:

# grep STEPHEN memo 3. STEPHEN FLEMING

# grep BRIAN memo 6. BRIAN LARA

You can accomplish the same task with just one fgrep command:

# fgrep “STEPHEN
> BRIAN” memo
3. STEPHEN FLEMING
6. BRIAN LARA

Note that carriage return is required between entries. Without the carriage return, the search would look for “STEPHEN BRIAN” on each line. With the return, it looks for a match to “STEPHEN” and a match to “BRIAN”.

Note also that quotation marks must be used around the targeted text. This is what differentiates the text from the filename (or filenames).

Instead of specifying search items on the command line, you can place them in a file and use the contents of that file to search other files. The -f option allows you to specify a master file containing search items for which you search frequently.

For example, imagine a file named “search_items” that contains two search items for which you intend to search:

# cat search_items STEPHEN BRIAN

The following command searches for “STEPHEN” and “BRIAN” in our earlier memo file:

# fgrep -f search_items memo 3. STEPHEN FLEMING 6. BRIAN LARA

egrep

egrep is a more powerful version of grep that allows you to search for more than one object at a time. Objects being searched for are separated by carriage returns (as with fgrep) or by the pipe symbol (|).

# egrep "STEPHEN > BRIAN" memo 3. STEPHEN FLEMING 6. BRIAN LARA

# egrep "STEPHEN | BRIAN" memo 3. STEPHEN FLEMING 6. BRIAN LARA

The two commands above do the same job.

Besides the capacity to search for multiple objects, egrep offers the ability to search for repetitions and groups:

* ? looks for zero repetitions or one repetition of the character that precedes the question mark.
* + looks for one or more repetitions of the character that precedes the plus sign.
* ( ) signifies a group.

For example, imagine that you can’t remember whether Brian’s surname is “Lara” or “Laras”.

# egrep "LARAS?" memo 6. BRIAN LARA

This search produces matches to both “LARA” and “LARAS”. The following search is a bit different:

# egrep "STEPHEN+" memo 3. STEPHEN FLEMING

It matches “STEPHEN”, STEPHENN”, STEPHENNN”, and so on.

If you are looking for a word plus one of its possible derivatives, include the distinguishing characters of the derivative in parentheses.

# egrep -i "electron(ic)?s" memo Sachin Tendulkar, who joins us from XYZ Consumer Electronics as a national account manager covering traditional mass merchants.

This finds a match for both “electrons” and “electronics”.

To summarize:

* A regular expression followed by + matches one or more occurrences of the regular expression.

* A regular expression followed by ? matches zero or one occurrence of the regular expression.

* Regular expressions separated by | or by a carriage return match strings that are matched by any of the expressions.

* A regular expression can be enclosed in parentheses ( ) for grouping.

* The command-line parameters you can use include -c, -f, -i, -l, -n, and -v.

The grep utilities: A real-world example

The grep family of utilities can be used with any system file in text format to find a match in a line. For example, to find the entries in the /etc/passwd file for a user named “root”, use the following:

# grep root /etc/passwd root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin

Because it looks for a match anywhere in the file, grep finds entries for both “root” and “operator”. If you want to find only the entry with the username “root”, you can modify the command as follows:

# grep "^root" /etc/passwd root:x:0:0:root:/root:/bin/bash

cut

With the cut utility, you can separate columns that could constitute data fields in a file. The default delimiter is the tab, and the -f option is used to specify the desired field.

For example, imagine a text file named “sample” with three columns that look like this:

one two three four five six seven eight nine ten eleven twelve

Now, apply the following command:

# cut -f2 sample

This will return:

two five eight eleven

If you change your command like so:

# cut -f1, 3 sample

It will return the opposite:

one three four six seven nine ten twelve

Several command-line options are available with this command. Besides -f, you should be familiar with these two:

* -c — Allows you to specify characters instead of fields.
* -d — Allows you to specify a delimiter other than the tab.

cut: Two real-world examples

The ls -l command shows the permissions, number of links, owner, group, size, date, and filenames of all the files in a directory — all separated by white space. If you’re not interested in most of the fields and want to see only the file owner, you can use the following command:

# ls -l | cut -d" " -f5 root 562 root root root root root root

This command displays only the file owner (the fifth field), ignoring every other field.

If you know the exact position at which the first character of the file owner begins, you can use -c option to display the first character of the file owner. Assuming that it begins with the 16th character, the following command returns the 16th character, the first letter of the owner’s name.

# ls -l | cut -c16 r r r r r r r

If you further assume that most users will use eight characters or fewer for their name, you can use the following command:

# ls -l | cut -c16-24

It will return those entries in the name field.

Now, assume that the name of the file begins with the 55th character, but that it is impossible to determine how many characters it takes up after that because some filenames are considerably longer than others. A solution is to begin with the 55th character and not specifying an ending character (meaning that the entire rest of the line is taken) as shown below:

# ls -l | cut -c55- a.out cscope-15.5 cscope-15.5.tar cscope.out memo search_items test.c test.s

Now, consider another scenario. To obtain a list of all the users on the system, you can pull only the first field from the /etc/passwd file used in an earlier example:

# cut -d":" -f1 /etc/passwd root bin daemon adm lp sync shutdown halt mail news uucp operator

To collect the usernames and their corresponding home directories, you can pull the first and sixth fields:

# cut -d":" -f1,6 /etc/passwd root:/root bin:/bin daemon:/sbin adm:/var/adm lp:/var/spool/lpd sync:/sbin shutdown:/sbin halt:/sbin mail:/var/spool/mail news:/etc/news uucp:/var/spool/uucp operator:/root

paste

The paste utility combines fields from files. It takes one line from one source and combines it with another line from another source.

For example, imagine that the content of a file named “fileone” is:

IBM Global Services

In addition, you have “filetwo” with this content:

United States United Kingdom India

The following command combines the contents of these files, as shown below:

# paste fileone filetwo
IBM United States
Global United Kingdom
Services India

If there were more lines in fileone than filetwo, then the pasting would continue, with blank entries following the tab.

The tab character is the default delimiter, but you can change it to anything else with the -d option.

# paste -d", " fileone filetwo IBM, United States Global, United Kingdom Services, India

You can also use the -s option to output all of fileone on a line, followed by a carriage return and then filetwo.

# paste -s fileone filetwo IBM Global Services United States United Kingdom India

join

join is a greatly enhanced version of paste. join works only if the files being joined share a common field.

For example, consider the two files you were using with the paste command previously. Here’s what happens when you try to combine them with join:

# join fileone filetwo

Note that there is nothing to display. The join utility must find a common field between the files in question, and by default it expects that common field to be the first.

To see how this works, try adding some new content. Assume that fileone now contains these entries:

aaaa Jurassic Park bbbb AI cccc The Ring dddd The Mummy eeee Titanic

And filetwo now contains the following:

aaaa Neil 1111 bbbb Steven 2222 cccc Naomi 3333 dddd Brendan 4444 eeee Kate 5555

Now, try that command again:

# join fileone filetwo aaaa Jurassic Park Neil 1111 bbbb AI Steven 2222 cccc The Ring Naomi 3333 dddd The Mummy Brendan 4444 eeee Titanic Kate 5555

The commonality of the first field was identified, and the matching entries were combined. But paste blindly took from each file to create the output; join combines only lines that match, and the match must be exact. For example, imagine you added a line to filetwo:

aaaa Neil 1111 bbbb Steven 2222 ffff Elisha 6666 cccc Naomi 3333 dddd Brendan 4444 eeee Kate 5555

Now, your command will produce this output:

# join fileone filetwo aaaa Jurassic Park Neil 1111 bbbb AI Steven 2222

As soon as the files no longer match, no further operations can be carried out. Each line in the first file is matched to the same and only the same line in the second file for a match on the default field. If matches are found, they are incorporated into the output; otherwise they are not.

By default, join looks only at the first fields for matches and outputs all columns, but you can change this behavior. The -1 option lets you specify which field to use as the matching field in fileone, and the -2 option lets you specify which field to use as the matching field in filetwo.

For example, to match the second field of fileone to the third field of filetwo, use the following syntax:

# join -1 2 -2 3 fileone filetwo

The -o option specifies output in the format {file.field}. Thus, to print the second field of fileone and the third field of filetwo on matching lines, the syntax is:

# join -o 1.2 -o 2.3 fileone filetwo

join: A real-world example

The most obvious way you could use join in the real world would be to pull the username and the corresponding home directory from the /etc/passwd file and the group name from the /etc/group file. Groups appear in the fourth field in numerical format in the /etc/passwd file. Similarly, they appear in the third field in the /etc/group file.

# join -1 4 -2 3 -o 1.1 -o 2.1 -o 1.6 -t":" /etc/passwd /etc/group root:root:/root bin:bin:/bin daemon:daemon:/sbin adm:adm:/var/adm lp:lp:/var/spool/lpd nobody:nobody:/ vcsa:vcsa:/dev rpm:rpm:/var/lib/rpm nscd:nscd:/ ident:ident:/home/ident netdump:netdump:/var/crash sshd:sshd:/var/empty/sshd rpc:rpc:/

awk

awk is one of the most powerful utilities in Linux. It is actually a programming language in and of itself and can be used with complex logic statements, as well as to simply pull out snippets of text. We’ll skip the details, but let’s quickly review the syntax and then walk through some real-world examples.

An awk command consists of a pattern and an action composed of one or more statements, as shown in the syntax below:

awk '/pattern/ {action}' file

Notice that:

* awk tests every record in the specified file (or files) for a pattern match. If a match is found, the specified action is performed.
* awk can act as a filter in a pipeline or take input from the keyboard (standard input) if no file or files are specified.

One useful action is to print the data! Here is how to reference fields in a record.

* $0 — The entire record
* $1 — The first field in the record
* $2 — The second field in the record

You can also pull multiple fields in a record, separating each field by a comma.

For example, to pull the sixth field from the /etc/passwd file, the command is:
# awk -F: '{print $6}' /etc/passwd /root /bin /sbin /var/adm /var/spool/lpd /sbin /sbin /sbin /var/spool/mail /etc/news /var/spool/uucp

Note that -F is the input field separator defined by the predefined variable FS. It is a blank space, in my case.

To pull the first and sixth fields from the /etc/passwd file, the command is:

# awk -F: '{print $1,$6}' /etc/passwd root /root bin /bin daemon /sbin adm /var/adm lp /var/spool/lpd sync /sbin shutdown /sbin halt /sbin mail /var/spool/mail news /etc/news uucp /var/spool/uucp operator /root

To print the file using a dash in place of the colon delimiter between fields, the command is:

# awk -F: '{OFS="-"}{print $1,$6}' /etc/passwd root-/root bin-/bin daemon-/sbin adm-/var/adm lp-/var/spool/lpd sync-/sbin shutdown-/sbin halt-/sbin mail-/var/spool/mail news-/etc/news uucp-/var/spool/uucp operator-/root

To print the file using a dash between fields, and print only the first and sixth fields in reverse order, the command is:

# awk -F: '{OFS="-"}{print $6,$1}' /etc/passwd /root-root /bin-bin /sbin-daemon /var/adm-adm /var/spool/lpd-lp /sbin-sync /sbin-shutdown /sbin-halt /var/spool/mail-mail /etc/news-news /var/spool/uucp-uucp /root-operator

head

The head utility prints the first part of each file (10 lines by default). It reads from standard input if no files are given, or if given a filename of -.

For example, if you want to extract the first two lines from your memo file, the command is:

# head -2 memo In order to better serve the needs of our mass market customers, ABC Publishing is integrating the groups selling to this channel for ABC General Reference and ABC Computer Publishing. This change will allow us to better coordinate our selling and marketing efforts, as well as simplify ABC's relationships with these customers in the areas of customer service, co-op management, and credit and collection. Two national account managers, Ricky Ponting and Greeme Smith, have joined the sales team as a result of these changes.

You can specify the number of bytes to display using the -c option. For example, if you want to read the first two bytes from the memo file, the command is:

# head -c 2 memo
In

tail

The tail utility prints the last part of each file (10 lines by default). It reads from standard input if no files are given, or if given a filename of -.

For example, if you want to extract the last two lines from your earlier memo, the command is:

# tail -2 memo

Please join me in welcoming each of our new team members.

You can specify the number of bytes to display using the -c option. For example, if you want to read the last five bytes from the memo file, the command is:

# tail -c 5 memo
ers.

Conclusion

Now you know how to use various utilities to extract data from standard Linux files. Once extracted, that data can be manipulated for viewing and printing or directed into other files or databases. Knowing how to use just this handful of tools can help you spend less time on mundane tasks and become a more efficient administrator.

No tags

Jul/07

Securing Debian Manual

No comments · Posted by nguyen in Linux Docs

Securing services running on your system

Services can be secured in a running system in two ways:

Making them only accessible at the access points (interfaces) they need to be in.

Configuring them properly so that they can only be used by legitimate users in an authorized manner.

Restricting services so that they can only be accessed from a given place can be done by restricting access to them at the kernel (i.e. firewall) level, configure them to listen only on a given interface (some services might not provide this feature) or using some other methods, for example the Linux vserver patch (for 2.4.16) can be used to force processes to use only one interface.

Regarding the services running from inetd (telnet, ftp, finger, pop3…) it is worth noting that inetd can be configured so that services only listen on a given interface (using service@ip syntax) but that’s an undocumented feature. One of its substitutes, the xinetd meta-daemon includes a bind option just for this matter. See xinetd.conf(5).

service nntp
{
socket_type = stream
protocol = tcp
wait = no
user = news
group = news
server = /usr/bin/env
server_args = POSTING_OK=1 PATH=/usr/sbin/:/usr/bin:/sbin/:/bin
+/usr/sbin/snntpd logger -p news.info
bind = 127.0.0.1
}

The following sections detail how specific individual services can be configured properly depending on their intended use.
5.1 Securing ssh

If you are still running telnet instead of ssh, you should take a break from this manual and change this. Ssh should be used for all remote logins instead of telnet. In an age where it is easy to sniff Internet traffic and get clear-text passwords, you should use only protocols which use cryptography. So, perform an apt-get install ssh on your system now.

Encourage all the users on your system to use ssh instead of telnet, or even better, uninstall telnet/telnetd. In addition you should avoid logging into the system using ssh as root and use alternative methods to become root instead, like su or sudo. Finally, the sshd_config file, in /etc/ssh, should be modified to increase security as well:

ListenAddress 192.168.0.1

Have ssh listen only on a given interface, just in case you have more than one (and do not want ssh available on it) or in the future add a new network card (and don’t want ssh connections from it).

PermitRootLogin no

Try not to permit Root Login wherever possible. If anyone wants to become root via ssh, now two logins are needed and the root password cannot be brute forced via SSH.

Port 666 or ListenAddress 192.168.0.1:666

Change the listen port, so the intruder cannot be completely sure whether a sshd daemon runs (be forewarned, this is security by obscurity).

PermitEmptyPasswords no

Empty passwords make a mockery of system security.

AllowUsers alex ref me@somewhere

Allow only certain users to have access via ssh to this machine. user@host can also be used to restrict a given user from accessing only at a given host.

AllowGroups wheel admin

Allow only certain group members to have access via ssh to this machine. AllowGroups and AllowUsers have equivalent directives for denying access to a machine. Not surprisingly they are called “DenyUsers” and “DenyGroups”.

PasswordAuthentication yes

It is completely your choice what you want to do. It is more secure to only allow access to the machine from users with ssh-keys placed in the ~/.ssh/authorized_keys file. If you want so, set this one to “no”.

Disable any form of authentication you do not really need, if you do not use, for example RhostsRSAAuthentication, HostbasedAuthentication, KerberosAuthentication or RhostsAuthentication you should disable them, even if they are already by default (see the manpage sshd_config(5)).

Protocol 2

Disable the protocol version 1, since it has some design flaws that make it easier to crack passwords. For more information read a paper regarding ssh protocol problems or the Xforce advisory.

Banner /etc/some_file

Add a banner (it will be retrieved from the file) to users connecting to the ssh server. In some countries sending a warning before access to a given system about unauthorized access or user monitoring should be added to have legal protection.

You can also restrict access to the ssh server using pam_listfile or pam_wheel in the PAM control file. For example, you could keep anyone not listed in /etc/loginusers away by adding this line to /etc/pam.d/ssh:

auth required pam_listfile.so sense=allow onerr=fail item=user file=/etc/loginusers

As a final note, be aware that these directives are from a OpenSSH configuration file. Right now, there are three commonly used SSH daemons, ssh1, ssh2, and OpenSSH by the OpenBSD people. Ssh1 was the first ssh daemon available and it is still the most commonly used (there are rumors that there is even a Windows port). Ssh2 has many advantages over ssh1 except it is released under a closed-source license. OpenSSH is completely free ssh daemon, which supports both ssh1 and ssh2. OpenSSH is the version installed on Debian when the package ssh is chosen.

You can read more information on how to set up SSH with PAM support in the security mailing list archives.
5.1.1 Chrooting ssh

Currently OpenSSH does not provide a way to chroot automatically users upon connection (the commercial version does provide this functionality). However there is a project to provide this functionality for OpenSSH too, see http://chrootssh.sourceforge.net, it is not currently packaged for Debian, though. You could use, however, the pam_chroot module as described in Restricting users’s access, Section 4.10.8.

In Chroot environment for SSH, Appendix G you can find several options to make a chroot environment for SSH.
5.1.2 Ssh clients

If you are using an SSH client against the SSH server you must make sure that it supports the same protocols that are enforced on the server. For example, if you use the mindterm package, it only supports protocol version 1. However, the sshd server is, by default, configured to only accept version 2 (for security reasons).
5.1.3 Disallowing file transfers

If you do not want users to transfer files to and from the ssh server you need to restrict access to the sftp-server and the scp access. You can restrict sftp-server by configuring the proper Subsystem in the /etc/ssh/sshd_config.

You can also chroot users (using libpam-chroot so that, even if file transfer is allowed, they are limited to an environment which does not include any system files.
5.1.4 Restricing access to file transfer only

You might want to restrict access to users so that they can only do file transfers and cannot have interactive shells. In order to do this you can either:

disallow users from login to the ssh server (as described above either through the configuration file or PAM configuration).

give users a restricted shell such as scponly or rssh. These shells restrict the commands available to the users so that they are not provided any remote execution priviledges.

5.2 Securing Squid

Squid is one of the most popular proxy/cache server, and there are some security issues that should be taken into account. Squid’s default configuration file denies all users requests. However the Debian package allows access from ‘localhost’, you just need to configure your browser properly. You should configure Squid to allow access to trusted users, hosts or networks defining an Access Control List on /etc/squid/squid.conf, see the Squid User’s Guide for more information about defining ACLs rules. Notice that Debian provides a minimum configuration for Squid that will prevent anything, except from localhost to connect to your proxy server (which will run in the default port 3128). You will need to customize your /etc/squid/squid.conf as needed. The recommended minimum configuration (provided with the package) is shown below:

acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
(…)
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
# Only allow purge requests from localhost
http_access allow purge localhost
http_access deny purge
# Deny requests to unknown ports
http_access deny !Safe_ports
# Deny CONNECT to other than SSL ports
http_access deny CONNECT !SSL_ports
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
http_access allow localhost
# And finally deny all other access to this proxy
http_access deny all
#Default:
# icp_access deny all
#
#Allow ICP queries from everyone
icp_access allow all

You should also configure Squid based on your system resources, including cache memory (option cache_mem), location of the cached files and the amount of space they will take up on disk (option cache_dir).

Notice that, if not properly configured, someone may relay a mail message through Squid, since the HTTP and SMTP protocols are designed similarly. Squid’s default configuration file denies access to port 25. If you wish to allow connections to port 25 just add it to Safe_ports lists. However, this is NOT recommended.

Setting and configuring the proxy/cache server properly is only part of keeping your site secure. Another necessary task is to analyze Squid’s logs to assure that all things are working as they should be working. There are some packages in Debian GNU/Linux that can help an administrator to do this. The following packages are available in Debian 3.0 and Debian 3.1 (sarge):

calamaris – Log analyzer for Squid or Oops proxy log files.

modlogan – A modular logfile analyzer.

sarg – Squid Analysis Report Generator.

squidtaild – Squid log monitoring program.

When using Squid in Accelerator Mode it acts as a web server too. Turning on this option increases code complexity, making it less reliable. By default Squid is not configured to act as a web server, so you don’t need to worry about this. Note that if you want to use this feature be sure that it is really necessary. To find more information about Accelerator Mode on Squid see the Squid User’s Guide – Accelerator Mode
5.3 Securing FTP

If you really have to use FTP (without wrapping it with sslwrap or inside a SSL or SSH tunnel), you should chroot ftp into the ftp users’ home directory, so that the user is unable to see anything else than their own directory. Otherwise they could traverse your root file system just like if they had a shell in it. You can add the following line in your proftpd.conf in your global section to enable this chroot feature:

DefaultRoot ~

Restart ProFTPd by /etc/init.d/proftpd restart and check whether you can escape from your homedir now.

To prevent ProFTPd DoS attacks using ../../.., add the following line in /etc/proftpd.conf: DenyFilter \*.*/

Always remember that FTP sends login and authentication passwords in clear text (this is not an issue if you are providing an anonymous public service) and there are better alternatives in Debian for this. For example, sftp (provided by ssh). There are also free implementations of SSH for other operating systems: putty and cygwin for example.

However, if you still maintain the FTP server while making users access through SSH you might encounter a typical problem. Users accessing anonymous FTP servers inside SSH-secured systems might try to log in the FTP server. While the access will be refused, the password will nevertheless be sent through the net in clear form. To avoid that, ProFTPd developer TJ Saunders has created a patch that prevents users feeding the anonymous FTP server with valid SSH accounts. More information and patch available at: ProFTPD Patches. This patch has been reported to Debian too, see Bug #145669.
5.4 Securing access to the X Window System

Today, X terminals are used by more and more companies where one server is needed for a lot of workstations. This can be dangerous, because you need to allow the file server to connect to the clients (X server from the X point of view. X switches the definition of client and server). If you follow the (very bad) suggestion of many docs, you type xhost + on your machine. This allows any X client to connect to your system. For slightly better security, you can use the command xhost +hostname instead to only allow access from specific hosts.

A much more secure solution, though, is to use ssh to tunnel X and encrypt the whole session. This is done automatically when you ssh to another machine. For this to work, you have to configure both the ssh client and the ssh server. On the ssh client, ForwardX11 should be set to yes in /etc/ssh/ssh_config. On the ssh server, X11Forwarding should be set to yes in /etc/ssh/sshd_config and the package xbase-clients should be installed because the ssh server uses /usr/X11R6/bin/xauth (/usr/bin/xauth on Debian unstable) when setting up the pseudo X display. In times of SSH, you should drop the xhost based access control completely.

For best security, if you do not need X access from other machines, switch off the binding on TCP port 6000 simply by typing:

$ startx — -nolisten tcp

This is the default behavior in Xfree 4.1.0 (the Xserver provided in Debian 3.0 and 3.1). If you are running Xfree 3.3.6 (i.e. you have Debian 2.2 installed) you can edit /etc/X11/xinit/xserverrc to have it something along the lines of:

#!/bin/sh
exec /usr/bin/X11/X -dpi 100 -nolisten tcp

If you are using XDM set /etc/X11/xdm/Xservers to: :0 local /usr/bin/X11/X vt7 -dpi 100 -nolisten tcp. If you are using Gdm make sure that the DisallowTCP=true option is set in the /etc/gdm/gdm.conf (which is the default in Debian). This will basically append -nolisten tcp to every X command line [39].

You can also set the default’s system timeout for xscreensaver locks. Even if the user can override it, you should edit the /etc/X11/app-defaults/XScreenSaver configuration file and change the lock line:

*lock: False

(which is the default in Debian) to:

*lock: True

FIXME: Add information on how to disable the screensavers which show the user desktop (which might have sensitive information).

Read more on X Window security in XWindow-User-HOWTO (/usr/share/doc/HOWTO/en-txt/XWindow-User-HOWTO.txt.gz).

FIXME: Add info on thread of debian-security on how to change config files of XFree 3.3.6 to do this.
5.4.1 Check your display manager

If you only want to have a display manager installed for local usage (having a nice graphical login, that is), make sure the XDMCP (X Display Manager Control Protocol) stuff is disabled. In XDM you can do this with this line in /etc/X11/xdm/xdm-config:

DisplayManager.requestPort: 0

For GDM there should be in your gdm.conf:

[xdmcp]
Enable=false

Normally, all display managers are configured not to start XDMCP services per default in Debian.
5.5 Securing printing access (the lpd and lprng issue)

Imagine, you arrive at work, and the printer is spitting out endless amounts of paper because someone is DoSing your line printer daemon. Nasty, isn’t it?

In any UNIX printing architecture, there has to be a way to get the client’s data to the host’s print server. In traditional lpr and lp, the client command copies or symlinks the data into the spool directory (which is why these programs are usually SUID or SGID).

In order to avoid any issues you should keep your printer servers especially secure. This means you need to configure your printer service so it will only allow connections from a set of trusted servers. In order to do this, add the servers you want to allow printing to your /etc/hosts.lpd.

However, even if you do this, the lpr daemon accepts incoming connections on port 515 of any interface. You should consider firewalling connections from networks/hosts which are not allowed printing (the lpr daemon cannot be limited to listen only on a given IP address).

Lprng should be preferred over lpr since it can be configured to do IP access control. And you can specify which interface to bind to (although somewhat weirdly).

If you are using a printer in your system, but only locally, you will not want to share this service over a network. You can consider using other printing systems, like the one provided by cups or PDQ which is based on user permissions of the /dev/lp0 device.

In cups, the print data is transferred to the server via the HTTP protocol. This means the client program doesn’t need any special privileges, but does require that the server is listening on a port somewhere.

However, if you want to use cups, but only locally, you can configure it to bind to the loopback interface by changing /etc/cups/cupsd.conf:

Listen 127.0.0.1:631

There are many other security options like allowing or denying networks and hosts in this config file. However, if you do not need them you might be better off just limiting the listening port. Cups also serves documentation through the HTTP port, if you do not want to disclose potential useful information to outside attackers (and the port is open) add also:

Order Deny,Allow
Deny From All
Allow From 127.0.0.1

This configuration file can be modified to add some more features including SSL/TLS certificates and crypto. The manuals are available at http://localhost:631/ or at cups.org.

FIXME: Add more content (the article on Amateur Fortress Building provides some very interesting views).

FIXME: Check if PDG is available in Debian, and if so, suggest this as the preferred printing system.

FIXME: Check if Farmer/Wietse has a replacement for printer daemon and if it’s available in Debian.
5.6 Securing the mail service

If your server is not a mailing system, you do not really need to have a mail daemon listening for incoming connections, but you might want local mail delivered in order, for example, to receive mail for the root user from any alert systems you have in place.

If you have exim you do not need the daemon to be working in order to do this since the standard cron job flushes the mail queue. See Disabling daemon services, Section 3.6.1 on how to do this.
5.6.1 Configuring a Nullmailer

You might want to have a local mailer daemon so that it can relay the mails sent locally to another system. This is common when you have to administer a number of systems and do not want to connect to each of them to read the mail sent locally. Just as all logging of each individual system can be centralized by using a central syslog server, mail can be sent to a central mailserver.

Such a relay-only system should be configured properly for this. The daemon could, as well, be configured to only listen on the loopback address.

The following configuration steps only need to be taken to configure the exim package in the Debian 3.0 release. If you are using a later release (such as 3.1 which uses exim4) the installation system has been improved so that if the mail transport agent is configured to only deliver local mail it will automatically only allow connections from the local host and will not permit remote connections.

In a Debian 3.0 system using exim, you will have to remove the SMTP daemon from inetd:

$ update-inetd –disable smtp

and configure the mailer daemon to only listen on the loopback interface. In exim (the default MTA) you can do this by editing the file /etc/exim.conf and adding the following line:

local_interfaces = “127.0.0.1″

Restart both daemons (inetd and exim) and you will have exim listening on the 127.0.0.1:25 socket only. Be careful, and first disable inetd, otherwise, exim will not start since the inetd daemon is already handling incoming connections.

For postfix edit /etc/postfix/main.conf:

inet_interfaces = localhost

If you only want local mail, this approach is better than tcp-wrapping the mailer daemon or adding firewalling rules to limit anybody accessing it. However, if you do need it to listen on other interfaces, you might consider launching it from inetd and adding a tcp wrapper so incoming connections are checked against /etc/hosts.allow and /etc/hosts.deny. Also, you will be aware of when an unauthorized access is attempted against your mailer daemon, if you set up proper logging for any of the methods above.

In any case, to reject mail relay attempts at the SMTP level, you can change /etc/exim/exim.conf to include:

receiver_verify = true

Even if your mail server will not relay the message, this kind of configuration is needed for the relay tester at http://www.abuse.net/relay.html to determine that your server is not relay capable.

If you want a relay-only setup, however, you can consider changing the mailer daemon to programs that can only be configured to forward the mail to a remote mail server. Debian provides currently both ssmtp and nullmailer for this purpose. In any case, you can evaluate for yourself any of the mail transport agents [40] provided by Debian and see which one suits best to the system’s purposes.
5.6.2 Providing secure access to mailboxes

If you want to give remote access to mailboxes there are a number of POP3 and IMAP daemons available.[41] However, if you provide IMAP access note that it is a general file access protocol, it can become the equivalent of a shell access because users might be able to retrieve any file that they can through it.

Try, for example, to configure as your inbox path {server.com}/etc/passwd if it succeeds your IMAP daemon is not properly configured to prevent this kind of access.

Of the IMAP servers in Debian the cyrus server (in the cyrus-imapd package) gets around this by having all access to a database in a restricted part of the file system. Also, uw-imapd (either install the uw-imapd or better, if your IMAP clients support it, uw-imapd-ssl) can be configured to chroot the users mail directory but this is not enabled by default. The documentation provided gives more information on how to configure it.

Also, you might want to run an IMAP server that does not need valid users to be created on the local system (which would grant shell access too), courier-imap (for IMAP) and courier-pop, teapop (for POP3) and cyrus-imapd (for both POP3 and IMAP) provide servers with authentication methods beside the local user accounts. cyrus can use any authentication method that can be configured through PAM while teapop might use databases (such as postgresql and mysql) for user authentication.

FIXME: Check: uw-imapd might be configured with user authentication through PAM too.
5.6.3 Receiving mail securely

Reading/receiving mail is the most common clear-text protocol. If you use either POP3 or IMAP to get your mail, you send your clear-text password across the net, so almost anyone can read your mail from now on. Instead, use SSL (Secure Sockets Layer) to receive your mail. The other alternative is SSH, if you have a shell account on the box which acts as your POP or IMAP server. Here is a basic fetchmailrc to demonstrate this:

poll my-imap-mailserver.org via “localhost”
with proto IMAP port 1236
user “ref” there with password “hackme” is alex here warnings 3600
folders
.Mail/debian
preconnect ’ssh -f -P -C -L 1236:my-imap-mailserver.org:143 -l ref
my-imap-mailserver.org sleep 15 /dev/null’

The preconnect is the important line. It fires up an ssh session and creates the necessary tunnel, which automatically forwards connections to localhost port 1236 to the IMAP mail server, but encrypted. Another possibility would be to use fetchmail with the SSL feature.

If you want to provide encrypted mail services like POP and IMAP, apt-get install stunnel and start your daemons this way:

stunnel -p /etc/ssl/certs/stunnel.pem -d pop3s -l /usr/sbin/popd

This command wraps the provided daemon (-l) to the port (-d) and uses the specified SSL certificate (-p).
5.7 Securing BIND

There are different issues that can be tackled in order to secure the Domain server daemon, which are similar to the ones considered when securing any given service:

configuring the daemon itself properly so it cannot be misused from the outside (see Bind configuration to avoid misuse, Section 5.7.1). This includes limiting possible queries from clients: zone transfers and recursive queries.

limit the access of the daemon to the server itself so if it is used to break in, the damage to the system is limited. This includes running the daemon as a non-privileged user (see Changing BIND’s user, Section 5.7.2) and chrooting it (see Chrooting the name server, Section 5.7.3).

5.7.1 Bind configuration to avoid misuse

You should restrict some of the information that is served from the DNS server to outside clients so that it cannot be used to retrieve valuable information from your organization that you do not want to give away. This includes adding the following options: allow-transfer, allow-query, allow-recursion and version. You can either limit this on the global section (so it applies to all the zones served) or on a per-zone basis. This information is documented in the bind-doc package, read more on this on /usr/share/doc/bind/html/index.html once the package is installed.

Imagine that your server is connected to the Internet and to your internal (your internal IP is 192.168.1.2) network (a basic multi-homed server), you do not want to give any service to the Internet and you just want to enable DNS lookups from your internal hosts. You could restrict it by including in /etc/bind/named.conf:

options {
allow-query { 192.168.1/24; } ;
allow-transfer { none; } ;
allow-recursion { 192.168.1/24; } ;
listen-on { 192.168.1.2; } ;
forward { only; } ;
forwarders { A.B.C.D; } ;
};

The listen-on option makes the DNS bind to only the interface that has the internal address, but, even if this interface is the same as the interface that connects to the Internet (if you are using NAT, for example), queries will only be accepted if coming from your internal hosts. If the system has multiple interfaces and the listen-on is not present, only internal users could query, but, since the port would be accessible to outside attackers, they could try to crash (or exploit buffer overflow attacks) on the DNS server. You could even make it listen only on 127.0.0.1 if you are not giving DNS service for any other systems than yourself.

The version.bind record in the chaos class contains the version of the currently running bind process. This information is often used by automated scanners and malicious individuals who wish to determine if one’s bind is vulnerable to a specific attack. By providing false or no information in the version.bind record, one limits the probability that one’s server will be attacked based on its published version. To provide your own version, use the version directive in the following manner:

options { … various options here …
version “Not available.”; };

Changing the version.bind record does not provide actual protection against attacks, but it might be considered a useful safeguard.

A sample named.conf configuration file might be the following:

acl internal {
127.0.0.1/32; // localhost
10.0.0.0/8; // internal
aa.bb.cc.dd; // eth0 IP
};

acl friendly {
ee.ff.gg.hh; // slave DNS
aa.bb.cc.dd; // eth0 IP
127.0.0.1/32; // localhost
10.0.0.0/8; // internal
};

options {
directory “/var/cache/bind”;
allow-query { internal; };
allow-recursion { internal; };
allow-transfer { none; };
};
// From here to the mysite.bogus zone
// is basically unmodified from the debian default
logging {
category lame-servers { null; };
category cname { null; };
};

zone “.” {
type hint;
file “/etc/bind/db.root”;
};

zone “localhost” {
type master;
file “/etc/bind/db.local”;
};

zone “127.in-addr.arpa” {
type master;
file “/etc/bind/db.127″;
};

zone “0.in-addr.arpa” {
type master;
file “/etc/bind/db.0″;
};

zone “255.in-addr.arpa” {
type master;
file “/etc/bind/db.255″;
};

// zones I added myself
zone “mysite.bogus” {
type master;
file “/etc/bind/named.mysite”;
allow-query { any; };
allow-transfer { friendly; };
};

Please (again) check the Bug Tracking System regarding Bind, specifically Bug #94760 (regarding ACLs on zone transfers). Feel free to contribute to the bug report if you think you can add useful information.
5.7.2 Changing BIND’s user

Regarding limiting BIND’s privileges you must be aware that if a non-root user runs BIND, then BIND cannot detect new interfaces automatically, for example when you put a PCMCIA card into your laptop. Check the README.Debian file in your named documentation (/usr/share/doc/bind/README.Debian) directory for more information about this issue. There have been many recent security problems concerning BIND, so switching the user is useful when possible. We will detail here the steps needed in order to do this, however, if you want to do this in an automatic way you might try the script provided in Sample script to change the default Bind installation., Appendix E.

Notice, in any case, that this only applies to BIND version 8. In the Debian packages for BIND version 9 (since the 9.2.1-5 version, available since sarge) the bind user is created and used by setting the OPTIONS variable in /etc/default/bind9. If you are using BIND version 9 and your name server daemon is not running as the bind user verify the settings on that file.

To run BIND under a different user, first create a separate user and group for it (it is not a good idea to use nobody or nogroup for every service not running as root). In this example, the user and group named will be used. You can do this by entering:

addgroup named
adduser –system –home /home/named –no-create-home –ingroup named \
–disabled-password –disabled-login named

Notice that the user named will be quite restricted. If you want, for whatever reason, to have a less restrictive setup use:

adduser –system –ingroup named named

Now you can either edit /etc/init.d/bind with your favorite editor and change the line beginning with

start-stop-daemon –start

to[42]

start-stop-daemon –start –quiet –exec /usr/sbin/named — -g named -u named

Or you can change (create it if it does not exit) the default configuration file (/etc/default/bind for BIND version and introduce the following:

OPTIONS=”-u named -g named”

Change the permissions of files that are used by Bind, including /etc/bind/rndc.key:

-rw-r—– 1 root named 77 Jan 4 01:02 rndc.key

and where bind creates its pidfile, using, for example, /var/run/named instead of /var/run:

$ mkdir /var/run/named
$ chown named.named /var/run/named
$ vi /etc/named.conf
[ ... update the configuration file to use this new location ...]
options { …
pid-file “/var/run/named/named.pid”;
};
[ ... ]

Also, in order to avoid running anything as root, change the reload line in the init.d script by substituting:

reload)
/usr/sbin/ndc reload

to:

reload)
$0 stop
sleep 1
$0 start

Note: Depending on your Debian version you might have to change the restart line too. This was fixed in Debian’s bind version 1:8.3.1-2.

All you need to do now is to restart bind via /etc/init.d/bind restart, and then check your syslog for two entries like this:

Sep 4 15:11:08 nexus named[13439]: group = named
Sep 4 15:11:08 nexus named[13439]: user = named

Voilà! Your named now does not run as root. If you want to read more information on why BIND does not run as non-root user on Debian systems, please check the Bug Tracking System regarding Bind, specifically Bug #50013: bind should not run as root and Bug #132582: Default install is potentially insecure, Bug #53550, Bug #52745, and Bug #128129. Feel free to contribute to the bug reports if you think you can add useful information.
5.7.3 Chrooting the name server

To achieve maximum BIND security, now build a chroot jail (see General chroot and suid paranoia, Section 5.10) around your daemon. There is an easy way to do this: the -t option (see the named(8) manpage or page 100 of Bind’s 9 documentation (PDF)). This will make Bind chroot itself into the given directory without you needing to set up a chroot jail and worry about dynamic libraries. The only files that need to be in the chroot jail are:

dev/null
etc/bind/ – should hold named.conf and all the server zones
sbin/named-xfer – if you do name transfers
var/run/named/ – should hold the PID and the name server cache (if
any) this directory needs to be writable by named user
var/log/named – if you set up logging to a file, needs to be writable
for the named user
dev/log – syslogd should be listening here if named is configured to
log through it

In order for your Bind daemon to work properly it needs permission in the named files. This is an easy task since the configuration files are always at /etc/named/. Take into account that it only needs read-only access to the zone files, unless it is a secondary or cache name server. If this is your case you will have to give read-write permissions to the necessary zones (so that zone transfers from the primary server work).

Also, you can find more information regarding Bind chrooting in the Chroot-BIND-HOWTO (regarding Bind 9) and Chroot-BIND8-HOWTO (regarding Bind 8). This same documents should be available through the installation of the doc-linux-text (text version) or doc-linux-html (HTML version). Another useful document is http://web.archive.org/web/20011024064030/http://www.psionic.com/papers/dns/dns-linux.

If you are setting up a full chroot jail (i.e. not just -t) for Bind in Debian, make sure you have the following files in it[43]:

dev/log – syslogd should be listening here
dev/null
etc/bind/named.conf
etc/localtime
etc/group – with only a single line: “named:x:GID:”
etc/ld.so.cache – generated with ldconfig
lib/ld-2.3.6.so
lib/libc-2.3.6.so
lib/ld-linux.so.2 – symlinked to ld-2.3.6.so
lib/libc.so.6 – symlinked to libc-2.3.6.so
sbin/ldconfig – may be deleted after setting up the chroot
sbin/named-xfer – if you do name transfers
var/run/

And modify also syslogd listen on $CHROOT/dev/log so the named server can write syslog entries into the local system log.

If you want to avoid problems with dynamic libraries, you can compile bind statically. You can use apt-get for this, with the source option. It can even download the packages you need to properly compile it. You would need to do something similar to:

$ apt-get source bind
# apt-get build-dep bind
$ cd bind-8.2.5-2
(edit src/port/linux/Makefile so CFLAGS includes the ‘-static’
option)
$ dpkg-buildpackage -rfakeroot -uc -us
$ cd ..
# dpkg -i bind-8.2.5-2*deb

After installation, you will need to move around the files to the chroot jail[44] you can keep the init.d scripts in /etc/init.d so that the system will automatically start the name server, but edit them to add –chroot /location_of_chroot in the calls to start-stop-daemon in those scripts or use the -t option for BIND by setting it in the OPTIONS argument at the /etc/default/bind (for version or /etc/default/bind9 (for version 9) configuration file.

For more information on how to set up chroots see General chroot and suid paranoia, Section 5.10.

FIXME: Merge info from http://people.debian.org/~pzn/howto/chroot-bind.sh.txt, http://www.cryptio.net/~ferlatte/config/ (Debian-specific), http://web.archive.org/web/20021216104548/http://www.psionic.com/papers/whitep01.html and http://csrc.nist.gov/fasp/FASPDocs/NISTSecuringDNS.htm.
5.8 Securing Apache

FIXME: Add content: modules provided with the normal Apache installation (under /usr/lib/apache/X.X/mod_*) and modules that can be installed separately in libapache-mod-XXX packages.

You can limit access to the Apache server if you only want to use it internally (for testing purposes, to access the doc-central archive, etc.) and do not want outsiders to access it. To do this use the Listen or BindAddress directives in /etc/apache/http.conf.

Using Listen:

Listen 127.0.0.1:80

Using BindAddress:

BindAddress 127.0.0.1

Then restart apache with /etc/init.d/apache restart and you will see that it is only listening on the loopback interface.

In any case, if you are not using all the functionality provided by Apache, you might want to take a look at other web servers provided in Debian like dhttpd.

The Apache Documentation provides information regarding security measures to be taken on Apache web server (this same information is provided in Debian by the apache-doc package).

More information on further restricting Apache by setting up a chroot jail is provided in Chroot environment for Apache, Appendix H.
5.8.1 Disabling users from publishing web contents

The default Apache installation in Debian permits users to publish content under the $HOME/public_html. This content can be retrieved remotely using an URL such as: http://your_apache_server/~user.

If you do not want to permit this you must change the /etc/apache/http.conf configuration file commenting out (in Apache 1.3) the following module:

LoadModule userdir_module /usr/lib/apache/1.3/mod_userdir.so

If you are using Apache 2.0 you must remove the file /etc/apache2/mods-enabled/userdir.load or restrict the default configuration by modifying /etc/apache2/mods-enabled/userdir.conf.

However, if the module was linked statically (you can list the modules that are compiled in running apache -l) you must add the following to the Apache configuration file:

Userdir disabled

An attacker might still do user enumeration, since the answer of the web server will be a 403 Permission Denied and not a 404 Not available. You can avoid this if you use the Rewrite module.
5.8.2 Logfiles permissions

Apache logfiles, since 1.3.22-1, are owned by user ‘root’ and group ‘adm’ with permissions 640. These permissions are changed after rotation. An intruder that accessed the system through the web server would not be able (without privilege escalation) to remove old log file entries.
5.8.3 Published web files

Apache files are located under /var/www. Just after installation the default file provides some information on the system (mainly that it’s a Debian system running Apache). The default webpages are owned by user root and group root by default, while the Apache process runs as user www-data and group www-data. This should make attackers that compromise the system through the web server harder to deface the site. You should, of course, substitute the default web pages (which might provide information you do not want to show to outsiders) with your own.
5.9 Securing finger

If you want to run the finger service first ask yourself if you need to do so. If you do, you will find out that Debian provides many finger daemons (output from apt-cache search fingerd):

cfingerd – Configurable finger daemon

efingerd – Another finger daemon for unix, capable of fine-tuning your output.

ffingerd – a secure finger daemon

fingerd – Remote user information server.

xfingerd – BSD-like finger daemon with qmail support.

ffingerd is the recommended finger daemon if you are going to use it for a public service. In any case, you are encouraged to, when setting it up through inetd, xinetd or tcpserver to: limit the number of processes that will be running at the same time, limit access to the finger daemon from a given number of hosts (using tcp wrappers) and having it only listening to the interface you need it to be in.
5.10 General chroot and suid paranoia

chroot is one of the most powerful possibilities to restrict a daemon or a user or another service. Just imagine a jail around your target, which the target cannot escape from (normally, but there are still a lot of conditions that allow one to escape out of such a jail). If you do not trust a user or a service, you can create a modified root environment for him. This can use quite a bit of disk space as you need to copy all needed executables, as well as libraries, into the jail. But then, even if the user does something malicious, the scope of the damage is limited to the jail.

Many services running as daemons could benefit from this sort of arrangement. The daemons that you install with your Debian distribution will not come, however, chrooted[45] per default.

This includes: name servers (such as bind), web servers (such as apache), mail servers (such as sendmail) and ftp servers (such as wu-ftpd). It is probably fair to say that the complexity of BIND is the reason why it has been exposed to a lot of attacks in recent years (see Securing BIND, Section 5.7).

However, Debian does provide some software that can help set up chroot environments. See Making chrooted environments automatically, Section 5.10.1.

Anyway, if you run any service on your system, you should consider running them as secure as possible. This includes: revoking root privileges, running in a restricted environment (such as a chroot jail) or replacing them with a more secure equivalent.

However, be forewarned that a chroot jail can be broken if the user running in it is the superuser. So, you need to make the service run as a non-privileged user. By limiting its environment you are limiting the world readable/executable files the service can access, thus, you limit the possibilities of a privilege escalation by use of local system security vulnerabilities. Even in this situation you cannot be completely sure that there is no way for a clever attacker to somehow break out of the jail. Using only server programs which have a reputation for being secure is a good additional safety measure. Even minuscule holes like open file handles can be used by a skilled attacker for breaking into the system. After all, chroot was not designed as a security tool but as a testing tool.
5.10.1 Making chrooted environments automatically

There are several programs to chroot automatically servers and services. Debian currently (accepted in May 2002) provides Wietse Venema’s chrootuid in the chrootuid package, as well as compartment and makejail. These programs can be used to set up a restricted environment for executing any program (chrootuid enables you to even run it as a restricted user).

Some of these tools can be used to set up the chroot environment easily. The makejail program for example, can create and update a chroot jail with short configuration files (it provides sample configuration files for bind, apache, postgresql and mysql). It attempts to guess and install into the jail all files required by the daemon using strace, stat and Debian’s package dependencies. More information at http://www.floc.net/makejail/. Jailer is a similar tool which can be retrieved from http://www.balabit.hu/downloads/jailer/ and is also available as a Debian package.
5.11 General cleartext password paranoia

You should try to avoid any network service which sends and receives passwords in cleartext over a net like FTP/Telnet/NIS/RPC. The author recommends the use of ssh instead of telnet and ftp to everybody.

Keep in mind that migrating from telnet to ssh, but using other cleartext protocols does not increase your security in ANY way! Best would be to remove ftp, telnet, pop, imap, http and to supersede them with their respective encrypted services. You should consider moving from these services to their SSL versions, ftp-ssl, telnet-ssl, pop-ssl, https …

Most of these above listed hints apply to every Unix system (you will find them if reading any other hardening-related document related to Linux and other Unices).
5.12 Disabling NIS

You should not use NIS, the Network Information Service, if possible, because it allows password sharing. This can be highly insecure if your setup is broken.

If you need password sharing between machines, you might want to consider using other alternatives. For example, you can setup an LDAP server and configure PAM on your system in order to contact the LDAP server for user authentication. You can find a detailed setup in the LDAP-HOWTO (/usr/share/doc/HOWTO/en-txt/LDAP-HOWTO.txt.gz).

You can read more about NIS security in the NIS-HOWTO (/usr/share/doc/HOWTO/en-txt/NIS-HOWTO.txt.gz).

FIXME (jfs): Add info on how to set this up in Debian.
5.13 Securing RPC services

You should disable RPC if you do not need it.

Remote Procedure Call (RPC) is a protocol that programs can use to request services from other programs located on different computers. The portmap service controls RPC services by mapping RPC program numbers into DARPA protocol port numbers; it must be running in order to make RPC calls.

RPC-based services have had a bad record of security holes, although the portmapper itself hasn’t (but still provides information to a remote attacker). Notice that some of the DDoS (distributed denial of service) attacks use RPC exploits to get into the system and act as a so called agent/handler.

You only need RPC if you are using an RPC-based service. The most common RPC-based services are NFS (Network File System) and NIS (Network Information System). See the previous section for more information about NIS. The File Alteration Monitor (FAM) provided by the package fam is also an RPC service, and thus depends on portmap.

NFS services are quite important in some networks. If that is the case for you, then you will need to find a balance of security and usability for your network (you can read more about NFS security in the NFS-HOWTO (/usr/share/doc/HOWTO/en-txt/NFS-HOWTO.txt.gz)).
5.13.1 Disabling RPC services completely

Disabling portmap is quite simple. There are several different methods. The simplest one in a Debian 3.0 system and later releases is to uninstall the portmap package. If you are running an older Debian version you will have to disable the service as seen in Disabling daemon services, Section 3.6.1, because the program is part of the netbase package (which cannot be de-installed without breaking the system).

Notice that some desktop environments (notably, GNOME) use RPC services and need the portmapper for some of the file management features. If this is your case, you can limit the access to RPC services as described below.
5.13.2 Limiting access to RPC services

Unfortunately, in some cases removing RPC services from the system is not an option. Some local desktop services (notably SGI’s fam) are RPC based and thus need a local portmapper. This means that under some situations, users installing a desktop environment (like GNOME) will install the portmapper too.

There are several ways to limit access to the portmapper and to RPC services:

Block access to the ports used by these services with a local firewall (see Adding firewall capabilities, Section 5.14).

Block access to these services using tcp wrappers, since the portmapper (and some RPC services) are compiled with libwrap (see Using tcpwrappers, Section 4.11). This means that you can block access to them through the hosts.allow and hosts.deny tcp wrappers configuration.

Since version 5-5, the portmap package can be configured to listen only on the loopback interface. To do this, modify /etc/default/portmap, uncomment the following line: #OPTIONS=”-i 127.0.0.1″ and restart the portmapper. This is sufficient to allow local RPC services to work while at the same time prevents remote systems from accessing them (see, however, Disabling weak-end hosts issues, Section 4.17.5).

5.14 Adding firewall capabilities

The Debian GNU/Linux operating system has the built-in capabilities provided by the Linux kernel . If you install a recent Debian release (default kernel installed is 2.6) you will have iptables (netfilter) firewalling available[46].
5.14.1 Firewalling the local system

You can use firewall rules as a way to secure the access to your local system and, even, to limit the outbound communications made by it. Firewall rules can also be used to protect processes that cannot be properly configured not to provide services to some networks, IP addresses, etc.

However, this step is presented last in this manual basically because it is much better not to depend solely on firewalling capabilities in order to protect a given system. Security in a system is made up of layers, firewalling should be the last to include, once all services have been hardened. You can easily imagine a setup in which the system is solely protected by a built-in firewall and an administrator blissfully removes the firewall rules for whatever reason (problems with the setup, annoyance, human error…), this system would be wide open to an attack if there were no other hardening in the system to protect from it.

On the other hand, having firewall rules on the local system also prevents some bad things from happening. Even if the services provided are configured securely, a firewall can protect from misconfigurations or from fresh installed services that have not yet been properly configured. Also, a tight configuration will prevent trojans calling home from working unless the firewalling code is removed. Note that an intruder does not need superuser access to install a trojan locally that could be remotely controlled (since binding on ports is allowed if they are not priviledged ports and capabilities have not been removed).

Thus, a proper firewall setup would be one with a default deny policy, that is:

incoming connections are allowed only to local services by allowed machines.

outgoing connections are only allowed to services used by your system (DNS, web browsing, POP, email…).[47]

the forward rule denies everything (unless you are protecting other systems, see below).

all other incoming or outgoing connections are denied.

5.14.2 Using a firewall to protect other systems

A Debian firewall can also be installed in order to protect, with filtering rules, access to systems behind it, limiting their exposure to the Internet. A firewall can be configured to prevent access from systems outside of the local network to internal services (ports) that are not public. For example, on a mail server, only port 25 (where the mail service is being given) needs to be accessible from the outside. A firewall can be configured to, even if there are other network services besides the public ones running in the mail server, throw away packets (this is known as filtering) directed towards them.

You can even set up a Debian GNU/Linux box as a bridge firewall, i.e. a filtering firewall completely transparent to the network that lacks an IP address and thus cannot be attacked directly. Depending on the kernel you have installed, you might need to install the bridge firewall patch and then go to 802.1d Ethernet Bridging when configuring the kernel and a new option netfilter ( firewalling ) support. See the Setting up a bridge firewall, Appendix D for more information on how to set this up in a Debian GNU/Linux system.
5.14.3 Setting up a firewall

The default Debian installation, unlike other Linux distributions, does not yet provide a way for the administrator to setup a firewall configuration throughout the default installation but you can install a number of firewall configuration packages (see Using firewall packages, Section 5.14.3.1).

Of course, the configuration of the firewall is always system and network dependant. An administrator must know beforehand what is the network layout and the systems he wants to protect, the services that need to be accessed, and whether or not other network considerations (like NAT or routing) need to be taken into account. Be careful when configuring your firewall, as Laurence J. Lane says in the iptables package:

The tools can easily be misused, causing enormous amounts of grief by completely crippling network access to a system. It is not terribly uncommon for a remote system administrator to accidentally lock himself out of a system hundreds or thousands of miles away. One can even manage to lock himself out of a computer who’s keyboard is under his fingers. Please, use due caution.

Remember this: just installing the iptables (or the older firewalling code) does not give you any protection, just provides the software. In order to have a firewall you need to configure it!

If you do not have a clue on how to set up your firewall rules manually consult the Packet Filtering HOWTO and NAT HOWTO provided by iptables for offline reading at /usr/share/doc/iptables/html/.

If you do not know much about firewalling you should start by reading the Firewalling and Proxy Server HOWTO, install the doc-linux-text package if you want to read it offline. If you want to ask questions or need help setting up a firewall you can use the debian-firewall mailing list, see http://lists.debian.org/debian-firewall. Also see Be aware of general security problems, Section 2.2 for more (general) pointers on firewalls. Another good iptables tutorial is http://iptables-tutorial.frozentux.net/iptables-tutorial.html.
5.14.3.1 Using firewall packages

Setting up manually a firewall can be complicated for novice (and sometimes even expert) administrators. However, the free software community has created a number of tools that can be used to easily configure a local firewall. Be forewarned that some of these tools are oriented more towards local-only protection (also known as personal firewall) and some are more versatile and can be used to configure complex rules to protect whole networks.

Some software that can be used to set up firewall rules in a Debian system is:

firestarter, a GNOME application oriented towards end-users that includes a wizard useful to quickly setup firewall rules. The application includes a GUI to be able to monitor when a firewall rule blocks traffic.

fwbuilder, an object oriented GUI which includes policy compilers for various firewall platforms including Linux’ netfilter, BSD’s pf (used in OpenBSD, NetBSD, FreeBSD and MacOS X) as well as router’s access-lists. It is similar to enterprise firewall management software. Complete fwbuilder’s functionality is also available from the command line.

shorewall, a firewall configuration tool which provides support for IPsec as well as limited support for traffic shaping as well as the definition of the firewall rules. Configuration is done through a simple set of files that are used to generate the iptables rules.

guarddog, a KDE based firewall configuration package oriented both to novice and advanced users.

knetfilter, a KDE GUI to manage firewall and NAT rules for iptables (alternative/competitor to the guarddog tool although slightly oriented towards advanced users).

bastille, this hardening application is described in Automatic hardening of Debian systems, Chapter 6. One of the hardening steps that the administrator can configure is a definition of the allowed and disallowed network traffic that is used to generate a set of firewall rules that the system will execute on startup.

Lots of other iptables frontends come with Debian; an extensive list comparing the different packages in Debian is maintained at the Firewalls page on the Debian wiki.

Notice that some of the packages outlined previously will introduce firewalling scripts to be run when the system boots. Test them extensively before rebooting or you might find yourself locked from the box. If you mix different firewalling packages you can have undesired effects, usually, the firewalling script that runs last will be the one that configures the system (which might not be what you intend). Consult the package documentation and use either one of these setups.

As mentioned before, some programs, like firestarter, guarddog and knetfilter, are administration GUIs using either GNOME or KDE (last two). These applications are much more user-oriented (i.e. for home users) than some of the other packages in the list which might be more administrator-oriented. Some of the programs mentioned before (like bastille) are focused at setting up firewall rules to protect the host they run in but are not necessarily designed to setup firewall rules for firewall hosts that protect a network (like shorewall or fwbuilder).

There is yet another type of firewall application: application proxies. If you are looking into setting up an enterprise-level firewall that does packet filtering and provides a number of transparent proxies that can do fine-grain traffic analysis you should consider using zorp, which provides this in a single program. You can also manually setup this type of firewall host using the proxies available in Debian for different services like for DNS using bind (properly configured), dnsmasq, pdnsd or totd for FTP using frox or ftp-proxy, for X11 using xfwp, for IMAP using imapproxy, for mail using smtpd, or for POP3 using p3scan. For other protocols you can either use a generic TCP proxy like simpleproxy or a generic SOCKS proxy like dante-server, tsocks or socks4-server. Typically, you will also use a web caching system (like squid) and a web filtering system (like squidguard or dansguardian).
5.14.3.2 Manual init.d configuration

Another possibility is to manually configure your firewall rules through an init.d script that will run all the iptables commands. Take the following steps:

Review the script below and adapt it to your needs.

Test the script and review the syslog messages to see which traffic is being dropped. If you are testing from the network you will want to either run the sample shell snippet to remove the firewall (if you don’t type anything in 20 seconds) or you might want to comment out the default deny policy definitions (-P INPUT DROP and -P OUTPUT DROP) and check that the system will not drop any legitimate traffic.

Move the script to /etc/init.d/myfirewall

Configure the system to run the script before any network is configured:

#update-rc.d myfirewall start 40 S . stop 89 0 6 .

This is the sample firewall script:

#!/bin/sh
# Simple example firewall configuration.
#
# Caveats:
# – This configuration applies to all network interfaces
# if you want to restrict this to only a given interface use
# ‘-i INTERFACE’ in the iptables calls.
# – Remote access for TCP/UDP services is granted to any host,
# you probably will want to restrict this using ‘–source’.
#
# chkconfig: 2345 9 91
# description: Activates/Deactivates the firewall at boot time
#
# You can test this script before applying with the following shell
# snippet, if you do not type anything in 10 seconds the firewall
# rules will be cleared.
#—————————————————————
# while true; do test=”"; read -t 20 -p “OK? ” test ; \
# [ -z "$test" ] && /etc/init.d/myfirewall clear ; done
#—————————————————————

PATH=/bin:/sbin:/usr/bin:/usr/sbin

# Services that the system will offer to the network
TCP_SERVICES=”22″ # SSH only
UDP_SERVICES=”"
# Services the system will use from the network
REMOTE_TCP_SERVICES=”80″ # web browsing
REMOTE_UDP_SERVICES=”53″ # DNS
# Network that will be used for remote mgmt
# (if undefined, no rules will be setup)
# NETWORK_MGMT=192.168.0.0/24
# Port used for the SSH service, define this is you have setup a
# management network but remove it from TCP_SERVICES
# SSH_PORT=”22″

if ! [ -x /sbin/iptables ]; then
exit 0
fi

fw_start () {

# Input traffic:
/sbin/iptables -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# Services
if [ -n "$TCP_SERVICES" ] ; then
for PORT in $TCP_SERVICES; do
/sbin/iptables -A INPUT -p tcp –dport ${PORT} -j ACCEPT
done
fi
if [ -n "$UDP_SERVICES" ] ; then
for PORT in $UDP_SERVICES; do
/sbin/iptables -A INPUT -p udp –dport ${PORT} -j ACCEPT
done
fi
# Remote management
if [ -n "$NETWORK_MGMT" ] ; then
/sbin/iptables -A INPUT -p tcp –src ${NETWORK_MGMT} –dport ${SSH_PORT} -j ACCEPT
else
/sbin/iptables -A INPUT -p tcp –dport ${SSH_PORT} -j ACCEPT
fi
# Remote testing
/sbin/iptables -A INPUT -p icmp -j ACCEPT
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -j LOG

# Output:
/sbin/iptables -A OUTPUT -j ACCEPT -o lo
/sbin/iptables -A OUTPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# ICMP is permitted:
/sbin/iptables -A OUTPUT -p icmp -j ACCEPT
# So are security package updates:
# Note: You can hardcode the IP address here to prevent DNS spoofing
# and to setup the rules even if DNS does not work but then you
# will not “see” IP changes for this service:
/sbin/iptables -A OUTPUT -p tcp -d security.debian.org –dport 80 -j ACCEPT
# As well as the services we have defined:
if [ -n "$REMOTE_TCP_SERVICES" ] ; then
for PORT in $REMOTE_TCP_SERVICES; do
/sbin/iptables -A OUTPUT -p tcp –dport ${PORT} -j ACCEPT
done
fi
if [ -n "$REMOTE_UDP_SERVICES" ] ; then
for PORT in $REMOTE_UDP_SERVICES; do
/sbin/iptables -A OUTPUT -p udp –dport ${PORT} -j ACCEPT
done
fi
# All other connections are registered in syslog
/sbin/iptables -A OUTPUT -j LOG
/sbin/iptables -A OUTPUT -j REJECT
/sbin/iptables -P OUTPUT DROP
# Other network protections
# (some will only work with some kernel versions)
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 0 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
echo 1 > /proc/sys/net/ipv4/ip_always_defrag
echo 1 > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route

}

fw_stop () {
/sbin/iptables -F
/sbin/iptables -t nat -F
/sbin/iptables -t mangle -F
/sbin/iptables -P INPUT DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P OUTPUT ACCEPT
}

fw_clear () {
/sbin/iptables -F
/sbin/iptables -t nat -F
/sbin/iptables -t mangle -F
/sbin/iptables -P INPUT ACCEPT
/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables -P OUTPUT ACCEPT
}

case “$1″ in
start|restart)
echo -n “Starting firewall..”
fw_stop
fw_start
echo “done.”
;;
stop)
echo -n “Stopping firewall..”
fw_stop
echo “done.”
;;
clear)
echo -n “Clearing firewall rules..”
fw_clear
echo “done.”
;;
*)
echo “Usage: $0 {start|stop|restart|clear}”
exit 1
;;
esac
exit 0

Instead of including all of the iptables rules in the init.d script you can use the iptables-restore program to restore the rules saved using iptables-save. In order to do this you need to setup your rules, save the ruleset under a static location (such as /etc/default/firewall)
5.14.3.3 Configuring firewall rules through ifup

You can use also the network configuration in /etc/network/interfaces to setup your firewall rules. For this you will need to:

Create your firewalling ruleset for when the interface is active.

Save your ruleset with iptables-save to a file in /etc, for example /etc/iptables.up.rules

Configure /etc/network/interfaces to use the configured ruleset:

iface eth0 inet static
address x.x.x.x
[.. interface configuration ..]
pre-up iptables-restore < /etc/iptables.up.rules

You can optionally also setup a set of rules to be applied when the network interface is down creating a set of rules, saving it in /etc/iptables.down.rules and adding this directive to the interface configuration:

post-down iptables-restore < /etc/iptables.down.rules

For more advanced firewall configuration scripts through ifupdown you can use the hooks available to each interface as in the *.d/ directories called with run-parts (see run-parts(8)).
5.14.3.4 Testing your firewall configuration

Testing your firewall configuration is as easy, and as dangerous, as just running your firewall script (or enabling the configuration you defined in your firewall configuration application). However, if you are not careful enough and you are configuring your firewall remotely (like through an SSH connection) you could lock yourself out.

There are several ways to prevent this. One is running a script in a separate terminal that will remove the firewall configuration if you don’t feed it input. An example of this is:

$ while true; do test=”"; read -t 20 -p “OK? ” test ; \
[ -z "$test" ] && /etc/init.d/firewall clear ; done

Another one is to introduce a backdoor in your system through an alternate mechanism that allows you to either clear the firewall system or punch a hole in it if something goes awry. For this you can use knockd and configure it so that a certain port connection attempt sequence will clear the firewall (or add a temporary rule). Even though the packets will be dropped by the firewall, since knockd binds to the interface and sees you will be able to work around the problem.

Testing a firewall that is protecting an internal network is a different issue, you will want to look at some of the tools used for remote vulnerability assessment (see Remote vulnerability assessment tools, Section 8.1) to probe the network from the outside in (or from any other direction) to test the effectiveness of the firewall configuation.

No tags

Jul/07

Sarg three scripts for daily, weekly and monthly statistics.

No comments · Posted by nguyen in Linux Docs

Hi,

here are three scripts for daily, weekly and monthly statistics. After the monthly statistics the access.log will be rotated.
This is based on Jeremy Lahners first script on your web page, many thanks to him.

Daily (Jeremy´s script):
————————————-

#!/bin/bash

#Get current date
TODAY=$(date +%d/%m/%Y)

#Get one week ago today
YESTERDAY=$(date –date “1 day ago” +%d/%m/%Y)

/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -o /usr/local/apache/htdocs/reports/daily -z -d $YESTERDAY-$TODAY

exit 0

Weekly
————————————-

#!/bin/bash

#Get current date
TODAY=$(date +%d/%m/%Y)

#Get one week ago today
YESTERDAY=$(date –date “1 week ago” +%d/%m/%Y)

/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -o /usr/local/apache/htdocs/reports/weekly -z -d $YESTERDAY-$TODAY

exit 0

Monthly

#!/bin/bash

#Get current date
TODAY=$(date +%d/%m/%Y)

#Get one week ago today
YESTERDAY=$(date –date “1 month ago” +%d/%m/%Y)

/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -o /usr/local/apache/htdocs/reports/monthly -z -d $YESTERDAY-$TODAY

/usr/local/squid/bin/squid -k rotate

exit 0

And the same for the admin who want the report as an email (example for daily)

#!/bin/bash

#Get current date
TODAY=$(date +%d/%m/%Y)

#Get one week ago today
YESTERDAY=$(date –date “1 day ago” +%d/%m/%Y)

/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -e user@site.com -z -d $YESTERDAY-$TODAY

exit 0
~

Cheers
Lars

No tags

Jul/07

SARG – Daily/Weekly/Monthly Squid usage reports creation tool

No comments · Posted by nguyen in Linux Docs

#!/bin/sh
# SARG – Daily/Weekly/Monthly Squid usage reports creation tool
# Written by Ugo Viti

# —————————————————————————-
#
# Copyright (C) 2005 Ugo Viti
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
# —————————————————————————-

# Thanks for enanchements to:
# – martijn
# – Stas Degteff https://sourceforge.net/users/stas_degteff/

VER=20050202

## What is this?
# sarg-reports (this file) is a simple bash script written to automate
# the SARG (a powerful squid log analyzer) reports and log management.
# Sarg it self, provide to end user a generic interface to create
# reports based on squid access log (begin of log to current date).
# sarg-reports (this script) is useful because it allow you to easly
# create and manage Daily, Weekly and Monthly reports.
# Try it, within 5 minutes you will be ready to rule
# using sarg-reports is very easy, read the following 3 steps to know how

## Requirements
# a) An unix system with bash shell (like GNU/Linux, FreeBSD, etc…)
# b) Squid – http://www.squid-cache.org
# c) Sarg – http://web.onda.com.br/orso/sarg.html

##
## Installation guide and configuration parameters
##

# 1) Download Squid and Sarg, Install, Configure and Tune
# they before continue reading

# 2) In root crontab (crontab -e) insert the following lines:
# (the today report creation time depend mostly of your squid server
# load average, tune it):
#
# — BEGIN ROOT CRONTAB —
# PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
# 00 08-18/1 * * * sarg-reports today
# 00 00 * * * sarg-reports daily
# 00 01 * * 1 sarg-reports weekly
# 30 02 1 * * sarg-reports monthly
# — END ROOT CRONTAB —
#
# REMEMBER: if you use logrotate, configure it to rotate the logs within MONTHLY basis,
# AFTER sarg-reports created the monthly html report.

# 3) Customize the following variables:
# (Please, configure accurately the sarg.conf file before)
#
# (SARG) The sarg executable location
# (CONFIG) The sarg main configuration file location
# (HTMLOUT) Location where will be saved the reports
# (PAGETITLE) The title of main index page
# (LOGOIMG) Image logo to view in main index page
# (LOGOLINK) HTTP web page link of logo
# (DAILY) Word ‘daily’ translation, translate it to your language
# (WEEKLY) Word ‘weekly’ translation, translate it to your language
# (MONTHLY) Word ‘monthly’ translation, translate it to your language
# (EXCLUDELOG1) Exclude text from cron emails
# + (normally, sarg, during cron activity, if it don’t find any valid records,
# (EXCLUDELOG2) it will output an error message (usually on ‘today’ reports).
# I don’t want to be warned by email about this, so, i wrote the ‘text’
# that will be never logged.
# This is useful to receive email of real problems only (enjoy that)

#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
## Italian Language
SARG=/usr/bin/sarg
CONFIG=/etc/sarg/sarg.conf
HTMLOUT=/var/www/html/admin/log/proxy
PAGETITLE=”Statistiche Proxy di $(hostname)”
LOGOIMG=http://www.initzero.it/images/initzero-logo.jpg
LOGOLINK=http://www.initzero.it
DAILY=Giornaliero
WEEKLY=Settimanale
MONTHLY=Mensile
EXCLUDELOG1=”SARG: Nessun records trovato.”
EXCLUDELOG2=”SARG: Fine”
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
## Russian Language
# SARG=”/usr/bin/sarg”
# CONFIG=/etc/sarg/sarg.conf
# HTMLOUT=/var/www/sarg-reports
# PAGETITLE=”óÔÁÔÉÓÔÉËÁ ÓÅÒ×ÅÒÁ ÐÒÏËÓÉ Squid” # russian koi8-r
# LOGOIMG=http://litek.ru/images/logotop.gif
# LOGOLINK=http://litek.ru
# DAILY=”åÖÅÄÎÅ×ÎÁÑ” # russian koi8-r
# WEEKLY=”åÖÅÎÅÄÅÌØÎÁÑ” # russian koi8-r
# MONTHLY=”åÖÅÍÅÓÑÞÎÁÑ” # russian koi8-r
#EXCLUDELOG1=”SARG: Records in file:”
#EXCLUDELOG2=”SARG: ïÔÞÅÔ ÕÓÐÅÛÎÏ ÓÇÅÎÅÒÉÒÏ×ÁÎ ×” # sarg.conf: language Russian_koi8
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

######################################################################
## The configuration is end, so don’t touch anything bellow

# TEMP Files
TMPFILE=/tmp/sarg-reports.$RANDOM
ERRORS=”${TMPFILE}.errors”

# Date Calc
MANUALDATE=$2
case “$(uname)” in
“FreeBSD”)
TODAY=$(date +%d/%m/%Y)
YESTERDAY=$(date -v-1d +%d/%m/%Y)
WEEKAGO=$(date -v-1w +%d/%m/%Y)
MONTHAGO=$(date -v-1m +01/%m/%Y)-$(date -v-1m +31/%m/%Y)
;;
“OpenBSD”)
TODAY=$(date +%d/%m/%Y)
YESTERDAY=$(date -r $((`date +%s` – 86400 )) +%d/%m/%Y)
WEEKAGO=$(date -r $((`date +%s` – 604800)) +%d/%m/%Y)
MONTHAGO=$(perl -e ‘@t=localtime(time); $y=$t[4]==0?$t[5]+1899:$t[5]+1900; $m=$t[4]==0?12:$t[4]; print “1/$m/$y-”,$m==2?$y%4>0?28:29:$m==4||$m==6||$m==9||$m==11?30:31 ,”/$m/$y\n”;’)
;;
*)
TODAY=$(date –date “today” +%d/%m/%Y)
YESTERDAY=$(date –date “1 day ago” +%d/%m/%Y)
WEEKAGO=$(date –date “1 week ago” +%d/%m/%Y)
MONTHAGO=$(date –date “1 month ago” +01/%m/%Y)-$(date –date “1 month ago” +31/%m/%Y)
;;
esac

# Fix for Red Hat 9 systems and coreutils prior to 5.0 version
export LC_ALL=C

# Main index.html creation
create_index_html ()
{
echo -e “\
\n\
\n\
\n\
\n\
\n\

\n\

$PAGETITLE
$DAILY
$WEEKLY
$MONTHLY

\n\

\n\
\n\
” > $HTMLOUT/index.html
}

# Functions
exclude_from_log ()
{
cat $ERRORS | grep -v “$EXCLUDELOG1″ | grep -v “$EXCLUDELOG2″
rm -f $TMPFILE*
}

manual ()
{
DAILYOUT=$HTMLOUT/$DAILY
mkdir -p $DAILYOUT
create_index_html
if [ -z "$MANUALDATE" ]
then
echo “No date given, please specify a valid date (DD/MM/YYYY)”
else
$SARG -f $CONFIG -d $MANUALDATE -o $DAILYOUT
fi
}

today ()
{
DAILYOUT=$HTMLOUT/$DAILY
mkdir -p $DAILYOUT
create_index_html
$SARG -f $CONFIG -d $TODAY -o $DAILYOUT >$ERRORS 2>&1
exclude_from_log
}

daily ()
{
DAILYOUT=$HTMLOUT/$DAILY
mkdir -p $DAILYOUT
create_index_html
$SARG -f $CONFIG -d $YESTERDAY -o $DAILYOUT >$ERRORS 2>&1
exclude_from_log
}

weekly ()
{
WEEKLYOUT=$HTMLOUT/$WEEKLY
mkdir -p $WEEKLYOUT
create_index_html
$SARG -f $CONFIG -d $WEEKAGO-$YESTERDAY -o $WEEKLYOUT >$ERRORS 2>&1
exclude_from_log
}

monthly ()
{
MONTHLYOUT=$HTMLOUT/$MONTHLY
mkdir -p $MONTHLYOUT
create_index_html
$SARG -f $CONFIG -d $MONTHAGO -o $MONTHLYOUT >$ERRORS 2>&1
exclude_from_log
}

case $1 in
manual)
manual
;;
today)
today
;;
daily)
daily
;;
weekly)
weekly
;;
monthly)
monthly
;;
*)
echo “SARG – Daily / Weekly / Monthly – Squid proxy usage reports creation tool”
echo “Written by Ugo Viti “
echo “Version: $VER”
echo
echo “Usage: $0 [OPTIONS]“
echo
echo “Allowed options:”
echo ” manual, Create Manual report”
echo ” today, Create Today report”
echo ” daily, Create Daily report”
echo ” weekly, Create Weekly report”
echo ” montly, Create Monthly report”
exit 0
esac

## HISTORY:
# 20050502 – Stas Degteff added support for non latin Charset and added support for OpenBSD
# 20030826 – FreeBSD support (thanks to martijn to let me coding on your FreeBSD server )
# 20030715 – Some cleanups
# 20030623 – Manual report creation
# 20030620 – Main Index creation
# 20030619 – Solved ’sort’ bug on Red Hat 9 systems
# 20030618 – First Version

## TODO:
# – Smarty weekly recognition…
# Like “begin of last week to end of last week”,
# doesn’t like this script do: “7 days ago to yesterday”
# – Monthly recognition isn’t so elegant (is very ugly, i know)
# – Suggestions are welcome
# – If you Rotate the squid logs before sarg-reports will run,
# it will not create any html reports
# (TIPS: Rotate the logs after sarg-reports)

No tags

Do, or do not. There is no ‘try’ |

Categories

Blogroll

Meta

Archive for July 28th, 2007

Simplify data extraction using Linux text utilities

Securing Debian Manual

Sarg three scripts for daily, weekly and monthly statistics.

SARG – Daily/Weekly/Monthly Squid usage reports creation tool

Find it!

Archives