A few days later I dug out my kindle and noticed the display was suffering from a split personality. The bottom half was telling me to switch it on, while the top half was still showing three little birds, outside my window, singing a sweet song. Oh dear. After a visit to the University of Google it became apparent to me that this problem is not uncommon. Weird, but not uncommon. Weird because you can run your finger across the screen and feel nothing; no cracks no blemishes. But it’s clearly broke. It may be silky smooth on the outside but it hides a shattered interior.

The forums on Amazon buzz with feel-good stories of customers phoning Amazon and getting great trade-in deals on their ‘just out of warranty’ kindles. My Kindle was not just out of warranty, it was exceedingly, comfortably and generously out of warranty, and had been for a couple of years. Still worth a try …

Well that didn’t go anywhere useful. Nice chap all the same, and together across the ether we visited the Amazon website where we discussed the nice shiny new kindles and he advised me that I could ‘buy’ one, at the price shown. Apparently bears also go to the bathroom in the woods. Ok, let’s go to ebay.

Aha … Now we’re cooking … This looks just the chap. A couple of days later a huge ball of bubble-wrap arrives at work, and somewhere inside, is a shiny new kindle screen. And there’s even a link to their Youtube video, yeah, well whatever. Why watch a youtube video when it’s NELUG night, and, well, you can run linux on a kindle can’t you? Apparently.

Richard takes the kindle to bits

That evening I turned up with a broken kindle, a new screen, and a random selection of small screwdrivers. I dumped them on the table and headed for the bar. A few minutes later I returned with my drink to find surgery was already underway. In the time it takes to say “a pint of Black Sheep please” Richard had prised open the cover, removed the battery, and was poking at various bits of the kindle’s anatomy with professional interest.

I made a perfunctory pretense of watching the instruction video on youtube but the Nelug hive mind was working quite well without it. With the new screen fitted re-assembly was, as they say, simply the reverse of disassembly. I stuck to the Black Sheep while Richard stuck to the screwdriver and, despite some frisky screws that had decided to go for a wander and wanted to live somewhere else for a while, it all ended well.

The broken screen on the left with the repaired Kindle on the right

We also spent some time diagnosing CUPS and wireless driver problems, and discussing the implications of Google’s recent announcement that is supporting OAuth 2.0 authentication for several of its APIs.

What is a Regular Expression?

Purpose

A regular expression is a flexible way of defining patterns of text. It is a formal language which is interpreted by a regular expression engine (which might be part of an application or a programming language) that parses input text and compares it to the regular expression, and then performs operations on text that matches the regular expression.

Common uses of regular expressions include:

• Matching text
• Substituting text
• Extracting text

Syntax

The basic syntax of a regular expression is /pattern/flags. The main part is the text pattern description, and the flags control the behaviour of the regular expression engine.

Different regular expression engines support different features, and also slightly vary in their syntax. After a overview of general regexp syntax we will look at some common applications and languages and how they support regular expressions.

Examples

%\d\d?/\d\d?/\d\d\d?\d?%

This will match something that looks like a date, in a format like dd/mm/yyyy or m/d/yy. Note that it does not check that it is a valid date, a string like 75/33/9876 would match. Also note that a percentage mark has been used as the regexp delimiter; this can be clearer when the pattern contains slashes.

/<p( [^>]*)?>.*?</p>/m

This regular will match a paragraph element and its contents in a HTML document.

/
  (?:
    (?:(-?\d{1,3})m {0,3}(-?\d{1,4})y (?:\(( {0,2}-?\d{1,2})\))?)
    |
    (?: {0,2} (?:(-?\d{1,3})\/(\d)|(Junct))[ ]
      - {0,2} (?:(-?\d{1,3})\/(\d)|(Junct))
    )
  )
  \s+
  (C|C&A&T|C & [AT]|L[234]|\*L4|SD[12])?
  \s+
  (?:\(((?:[ -]\d{2})|(?:\d\.\d))\))?
  \s+
  (-?\d+)?
  \s+
  (ALIG35|AL70|GAUGE|MT70|[LR]TOP
   | TW[35]M|CYC(?:[69]_|1[38])(?:BO|[LR]T))
  \s*=?\s* (-?\d{1,3}\.\d+)mm
  (?:\(1: ?(\d{2,3})\))?
  \s+
  \[ {0,2}(\d{1,3})\]
  \s*(.*)
  (?: +> :)_+ *_+:_+\/_+\/_+
  \s+
  (?:to +(?:(-?\d{1,3})m {0,3}(-?\d{1,4})y[ ]
     (?:\(( {0,2}-?\d{1,2})\))?): *(\d+)cycles)?
  \s+
  ((?:P )?(?:IN)?VALID (?: BUT OFF ROUTE)?|OFF ROUTE|UNVERIFIED)?
/gmx

This is a much more complex example. It is a regular expression that was written to match text in reports produced by a legacy system. These reports had been designed to be printed and read; by using a regular expression it was possible to parse the report and extract the important information from it. This regular expression matches a group of lines in the report and captures the bits of data that we are interested in. It would be possible to use other methods to parse this report, but the flexibility of regular expressions make it well suited to cope with the quirks of the report formatting produced by different versions of the legacy software; the use of alternation and variable matches means that this regexp can match all formats of the report instead of having to rewrite the parsing code for each version.

Regexp elements

Characters

Normal characters

Normal characters match themselves only.

• a b c X Y Z
• 0 1 2 3 4 5 6 7 8 9
• " _ = #

Special characters

More exoctic characters are matched using character sequences.

.

The dot character will match almost any single character. It does not usually match line break characters, unless the /g flag is set.

\* \? \} \[ \] \/ \\ \^ \$

If you need to match a literal character that has a special meaning in regular expresssions then it needs to be escaped using a backslash.

\n \t \e \a

There are several predefined sequences for non-printable characters. \n is a new line character, \t is a tab, \e is an escape character and \a is a bell. These will be familiar to anyone who has used C or many other programming languages.

\xB0 \u0260

Some regular expression engines allow arbitrary hexadecimal or Unicode code points to be represented using a \x00 or \u0000 syntax.

Character classes

By using [square brackets] you can match any of several different characters.

Collections of characters

[abc]

The simplest form is a list of characters in square brackets, this will match any one of those characters.

[0-9]

[a-z]

[0-9a-zA-Z]

To make it simpler to match a large number of possible characaters you can specify ranges.

[-+0-9]

Simple characters and ranges can be combined as shown above. Note that, due to its special meaning for ranges, to match a literal hyphen character then you can place it at the start of a character class (alternatively you can escape it with a backslash).

[^abc]

Negation is done by having a caret at the start of a character class. The above example will match any character apart from a, b, or c.

Pre-defined character classes

There are many predefined shorthand sequences for commonly used character classes.

\d

Any digit.

\d

Any character other than a digit.

\s

Any space character, e.g. space, tab.

\S

Any non-space character.

\w

Any word character. The definition of word characters can vary, but it usually means any letter, any digit, or an underscore.

\W

Any non-word character.

[[:alpha:]]

Any letter character. This is an example of a POSIX character class. Note the double square brackets used here; the POSIX character class is [:alpha:] which can only be used inside the normal square brackets for character classes. POSIX character classes can be combined with other elements within a character class, e.g. [[:alpha::]ab[:digit:]].

Repetition

Quantifiers are used to control repetetive matching. Greedy quantifiers will try and match as much text as possible, lazy quantifiers will try and match as little as possible. Lazy quantifiers are used much less frequently than greedy quantifiers.

Normal quantifiers

ab?c

The question mark character will match either zero or one occurrence of the preceding expression. The above example will match either ac or abc, preferring the latter if possible.

ab*c

The asterisk character matches zero or more occurrences. The above example will match ac, abc, abbc, abbbc, …

ab+c

The plus character matches one or more occurrences. The above example will match abc, abbc, abbbc, …

Range quantifiers

ab{3}c

A number inside braces indicates an exact number of occurrences. The above only will match abbbc.

ab{2,4}c

Two numbers inside braces, separated by a comma, indicates a range of occurrences. This example will match abbc, abbbc, or abbbbc.

ab{2,}c

Omitting the second number, but keeping the comma, gives a minimum number of occurrences. This example will match abbc, abbbc, abbbbc, …

ab{,3}c

Omitting the first number, gives a maximum number of occurrences. This example will match ac, abc, abbc, or abbbc.

Lazy quantifiers

ab??c

Like ab?c this will match either ac or abc, but the double question mark will make it prefer to match the former if this is possible.

ab*?c

This will match the same set of possibilities as ab*c, but if there are several possible matches then it will match as few b characters as it can.

ab+?c

This is the lazy equivalent of ab+c.

ab{1,2}?c

Similarly, putting a question mark after a range quantifier makes it lazy.

Alternation, grouping and matching

Alternation

a|b

Matching one a set of possible different is done by using the pipe operator. This will match either a or b.

foo|bar

The alternation operator has very low precedence, in particular lower than a sequence of characters. This means that this example will match either foo or bar, not fooar or fobar.

foo|bar|baz

Matching one of more than two possibilities is simply done by using multiple pipe operators. This will match any one of foo, bar, or baz.

Grouping and matching

foo(bar)?
Parentheses group a set of characters together. Here the ? quantifier applies to everything inside the brackets, so this will match either foo or foobar.
foo(bar|foo)
Parentheses can be combined with other operators, such as the pipe alternation operator. This will match either foobar or foofoo.
(fooba[rz])
As well as grouping characters together, parentheses are used to capture elements within a regular expression which can then be examined later on. This will match either foobar or foobaz and the matching text will be captured; in sed it will be stored as /1, in perl in the variable $1.

Grouping without matching

(?:foo)

If you want to group a set of characters together without capturing them, then the (?:…) operator will do this.

Positional markers

As well as matching text itself, you can control where the text occurs by using positional markers. These markers do not match any text themselves, but control where the other patterns in the regular expression are able to match text.

Beginning/end of lines

^foo

^ matches the start of a line or piece of text. This example will only match foo if it is at the start of a line.

bar$

$ matches the end of a line or the end of the text. This will match bar when it is at the end of a line.

Beginning and end of words

\b

This batches word boundaries. In the string foo bar it will match the start of the string, between the o and the space at the end of the word foo, between the space and the b at the start of the word bar, and at the end of the string.

\B

This is the opposite of /b and will match anywhere other than a word boundary, i.e. in the middle of words, and within sequences of non-word characters.

Lookaround

foo(?=bar)

This is a positive lookahead: it will match if the text contains foobar, but will only match the foo part, and not the bar part.

foo(?!bar)

This is a negative lookahead: it will match foo, unless it is immediately followed by bar.

(?<=foo)bar

This is a positive lookbehind: it will match the text bar, but only if it occurs as foobar. The text foo will not be part of the match.

(?<!foo)bar

This is a negative lookbehind: it will match bar unless it is preceded by foo.

Flags

Flags controls the overall behaviour of the regular expression.

i

The i (insensitive) flag tells the regular expression to match in a case insensitive manner. /foo/ will only match foo, but /foo/i will also match FOO, fOo, and so on.

g

The g (global) flag tells the regexp engine to match all possible instances of the regular expression. Normally it will stop after the first match, but if this flag is set then it will look for any further matches.

m

The m (multiline) flag is for regular expressions that span more than one line of text. Normally the match has to be on a single line, but if this flag is set then the match can span several lines. This also changes the behaviour of the dot character class; it normally does not match line end characters, but will if the m flag is set.

x

Unlike the other flags, this does not alter the behaviour of the regexp engine. Instead it allows you to write more legible regular expressions by splitting them across multiple lines: the lines will be concatenated with leading and trailing white space ignored. The earlier example used this flag to break up a very long regular expression.

Programs and Languages

grep

grep is a simple program usually used to extract lines from text files that match a given pattern. It is often used to match plain character sequences, so there are very few special characters: most regular expression operators have to be preceded by a backslash to give them their normal meaning. Exceptions to this are the * quantifier and the ^ and $ anchors which work as normal.

Metacharacters

. \n \t \s \S \w \W
POSIX character classes, e.g. [[:digit:]]

Repetition

\? * \+ \{n,m\}

Alternation and grouping

\| \(…\)

Anchoring

^ $ \b \B \< \>

Examples

grep 'FIXME\|TODO' */*.p[lm]

This will print any lines containing either FIXME or TODO from perl files.

egrep

Grep also has an extended mode which removes makes most of the operator characters behave as normal, so you do not need to prefix them with a backslash like in its basic mode. If you are using anything other than very simple regular expressions with grep then is best to use this mode.

Metacharacters

. \n \t \s \S \w \W
POSIX character classes, e.g. [[:digit:]]

Repetition

? * + {n,m}

Alternation and grouping

| (…)

Anchoring

^ $ \b \B \< \>

Examples

grep -E 'FIXME|TODO' **/*.p[lm]

sed

sed performs operations on streams of characters. The most common operation is to replace strings, but many more powerful things are possible. Its regexp syntax is very similar to the basic mode of grep.

Metacharacters

. \n \t \s \S \w \W
POSIX character classes, e.g. [[:digit:]]

Repetition

* \? \+ \{m,n\}

Alternation and grouping

\| \( \)

Anchors

^ $ \b \B \` \'

Examples

sed 's%\(\d\d\)/\(\d\d\)/\(\d\d\)%20\3-\1-\2%'

This will transform dates from the format mm/dd/yy to the format yyyy-mm-dd, assuming that the date is in the 21st century.

sed '/^__END__$/,$d' foo.pl

This will strip the perlpod, and anything else that follows a __END__ line, from a perl file.

sed 's/^\s\+//;s/\s\+$//'

This strips all leading and trailing spaces from text.

sed 's/^\s*\(.*\S\)\?\s*$/\1/'

This also strips leading and trailing spaces from text. The previous example uses two statements, one for leading space, and one for trailing space; this one using a single statement using a backreference. This approach is much less efficient and will be several order of magnitudes slower than the previous example due to the increased memory requirements from the backreference.

perl

Perl has by far the most comprehensive support for regular expression features. Many features appear first in Perl before being copied by other languages and programs.

The Perl regular expression syntax is used in many applications and other programming languages through the PCRE library. This library is used by PHP, the Apache webserver, the Exim mailserver, and many others.

Metacharacters

All metacharacters are supported.

Repetition

? * + {n,m} ?? *? +? {n,m}?

Alternation and grouping

| (…) (?:…)

Anchors

^ $ \b \B (?=…) (?!…) (?<=…) (?<!…)

Perl Example 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

if ( /^ *ELR : +([A-Z]{3}\d?|[A-Z]{2}\d{2})/o ) { $elr = $1; } elsif ( /^ *Track Id : +\d(4})/o ) { $tid = $1; } elsif ( /^ *\d{1,3}.\d{4}/o) { my @data = unpack($template, $_); for (my $i = @data; $i >= 0; --$i) { if ($i % 2 == 0) { # every other element is a separator -- delete these splice(@data, $i, 1); } else { # remove leading/trailing spaces $data[$i] =~ s/^ +//; $data[$i] =~ s/ +$//; } print $elr, $tid, @data; }

Perl Example 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

# decodes a standard deviation value my %errcodes = ( NA => -1, NF => -2, NV => -3, SS => -4, ST => -5 );   sub sdval { my $val = $_[0]; if ($val =~ m/\d\.\d/) { return $val; } elsif ($val =~ m/\*\*/) { return 10; } elsif ($val =~ m/($errcodes)/) { return $errcodes{$1}; } else { return ""; } }

Further reading

• perldoc perlretut
• Mastering Regular Expressions by Jeffrey E F Friedl.
  2nd edition published by O'Reilly, 2002.

---

Based on a talk presented by Oliver Burnett-Hall at Durham LUG on 17 February 2008.

]]> http://www.nelug.org.uk/a-brief-introduction-to-regular-expressions/feed/ 0 http://www.nelug.org.uk/durham-linux-user-group/ http://www.nelug.org.uk/durham-linux-user-group/#comments Mon, 16 Apr 2012 13:32:19 +0000 admin http://www.nelug.org.uk/?p=20 Continue reading →]]> We are in the process of updating this site… Apologies if the content you were looking for is no longer here… If you want to help, please get in touch.

You can join the mailing list by going to http://mailman.lug.org.uk/mailman/listinfo/durham

Meetings are held in Durham (In the bar at Durham Rowing Club – See the location page).

We meet on the 3rd Tuesday of every month, from around 19:30, until around 22:30

]]> http://www.nelug.org.uk/durham-linux-user-group/feed/ 0 http://www.nelug.org.uk/digital-audio-essentials/ http://www.nelug.org.uk/digital-audio-essentials/#comments Wed, 18 Oct 2006 12:44:15 +0000 olly-bh http://www.nelug.org.uk/?p=155 Continue reading →]]>
Title: Digital Audio Essentials: A Comprehensive Guide to Creating, Recording, Editing, and Sharing Music and Other Audio
Author: Bruce Fries, Marty Fries
Price: £24.95
Publisher: O’Reilly
Published: May 2005
Reviewed by: Dougie Nisbet
Review date: October 2006
Rating: 4/5

Overview

This book presents itself an interesting challenge in that it attempts to present a comprehensive guide to digital audio in a book that isn’t the size of a brick. The authors do this successfully and the book is a worthwhile read.

General

The narrative is nicely pitched and soon settles down comfortably somewhere between a Dummies Guide and a dry reference book. The authors use simple elegant language to get across their message without sounding condescending or trivialising. They do not fall into the trap of presenting pages of space-wasting screen dumps and where they are used they are used sparingly and sensibly.

The book’s great strength is in astutely judging how much detail to delve into each subject. It does this in such a manner that each section can be read in as little or as much depth as desired, without any feeling that the narrative flow has been interrupted or that by skimming a particular section it will cause problems later. It is a well-paced book in the category of an enjoyable cover-to-cover read rather than a dry reference work.

The authors often refer to subjects that will be covered in more detail later in the book, rather than simply cross-referencing. This strikes a nice balance; you don’t feel compelled to jump forward to the referenced text that will be reached in due course anyway. This gives a good feeling of narrative flow.

Advice is always practical. For example, on ‘Purchasing’ the book explains the importance of satisfying your own functional requirements and the diminishing returns of extra expenditure. Chapter 2 gives very good advice on how to choose hardware sensibly and how to go about intelligent cost-effective upgrades without getting sucked into impulse buys based on faster processors and higher clock-cycles.

The section on Minimizing Noise (chapter 4, page 61) is well explained and practical.

I found Chapter 8 a bit of a stumble. An abrubt blast of technical talk that could be quite challenging if the reader did not have some background experience of audio. However, as befits the style of the book, you can re-read or skim as desired without spoiling the overall narrative flow of the book.

The explanation of Digital Rights Management and file formats is well discussed and illuminating.

There is some repetition throughout the book but this tends to be more useful that irritating as it helps consolidate earlier explained material.

On page 160 Linux is mentioned in the discussion of formats, but barely mentioned elsewhere where a brief reference could be useful. There is a slight inconsistency here as the authors make clear that the book is geared towards Mac and Windows XP platforms, so it seems a bit pointless making the Linux reference at all. But since it has been mentioned, there are several opportunities in the book where a short reference could be made to Linux without any need to go into detail that would allow the reader to explore further if required. For example, it would take negligable space to mention in passing that there is a sizeable range of media players available on Linux.

The discussion on Perceptual Encoding in Chapter 10 is very interesting and clearly explained.

I found the section on Equalization (page 237) quite hard work. Perhaps too much was crammed in here in too short a space, and a longer, gentler explanation would have worked better. However I’m sure a few re-reads would have helped too.

In the section on Noise Reduction and recording from vinyl, I feel the authors should have made the point in passing that in many cases it is worth checking to see if the piece of work is already available as a digitally remastered CD. Even the most enthusiastic hobbyist may find that the time required and quality achieved from transferring vinyl to digital is not worth the effort when a professionally produced CD can be bought for a modest sum.

The book is full of interesting nuggets of information just waiting to be stumbled upon. I was intrigued to read about the lifetime of recorded versus unrecorded CD-Rs (p. 281), something that would never have occurred to me.

The wrapping up of the book with an in-depth explanation of internet streaming radio was of little interest to me and was the only part of the book that I felt was covered in too much detail. I was left wondering who the target audience was.

Niggles

On page 18, chapter 2, it might be worth mentioning that RAM upgrades can be limited by the BIOS or OS even if there are free slots.

The section on ripping (chapter 4, page 49) makes no mention of the perils of Copy Protected CDs. For a book that is generally very good at warning the reader of the pitfalls and legalities of ripping, this should be included.

The explanation on spyware (chapter 5, page 95) mentions always clicking on the ‘No’ button on intrusive pop-up windows. With an increasing number of adverts now masquerading as pop-up windows with dialogues, where the whole pop-up is a clickable link, it would be safer to click on the “X” in the top-right hand corner of the window, or even close down the browser entirely.

Normally the US bias is not an issue. However there are times where the authors could give a nod to other countries and terminologies. For example, in the section on Broadcast Radio (chapter 6, page 99) it would be helpful to explain if or where the acronyms differed from other English speaking countries. The section on Broadcast Radio should certainly mention the excellent advert-free streaming and “Listen-Again” services available from the BBC RadioPlayer website.

The introduction to the book states clearly that it is written with the Windows and MAC user primarily in mind, so it is churlish to complain that there are no references to other operating systems. However it wouldn’t take much more space to mention that software such as RealPlayer is also available for Linux (chapter 6, page 103)

Summary

A nicely balanced book that can be read cover-to-cover as well as being retained as a reference work.

]]> http://www.nelug.org.uk/digital-audio-essentials/feed/ 0 http://www.nelug.org.uk/book-review-802-11-wireless-networks-the-definitive-guide-second-edition/ http://www.nelug.org.uk/book-review-802-11-wireless-networks-the-definitive-guide-second-edition/#comments Sun, 18 Dec 2005 13:50:34 +0000 olly-bh http://www.nelug.org.uk/?p=160 Continue reading →]]>
Title: 802.11 Wireless Networks: The Definitive Guide (Second Edition)
Author: Matthew S. Gast
Price: £31.95
Publisher: O’Reilly
Published: April 2005
Reviewed by: Martin Ward
Review Date: December 2005
Rating: 5/5


Calling your book “The Definitive Guide” sets the bar high at the start, and Gast does well to live up to his title and provide virtually everything you need to know about 802.11 networking.

If you are a wardriver looking for plans to make antennae out of Pringles cans, then you won’t find them here (but they are readily available on the Internet!) If you need to set up a wireless network of any size, or are just curious about how they actually work, then this is the book for you.

Book Layout

After an Introduction and Overview in Chapters 1 and 2, Chapter 3 looks at MAC (Media Access Controller) Fundamentals: everything between the device driver and the radio.

Chapter 4 describes frame layouts.

Chapter 5 discusses WEP (the so-called Wired Equivalent Privacy) and its many shortcomings.

Chapters 6 and 7 disciss 802.1X and 802.11i respectively.

Chapter 8 deals with management functions: scanning, preauthentication, association, power conserving etc.

Chapter 9 deals with PCF (Point Coordination Function), an as-yet unimplemented standard for contention-free access to the medium.

Chapters 10 to 14 deal with the physical aspects of RF propagation, frequency hopping and modulation techniques and so on.

Chapter 15 describes the competing proposed 802.11n standards, which will offer over 100Mbs of throughput over a wireless link.

Chapters 16 to 19 give detailed information on setting up wireless hardware and drivers on Windows Mac and Linux systems.

Chapters 20 and 21 discuss access points and the options for organising the logical wireless network architecture.

Chapter 22 discusses the important issue of the security architecture.

Chapter 23 describes site planning issues and some of the project management issues.

Chapters 24 and 25 look at network analysis and performance tuning.

Finally Chapter 26 concludes.

The book is very detailed and thorough, with a lot of useful information. There is some repetition: I lost count of the number of places where the reader is warned of the broken nature of static WEP security, for example! But this is probably inevitable in a book of this size and scope, when there is some attempt to make the various parts of the book self-contained.

Overall, “802.11 Wireless Networks: The Definitive Guide” lives up to its name and is highly recommended.

]]> http://www.nelug.org.uk/book-review-802-11-wireless-networks-the-definitive-guide-second-edition/feed/ 0 http://www.nelug.org.uk/net-booting-a-diskless-sun3-from-a-linux-server/ http://www.nelug.org.uk/net-booting-a-diskless-sun3-from-a-linux-server/#comments Wed, 01 Sep 2004 21:40:58 +0000 olly-bh http://www.nelug.org.uk/?p=89 Continue reading →]]> My system consists of a Linux server (basically Slackware-3.6 but with kernel 2.2.10 and various other irrelevant package upgrades) and 2 Sun3′s, only one of which is ever in use at any given time (thus they use the same swap partition on the server – later).

The Sun3′s are diskless and hence boot from the Linux box, and mount all filesystems from it. The Sun’s run NetBSD 1.3.2 largely because the Sun 3 port of Linux was not very stable or mature when I set things up.

OK, some basic info for future use:

Host	Hostname	Ethernet addr	IP addr
Linux PC	moriah	00:00:C0:8C:AB:2E	192.168.200.1
Sun 3/60	hermon	08:00:20:00:49:16	192.168.200.2
Sun 3/50	carmel	08:00:20:06:1f:50	192.168.200.3

The boot process for the Suns is basically as follows:

1. tell the machine to boot from the ethernet device
2. it gets its IP address by rarp
3. it uses tftp to get a 2nd stage boot program **whose name is the IP address of the client** from the server
4. the 2nd stage boot loader gets boot parameters from the bootp server
5. the boot parameters specify where the root filesystem, containing the kernel, is located.
6. The boot loader nfs-mounts this remote root f/s and boots the kernel.
7. Once it has its root f/s the boot process is pretty standard as per a diskful machine.

OK, in more detail:

Power on machine, hit L1-A if necessary to get a monitor prompt

Enter boot command, specifying ethernet interface as the boot device:

> b le(0,0,0)

First thing that happens is the machine broadcasts rarp requests saying “This is my ethernet addr, can somebody tell me my IP address please”.

My linux kernel is compiled with rarp support, and the kernel rarp table is populated at boot time by the following commands in rc.local:

rarp -s hermon  08:00:20:00:49:16
rarp -s carmel  08:00:20:06:1F:50

(/etc/hosts on moriah contains entries for hermon and carmel so the names are resolvable to numeric addresses)

(the forward-mapping arp table is also set up at boot time on moriah by "arp -f /etc/ethers" command – sadly rarp has no corresponding -foption so each entry must be added individually).

On receiving a response from moriah to the rarp request, the Sun contacts the tftp server on moriah and attempts to download the file /tftpboot/<my-IP-addr> from it. /tftpboot on moriah looks like:

moriah% ls -lF /tftpboot/
total 16
lrwxrwxrwx   1 root     root            7 Aug  3 23:00 C0A8C802 -> netboot*
lrwxrwxrwx   1 root     root            7 Aug  3 23:00 C0A8C803 -> netboot*
-rwxr-xr-x   1 root     root        15360 Aug  1 22:56 netboot*

where netboot is the NetBSD stage 2 boot program. IP addresses are in hex as you can see.

The tftp server in.tftpd runs from inetd on moriah – the relevant line from /etc/inetd.conf is:

moriah% grep tftp /etc/inetd.conf
tftp    dgram   udp     wait    nobody  /usr/sbin/in.tftpd in.tftpd

and from /etc/services:

moriah% grep tftp /etc/services
tftp            69/udp

Once dowloaded, the stage 2 boot program executes: its first task is to get its boot parameters from the bootp server. I had trouble getting the bootpd program which came with Slackware to work so I replaced it with bootparamd from the NetKit. bootparamd runs as a daemon, but could just as well run from inetd. It uses the file /etc/bootparams to hold boot config information:

moriah% cat /etc/bootparams
hermon  root=moriah:/export/root/hermon swap=moriah:/export/swap
carmel  root=moriah:/export/root/carmel swap=moriah:/export/swap

All that is specified in this case is the root and swap partitions for each of the 2 Sun’s – as I said they share a swap partition as only one is ever up at any one time.

From then on it’s plain sailing (you must of course have nfs working on the server so the clients can mount the filesystems). The boot loader mounts the root fs from the server, and loads the kernel (which in the case of NetBSD must be /netbsd – I only found this out by trial and error). The root fs in my case contains everything, but there’s no reason why you shouldn’t nfs mount other filesystems in the usual way of course once the kernel boots. Here’s the contents of /export/root/hermon, FWIW:

moriah% ls -lF /export/root/hermon
total 3955
drwxr-xr-x   2 root     root         1024 Jan  5  1998 altroot/
drwxr-xr-x   2 root     root         1024 Jan  5  1998 bin/
-rw-------   1 root     root       458752 Aug 10 19:26 core
drwxr-xr-x   3 root     root         5120 Aug 17 01:35 dev/
drwxr-xr-x   8 root     root         2048 Aug 14 15:26 etc/
-rw-------   1 root     root        16656 Aug 13 15:58 getty.core
drwxr-xr-x   3 root     root         1024 Aug 13 15:55 home/
drwxr-xr-x   2 root     root         1024 Dec  6  1997 mnt/
lrwxrwxrwx   1 root     root           13 Aug 14 05:56 netbsd -> netbsd.hermon*
-rw-r--r--   1 root     root      1025115 Aug  2 21:34 netbsd-gen
-rw-r--r--   1 root     root       769619 Aug  2 21:34 netbsd-inst
-rw-r--r--   1 root     root       983596 Aug  2 21:35 netbsd-rd
-rwxr-xr-x   1 root     root       750847 Aug 14 05:56 netbsd.hermon*
drwxr-xr-x   2 root     root         1024 Aug 10 00:21 proc/
drwxr-xr-x   3 root     root         1024 Aug 10 00:03 root/
drwxr-xr-x   2 root     root         2048 Jan  5  1998 sbin/
drwxr-xr-x   2 root     root         1024 Dec 19  1997 stand/
drwxr-xr-x   2 root     root         1024 Aug 13 16:06 swap/
lrwxrwxrwx   1 root     root           11 Aug 10 00:02 sys -> usr/src/sys/
drwxrwxrwt   3 root     root         1024 Aug 17 01:18 tmp/
drwxr-xr-x  15 root     root         1024 Aug 13 15:45 usr/
drwxr-xr-x  19 root     root         1024 Dec  6  1997 var/

There’s a few old kernels lying aroud, the one in use I configured and built for this machine. The only other thing that I had trouble figuring out was how to tell the Sun to swap on the server (on another partition). Here is /export/root/hermon/etc/fstab which specifies this (in this case no other f/s are mounted as the root contains everything):

moriah% cat /export/root/hermon/etc/fstab

moriah:/export/swap none swap sw,nfsmntpt=/swap

---

Eddy Younger <eddy@shofar.uklinux.net>

]]> http://www.nelug.org.uk/net-booting-a-diskless-sun3-from-a-linux-server/feed/ 0 http://www.nelug.org.uk/linux-firewalls/ http://www.nelug.org.uk/linux-firewalls/#comments Thu, 11 Jan 2001 23:40:59 +0000 olly-bh http://www.nelug.org.uk/?p=79 Continue reading →]]> Linux Firewalls

What is a firewall?

• A firewall is a controlled gateway between one network and another (i.e. an intranet and the internet).
• It is not a universal panacea for computer security. You must follow other good security practices.

Why Firewalls?

• You cannot trust everyone. Some people take pleasure in hacking into machines. Not all are malicious but some are!
• Your computer holds private/confidential data an you have a duty to protect it.
• You want to limit access from within your private network to specific external information/services (i.e. not mpeg3′s)
• You want to monitor/record traffic for audit/security purposes. Beware of privacy laws!

Types of Firewall

• Gateway based – you have to log onto a gateway machine which has restricted (if any) internal access and connect to external sites from the gateway.
• Service based – A variation on the gateway based approach is to restrict the services which are available on machines which are visible externally.
• Proxy based – provide external services via proxies which are accessible from within the private network. Only the proxies will have access to the external network.
• Packet based – packets filters can control which network packets are forwarded between networks and may make access decisions based on the contents of the packets.
• Masquerading – where all packets from a private network are rewritten in such a way that they appear to come from a single firewall (gateway) machine.

Gateway based

A gateway system operates by having a gateway machine which does not forward packets at all. The user must log onto the gateway machine and from there access external sites.

In this setup the gateway machine typically has a very restricted set of services which are necessary for accessing external sites. You will often find that it does not export or import filesystems (via NFS) and that many system commands have been removed to limit the range of tools that a hacker can exploit. It is also typical that filesystems are mounted read-only to stop any trojan horse based attacks etc.

Accounts are strictly controlled on gateway machines and may require explicit authorisation to use them. In addition to this password ageing may be enabled ensuring that passwords are changed on a regular basis.

Limited Service based

Limiting services revolves around configuring system daemons to ensure that only the necessary daemons/services are available. In particular you may choose not to run services which give away information about users and system configuration. For example:

• finger – the finger service can give information about who is logged into your system, when they last logged in and other bits of information which are useful to hackers.
• ident – information about users.
• tftpd – allows access to (certain) files without any form of authorisation.

These services/daemons are typically started at boot time or dynamically from inetd. The configuration/startup scripts are typically found in:

• /etc/inetd.conf
• /etc/rc.d

In general you should be very careful about giving external access to user daemons. Take special care over database daemons and similar user tools which may accept connections from anybody who cares to try.

tcpd

A simple but powerful method of protecting internet services is to use the tcpd wrapper program which is invoked in place of the normal service binary and first vets a connection to ensure that it is appropriate before invoking the normal binary with that connection. tcpd can only be used in place of daemons which have a one-to-one mapping between network connection and executable. In other words it is not possible to substitute a long running daemon (e.g. a database) with tcpd. The database must be configured correctly in this case.

hosts_access(5)

tcpd is primarily configured via the files /etc/hosts.allow and /etc/hosts.deny. These files contain a describe what action tcpd must take for specific services with connections coming from certains hosts/users.

An example which denies access to all hosts is

       /etc/hosts.deny:
          ALL: ALL

To allow authorised hosts to connect /etc/hosts.allow can be populated with entries similar to:

       /etc/hosts.allow:
          ALL: LOCAL @some_netgroup
          ALL: .foobar.edu EXCEPT terminalserver.foobar.edu

The full configuration language is much more expressive than that described above and involve scripts which could be used to perform a reverse search on the connecting host to find out more information. A full description of this can be found in the hosts_access manual page which is found in section 5 of the manuals.

tcpdchk

The program tcpdchk can be used to check the configuration of tcpd to ensure that mistakes are not present which may provide security problems. It even checks if tcpd protection is appropriate for a particular daemon (note this is heuristic based – it is not magic!).

Proxy based

Proxies are very popular and are typically used to limit network usage by caching frequently accessed information and thus reducing bandwidth requirements.

In addition to this use it is possible to use proxies as controlled firewall breaches. By doing this you can monitor external traffic (for security purposes of course) and provide hackers with a limited scope for attacking your system.

Popular proxies include:

• tis firewall toolkit (www.tis.com) – includes sendmail, ftp, telnet proxies (amongst others) and also provides a generic proxy which can be used for other daemons.
• socks – provides restricted access for many common tcp (and udp?) based services
• squid – web based cacheing

Packet based

A Packet based firewall involves selectively passing packets between different network interfaces based on the type of packet, source, destination and even the status of a connection.

Linux provides the IP chains facility (configured by the ipchains command) which allows a user to give the kernel about what should and should not be allowed between different interfaces.

A sample firewall configuration file is shown below. Note that this includes entries for IP masquerading.

Masquerading

An extension to packet based filtering is IP masquerading. Using this mechanism all external traffic has its IP headers rewritten in such a way that it appears that the packet has come from the gateway machine. For any external packets which are coming into the network the rewriting is reversed and the packet headers are rewritten to ensure that they go back to the appropriate internal machine. This hides all information about the internal network and makes it difficult for external sites to target specific machines within the network.

A common use for this technique is to allow an internal network which does not have proper addresses assigned to them to go via a single machine which does have an address assigned to it.

Sample IP chains configuration

#!/bin/sh
#
# Firewall rules.
# Invoked by rc.M after rc.inet?
#
# EJY 24/12/2000
#
# Flush all chains: start from a clean config.
/sbin/ipchains -F
/sbin/ipchains -X

# Create chain for input on ppp interface(s)
/sbin/ipchains -N ppp-in

# Default policy is deny on forward chain
/sbin/ipchains -P output ACCEPT 
/sbin/ipchains -P forward DENY 

# We trust anyone on the local ethernet,
# anything on ppp we don't
/sbin/ipchains -A input -i eth0 -j ACCEPT
/sbin/ipchains -A input -i ppp+ -j ppp-in

# Activate TOS mangling
/sbin/ipchains -A output -p tcp -d 0.0.0.0/0 telnet -t 0x01 0x10
/sbin/ipchains -A output -p tcp -d 0.0.0.0/0 ftp -t 0x01 0x10
/sbin/ipchains -A output -p tcp -s 0.0.0.0/0 ftp-data -t 0x01 0x08

# Deny spoofed addresses 
/sbin/ipchains -A ppp-in -j DENY -s 192.168.0.0/16
/sbin/ipchains -A ppp-in -j DENY -s 172.16.0.0/12
/sbin/ipchains -A ppp-in -j DENY -s 10.0.0.0/8

# Ditch multicast silently
/sbin/ipchains -A ppp-in -j DENY -d 224.0.0.0/4

# Do masquerading
/sbin/ipchains -A forward -j MASQ -s 192.168.200.0/24 -d ! 192.168.200.0/24

# Allow auth connections, as they're almost always requd.
/sbin/ipchains -A ppp-in -p tcp --destination-port auth -j ACCEPT

# 
# Log the skript kiddiez, just for fun...
#
/sbin/ipchains -A ppp-in -j DENY -l -p udp --destination-port 53
/sbin/ipchains -A ppp-in -j DENY -l -p udp --destination-port ftp
/sbin/ipchains -A ppp-in -j DENY -l -p tcp --destination-port 53
/sbin/ipchains -A ppp-in -j DENY -l -p tcp --destination-port ftp

# Deny all priveleged ports (previous rules have precedence ...
/sbin/ipchains -A ppp-in -p tcp --destination-port 1:1023 -j DENY
/sbin/ipchains -A ppp-in -p udp --destination-port 1:1023 -j DENY

# Block certain high-number ports, with logging ...

# Block NFS-server
/sbin/ipchains -A ppp-in -j DENY -l -p tcp -i  ppp0 --destination-port 2049
/sbin/ipchains -A ppp-in -j DENY -l -p udp -i  ppp0 --destination-port 2049

# Block X-server
/sbin/ipchains -A ppp-in -j DENY -l -p tcp -i  ppp0 --destination-port 6000:6016
/sbin/ipchains -A ppp-in -j DENY -l -p udp -i  ppp0 --destination-port 6000:6016

# Block IJB and Squid
/sbin/ipchains -A ppp-in -j DENY -l -p tcp -i  ppp0 --destination-port 8000
/sbin/ipchains -A ppp-in -j DENY -l -p udp -i  ppp0 --destination-port 8000
/sbin/ipchains -A ppp-in -j DENY -l -p tcp -i  ppp0 --destination-port 8080
/sbin/ipchains -A ppp-in -j DENY -l -p udp -i  ppp0 --destination-port 8080
#
# Block various known trojans
#
# ipchains -A ppp-in -p tcp --destination-port asp -j DENY -l
# ipchains -A ppp-in -p udp --destination-port asp -j DENY -l
ipchains -A ppp-in -p tcp --destination-port 31337 -j DENY -l
ipchains -A ppp-in -p udp --destination-port 31337 -j DENY -l
ipchains -A ppp-in -p tcp --destination-port 1243 -j DENY -l
ipchains -A ppp-in -p udp --destination-port 1243 -j DENY -l

# Allow remaining high-numbered ports ...
/sbin/ipchains -A ppp-in -p tcp --destination-port 1024: -j ACCEPT
/sbin/ipchains -A ppp-in -p udp --destination-port 1024: -j ACCEPT

A more complex ipchains example

This configuration was produced automatically by Robert Ziegler’s wonderful Linux firewall design tool

#!/bin/sh

# ----------------------------------------------------------------------------
# Copyright (C) 1997, 1998, 1999, 2000  Robert L. Ziegler
#
#  Permission to use, copy, modify, and distribute this software and its
#  documentation for educational, research, private and non-profit purposes,
#  without fee, and without a written agreement is hereby granted. 
#  This software is provided as an example and basis for individual firewall
#  development.  This software is provided without warranty.
#
#  Any material furnished by Robert L. Ziegler is furnished on an 
#  "as is" basis.  He makes no warranties of any kind, either expressed 
#  or implied as to any matter including, but not limited to, warranty 
#  of fitness for a particular purpose, exclusivity or results obtained
#  from use of the material.
# ----------------------------------------------------------------------------

#  /etc/rc.d/rc.firewall
#  Invoked from /etc/ppp/ip-up, or
#  from /sbin/ifup-local, or
#  from /etc/sysconfig/network-scripts/ifup-post.

echo "Starting firewalling... "

# ----------------------------------------------------------------------------
#  Some definitions for easy maintenance.
#  EDIT THESE TO SUIT YOUR SYSTEM AND ISP.

EXTERNAL_INTERFACE="ppp0"               # Internet connected interface
LOOPBACK_INTERFACE="lo"                 # or your local naming convention
LOCAL_INTERFACE_1="eth0"                # internal LAN interface

IPADDR=$(/sbin/ifconfig | /bin/grep P-t-P | /usr/bin/cut -c 21-35)
LOCALNET_1="192.168.1.0/24"             # whatever private range you use

ANYWHERE="any/0"                        # match any IP address

DHCP_SERVER="any/0"
NAMESERVER_1="any/0"                    # everyone must have at least one

SMTP_SERVER="any/0"                     # Your ISP mail gateway. Your relay.
POP_SERVER="any/0"              # Your ISP pop mail server.
IMAP_SERVER="any/0"             # Your ISP imap mail server.

LOOPBACK="127.0.0.0/8"                  # reserved loopback address range
CLASS_A="10.0.0.0/8"                    # class A private networks
CLASS_B="172.16.0.0/12"                 # class B private networks
CLASS_C="192.168.0.0/16"                # class C private networks
BROADCAST_SRC="0.0.0.0"                 # broadcast source address
BROADCAST_DEST="255.255.255.255"        # broadcast destination address
PRIVPORTS="0:1023"                      # well known, privileged port range
UNPRIVPORTS="1024:65535"                # unprivileged port range

# ----------------------------------------------------------------------------

NFS_PORT="2049"                         # (TCP/UDP) NFS
SOCKS_PORT="1080"                       # (TCP) Socks
OPENWINDOWS_PORT="2000"                 # (TCP) openwindows

# X Windows port allocation begins at 6000 and increments to 6063
# for each additional server running.
XWINDOW_PORTS="6000:6063"               # (TCP) X windows

# traceroute usually uses -S 32769:65535 -D 33434:33523
TRACEROUTE_SRC_PORTS="32769:65535"
TRACEROUTE_DEST_PORTS="33434:33523"

# ----------------------------------------------------------------------------
# Default policy is DENY
# Explicitly accept desired INCOMING & OUTGOING connections

    # Remove all existing rules belonging to this filter
    ipchains -F

    # Set the default policy of the filter to deny.
    ipchains -P input  DENY
    ipchains -P output REJECT
    ipchains -P forward REJECT

    # set masquerade timeout to 10 hours for tcp connections
    ipchains -M -S 36000 0 0


# ----------------------------------------------------------------------------

    # Enable IP Forwarding, if it isn't already
    echo 1 > /proc/sys/net/ipv4/ip_forward

    # Enable TCP SYN Cookie Protection
    echo 1 > /proc/sys/net/ipv4/tcp_syncookies

    # Enable always defragging Protection
    echo 1 > /proc/sys/net/ipv4/ip_always_defrag

    # Enable broadcast echo  Protection
    echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

    # Enable bad error message  Protection
    echo 1 > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses

    # Enable IP spoofing protection
    # turn on Source Address Verification
    for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
        echo 1 > $f
    done

    # Disable ICMP Redirect Acceptance
    for f in /proc/sys/net/ipv4/conf/*/accept_redirects; do
        echo 0 > $f
    done

    # Disable Source Routed Packets
    for f in /proc/sys/net/ipv4/conf/*/accept_source_route; do
        echo 0 > $f
    done

    # Log Spoofed Packets, Source Routed Packets, Redirect Packets
    for f in /proc/sys/net/ipv4/conf/*/log_martians; do
        echo 1 > $f
    done


    # These modules are necessary to masquerade their respective services.
    /sbin/modprobe ip_masq_ftp

# ----------------------------------------------------------------------------
# LOOPBACK

    # Unlimited traffic on the loopback interface.

    ipchains -A input  -i $LOOPBACK_INTERFACE  -j ACCEPT 
    ipchains -A output -i $LOOPBACK_INTERFACE  -j ACCEPT 

# ----------------------------------------------------------------------------
# Unlimited traffic within the local network.

    # All internal machines have access to the fireall machine.

    ipchains -A input  -i $LOCAL_INTERFACE_1 -s $LOCALNET_1 -j ACCEPT 
    ipchains -A output -i $LOCAL_INTERFACE_1 -d $LOCALNET_1 -j ACCEPT 

# ----------------------------------------------------------------------------
# Masquerade internal traffic.

    # All internal traffic is masqueraded externally.
    ipchains -A forward -i $EXTERNAL_INTERFACE -s $LOCALNET_1 -j MASQ

# ----------------------------------------------------------------------------
# SPOOFING & BAD ADDRESSES
# Refuse spoofed packets.
# Ignore blatantly illegal source addresses.
# Protect yourself from sending to bad addresses.

    # Refuse incoming packets pretending to be from the external address.
    ipchains -A input  -i $EXTERNAL_INTERFACE  -s $IPADDR -j DENY -l

# ----------------------------------------------------------------------------
# NOTE:
#      The symbolic names used in /etc/services for the port numbers vary by
#      supplier.  Using them is less error prone and more meaningful, though.

# ----------------------------------------------------------------------------
# TCP UNPRIVILEGED PORTS
# Avoid ports subject to protocol & system administration problems.

    # NFS: establishing a TCP connection
    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $NFS_PORT -j DENY -l
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $NFS_PORT -j REJECT 

    # openwindows: establishing a connection
    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $OPENWINDOWS_PORT -j DENY -l
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $OPENWINDOWS_PORT -j REJECT 


    # Xwindows: establishing a connection
    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $XWINDOW_PORTS -j DENY -l
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $XWINDOW_PORTS -j REJECT 

    # SOCKS: establishing a connection
    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $SOCKS_PORT -j DENY -l
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp -y \
             --destination-port $SOCKS_PORT -j REJECT 

# ----------------------------------------------------------------------------
# UDP UNPRIVILEGED PORTS
# Avoid ports subject to protocol & system administration problems.

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             --destination-port $NFS_PORT -j DENY -l

    # UDP INCOMING TRACEROUTE
    # traceroute usually uses -S 32769:65535 -D 33434:33523

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             --source-port $TRACEROUTE_SRC_PORTS \
             --destination-port $TRACEROUTE_DEST_PORTS -j DENY -l

# ----------------------------------------------------------------------------

    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp ! -y \
             -d $IPADDR -j ACCEPT 

    # ------------------------------------------------------------------

    # DNS server (53)
    # ---------------

    # DNS forward-only nameserver
    # ---------------------------

    # forward-only can use regular TCP protocol to forwarders

    ipchains -A output -i $EXTERNAL_INTERFACE -p udp  \
             -s $IPADDR 53 \
             -d $NAMESERVER_1 53 -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             -s $NAMESERVER_1 53 \
             -d $IPADDR 53 -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             -d $NAMESERVER_1 53 -j ACCEPT 

    # ------------------------------------------------------------------

    # HTTP client (80)
    # ----------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 80 -j ACCEPT 

    # ------------------------------------------------------------------

    # HTTPS client (443)
    # ------------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 443 -j ACCEPT 

    # ------------------------------------------------------------------

    # POP client (110)
    # ----------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 110 -j ACCEPT 

    # ------------------------------------------------------------------

    # IMAP client (143)
    # -----------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             -d $IMAP_SERVER 143 -j ACCEPT 

    # ------------------------------------------------------------------

    # SMTP client (25)
    # ----------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             -d $SMTP_SERVER 25 -j ACCEPT 

    # ------------------------------------------------------------------

    # TELNET client (23)
    # ------------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 23 -j ACCEPT 

    # ------------------------------------------------------------------

    # AUTH server (113)
    # -----------------

    # Reject, rather than deny, the incoming auth port. (NET-3-HOWTO)
    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp  \
             --source-port $UNPRIVPORTS \
             -d $IPADDR 113 -j REJECT 


    # AUTH client (113)
    # -----------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 113 -j ACCEPT 

    # ------------------------------------------------------------------

    # WHOIS client (43)
    # -----------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 43 -j ACCEPT 

    # ------------------------------------------------------------------

    # FINGER client (79)
    # ------------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 79 -j ACCEPT 

    # ------------------------------------------------------------------

    # FTP client (21)
    # ---------------

    # outgoing request
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 21 -j ACCEPT 


    # PORT mode data channel
    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp  \
             --source-port 20 \
             -d $IPADDR $UNPRIVPORTS -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp ! -y \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port 20 -j ACCEPT 


    # PASSIVE mode data channel creation
    ipchains -A output -i $EXTERNAL_INTERFACE -p tcp  \
             -s $IPADDR $UNPRIVPORTS \
             --destination-port $UNPRIVPORTS -j ACCEPT 

# ----------------------------------------------------------------------------
# UDP accept only on selected ports
# ---------------------------------


    # DHCP client (67, 68)
    # --------------------

    # allow dhcp server (67) to connect to dhcp client (68)
    # Note: the DHCP server is the only externel source of broadcast
    #       messages we should see, ever.

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             -s $DHCP_SERVER 67 \
             -d $IPADDR 68 -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p udp  \
             -s $IPADDR 68 \
             -d $DHCP_SERVER 67 -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             -s $DHCP_SERVER 67 \
             -d $BROADCAST_DEST 68 -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p udp  \
             -s $BROADCAST_SRC 68 \
             -d $DHCP_SERVER 67 -j ACCEPT 

    # Getting renumbered
    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             -s $BROADCAST_SRC 67 \
             -d $BROADCAST_DEST 68 -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p udp  \
             -s $BROADCAST_SRC 68 \
             -d $BROADCAST_DEST 67 -j ACCEPT 

    # As a result of the above, we're supposed to change our IP address with
    # this message, which is addressed to our new address before the dhcp
    # client has received the update.

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             -s $DHCP_SERVER 67 \
             --destination-port 68 -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             --source-port 67 \
             -d $IPADDR 68 -j DENY -l

    # ------------------------------------------------------------------

    # OUTGOING TRACEROUTE
    # -------------------
    ipchains -A output -i $EXTERNAL_INTERFACE -p udp  \
             -s $IPADDR $TRACEROUTE_SRC_PORTS \
             --destination-port $TRACEROUTE_DEST_PORTS -j ACCEPT -l

# ----------------------------------------------------------------------------
# ICMP

    #    To prevent denial of service attacks based on ICMP bombs, filter
    #    incoming Redirect (5) and outgoing Destination Unreachable (3).
    #    Note, however, disabling Destination Unreachable (3) is not
    #    advisable, as it is used to negotiate packet fragment size.

    # For bi-directional ping.
    #     Message Types:  Echo_Reply (0),  Echo_Request (8)
    #     To prevent attacks, limit the src addresses to your ISP range.
    # 
    # For outgoing traceroute.
    #     Message Types:  INCOMING Dest_Unreachable (3), Time_Exceeded (11)
    #     default UDP base: 33434 to base+nhops-1
    # 
    # For incoming traceroute.
    #     Message Types:  OUTGOING Dest_Unreachable (3), Time_Exceeded (11)
    #     To block this, deny OUTGOING 3 and 11

    #  0: echo-reply (pong)
    #  3: destination-unreachable, port-unreachable, fragmentation-needed, etc.
    #  4: source-quench
    #  5: redirect
    #  8: echo-request (ping)
    # 11: time-exceeded
    # 12: parameter-problem

    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type echo-reply \
             -d $IPADDR -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type destination-unreachable \
             -d $IPADDR -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type source-quench \
             -d $IPADDR -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type echo-request \
             -d $IPADDR -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type time-exceeded \
             -d $IPADDR -j ACCEPT 

    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type parameter-problem \
             -d $IPADDR -j ACCEPT 


    ipchains -A output -i $EXTERNAL_INTERFACE -p icmp  \
             -s $IPADDR echo-reply -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p icmp  \
             -s $IPADDR fragmentation-needed -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p icmp  \
             -s $IPADDR source-quench -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p icmp  \
             -s $IPADDR echo-request -j ACCEPT 

    ipchains -A output -i $EXTERNAL_INTERFACE -p icmp  \
             -s $IPADDR parameter-problem -j ACCEPT 

# ----------------------------------------------------------------------------
# Enable logging for selected denied packets

    ipchains -A input  -i $EXTERNAL_INTERFACE -p tcp  -j DENY -l

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             --destination-port $PRIVPORTS -j DENY -l

    ipchains -A input  -i $EXTERNAL_INTERFACE -p udp  \
             --destination-port $UNPRIVPORTS -j DENY -l


    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type 5 -j DENY -l
    ipchains -A input  -i $EXTERNAL_INTERFACE -p icmp  \
             --icmp-type 13:255 -j DENY -l

    ipchains -A output -i $EXTERNAL_INTERFACE  -j REJECT -l

# ----------------------------------------------------------------------------

echo "done"

exit 0

---

Richard Mortimer

Last modified: Thu Jan 11 23:40:46 GMT 2001

]]> http://www.nelug.org.uk/linux-firewalls/feed/ 1 http://www.nelug.org.uk/configuring-email-systems/ http://www.nelug.org.uk/configuring-email-systems/#comments Wed, 07 Jun 2000 06:09:55 +0000 admin http://www.nelug.org.uk/?p=51 Continue reading →]]> NELUG, 7/6/2000

Eddy Younger (eddy@shofar.uklinux.net)

Software Components of the Email System

There are principally three classes of software components involved in transfering a mail message from the sender to the recipient:

1. MUA – mail user agent, used to read, compose and post mail.
2. MTA – mail transport agent. MTA’s at the source and destination hosts (and possibly also intermediate hosts) pass the messages from one to another
3. MDA – mail delivery agent. At the ultimate destination host, the MDA receives the message from the local MTA and delivers it to its ultimate destination, usually the recipient’s mailbox file.

In the modern world almost all email transport is achieved using SMTP – the Simple Mail Transport Protocol – or its Extended variant ESMTP. MTA’s speak to each other in (E)SMTP. You can send mail without using a MUA if you wanted to, by talking SMTP directly to the MTA, and it used to be possible to do all kinds of nefarious things by doing so, though thankfully most mail servers are much more secure in these days of Skript Kiddies and Spam.

MUA’s

Reading Mail:

Commonly there are three methods to access the mailbox:

1. read the mailbox directly (/bin/mail, most Unix mail clients);
2. POP3 – Post-Office Protocol. Mail reader communicates with POP3 server which accesses the mailbox and transfers messages to the mail reader, which re-files them locally;
3. IMAP – another client/server system. The server is capable of transfering header information to the client without the message body, and of deleting messages from the mailbox on behalf of the client. The messages need only be uploaded to be read or filed.

Modern Unix mail clients (pine, mutt, emacs VM) can handle POP, IMAP or a local mailbox. Others such as Netscape need to talk to a POP or IMAP server, even if the mailbox is local.

Sending mail:

MUA hands mail on to a MTA. This may be on the local host or on a remote host (maybe at your ISP), depending upon which MUA you’re using and how it’s configured. The MUA generally uses SMTP to talk to the MTA; in rarer (rarer on Unix, that is) case POP may be used to post mail as well as to read it.

MTA’s

Sendmail is the most widely used, most powerful, probably the most efficient, and (formerly) the hardest to configure. It originates from a time when memory and CPU bandwidth were expensive, and when there were many and varied networks other than the Internet (BITNET, EARN, JANET, UUCP …). Sendmail is capable of acting as a gateway between such different networks, as a mail-to-fax or mail-to usenet gateway, etc. It is very resource-efficient. Virtually every piece of mail which moves on the backbone is handled by sendmail at some point (chalk up one more for Open Source Software).

Since the virtual demise of all other networks, numerous other MTA’s with more limited capabilities have come along, e.g Exim, qmail, Postfix. These tend to lack the gateway capabilities of sendmail, for example, but as a result are simpler, and regarded by some as more secure. A major selling point seems to be that their configuration files are more readable than sendmail’s, which is undoubtably true but largely irrelevant as we shall see. Some very large sites are now running on these alternative MTA’s however – Freeserve runs on Exim for example.

MDA’s

The role of the MDA is to accept a message from the MTA on the final destination host and deliver it to its destination, usually a user’s mailbox. On Unix systems (ATT variants) the MDA for mail to a user’s mailbox is usually /bin/mail (though interestingly the MDA in the default sendmail.cf on Slackware Linux is procmail). For mail to a program (where allowed) it is /bin/sh.

The obligatory diagram




This picture may be complicated by

1. DNS MX records for the destination
2. Aliases, either at the source or destination

MX records

Once the MTA has deciphered the destination address, it does a name-server lookup (usually DNS, but maybe NIS or whatever) to find it’s IP address. MX records are special DNS records used by the mail system: a destination (e.g durham.ac.uk) may one or more MX records associated with it which point to the IP adresses of hosts which will recieve mail on behalf of that destination. In such cases, the destination may be a host which is incapable of receiving mail (no MTA), or may not exist as a physical machine at all (e.g. durham.ac.uk or shofar.uklinux.net). The MX records for these destination point to a real machine to which mail will actually be routed (a destination may have several MX records, possibly with different preference values).

Aliases

There are two classes of aliases:

1. system wide, defined in /etc/aliases or the mail.aliases nis map; these are expanded by the MTA
2. user’s aliases, usually defined in ~/.mailrc; these are expanded by the MUA when the message is posted

Configuring sendmail

The main configuration file for sendmail, by default /etc/sendmail.cf, is large and difficult to read even in the simplest configuration. For this reason sendmail has a reputation for being hard to configure. However, nowadays sendmail.cf files may be generated using the m4 macro processor from a source file of typically less than a dozen lines by calling out the pre-defined macros which form part of the sendmail distribution. Directly editing sendmail.cf is only necessary in unusual configurations.

sendmail.cf files contain a set of definitions (which essentially set values for macros used elsewhere) and rewriting rules which rewrite the headers of mail messages – most particularly the address portions – before delivering them to the destination (or next) MTA. The rules consist of a LHS which is a pattern to match against the input, and a RHS which defines how a matching input should be re-written. A rule is applied recursively until the match fails (or it is explicitly terminated), and rules may “call” other rules. The “entry point” is rule 3, confusingly enough.

You can now configure sendmail without knowing or understanding anything about the syntax of sendmail.cf

Configuring email on a dial-up host

There are two basic approaches, with various possible points in between:

Local MUA only

In this scenario, there is no mail service at all on the local machine. A mail client is used which can download mail from the dial-up service and can hand over sent mail directly to the MTA on the server. This is the approach taken by ISP’s in the software they supply to end users. This will work fine if you don’t need anything sophisticated like multi-drop mailbox capabilities, or to provide mail services to a lan through a dial-up gateway (though even that may be possible up to a point). Given the capabilities of a Linux box it’s pretty uninspiring though.

A real email service

As usual, there’s more than one way of doing it. We have to cope with the fact that the connection from the server to the internet (the dial-up connection) comes and goes. We will also usually need to deal with the fact that

1. we only have one mailbox at the ISP into which all mail for <any_user>@<ourname>.isp.co.uk is delivered.
2. we have to collect email via POP or IMAP, and not SMTP through our local MTA.

What follows is a case study based on my own setup, where one machine acts as a mail hub and also a dial-up gateway for a local area network. Sending mail to the outside world, and collecting mail from two separate ISP’s, are dealt with separately owing to the constraints of ISP’s and dial-up links.

Sending mail

Sending of mail is dealt with by MTA’s (in this case, sendmail) running on the LAN clients and the mail hub. Mail clients (MUA’s) hand over mail to the MTA on the local machine (in the case of Netscape mail and friends they could hand it straight to the mail hub). Sendmail on the clients uses the dumbest possible configuration, known as the “nullclient” configuration after the m4 macro used to generate it:

include(`../m4/cf.m4')                                 include the boilerplate 
                                                       sendmail.cf content
VERSIONID(`@(#)generic-shofar.mc 8.9 (Berkeley)')      gets included as a comment 
                                                       in the .cf file
OSTYPE(linux)dnl                                       tells m4 which definitions 
                                                       to use for platform-specific 
                                                       paths, etc
DOMAIN(generic)dnl                                     include vanilla domain-
                                                       specific definitions
FEATURE(nullclient,moriah)                             hand over all mail to the 
                                                       MTA on the machine called
                                                       "moriah"

m4 configuration file for dumb lan clients

This generates a sendmail.cf file of 642 lines. The only support this provides is for forwarding all mail to the named mail hub. No aliases are expanded on the originating host, nor are .forward files processed – this implies that user acounts should be visible on the mail hub if .forward files are to be used.

The sendmail configuration on the mail hub needs to be more complicated. It’s major job is to rewrite senders’ mail addresses to be that of the ISP mailbox, in order that mail can be replied-to, and also to prevent anti-spam and anti-spoofing rules from rejecting the message. Well-configured MTA’s will refuse to accept messages whose “From:” address does not resolve to a valid host or MX record – this presents a problem for hosts or LAN’s with only dial-up connections, which should be using network addresses from the defined set of private ranges, and whose local host/domain names are not resolvable out in the real world. The second (though optional) function of the mail hub’s configuration is to define a “smart host” within the ISP to which all outgoing mail is passed. This has the advantage that if the ultimate destination of a message is temporarilly unavailable, the smarthost and not our mail hub will take care of retrying the delivery. It also means that if the link to the destination is very slow we don’t have to stay connected until the delivery is complete – the smarthost will accept the mail and we can hang up the phone and allow it to take care of the delivery. The disadvantage is that if delivery fails we won’t find out until we next connect to dowload our mail. Note that some ISP’s (e.g. Freeserve) interpose a smarthost whether you ask for one or not.

include(`../m4/cf.m4')
VERSIONID(`@(#)mailhub-shofar.mc 8.8 (Berkeley)5/19/98')
OSTYPE(linux)dnl
DOMAIN(generic)dnl
define(SMART_HOST,smtp:mail.uklinux.net)          define the smart host and the 
                                                  mailer thru' which to access it
MASQUERADE_AS(shofar.uklinux.net)dnl              From: and Reply-To: addresses must 
                                                  be rewritten as this
MASQUERADE_DOMAIN(shofar.org.uk)dnl               all addresses in this domain are to 
                                                  be masqueraded as above
FEATURE(allmasquerade)dnl                         causes local aliases and To: 
                                                  addresses in above domain to 
                                                  be masqueraded
FEATURE(masquerade_entire_domain)dnl              rewrite <any_host>.shofar.org.uk to 
                                                  be shofar.uklinux.net
FEATURE(masquerade_envelope)dnl                   masquerade the envelope as well as 
                                                  the From: line (see below)
MAILER(local)dnl                                  mail will either be delivered to a 
                                                  local MDA
MAILER(smtp)dnl                                   or transferred via SMTP

m4 configuration file for dial-up mail hub

Most of the masquerade features in this configuration are the result of the fact that I am using a bogus domain name (shofar.org.uk) on the local LAN. We want to be able to send mail to aliases defined locally (e.g. NELUG-announce) and have that rewritten as <local_alias_name>@shofar.uklinux.net ( hence FEATURE(allmasquerade) ), so that recipients can “reply-to-all” and it will work (but probably not the way they expect). We also want mail from any host on the LAN to have its address masqueraded ( FEATURE(masquerade_entire_domain) ). Most importantly, we want the envelope From address (not just the From and Reply_To in the header) to be masqueraded, as this is the address used to

1. return bounces; and
2. check for spoofed sender-addresses

It is therefore imperative that the envelope From address is resolvable to a valid address, which “eddy@shofar.org.uk” definitely is not (outside my private LAN).

Thus, all mail originating in the LAN is passed to sendmail on the mail hub, who will attempt to transfer the messages to the smarthost. If this fails, either because the dialup link isn’t up or the smarthost is broken, sendmail places the messages in a queue (by default /var/spool/mqueue), and will retry each message at intervals (the -q commabd-line parameter), warn the sender after a specified (in sendmail.cf) period, and give up and return the mail if a time limit (you guessed…) is exceeded. Because it is more usual for the link to be down than up, we don’t want to rely on sendmail happening to retry at a time when the link is actually there. So, one thing that the ip-up script does (whenever the link comes up, remember…) is to run “sendmail -q” which causes an instance of sendmail to start, process all queued message once, then terminate. Provided the smarthost isn’t broken, this will cause the queue to be flushed.

Generating the .cf file

Simplicity itself. Assuming you’ve created the m4 file and named it “<hostname>.mc” – a good idea if you’re going to produce config. files for several hosts – in the cf/cf directory of the sendmail distribution, simply “make <hostname>.cf” in the same directory, and your half-dozen lines of m4 source will be translated into many hundred lines of mystic runes. Copy the resulting “host.cf” to /etc/sendmail.cf on the target machine, kill and restart sendmail if it’s running in daemon mode. No, you don’t need to reboot, and you don’t need to insert the distribution cd several times over.

Receiving mail

To receive mail, we want to poll the mail services at two different ISP’s, which support only POP for mail retrieval, and deliver the messages to the appropriate mailboxes on the local mail hub. Client machines on the LAN all mount the mail spool directory (/var/mail) via NFS and so have direct access to the mailboxes. To provide for users of Netscape mail (or Windoze mail clients) we’d need to run a POP server on the mail hub, but up with that I no longer have to put.

The ISP mailboxes are multi-drop, i.e. if I requested an email address of “eddy@shofar.uklinux.net”, then “<any_user_name_at_all>@shofar.uklinux.net”, is a valid email address, and the mail will end up in the same mailbox at the ISP.

To collect mail, we use Eric Raymond’s most excellent fetchmail program, which deals quite nicely with all these requirements. It can be configured to poll a number of mailboxes (using POP3 or IMAP), retrieve the mail and forward via sendmail to either a single local user, or to map user-names in the recipient address of the message on a one-to-one basis to users on the local system and forward mail accordingly. User names in the mail address need not be the same as those on the local mail hub – fetchmail takes care of the mapping.

Fetchmail configuration

Fetchmail is run from the ip-up script whenever the dial-up link comes up; it is configured to poll the mailboxes at 5 minute intervals while the link remains up. The configuration file is .fetchmailrc (in root’s home directory since fetchmail runs as root from ip-up). In order for it to run without intervention, the .fetchmailrc file contains passwords to log in to the POP3 accounts, which represents a security risk in some environments despite it being unreadable except by root.

# Configuration created 11th May 2000 by hand
#
set syslog                        #error logging via syslog
set postmaster "postmaster"       #last-resort recipient if mappings fail
set no bouncemail                 #don't bounce mail to sender on error
#
# "Single-drop" ISP mailbox
#
poll pop.freeserve.net with proto POP3
       user "shofar.freeserve.co.uk" there with password "**********" 
       is eddy here options flush 
#
# "Multi-drop" ISP mailbox
#
poll mail.uklinux.net with proto POP3 aka shofar.uklinux.net
       user "shofar" with password "**********" to eddy nelug=eddy NELUG-admin 
nelug-approval=eddy NELUG-list root keith here options flush

.fetchmailrc for dial-up mail hub

The first mailbox (pop.freeserve.net) is a simple case where we want all mail to be delivered to a single user on the local machine, regardless of what the username in the mail address might be. In the second case, the “to” keyword defines this to be a multidrop mailbox, and we list recipient names corresponding to local user names: remote-name=local-name specifies a mapping from the name in the mail address to a local user name; names without a corresponding “=local-name” mean that this name in the mail address should be mapped to the same local user name (equivalent for example to “eddy=eddy”, “NELUG-admin=NELUG-admin” in the above). Any mail address containing a username which is not listed will be delivered to the username specified in the postmaster variable set in the preamble (in this case “postmaster”).

In the second case, fetchmail will actively parse the mail headers in order to determine who the mail was addressed to (unlike the first case where it doesn’t care, it just forwards everything to “eddy”). For this reason, we need to specify using the “aka” keyword the host part of the mail address we should expect to find, as this is different from the hostname of the machine we are retrieving the mail from.

Debugging

Sendmail (and fetchmail if you “set syslog”) log messages via syslog to say what their doing. Sendmail in particular logs messages about every mail message it processes.

Example:

Jun  6 18:13:42 moriah fetchmail[295]: POP3> TOP 3 99999999 
Jun  6 18:13:42 moriah fetchmail[295]: POP3< +OK Top of message follows 
Jun  6 18:13:42 moriah fetchmail[295]: reading message 3 of 4 (3512 octets) 
Jun  6 18:13:42 moriah fetchmail[295]: SMTP> MAIL FROM:<ey86798@herring.uk.sun.c
om> BODY=7BIT SIZE=3512 
Jun  6 18:13:42 moriah fetchmail[295]: SMTP< 250 <ey86798@herring.uk.sun.com>...
 Sender ok 
Jun  6 18:13:42 moriah fetchmail[295]: SMTP> RCPT TO:<eddy@localhost>
Jun  6 18:13:42 moriah fetchmail[295]: SMTP< 250 <eddy@localhost>... Recipient o
k 
Jun  6 18:13:42 moriah fetchmail[295]: SMTP> DATA 
Jun  6 18:13:42 moriah fetchmail[295]: SMTP< 354 Enter mail, end with "." on a l
ine by itself 
Jun  6 18:13:43 moriah fetchmail[295]: SMTP>. (EOM) 
Jun  6 18:13:43 moriah sendmail[341]: SAA00341: from=<ey86798@herring.uk.sun.com
>, size=3530,, pri=33530, nrcpts=1, msgid=<14652.56766.948321.273973@her
ring>, bodytype=7BIT, proto=ESMTP, relay=root@localhost [127.0.0.1]
Jun  6 18:13:43 moriah fetchmail[295]: SMTP< 250 SAA00341 Message accepted for d
elivery 
Jun  6 18:13:43 moriah fetchmail[295]:  flushed 
Jun  6 18:13:43 moriah fetchmail[295]: POP3> DELE 3^M 
Jun  6 18:13:44 moriah fetchmail[295]: POP3< +OK Message deleted 
Jun  6 18:13:44 moriah sendmail[342]: SAA00341: to=<eddy@localhost>, delay=00:00
:02, xdelay=00:00:01, mailer=local, stat=Sent
Jun  6 18:13:44 moriah sendmail[335]: SAA00334: to=eddy             , delay=00:0
0:06, xdelay=00:00:05, mailer=local, stat=Sent

This shows fetchmail retrieving a message via POP3, forwarding it to sendmail for delivery, and subsequently deleting the message from the POP server. It also shows the messages logged by sendmail on receipt and delivery of the message (the two sets of messages are interleaved in the file – a not uncommon occurrence in log files).

Sendmail may also be run from the command line is debugging modes to allow one to see exactly how it’s processing addresses, for example to produce a trace of the rewrite rules as they fire, and to trace the rewriting of the address. See the bat book or the red book for details of debugging flags.

The mail headers themselves contain valuable information which can help debug bounces or mail forwarding loops. Each MTA adds its own “Received…” line as it processes the message. For sendmail, the format of the Received: line is defined in sendmail.cf.

Example:

From paulmidgley@freeuk.com  Sat Jun  3 20:58:33 2000
Received: from localhost (root@localhost [127.0.0.1])
        by moriah.shofar.org.uk (8.9.3/8.9.1) with ESMTP id UAA00617
        for <eddy@localhost>; Sat, 3 Jun 2000 20:58:31 +0100
Received: from mail.uklinux.net
        by localhost with POP3 (fetchmail-5.1.2)
        for eddy@localhost (multi-drop); Sat, 03 Jun 2000 20:58:32 +0100 (BST)
Received: from scrabble.freeuk.net (scrabble.freeuk.net [212.126.144.6])
        by www.uklinux.net (8.9.3/8.8.7) with ESMTP id SAA09791
        for <eddy@shofar.uklinux.net>; Sat, 3 Jun 2000 18:26:30 +0100
Received: from [212.126.152.202] (helo=z4t4z8)
        by scrabble.freeuk.net with smtp (Exim 3.12 #1)
        id 12yHgW-00022v-00
        for eddy@shofar.uklinux.net; Sat, 03 Jun 2000 18:26:08 +0100
Message-ID: <001301bfcd7f$fb61e220$ca987ed4@z4t4z8>
MIME-Version: 1.0
Content-Type: text/plain;
        charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Lookout Excess 5.00.2615.200
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
From: "Paul Midgley" <paulmidgley@freeuk.com>
To: "Eddy Younger" <eddy@shofar.uklinux.net>
Subject: Internet access
Date: Sat, 3 Jun 2000 18:18:25 +0100

Here we can see the route the message took from sender to recipient and the MTA’s involved at each stage. The message was generated by something called “Microsoft Outlook (??)”. It was transferred first of all to scrabble.freeuk.net which was running Exim 3.12. scrabble.freeuk.net passed the message to www.uklinux.net, which is running sendmail v8.9.3: this is despite the fact that the message was addressed to shofar.uklinux.net – this probably indicates that there is an MX record for shofar.uklinux.net pointing to www.uklinux.net. (The version numbers in parentheses are sendmail-version/config-file-version. config-file-version is actually the value of the macro Z defined in the config file). The message has now reached its destination according to the address in the envelope, but this is not the end of its travels, as fetchmail now comes along and removes it from the mailbox, rewrites its envelope address and re-inserts it into the email system by passing it to sendmail (v 8.9.3) on moriah.shofar.org.uk. As the envelope To address has been rewritten by fetchmail to be the local host, sendmail delivers it locally. It would be perfectly possible though for the envelope address written by fetchmail to be a local alias which expanded to a remote address, and hence for the message to be re-routed over the internet – this is how the old NELUG mailing list worked.

Example 2: the above config. in action

From owner-nelug-eddy=shofar.uklinux.net@lists.lug.org.uk  Thu Jun  1 21:12:45 2000
Received: from localhost (root@localhost [127.0.0.1])
        by moriah.shofar.org.uk (8.9.3/8.9.1) with ESMTP id VAA00390
        for <eddy@localhost>; Thu, 1 Jun 2000 21:12:44 +0100
Received: from mail.uklinux.net
        by localhost with POP3 (fetchmail-5.1.2)
        for eddy@localhost (multi-drop); Thu, 01 Jun 2000 21:12:44 +0100 (BST)
Received: from lug.org.uk (qmailr@lug.org.uk [195.92.249.253])
        by www.uklinux.net (8.9.3/8.8.7) with SMTP id VAA29981
        for <eddy@shofar.uklinux.net>; Thu, 1 Jun 2000 21:24:38 +0100
Received: (qmail 13274 invoked by uid 300); 1 Jun 2000 20:24:19 -0000
Delivered-To: nelug@lists.lug.org.uk
Received: (qmail 13265 invoked from network); 1 Jun 2000 20:24:19 -0000
Received: from ns0.uklinux.net (HELO www.uklinux.net) (root@212.1.130.10)
  by lug.org.uk with SMTP; 1 Jun 2000 20:24:19 -0000
Received: from moriah.shofar.org.uk (root@ppp-1-201.cvx1.telinco.net [212.1.136.201])
        by www.uklinux.net (8.9.3/8.8.7) with ESMTP id VAA29961;
        Thu, 1 Jun 2000 21:24:13 +0100
Received: (from eddy@localhost)
        by moriah.shofar.org.uk (8.9.3/8.9.1) id UAA00333
        for NELUG-announce; Thu, 1 Jun 2000 20:55:21 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <14646.40424.81455.594113@moriah.shofar.org.uk>
X-Mailer: VM 6.72 under 21.1 (patch 8) "Bryce Canyon" XEmacs Lucid
Precedence: bulk
Reply-To: nelug@lists.lug.org.uk
From: Eddy Younger <eddy@shofar.uklinux.net>
Sender: owner-nelug@lists.lug.org.uk
To: NELUG-announce@shofar.uklinux.net
Subject: [nelug] Reminder - NELUG meeting, Weds. 7th June
Date: Thu, 1 Jun 2000 18:31:20 +0100 (BST)

Recommended Reading

1. The Bat Book: “Sendmail” – Costales, Allman & Rickert, O’Reilly 1993.
2. The red book: “Unix System Administration Handbook” – Nemeth, Snyder, Seebass & Hein, Prentice-Hall 1995
3. The Crab Book; “TCP/IP Network Administration”, – Hunt, Craig, O’Reilly.
4. cf/README file from the sendmail distribution;
5. /usr/doc/fetchmail*/*, fetchmail man page.
6. RFC 821 (SMTP definition); RFC 822 (mail mesage structure); RFC 1425 (ESMPT definition); RFC 974 (DNS MX records and mail routing)
]]> http://www.nelug.org.uk/configuring-email-systems/feed/ 0 http://www.nelug.org.uk/dial-up-networking/ http://www.nelug.org.uk/dial-up-networking/#comments Wed, 12 Apr 2000 11:48:39 +0000 olly-bh http://www.nelug.org.uk/?p=115 Continue reading →]]> NELUG, 12/4/2000

Eddy Younger <eddy@shofar.uklinux.net>

Introduction

• Dial-up networking is a special case of TCP/IP (usually…) networking over point-to-point connections.
• Two machines are connected to one another via serial lines
• Complicated by the need initially to establish a connection thru’ a modem
• See also PLIP (parallel-port connection)

PPP

PPP – point-to-point protocol, consists of a scheme for transporting IP packets over a serial line, a protocol to control the establishment of the link and negotiation of link parameters – the Link Control Protocol (LCP), and a set of Network Control Protocols (NCP) which configure different network-layer protocols – Linux ppp supports both IP and IPX at the network level. Includes a variety of data-compression mechanisms, authentication mechanisms, address assignment etc. Earlier mechanism – SLIP (Serial-line IP) – now deprecated: no link control/negotiation scheme meant that the two ends of the link had to have parameters fixed in advance and known to the other end.

There are two parts to the Linux PPP implementation:

1. The ppp driver, a kernel module (or compiled-in). Implements the ppp encapsulation and LCP protocols. Sits between the IP layer and the RS-232 data-link layer.
2. pppd – the ppp daemon. Responsible for bringing up a ppp instance on specified serial device, and handles negotiation of link parameters with the peer system. In the dial-up case, pppd invokes an external program – typically chat or dip – to initiate the connection. Once the link is up and negotiation is complete, pppd retires into the background until kicked into action by further LCP messages, or to tear down the connection.

Cast of characters


ppp layer appears to the IP layer above like just another network device.

chat

Simple program used by pppd to talk to the modem in order to dial up, and optionally log in to the remote machine, start ppp on the remote machine, etc. Simple expect/response dialog program; a chat script consists of pairs of “expected string”/”response string” lines and a few simple commands:

ABORT BUSY
ABORT "NO CARRIER"
ABORT VOICE
ABORT "NO DIALTONE"
ABORT "NO ANSWER"
"" ATZ
OK ATDT08456621598
CONNECT ""
ogin: <user-name>
assword: <your-password>
% ppp

chat scripts may be used to log in to remote systems which issue “login:” and “password:” prompts, as in this example, but this is unusual for ISP’s and is less secure than CHAP authentication (see later).

dip

More sophisticated chat-like program, with inbuilt simple scripting language, and wider command set. Allows user interaction for example to enter user name and password at appropriate point in the dialogue. Useful for token-card users

wvdial

More sophisticated still. Can function as a chat or dip replacement (subservient to pppd), or can function as the controlling process – initiating the connection and then invoking pppd. In this role it first dials up via the modem and attempts to analyse the data coming back from the modem in order to decide how to log into the remote system and establish the ppp connection as necessary. Configured by a file (/etc/wvdial.conf) consisting of a number of “sections” containing parameters such as phone number and login name for each of possible several remote connections (ISP’s).

pppd options

The correct pppd options are probably the most important and most troublesome aspect of dial-up networking. Options may be specified in the global options file (/etc/ppp/options unless otherwise specified by file <filename> argument to pppd), the file .ppprc in the user’s home directory. the device-specific options file (/etc/ppp/options.<tty-name>, eg. /etc/ppp/options.ttyS0, …./options.modem, etc), or on the pppd command-line, and are processed in that order. Different distributions provide several configuration programs to insulate the user from the business of writing options files directly, however doing it yourself is very easy, more instructive and arguably more reliable.

Common options

call <isp>

read options from the file /etc/ppp/peers/<isp>. This is the simplest way to configure pppd for a number of remote connections. The file /etc/ppp/peers/<isp> will contain options specific to the remote host, /etc/ppp/options will contain options common to all connections. e.g. pppd call uklinux

device e.g. /dev/modem

specify the device to communicate over

user <username>

use username to authenticate the local system to the remote

demand

enable demand dialing. pppd will configure the ppp device, but will not attempt to dial in to the remote system until it detects traffic trying to use the link. Certain classes of traffic may be ignore by use of the active-filter option (see man page for more details).

idle <seconds>

the link will be dropped of there is no traffic across it for the specified number of seconds.

defaultroute

add a default route via the remote system to the routing table when the link is established. This is almost always required for a home system using a dial-up link as its only internet connection.

usepeerdns

ask the peer to provide up to 2 DNS server addresses when the link is established. These are passed to the ip-up script as environment variables DNS1 and DNS2; ip-up can use these to configure the local resolver to use these nameservers. Usually only essential if your ISP’s nameserver addresses are not fixed.

ipcp-accept-local

ipcp-accept-remote

allow the remote system to set the IP address of both our end and its end of the ppp link. Both are typically required as IP addresses are typically assigned dynamically by ISP’s (unless you have e.g. a Demon account with a static IP address assigned to you). (IPCP is the NCP specific to IP).

connect <command>

specify the command to be used to bring up the serial link. Typically this will be a chat or dip command.

no-chap, no-pap

refuse to authenticate using chap or pap, respectively.

novj, noccp

disable different compression schemes (maybe useful for debugging).

ip-up/ip-down scripts

These live in /etc/ppp also, and are run respectively when the IP connection is established and broken.

Authentication

ppp supports two authentication protocols, CHAP (challenge-handshake authentication protocol) and PAP (password authentication protocol). Each of the two communicating systems may be required to autenticate itself to the peer using either or only one of these protocols.

CHAP

CHAP is the more secure of the two authentication methods, as at no time is the password or secret transmitted over the link. The system demanding authentication sends a challenge string to the peer, which encrypts the challenge string using the secret as a key. The encrypted challenge is then sent back to the first system which decrypts it using its stored copy of the secret. If the decrypted version matches the original challenge then both systems used the same secret as the key and the authentication succeeds.

PAP

PAP is a more simple autentication scheme wherein one system authenticates itself to the other by sending an identifier/password pair, much like a user logging in to a Unix system with a user name and password. It is less secure than CHAP as the password is transmitted unencrypted and hence theoretically open to snooping if the link is compromised.

PAP and CHAP use secrets files in /etc/ppp to store authentication strings (/etc/ppp/pap-secrets and /etc/ppp/chap-secrets respectively). Both of these files have the same format, consisting of lines of the form user-id peer-system-name secret.

In general, systems will agree to authenticate using whichever scheme the peer system tries to use. The no-chap or no-pap options to pppd will cause it to refuse to use those protocols.

Examples

/etc/ppp/options:

# /etc/ppp/options
#
# Don't escape control chars
asyncmap 0
# Enable hardware handshaking
crtscts
# Enable connect-on-demand
demand
# Drop link after 2 mins idle
idle 120
# No default IP address
noipdefault
# Don't require remote system to authenticate itself
noauth
# Use ppp link as default route
defaultroute
# Accept local/remote IP addresses from remote system
ipcp-accept-remote
ipcp-accept-local
# verbose debugging from ppp driver
kdebug 7
# lock the serial device while link is up
lock

/etc/ppp/peers/uklinux:

# device and line-speed to use
/dev/modem 115200  
# user name for authentication
user shofar 
# specify chat script to bring up connection
connect 'usr/sbin/chat -v -f /etc/ppp/chat-uklinux' 
# turn on pppd debug messages to syslogd
debug

/etc/ppp/chat-uklinux:

ABORT BUSY
ABORT "NO CARRIER"
ABORT VOICE
ABORT "NO DIALTONE"
ABORT "NO ANSWER"
"" ATZ
OK ATDT08456621598
CONNECT ""

/etc/ppp/chap-secrets:

shofar.freeserve.co.uk * <my-password>
shofar * <another-password>

/etc/ip-up:

#!/bin/sh
# Pick up my mail from the ISP servers
fetchmail  -v
# send any mail queued on the local system
/usr/bin/sendmail -q

N.B. Using kdebug 7 option causes every frame to be logged by the ppp driver (usually to /var/log/debug); this not only generates voluminous log files but also slows down the driver – and hence the whole system. If you set this for debugging purposes, remove it as soon as things are working properly !

Similar chat-freeserve and /etc/ppp/peers/freeserve files exist for my freeserve account. Bringing up the link is simply a matter of

pppd call uklinux  (or pppd call freeserve)

The demand option means that dial-up only happens when something actually wants to use the link.

In contrast, linuxconf on a RedHat 6.1 system uses 6 or 7 files and of the order of 100′s of lines of shell to configure a ppp connection – difficult to follow and debug if anything goes wrong.

Dial-up link as an internet gateway

A local-area network might well wish to use a machine with a dial-up connection as an internet gateway. There are (at least) two ways to approach this depending upon the level of access required from the LAN:

1. Mail and WWW access only. Disable IP-forwarding on the gateway. Run a web proxy (squid, IJB or similar) on the gateway. All other machines on the LAN have their browsers set up to use the gateway as a proxy. Use fetchmail (run periodically as a cron job) to download mail periodically. Run pop server on the gateway, or mount /var/mail on clients via nfs. Set gateway as SMTP host for client mailers or use nullclient sendmail.cf files on client machines. On gateway run sendmail -q under cron periodically to send queued mail. In this scenario, no IP traffic is routed through the gateway machine.
2. Full internet access. Set up IP-masquerading on the gateway. All clients on LAN have default route set to gateway, and gateway as DNS server. Use defaultroute and usepeerdns pppd options (or run bind) on gateway.

Debugging

1. Use debug and kdebug options to pppd and ppp respectively. These cause verbose logging thru’ syslog and klogd, which usually will end up in /var/adm/messages (or possibly /var/log/syslog, depending upon your distribution) and /var/log/debug respectively.
2. Run tcpdump on the ppp interface (tcpdump -i ppp0) to look at IP traffic on the link (if you get the link up !).
3. Usual networking utilities (netstat, ifconfig) will work on the ppp device once the IP link is up:

# ifconfig -a
dummy     Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          BROADCAST NOARP  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 

eth0      Link encap:Ethernet  HWaddr 00:C0:F0:45:83:C9  
          inet addr:192.168.200.1  Bcast:192.168.200.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:655 errors:0 dropped:0 overruns:0 frame:0
          TX packets:646 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          Interrupt:11 Base address:0x6100 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:3924  Metric:1
          RX packets:1824 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1824 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 

ppp0      Link encap:Point-to-Point Protocol  
          inet addr:212.1.156.19  P-t-P:10.112.112.112  Mask:255.255.255.255
          POINTOPOINT NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:42 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10

Sample debug output (from /var/adm/messages):

Apr 11 20:47:15 moriah kernel: registered device ppp0
Apr 11 20:47:15 moriah pppd[410]: pppd 2.3.10 started by root, uid 0
Apr 11 20:47:15 moriah kernel: ppp_ioctl: set dbg flags to 10000
Apr 11 20:47:15 moriah kernel: ppp_tty_ioctl: set xmit asyncmap 0
Apr 11 20:47:15 moriah kernel: ppp_ioctl: set flags to 10000
Apr 11 20:47:15 moriah kernel: ppp_ioctl: set mru to 5dc
Apr 11 20:47:15 moriah kernel: ppp_tty_ioctl: set rcv asyncmap 0
Apr 11 20:47:15 moriah pppd[410]: Using interface ppp0
Apr 11 20:47:15 moriah pppd[410]: local  IP address 192.168.200.1
Apr 11 20:47:15 moriah pppd[410]: remote IP address 10.112.112.112
Apr 11 20:48:09 moriah pppd[410]: Starting link
Apr 11 20:48:10 moriah chat[414]: abort on (BUSY)
Apr 11 20:48:10 moriah chat[414]: abort on (NO CARRIER)
Apr 11 20:48:10 moriah chat[414]: abort on (VOICE)
Apr 11 20:48:10 moriah chat[414]: abort on (NO DIALTONE)
Apr 11 20:48:10 moriah chat[414]: abort on (NO ANSWER)
Apr 11 20:48:10 moriah chat[414]: send (ATZ^M^M)
Apr 11 20:48:10 moriah chat[414]: expect (OK)
Apr 11 20:48:10 moriah chat[414]: ATZ^M^M
Apr 11 20:48:10 moriah chat[414]: OK
Apr 11 20:48:10 moriah chat[414]:  -- got it 
Apr 11 20:48:10 moriah chat[414]: send (ATDT08456621598^M)
Apr 11 20:48:10 moriah chat[414]: expect (CONNECT)
Apr 11 20:48:10 moriah chat[414]: ^M
Apr 11 20:48:28 moriah chat[414]: ATDT08456621598^M^M
Apr 11 20:48:28 moriah chat[414]: CONNECT
Apr 11 20:48:28 moriah chat[414]:  -- got it 
Apr 11 20:48:28 moriah chat[414]: send (^M)
Apr 11 20:48:29 moriah kernel: ppp_ioctl: get unit: 0
Apr 11 20:48:29 moriah kernel: ppp_ioctl: set flags to 10000
Apr 11 20:48:29 moriah kernel: ppp_tty_ioctl: set xasyncmap
Apr 11 20:48:29 moriah kernel: ppp_tty_ioctl: set xmit asyncmap ffffffff
Apr 11 20:48:29 moriah kernel: ppp_ioctl: set flags to 10000
Apr 11 20:48:29 moriah kernel: ppp_ioctl: set mru to 5dc
Apr 11 20:48:29 moriah kernel: ppp_tty_ioctl: set rcv asyncmap ffffffff
Apr 11 20:48:29 moriah pppd[410]: Serial connection established.
Apr 11 20:48:29 moriah pppd[410]: Connect: ppp0 <--> /dev/modem
Apr 11 20:48:30 moriah kernel: ppp_tty_ioctl: set xmit asyncmap a0000
Apr 11 20:48:30 moriah kernel: ppp_ioctl: set flags to f010003
Apr 11 20:48:30 moriah kernel: ppp_ioctl: set mru to 5dc
Apr 11 20:48:30 moriah kernel: ppp_tty_ioctl: set rcv asyncmap 0
Apr 11 20:48:30 moriah kernel: ppp_ioctl: set flags to f010043
Apr 11 20:48:30 moriah kernel: PPP Deflate Compression module registered
Apr 11 20:48:31 moriah kernel: ppp_ioctl: set maxcid to 16
Apr 11 20:48:31 moriah kernel: ppp_ioctl: set flags to f01004f
Apr 11 20:48:31 moriah fetchmail[418]: 5.1.2 querying pop.freeserve.net (protoco
l POP3) at Tue, 11 Apr 2000 20:48:31 +0100 (BST)

Recommended Reading

1. ppp-HOWTO, ISP-HOWTO, Serial-HOWTO
2. /usr/doc/ppp-*/*
3. man pages: pppd, chat, dip, wvdial
4. The Crab Book – Hunt, Craig: TCP/IP Network Administration, O’Reilly.
]]> http://www.nelug.org.uk/dial-up-networking/feed/ 0