Subject: Proxy arp - theory and examples, was RE: [Ipchains] Configuration
From: William Stearns <[email-masked]>
To: Josh Sides <[email-masked]>, NARULA PRASHANT <[email-masked]>
Cc: ML-ipchains <[email-masked]>, William Stearns <[email-masked]>
Date: Sat, 18 Dec 1999 16:28:31 -0500 (EST)

Good afternoon, Josh and Narula,

On Sat, 18 Dec 1999, Josh Sides wrote:

> I'm sorry I should have read thru your response a second time.  I have read
> the Ipchains Howto and the Masquerade howto several times.  I don't think I
> have read about anything about this.  Where can I read more?

	(See the resources at the end of this email)

> William Stearns:
> 	2) If they won't do this and you're left with only a single block
> of addresses between your firewall and their router, you can still move
> some of those addresses to a separate server network.  In this case, your
> firewall will perform proxy arp for those machines.  This means the
> firewall will, at the Ethernet level, stand in for those machines so that
> the router thinks they're actually on the original cable when in fact
> they're on the server lan.

	I'll present a little of the background and then try to apply it
to Narula's configuration below.

	Proxyarp is a technique that allows a machine that should be on an
Ethernet cable to actually be somewhere else.  A proxy machine stands in
for the missing box and takes responsibility for getting the packets to
the missing box.
	Proxyarp is really a way of working around incorrect or incomplete
routing tables.  I believe any problem solved with proxyarp could also be
solved by putting correct routing tables in place.  If one has an
uncooperative ISP that won't update their routing, it can come in handy.
	Proxyarp is also used for the router accepting incoming ppp
connections at an ISP; when a client calls and connects, the ISP's router
can simply start standing in for that client on it's Ethernet cable.

	Back to the client end, here's a common example:

           .2  .3  .4  .5
            |   |   |   |
ISP Router ------------------ Linux box
        .1   12.13.14.X/24    .10
       (A)                    (B)

	Four servers and a Linux box on an Ethernet hub with real IP's.
The ISP's router believes that all the machines from 12.13.14.1 to
12.13.14.254 are on the Ethernet network.
	The (A) and (B) are the Ethernet cards - we'll need these later.
I did not include the fact that on the left hand side of the ISP's router
is the connection to the Internet; this might be a dial up connection or a
dedicated line.  Picture an arrow pointing left to the Internet from the
ISP's router.

	Lets say that we want to move two of the boxes behind the Linux
box.  It could be that you want to protect servers .4 and .5 behind the 
Linux box so it can act as a firewall for them.  Maybe those machines are
moving to a different area of the building and you need to put them on a
different hub.  Here's the new layout:

           .2  .3                    >> .4  .5
            |   |                    >>  |   |
ISP Router ------------------ Linux box ----------------
        .1   12.13.14.X/24    .10   .11 (also 12.13.14.X/24 addresses)
       (A)                    (B)   (C)

	The .4 and .5 servers are on the other side of the Linux box now.
Here's the problem.  If .2 or .3 need to talk to .4 or the router (.1) has
a packet destined to .4, they'll send out a broadcast on the left cable
saying "What Ethernet card has .4?" (*1)  If .4 was still on the left hand
Ethernet network, it would respond with it's own mac (Ethernet card)
address and the IP packets would flow.  Unfortunately, .4 is no longer on
that cable to hear the ARP request, so .1, .2, and .3 think that .4
doesn't exist. (*2)

	There are two ways to fix this.  The first is to explicitly tell
.1, .2 and .3 that .4 and 5 can be found on the other side of .10.  In
other words, add a routing table entry that says "Give all packets with a
destination address of .4 to .10 and let .10 figure out how to get them to
.4".  If .2 or .3 were Linux boxes (you could add these routes to almost
any IP capable OS, but I'm just giving a Linux example), you could add the
following to their startup scripts (*3):

route add -host 12.13.14.4 gw 12.13.14.10
route add -host 12.13.14.5 gw 12.13.14.10

	These routes would take precedence over the existing 

route add -net 12.13.14.0 netmask 255.255.255.0 dev eth0

	that already exist because the host routes point at a smaller
number of machines (1 each) than the network route (for 256 machines).
See "man route".
	Now .2 and .3 understand that they can't directly reach .4 or .5
on their Ethernet cable.  Instead, they give all packets destined for .4
to .10 and hope that it can find .4, which it can.

	That's fine for the .2 and .3 Linux boxes, but what about the
router?  The ISP owns it and isn't willing to let us play around with it's
configuration.
	OK, we'll let the router believe that all 254 machines are still
on the left hand Ethernet cable.  And we'll lie to it.  :-)
	We tell .10 that whenever it hears an ARP request like "I'm
12.13.14.1, can someone tell me what Ethernet card has 12.13.14.4?", it
should lie and respond "Yes .1, .4 can be found at my (B) Ethernet card.".
.1 will then start giving all packets that should have gone to .4 to .10's
(B) Ethernet card instead, thinking it's talking to .4.  In fact, those
packets have to go through .10 and have one more cable to cross before
they reach .4, but .1 doesn't care.  (Note: at this point, the Linux box
is a router - it's forwarding packets from one IP interface to another.
Make sure you have IP forwarding turned on or the packets won't get passed
back and forth.)
	Assuming (B) is eth0 and has a mac address of 00:10:5A:CC:97:BF,
here's what you'd type on the Linux box to make this happen:

route add -host 12.13.14.4 dev eth1
route add -host 12.13.14.5 dev eth1

	Translation: If I need to find .4 or .5, look out the (C)
interface instead of the (B) where I'd normally look for 12.13.14.0/24
addresses.

arp -i eth0 -s 12.13.14.4 00:10:5A:CC:97:BF pub
arp -i eth0 -s 12.13.14.5 00:10:5A:CC:97:BF pub

	Translation: If I hear incoming arp requests on eth0 (the left
hand Ethernet network for .4 or .5, respond that those machines can be
reached through the (B) Ethernet card.  See "man arp" for more details.

	If you use this second approach (proxyarp), you no longer have to
make any routing changes to .1, .2, or .3.  You do have to make a little
bit of a change to .4 and .5, however.
	After booting, you need to do the following on .4:
route del -net 12.13.14.0 netmask 255.255.255.0 dev eth0
route add -host 12.13.14.11 dev eth0
route add -host 12.13.14.5 dev eth0
route del default gw 12.13.14.1
route add default gw 12.13.14.11

	Translation: take away the incorrect assumption that the entire
class C is on the right hand Ethernet cable, register the fact that .11
and .5 _are_ on the right hand Ethernet cable, and hand off all packets
going anywhere else to .11 and not the .1 we would have used if we were
still on the left hand cable..

	.5 would need an almost identical set (the only change being
that it needs to be told that _.4_ is on it's cable):
route del -net 12.13.14.0 netmask 255.255.255.0 dev eth0
route add -host 12.13.14.11 dev eth0
route add -host 12.13.14.4 dev eth0
route del default gw 12.13.14.1
route add default gw 12.13.14.11

	Why all this work and effort?  Because you have machines (at least
the .1 router) that don't have a correct view of how you network is laid
out.  You need to get another machine (.10) to lie about how to reach the
missing machines (.4 and .5) so that .1 can reach .4 and .5 .

	Now to Narula's question and a real world example of the above:

> [Narula]
> dear william, i read your post on the ipchains mailing list. can you
> please tell me how to implement proxy arp on RH6.0 kernel 2.2.5.  
> Will this approach be able to solve my problem which is as below: I
> have a real class c network on my external network: 209.100.10.0/24.
> The ip address of the router is 209.100.10.1 and the ip address of the
> external interface of the Linux firewall is 209.100.10.2. My internal
> network is 172.16.0.0/16. The ip address of the internal interface of
> the Linux firewall is 172.16.1.1. I have assigned my servers
> 172.16.10.1-255 and my client workstations 172.16.11.1-172.16.30.255.
> Using ip masq'ing, all machines can see the Internet with no problems.
> The reason I am needing nat is so that I can map real ip addresses to
> my servers so they will be visible accessible from the internet, yet
> still behind my firewall.
>
> I want to take the real ip addresses 209.100.10.11-20 and map them to
> 172.16.10.11-20.
>
> Address match ups will look like this:
>
> 209.100.10.11 => 172.16.10.11
> 209.100.10.12 => 172.16.10.12
> 209.100.10.13 => 172.16.10.13
> 209.100.10.14 => 172.16.10.14
> 209.100.10.15 => 172.16.10.15
> 209.100.10.16 => 172.16.10.16
> 209.100.10.17 => 172.16.10.17
> 209.100.10.18 => 172.16.10.18
> 209.100.10.19 => 172.16.10.19
> 209.100.10.20 => 172.16.10.20
>
> Simply put, if someone tries to access a web server at 209.100.10.14,
> the Linux firewall will translate the address through to 172.16.10.14.
> Also, I cannot use ip-masq to return packets because they need to be
> returned from 209.100.10.14, not the external interface of my firewall
> 209.100.10.1 for this to work properly.

	I'll assume eth0 is your outside interface on the Linux firewall
and that eth1 is the inside interface.
	The above choices still apply.  You can do any of the following:
	- Tell your router that packets destined to these ten machines
need to be handed to the Linux firewall (209.100.10.2).  Give the Linux
box static routes to these machines on the internal network.

Router:
route add -host 209.100.10.11 gw 209.100.10.2 (repeat for the rest. You'll
need to figure out the syntax to use for your particular router, but the
idea is that it should hand off packets to 209.100.10.11, netmask
255.255.255.255 to 209.100.10.2, often called the gateway.)

Linux firewall:
route add -host 209.100.10.11 dev eth1 (repeat for the rest)
ifconfig eth1:1 209.100.10.3 netmask 255.255.255.255 up

Servers:
Give them the real IP's instead of reserved IP's.
route del -net 209.100.10.0/24 dev eth0
route add -host 209.100.10.11 dev eth0 (repeat for all other servers)
route del default gw 209.100.10.1
route add default gw 209.100.10.3

	- Set up proxy arp so that the Linux firewall acts as a stand-in
on the outside cable for the machines that are now on the inside network
(.11-.20).

Router:
(nothing to do)

Linux firewall:
route add -host 209.100.10.11 dev eth1 (repeat for the rest)
ifconfig eth1:1 209.100.10.3 netmask 255.255.255.255 up
arp -i eth0 -s 209.100.10.11 00:10:5A:CC:97:BF pub (repeat for all, use
the mac address for the Linux firewall's eth0 card instead of the example)

Servers:
Give them the real IP's instead of reserved IP's.
route del -net 209.100.10.0/24 dev eth0
route add -host 209.100.10.11 dev eth0 (repeat for all other servers)
route del default gw 209.100.10.1
route add default gw 209.100.10.3

	Note that the only difference between the first approach and the
second is how the router knows to hand packets destined for the internal
servers to the Linux firewall - in the first example, it's told this
explicitly, in the second proxyarp example, the Linux firewall convinces
the router that it can get packets to the servers.

	- Give the Linux box the .11-.20 IP addresses and tell it to
perform NAT to the boxes on the Internal lan.
(Sorry - not all that familiar with how to do formal NAT.  It involves
bringing up all the real IP's on the Linux firewall itself and using the
"ip" command from the "iproute" package to tell the kernel to translate
addresses from the real back to the reserved addresses on the servers.)

	- One more possibility, if you only have a few ports that need to
be handed back to the servers, is to use the single real address on your
Linux firewall and do port forwarding back to the servers on reserved
addresses.  This is probably not appropriate if you have more than one web
server, more than one mail server, or more than one (any IP service)
server.  See the IP Masquerading HOWTO and the ipmasq resource page for
more information on port forwarding.

	The above examples haven't been tested - you'll almost certainly
have to do some tweaking to make them work.  If things don't work at
first, use ping to see which machines can reach which machines.  Examine
the routing tables on all machines involved to see if the outgoing ping
requests (icmp echo requests) can get all the way to the destination
machine and the ping replies (icmp echo replies) can get all the way back.
	Cheers,
	- Bill

*1 This is called an ARP (Address Resolution Protocol) query.  .1 sends
out an Ethernet broadcast packet asking what Ethernet card is associated
with 12.13.14.4.  .4 is supposed to respond with "I can be found at Mac
address 04:05:06:07:08:09" so that .1 can now direct IP packets for .4 to
that mac address instead of having to broadcast everything to all Ethernet
cards.  Run tcpdump on an Ethernet interface sometime ("tcpdump -i eth0
-tn") and you'll see the ARP requests and responses from time to time.

*2 Note that ARP packets stay confined to a single lan - we can't ask the
Linux box in the center to carry the ARP query over to the right hand
cable.

*3  All of the commands listed here can go either in your network startup
script (perhaps /etc/rc.d/init.d/network for RedHat) or in
/etc/rc.d/rc.local .  They should be in place before any other services
need to reach other machines, so /etc/rc.d/init.d/network is preferred.
Place them after the other commands that bring up interfaces in that file.

Resources:
- The Network Administrators Guide (NAG - see "Address Resolution" and
"Checking the ARP Tables")
http://metalab.unc.edu/linux/LDP
- The IP Masquerading HOWTO, the Proxy ARP Subnetting HOWTO and the
(unmaintained) Proxy-arp mini HOWTO at:
http://metalab.unc.edu/linux/HOWTO/
- "man arp", "man route", the documentation in the iproute or iproute2
package:
ftp://ftp.inr.ac.ru/ip-routing/

---------------------------------------------------------------------------
	I'm not tense, just terribly, terribly alert.
(Courtesy of "Michael J. Dark" <[email-masked]>)
--------------------------------------------------------------------------
William Stearns ([email-masked]).  Mason, Buildkernel, named2hosts, 
and ipfwadm2ipchains are at: http://www.pobox.com/~wstearns/
--------------------------------------------------------------------------