This page last modified: Jul 08 2005
title:Ethernet setup and problem solving with Linux and Fedora
keywords:ethernet,ether,net,red,hat,red hat,fedora,linux,osx,cable,interface,network,eth0,eth1,kudzu,3c509,3c905,modem,hub,switch,ping,iptables,firewall,masquerade,lan,wan,isp,fails,disabled,problem,data,packet,tcp
description:Example of working firewall and local area network with a private ethernet subnet. Includes some problems you may encounter, and how to diagnose them.
Last updated July 7, 2005.
This document gives working settings for ethernet under Red Hat/Fedora
Linux (and other simliar variants). Differences among Linux variants
is probably limited to file name and locations.
These examples cover a firewall machine with 2 interfaces, and a LAN
machine behind the firewall on a private 192.168 or 10.10
network. I've tested these and they work (assuming I haven't made any
typos - I wasn't about to put in my actual ip addresses, so I'm using
example.com for everything). We'll name the firewall machine "hera"
and the LAN machine "zeus".
The more recent graphical network tools are good, but if your network
is not working, I suggest not using the tools and editing the text
files just to be sure.
You'll need to stop, start, and restart your network. After any change
to any file, restart.
/etc/rc.d/init.d/network stop
/etc/rc.d/init.d/network start
/etc/rc.d/init.d/network restart
Starting/restarting should start all interfaces and iptables. See
diagnostics below for questions about iptables.
Below are the settings for zeus, which is one of several machines
behind the firewall. All the computers in this example have static ip
addresses.
Notice that zeus has newer (and smarter) onboard ethernet (built into
the motherboard). It gives out more information than we get from
hera's eth0. The interface on zeus happily autonegotiates a 100BaseT
connection to the switch. I have a small 8 port switch (not a hub) as
the core of my LAN.
[root@zeus ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=zeus
GATEWAY=10.10.1.8
[root@zeus root]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
BROADCAST=10.10.1.255
IPADDR=10.10.1.4
NETMASK=255.255.255.0
NETWORK=10.10.1.0
ONBOOT=yes
TYPE=Ethernet
[root@zeus root]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 10.10.1.8 0.0.0.0 UG 0 0 0 eth0
[root@zeus ~]# mii-tool eth0
eth0: negotiated 100baseTx-FD flow-control, link ok
[root@zeus ~]# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0x00000001 (1)
Link detected: yes
[root@zeus ~]#
Below are the settings for hera, which is the firewall machine with
two interfaces. I'm pretty sure the firewall machine must have a
static ip address.
You can see that the eth0 (on the motherboard) is somewhat crude. No
settings are available. It also appears that either my DSL modem or eth0
can't autonegotiate. On the other hand, the eth1 (a newer PCI ethernet
port) gives us all kinds of information, and it is quite happy to
autonegotiate 100BaseT with my (reasonably) smart ethernet switch that
runs the LAN behind the firewall.
[root@hera root]# cat /etc/resolv.conf
nameserver 192.0.34.43
nameserver 193.0.0.236
[root@hera root]# cat /etc/sysconfig/network
NETWORKING=yes
GATEWAYDEV=eth0
FORWARD_IPV4=yes
HOSTNAME=example.com
GATEWAY=192.0.34.166
[root@hera root]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.0.34.255
IPADDR=192.0.34.166
NETMASK=255.255.255.0
NETWORK=192.0.34.0
ONBOOT=yes
[root@hera root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
BROADCAST=10.255.255.255
IPADDR=10.10.1.8
NETMASK=255.0.0.0
NETWORK=10.0.0.0
ONBOOT=yes
[root@hera root]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.0.34.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 192.0.34.166 0.0.0.0 UG 0 0 0 eth0
[root@hera root]# mii-tool eth0
eth0: autonegotiation failed, link ok
[root@hera root]# ethtool eth0
Settings for eth0:
No data available
[root@hera root]# mii-tool eth1
eth1: negotiated 100baseTx-FD, link ok
[root@hera root]# ethtool eth1
Settings for eth1:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Current message level: 0x000020c1 (8385)
Link detected: yes
[root@hera root]#
The settings above in ifcfg-eth1 work fine, but knowing what I know now, I'd
probably make it more specific to the 10.10.1.0 network/subnet.
DEVICE=eth1
BOOTPROTO=static
BROADCAST=10.10.1.255
IPADDR=10.10.1.8
NETMASK=255.255.255.0
NETWORK=10.10.1.0
ONBOOT=yes
Diagnosing problems.
These things will make your network fail:
- bad cables
- incorrect cable (crossover instead of straight thru)
- device conflicts (often a modem will confict with the ethernet card)
- interface not working (bad card)
- interface not compatible with your hardware
- incorrect routing due to incorrect config settings
- two interfaces with the same ip address
- hub/switch autonegotiation fails
- incorrect gateway keeps LAN from accessing the Internet
- no ip forwarding or NAT prevents LAN from accessing the Internet
- iptables (firewall) prevents communication
- wrong driver
- kudzu conflicts with ethernet hardware
I've categorized problems by the general nature of their cause.
Data doesn't go from CPU to interface:
- device conflicts (remove conflicting device)
- interface not working (buy a new ethernet card)
- hardware not compatible (buy a new ethernet card, or a new computer)
- iptables (firewall) blocking (edit iptables config, open the port/protocol)
- wrong driver (download newer/correct driver)
- kudzu conflicts (disable kudzu with ntsysv or chkconfig)
Data gets to the interface, but not to the hub:
- bad cables (buy a new cat5 cable)
- two interfaces with the same ip address (change one to a different subnet)
- hub/switch autonegotiation fails (disable autonegotiate; I've only
had to do this under OSX, and it has a GUI with "manual" ethernet
settings. I guess Linux uses ethtools and/or mii-tools)
Data gets to the hub and apparently to the second computer, but not
from second computer's interface to its CPU:
- same as "data doesn't go from CPU to interface" above.
LAN computer can't get thru the firewall to the internet:
- incorrect gateway (it is the ip address of the gateway computer on
the same subnet as the computer using the gateway)
- no ip forwarding or NAT (enable ipv4 forwarding, enable masquerading
in iptables, enable NAT)
- iptables (firewall) blocking (add a rule to your iptables config to
allow the connection type you need)
Two interfaces:
I'm beginning to think that if you have two interfaces, they are
required to connect to different subnets, i.e. eth0 10.10.1.1 eth1
10.10.2.1 where eth0 talks to the .1 subnet and eth1 talks to the .2
subnet. Both interfaces using different ip addresses on the same
subnet seems to make a mess of the routing tables. Also, there is no
reason I can see to have two interfaces talking to the same network
(it is illogical). I have discovered that sometimes disabling an
interface (either in the bios setup or by removing its config files)
causes more trouble, and in at least one case I just told the unused
interface to talk to a different subnet (a subnet with no other
computers) and it was fine.
See "interface swapping" below.
Mostly I use ping to test connections. If ping works and ssh doesn't,
then the problem is almost certainly iptables.
I also use ping to make the LED blink on the interfaces and the hub. I
assume that a blinking light means that data is succesfully reaching
that point.
Here's what good output from ping looks like. For a little test I just
use ping -c1 (only send one ping). When I'm crawling around behind
computers and watching LEDs on hubs and switches, I just let ping
run. Never use ping in flood mode. Note that ping can be blocked by
iptables, and that many ISPs and some hosts will not return ping
packets.
[root@hera root]# ping -c1 10.10.1.8
PING 10.10.1.8 (10.10.1.8) from 10.10.1.8 : 56(84) bytes of data.
64 bytes from 10.10.1.8: icmp_seq=1 ttl=64 time=0.095 ms
--- 10.10.1.8 ping statistics ---
1 packets transmitted, 1 received, 0% loss, time 0ms
rtt min/avg/max/mdev = 0.095/0.095/0.095/0.000 ms
[root@hera root]# ping -c1 10.10.1.4
PING 10.10.1.4 (10.10.1.4) from 10.10.1.8 : 56(84) bytes of data.
64 bytes from 10.10.1.4: icmp_seq=1 ttl=64 time=0.236 ms
--- 10.10.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% loss, time 0ms
rtt min/avg/max/mdev = 0.236/0.236/0.236/0.000 ms
[root@hera root]#
Interface swapping
------------------
When using two ethernet interfaces (and probably any N interfaces of
any sort), the interface names will change if the modules load in a
different order. This can happen if the hardware was changed or the
kernel software/drivers (?) updated.
With two interfaces you should probably include each interface's MAC
address in ifcfg-ethN to assure that rebooting doesn't swap
interfaces. Using the gui tool system-config-network-gui (or whatever
it is called on your flavor of Linux), under the device configuration,
"Hardware Device" tab, check the "Bind to MAC address" box. If you
can't remember the MAC address, just click the "Probe" button.
If you are editing your config files manually, include one line like
this in your /etc/sysconfig/network-scripts/ifcfg-ethN
HWADDR=00:10:4B:35:B3:13
I've seen a report of an obscure bug with dhclient that didn't like an
HWADDR line, even when commented out. Also the "nameif" command (see
man nameif) can help prevent swapping of interfaces.
Redhat reports an (old, fixed) bug that the HWADDR didn't prevent swapping,
and instead just messed things up when the ports were swapped.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=101566
Other hints and info
--------------------
I'm not quite sure what arp does, but it seems like the arp -a command
prints the name of the gateway router.
arp -a