View source for Iproute2

'''Advanced Routing and iproute2 - the ip routing utility'''
-------------------------------------------------------------------------------
iproute2.txt research document, details may be sketchy and incomplete.    v0.02
-------------------------------------------------------------------------------

== General Notes and Information ==

ifconfig and route use iproute2 syscalls.  However, the 'ip' tool is the key
to access advanced iproute2 syscalls.

 [ Usage examples of the ip tool ]
 
 ip link list                    : shows links including MAC but not IP address
 ip address show                 : more info including IP and queueing disipline
 ip route show                   : equivalent to route -n except uses iproute2


                                
-------------------------------------------------------------------------------
Address Resolution Protocol (ARP) resolves the hardware address of another
machine on the same local network.  ARP determines where a machine is at on
the LAN (really it determines what the MAC is of that machine).

synonymous terminology: MAC address = location = hardware address

IP address is Layer 3 networking, Media Access Control (MAC) is Layer 2.

An IP address knows not the location of a machine, ARP does.  Machines on the
Internet have DNS names which resolve to IP addresses, not to be confused with
knowing the location (MAC address).

When computer A wants to find computer B on the same LAN, computer A does an
ARP broadcast with IP (layer 3).  Computer B with matching IP answers with its
MAC address.  Further communication is layer 2 by MAC.  The ARP entry for 
computer B remains in the ARP cache of computer A for a limited duration.

When computer A wants to find computer C, which is on another network across
the Internet, 'A' knows the subnet is different from its own and therefore 
references its gateway (the router) and asks it for the location (MAC) of 
computer C.  Through Internet routing the local router finds the remote
router which is the gateway for computer C and asks it what the MAC address
is for computer C.  If the remote router does not know, it does an ARP
broadcast on the LAN of computer C.  Computer C answers telling its router
its MAC address.  That remote router then sends the MAC back to the local
router for computer A and informs computer A of the MAC address for 'C'.
Further communication is layer 2 from local machine to local router and layer 
3 between routers.

synonymous terminology: arp cache = neighbor cache = neigh

 [ Usage examples of ip relating to ARP ]
 
 ip neigh show                    : view current ARP cache table
 ip neigh delete X.X.X.X dev ethX : delete IP X.X.X.X from APR cache 



-------------------------------------------------------------------------------
Of the three routing tables which are part of iproute2, the traditional 'route'
command only modifies the main table.  The 'ip' tool can modify all three 
tables but modifies the main table by default.

 ip route ls                     : shows only the main route table
 ip rule list                    : display current route rules and priority
 ip route list table local       : shows necessary stuff in the local table
 ip route list table main        : same as 'ip route ls'
 ip route flush cache            : clear all route cache, do after modification

You can create your own route tables.  To create a custom table:


 echo 200 tablename >> /etc/iproute2/rt_tables
(''above: creates a route table in rt_tables'')


 ip rule add from X.X.X.X table tablename
(''above: source IP or computer on LAN'')


 ip route add default via Y.Y.Y.Y dev ethX table tablename
(''above: assigns route for computer on LAN'')



-------------------------------------------------------------------------------
Example: Two Internet Providers - Multihoming

A business may have two Internet providers.  To set up routing for two ISPs on
linux consider this example a generic guide.

 192.168.0.1 = IP of Internal network (irrelevant) on eth1
 64.21.10.250 = IP of first ISP on eth0
   gw 64.21.10.1  network 255.255.255.0
 128.42.20.250 = IP of second ISP on eth2
   gw 128.42.20.1 network 255.255.255.0

1. create two tables and set up routing

 ip route add 255.255.255.0 dev eth0 src 64.21.10.250 table T1
 ip route add default via 64.21.10.1 table T1
 ip route add 255.255.255.0 dev eth2 src 128.42.20.250 table T2
 ip route add default via 128.42.20.1 table T2

2. set up main routing table

 ip route add 255.255.255.0 dev eth0 src 64.21.10.250
 ip route add 255.255.255.0 dev eth2 src 128.42.20.250

3. set the preference for the default route

 ip route add default via 64.21.10.1

4. routing rules for interfaces

 ip rule add from 64.21.10.250 table T1
 ip rule add from 128.42.20.250 table T2

*. load balancing between the two providers

 ip route add default scope global nexthop via 64.21.10.1 dev eth0 weight 1 nexthop via 128.42.20.1 dev eth2 weight 1

-------------------------------------------------------------------------------
Thu Oct 23 11:48:27 CDT 2003

&nbsp;

== ADSL Rate Limit Experiment ==

 <nowiki>#  All Rates are in Kbits, so in order to gets Bytes divide by 8</nowiki>
 <nowiki>#  e.g. 25Kbps == 3.125KB/s</nowiki>
 <nowiki>#</nowiki>
 <nowiki>#  eth1 ==> is local network e.g. 192.168.0.0/24</nowiki>
 <nowiki>#  eth0 ==> is Internet network e.g. Your ISP</nowiki>
 <nowiki></nowiki>
 <nowiki>DNLD=128Kbit</nowiki>
 <nowiki>DWEIGHT=15Kbit          # DOWNLOAD Weight Factor ~ 1/10 of DOWNLOAD Limit</nowiki>
 <nowiki>UPLD=25KBit             # UPLOAD Limit</nowiki>
 <nowiki>UWEIGHT=2Kbit           # UPLOAD Weight Factor</nowiki>
 <nowiki></nowiki>
 <nowiki>tc qdisc del dev eth0 root</nowiki>
 <nowiki>tc qdisc del dev eth1 root</nowiki>
 <nowiki>#tc qdisc add dev eth1 root handle 1: cbq avpkt 1500 bandwidth 100mbit</nowiki>
 <nowiki></nowiki>
 <nowiki>#tc class add dev eth1 parent 1: classid 1:1 cbq rate 256kbit allot 1000 prio 5 bounded isolated</nowiki>
 <nowiki>#tc filter add dev eth1 parent 1: protocol ip prio 16 u32 match ip dst 192.168.30.83 flowid 1:1</nowiki>
 <nowiki></nowiki>
 <nowiki>#tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000 mpu 64</nowiki>
 <nowiki>#tc class add dev eth0 parent 10:0 classid 10:1 cbq rate $UPLD weight $UWEIGHT allot 1514 prio 1 avpkt 1000 bounded</nowiki
>
 <nowiki>#tc filter add dev eth0 parent 10:0 protocol ip handle 3 fw flowid 10:1</nowiki>
 <nowiki></nowiki>
 <nowiki>#tc qdisc add dev eth1 root handle 11: cbq bandwidth 100Mbit avpkt 1000 mpu 64</nowiki> 
 <nowiki>#tc class add dev eth1 parent 11:0 classid 11:1 cbq rate $DNLD weight $DWEIGHT allot 1514 prio 1 avpkt 1000 bounded</nowiki
>
 <nowiki>#tc filter add dev eth1 parent 11:0 protocol ip handle 4 fw flowid 11:1</nowiki>
 <nowiki></nowiki>
 <nowiki>echo "1"</nowiki>
 <nowiki>tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000 mpu 64</nowiki>
 <nowiki>tc class add dev eth0 parent 10:0 classid 10:1 cbq rate $UPLD weight $UWEIGHT allot 1514 prio 1 avpkt 1000 bounded</nowiki>
 <nowiki>tc filter add dev eth0 parent 10:0 protocol ip prio 16 u32 match ip dst 192.168.30.83 flowid 10:1</nowiki>
 <nowiki></nowiki>
 <nowiki>echo "2"</nowiki>
 <nowiki>tc qdisc add dev eth1 root handle 11: cbq bandwidth 100Mbit avpkt 1000 mpu 64</nowiki>
 <nowiki>tc class add dev eth1 parent 11:0 classid 11:1 cbq rate $DNLD weight $DWEIGHT allot 1514 prio 1 avpkt 1000 bounded</nowiki>
 <nowiki>tc filter add dev eth1 parent 11:0 protocol ip prio 16 u32 match ip dst 192.168.30.83 flowid 11:1</nowiki>
 <nowiki></nowiki>

&nbsp;

== Bandwidth Limiting with IP Masquerade ==

=== Introduction ===

This document is meant for IP Masquerade users who want to limit specific host's bandwidth.
The example made throughout the document is a aDSL line (640Kbits download / 160Kbits  upload) where
the DHCP hosts of the subnet are bandwidth limited and also forced through a cachine proxy.

My router specs are as follows:

:Distribution:
:*Debian 2.2 (stable)
::Technically, its Debian 3.0 (unstable), buy only the sysinit scripts, bind, sendmail, dhcp, nfs, samba and X are kept at the bleeding edge. (All the major 'Services').
:Hardware:
:*CPU: 466Mhz Celeron
:*RAM: 192MB
:*DISK: 40GB 7200 RPM Maxtor ATA100, 2GB IBM 5400 RPM ATA33

:Services:
:*DHCP Server
:*DNS Server
:*Mail Server
:*IP Masquerade (port fowarding also)
:*Caching Proxy (Squid)
:*Samba server
:*NFS Server

=== Requirements ===

The requirements here will only pertain to setting up bandwidth limiting and NOT IP Masquerading, DHCP Server, NFS Server, etc.

Kernel Requirements:

All IPTABLES support needed for IP Masquerade plus
  CONFIG_IP_NF_CONNTRACK
  CONFIG_IP_NF_TARGET_MARK --> This is for marking packets. We are going to mark the packets we want limited with this

QoS and/or fair queueing
  CONFIG_NET_SCH_CBQ
  CONFIG_NET_CLS_FW

Now I usually compile all the IPTABLES and QoS and/or fair queueing stuff as modules, but lsmod only shows those above as in use.

Software Requirements:

*IPTABLES --> http://www.iptables.org/
*iproute2 --> ftp://ftp.inr.ac.ru/ip-routing/  --> Most distributions come with this defaultly installed
 
=== Setting Bandwidth ===

I set my bandwidth in a shell script for the purpose of having it set on boot time and for ease and stopping and starting.

 <nowiki>#!/bin/bash</nowiki>
 <nowiki>#</nowiki>
 <nowiki>#  All Rates are in Kbits, so in order to gets Bytes divide by 8</nowiki>
 <nowiki>#  e.g. 25Kbps == 3.125KB/s</nowiki>
 <nowiki>#</nowiki>
 <nowiki>TC=/sbin/tc</nowiki>
 <nowiki>DNLD=150Kbit              # DOWNLOAD Limit</nowiki>
 <nowiki>DWEIGHT=15Kbit         # DOWNLOAD Weight Factor ~ 1/10 of DOWNLOAD Limit</nowiki>
 <nowiki>UPLD=25KBit                # UPLOAD Limit</nowiki>
 <nowiki>UWEIGHT=2Kbit           # UPLOAD Weight Factor</nowiki>
 <nowiki> </nowiki>
 <nowiki>tc_start() {</nowiki>
 <nowiki></nowiki>
 <nowiki>    $TC qdisc add dev eth0 root handle 11: cbq bandwidth 100Mbit avpkt 1000 mpu 64</nowiki>
 <nowiki>    $TC class add dev eth0 parent 11:0 classid 11:1 cbq rate $DNLD weight $DWEIGHT allot 1514 prio 1 avpkt 1000 bounded</nowiki>
 <nowiki>    $TC filter add dev eth0 parent 11:0 protocol ip handle 4 fw flowid 11:1</nowiki>
 <nowiki></nowiki>
 <nowiki>    $TC qdisc add dev eth1 root handle 10: cbq bandwidth 10Mbit avpkt 1000 mpu 64</nowiki>
 <nowiki>    $TC class add dev eth1 parent 10:0 classid 10:1 cbq rate $UPLD weight $UWEIGHT allot 1514 prio 1 avpkt 1000 bounded</nowiki>
 <nowiki>    $TC filter add dev eth1 parent 10:0 protocol ip handle 3 fw flowid 10:1</nowiki>
 <nowiki></nowiki>
 <nowiki>}</nowiki>
 <nowiki> </nowiki>
 <nowiki>tc_stop() {</nowiki>
 <nowiki></nowiki>
 <nowiki>    $TC qdisc del dev eth0 root</nowiki>
 <nowiki>    $TC qdisc del dev eth1 root</nowiki>
 <nowiki></nowiki>
 <nowiki>}</nowiki>
 <nowiki></nowiki>
 <nowiki>tc_restart() {</nowiki>
 <nowiki></nowiki>
 <nowiki>    tc_stop</nowiki>
 <nowiki>    sleep 1</nowiki>
 <nowiki>    tc_start</nowiki>
 <nowiki></nowiki>
 <nowiki>}</nowiki>
 <nowiki></nowiki>
 <nowiki>tc_show() {</nowiki>
 <nowiki></nowiki>
 <nowiki>    echo ""</nowiki>
 <nowiki>    echo "eth0:"</nowiki>
 <nowiki>    $TC qdisc show dev eth0</nowiki>
 <nowiki>    $TC class show dev eth0</nowiki>
 <nowiki>    $TC filter show dev eth0</nowiki>
 <nowiki>    echo ""</nowiki>
 <nowiki></nowiki>
 <nowiki>    echo "eth1:"</nowiki>
 <nowiki>    $TC qdisc show dev eth1</nowiki>
 <nowiki>    $TC class show dev eth1</nowiki>
 <nowiki>    $TC filter show dev eth1</nowiki>
 <nowiki>    echo ""</nowiki>
 <nowiki></nowiki>
 <nowiki>}</nowiki>
 <nowiki></nowiki>
 <nowiki>case "$1" in</nowiki>
 <nowiki></nowiki>
 <nowiki>  start)</nowiki>
 <nowiki></nowiki>
 <nowiki>    echo -n "Starting bandwidth shaping: "</nowiki>
 <nowiki>    tc_start</nowiki>
 <nowiki>    echo "done"</nowiki>
 <nowiki>    ;;</nowiki>
 <nowiki></nowiki>
 <nowiki>  stop)</nowiki>
 <nowiki></nowiki>
 <nowiki>    echo -n "Stopping bandwidth shaping: "</nowiki>
 <nowiki>    tc_stop</nowiki>
 <nowiki>    echo "done"</nowiki>
 <nowiki>    ;;</nowiki>
 <nowiki></nowiki>
 <nowiki>  restart)</nowiki>
 <nowiki></nowiki>
 <nowiki>    echo -n "Restarting bandwidth shaping: "</nowiki>
 <nowiki>    tc_restart</nowiki>
 <nowiki>    echo "done"</nowiki>
 <nowiki>    ;;</nowiki>
 <nowiki></nowiki>
 <nowiki>  show)</nowiki>
 <nowiki>   </nowiki>
 <nowiki>    tc_show()</nowiki>
 <nowiki>    ;;</nowiki>
 <nowiki></nowiki>
 <nowiki>  *)</nowiki>
 <nowiki></nowiki>
 <nowiki>    echo "Usage: /etc/init.d/tc.sh {start|stop|restart|show}"</nowiki>
 <nowiki>    ;;</nowiki>
 <nowiki></nowiki>
 <nowiki>esac</nowiki>
 <nowiki></nowiki>
 <nowiki>exit 0</nowiki>
 <nowiki></nowiki>

Now lets go through that.

At the top are some variables which make it easier to change the entire script with on change.

THIS SCRIPT IS ONLY ** ONE ** RULESET.

You can add as many rules as you like. I don't run a ISP, I only needed to restrict a few people.

This scripts assumes 2 things.
#eth0 ==> is local network e.g. 192.168.0.0/24
#eth1 ==> is Internet network e.g. Your ISP

Stop, Restart, and Show are self explanitory. I will not explain them.

Start:
 $TC qdisc add dev eth0 root handle 11: cbq bandwidth 100Mbit avpkt 1000 mpu 64
 $TC class add dev eth0 parent 11:0 classid 11:1 cbq rate $DNLD weight $DWEIGHT allot 1514 prio 1 avpkt 1000 bounded
 $TC filter add dev eth0 parent 11:0 protocol ip handle 4 fw flowid 11:1
 
 $TC qdisc add dev eth1 root handle 10: cbq bandwidth 10Mbit avpkt 1000 mpu 64
 $TC class add dev eth1 parent 10:0 classid 10:1 cbq rate $UPLD weight $UWEIGHT allot 1514 prio 1 avpkt 1000 bounded
 $TC filter add dev eth1 parent 10:0 protocol ip handle 3 fw flowid 10:1

First the top 3 three lines are for the Download bandwidth.

The first line creates and "PARENT" qdisc:
  --> $TC qdisc add dev eth0 root handle 11: cbq bandwidth 100Mbit avpkt 1000 mpu 64
man tc for more information, but basically its the device definition. 100Mbit Netgear card.

Next line creates the child qdisc with will have the actual download limit in it.
  --> $TC class add dev eth0 parent 11:0 classid 11:1 cbq rate $DNLD weight $DWEIGHT allot 1514 prio 1 avpkt 1000 bounded
man tc and friends again if you need any more definition than that. Personally I don't know what all it means or any optimal settings.

I just used the suggested man page defaults.

The last line and VERY important line, sets the handle on the child qdisc.

It sets the handle to 4.  Later we will use iptables to MARK the download packets with a 4 mark, and thats how the child qdisc will know which packets to limit and which to let alone.

The Bottom three lines are for the uploading bandwidth.

Its the same as the download, but the device has changed from eth0 to eth1, the variables have changed from $DNLD to UPLD etc, and
the handle has changed from 4 to 3.

Marking Packets for Limiting

Ok now you are ready to mark packets as they come through in order to limit them.

I just added the next few lines to my rc.firewall.

 # Mark packets to route
 # Upload marking
 $IPTABLES -t mangle -A FORWARD -s 192.168.0.128/29 -j MARK --set-mark 3
 $IPTABLES -t mangle -A FORWARD -s 192.168.0.6 -j MARK --set-mark 3
 
 # Download marking
 $IPTABLES -t mangle -A POSTROUTING -s ! 192.168.0.0/24 -d 192.168.0.128/29 -j MARK --set-mark 4
 $IPTABLES -t mangle -A POSTROUTING -s ! 192.168.0.0/24 -d 192.168.0.6 -j MARK --set-mark 4

Here I mark the following hosts:
 192.168.0.6
 192.168.0.128/29 =>
         192.168.0.128
         192.168.0.129
         192.168.0.130
         192.168.0.131
         192.168.0.132
         192.168.0.133
         192.168.0.134
         192.168.0.135

I hope no one needs help with Subnet masks. :)

Downloads are marked 4 and uploads are marked 3

29 means 29 out of the 32 bits are marked. i.e. 11111111.11111111.11111111.11111000 is the subnet mask ==> 255.255.255.248 which means 00000111 hosts allowed in subnet or 8 hosts: 0 through 7 + 192.168.0.128 SIMPLE.

=== Bandwidth Monitoring ===

Freshmeat.

These following tools are recommended:
#bwm ==> very simple, ncurses based, for quick and easy overall network summary.
#iptraf ==> very robust, ncurses based, my favorite, without kernel patch it lets you monitor specific host based on MAC Addresses
#connmon ==> ncurses and gtk interfaces. With kernel patch you can monitor individual ip bandwidths

=== Conclusion ===

It took me a long time to figure this crap out.

I learned plenty from the cbq-init script from freshmeat. thanx.

Email if you have anymore questions, fixes, complaints.

Written by Joe Roback <joe@roback.cc >

&nbsp;

== References ==

*Anubis ISPToolz - Anubis ISP Tools: Bandwidth & IP System
:http://anubis-tools.sourceforge.net/

*Software and Tools - Traffic Control HOWTO
:http://www.telenovela-world.com/~spade/linux/howto/Traffic-Control-HOWTO/software.html

*Implementation - ADSL Bandwidth Management HOWTO
:http://www.tldp.org/HOWTO/ADSL-Bandwidth-Management-HOWTO/implementation.html

*Bandwidth Limiting with IP Masquerade - Howto *
:http://www.szabilinux.hu/bandwidth/

*Linux.com - The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads
:http://www.linux.com/howtos/Adv-Routing-HOWTO/lartc.cookbook.ultimate-tc.shtml

&nbsp;

&nbsp;

[[Category:Computer_Technology]]
[[Category:Linux]]
[[Category:Networking]]