Ethloop - local simulator for testing Linux QoS disciplines
Martin Devera aka devik (firstname.lastname@example.org)
Last updated: 3.5.2002
What is it
Ethloop is very simple and versatile packet generator and measurer
which I use to test HTB qdisc. It is developement tool and you take
it as such. It accepts controling program at stdin, executes it
and produces flow rate table at stdout.
The generated rate is output on interface(s) selected in program
and data are recieved on all interfaces in a system. You sometimes
need to set promisc flag on recieving interfaces.
Generated packets are ethernet frames with protocol set to ETH_P_CUST
so that you can safely inject them into existing LANs. MAC addresses
are set to MACs of sending/recieving interface (may be the same).
When you have only one ethernet card and still want to test on real hw
(not on lo device) you can create loopback connector. It is normal TP
8 pin connector where you interconnect pins 1-3 and 2-6. It works for
both 10 and 100Mbit and full duplex.
Other interface setup
You don't need to set IP adress for used interfaces. Ethloop's packets
are not IP packets. It will work with loopback (lo) device on linux,
with one eth equiped with wired loopback connector (see above) or more
eth cards in single computers if these are interconnected by hub/switch
Don't forget ifconfig ethX up if you are not using that interface
The controling program is line based text file. Each line has fixed
format and comment lines behins with #. The format of a line is:
5000 R 2 4500000
Number of spaces is optional and tabs can be used. Input is parsed
by sscanf "%u %c %u %n". The first number is time in
miliseconds from program start. The second character determines
operation to do. Third number is flow id (see below)
and remainder of line is operation dependent. However the remainder can be text
or number and numbers are allowed in decimal or hex (if starting with 0x).
Number can end with k to multiply it by 1024.
The line above means: at time 5sec set rate of flow 2 to 4.5 MBps.
Lines are first read into struct progdata in memory (performance reasons)
and should be in increasing time order. Ethloop performs no sorting
by time so be carefull about it.
Each line is executed at its time and can change ongoing traffic generators.
Flow is elementary object in Ethloop. Flow has many properties as sending rate,
rate jitter, incoming and outgoing interface .. see struct flowtab for
complete list if you are interested.
Each flow is really "flow" of packets with some properties. Ethloop generates the flow
and measures some properties of recieved packets which belongs to the flow. Each
packet carries information about flowid inside. You can exploit it with u32 QoS filter
but there are better ways.
See command reference later in this document to understand all flow related parameters.
Let's see simple example. We want to simulate two companies sharing single link
with HTB qdisc. We will use interface lo for test (everyone has one). We will
add TBF to the lo interface with 100kbps rate.
tc qdisc add dev lo root tbf rate 100kbps burst 10k limit 10k
Now let's create simple simulation program. Both companies will start to transmit
packets at 100kbps and second one will change to 50kbps and then to 10kbps later.
# Two flow simulation
# lo will be used to TX and RX
0 i 0 lo
0 i 1 lo
# set simulation packet size to 1k
0 S 0 1k
0 S 1 1k
# start both flows
0 R 0 100k
0 R 1 100k
# flow 1 to 50k and 10k later
2000 R 1 50k
4000 R 1 10k
# finish at 6sec
6000 X 0 0
Store it into file prog and run. You should see:
[root@devix ethloop]# ./ethloop < prog
read program 9 commands
store 0 at 508 ms
store 1 at 1007 ms
store 2 at 1507 ms
store 3 at 2007 ms
store 4 at 2507 ms
store 5 at 3007 ms
store 6 at 3507 ms
store 7 at 4007 ms
store 8 at 4506 ms
store 9 at 5006 ms
store 10 at 5505 ms
store 11 at 6017 ms
0.5 53506 42798 49 0 53506 43610 63 0
1.0 67953 62392 84 0 67953 36488 97 0
1.5 70691 68129 84 0 70691 33971 99 0
2.0 71781 68019 87 0 71781 34555 92 0
2.5 71999 70288 80 0 56294 31469 84 0
3.0 72834 72428 88 0 53234 31278 89 0
3.5 72113 72016 88 0 51796 30921 90 0
4.0 72052 69469 84 0 51534 33443 87 0
4.5 69599 78582 2 0 20099 19362 38 0
5.0 71856 73988 0 0 12486 12311 9 0
5.5 72045 72551 0 0 10819 10778 2 0
6.0 71088 71208 0 0 10504 10494 0 0
The "store" status is written to stderr and table into stdout. You
can do "./ethloop < prog > output". Now how to interpret output
table. First column is time in seconds (0.5 time increment is fixed).
Then for each flow there are four fields. In order these are: send
rate, recieve rate, delay and arbitrary field. Send rate should be
the same as you specified in the program and recieved is all what
was not dropped (thus should be lower). Delay is how long the packet
was in some queues. All these fields are averaged by EWMA with 1 sec
The arbitrary four field in each flow has different meaning - generaly
it is copy of k1 .. k4 areas in packets - I use is to measure some qdisc
interna parameters. Kernel sets these to something and I can display
it. In code you can also with small change allow jitter to be displayed
here (the first delay derivative).
Probably you want to see it graphically. Simple - store ethloop's output
into file "output", run gnuplot and type:
gnuplot> set data style lp
gnuplot> plot 'output' using 1:3,'output' using 1:7,'output' using 1:($3+$7)
gnuplot> set term gif small size 320,200
gnuplot> set out 'test.gif'
You should see output on your terminal and also test.gif should be written
as you see here. Also here you see what it is good idea to use SFQ - the
graphs are not so fair, are they ?
For each command I describe format of its parameter first then function.
- X Both flowid and argument are ignored, exit Ethloop and output
- i String. Set interface as incoming and outgoing for the flow.
- t String. Override outgoing interface only.
- S Number. Byte size of each packet in this flow.
- J Number. Jitter of packet size in bytes.
- R Number. Transmitter rate in bytes per second.
- P Number. Priocode set by setsockopt(sock,SOL_SOCKET,SO_PRIORITY,...). It
is very convenient way to test qdisc because majority of classfull qdisc
will use classid stored in priority field. To say that this flow should
go to class 3:4f use P 0x3004f. This way you need no filters.
- G Number. Priority jitter. The priority will be randomly changed in range
defined by P code and P+G. It allows you easy testing of qdisc with thousand
of active classes by naming them 1:7,1:8,1:9,1:a,1:b....1:fff for example
and set P 0x10007, G 0xff8 in one flow.
- s String. System command. It allows you to execute arbitrary command at some
time. Flowid is irelevant here and should be 0. system (2) call is used.
It is nice if you want to change setup of qdisc in the same measurement
(5000 s 0 tc class change root classid 1:1 prio 2).
here is archive. Untar, make and go.
here is archive with program and script I used to generate HTB graphs.