Investigation of Performance of TCP over ATM
NSERC
Undergraduate Research Scholarship 2000
William
M. Miles, Mary E. Kaye and Brent R. Petersen
Department of Electrical and Computer Engineering
University of New Brunswick
P. O. Box 4400
15 Dineen Drive, Head Hall, Room D-41
Fredericton, New Brunswick, Canada
E3B 5A3
Technical Report
September 4, 2000.
Table of
Contents 2.2 Asynchronous Transfer Mode (ATM) 2.3.3 Alternatives to Classical IP over
ATM 4.1.1 Selection of Base Kernel 4.1.2 ATM Services Under Linux 4.1.3 ATM Service Initialization 4.2 TCP over ATM Protocol Definition 4.2.3 Disconnection and Error Handling 4.3.1 Modification of TCP options 5. Conclusions and
Recommendations ATM technologies are dropping in price, and are becoming
increasingly available for entire networks down to the workstation level. ATM is an inherently connection-based
protocol. However, existing high-level
network protocols typically assume a broadcast network and require complex
software support to operate on an ATM network.
With the specific case of the TCP/IP suite, the overhead is further
compounded by the fact that TCP is also connection-oriented. Thus we have a connection-oriented
application (TCP) operating on an emulated broadcast network (IP) in turn
running over connection-oriented hardware (ATM). This configuration is inherently wasteful, especially because IP
headers are large in comparison to ATM cells.
By building ATM support directly in to the TCP layer, substantial
performance improvements may be made as both the header and processing overhead
of the IP layer may be bypassed. Modern protocol suites often try to combine the features of
these two paradigms to maximize the utility to the user. The very common TCP/IP protocol suite,
developed in the 1970s and 1980s, is no exception: the TCP (Transfer Control
Protocol) is designed to create reliable virtual connections over a packet
switching network; while the IP (Internet Protocol) provides the services to
operate that packet switching network.
TCP is responsible for the application interface, allowing programs to
define connections that can be trusted to remote machines. IP is responsible only for routing packets
from point A to point B; it is connectionless, and knows nothing of the data
each packet contains. While TCP/IP can be adapted to run on nearly any hardware configuration,
IP does make certain assumptions about the network architecture. An IP network is broken down into subnets
with related addresses and routers (known as gateways) connecting the subnets
together. Each machine on a subnet
expects to be able to contact other machines on its subnet without additional
routing; that is, it can simply send a packet to the hardware interface and
expect direct delivery directly. It
also expects to be able to broadcast a packet to all machines on the
subnet. On broadcast media such as
Ethernet or token ring networks, this is trivial; however, Asynchronous
Transfer Mode (ATM), a relatively new networking hardware protocol, does not
support broadcasting, and so a traditional TCP/IP implementation is not
possible. ATM is a protocol designed to take advantage of modern
communications techniques and hardware.
ATM operates very much like TCP in that it is inherently
connection-oriented, but operates over a packet switching network. ATM switches move tiny packets called
‘cells’ at extremely high speeds, routing them based on pre-established
connections; this arrangement combines the rapid transit speed of
circuit-switching networks with the bandwidth efficiency of packet-switching
networks. As the reliability of modern
communications hardware is now extremely high, ATM does very little in the way
of error checking to minimize overhead and improve throughput. Should an error be detected, ATM switches
will simply drop (ignore) the packet; thus reliability controls must be
performed by a higher-level protocol.
Additionally, ATM contains inherent support for voice and video
transmission, allowing many different applications to share a common ATM
network with minimal degradation. Since there is no support for packet broadcasting on ATM
networks, several different schemes have been used to run the TCP/IP protocol
suite over an ATM network. The most
straightforward is called ‘Classical IP over ATM’ (CLIP). CLIP requires that one machine with a known
address (either standard or manually entered at all stations) act as an address
server for the network. Using a protocol
called ATMARP (ATM Address Resolution Protocol), all machines on that subnet
must contact this machine to find out the ATM address registered to the IP
address it wishes to contact. The
client must then create an ATM connection directly to that machine, which it
passes all IP traffic destined there across.
For broadcast IP packets, the client retrieves the list of all ATM
addresses for that subnet, and contacts each machine individually to deliver
the packet. The other schemes used to facilitate IP traffic over an ATM
network are equally roundabout. LANE
(Local Area Network Emulation) operates nearly identically as CLIP, except that
instead of handling just IP traffic it is generalized to emulate a broadcast
network device, and thus any broadcast network protocol (IP, IPX, Netbeui, etc.)
may be used over it. This simply adds
yet another layer of software and emulation that must be executed each time a
packet is sent. MPOA (MultiProtocol
Over ATM) is another generalized broadcast network system, but operates more
directly with the protocol rather than creating an emulated network
device. Both LANE and MPOA require
servers at known addresses and more processing than CLIP, but allow other
broadcast protocols to be run in tandem over the same virtual circuits. The irony emerges when one considers the TCP protocol. TCP was designed to operate in conjunction
with IP, providing a simple, reliable, connection-oriented interface for
applications. TCP essentially converts
an incoming data stream into packets, attaches flow control and reliability
information, and then employs IP to carry those packets to their
destination. When using TCP in an ATM
environment, we are therefore encapsulating a connection-oriented system (TCP)
on a connectionless system (IP) on a connection-oriented system (ATM). It
is possible to reduce the overhead of TCP connections in an ATM environment by mapping
them directly to ATM connections. Not
only will bandwidth be saved by not sending the IP headers with each packet,
but the processing overhead of creating and parsing the IP headers will also be
eliminated. This system, called TCP
over ATM (T/A), will provide a shortcut bypassing the IP layer as shown in
Figure 1. To facilitate this end, a new protocol must be designed to
manage the setup, status, and teardown of ATM connections that correspond to
TCP connections. Since both TCP and
ATM are connection-oriented protocols, this will be relatively simple. The new protocol must be able to identify
the ATM address of each end station, determine when a TCP connection is being
opened to a machine that can support T/A connections, open an ATM connection
when a TCP connection is being established, handle any errors that occur while
the connection is opening or established, and tear down the connection when the
TCP session is terminated. Over the course of this project, steps towards writing a
rudimentary implementation of T/A were taken.
This implementation will be completed as part of the requirements for
the author’s undergraduate thesis. At
that time, performance benchmarks will be taken to analyze the practicality of
continued work in this direction and the results published. If the improvements are deemed sufficient,
then a draft submission to the appropriate standards organization will be made. The Linux 2.4.0-pre5 kernel was used as the base for
implementing T/A for this project.
This selection was made because it is entirely open-source, ATM services
are available for this kernel, and a special software driver is available that
emulates ATM hardware using traditional Ethernet hardware, thus reducing the
costs of development. The source code
for the base kernel may be downloaded at http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.0-test5.tar.bz2 ATM services under Linux are operated through a group of
daemons that interact with the kernel.
These daemons are not included as part of the server sources; they must
be downloaded separately from http://icawww1.epfl.ch/linux-atm/,
the ATM on Linux home page. The version
0.78 ATM services were used during this experiment. Figure 2 details the interaction of the kernel and the daemons to
operate a CLIP system. Blue lines indicate the flow of data from the application;
orange lines indicate the flow of control information. The purple bubbles are responsible for
maintaining the devices connected to.
Red lines indicate the flow of data back through the virtual ATM device. The following table details the function of each module. Function Application User application creating a TCP connection. Standard Linux TCP/IP stack for internetwork
communications. IP interfaces Active interfaces that support IP. ifconfig InterFace CONFIG. Administers
properties of hardware interfaces supporting IP. eth0 Ethernet hardware interface. atm0 CLIP interface for ATM. atmarp Administration program that enables CLIP interfaces. atmarpd ATM ARP Daemon. Responsible
for providing IP to ATM name resolution.
May maintain master ARP list for subnet, or may be a client that
connects to a master ARP list elsewhere. ATM Address List List of ATM addresses known to be reachable through ATM
interface. ilmid Daemon supporting ILMI, a protocol to allow ATM devices to
identify the ATM addresses of other nearby devices. Not used in this project. atmaddr Manual ATM address list maintenance command. atmsigd ATM SIGnaling Daemon.
Handles the control channel for switched virtual circuit operation,
communicating circuit setup and disconnection information to the ATM switch. ATM Devices Active kernel ATM device drivers. atmtcp Daemon that uses a TCP connection to emulate an ATM
device. The following setup scripts were used to open the virtual
ATM connection between the two test machines, named ‘compeng6’ and ‘vega’. An ATM hosts file has previously been set up
containing fake ATM addresses for both machines. This file substitutes the fake ATM addresses for the strings
‘compeng6-atm’ and ‘vega-atm’ during the setup scripts. Function atmtcp virtual listen & Begin virtual ATM service. Listen for incoming connection. Initialize ATM signalling daemon as network host. atmaddr –a compeng6-atm Bind the ATM address for ‘compeng6-atm’ to the local
machine. atmarpd –b Activate the ATM ARP daemon. atmarp –c atm0 Create the atm0 IP interface using CLIP. ifconfig atm0 10.0.0.2 up Activate the atm0 interface using IP address 10.0.0.2 Function atmtcp virtual connect 131.202.135.34 & Begin virtual ATM services. Connect to 131.202.135.34 (compeng6). Initialize ATM signalling daemon as network client. atmaddr –a vega-atm Bind the ATM address for ‘vega-atm’ to the local machine. atmarpd –b Activate the ATM ARP daemon. atmarp –c atm0 Create the atm0 IP interface using CLIP. ifconfig atm0 10.0.0.3 up Activate the atm0 interface using IP address 10.0.0.3 atmarp –s 10.0.0.2 compeng6-atm arpsrv Configure the ATM ARM daemon to map IP 10.0.0.2 to
compeng6-atm, and use it as the ATMARP server. These scripts, when run, set up a virtual ATM connection
using readily available Ethernet hardware and activate CLIP across it,
providing a test bed for the T/A protocol. TCP uses a three-way handshake to establish a
connection. First, a packet with the
SYN (synchronize, or connect) flag is sent to the remote machine to request
that a communications channel be opened.
If the remote machine is available to accept this incoming connection,
it will reply with a packet with both the SYN and ACK (acknowledge) flags
set. The initiating machine, to
complete the connection, replies with a packet with just the ACK flag set. This handshaking system provides an ideal
mechanism to add T/A functionality. In
opening a T/A connection, it was decided that an option would be added to the
SYN packet, which would then allow the remote machine the option of
establishing a T/A connection if it so desired. If the remote machine is incapable of making a T/A connection, it
can simply pass the SYN-ACK packet back via regular IP. It was originally planned that this option
would contain the ATM address of the local machine, allowing the remote machine
to create the connection without performing an address lookup; however, a
public ATM address is too large to fit in a TCP option. Thus, only a single
byte is sent, and it is up to the remote machine to determine the ATM address
to connect to. In a pure IP environment, all three of these packets will
travel via IP. Under T/A, the original
SYN packet will also contain a TCP option requesting that a T/A connection be
made, if possible. The remote host will
then attempt to determine the ATM address of the initiating machine, if
possible. If a valid address is
available, the remote machine will then open an ATM connection to the
initiating machine. Once the ATM
connection has been established, the SYN-ACK packet and all further TCP packets
will be passed along it. Should this
ATM connection fail at any time, or in the event that it cannot be established,
the TCP packets will revert to using IP for transport. Both directions of the ATM connection must remain open while
the TCP connection is active, even if one side of the TCP connection is closed;
it must be possible for acknowledgements to flow back. Thus, disconnection of the T/A connection
may only take place when both sides of the connection are closed or when an
error occurs. In the event of an error,
this initial implementation of T/A will immediately revert back to using
standard IP to communicate and disconnect the T/A connection. Future implementations may wish to provide
more sophisticated error recovery, but as the purpose of this project is simply
to benchmark the concept a conservative implementation was selected. The first
step taken was the addition of the new TCP option. A value of ‘12’ was selected for this experiment; any TCP packet
with this value as a single-byte option it its header will be interpreted as a
request to open a T/A connection.
Support for the generation of this value based on the value of a
user-controllable TCP socket option was also added, to provide the ability to
choose whether or not to use T/A to user applications. Both the header generation and header
analysis functions were modified to support this new option. To handle
each T/A connection, a series of new functions and data structures were
defined. First, an ATM circuit
structure was added to the TCP status structures, allowing each TCP connection
to have an independent T/A connection. Second, the entry points for the new T/A functionality were
identified: when a new SYN packet was generated with the new TCP option, the
system must be configured to accept a new incoming T/A connection; when a SYN
packet is received with the new option and the system is capable of opening a
new T/A connection, it must identify the address to open the connection to,
then open it; when a packet is sent while there is an active T/A connection, it
must be sent along it; when a packet is received over a T/A connection, it must
be passed up to the TCP stack; and when the socket is closed, the T/A
connection must also be closed. A set
of new functions was defined in a new C file to handle the majority of these
cases. The receiving function must be
embedded in the ATM receive code, while the others may fit in the TCP code;
consequently, it was left to last. As both
the connect packets and outgoing packets ultimately pass through the IP layer,
the uppermost function of the IP layer was modified to call a function named
tcpatm_handle to determine whether any additional T/A processing was
required. If tcpatm_handle returned
true, then it will have already handled the packet and it need not be sent through
IP; if false, then send the IP packet normally. tcpatm_handle analyzes outgoing packets for which T/A is
enabled. If it is a SYN packet, it
calls a function named tcpatm_listen to prepare for an incoming T/A connection;
if it is a SYN-ACK packet, it calls tcpatm_open to identify the target machine
and open the T/A connection to it; if it is any other packet and a T/A
connection is established, then call tcpatm_send to send the packet along that
connection. In any other case, set the
socket as not using T/A to avoid unnecessary further processing for future
packets. tcpatm_listen, tcpatm_open,
and tcpatm_send are not yet written, however, and simply return success. One of the
issues that arose during the consideration of the content of tcpatm_listen and
tcpatm_open is that in order to establish an ATM connection for T/A, one must
specify the destination system address.
It would be ideal to be able to send this address along with the TCP
connection packets; however, ATM addresses are much too long to fit in a TCP
header. While it is possible to define
a new protocol under IP to communicate this address, this would overcomplicate
this experiment. Thus, for simplicity,
IP over ATM protocol independence was sacrificed; a simple ATMARP lookup was
coded into a prototype tcpatm_open function, as this implementation will only
be expected to operate in conjunction with CLIP. Once the
functions for establishing T/A connections and sending packets along them are
completed, the next problem will be identifying incoming T/A packets and
passing them to the TCP stack. All of
the above problems were handled by simple modifications to the well-documented
TCP code. However, installing support
to recognize T/A connections will be more complicated. One possible solution is to write a daemon
that will handle all T/A connections; however, this would require a
kernel-daemon interface, which adds another level of error-control that must be
built in. A better solution is to embed
some support directly in to the kernel; however, the ATM code is not
well-documented and subject to change.
This problem must be addressed as the test implementation is completed. TCP over ATM provides a mechanism for reducing the overhead
of TCP connections in an ATM environment.
It is a logical step, given that ATM is available to machines running
TCP/IP and the complexities of IP over ATM.
No previous work has been done in this direction. A test implementation is under way to
analyze the performance benefits precisely.
Should this analysis reveal any significant benefit, it is recommended
that TCP over ATM be put forward to Internet standards organizations so that
others may benefit from this technology. The complete kernel source with the modifications made at
the time this document was written is available at http://vega.dyndns.org/TCPATM/linux.atmtest.tar.bz2 A copy of just the modified source and
header files is available at http://vega.dyndns.org/TCPATM/changed.tar.bz2 Future updates will be posted at http://vega.dyndns.org/TCPATM All modifications were added within
conditional compilation blocks to allow easy updating to new kernel versions;
the flag used was named TCPATM. Printed
source files are not included here due to excessive length (>200 pages).
We gratefully acknowledge the donations of ATM switching equipment and
financial support provided by Canadian Marconi Corporation and the Natural
Sciences and Engineering Research Council of Canada (NSERC).
1. Introduction
2. Background
2.1 TCP/IP
2.1.1 Introduction
2.1.2 Concepts
2.2 Asynchronous
Transfer Mode (ATM)
2.3.1 Introduction
2.3.2 Classical
IP over ATM
2.3.3 Alternatives
to Classical IP over ATM
2.3.3 TCP
with ATM
3. TCP
over ATM
3.1 Concept
3.2
Implementation Overview
3.3
Project Goals
4.0 Implementation
Details
4.1
Experimental Kernel
4.1.1 Selection
of Base Kernel
4.1.2 ATM
Services Under Linux
Module
4.1.3 ATM
Service Initialization
Host Setup (compeng6)
Command
Client Setup (vega)
Command
4.2 TCP
over ATM Protocol Definition
4.2.1 Callback
4.2.2 Connection
Sequence
4.2.3 Disconnection
and Error Handling
4.3 Implementation
Details
4.3.1 Modification of TCP options
4.3.2 Entry Points
4.3.3 Function Breakdown
4.3.4 Callback address
4.3.5
System
input
5. Conclusions
and Recommendations
A. Appendix
– Source Code