AOH :: GTTUTOR2.TXT|
Telecomm programming tutor #2
Following is a second conversational 'chat' between James Davis
and Raymond Wood designed as a follow-up of the first one. It
takes on the form of a tutorial again due to the high number of
requests for same following the first one we released.
Note, this is an update of the original text in that I discovered
an inadvertant error in the original that confused SEAlink and
Zmodem relative to their implementation of network flow control.
D: Shall we start this off with a kind of outline as to where I
think we will go with it? We discussed many fundamentals
involved with communications in the first tutorial and ended up
discussing several of the more popular file transfer protocols.
This session will go farther into the area of file transfer
protocols, technology such as the 9600 bit per second modems and
error correcting modems with MNP or ARQ, and how one goes about
intelligently selecting a protocol given a basic understanding of
their environment. For example, while Ymodem was described as
the 'King of the hill' when it comes to performance, that is not
true if you are using one of the packet switching networks. It
is also not true at 9600 bits per second.
W: You mentioned 9600 and MNP. I thought that there was no
industry standard for 9600 and that it is only practical if the
other end is talking the same language with the same hardware?
Also that MNP was implemented in the hardware of the
modem...where am I wrong ?
D: You're not wrong. GT PowerComm (12.20) now supports 9600
baud. I believe the newest version of Qmodem (3.0) does as well.
Paul Meiners, the author of GT PowerComm, has a USRobotics
HST9600 baud modem and he is using it every day. I, too, have a
USR HST9600 as well as a Microcom MNP modem that I am testing.
There are two quite different error correction methods in use at
this time. MNP (Microcom Networking Protocol) which was
developed by Microcom and ARQ (a general term used by USR to mean
Automatic Retry Request protocols - theirs being specifically
called USR-HST [High Speed Technology]) and these two methods are
totally incompatible. Even the methods used to modulate 9600
baud signals appears to be incompatible. However, we have
successfully connected these two different brands of modems in
'reliability' mode. The USR has the ability to 'fallback' to MNP
at 1200 or 2400 baud where MNP has established a standard. (Of
course, that makes sense for our PCP users).
We have also connected with other USR HST9600 modems and seen
that we have outstanding performance at 9600 baud. (We have
cruised along at about 945 cps during transfers of more than 3
million bytes so far). Further, GT is such an efficient comm
program that we are able to drive these modems at 19,200 bits per
second from the systems while the modem is communicating at 9600
to another modem - for additional performance. It is for this
very reason that we had to implement flow control - so the
transmitting modem does not overrun. I will discuss this in more
detail a little later in this tutorial.
So, while you are correct that there is no standard at 9600 baud,
that does not mean that 9600 baud modems are necessarily
impractical. We are determining to what extent it is a problem.
What concerns me the most is the different modulation methods.
Nevertheless, it will not stop our support of 9600 baud.
Finally, you are right again, MNP (ARQ) is a hardware function -
but it can and should be a transparent one. I note, for example,
that since I began testing these modems I have connected with
several (many) others and, as a result,totally eliminated the
line noise that was present prior to the MNP connection - ie,
there appears to be more to MNP than just error free file
transfers. Thus, we must look at it. And, in doing so, we will
test the various non-error checking protocols that are used in
such environments (Ymodem-G, for example). It is as much a
learning curve for us as for the users - we just MUST do it
behind the scenes for credibility sake.
W: I understand the necessity to stay up with technological
advances affecting your your product. What I am not to clear on
is exactly what is MNP or ARQ and why have they come about. Can
you shed some light on this?
D: Since 2400 baud modems are NOT really 2400 'baud' - they are
2400 bits per second, 1200 baud modems - it has been clear that
the limit of reliable communications in terms of speed using the
bandwidth of the existing telephone circuitry has not been
reached. However, it is also clear that as we more densely pack
information within that bandwidth the incidence of errors
increases. The manufacturers investigated, starting with
Microcom, various error detection and recovery methods that were
hardware assisted. That was the the birth of MNP (Microcom
Networking Protocol). There has been an evolution in that
technology which results in several 'levels' of MNP available
today. The higher the level, the more function is included. At
any level, MNP merely insures that the data received by the modem
is what was sent by the sending modem. That is INSUFFICIENT, in
my opinion. The only valid scenario is one in which the
receiving COMPUTER is insured that it received accurately what
the sending COMPUTER sent. There are cables, ports, circuits,
timings, etc. that MNP DOES NOT CHECK. Thus, it seems that a
combination of software and hardware error detection and
correction methods is necessary.
Almost all file transfer protocols check what I believe is
necessary - computer to computer accuracy. What, then, is the
advantage of MNP? Well, to begin with, it SHOULD be more
efficient. If the software need only be concerned with data
bytes and not CRC and other control bytes, then it should be
faster. Further, the newer levels of MNP are more efficient than
you might have guessed. They strip off the start bit and the
stop bits from each byte, for example, and that increases
transfer performance by 20% (8 bits per byte rather than 10).
Further, they send 'compressed' data via internal algorithms
which increases performance even more. On the other side of the
ledger, MNP and ARQ technology has some built in disadvantages
from a performance point of view, they are, after all, no longer
just high speed pipes but are now full computers (usually Z80's)
and are prone to modest slowdowns at the higher speeds.
Nevertheless, at 9600 'baud' it is possible to obtain about 1100
cps rather than 960 and at 2400 'baud' it is possible to obtain
upwards of 290 cps rather than 240.
Not to forget, as I mentioned earlier, MNP is active at all times
while protocol transfers are active only during a transfer -
thus, line noise is effectively filtered out even while we are
chatting. There are several possible advantages, and a few
disadvantages - not the least of which is the lack of standards.
W: Jim, I understand what you just said and from that it would
seem that MNP is needed at both ends to do the job. Is that
correct? Also is MNP proprietary for just Microcom modems?
D: It is obviously true that MNP (or ARQ) must exist on both
ends to be functional. When my Microcom modem connects with a
non-MNP modem it recognizes that fact and reverts to being a
standard Hayes compatible modem. Further, when the USR HST
connects with a Microcom that has MNP, there is a fallback in
baud rates to 2400 baud in both modems so that they can
communicate using MNP. That is likely to be overridden by the
users, however, via disabling MNP or ARQ in such situations. (My
opinion only). However, it is reasonably certain that 9600 baud
connections cannot be established without error correction being
functional. Further, while Microcom MNP is wider used than ARQ
(USR's method), the USR method of supporting both (at different
baud rates) is more flexible and argues for USR. It may be that
we obtained the wrong 9600 baud modems at this time. It is part
of the testing and learning process.
As to the proprietary nature of MNP, according to USRobotics,
Microcom has placed at least the first three levels of MNP into
the public domain. It is certain that they have been generous in
licensing out at least the lower 'levels' to other manufacturers.
What alternative do they have? Unless a standard evolves, these
are contests that damage the future, not advances it.
W: It seems obvious that standards in this area are to the
advantage of all concerned. Is there a standards organization
looking into this? I would like to have 9600 baud capability and
error free transmission. However, I would also like to
communicate with whomever without having to worry about whats at
the other end. Do you see what I am concerned about?
D: Of course. It is a paraphrase of my earlier discussion. I
think the only 'standards organization' that is effective is
called the marketplace. The huge power of the Hayes
organization, because of its modem standard, is likely to be the
telling blow to other manufacturers - when they finally put there
own 9600 baud technology - may well become the new standard.
Because of this I believe it is premature to buy 'long' in such
security issues as USRobotics and Microcom.
W: Whenever I talk to the Hayes people at a convention or trade
show, they know or say nothing about 9600 development. I do not
know if this is just policy or not. I think that when they do
introduce 9600 that it would not necessarily mean that whatever
they do will be the standard. I may be naive, but I would like
to believe that will be the case. I say this only because others
are active in meeting a need and they are not or appear not to
D: No argument there. My point remains valid only if Hayes does
something in the near term. Intel saw what happens when they get
over confident and let competition pass them by when they first
put the 8080 micro-computer chip into the marketplace. They had
it made, save only that the Z80 took it ALL away from them. It
was an awfully long time before they we were able to come back
and Motorola nearly did it to them again. So, while Hayes has by
far the largest visible shelf space in the industry at the
moment, USR (my guess) or Microcom could steal it away from lack
of responsive attention on their part.
W: It would seem that you need compatible hardware above 2400
baud and compatible software as well for truly effective and
increased performance. Does Paul Meiners' Megalink protocol tie
into this somehow?
D: Megalink is an extremely efficient protocol particularly
designed for the network environments like PCP and the higher
baud rates. It is 'network friendly', which means that it
recognizes and honors flow control imposed by the network. For
efficiency it uses 512 byte packets (4 blocks), it is a full
streaming protocol, which means it does not ever stop sending
unless it receives a NAK saying a packet was received in error,
and it is batch oriented. It uses block 0 header information, as
do all the '...link' protocols so that the resulting file is the
same size and properly time and date stamped, and it uses 32 bit
CRC rather rather than 16.
I think it is time to go back to the earlier tutorial and add
some more concepts at this time.
Since our last discussion there has been increased popularity in
two relatively new file protocols. The first of these is called
SEAlink and the second is Zmodem. You will recall in the earlier
discussion that 'windowing' techniques are beginning to become
available in the file transfer protocols. There is now a
Windowing Kermit, for example, as well as WXmodem. These
programs attempt to obtain better performance by avoiding the
start-stop approach used by earlier protocols where after sending
a packet of data the transmitter would stop and wait for an
Acknowledgment that the packet had been properly received before
sending the next one. Windowing protocols assume that the
packets are being received without error and do not wait between
packets. The receiving systems DO send ACK signals, its just
that the transmitter is not waiting for them. Assuming all is
well, time has been saved as a result. When an error does occur,
a NAK is returned to the transmitter and associated with that
signal is the packet number that was in error. Assuming the
transmitter still has that packet at its disposal it merely
retransmits it and proceeds.
That is the limit, of course. In order to be able to retransmit
a packet it must still be in the transmit buffer and the buffer
has a finite length. All windowing protocols set a maximum
'window size'. This means that there can be no more than 'x'
packets sent without a reply before the transmitter is forced to
wait for that reply else error recovery would not work. This is
no big deal at 1200 baud, but at 2400 and above it is really
SEAlink is a windowing protocol. It has as an added advantage
over WXmodem, for example, two important features: it uses 3 byte
CRC for increased reliability and it uses a window size of 6
rather than the 4 used by WXmodem. It is NOT 'network-friendly'.
What is 'network-friendly'? It is a design that recognizes and
honors XON/XOFF signals that are placed on a packet switching
network when that network (like PC Pursuit) becomes so busy that
it is nearly choking on data. When the network places an XOFF on
the line, a network-friendly recognizes it for what it is rather
than a coincidental configuration of bits in a byte of data and
stops sending data! It stops until it receives an XON from the
network. Why is that important? Well, it is my experience that
a huge number of subscribers now exists for PCP. Forcing a
network to exceed its ability to handle data could only crash the
network. PCP would not allow that. They have intelligent node
controllers that selectively will abort a 'hog' link that does
not honor its earlier 'request' to wait a little (via XOFF).
Thus, using a protocol that is not network-friendly is like
saying: "I don't care if I am a hog. And, if you don't like it,
then abort me." As usage continues to increase, the network will
oblige that attitude.
The result of being network-friendly is two fold in terms of
'hits' against performance: 1) while you are waiting for the
network to send you an XON you are not sending data and 2) there
are MANY extra bytes of control information that definitionally
must be sent along with your data.
Let me explain that last point as it is not obvious, I know.
XOFF and XON are simply bytes, just like the letter 'A' or the
digit '4'. If no data file contained those bytes then it would
be easy to implement a network-friendly protocol. Recall,
however, that it is almost always true that data is sent in some
form of archive or compressed format. The resulting bytes can
have ANY configuration despite what the un-archived or un-
compressed file looks like. In other words, the odds are
essentially 100% that the data files that you send consist of
probably many bytes that look like XOFF or XON. That cannot be
allowed to happen. The protocol finds all such bytes and
encapsulates them in what is called an escape sequence that
consists of a special byte (usually the DLE character) followed
by a 'folded' duplicate of the byte that needed to be camouflaged
(the XON or XOFF). Folding merely means that the byte is
transmogrified in some way (usually via being sent as a
compliment - XORed with all 1's). Further, the DLE character
itself must also be escape sequenced for this method to work. It
is a random process that results in indeterminate performance for
any particular file. That is, if a file had none of these three
special byte combinations in it, then the time to transmit it
would be minimal where a file that happened to have many of them
will have that many more bytes to send in order to escape
sequence it. In such a case the file would take longer to
transmit than the first. Same protocol, different performance.
On balance, the designers of SEAlink did an excellent job. The
performance of SEAlink is essentially as good as WXmodem yet it
is more reliable and uses the 'link' header. Why is SEAlink becoming
so popular? Because it is a protocol supported under a BBS
system called OPUS which is quickly replacing most of the old
FIDO systems all over the country. It is a good protocol.Because
it is not network-friendly it does not bother with (it doesn't have
to) escape coding anything. That is probably a fatal mistake to
its future particularly as the networks get crowded.
The next one of interest is called Zmodem. This is almost always
found as an external protocol. That means it is included in a
file (DSZ.EXE) that is shelled to by the host or terminal
communications program when it is needed. As such, it requires a
lot of memory compared to the internal protocols. But because of
that, it is easy to install as a protocol offering of many BBS
systems. There is another and more significant difference
between Zmodem and the other protocols we have already discussed
so far. Instead of being start-stop in nature, and instead of
being windowing, it is a streaming protocol. A streaming
protocol does not expect to get ANY ACK signals back from the
receiver until the transfer is complete and successful. If an
error occurs it will receive a NAK and it is up to the
transmitter to insure that it can recover from any NAK received.
Thus, because it is not a windowed protocol it never stops
transmitting unless there is an error. That means it should be
faster than even the windowing protocols.
Zmodem uses 32-bit CRC for reliability and it is network-friendly.
In some ways it is not very user-friendly, however. For example,
in every other protocol there is a way to
terminate the transfer should you wish to do so while it is in
progress. The usual manner is to press Cntl-X one or two times
and wait till the other end recognizes the abort request and
finally stops. In the case of Zmodem you must do so 5! times in
a row to stop it. I suggest that not 1 user in a thousand knows
that. It is a popular protocol as a result of its performance on
the packet switching networks. Incidentally, they
also escape sequenced a fourth byte - the SYN. It is for rather
obscure reasons and I believe a mistake.
Included in GT PowerComm 12.20 is the newest file transfer
protocol. It is called MegaLink. It uses 32-bit CRC, it is
network-friendly, is faster than Sealink, and like all the 'link'
named protocols it uses a header record that results in exact
size and proper time and date stamping of the resulting file when
received. Most interesting about MegaLink is how well it
performs at the very highest baud rates. Running comparative
tests of four different protocols, all sending the same 880K file
to the same machine and at 9600 baud, I obtained the following
WXmodem 60.4 % efficiency 580 cps
SEAlink 75.6 % 725 cps
Ymodem 77.6 % 744 cps
MegaLink 98.5 % 945 cps
In order, WXmodem did so poorly for two reasons: at 9600 baud its
window limit of 4 is the same as not having a windowing technique
at all. Second, there are ACK signals coming back for each
packet sent. In the 9600 baud arena, the transmission is only
9600 baud in one direction and only 300 baud in the other! It is
transparent, more or less, to the users as the modems
automatically change which direction is at 9600 baud based on the
volume of data that needs to be sent in each direction at any one
time. Further, while one character (the ACK itself at 300 baud
is not significant, the ACK/NAK response is actually either two
or three bytes rather than one as you might expect. The
additional byte(s) is for packet number (and it's compliment).
SEAlink is being driven about as fast as it can go. It is not as
fast as Ymodem because of the small window it uses (like WXmodem)
and because it has so many more characters to transmit because it
is network-friendly (escape sequences).
Ymodem is going as fast as it can. It is effected primarily
because of the start-stop nature of its function and the fact
that the ACK/NAKs are coming back at 300 baud. Here we see
clearly an indication that the days of the start-stop protocols
As an aside, Ymodem-G would have performed MUCH better because it
has no error control whatever, thus it has fewer bytes to
transmit and no turnaround delays. Remember, however, that error
correcting modems are only capable of insuring that the data sent
from one modem is received reliably by the other. As will be
seen in the discussion later about Zmodem's total failure,
Ymodem-G would not have reliably worked in this test.
It is interesting that Zmodem failed altogether at 9600 baud.
The reason is a little subtle and it leads to the next thing I
wanted to discuss anyway.
I earlier mentioned that the MNP and ARQ modems are able to strip
the start and stop bits from bytes, (they must, thus, be in
synchronous mode rather than asynchronous), and that they also
may use a form of compression beyond that for performance
reasons. I further stated that at 9600 baud the modem I was
using was able to perform at 1100 cps rather than 960. This may
have caused you to ponder: if the modem is connect to the
computer at 9600 baud that means the computer can only send 960
characters per second to the modem for subsequent transmission.
So how can the modem send it any faster than it receives it?
The answer is that it cannot do so. The method to use to obtain
these extraordinary performances is to connect your computer to
the modem at 19,200 baud and utilize a buffer in the modem to
match up the input with the output. Naturally, as the data is
arriving at the modem much faster than it is leaving, there must
be a way to stop the input. Well, you guessed it, we use flow
control just like the networks when they are getting choked. In
particular, we sense that the modem's Clear To Send signal is on
or off. When off, we stop sending data to it and when on, we
instantly start cramming data at it at 19,200 baud. In this way,
the modem is able to send data at 1100 cps. Naturally, the modem
must be able to control its CTS signal for this to work.
USRobotics HST is capable of doing so.
I showed you what happened to Zmodem when we tried to transfer to
it at in excess of 9600 baud - it failed. That is not entirely
the fault of Zmodem, however. Unless the receiving system is of
the AT class of computers you will probably find that regardless
of what kind of software you are using with it, the modem is
faster than the computer's ability to feed it or eat from it!!
Now that is amazing, isn't it? We now have modems that are paced
by the computer they are attached to instead of the other way
Incidentally, unless the receiving computer is connect to the
receiving modem at 19,200 instead of 9600 baud, and has
implemented some form of flow control to signal the sending modem
that it's buffer is full, 1100 cps transmissions to it will
naturally fail when the buffer is overflowed.
The entire AOH site is optimized to look best in Firefox® 3 on a widescreen monitor (1440x900 or better).
Site design & layout copyright © 1986- AOH
We do not send spam. If you have received spam bearing an artofhacking.com email address, please forward it with full headers to firstname.lastname@example.org.