OVERVIEW OF BELLCORE METROCORE\*(Tm NETWORK


A. Albanese, M. W. Garrett, A. Ippoliti,
M. A. Karr, M. Maszczak, and D. Shia

Bell Communications Research
Morristown, New Jersey 07960, USA
(201) 829-4291


The Bellcore METROCORE\*(Tm network is a test bed prototype of a
metropolitan area network (MAN) for exploring new network concepts
and technology applications in broadband communications.
The present design and status of the METROCORE network
research prototype are outlined.
This system operates at 150 Mb/s with the potential for
upgrading to 2.4 Gb/s.
A basic hardware architecture has been designed which consists
of a network interface (media access control (MAC) layer), and
several independent, modular units that interface to various services.
At the MAC layer, integrated services are supported by giving priority to
those traffic types requiring bounded delay.
The organization is modular, allowing components and services to be added
or improved while redesigning the minimum amount of hardware.
The software network architecture has also been designed and it includes
a network controller to configure, monitor, and administer the network.

The Bellcore METROCORE\*(Tm

METROCORE is a trademark of Bell Communications Research, Inc.

This paper will be presented at the IFIP WG 6.4 Workshop HSLAN'88,
April 14-15, 1988, Liege, Belgium.

network is a Metropolitan Area Network (MAN)
prototype using an enhanced version of Fasnet [LIMB82]
as its Media Access Control (MAC) protocol.
This prototype
serves as a test-bed for research in the integration of
diverse services onto a unified packet switched network.
The protocol provides special access for traffic with
delay constraints.
There is also a mechanism for ensuring
fairness among the active nodes.
Nodes are connected to each other by a dual bus network
similar to the architecture being considered by IEEE 802.6.
Rather than using passive taps as in the original Fasnet proposal,
active regeneration is used with optical fault bypass devices
and a stand by fiber optic link to insure fail-safty (Figure^1).



"Metrocore implementation with four nodes."


The hardware designed for each node includes a
MAC layer circuit which interfaces the node to the
network.
This exchanges packets across a specialized internal
communication bus with a variety of "service processors," (SPs)
which serve as interfaces to the services running on the network.
The internal bus is called the "Fast Packet Bus" (FPB), and
is implemented in a similar way to the backplane of a double VME card cage.
However, only the physical VME specification is used since
the VME electrical definition,
like most computer backplanes,
can not provide as much bandwidth as we desire for this application.
Each SP ideally occupies one board, allowing several services
to be present in a node.
In addition to the MAC circuit and the SPs, there
is a node controller circuit which keeps track of statistics
for billing, management, administration, and operations and takes care
of diagnostics and fault control.

The service processors we anticipate include services such as
telephone, television, computers, disk servers, and a LAN interconnection.
The LAN service processor was designed first since
it is an important service to explore in the near term,
and because this allows us to connect (indirectly) anything that
already uses Ethernet.
This processor consists of five modular circuits implemented in
four boards that may be modified for use in other SPs.
This processor implements a transparent
protocol which encapsulates Ethernet packets for transport
across Fasnet and retransmission on a remote Ethernet.
Other suggested protocols for service integration for interconnected
LANs [FRAT87] and routing for interconnected MANs [STRG87a,b]
have not been implemented yet.

Figure^2 shows the system block diagram.  Each component is described
below, followed by brief discussions of the software architecture,
and a Gb/s network enhancement.





"Metrocore node."


"Optical Transmission System"
We are presently using an off-the-shelf transmission system
with single mode optical fibers at 150 Mb/s and
a wavelength of 1.3 &mu m&.
This system was chosen because it provides a virtual clear channel,
and can tolerate the long strings of consecutive zeros which occur between
the head-end node (which sends blank packets) and the node that uses
a packet by filling in the empty data field.
The system does its own scrambling and clock recovery.
Connections between nodes are point-to-point, but
an optical fail-safe devices may be inserted to make the node appear
transparent in case of a local node failure [ALBA82, LOH86].
A stand by fiber optical link allows network reconfiguration in case one of
the links is not functioning.
"MAC Chip"
This chip is the major component of the MAC circuit.
The MAC chip executes the media access layer protocol and thus governs the
flow of data onto the network.
Two chips are needed for the two directions of traffic on the network.
The design is implemented as a
semi-custom chip manufactured by Motorola, using emitter coupled logic
(ECL) technology for high-speed performance.
The maximum speed was measured at 310^Mb/s.
The MAC chip replaces a whole board of SSI/MSI chips in a
previous prototype.

The MAC chip reads the various fields of the packet header to
execute proper transmission, and signals interface circuits
for reception of packets.
Other functions include synchronization of the chip to the incoming packet
stream, generation of timing signals and 1 &times& 16 serial to parallel
conversion.
Each chip contains the circuitry needed to execute special
functions associated with the end nodes.
Thus, as a fail-safe mechanism,
a middle node may become the head-end if the original head-end fails or
if the network becomes severed.

The chip was designed and simulated on a Daisy CAD workstation using
a library of components made available from Motorola.
The Motorola
computer system was then used for the final timing simulation, race-condition
test, placement and routing of the metal paths.
This chip is equivalent to 2500 ECL gates and comes in a 149-pin PGA
package.  98% of the available circuitry and pins were used.

"MAC Circuit"
The MAC circuit consists of MAC chips, address recognition circuits
(address filters) , bus arbiters, a head-end enable detection circuit
(activity detect), a head-end access field generator,
ECL/TTL converters, control circuitry, and a power-up reset
circuit (see Figure^3).
There is generally two of everything due to the two unidirectional
busses used in Fasnet.



"Medium Access Control Circuit."


Each MAC chip feeds incoming data, in a 16-bit parallel word, into an
address recognition circuit which compares the address field
of the packet to a group address and a single node address.
Thus a node may be addressed individually or as part of a multi-cast group.
The link-layer address is structured so that the first bit (most
significant) indicates group or node address (0/1), the next
11 bits identify the node or group, and the last four constitute
a sub-address indicating the intended service processor.
The address circuit is implemented
in TTL logic with ECL/TTL converters between the MAC chip and itself.
Data and control information is passed on to the service processors
through the two outgoing paths to the Fast Packet Bus.

Four bus arbiter circuits are included for the four incoming paths
from the FPB.
These paths carry data destined for transmission
from the service processors to the MAC chips.
There are "data" and "voice" paths for each of two MAC chips.
(The data and voice paths are not limited to those services but are used
to differentiate between traffic which can tolerate network delays (low
priority) and those which can not (high priority).
The paths are separated
because the two Fasnet lines operate independently, and the two cycles
(for different priority traffic) are also independent.
The bus arbiters choose which of competing service processors gets
next access to a given path of the FPB, and therefore to the network.
The choice is made using a simple round-robin scheme.
Processors using the same FBP path
are not further prioritized, although this could be done
using a more complex bus arbiter circuit.

An activity detection circuit monitors the
synchronization at the receiver to discover whether
the node is to be head-end or not.
The head-end node will
have a desable receiver for the line that it heads and thus
would always remain head-end.
A middle node (Figure^1) will become
head-end in the event of a failure just upstream of itself.
If the receiver does not provide a carrier detect signal,
the MAC synchronization function could be used:
When the MAC fails to synchronize for 2 msec (for example),
it turns on its head-end function, and then monitors the incoming
data stream for a valid Fasnet synchronization pattern.
Reception of such a pattern would then turn off the head-end
function for that node returning to its middle node function.

The access field generator circuit (see Figure^3), implemented as
a PLA (programmable logic array) is required if a node is a head-end.
This takes care of "turning around" the start and end bits of the Fasnet
access field, which in turn operate the network cycles.
Every
node which might be head-end must have this.
It is possible
to avoid putting this circuit in every node; the tradeoff being
that all nodes between the fault and the first head-end-capable
node would go down.

The MAC chips require information to determine system parameters such
as packet size etc.
These, as well as the node addresses are stored
in a small register and are generally programmed on power-up by a control
circuit in the node.

The remaining functions in Figure^3 are the power-up reset circuit
which holds reset high until the power supply voltage stablizes;
and a local oscillator which serves as system clock in head-end nodes.
"Fast Packet Bus"
The backplane of the card cage has two 96-pin connectors which carry
the two incoming paths to the processors, and the four outgoing paths
to the MAC circuit.
Several pins also supply power and ground (TTL) to the boards.
In addition, there is a synchronous serial line which allows the
node controller SP to down load and read the stored MAC parameters.

On each incoming path, there are 16 bits of data; a four bit sub-address
which identifies the processor to receive the packet; a four bit control
field which comes from the beginning of the information part of the
packet, and may be used optionally to give the SP a code already parsed
from the packet.  A signal called
"Packet Ready" indicates the beginning of a new, valid packet, and
two clock signals (at about 10 MHz and 20 MHz) are given from which
the MAC circuit's two phase clock (&phi&1 and &phi&2) are derived.

On each outgoing path there are again, 16 bits of data.
A strobe
signal from the MAC chip prompts each word from the processor.
Each processor (up to four) has its own set of handshaking lines with
the bus arbiter.
If more than four processors use a single path, then this handshaking
may be daisy-chained on every fourth processor.
The processor wishing to
transmit raises its "request" line; the bus arbiter selects the next
requesting processor in turn and responds with an "acknowledge."
The processor
then raises its first word of data together with
"data ready."
When the MAC gets access to the network, it will strobe the processor
for the following words.

This type of internal node bus uses a lot of pins (192).
The choice could be made to reduce the number of wires by time multiplexing
signals of the various buses.
This however, would require a large and
complex buffer on the MAC circuit, which would be more difficult
than having a large backplane bus.
The present FPB has the advantage
of not presenting any bottleneck at all beyond that caused by
the node access to the network (i. e. same-priority, same-direction packets
must compete for the network at the MAC chip anyway).
"Node Controller Circuit"
When a node is powered up, all processors as well as the
MAC circuit must know the node's address.
This circuit is responsible for
providing this information as well as certain MAC chip parameters.
This circuit would also monitor
statistics about the node's activities for billing etc.
There may
be a central control node for the whole network with which this circuit would
communicate to reconfigure addresses, diagnose network failures or exchange
statistical information.
The controller circuit would
therefore be able to construct Fasnet packets and transmit and receive
like any other service processor.
All the functions of the node controller were implemented in software that run
in the 68000 of the Ethernet Circuit in the LAN processor (see Figure^4).
"LAN Service Processor"
This processor does the function of a LAN/MAN bridge (see Figure^4),
it carries packets transparently between
two Ethernet LANs which may be separated by considerable distance.
A description of the functionality and philosophy of design
may be found in [ALBA86, DEGR86].
The LAN service processor consists of five circuits implemented on four boards,
which could probably be integrated down to one board using a VLSI technology
like that used for the MAC chip.
Each circuit performs a special task:
(1) Input Buffer, (2) Output Buffer,
(3) Depacketizer, (4) Packetizer, and (5) the Ethernet Circuit
and Node Controller.
These are general building blocks for any service processor.
The buffers are rather large, since a LAN is expected to generate more
traffic than a computer interface would, for example, but the design
can be scaled down and used in any case.
The packetizer and depacketizer
are bit-sliced processors offering very high throughput for communications,
but weak processing power.
The Ethernet circuit is a specialized processor
with potential for complex processing.
It can be tailored for another
purpose by substituting another interface for the Ethernet dependent
parts.
This processor is not, however, suited as a general purpose
computer because of the limited memory configuration.
Rather, it should be seen as a powerful dedicated controller.



"LAN Service Processor and Node Controller."


.H 2 "Input and Output Buffer Circuits"
The input buffer connects to the incoming paths of the FPB.
Each of the two large
circular buffers are synchronized to their respective MAC clocks.
Read
and write pointers are generated for each buffer, and the circuit
monitors whether the buffers are full or empty by
comparing the values of the two pointers.
The buffers themselves are
16K &times& 16 RAM chips with very fast (35 nsec) access times.
In this circuit, the 2-phase clock from the FPB is decoded and used.

On the other side, the buffers feed the depacketizer circuit, which uses
the clock of whichever input buffer it is reading at the time.
Two finite state machines (PLAs) which are referred to as the
"producer" and "consumer," govern the flow of data into
and out of the buffer, respectively.

The output buffer takes data from the packetizer circuit and stores it for
transmission on the outgoing FPB.
Many packets may be stored in
the single buffer along with information about which outgoing path(s)
each one is to be transmitted on.
Packets destined for both directions
on the network must be transmitted on each direction separately, since
the two MAC chips
cannot possibly be synchronized.
"Depacketizer and Packetizer Circuits"
The depacketizer and packetizer circuits are almost identical in hardware
but different in software.
The processor consists of an AMD
2910 bit-sliced sequencer and an AMD 2116 ALU chip.
The function performed
involves stripping Fasnet headers from incoming packets and storing the
Ethernet fragments away in a DRAM on the Ethernet Processor circuit, and
then indicating to that circuit when a packet is completely reconstituted
and ready for retransmission on Ethernet.

The depacketizer circuit also contains an Ethernet address filter, which
now consists of a hashing algorithm, but would ideally be implemented
as a Content Addressable Memory (CAM), which may
be available in the future.
Ethernet packets are filtered so that only the ones intended for local
Ethernet nodes are retransmitted.
Others are dropped.

The packetizer is the same processor as used in the depacketizer circuit,
except that the task here is to transform whole Ethernet packets into
bursts of Fasnet packets.
The Ethernet packets are completely and
transparently encapsulated.
Addresses are not converted.  Headers are
added indicating Fasnet addresses and control information.
An Ethernet
address filter lets only those packets not destined for local Ethernet
nodes to escape to the MAN.
"Ethernet Processor Circuit"
This circuit consists mainly of two dynamic RAM controller chips, with their
respective banks of DRAM, which interface to the packetizer and depacketizer
circuits;
an AMD Local Area Network Controller for Ethernet (LANCE) chip; and
a 68000 microprocessor chip.
The 68000 takes care of initialization and
memory management, and the LANCE, which is itself a dedicated
microprocessor, takes care of getting traffic onto and
off of the Ethernet.
In addition, there are two UARTs on the 68000 which
allow a terminal and a modem to be connected to the node
for diagnostic purposes.
"Network Software Architecture"
For network management and operation purposes each network node
is structured according to the Open System Interconnection (OSI)
Reference Model to support communications between all the
nodes and the network controller.
Figure^5 shows the software network architecture.
The Management Control Unit consists of network management
and node management units.


.FG "Network Software Architecture."


The Network Management unit monitors the status of the network, collects
statistical data from each node and  stores it on a disk, provides hard
copies of network statistics, and handles network set-up
and configurations of the Ethernet LAN interconnections via a human
administrator.

The Node Management Unit performs self-test on power up and upon receiving 
requests from the Network Management Unit.
It also provides remote reset and 
download capabilities to facilitate software updates and maintenances.
It maintains configuration data and collects statistical data about local
operations.
Upon receiving requests from the Network Management Unit,
it reports the most recent information about the node operations.
It also monitors any fault events on the node.
When a fault happens,
it attempts to either correct the fault problem or to collect more information.
In any case the fault is reported to the Network Management Unit along with
the result of attempted recovery.

In order to communicate reliably to each other, the Network Management Unit
and the Node Management Unit require the Transport layer services.
The Transport layer uses the Data Link layer services to handle
connection setup and termination, flow control,
multiplexing, fragmentation, and error detections and corrections.
The Data Link layer deals with sending and receiving packets to and from
other nodes without bit pattern errors. 

Please note that these reliable 
services are provided only for the management purposes.
They do not interfere with transparent wiring services offered
by the LAN processor.
In another word, these protocols 
are logically separated from the transparent wiring services.
"Multi-Gigabit/second MAC Circuit"
This work investigates how to share the multigigabit/second
transport capabilities offered by lightwave systems among a multitude
of users and how to integrate the many broadband and narrowband services
that a user may require.
We have demonstrated a 1.2 Gb/s system to provide packet
communications in the distributed switching system shown in Figure^6.
The system provides multiple access at 1.2 Gb/s and consists
of a unique node architecture that combines available
lightwave, gallium arsenide, and silicon technologies [KARR87].


"Gigabit/second access interface."


Gallium arsenide (GaAs) multiplexer and demultiplexer ICs represent
today's most advanced technology.
Silicon very large scale integrated (VLSI) circuits have up to 100,000
transistors on a chip,
but generally do not operate in the Gb/s range.
Commercially available GaAs ICs will work above 1 Gb/s
and their circuit densities are rapidly advancing.

We use 8:1/1:8 multiplexer/demultiplexer GaAs ICs made by
Gigabit Logic Inc., together
with our silicon MAC IC to divide the 1.2 Gb/s bandwidth into
8 channels of 150 Mb/s.
These devices are considered "medium-scale integration" (MSI)
and contain about 200 gates.
The data is controlled by a packet switching protocol using control fields
only on one of the channels [GARR86].
The protocol, based on Fasnet, is executed by the Media Access Control (MAC)
chip.
Other channels carry only data and are synchronized to the control channel.
Thus each user has access to the full 1.2 Gb/s capacity on a per-packet basis.

Some gallium arsenide integrated circuit components are becoming available
in the 2 Gb/s range.
A 16:1 multiplexer and a 1:16 demultiplexer operating at 2.4 Gb/s were built
which could
upgrade the 150 Mb/s MAC circuit to 2.4 Gb/s.
Metropolitan area network technology enables users currently
connected to local area networks, geographically limited
within one building or office complex,
to extend their range of networking interconnectivity.
Large corporations that have several LANs operational at separate sites
interspersed within an urban area will be able to utilize their resources
more effectively and make them more widely available through the use of
MAN networking.

The technology is demonstrated in a laboratory test bed.
Working prototypes demonstrate that present technology is capable to
interconnect 10 Mb/s LANs over MANs implemented at 150 Mb/s
and to be upgraded to 1.2 Gb/s in the near future.

"REFERENCES"


[ALBA82]
A. Albanese,
"Fail-Safe Nodes for Lightguide Digital Networks,"
Bell System Technical Journal, Vol 61, Nr 2, pp 247-256, February 1982.
[ALBA86]
A. Albanese, G. DeGrandi, M.W. Garrett, "An Architecture for
Transparent MAN/LAN Gateways," IEEE International Conference on
Communications, Toronto, Canada, June 1986.
[DEGR86]
G. DeGrandi, M.W. Garrett, A. Albanese, T.H. Lee, "The Design and
Implementation of a Transparent MAN/LAN Gateway," Proc.
EFOC/LAN'86, Information Gatekeepers Inc., Amsterdam June 1986.
[FRAT87]
L. Fratta and A. Albanese,
"Service Integration for Interconnected LANs,"
The Third International Conference on Data Communication Systems and Their
Performance, Rio de Janeiro, Brazil, June 22, 1987.
[GARR86]
M.W. Garrett, J.O. Limb, A. Albanese, "Multiple Gb/s Fiber-Optic
Metropolitan Area Network," IEEE International Conference on
Communications, Toronto, Canada, June 1986.
[KARR87]
M. Karr, A. Albanese, M. Garrett, K.W. Loh, M. Maszczak,
"Experimental Gigabit Packet Communications Network,"
IEEE Optical Fiber Communications Conference,
Reno NV, February 1987.
[LIMB82]
J.O. Limb, C. Flores, "Description of Fasnet - A
Unidirectional Local Area Communications Network," Bell
Syst Tech Journal Vol 61, No 7, p 1413, Sept 1982.
[LOH86]
K. W. Loh, M. Karr, A. Albanese, W. C. Young, L. Curtis,
J. Baran, and L. McCaughan,
"Fail-Safe Nodes for Fiber Networks,"
Optical Fiber Conference, Atlanta, Georgia, February 24-26, 1986.
[STR87a]
L. Strigini, L. Fratta, and A. Albanese,
"Multicast Services on High-Speed Interconnected LANs,"
1987 Workshop on High Speed LANs,
Aachen, West Germany,
February 16-17, 1987.
[STR87b]
L. Strigini, L. Fratta, and A. Albanese,
"Explicit Offset Routing for Interconnecting High-Speed Networks,"
EFOC/LAN 87,
Basel Switzerland,
June 3-5, 1987.