CN103916326B - System, method and equipment for data center - Google Patents

Abstract

A system, method and apparatus for a data center. In an embodiment of the present disclosure, the device includes a first edge device that may have a packet processing module. The first edge device is configured to receive packets. The packet processing module of the first edge device may be configured to generate a plurality of cells based on the packet. The second edge device has a packet processing module configured to reassemble the packet based on the plurality of cells. A multi-stage switch fabric can be coupled to the first edge device and the second edge device. A multilevel switch fabric can define a single logical entity. A multi-stage switch fabric can have multiple switch modules. Each switch module of the plurality of switch modules has a shared storage device. The multi-stage switch fabric can be configured to switch multiple cells such that multiple cells are sent to the second edge device.

Description

Systems, methods and devices for data centers

This application is a divisional application of a Chinese patent application with an application date of September 11, 2009, an application number of 200910246898.X, and an invention title of "System, Method and Equipment for Data Center".

Cross references to related patent applications

This patent application claims priority to U.S. Patent Application No. 61/098516, filed September 19, 2008, entitled "Systems, Apparatus and Methods for a Data Center" and interests; also claiming U.S. Patent Application No.61/096209 entitled "Methods and Apparatus Related to Flow Control within a Data Center (involving methods and devices for flow control in a data center)" and filed on September 11, 2008 priority and interest; both are fully incorporated herein by reference.

This patent application is entitled "Methods and Apparatus for Transmission of Groups of Cell via a Switch Fabric (methods and devices for transmitting cell groups via a switch fabric)" and filed on December 24, 2008, U.S. Patent Application No.12/ Partial continuation of 343728; a U.S. patent filed on December 29, 2008 entitled "System Architecture for a Scalable and Distributed Multi-Stage Switch Fabric (system architecture for scalable and distributed multi-stage switch fabric)" Application No. 12/345500 is a continuation in part; is entitled "Methods and Apparatus Related to a Modular Switch Architecture (methods and devices involving modular switch architecture)" and filed on December 29, 2008, U.S. Patent Application No. Continuation-in-Part of .12/345502; is entitled "Methods and Apparatus for Flow Control Associated withs Multi-Stage Queue" and filed September 30, 2008 , claiming priority to U.S. Patent Application No. 6I/096209, filed September 11, 2008, entitled "Methods and Apparatus Related to Flow Control within a Data Center" Continuation-in-Part of U.S. Patent Application No. 12/242224 of interest; is titled "Methods and Apparatus for Flow-Controllable Multi-Staged Queues (method and apparatus for flow-controllable multi-stage queues)" and filed in 2008 Submitted on September 30, 2008, U.S. Patent Application No. Continuation-in-Part of US Patent Application No. 12/242230 having priority and benefit of 61/096209. Each of the above-mentioned applications is hereby incorporated by reference in its entirety.

This patent application is also titled "Methods and Apparatus Related to Any-to-Any Connectivity within a Data Center" and filed on June 30, 2009, U.S. Patent Application No. Partial continuation of 12/495337; U.S. Patent Application No. entitled "Methods and Apparatus Related to Lossless Operation within a Data Center" and filed on June 30, 2009. Continuation-in-Part of 12/495344; U.S. Patent Application entitled "Methods and Apparatus Related to Low Latercy within a Data Center" and filed June 30, 2009 Partial continuation of No.12/495358; is titled "Methods and Apparatus Related to Flow Control within a Data Center Switch Fabric" and filed on June 30, 2009 Continuation-in-Part of U.S. Patent Application No. 12/495361; entitled "Methods and Apparatus Related to Virtualization of Data Center Resources" and filed June 30, 2009 A continuation-in-part of U.S. Patent Application No. 12/495,364. Each of the above-mentioned applications is hereby incorporated by reference in its entirety.

technical field

Embodiments relate generally to data center equipment, and more specifically to architectures, devices, and methods for data center systems having switch cores and edge devices.

Background technique

Known architectures for data center systems involve unwieldy and complex approaches, increasing the overhead and latency of such systems. For example, some known data center networks consist of three or more switching layers, where Ethernet and/or Internet Protocol (IP) packet processing is performed at each layer. Packet processing and queuing overhead is unnecessarily repeated at each layer, directly increasing overhead and end-to-end latency. Similarly, such known data center networks do not typically scale in a cost-effective manner: for a given data center system, an increase in the number of servers typically requires additional ports, resulting in more ports being added at each tier of the data center system. device of. Such poor scalability adds overhead to such data center systems.

Accordingly, a need exists for improved data center systems including improved architectures, devices and methods.

Contents of the invention

In one embodiment, a communications device includes a first edge device that may have a packet processing module. The first edge device may be configured to receive packets. The packet processing module of the first edge device may be configured to generate a plurality of information elements based on the packet. The second edge device may have a packet processing module configured to reassemble the packet based on the plurality of cells. A multi-stage switch fabric can be coupled to the first edge device and the second edge device. The multilevel switch fabric can define a single logical entity. The multi-stage switch fabric may have multiple switch modules. Each switch module of the plurality of switch modules has a shared storage device. The multi-stage switch fabric can be configured to switch multiple cells such that multiple cells are sent to the second edge device.

According to an aspect of an embodiment of the present disclosure, there is provided a communication device, including: a switching core, the switching core defines a single logical entity and has a multi-stage switching structure physically distributed across multiple racks, the multi-stage a switch fabric having a plurality of input ports and a plurality of output ports, the switch core is configured to be coupled to a plurality of peripheral processing devices via the plurality of input ports and the plurality of output ports, the switch core is configured to Non-blocking connectivity is provided at line rate between a first peripheral processing device arranged within a first bay and a second peripheral processing device arranged within a second bay.

According to an embodiment of the present disclosure, the plurality of peripheral processing devices include at least one peripheral processing device with virtualized resources and at least one peripheral processing device without virtualized resources.

According to an embodiment of the present disclosure, the number of the plurality of input ports and the plurality of output ports is greater than 1000, each of the plurality of input ports and each of the plurality of output ports Both are configured to operate at a speed of no less than 10Gb/s.

According to an embodiment of the present disclosure, both the first peripheral processing device and the second peripheral processing device are one of a storage node device, a computing node device, a service node device or a router.

According to an embodiment of the present disclosure, the plurality of peripheral processing devices includes a third peripheral processing device, and the switch core is configured to communicate at a line rate between the second peripheral processing device and the third peripheral processing device. providing non-blocking connectivity, the switch core configured to receive a first packet associated with the first peripheral processing device, the switch core configured to sequence based on cells associated with the first packet sending a second packet to the second peripheral processing device and sending a third packet to a third peripheral processing device, the multi-stage switch fabric configured to transmit from an input port of the plurality of input ports to the output port The output port in sends the cell.

According to an embodiment of the present disclosure, both the first peripheral processing device and the third peripheral processing device are one of a storage node device, a computing node device, a service node device, or a router; and the second peripheral processing device is at least one of a firewall device, an intersection detection device, or a load balancing device.

According to another aspect of the embodiments of the present disclosure, a communication device is provided, including: a switch core, the switch core has a multi-stage switch structure physically distributed across multiple racks, and the multi-stage switch structure has multiple An input port and a plurality of output ports, the switching core is configured to be coupled to a plurality of peripheral processing devices via the plurality of input ports and the plurality of output ports, the switching core is configured to be at a line rate for the Each peripheral processing device in the plurality of peripheral processing devices provides connectivity to each remaining processing device in the plurality of peripheral processing devices, thereby enabling each output port in the plurality of output ports to be controlled by the Each of the plurality of peripheral processing devices is equally accessed via one of the plurality of input ports.

According to an embodiment of the present disclosure, the plurality of peripheral processing devices include at least one peripheral processing device coupled to the switching core via an Ethernet connection and at least one peripheral processing device coupled to the switching core via a non-Ethernet connection Processing device.

According to an embodiment of the present disclosure, the plurality of peripheral processing devices includes at least one peripheral processing device using layer 3 routing and at least one peripheral processing device using layer 4 to layer 7 devices.

According to another aspect of the embodiments of the present disclosure, there is provided a communication device, including: a switch core, the switch core defines a single logical entity and has a multi-level switch structure, and the multi-level switch structure has multiple rack-distributed stages, the plurality of stages collectively have a plurality of input ports and a plurality of output ports, the switching core is configured to be coupled to a plurality of peripheral processing means, the switch core configured to allow a plurality of cells associated with a packet to enter the plurality of cells when delivery of the plurality of cells can be substantially guaranteed without loss through the multi-stage switch fabric input port of the plurality of input ports.

According to one embodiment of the present disclosure, the plurality of peripheral processing devices includes a first peripheral processing device configured to communicate with a Fiber Channel protocol and a second peripheral processing device configured to communicate with a Fiber Channel over Ethernet protocol.

According to an embodiment of the present disclosure, the multi-stage switching fabric is configured as a deterministic network.

According to an embodiment of the present disclosure, the multi-stage switch fabric is configured as a deterministic network, so that when the plurality of cells can be transmitted to one output port among the plurality of output ports at a predetermined time, the The multi-stage switch fabric allows the packet to enter the input port.

According to an embodiment of the present disclosure, the switch core is configured to send a plurality of information elements associated with the packet from the input port to a first output port and a second output port of the plurality of output ports , without performing packet loss processing at at least one of the plurality of stages of the multi-stage switch fabric.

According to an embodiment of the present disclosure, the switch core includes a plurality of edge devices coupled to the multi-stage switch structure via the plurality of input ports and the plurality of output ports, and the plurality of edge devices are coupled to To the plurality of peripheral processing devices, each edge device of the plurality of edge devices is configured to receive the packet and define the plurality of information elements based on the packet.

According to an embodiment of the present disclosure, the switch core is configured to send a plurality of packets with the associated cells without performing packet loss processing at at least one of the plurality of stages.

According to another aspect of the embodiments of the present disclosure, there is provided a communication device, including: a switch core, the switch core defines a single logical entity and has a switch fabric, and the switch fabric has multiple physical distribution across multiple racks stages, the multi-stage switch fabric has a plurality of input ports and a plurality of output ports, the switch core is configured to be coupled to a plurality of peripheral processing devices via the plurality of input ports and the plurality of output ports, The switch core is configured to receive a packet from an input port of the plurality of input ports, the switch core is configured to pass from the input port to an output port of the plurality of output ports via the plurality of stages A plurality of cells associated with the packet is transmitted without performing packet loss processing at at least one of the plurality of stages of the switch fabric.

According to one embodiment of the present disclosure, the multistage switch fabric is configured as a deterministic network such that only when delivery of a plurality of cells associated with a packet within the switch fabric can be substantially guaranteed without loss , to allow packets from input ports in the plurality of input ports.

According to an embodiment of the present disclosure, the output port is a first output port, and the switch core is configured to transmit from the input port to the first output port and the second output port among the plurality of output ports Sending the plurality of cells associated with the packet.

According to an embodiment of the present disclosure, the switch core includes a plurality of edge devices coupled to the multi-stage switch structure via the plurality of input ports and the plurality of output ports, and the plurality of edge devices are coupled to To the plurality of peripheral processing devices, each edge device of the plurality of edge devices is configured to receive the packet and define the plurality of information elements based on the packet.

According to another aspect of an embodiment of the present disclosure, there is provided a communication device, including: a switch core, the switch core defines a single logical entity and has a multi-stage switch structure configured as a deterministic network, the multi-stage switch structure has a plurality of input ports and a plurality of output ports, the switch core is configured to be coupled to a plurality of peripheral processing devices via the plurality of input ports and the plurality of output ports, the switch core is configured to receive from the An input port of a plurality of input ports receives a packet from which the switch core is configured to transmit a plurality of cells associated with the packet to an output port of the plurality of output ports.

According to an embodiment of the present disclosure, the multi-stage switching fabric is physically distributed across multiple racks.

According to one embodiment of the present disclosure, the multistage switch fabric is configured as a deterministic network such that only when delivery of a plurality of cells associated with a packet within the switch fabric can be substantially guaranteed without loss , to allow packets from input ports in the plurality of input ports.

According to an embodiment of the present disclosure, the multi-stage switch fabric is configured as a deterministic network such that when the plurality of cells associated with packets can be transmitted to one of the plurality of output ports at a predetermined time When outputting a port, the switch core receives packets from an input port of the plurality of input ports.

According to an embodiment of the present disclosure, the switch core includes a plurality of edge devices coupled to the multi-stage switch structure via the plurality of input ports and the plurality of output ports, and the plurality of edge devices are coupled to To the plurality of peripheral processing devices, each edge device of the plurality of edge devices is configured to receive the packet and define the plurality of information elements based on the packet.

According to an embodiment of the present disclosure, the switch core is configured to send a plurality of cells associated with the packet from the input port to the output port via a plurality of stages of the multi-stage switch fabric, and Packet loss processing need not be performed at at least one of the plurality of stages.

According to another aspect of the embodiments of the present disclosure, a communication device is provided, including: a switch core, the switch core has a multi-stage switch structure physically distributed among multiple racks, and the multi-stage switch structure has a plurality of input buffers and a plurality of output ports, the switch core configured to be coupled to a plurality of edge devices; and a controller that does not require software during operation but is implemented in hardware and requires software implementation during configuration and monitoring , the controller is coupled to the plurality of input buffers and the plurality of output ports, the controller is configured to when congestion at one of the plurality of output ports is foreseen and at the A flow control signal is sent to one of the plurality of input buffers before congestion within the switch core occurs.

According to an embodiment of the present disclosure, the controller is configured to perform end-to-end on the input buffer and the output port independently of intra-fabric flow control for the multi-stage switch fabric of the switch core flow control.

According to an embodiment of the present disclosure, the controller is configured to perform end-to-end flow control on the input buffer and the output port independently of the flow control on the plurality of edge devices.

According to an embodiment of the present disclosure, a plurality of peripheral processing devices configured to be coupled to the plurality of edge devices, the controller configured to control the The input buffer and the output port perform end-to-end flow control.

According to an embodiment of the present disclosure, the controller is configured to perform end-to-end flow control such that cells are buffered at the input buffer for a period of time before being sent to the output port, the time being equal to The end-to-end flow control is associated.

According to an embodiment of the present disclosure, the controller is configured to be independent of the cell segment buffered at one stage of the multi-stage switch fabric and independently of the cell segment buffered at one of the plurality of edge devices. performing end-to-end flow control on cells buffered at said input buffer.

According to one embodiment of the present disclosure, the controller is configured to perform end-to-end flow control on cells buffered at the input buffer independently of flow control mechanisms associated with Ethernet.

According to another aspect of the embodiments of the present disclosure, there is provided a communication device, including: a switch core, the switch core has a multi-stage switch structure physically distributed among multiple racks, and the multi-stage switch structure is divided into configured to receive a plurality of cells associated with a packet and configured to switch a plurality of cell segments based on the plurality of cells; a plurality of edge devices coupled to the switching core, one of the edge devices an edge device configured to receive the packet, the edge device configured to send the plurality of cells to the multi-stage switch fabric; and a controller coupled to the multi-stage switch fabric, the controller being It is configured to perform flow control on the plurality of cells independently of flow control for the plurality of edge devices and intra-fabric flow control for the multi-stage switch fabric.

According to one embodiment of the present disclosure, the controller does not require software during operation but is implemented in hardware, and requires software implementation during configuration and monitoring.

According to an embodiment of the present disclosure, the multi-stage switch fabric has a plurality of input buffers and a plurality of output ports, and the controller is configured to, when congestion at one of the plurality of output ports is eliminated, A flow control signal is sent to one of the plurality of input buffers in anticipation and before congestion within the switch core occurs.

According to an embodiment of the present disclosure, the multi-stage switch fabric has a plurality of input buffers and a plurality of output ports, the controller is configured to be independent of the flow control mechanism associated with End-to-end flow control is performed on cells buffered at one of the input buffers.

According to another aspect of the embodiments of the present disclosure, there is provided a communication device, including: a switch core, the switch core has a multi-level switch structure; a plurality of first peripheral processing devices, coupled to the In the multi-level switch structure, each peripheral processing device in the first plurality of peripheral processing devices is a storage node with virtualized resources, and the virtualized resources of the first plurality of peripheral processing devices are jointly defined by the virtual storage resources interconnected by the switch core; and a second plurality of peripheral processing devices coupled to the multi-level switch fabric through a plurality of connections with protocols, each peripheral in the second plurality of peripheral processing devices The processing device is a storage node having virtualized resources of the second plurality of peripheral processing devices collectively defining virtual computing resources interconnected by the switching core.

According to one embodiment of the present disclosure, each peripheral processing device of the first plurality of peripheral processing devices has virtualized resources, and each peripheral processing device of the first plurality of peripheral processing devices is configured such that its virtualized resources capable of being replaced by virtualized resources from remaining peripheral processing devices in the first plurality of peripheral processing devices; and each peripheral processing device in the second plurality of peripheral processing devices has a virtualized resource, the Each peripheral processing device of the second plurality of peripheral processing devices is configured such that its virtualized resources can be replaced by virtual resources from remaining peripheral processing devices of the second plurality of peripheral processing devices.

According to one embodiment of the present disclosure, the first plurality of peripheral processing devices is associated with a packet-based communication protocol and is associated with a security protocol; and the second plurality of peripheral processing devices is associated with a packet-based communication protocol and is associated with a associated with a security protocol.

According to another aspect of the embodiments of the present disclosure, a communication device is provided, including: a switching core, the switching core has a multi-level switching structure, and the switching core is configured to be logically divided into a first virtual switching core and a second virtual switching core. a virtual switch core; a plurality of peripheral processing devices coupled to the multi-level switch fabric, the plurality of peripheral processing devices having a first subset of peripheral processing devices operably coupled to the first virtual switch core and a second subset of peripheral processing devices operatively coupled to the second virtual switch core.

According to one embodiment of the present disclosure, the switching core is configured such that the first virtual switching core and the second virtual switching core are administratively managed independently of each other.

According to an embodiment of the present disclosure, the switching core is configured such that the first virtual switching core has a bandwidth independent of the bandwidth of the second virtual switching core.

According to one embodiment of the present disclosure, the switching core is configured such that the first virtual switching core has bandwidth and administrative management independent from that of the second virtual switching core.

According to an embodiment of the present disclosure, the switching core is configured such that the first virtual switching core operates using a layer 2 protocol, and the second virtual switching core operates using a layer 2 protocol and a layer 3 protocol.

According to one embodiment of the present disclosure, the first peripheral processing device subset has virtual resources, and the second peripheral processing device subset has virtual resources.

According to an embodiment of the present disclosure, the first peripheral processing device subset includes a peripheral processing device that is one of a computing node, a storage node, a service node device, and a router, and includes a computing node, a storage node, a service node device and a peripheral processing device of the remaining one of the routers; and the second subset of peripheral processing devices includes a peripheral processing device that is one of a computing node, a storage node, a service node device, and a router, and includes computing nodes, storage A peripheral processing device of the remaining one of the node, the service node device and the router.

Description of drawings

Figure 1 is a system block diagram of a data center (DC) according to one embodiment.

Figure 2 is a schematic diagram illustrating an example of a portion of a data center with arbitrary connectivity, according to one embodiment.

Figure 3 is a diagram illustrating logical groups of resources associated with a data center, according to one embodiment.

Figure 4A is a schematic diagram illustrating a switch fabric that can be included in a switch core, according to one embodiment.

FIG. 4B is a schematic diagram illustrating a swap table that can be stored in the memory module shown in FIG. 4A according to one embodiment.

Figure 5A is a schematic diagram illustrating a switch fabric system, according to one embodiment.

Figure 5B is a schematic diagram illustrating an input/output module, according to one embodiment.

Figure 6 is a schematic diagram illustrating a portion of the switch fabric system of Figure 5A, according to one embodiment.

Figure 7 is a schematic diagram illustrating a portion of the switch fabric system of Figure 5A, according to one embodiment.

8 and 9 show front and rear views, respectively, of a housing for covering a switch fabric, according to one embodiment.

Figure 10 shows a portion of the housing of Figure 8, according to one embodiment.

11 and 12 are schematic diagrams showing switch fabrics in a first configuration and a second configuration, respectively, according to another embodiment.

Figure 13 is a schematic diagram illustrating data flow associated with a switch fabric, according to one embodiment.

FIG. 14 is a schematic diagram illustrating flow control in the switch fabric shown in FIG. 13 according to one embodiment.

Figure 15 is a schematic diagram illustrating a buffer module according to one embodiment.

Figure 16A is a schematic block diagram of an ingress scheduling module and an egress scheduling module configured to coordinate cell group transmission via a switch fabric of a switching core, according to one embodiment.

Figure 16B is a signaling flow diagram illustrating signaling involved in cell group transmission, according to one embodiment.

Figure 17 is a schematic block diagram illustrating two groups of cells queued at an ingress queue arranged on the ingress side of a switch fabric, according to one embodiment.

Figure 18 is a schematic block diagram illustrating two groups of cells queued at an ingress queue arranged at the ingress side of a switch fabric according to another embodiment.

Figure 19 is a flowchart illustrating a method of scheduling cell group transmission via a switch fabric, according to one embodiment.

Figure 20 is a signaling flow diagram illustrating the processing of request sequence values associated with transmission requests, according to one embodiment.

Figure 21 is a signaling flow diagram illustrating response sequence values associated with transmission responses, according to one embodiment.

Figure 22 is a schematic block diagram illustrating a multi-level flow controllable queue according to one embodiment.

Figure 23 is a schematic block diagram illustrating a multi-level flow controllable queue according to one embodiment.

Figure 24 is a schematic block diagram illustrating a destination control module configured to define flow control signals associated with a plurality of receive queues, according to one embodiment.

Figure 25 is a schematic diagram illustrating flow control packets, according to one embodiment.

detailed description

FIG. 1 is a schematic diagram illustrating a data center (DC) 100 (eg, super data center, idealized data center) according to one embodiment. Data center 100 includes a switch core (SC) 180 operatively connected to four types of peripheral processing devices 170 : compute nodes 110 , service nodes 120 , routers 130 , and storage nodes 140 . In this embodiment, a data center management (DCM) module 190 is configured to control (eg, manage) the operation of the data center 100 . In some embodiments, data center 100 can be referred to as a data center. In some embodiments, the peripheral processing device may include one or more virtual resources such as virtual machines.

Each peripheral processing device 170 is configured to communicate via the switch core 180 of the data center 100 . In particular, switch core 180 of data center 100 is configured to provide arbitrary connectivity between peripheral processing devices 170 with relatively low latency. For example, switch core 180 can be configured to send (eg, transfer) data between one or more compute nodes 110 and one or more storage nodes 140 . In some embodiments, switch core 180 can have at least hundreds or thousands of ports (eg, egress ports and/or ingress ports) through which peripheral processing device 170 can send and/or receive data. Peripheral processing device 170 includes one or more network interface devices (e.g., a network interface card (NIC), 10 Gigabit (Gb) Ethernet centralized network adapter (CNA) device) through which peripheral processing device 170 is capable of sending signals Signals are received to and/or from switch core 180 . Signals can be sent to and/or received from switch core 180 via physical and/or wireless links operatively coupled to peripheral processing device 170 . In some embodiments, peripheral processing device 170 can be configured to be based on one or more protocols (e.g., Ethernet protocol, Multi-Protocol Label Switching (MPLS) protocol, Fiber Channel protocol, Fiber-channel over Ethernet protocol (fibre-channel protocol) -over Ethernet protocol), Infiniband-related protocol (Infiniband-related protocol)) to send data to the switching core 180 and/or receive data from the switching core 180.

In some embodiments, switch core 180 may be (eg, capable of) a single consolidated switch (eg, a single large-scale consolidated L2/L3 switch). In other words, switch core 180 can be configured to operate as a single logical entity (eg, individual logical network elements) as opposed to, eg, a collection of different network elements configured to communicate with each other via Ethernet connections. Switch core 180 can be configured to connect (eg, facilitate communication between) compute nodes 110 , storage nodes 140 , service nodes 120 , and/or routers 130 in data center 100 . In some embodiments, switch core 180 can be configured to communicate via an interface device configured to transmit data at a rate of at least 10 Gb/s. In some embodiments, switch core 180 can be configured to communicate via an interface device (eg, a Fiber Channel interface device) configured to communicate at, for example, 2Gb/s, 4Gb/s, 8Gb/s, 10Gb/s, 40Gb/s, 100Gb/s and/or faster link rates to send data.

While switch core 180 may be logically centralized, implementation of switch core 180 may be highly distributed, eg, for reliability. For example, portions of switch core 180 may be physically distributed across, eg, many racks. In some embodiments, for example, a processing stage of switch core 180 can be included in a first rack and another processing stage of switch core 180 can be included in a second rack. Both processing stages can logically be combined and exchanged as separate parts. More details about the architecture of switch core 180 will be described in conjunction with FIGS. 4 to 13 .

As shown in FIG. 1 , switch core 180 includes edge portion 185 and switch fabric 187 . Edge portion 185 may include an edge device (not shown) capable of functioning as a gateway device between switch fabric 187 and peripheral processing device 170 . In some embodiments, edge devices in edge portion 185 can collectively have thousands of ports (e.g., 100,000 ports, 500,000 ports) through which data from peripheral processing device 170 can be sent in (e.g., Routing) to and/or from one or more portions of the switch core 180. In some embodiments, edge devices can be referred to as access switches, network appliances, and/or input/output modules (eg, as shown in FIGS. 5A and 5B ). In some embodiments, an edge device can be included in, for example, a top-of-rack (TOR) of a rack.

Data can be transferred based on different platforms at the peripheral processing device 170, the switching core 180, the switching fabric 187 of the switching core 180, and/or the edge portion 185 of the switching core 180 (for example, at an edge device included in the edge portion 185) deal with. For example, communications between the one or more peripheral processing devices 170 and the edge devices at the edge portion 185 may be based on data packet streams defined by an Ethernet protocol or a non-Ethernet protocol. In some embodiments, various data processing can be performed on edge devices within edge portion 185 rather than within switch fabric 187 of switch core 180 . For example, data packets can be parsed into cells at an edge device of edge portion 185 and the cells sent from the edge device to switch fabric 187 . Cells can be parsed into segments and sent within switch fabric 187 as fragments (which can also be referred to as flits in some embodiments). In some embodiments, data packets can be parsed into cells at a portion of switch fabric 187 . In some embodiments, congestion resolution and/or scheduling of data (e.g., cell) transfers via switch fabric 187 can be implemented at edge devices (e.g., access switches) within edge portion 185 of switching center 180 or execute. However, congestion resolution and/or data transmission scheduling may not be performed in the modules defining the switch fabric 187 . Further details relating to data packet, cell and/or fragment processing within components of the data center are described below. For example, more details related to cell processing will be described in connection with at least FIG. 16A through FIG. 21 .

In some embodiments, edge devices within edge portion 185 can be configured to classify, for example, data packets received at switching core 180 from peripheral processing device 170 . In particular, edge devices within edge portion 185 of switch core 180 can be configured to perform Ethertype classification, which may include classification based on, for example, Layer 2 Ethernet addresses (e.g., Media Access Control (MAC) addresses) and/or Classification of Layer 4 Ethernet addresses such as Universal Datagram Protocol (UDP) addresses. In some embodiments, the destination may be determined based on, for example, the classification of the packet at the edge portion 185 of the switch core 180 . For example, a first edge device can identify a second edge device as the packet's destination based on the packet's classification. Packets can be parsed into cells and sent from the first edge device to the switch fabric 187 . Cells can be switched through the switch fabric 187 so that they can be sent to the second edge device. In some embodiments, cells can be switched through switch fabric 187 based on information pertaining to the destination and associated with the cells.

Security policies on the switch core 180 can be applied more efficiently because classification is performed at a separate logical layer of the switch core 180 , at the edge portion 185 of the switch core 180 . In particular, many security policies can be applied at the edge portion 185 of the switch core 180 during classification in a relatively uniform and seamless manner.

Further details related to packet classification within a data center are described in connection with, for example, FIGS. 5A , 5B and 19 . Additional details relating to Packet Classification Associated within a Data Center are at "Methods and Apparatus Related to Packet Classification Associated with a Multi-Stage Switch" and published September 2008 U.S. Patent Application Serial No. 12/242168 filed on September 30 and entitled "Methods and Apparatus for Packet Classification Based on Policy Vectors (methods and devices for packet classification based on policy vectors)" and filed on September 30, 2008 described in Patent Application Serial No. 12/242,172, both of which are fully incorporated herein by reference.

Switch core 180 can be defined such that classification of data (eg, data packets) is not performed in switch fabric 187 . Therefore, although the switch fabric 187 may have multiple levels, the multiple levels do not require a topological hop in which data classification is performed, and the switch fabric 187 can define individual topological hops. Alternatively, destination information determined based on classification at an edge device (eg, an edge device within edge portion 185 of switch core 180 ) can be used for switching (eg, switching of cells) within switch fabric 187 . Further details related to switching within switch fabric 187 are described in connection with, for example, FIGS. 4A and 4B.

In some embodiments, processing related to classification may be performed at a classification module (not shown) included in an edge device (eg, an input/output module). Parsing of packets into cells, scheduling of cell transmission via switch fabric 187, reassembly of packets and/or cells, and/or etc. implement. In some embodiments, the classification module may be referred to as a packet classification module, and/or the processing module may be referred to as a packet processing module. More details related to the edge device including the classification module and the processing module will be described in connection with Fig. 5B.

In some embodiments, one or more portions of data center 100 may be (or may include) hardware-based modules (e.g., application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays ( FPGA)) and/or software-based modules (eg, computer code modules, processor-readable instruction sets executable on a processor). In some embodiments, one or more functions related to data center 100 may be included in different modules and/or combined into one or more modules. For example, data center management module 190 may be a combination of hardware modules and software modules configured to manage resources within data center 100 (eg, resources of switch core 180)

One or more compute nodes 110 may be general purpose compute engines that can include, for example, processors, memory, and/or one or more network interface devices (eg, network interface cards (NICs)). In some embodiments, processors in compute node 110 may be part of one or more cache coherent domains.

In some embodiments, compute nodes 110 may be host devices, servers, and/or the like. In some embodiments, one or more compute nodes 110 can have virtualized resources such that any compute node 110 (or portion thereof) can be used in place of any other compute node 110 (or portion thereof) within the data center 100 .

One or more storage nodes 140 may be devices including, for example, processors, memory, locally attached disk storage, and/or one or more network interface devices. In some embodiments, storage nodes 140 can have dedicated modules (eg, hardware modules and/or software modules) configured to enable, for example, one or more computing nodes 110 to read data from one or more Store the data of the node 140 or write the data to one or more storage nodes 140 . In some embodiments, one or more storage nodes 140 may have virtualized resources such that any storage node 140 (or portion thereof) may be used in place of any other storage node 140 (or portion thereof) within data center 100 .

One or more service nodes 120 may be Open Systems Interconnection (OSI) Layer 4 through Layer 7 devices, which may include, for example, a processor (e.g., a network processor), memory, and/or one or more network interfaces devices (eg, 10Gb Ethernet devices). In some embodiments, service nodes 120 may include hardware and/or software configured to perform computations on relatively heavy network workloads. In some embodiments, service node 120 may be configured to perform computations on a per-packet basis in a relatively efficient (eg, more efficient than performing on, eg, compute node 110 ) manner. Computations may include, for example, stateful firewall computations, intrusion detection and prevention (IDP) computations, Extensible Markup Language (XML) accelerated computations, Transmission Control Protocol (TCP) terminal computations, and/or application level load balancing computations. In some embodiments, one or more service nodes 120 may have virtualized resources such that any service node 120 (or portion thereof) can be used in place of any other service node 120 (or portion thereof) within the data center 100 .

One or more routers 130 can be network devices configured to connect at least a portion of data center 100 to another network (eg, the global Internet). For example, as shown in FIG. 1 , switch core 180 can be configured to communicate with network 135 and network 137 through router 130 . Although not shown, in some embodiments, one or more routers 130 are capable of enabling communication between components within data center 100 (eg, peripheral processing devices 170 , portions of switch core 180 ). Communication can be defined based on, for example, layer 3 routing protocols. In some embodiments, one or more routers 130 can have one or more network interface devices (e.g., 10Gb Ethernet devices) through which routers 130 can communicate to and/or from, for example, switch core 180 and/or Other peripheral processing devices 170 send and/or receive signals.

More details concerning virtualized resources within a data center are presented in "Methods and Apparatus for Determining a Network Topology During Network Provisioning" and filed December 30, 2008 Co-pending U.S. Patent Application No. 12/346623, entitled "Methods and Apparatus for Distributed Dynamic Network Provisioning (method and apparatus for dynamic network provisioning distribution)" and filed on December 30, 2008. Patent Application No. 12/346632, and co-pending U.S. Patent Application No. .12/346630, all of which are incorporated herein by reference.

As described above, switch core 180 can be configured to function as a single general purpose switch capable of connecting any peripheral processing device 170 within data center 100 to any other peripheral processing device 170 . In particular, switch core 180 can be configured to provide arbitrary connectivity between peripheral processing devices 170 (e.g., a relatively large number of peripheral processing devices 170) and switch core 180, except those delayed by the bandwidth of the network interface device and by speed-of-light signaling. (also known as speed-of-light latency) imposes substantially no visible constraints, the network interface device connects the peripheral processing device 170 to the switch core 180 . In other words, switch core 180 can be configured such that each peripheral processing device 170 appears to be directly interconnected to all other peripheral processing devices within data center 100 . In some embodiments, switch core 180 can be configured to enable peripheral processing device 170 to communicate at a line rate (or substantially at line rate) via switch core 180 . A schematic representation of any connectivity is shown in FIG. 2 .

Furthermore, the switch core 180 can handle the migration of virtual resources, eg, between any peripheral processing devices 170 in communication with the switch core 180 in a desired manner because the switch core 180 functions as a single logical entity. Accordingly, virtual resource migration scope within peripheral processing device 170 may span substantially all ports coupled to switch core 180 (eg, all ports of edge devices 185 of switch core 180 ).

In some embodiments, provisioning associated with virtual resource migration may be handled in part by a network management module. A centralized network management entity or network management module can cooperate with network devices (eg, portions of switching core 180) to collect and manage network topology information. For example, the network device can push information about resources (virtual and physical) currently operationally coupled to the network device to the network management module, as the resources are attached or independent of the network device. An external management entity such as a peripheral processing device management tool (e.g., a server management tool) and/or a network management tool can communicate with the network management module to send network provisioning instructions to network devices and other resources in the network without requiring static configuration of the network. describe. Such a system avoids the difficulties of static network description and network performance degradation caused by other types of peripheral processing devices 170 and network management systems.

In one embodiment, a server management tool or external management entity communicates with the network management module to provide network devices with virtual resources associated with peripheral processing device 170 and to determine operational status or conditions (e.g., running, suspended, or migrated) and virtual resources position in the network. The virtual resource may be a virtual machine executing on a peripheral processing device 170 (eg, a server) coupled to the switch fabric via an access switch in a data center (eg, an access switch included in edge portion 187). Many types of peripheral processing devices 170 can be coupled to the switch fabric via access switches.

Rather than relying on a static network description for network topology information discovery and/or management (including binding of virtual resources onto network devices), the network management module communicates and cooperates with access switches and external management entities to discover or determine network topology information. After initializing (and/or starting) the virtual machine on the host (and/or other type of peripheral processing device 170 ), the external management entity can provide the network management module with the virtual machine's device identifier. The device identifier can be, for example, the Media Access Protocol (“MAC”) address of the network interface of the virtual machine or peripheral processing device 170, the name of the virtual machine or peripheral processing device 170, a globally unique identifier (“GUID”), and and/or a universally unique identifier (“UUID”) of a virtual resource or peripheral processing device 170 . A GUID need not be globally unique across all networks, virtual resources, peripheral processing devices 170, and/or network devices, but it is unique within a network or network segment managed by a network management module. Furthermore, the external management entity can provide instructions for providing instructions to ports of the access switch to which the peripheral processing device 170 managing the virtual machines is connected. The access switch can detect that a virtual machine has been initialized, started, and/or moved to the peripheral processing device 170 . After detecting the virtual machine, the access switch can query the peripheral processing device 170 for information about the peripheral processing device 170 and/or the virtual machine, including, for example, a device identifier of the peripheral processing device 170 or the virtual machine.

The access switch can interrogate or request information such as the device identifier of a virtual machine that is configured via The above protocol communication. Alternatively, the virtual machine may broadcast information about itself (including the virtual machine's device identifier) using, for example, Ethernet or IP broadcast packets after detecting that it has been connected to the access switch.

The access switch then pushes the virtual device's device identifier (sometimes called a virtual device identifier) and, in some embodiments, other information received from the virtual machine to the network management module. In addition, the access switch can push the device identifier of the access switch and the port identifier of the port of the access switch to the network management module, and the peripheral processing device 170 controlling the virtual machine is connected to the access switch. This information function serves as a description of the location of the virtual machine in the network and defines the binding of the virtual machine to the peripheral processing means 170 for the network management module and external management entities. In other words, upon receiving this information, the network management module can be able to associate the device identifier of the virtual machine (and/or the peripheral processing device 170 operating the virtual machine) with a specific port on a specific access switch ) is connected to that particular access switch.

The device identifier of the virtual machine, the device identifier of the access switch, the port identifier and the provisioning instructions provided by the external management entity can be stored in a memory accessible to the network management module. For example, the device identifier of the virtual machine, the device identifier of the access switch, and the port identifier can be stored in a memory configured as a database so that a query based on the database of device identifiers of the virtual machine returns the device of the access switch identifiers, port identifiers, and provisioning instructions.

Because the network management module can associate a virtual machine's location in the network based on the device identifier of the virtual machine, an external management entity need not pay attention to the topology of the network or tie the virtual machine to the peripheral processing device 170 to provide network resources (e.g., network appliances, virtual machines, virtual switches, or physical servers). In other words, the external management entity is as agnostic as the interconnection in the network and the location of the virtual machine in the network (e.g., at which port of which access switch, on which peripheral processing device 170 in the network), and can be based on The device identifiers of the virtual machines controlled by the peripheral processing device 170 in the network provide an access switch in the network. In some embodiments, the external management entity can also provide the physical peripheral processing device 170 . Furthermore, because the network management module dynamically determines and manages network topology information, external management entities do not rely on a static description of the network for the provisioning network.

As used in this specification, provisioning may include various types or forms of device and/or software module settings, configurations and/or adjustments. For example, provisioning may include configuring a network device, such as a network switch, based on a network policy. More particularly, for example, network provisioning may include: configuring the network device to operate as a layer 2 or layer 3 network switch; changing a routing table of the network device; updating security policies and/or device address or device identifier; select which network protocol implementation the network device uses; set network segment identifiers such as virtual local area network ("VLAN") tagging for network device ports; and/or apply access control lists (" ACL") to the network device. The network switch can be provided or configured such that rules and/or access restrictions defined by network policies are applied to data packets passing through the network switch. In some embodiments, virtual appliances are provided. A virtual device may be, for example, a software module implementing a virtual switch, virtual router, or virtual gateway configured to operate as an intermediary between physical networks and which is controlled by a host device, such as peripheral processing device 170 . In some embodiments, provisioning can include establishing a virtual port or connection between a virtual resource and a virtual device.

Figure 2 is a schematic diagram illustrating an example of a portion of a data center with arbitrary connectivity, according to one embodiment. As shown in FIG. 2 , a peripheral processing device PD (from the group of peripheral processing devices 210 ) is connected to each peripheral processing device 210 via a switch core 280 . In some embodiments, only the connections from the peripheral processing device PD to other peripheral processing devices 210 (other than the peripheral processing device PD) are shown for clarity.

In some embodiments, the switch core 280 is defined such that the switch core 280 is fair in the sense that the bandwidth of the destination link between the peripheral processing device PD and other peripheral processing devices 210 is substantially reasonably Shared among competing peripheral processing devices 210 . For example, when some (or all) of the peripheral processing devices 210 shown in FIG. 2 attempt to access the peripheral processing device PD at a given time, the bandwidth (e.g., instant bandwidth) will be substantially equal. In some embodiments, switch core 280 can be configured such that some (or all) peripheral processing devices 210 can communicate with peripheral processing device PD at full bandwidth (e.g., the full bandwidth of peripheral processing device PD) and/or in a non-blocking manner. way of communication. Furthermore, switch core 280 can be configured such that access to peripheral processing device PD by a peripheral processing device (from peripheral processing device 210) may not be blocked by other links between other peripheral processing devices and peripheral processing device PD (e.g., exists or attempts to) limit.

In some embodiments, the properties of switch core 280, arbitrary connectivity, low latency, fairness, and/or the like, enable connection to (e.g., communication with) switch core 280 of a given type (e.g., storage node type, Compute node type) peripheral processing devices 210 can be treated interchangeably (eg, independent of location relative to other processing devices 210 and switch core 280 ). This can be referred to as interchangeability, and can contribute to the efficiency and simplicity of data centers that include switching cores 280 . Even though switch core 280 may have a large number of ports (e.g., over 1000 ports), switch core 280 can still have arbitrary connectivity and/or fairness properties so that each port can operate at a relatively high speed (e.g., at operation at speeds exceeding 10Gb/s). This can be achieved without the need for specialized interconnects, such as those involved in supercomputers, and/or without complete knowledge of all communication modes. Further details relating to switching core architecture with arbitrary connectivity and/or fairness will be described at least in part in connection with FIGS. 4 to 13 .

Referring back to FIG. 1 , in some embodiments, data center 100 is configured to allow flexible oversubscription. In some embodiments, through flexible oversubscription, the relative cost of network infrastructure (eg, involving switching core 180 ) can be reduced relative to costs such as computation and storage. For example, the resources (e.g., all resources) within the switching core 180 of the data center 100 can operate as flexibly pooled resources such that underutilized resources associated with a first application (or set of applications) are at the peak of a second application, for example The processing period can be dynamically provisioned by the second application (or set of applications). Accordingly, the resources (or subsets of resources) of data center 100 can be configured to handle oversubscription more efficiently than if resources were allocated strictly as storage resources allocated to specific applications (or sets of applications). If managed as a storage resource, oversubscription can only be implemented within the storage resource, rather than across the entire data center 107, for example. In some embodiments, one or more protocols and/or components in data center 100 can be based on open standards (e.g., Institute of Electrical and Electronics Engineers (IEEE) standards, Internet Engineering Task Force (IETF) standards, International Information Technology Standards Committee (INCITS) standard).

In some embodiments, data center 100 can support a security model that allows a wide range of policies to be enforced. For example, data center 100 may support a no-communication policy, where applications reside in separate virtual data centers in data center 100, but can share the same physical peripheral processing devices (e.g., compute nodes 100, storage nodes 140) and network infrastructure (e.g., switch core 180). In some configurations, data center 100 can support multiprocessing of the same application portion and require almost unlimited communication. In some configurations, data center 100 can support policies requiring, for example, deep packet inspection, stateful firewalls, and/or stateless filters.

The data center 100 can have end-to-end application to application latencies (also referred to as end-to-end latencies) defined based on source latencies, zero load latencies, congestion latencies, and destination latencies. In some embodiments, source latency may be, for example, time spent during source peripheral processing (eg, time spent by software and/or a NIC). Similarly, destination latency may be, for example, time spent during processing by the destination peripheral processing device (eg, time spent by software and/or a NIC). In some embodiments, zero load latency may be the speed of light latency plus processing and store-and-forward latency within, for example, data center 180 . In some embodiments, the congestion latency may be, for example, a queuing delay caused by congestion in the network. Data center 100 can have a desired application performance with low end-to-end latency enabling applications that are latency sensitive to, for example, applications with real-time constraints and/or with advanced internal processing communication requirements.

The zero-load latency of the switch core 180 can be significantly reduced compared to the core portion of the data center with Ethernet hop-based interconnects. In some embodiments, for example, switch core 180 can have a zero-load latency of less than 6 microseconds from a switch core 180 input port to a switch core 180 output port (except for the speed of light latency). In some embodiments, for example, switch core 180 can have zero load latencies (excluding congestion latencies and lightspeed latencies) of less than 12 microseconds. Ethernet-based data center cores have significantly high latencies due to, for example, undesired levels of congestion (eg, between links). Congestion within portions of the Ethernet-based data center core may be exacerbated by the inability of the Ethernet-based data center core (or a management device associated with the Ethernet-based data center core) to handle the congestion in an undesired manner. Furthermore, latency within an Ethernet-based data center core can be non-uniform because the core can have a different number of hops between different source-destination pairs and/or between many store-and-forward switching nodes , performing classification of data packets in the store-and-forward switching node. In contrast, the classification of the switch core 180 is performed at the edge portion 185 , not at the switch fabric 187 , and the switch core 180 has a deterministic cell-based switch fabric 187 . For example, cell processing latencies through switch fabric 187 (rather than cell paths through switch fabric 187) may be predictable.

Switch core 180 of data center 100 is capable of providing lossless end-to-end packet transfer based at least in part on flow control mechanisms implemented within data center 100 . For example, data (eg, data pertaining to data packets) transmission scheduling via the switch fabric 187 is performed on a cell basis using a request authorization mechanism (also referred to as a request authentication mechanism). In particular, a cell is sent to switch fabric 187 (eg, from edge portion 185 to switch fabric 187) after a request to send the cell has been authorized on a substantially authorized transfer (lossless) basis. Once admitted to switch fabric 187, cells are processed in switch fabric 187 as fragments. The flow of fragments within switch fabric 187 can further be controlled such that fragments are not lost when congestion within switch fabric 187 is detected, for example. More details concerning cell and fragment processing within switch core 180 are described below.

Additionally, the data flow from each peripheral processing device 170 through switch fabric 187 can be isolated from the data flow through switch fabric 187 from the remaining peripheral processing devices 170 . In particular, data congestion at one or more peripheral processing devices 170 does not affect in an undesired manner the flow of data through the switch fabric 187 of the switch core 180, because at the edge portion 185 of the switch core 180, send requests have been authorized to work , the cell is only sent to the switch fabric 187 of the switch core 180. For example, a high level of data traffic at the first peripheral processing device 170 may be handled based on a request authorization congestion resolution mechanism so that high levels of data traffic at the first peripheral processing device 170 will not adversely affect the second peripheral processing device 170 is connected to a separate logical entity of switching core 180. In other words, when admitted to the switch fabric 187 of the switch core 180, traffic associated with the first peripheral processing device 170 will be isolated (eg, from a congestion perspective) from traffic associated with the second peripheral processing device 170. traffic.

Likewise, the flow of data packets in the switching core 180 that can be parsed into cells and fragments can be controlled at the peripheral processing device 170 based on a fine grain flow control mechanism. In some embodiments, fine-grained flow control is performed based on the stages of the queue. A fine-grained type of flow control can prevent (or substantially prevent) head-of-line blocking that leads to poor network utilization. Fine-grained flow control can also be used to lower (or reduce) latency within switch core 180 . In some embodiments, fine-grained flow control enables high-performance blocks to send and receive disk traffic to and from peripheral processing devices 170 that cannot use Ethernet and Internet (IP ) network is implemented in the desired manner. Further details concerning fine-grained flow control are described in connection with FIGS. 22 to 25 .

In some embodiments, data center 100, and in particular switch core 180, may have a modular architecture. In particular, the switching core 180 of the data center 100 can be implemented starting at a small scale and can be expanded (eg, added expansion) as needed. The switch core 180 can be expanded without substantially interrupting the continuous operation of the existing network and/or can be expanded without constraints on the physical placement of new devices on the switch core 180 .

In some embodiments, one or more portions of switch core 180 can be configured to operate based on a virtual private network ("VPN"). In particular, switch core 180 can be partitioned such that one or more peripheral processing devices 170 can be configured to partition communication via virtualized switching core 180 overlapping or non-overlapping. Switch core 180 can also be decomposed into virtualized resources with separate or overlapping subsets. In other words, the switch core 180 may be an individual switch that can be divided in a flexible manner. In some embodiments, the method enables extended networking at once within the consolidated switch core 180 of the data center 100 . This is in contrast to a data center, which may be a collection of independently scalable networks, each with customized and/or specific resources. In some embodiments, the network resources defining switch core 180 can be consolidated so that they can be used efficiently.

In some embodiments, data center management module 190 can be configured to define virtual multiple levels of physical (and/or virtual) resources that define data center 100 . For example, data center management module 190 is configured to define virtual multi-levels that can represent the application breadth of data center 100 . In some embodiments, the lower level (of the two levels) may include a Virtual Application Cluster (VAC), which may be assigned to (eg, controlled by) one or more entities (eg, management entities, A collection of physical (or virtual) resources for a single application of a financial system). The higher level (of the two levels) may include a virtual data center (VDC), which may include a set of VACs belonging to (eg, controlled by) one or more entities. In some embodiments, data center 100 includes multiple VACs, each of which may belong to a different management entity.

FIG. 3 is a schematic diagram illustrating a logical grouping 300 of resources associated with a data center, according to one embodiment. As shown in FIG. 3, logical group 300 includes virtual data center VDC1, virtual data center VDC2, and virtual data center VDC3 (collectively referred to as VDC). Likewise, as shown in FIG. 3 , each VDC includes a virtual application cluster VAC (for example, VAC32 in VDC3). Each VDC embodies a logical grouping of physical or virtual portions of a data center such as data center 100 shown in FIG. 1 (eg, portions of switching cores, portions of peripheral processing devices, and/or virtual machines within peripheral processing devices). For example, each VAC within a VDC embodies a logical grouping of peripheral processing devices such as compute nodes. For example, VDC1 may embody a logical grouping of physical data center portions, while VAC22 embodies a logical grouping of peripheral processing devices 370 within VDC1. As shown in Figure 3, each VDC can be managed based on a set of policies PY (also referred to as business rules) that can be configured, for example, to define the allowable range of operating parameters on applications running within the VDC. In some embodiments, VDCs can be referred to as the first tier of logical resources, while VACs are referred to as the second tier of logical resources.

In some embodiments, a VDC (and VAC) may be established so that resources associated with a data center are managed in a desired manner, for example, by an entity that uses (e.g., leases, owns, communicates through) the data center's A manager of resources and/or data center resources. For example, VDC1 may be a virtual data center associated with a financial institution, while VDC2 may be a virtual data center associated with a telecommunications service provider. Thus, policy PY1 can be defined through financial regulations so that VDC1 (and the physical and/or virtual data center resources associated with VDC1) can manage VDC2 (and the physical and/or virtual data center resources associated with VDC2) differently than policy PY2 Resources) are managed in a manner that the PY2 policy is defined by the telecommunications service provider. In some embodiments, one or more policies (e.g., part of policy PY1) are established by the network administrator to, when implemented, provide Information Security and/or Firewalls.

In some embodiments, policies can be associated with (or integrated in) data center management (not shown). For example, VDC2 can be managed based on policy PY2 (or a subset of policy PY2). In some embodiments, data center management can be configured, for example, to monitor real-time performance of applications within a VDC and/or can be configured to automatically allocate or de-allocate resources to meet corresponding policies for applications within a VDC. In some embodiments, policies can be configured to operate based on time thresholds. For example, one or more policies can be configured to operate based on periodic events (eg, predictable periodic events) in which parameter values (eg, traffic levels) change, eg, during particular times of day or days of the week.

In some embodiments, policies can be defined based on a high-level language. Thus, policies can be specified in a relatively accessible manner. Examples of policies include information security policies, fault isolation policies, firewall policies, performance assurance policies (eg, policies related to service levels enforced by applications), and/or other management policies related to information protection or access (eg, management isolation policies).

In some embodiments, policies can be enforced at a packet classification module that can be configured, for example, to classify data packets (e.g., IP packets, session control protocol packets, media packets, data packets defined at a peripheral processing device) ). For example, policies can be enforced within the packet classification module of the access switch within the edge portion of the switch core. Classification can include any processing performed so that data packets can be processed within a data center (eg, a switching core of a data center) based on policy. In some embodiments, a policy includes one or more policy conditions associated with instructions that can be executed. A policy may be, for example, a policy to route a data packet to a specific destination (instruction) if the data packet has a specific type of network address (policy condition). Packet classification can include determining whether policy conditions have been met so that the instruction can be executed. For example, one or more portions of a data packet (eg, realm, payload, address portion, port portion) can be analyzed by the packet classification module based on policy conditions defined within the policy. When a policy condition is satisfied, the data packet can be executed based on the instructions associated with the policy condition.

In some embodiments, one or more portions of logical grouping 300 can be configured to operate in a "lights out" mode from multiple remote locations—such as a separate location for each VDC and one or Two master locations to control the logical group 300. In some embodiments, a data center having logical groups such as shown in FIG. 3 can be configured to operate without requiring personnel to be physically present on the data center side. In some embodiments, the data center has sufficient redundant resources to accommodate failures, such as failure of one or more peripheral processing devices (e.g., within a VAC), failure of a data center management module, and/or or failure of the exchange core component. When monitoring software within the data center (eg, within the data center's data center management) indicates that the failure has reached a predetermined threshold, personnel can be notified and/or dispatched to replace the failed component.

As shown in Figure 3, VDCs may be logical groups independent of each other. In some embodiments, resources (e.g., virtual resources, physical resources) of a data center (e.g., shown in FIG. 1 ) can be partitioned into different logical groups 300 (e.g., , different layers of the logical group). In some embodiments, two or more VDCs of logical group 300 overlap. For example, the first VDC can share data center resources (eg, physical resources, virtual resources) with the second VDC. In particular, a portion of the switch core of the first VDC can be shared with the second VDC. In some embodiments, for example, resources included in a VAC of a first VDC can be included in a VAC of a second VDC.

In some embodiments, one or more VDCs can be defined manually (eg, manually by a network administrator) and/or automatically (eg, automatically based on policy). In some embodiments, VDC can be configured to change (eg, change dynamically). For example, a VDC (e.g., VDC1) can include a particular set of resources within one time period and can include a different set of resources (e.g., mutually independent resource sets, overlapping resource sets).

In some embodiments, one or more portions of a data center can be provisioned dynamically in response to, prior to, or during changes involving a VDC (eg, migration of portions of a VDC like its virtual machines). For example, a switching core of a data center can include a plurality of network devices, such as network switches, each storing a configuration template database including service instructions provided and/or requested by virtual machines. When a virtual machine migrates to and/or on a server connected to a network switch port of the switching core and/or initializes or starts, the server can send to the network switch an identifier related to the service provided by the virtual machine. The network device can select a configuration template from the configuration template database based on the identifier and provision ports and/or servers based on the configuration template. In this way, the task of provisioning network ports and/or devices can be distributed among the network switches in the switching core (e.g., in an automatic manner, without redefining the template distribution), and can be distributed as virtual machines dynamically or resources are distributed among peripheral processing devices Migrating between.

In some embodiments, provisioning can include multiple types or forms of device and/or software module settings, configurations, and/or adjustments. For example, provisioning can include configuring network devices, such as network switches, within the data center based on a policy such as one of the policies PY shown in FIG. 3 . More particularly, for example, provisioning involving a data center can include one or more of: configuring a network device to operate as a network router or a network switch; changing a routing table of a network device; updating security policies and/or operably coupling Address or identifier of a device connected to a network device; select which network protocol the network device will implement; set a network segment identifier such as a virtual local area network ("VLAN") tag for a network device port; and/or apply access control list ("ACL") to the network device. A portion of a data center can be provisioned or configured such that rules and/or access restrictions defined by policies (eg, PY3) are applied (eg, applied by classification processing) to data packets passing through the portion of the data center.

In some embodiments, virtual resources associated with a data center can be provisioned. A virtual resource may be, for example, a software module implementing a virtual switch, a virtual router, or a virtual gateway configured to operate as an intermediary between a physical network and a virtual resource controlled by a host device such as a server. In some embodiments, virtual resources may be controlled by a host device. In some embodiments, provisioning may include establishing a virtual port or connection between the virtual resource and the virtual appliance.

More details dealing with virtualized resources in the data center are in "Method and Apparatus for Determining a Network Topology During Network Provisioning" and filed on December 30, 2008 Co-pending U.S. Patent Application No. 12/346623, entitled "Methods and Apparatus for Distributed Dynamic Netowrk Provisioning (Methods and Apparatus for Dynamic Network Provisioning Distribution)" and filed on December 30, 2008 U.S. Patent Application No. 12/346632, Co-pending U.S. Patent Application No. 12, entitled "Methods and Apparatus for Distributed Dynamic Network Provisioning" and filed December 30, 2008 /346630, all of which are incorporated herein by reference.

FIG. 4A is a schematic diagram illustrating a switch fabric 400 that may be included in a switch core, according to one embodiment. In some embodiments, switch fabric 400 can be included in a switch core such as switch core 180 shown in FIG. 1 . As shown in FIG. 4A , switch fabric 400 is a three-stage, non-blocking Clos (Clos) network and includes a first stage 440 , a second stage 442 and a third stage 444 . The first stage 440 includes modules 412 (each of which can be referred to as a switching module or a cell switch). Each module 412 of the first stage 440 is an integration of electronic components and circuits. In some embodiments, for example, each module is an application specific integrated circuit (ASIC). In other embodiments, multiple modules are contained on a single ASIC. In some embodiments, each module is an integration of discrete electronic components. In some embodiments, a switch fabric with multiple stages can be referred to as a multi-stage switch fabric.

In some embodiments, each module 412 of the first stage 440 may be a cell switch. Cell switches can be configured to efficiently redirect data (eg, fragments) as it flows through switch fabric 400 . In some embodiments, for example, each module 412 of the first level can be configured to redirect data based on information included in a switching table. In some embodiments, the redirection of data, such as cells within a switch fabric 400 level, can be referred to as switching (eg, data switching) or, if the data is in the form of cells in the switch fabric 400, as a cell switch. In some embodiments, switching within modules of switch fabric 400 may be based on, for example, information associated with data (eg, headers). The switching performed by the modules of switch fabric 400 may differ from the Ethertype classification performed inside edge devices (eg, edge devices within edge portion 185 of switch core 180 shown in FIG. 1 ). In other words, switching within modules of switch fabric 400 may not be based on, for example, Layer 2 Ethernet addresses and/or Layer 4 Ethernet addresses. More details related to exchange table based data exchange will be described in conjunction with FIG. 4B.

In some embodiments, each cell switch also includes a plurality of input ports operatively coupled to write interfaces of a memory buffer (eg, a cut-through buffer). In some embodiments, a memory buffer is included in the buffer module. Similarly, the set of output ports can be operably coupled to the read interface of the memory buffer. In some embodiments, the memory buffer may be using on-chip static random access memory (SRAM) to provide sufficient bandwidth to all input ports for writing one incoming cell (e.g., a portion of a data packet) per time period and to All output ports provide sufficient bandwidth to read the shared memory buffer of one shifted-out cell per time period. Each cell switch operates like a crossbar switch that can be configured after each time period.

In some embodiments, memory buffers (e.g., portions of memory buffers associated with specific ports and/or flows) are of sufficient size (e.g., length) for modules in switch fabric 400 (e.g., module 412) To perform switching (eg, cell switching, data switching) and/or data (eg, cell) synchronization. However, the store buffer may have an insufficient size (and/or too low processing latency) for a module (eg, module 412 ) within switch fabric 400 to implement congestion resolution. Congestion resolution, such as a grant/request mechanism, can be implemented, for example, at an edge device (not shown) associated with a switch core, but not within switch fabric 400 that uses memory buffers for queuing data related to congestion resolution. implemented within the module. In some embodiments, one or more memory buffers within a module (e.g., module 414) have insufficient size (and/or too low processing latency) for, for example, reorganization of data at the module (e.g., cells). More details related to the shared memory buffer will be combined with accompanying drawing 15 and titled "Methods and Apparatus Related to a Shared Memory Buffer for Variable-Sized Cells (relate to the method and apparatus of the shared memory buffer of variable-sized cells)" and Described in co-pending US Patent Application No. 12/415517, filed March 31, 2009, which is incorporated herein by reference in its entirety.

In an alternate embodiment, each module of the first stage may be a crossbar switch having an input port and an output port. Multiple switches within a crossbar switch connect every input bar to every output bar. When a switch within a crossbar switch is in the "on" position, the input is operatively coupled to the output and data can flow. Alternatively, when a switch within a crossbar switch is in the "off" position, inputs are not operably coupled to outputs and data does not flow. In this way, the switches within the crossbar switch control which input rod is operatively coupled to the output rod.

Each module 412 of the first stage 440 includes a set of input ports 460 configured to receive data as it enters the switch fabric 400 . In this embodiment, each module 412 of the first stage 440 includes the same number of input ports 460 .

Similar to first stage 440 , second stage 442 of switch fabric 400 includes modules 414 . The modules 414 of the second stage 442 are structurally similar to the modules 412 of the first stage 440 . Each module 414 of the second stage 442 is operatively coupled to each module of the first stage 440 through a data path 420 . Each data path 420 between each module of the first stage 440 and each module 414 of the second stage 442 is configured to facilitate the transfer of data from the module 412 of the first stage 440 to the module 414 of the second stage 442 .

The data path 420 between the modules 412 of the first stage 440 and the modules 414 of the second stage 442 may be constructed in any manner configured to cause data to pass from the modules 412 of the first stage 440 to the modules 414 of the second stage 442 in a desired manner. way (for example, in an efficient way) to transmit. In some embodiments, for example, the data paths are optical connectors between modules. In other embodiments, the data paths are within the midplane. Such a midplane may be similar to that described here in more detail. Such a midplane can be effectively used to connect every module of the second stage to every module of the first stage. In other embodiments, the modules are contained in a single chip package and the data path is an electronic trace.

In some embodiments, switch fabric 400 is a non-blocking Clos (Clos) network. As such, the number of modules 414 of the second stage 442 of the switch fabric 400 varies based on the number of input ports 460 of each module 412 of the first stage 440 . In a rearrangeable non-blocking Clos (Klos) network (for example, a Benes (Barnes) network), the number of modules 414 of the second stage 442 is greater than or equal to the number of input ports 460 of each module 412 of the first stage 440 number. Thus, if n is the number of input ports 460 of each module 412 of the first stage 440 and m is the number of modules 414 of the second stage 442, m≧n. In some embodiments, for example, each module of the first stage has 5 input ports. Thus, the second level has at least 5 modules. All 5 modules of the first stage are operatively coupled to all 5 modules of the second stage through data paths. In other words, every module on the first stage can send data to any module on the second stage.

Third stage 444 of switch fabric 400 includes module 416 . Module 416 of third stage 444 is structurally similar to module 412 of first stage 440 . The number of modules 416 of the third stage 444 is equal to the number of modules 412 of the first stage 440 . Each module 416 of the third stage 444 includes an output port 462 configured to allow data to be sent out of the switch fabric 400 . Each module 416 of the third stage 444 includes the same number of output ports 462 . Furthermore, the number of output ports 462 per module 416 of the third stage 444 is equal to the number of input ports 460 per module 412 of the first stage 440 .

Each module 416 of the third stage 444 is connected to each module 414 of the second stage 442 by a data path 424 . The data path 424 between the module 414 of the second stage 442 and the module 416 of the third stage 444 is configured to facilitate the transfer of data from the module 414 of the second stage 442 to the module 416 of the third stage 444 .

The data path 424 between the module 414 of the second stage 442 and the module 416 of the third stage 444 can be constructed in any manner to be configured to effectively drive data from the module 414 of the second stage 442 to the module of the third stage 444 416 send. In some embodiments, for example, the data paths are optical connectors between the modules. In other embodiments, the data path is within the midplane. Such a midplane is similar to that described in detail here. Such a midplane can be effectively used to connect every module of the second stage to every module of the third stage. In another embodiment, the modules are contained in a single chip package and the data paths are electronic traces.

Figure 4B is a schematic diagram illustrating a swap table 49 that can be stored in memory 498 of the module shown in Figure 4A, according to one embodiment. A module such as one of the second stage modules 414 shown in FIG. 4A (eg, a switching module) can be configured to perform cell switching based on a switching table, such as switching table 49 shown in FIG. 4B. For example, switching table 49 (or a similarly configured switching table) can be used by (and/or included in) modules in one level of modules, for example, to determine whether a cell can be sent via a module in another level of modules to its destination. In some embodiments, a module through which a cell can be sent to its destination is referred to as a switch destination. In particular, the switching destination may be looked up in the switching table 49 based on destination information including eg cells (which can be determined outside the switching fabric 400).

Interchange table 49 includes binary values (e.g., binary value "1", binary value "0") that indicate whether one or more destinations represented by destination values DT1 through DTk (shown in row 47) can pass through One or more modules (which can be located adjacently) represented by module values SM1 to SMM (shown in column 48) arrive. In particular, the exchange table 49 includes a binary value "1" when a destination in a column comprising a binary value (eg, destination DT1) can be reached via a module (eg, module SM2) in a row intersecting the column . The exchange table 49 includes a binary value "0" when a destination in a column including a binary value cannot be reached via a module in a row intersecting the column. For example, a binary value of "1" in each entry at 46 indicates that if a module (including the switch table 49) sends data to a module represented by module values SM1 to SM3, then the data can eventually be sent to the module represented by destination value DT3 destination. In some embodiments, the modules can be configured to randomly select one of the module groups represented by the module values SM1 to SM3 (which are swap destinations) and can send data to the selected module so that the data can be Send to the destination indicated by the destination value DT3.

In some embodiments, the destination value 47 may be a destination port value associated with an edge device (eg, an access switch), eg, a switching core, a server communicating with the edge device, or the like. In some embodiments, the destination value (which corresponds to the at least one destination value 47 included in the switching table 49) may be based on, for example, the packet classification included in the information element and the information element (e.g., included in the information element header) associated. Thus, the destination value associated with the cell can be used by the module to query the switching destination using the switching table 49 . Packet classification can be performed at the edge devices (eg, access switches) of the switching core.

In some embodiments, memory (and such a switch table 49) can be included in a modular system of one or more modules. In some embodiments, the switch table 49 can be associated with more than one input port and/or more than one output port of the module system (or systems). More details related to the module system will be described in connection with FIG. 7 .

Figure 5A is a schematic diagram illustrating a switch fabric system 500, according to one embodiment. The switch fabric system 500 includes a plurality of input/output modules 502 , a first cable set 540 , a second cable set 542 and a switch fabric 575 . Switch fabric 575 includes a first switch fabric portion 571 disposed within enclosure 570 or a chassis, and a second switch fabric portion 573 disposed within enclosure 572 and the chassis.

The input/output module 502 (which may be, for example, an edge device) is configured to send data and/or receive data to and/or from the first switch fabric part 571 and/or the second switch fabric part 573 . Additionally, each input/output module 502 includes parsing functionality, sorting functionality, forwarding functionality, and/or queuing and scheduling functionality. In this way, packet parsing, packet classification, packet forwarding, and packet queuing and scheduling all occur before data packets enter the first switch fabric portion 571 and/or the second switch fabric portion 573 . Accordingly, these functions need not be performed at every level of the switch fabric 575, and each module of the switch fabric portions 571, 573 (described in further detail herein) need not include the capability to perform these functions. This may reduce cost, power consumption, cooling requirements and/or physical footprint requirements per module of the switch fabric portions 571,573. This also reduces the latency associated with the switch fabric. In some embodiments, for example, the end-to-end latency (i.e., the time required to send data from an I/O module to another I/O module through the switch fabric) can be compared to the end-to-end latency of a switch fabric system using the Ethernet protocol. End-to-end latency is lower. In some embodiments, the throughput of the switch fabric portions 571, 573 is constrained only by the connection density of the switch fabric system 500 and not by power and/or thermal limitations. In some embodiments, I/O module 502 (and/or functionality associated with I/O module 502 ) can be included in an edge device, eg, within an edge portion of a switch core as shown in FIG. 1 . Parsing function, classification function, forwarding function and queuing and dispatching function can be similar to "Methods and Apparatus Related to Packet Classification Associated with a Multi-Stage Switch (involving the method and equipment of grouping classification related to multi-stage switching)" and in U.S. Patent Application Serial No. 12/242168 filed September 30, 2008 and entitled "Methods and Apparatus for Packet Classification Based on Policy Vectors" and filed September 30, 2008 implements the functions disclosed in US Patent Application Serial No. 12/242,172, both of which are fully incorporated herein by reference.

Each input/output module 502 is configured to connect a first end of a first cable set 540 cable to a first end of a second cable set 542 cable. Each cable 540 is deployed between an input/output module 502 and a first switch fabric portion 571 . Similarly, each cable 542 is deployed between an input/output module 502 and a second switch fabric portion 573 . Using the first set of cables 540 and the second set of cables 542, each input/output module 502 is capable of sending and/or receiving data to and/or from the first switch fabric portion 571 and/or the second switch fabric portion 573, respectively.

The first set of cables 540 and the second set of cables 542 can be composed of any material suitable for transferring data between the input/output module 502 and the switch fabric portions 571,573. In some embodiments, for example, each cable 540, 542 is comprised of multiple optical fibers. In such an embodiment, each cable 540, 542 may have 12 transmit and 12 receive fibers. The 12 transmit fibers of each cable 540, 542 may include 8 fibers for transmitting data, 1 fiber for transmitting control signals, and 3 fibers for extending data capacity and/or for redundancy . Similarly, the 12 receive fibers of each cable 540, 542 may include 8 fibers for transmitting data, 1 fiber for transmitting control signals, and 3 fibers for extending data capacity and/or for redundancy. remaining fiber. In other embodiments, any number of optical fibers may be included in each cable.

The first switch fabric portion 571 and the second switch fabric portion 573 are used together for redundancy and/or greater capacity. In other embodiments, only one switch fabric portion is used. In still other embodiments, more than 2 switch fabric sections are used for increased redundancy and/or greater capacity. For example, 4 switch fabric sections can be operatively coupled to each I/O module by eg 4 cables. The second switch fabric portion 573 is structurally and functionally similar to the first switch fabric 571 . Therefore, only the first switch fabric portion 571 is described in detail here.

Figure 5B is a schematic diagram illustrating an input/output module 502, according to one embodiment. As shown in FIG. 5B , the input/output module 502 includes a classification module 596 , a processing module 597 , and a memory 598 . Classification module 596 can be configured to perform data classification, such as Ethertype classification of packets.

Various types of data processing can be performed at the processing module 597 . For example, data, such as packets, can be parsed at processing module 597 into cells. In some embodiments, congestion resolution can be implemented at processing module 597 and/or data (e.g., cell) transmission scheduling via a switch fabric (e.g., switch fabric 400 shown in FIG. 4A ) can be performed at processing module 597 . The processing module 597 can also be configured to concatenate information (for example, header information, destination information, source information) into, for example, a cell payload, which can be used to pass through a switch fabric (for example, the switch shown in FIG. 4A Structure 400) switches cells (based on the switching table shown in FIG. 4B).

When data processing is performed at classification module 596 and/or processing module 597 , one or more portions of data (eg, packets, cells) can be stored (eg, queued) in memory 598 . For example, data parsed into cells can be queued at memory 598 while processing module 597 performs processing related to congestion resolution. Accordingly, memory 598 may be of sufficient size to implement congestion resolution as described in FIGS. 16A-21 .

FIG. 6 shows a portion of the switch fabric system 500 of FIG. 5A including the first switch fabric portion 571 in more detail. The first switch fabric portion 571 includes an interface card 510 associated with the first and third stages of the first switch fabric portion 571; an interface card 516 associated with the second stage of the first switch fabric portion 571; and Mid-plane 550. In some embodiments the first switch fabric portion 571 includes eight interface cards 510 associated with the first and third stages of the first switch fabric, and eight interface cards 516 associated with the first stage of the first switch fabric. Secondary association. In other embodiments, a different number of interface cards associated with the first and third levels of the first switch fabric and/or a different number of interface cards associated with the second level of the first switch fabric may be used.

As shown in FIG. 6 , each I/O module 502 is operably coupled to the interface card 510 via one cable of the first cable set 540 . In some embodiments, for example, eight interface cards 510 are each operatively coupled to sixteen input/output modules 502, as described in greater detail herein. Thus, the first switch fabric portion 571 can be coupled to 128 input/output modules (16×8=128). Each of the 128 input/output modules 502 is capable of sending and receiving data to and from the first switch fabric portion 571 .

Each interface card 510 is connected to each interface card 516 via a midplane 550 . As such, each interface card 510 is capable of sending and receiving data to and from each interface card 516, as described in greater detail herein. Using midplane 550 to connect interface card 510 to interface card 516 reduces the number of cables used to connect first switch fabric portion 571 stages.

Figure 7 shows the first interface card 510', the midplane 550, and the first interface card 516' in greater detail. Interface card 510' is associated with the first and third stages of the first switch fabric portion 571, and interface card 516' is associated with the second stage of the first switch fabric portion 571. Each interface card 510 is similar in structure and function to the first interface card 510'. Similarly, each interface card 516 is similar in structure and function to the first interface card 516'.

First interface card 510' includes a plurality of cable connector ports 560, a first module system 512, a second module system 514, and a plurality of midplane connector ports 562. For example, Figure 7 shows a first interface card 510' having 16 cable connector ports 560 and 8 midplane connector ports 562. Each cable connector port 560 of the first interface card 510' is configured to receive a second end of a cable from the first cable set 540. Thus, as described above, the 16 cable connector ports 560 on each of the 8 interface cards 510 are used to receive 128 cables (16 x 8 = 128). Although 16 cable connector ports 560 are shown in FIG. 7, in other embodiments, any number of cable connector ports can be used so that each cable of the first set of cables can pass through the first The cable connector ports in the switch fabric are received. For example, if all 16 interface cards are used, each interface card can include 8 cable connector ports.

Each of the first module system 512 and the second module system 514 of the first interface card 510' includes modules of the first stage of the first switch fabric part 571 and modules of the third stage of the first switch fabric part 571. In some embodiments, 8 of the 16 cable connector ports 560 are operatively coupled to the first module system 512 and the remaining 8 cable connector ports of the 16 cable connector ports 560 are operatively The ground is coupled to the second module system 514 . Both the first module system 512 and the second module system 514 are operably coupled to each of the eight midplane connector ports 562 of the interface card 510'.

The first module system 512 and the second module system 514 of the first interface card 510' are ASICs. The first system of modules 512 and the second system of modules 514 are instances of the same ASIC. In this way, manufacturing costs can be reduced since multiple instances of a single ASIC can be produced. In addition, modules of the first stage of the first switch fabric part 571 and modules of the third stage of the first switch fabric are included on each ASIC.

In some embodiments, each of eight midplane connector ports 562 has twice the data capacity of each of sixteen cable connector ports 560 . Thus, each of the 8 midplane connector ports 562 has 16 data send and 16 data receive connections instead of 8 data send and 8 data receive connections. Thus, the bandwidth of 8 midplane connector ports 562 is equal to the bandwidth of 16 cable connector ports 560 . In other embodiments, each midplane connector port has 32 data transmit and 32 data receive connections. In such an embodiment, each cable connector port has 16 data transmit and 16 data receive connections.

The eight midplane connector ports 562 of the first interface card 510' are connected to the midplane 550. The midplane 550 is configured to connect each interface card 510 associated with the first and third stages of the first switch fabric portion 571 to each interface card 516 associated with the second stage of the first switch fabric portion 571 . In this way, the midplane 550 ensures that each midplane connector port 562 of each interface card 510 is connected to a midplane connector port 580 of a different interface card 516 . In other words, no two identical midplane connector ports of interface card 510 are operatively coupled to the same interface card 516 . In this manner, midplane 550 allows each interface card 510 to send and receive data to and from any one of eight interface cards 516 .

Although FIG. 7 shows a schematic diagram of first interface card 510', midplane 550, and first interface card 516', in some embodiments, first interface card 510, midplane 550, and first interface card 516 are physically located Similar to horizontal position interface card 620, midplane 640, and vertical position interface card 630, respectively, as shown in FIGS. 5-7 and described in further detail herein. Thus, the modules associated with the first stage and the modules associated with the third stage (both on interface card 510) are on one side of the midplane, while the modules associated with the second stage (on interface card 516) are on The opposite side of the midplane 550. Such a topology allows each module associated with the first stage to be operatively coupled to each module associated with the second stage, and each module associated with the second stage to be operatively coupled to a module associated with the third stage related to each module.

The first interface card 516' includes a plurality of midplane connector ports 580, a first module system 518, and a second module system 519. A plurality of midplane connector ports 580 are configured to send and receive data to and from any interface card 510 via the midplane 550 . In some embodiments, the first interface card 516' includes eight midplane connector ports 580.

The first module system 518 and the second module system 519 of the first interface card 516' are operably coupled to each midplane connector port 580 of the first interface card 516'. Thus, through the midplane 550, each module system 512, 514 associated with the first and third stages of the first switch fabric portion 571 is operatively coupled to the second stage associated with the first switch fabric portion 571. Each module system 518,519. In other words, each module system 512, 514 associated with the first and third levels of the first switch fabric portion 571 can communicate to and from each module system 518 associated with the second level of the first switch fabric portion 571, 519 send data and receive data and vice versa. In particular, a module associated with a first level within module system 512 or 514 can send data to a module associated with a second level within module system 518 or 519 . Similarly, a module associated with a second level within module system 518 or 519 can send data to a module associated with a third level within module system 512 or 514 . In other embodiments, modules associated with the third stage can send data and/or control signals to modules associated with the second stage, and modules associated with the second stage can send data and/or control signals to modules associated with the first stage. The module sends data and/or control signals.

In an embodiment where each module of the first stage of the first switch fabric portion 571 has 8 inputs (i.e., two modules per interface card 510), the second stage of the first switch fabric portion 571 has at least 8 modules for In the first switch fabric portion 571 to maintain reschedulable non-blocking. Thus, the second stage of the first switch fabric portion 571 has at least 8 modules and is rearrangeable without blocking. In some embodiments, twice as many modules as the second level are used to facilitate scaling of switch fabric system 500 from a 3-level switch fabric to a 5-level switch fabric, as described in further detail herein. In such a five-level switch fabric, the second level supports twice the switching throughput of the second level in the three-level switch fabric of the switch fabric system 500 . For example, in some embodiments, the 16 modules of the second level can be used to facilitate future expansion of the switch fabric system 500 from a three-level switch fabric to a five-level switch fabric.

The first module system 518 and the second module system 519 of the first interface card 516' are ASICs. The first system of modules 518 and the second system of modules 519 are instances of the same ASIC. Furthermore, in some embodiments, the first module system 518 and the second module system 519 associated with the second stage of the first switch fabric portion 571 are also used for the first stage and the third stage of the first switch fabric portion 571 An instance of an ASIC of the first module system 512 and the second module system 514 of the associated first interface card 510'. Thus, since multiple instances of a single ASIC can be used for each modular system of the first switch fabric portion 571, manufacturing overhead can be reduced.

In use, data is transferred from the first input/output module 502 to the second input/output module 502 via the first switch fabric portion 571 . The first input/output module 502 sends data to the first switch fabric part 571 via the cables of the first cable set 540 . Data is sent through the cable connector port 560 of one of the interface cards 510'

A first level module within a modular system 512 or 514 forwards data to the modular system by sending data through one of the connector ports 562 in the midplane of the interface card 510', the midplane 550, and to the interface card 516' Second level modules within 518 or 519. Data enters the interface card 516' through the midplane connector port 580 of the interface card 516'. The data is then sent to a second level module within the module system 518 or 519 .

The second level module determines how the second I/O module 502 connects via the midplane 550 and redirects data back to the interface card 510'. Because each module system 518 or 519 is operably coupled to each module system 512 and 514 on the interface card 510', the second-level modules within the module system 518 or 519 can determine which of the module systems 512 or 514 The tertiary module is operatively coupled to the second input/output module and sends data accordingly.

Data is sent to tertiary modules within the module system 512, 514 on the interface card 510'. The tertiary module then sends data to the second I/O module of the I/O module 502 via the cables of the first cable set 540 through the cable connector port 560 .

In other embodiments, instead of the first-level module sending data to individual second-level modules, the first-level module divides the data into separate parts (e.g., cells) and forwards a portion of the data to each second-level module , the first level modules are operatively coupled to the second level modules (eg, in this embodiment each second level module receives a portion of the data). Each second level module then determines how the second input/output modules are connected and redirects portions of the data back to individual third level modules. The third level module then reconstructs parts of the received data and sends the data to the second input/output module.

8-10 illustrate an enclosure 600 (ie, a rack) for housing a switch fabric (eg, first switch fabric portion 571 as described above) according to one embodiment. The housing 600 includes a casing 610 , a midplane 640 , an interface card 620 in a horizontal position, and an interface card 630 in a vertical position. FIG. 8 shows a front view of housing 610 in which eight horizontal positions of interface cards 620 deployed within housing 610 can be seen. FIG. 9 shows a rear view of the housing 610 in which the eight vertical positions of the interface cards 630 deployed within the housing 610 can be seen.

Each horizontally positioned interface card 620 is operatively coupled to each vertically positioned interface card 630 via a midplane 640 (see FIG. 10 ). The midplane 640 includes a front surface 642, a rear surface 644, and an array of receptacles 650 connecting the front surface 642 and the rear surface 644, as described below. As shown in FIG. 10 , interface card 620 in a horizontal position includes a plurality of midplane connector ports 622 that connect to receptacles on a front surface 642 of midplane 640 . Similarly, interface card 630 in a vertical position includes a plurality of midplane connectors 632 that connect to receptacles on rear surface 644 of midplane 640 . In this manner, the plane defined by each horizontally positioned interface card 620 intersects the plane defined by each vertically positioned interface card 630 .

The receptacles 650 of the midplane 640 operably couple each of the horizontally positioned interface cards 620 to each of the vertically positioned interface cards 630 . Jack 650 facilitates signal transmission between horizontal position interface card 620 and vertical position interface card 630 . In some embodiments, for example, receptacle 650 may be a multi-pin connection configured to receive a multiple pin-connector placed on midplane connector ports 622, 632 of interface cards 620, 630 connector, an empty tube that allows the horizontal position interface card 620 to be directly connected to the vertical position interface card 630, and/or any other device configured to operatively couple the two interface cards. Using such a midplane 640, each horizontal location interface card 620 is operatively coupled to each vertical location interface card 630 without requiring routing connections (eg, electrical traces) on the midplane.

Figure 10 shows a midplane including all 64 jacks 650 in an 8x8 array. In such an embodiment, eight horizontal position interface cards 620 can be operably coupled to eight vertical position interface cards 630 . In other embodiments, any number of jacks can be included in the midplane and/or any number of horizontally positioned interface cards can be coupled to any number of vertically positioned interface cards through the midplane.

If the first switch fabric part 571 is located in the enclosure 600, for example, each interface card 510 associated with the first stage and the third stage of the first switch fabric part 571 may be horizontally positioned and connected to the first stage of the first switch fabric part 571. Each interface card 516 associated with the second level may be a vertical position. In this way, each interface card 510 associated with the first stage and the third stage of the first switch fabric part 571 can be easily connected to each interface card associated with the second stage of the first switch fabric part 571 through the midplane 640 Interface card 516. In other embodiments, each interface card associated with the first and third levels of the first switch fabric portion is in a vertical position and each interface card associated with the second level of the first switch fabric portion is in a horizontal position. In another embodiment, each interface card associated with the first and third stages of the first switch fabric may be placed at any angle relative to the enclosure, and each interface card associated with the second stage of the first switch fabric An interface card may be orthogonal to the angular position of the interface cards associated with the first and third stages of the first switch fabric portion relative to the enclosure.

11 and 12 are schematic diagrams illustrating a switch fabric 1100 in a first configuration and a second configuration, respectively, according to one embodiment. Switch fabric 1100 includes a plurality of switch fabric systems 1108 .

Each switch fabric system 1108 includes a plurality of input/output modules 1102, a first cable set 1140, a second cable set 1142, a first switch fabric portion 1171 disposed within an enclosure 1170, and a second switch fabric portion disposed within an enclosure 1172 Structural part 1173. Each switch fabric system 1108 is similar in structure and function. Furthermore, input/output module 1102, first cable set 1140, and second cable set 1142 are structurally and functionally similar to input/output module 202, first cable set 240, and second cable set 242, respectively.

When switch fabric 1100 is in the first configuration, first switch fabric portion 1171 and second switch fabric portion 1173 of each switch fabric system 1108 function similarly to first switch fabric portion 571 and second switch fabric portion 573 described above. Thus, when the switch fabric 1100 is in the first configuration, the first switch fabric portion 1171 and the second switch fabric portion 1173 operate as independently existing three-level switch fabrics. Thus, when switch fabric 1100 is in the first configuration, each switch fabric system 1108 acts as a stand-alone switch fabric system and is not operably coupled to the other switch fabric systems 1108 .

In a second configuration ( FIG. 12 ), switch fabric 1100 further includes a third set of cables 1144 and a plurality of connecting switch fabrics 1191 , each located within enclosure 1190 . Housing 1190 may be similar to housing 600 described in detail above. Each switch fabric portion 1171 , 1173 of each switch fabric system 1108 is operatively coupled to each connection switch fabric 1191 via a third set of cables 1144 . As such, each switch fabric system 1108 is operably coupled to the other switch fabric systems 1108 via connection switch fabric 1191 when switch fabric 1100 is in the second configuration. Thus, the switch fabric 1100 in the second configuration is a 5-level Clos network.

The third set of cables 1144 can be composed of any material suitable for transferring data between the switch fabric portions 1171 , 1173 and the connecting switch fabric 1191 . In some embodiments, for example, each cable 1144 is comprised of multiple optical fibers. In such an embodiment, each cable 1144 may have 36 transmit and 36 receive fibers. The 36 transmit optical fibers of each cable 1144 may include 32 optical fibers for transmitting data, and 4 optical fibers for expanding data capacity and/or for redundancy. Similarly, the 36 receive fibers of each cable 1144 include 32 fibers for transmitting data, and 4 fibers for expanding data capacity and/or for redundancy. In other embodiments, any number of optical fibers can be included in each cable. By using cables with an increased number of optical fibers, the number of cables used can be effectively reduced.

As discussed above, flow control can be performed within the switching fabric of, for example, a data center. Figures 13 and 14, and accompanying description, are schematic diagrams illustrating flow control within a switch fabric. In particular, Figure 13 is a schematic diagram illustrating data flow associated with switch fabric 1300, according to one embodiment. Switch fabric 1300 shown in FIG. 13 is similar to switch fabric 400 shown in FIG. 4A and can be implemented in a data center such as data center 100 shown in FIG. 1 . In this embodiment, switch fabric 1300 is a 3-stage non-blocking Clos network and includes a first stage 1340 , a second stage 1342 , and a third stage 1344 . First stage 1340 includes module 1312 , second stage 1342 includes module 1314 , and third stage 1344 includes module 1316 . In some embodiments, the switch fabric 1300 may be a cell switch switch fabric and each module 1312 of the first stage 1340 may be a cell switch. Each module 1312 of the first stage 1340 includes a set of input ports 1360 configured to receive data as it enters the switch fabric 1300 . Each module 1316 of third stage 1344 includes an output port 1362 configured to allow data to exit switch fabric 1300 . Each module 1316 of the third stage 1344 includes the same number of output ports 1362 .

Each module 1314 of the second stage 1342 is operatively coupled to each module of the first stage 1340 through a unidirectional data path 1320 . Each unidirectional data path 1320 between each module of the first stage 1340 and each module 1314 of the second stage 1342 is configured to facilitate the transfer of data from the module 1312 of the first stage 1340 to the module of the second stage 1342 1314. Because the data path 1320 is unidirectional, it does not cause data to be transferred from the module 1314 of the second stage 1342 to the module 1312 of the first stage 1340 . Such a unidirectional data path 1320 costs less, uses fewer data connections, and is easier to implement than a similar bidirectional data path.

Each module 1316 of the third stage 1344 is operatively coupled to each module 1314 of the second stage 1342 by a unidirectional data path 1324 . Each unidirectional data path 1324 between a module 1314 of the second stage 1342 and a module 1316 of the third stage 1344 is configured to facilitate the transfer of data from the module 1314 of the second stage 1342 to the module 1316 of the third stage 1344 . Because data path 1324 is unidirectional, it does not cause data to pass from module 1316 of third stage 1344 to module 1314 of second stage 1344 . As noted above, such a unidirectional data path 1324 is less expensive and uses less area than a similar bidirectional data path.

Unidirectional data path 1320 between module 1312 of first stage 1340 and module 1314 of second stage 1342 and/or unidirectional data path between module 1314 of second stage 1342 and module 1316 of third stage 1344 Can be constructed in any manner configured to effectively facilitate data transfer. In some embodiments, for example, the data paths are optical connectors between modules. In other embodiments, the data path is within the midplane connector. Such a midplane connector may be similar to the midplane connector described in FIGS. 8 to 10 . Such midplane connectors can be effectively used to connect every module of the second stage to every module of the third stage. In other embodiments, the modules are contained in separate chip packages and the unidirectional data paths are electronic traces.

Each module 1312 of the first stage 1340 is in physical proximity relative to a corresponding module 1316 of the third stage 1344 . In other words, each module 1312 of the first stage 1340 is paired with a module 1316 of the third stage 1344 . For example, in some embodiments, each module 1312 of the first stage 1340 is in the same chip package as the module 1316 of the third stage 1344 . A bidirectional flow control path 1322 exists between each module 1312 of the first stage 1340 and a corresponding module 1316 of the third stage 1344 . The flow control path 1322 allows a module 1312 of the first stage 1340 to send a flow control indicator to a corresponding module 1316 of the third stage 1344, and vice versa. As described in further detail herein, this allows any module at any level of the switch fabric to send a flow control indicator to the module it is sending data to. In some embodiments, bidirectional flow control path 1322 is constructed from two separate unidirectional flow control paths. Two separate unidirectional flow control paths allow flow control indicators to pass between module 1312 of the first stage 1340 and module 1316 of the third stage 1344 .

FIG. 14 is a diagram illustrating flow control in the switch fabric 1300 shown in FIG. 13 according to one embodiment. In particular, the schematic diagram shows a detailed view of the first row 1310 of the switch fabric 1300 shown in FIG. 13 . The first row includes a module 1312' of the first stage 1340, a module 1314' of the second stage 1342, and a module 1316' of the third stage 1344. The module 1312' of the first stage 1340 includes a processor 1330 and a memory 1332. Processor 1330 is configured to control receiving and sending data. The memory 1332 is configured to buffer data when the modules 1314' of the second stage 1342 are not yet able to receive data and/or the modules 1312' of the first stage 1340 are not yet able to transmit data. In some embodiments, for example, if module 1314' of second stage 1342 has sent an abort indicator to module 1312' of first stage 1340, module 1312' of first stage 1340 buffers data until module 1312' of second stage 1342 Module 1314' is capable of receiving data. Similarly, in some embodiments, the modules 1312' of the first stage 1340 can buffer data when the modules 1312' receive multiple data signals (e.g., from multiple input ports) at substantially the same time. In such an embodiment, if only a single data signal can be output by module 1312' at a given time (e.g., every clock cycle), then other received data signals can be buffered. Similar to modules 1312' of first stage 1340, each module within switch fabric 1300 includes a processor and memory.

Both the module 1312' of the first stage 1340 and the module 1316' of the third stage 1344 paired with it are included on the first chip package 1326. This allows the flow control path 1322 between the module 1312' of the first stage 1340 and the module 1316' of the third stage 1344 to be easily constructed. For example, the flow control path 1322 may be a trace on the first chip package 1326 between the module 1312' of the first stage 1340 and the module 1316' of the third stage. In other embodiments, the modules of the first level and the modules of the third level are on separate chip packages but in close proximity to each other, which still allows a flow control path between them without using extensive wiring and/or long A trajectory can be created.

The modules 1314' of the second stage 1342 are included on the second chip package 1328. The unidirectional data path 1320 between the module 1312' of the first stage 1340 and the module 1314' of the second stage 1342, and the unidirectional data path 1320 between the module 1314' of the second stage 1342 and the module 1316' of the third stage 1344 A data path 1324 operably connects the first chip package 1326 to the second chip package 1328 . Although not shown in Figure 14, module 1312' of the first stage 1340 and module 1316' of the third stage 1344 are also connected to each module of the second stage by unidirectional data paths. As noted above, unidirectional data paths can be constructed in any manner configured to effectively facilitate the transfer of data between modules.

Flow control path 1322 and unidirectional data paths 1320, 1324 can be effectively used to send flow control indicators between modules 1312', 1314', 1316'. For example, if the module 1312' of the first stage 1340 is sending data to the module 1314' of the second stage 1342 and the amount of data in the buffer of the module 1314' of the second stage 1342 exceeds a threshold, then the module of the second stage 1342 Module 1314' can send a flow control indicator to module 1316' of third stage 1344 via unidirectional data path 1324 between module 1314' of second stage 1342 and module 1316' of third stage 1344. This flow control indicator triggers module 1316' of third stage 1344 to send a flow control indicator to module 1312' of first stage 1340 via flow control path 1322. The flow control indicator sent from the module 1316' of the third stage 1344 to the module 1312' of the first stage 1340 causes the module 1312' of the first stage 1340 to stop sending data to the module 1314' of the second stage 1342. Similarly, a flow control indicator sent from module 1314' of second stage 1342 to module 1312' of first stage 1340 via module 1316' of third stage 1344 requests flow from module 1312' of first stage 1340 to module 1312' of second stage 1340. Module 1314' of stage 1342 sends data (ie, continues to send data).

The two-level switch fabric within the same chip pack with on-chip bi-directional flow control paths between them minimizes connections between individual chip packs that are bulky and/or require large volumes. In addition, two stages within the same package with an on-chip bi-directional flow control path between them allows the data path between chip packages to be unidirectional while providing flow control communication capability between the transmit module and the receive module . More details concerning bidirectional flow control paths within a switch fabric are in co-pending U.S. patent application entitled "Flow Control in a Switch Fabric" and filed on December 29, 2008 No. 12/345490, which is incorporated herein by reference in its entirety.

As described in connection with Figures 13 and 14, buffer modules can be included within modules at the switch fabric level. More details relating to buffer modules that can be included, for example, within a switch fabric level will be described in connection with FIG. 15 .

FIG. 15 is a schematic diagram illustrating a buffer module 1500 according to one embodiment. As shown in FIG. 15, data signals S0 through SM are received at buffer module 1500 on input side 1580 of buffer module 1500 (eg, through input port 1562 of buffer module 1500). After buffer module 1500 processing, data signals S0 through SM are sent from buffer module 1500 on output side 1585 of buffer module 1500 (eg, through output port 1564 of buffer module 1500 ). Each of the data signals S0 to SM can define a channel (also referred to as a data channel). Data signals S0 through SM can collectively be referred to as data signals 1560 . Although the input side 1580 of the buffer module 1500 and the output side 1585 of the buffer module 1500 are shown as different physical sides of the buffer module 1500, the input side 1580 of the buffer module 1500 and the output side 1585 of the buffer module 1500 are logically defined and Various physical configurations of buffer module 1500 are not excluded. For example, one or more input ports 1562 and/or one or more output ports 1564 of buffer module 1500 can be physically located on any side (and/or the same side) of buffer module 1500 .

Buffer module 1500 can be configured to process data signal 1560 such that data signal 1560 processing latency through buffer module 1500 can be relatively small and substantially constant. Therefore, since the data signal 1560 is processed through the buffer module 1500, the bit rate of the data signal 1560 can be substantially unchanged. For example, the data signal S2 processing latency through the buffer module 1500 may be a substantially constant number of clock cycles (eg, a single clock cycle, several clock cycles). Accordingly, the data signal S2 may be time-shifted by a number of clock cycles, and the bit rate of the data signal S2 sent to the input side 1580 of the buffer module 1500 will be substantially the same as that sent from the output side 1585 of the buffer module 1500 The bit rate of the data signal S2 is the same.

Buffer module 1500 can be configured to modify the bit rate of one or more data signals 1560 in response to one or more portions of flow control signal 1570 . For example, buffer module 1500 can be configured to delay data signal S2 received at buffer module 1500 in response to a portion of flow control signal 1570 indicating that data signal S2 should be delayed for a particular period of time. In particular, buffer module 1500 can be configured to store (e.g., hold) one or more portions of data signal S2 until buffer module 1500 receives an indicator (e.g., flow control part of signal 1570). Accordingly, the bit rate of the data signal S2 sent to the input side 1580 of the buffer module 1500 is different (eg, substantially different) than the bit rate of the data signal S2 sent from the output side 1585 of the buffer module 1500 .

In some embodiments, processing at the buffer module 1500 can be performed at memory banks based on, for example, variable-sized cell segments. For example, in some embodiments, segments of cells can be processed during the allocation process by different memory banks (eg, Static Random Access Memory (SRAM) banks) included within buffer module 1500 . Memory banks can collectively define a shared memory buffer. In some embodiments, segments of a data signal can be allocated to memory banks in a predefined manner (eg, in a predefined pattern according to a predefined algorithm) during the allocation process. For example, in some embodiments, leading segments of the data signal 1560 can be processed in portions of the buffer module 1500 (e.g., specific memory banks of the buffer module 1500) that are related to the traces processed within the buffer module 1500. Several parts of the trailing segments are different. In some embodiments, the segments of data signal 1560 can be processed in a particular order. In some embodiments, for example, each segment of data signal 1560 may be processed based on its respective location within a cell. After the cell segments have been processed through the shared memory buffer, the cell segments can be ordered and sent from the buffer module 1500 during the reassembly process.

In some embodiments, for example, the read multiplexing module of buffer module 1500 can be configured to reassemble segments associated with data signal 1560 and send (eg, transmit) data signal 1560 from buffer module 1500 . The reassembly process can be defined based on a predefined methodology for allocating segments to the memory banks of the buffer module 1500 . For example, the read multiplexing module can be configured to first read the boot segment associated with the cell from the boot bank in a polled fashion (since the segment is written in a polled fashion), and then read the boot segment associated with the cell from the track bank Trace fragments associated with cells are read in a polling fashion. Therefore, very few, if any, control signals need to be sent between the write multiplexer and the read multiplexer. More details concerning fragment handling (e.g., fragment allocation and/or fragment reassembly) can be found in the document entitled "Methods and Apparatus Related to Shared Memory Buffer for Variable-SizedCells" and apparatus)" and described in co-pending U.S. Patent Application No. 12/415,517, filed March 31, 2009, which is incorporated herein by reference in its entirety.

16A is a schematic block diagram of an ingress scheduling module 1620 and an egress scheduling module 1630 configured to coordinate transmission of groups of cells via the switch fabric 1600 of the switching core 1690, according to one embodiment. Coordination may include, for example, scheduling transmission cell groups via switch fabric 1600, tracking requests and/or responses involving transmission cell groups, and the like. Ingress scheduling module 1620 can be included on the ingress side of switch fabric 1600 and egress scheduling module 1630 can be included on the egress side of switch fabric 1600 . Switch fabric 1600 can include ingress stage 1602 , intermediate stage 1604 , and egress stage 1606 . In some embodiments, the switch fabric 1600 can be defined based on a Clos network architecture (e.g., a non-blocking Clos network, a strictly non-blocking Clos network, a Benes (Barnes) network), and the switch fabric 1600 Can include data plane and control plane. In some embodiments, switch fabric 1600 may be a core portion of a data center (not shown), which can include a network or device interconnect.

As shown in FIG. 16A , input queues IQ1 through IQK (collectively referred to as ingress queues 1610 ) can be located on the ingress side of switch fabric 1600 . Ingress queue 1610 can be associated with ingress stage 1602 of switch fabric 1600 . In some embodiments, ingress queue 1610 can be included in a line card. In some embodiments, ingress queue 1610 can be located outside of switch fabric 1600 and/or outside of switch core 1690 . Each ingress queue 1610 may be a first-in-first-out (FIFO) type queue. Although not shown, in some embodiments, each ingress queue IQ1 through IQK may be associated (eg, uniquely) with an input/output port (eg, a 10Gb/s port). In some embodiments, each ingress queue IQ1 through IQK can be of sufficient size to implement congestion resolution, such as request grant congestion resolution. For example, input queue IQK-1 can be of sufficient size to hold a cell (or group of cells) until a request grant congestion scheme has been performed for the cell (or group of cells).

As shown in FIG. 16A , output ports P1 through PL (collectively referred to as output ports 1640 ) can be located on the output side of switch fabric 1600 . Output port 1640 may be associated with output stage 1606 of switch fabric 1600 . In some embodiments, output port 1640 can be referred to as a destination port.

In some embodiments, input queue 1610 can be included on one or more input line cards (not shown) located outside of input stage 1602 of switch fabric 1600 . In some embodiments, output ports 1640 may be included in one or more output line cards (not shown) located outside of output stage 1606 of switch fabric 1600 . In some embodiments, one or more input queues 1610 and/or one or more output ports 1640 can be included in one or more stages of switch fabric 1600 (eg, input stage 1602 ). In some embodiments, the output scheduling module 1620 can be included in one or more output line cards and/or the input scheduling module 1630 can be included in one or more input lines. In some embodiments, each line card (eg, output line card, input line card) associated with switch core 1690 may include one or more scheduling modules (eg, output scheduling module, input scheduling module).

In some embodiments, input queue 1610 and/or output port 1640 can be included in one or more gateway devices (not shown) between switch fabric 1600 and/or peripheral processing devices (not shown). One or more gateway devices, switch fabric 1600, and/or peripheral processing devices can collectively define at least a portion of a data center (not shown). In some embodiments, one or more gateway devices may be edge devices within the edge portion of switch core 1690 . In some embodiments, switch fabric 1600 and peripheral processing devices can be configured to process data based on different protocols. For example, the peripheral processing devices can include, for example, one or more master devices that can be configured to communicate based on the Ethernet protocol and the switch fabric 1600, which can be a cell-based fabric (e.g., configured to execute one or more virtual resource host, web server). In other words, one or more gateway devices can provide access to switch fabric 1600 to other devices configured to communicate via one protocol, which can be configured to communicate via another protocol. In some embodiments, one or more gateway devices can be referred to as access switches or network devices. In some embodiments, one or more gateway devices can be configured as routers, hub devices, and/or bridge devices.

In this embodiment, for example, the input scheduling module 1630 can be configured to define the cell group GA queued at the input queue IQ1 and the cell group GC queued at the input queue IQK-1. The cell group GA is queued at the front of the input queue IQ1, and the cell group GB is queued after the cell group GA in the input queue IQ1. Since input queue IQ1 is a FIFO type queue, cell group GB cannot be sent via switch fabric 1600 until cell group GA has been sent from input queue IQ1. The cell group GC is queued at the front of the input queue IQK-1.

In some embodiments, portions of input queue 1610 can be mapped to (eg, assigned to) one or more output ports 1640 . For example, input queues IQ1 to IQK-1 can be mapped to output port P1, so that all cells 310 queued at input ports 1Q1 to IQK-1 will be scheduled by input scheduling module 1620 for transmission to output port P1 via switch fabric 1600. Similarly, input queue IQK can be mapped to output port P2. This mapping can be stored in memory (eg, memory 1622 ) as, for example, a look-up table that input scheduling module 1620 can access when scheduling (eg, requesting) transmission of groups of cells.

In some embodiments, one or more input queues 1610 can be associated with priority values (also referred to as transmission priority values). The input scheduling module 1620 can be configured to schedule transmission of cells from the input queue 1610 based on a priority value. For example, because input queue IQK-1 can be associated with a higher priority value than input queue IQ1, input scheduling module 1620 can be configured to request cell group GC to transfer to before requesting cell group GA to transfer to output port P1. output port P1. Priority values can be defined based on service levels (eg, Quality of Service (QoS)). For example, in some embodiments, different types of network traffic may be associated with different service levels (and different priorities). For example, storage traffic (eg, read and write traffic), inter-processor communication, media signaling, session layer signaling, etc. are each associated with at least one service level. In some embodiments, the priority value can be based on, for example, the IEEE 802.1qbb protocol, which defines a priority-based flow control policy.

In some embodiments, one or more input queues 1610 and/or one or more output ports 1640 may be suspended. In some embodiments, one or more input queues 1610 and/or one or more output ports 1640 can be paused so that cells are not lost. For example, if output port P1 is temporarily unavailable, cell transmission from input queue IQ1 and/or input queue IQK-1 can be suspended so that cells at output port P1 are not lost due to output port P1 being temporarily unavailable. In some embodiments, one or more input queues 1610 may be associated with a priority value. For example, if output port P1 is congested, the transfer of cells from input queue IQ1 to output port P1 may be suspended, but not the transfer of cells from input queue IQK-1 to output port P1 because input queue IQK-1 can communicate with Associated with a higher priority value than input queue 1Q1.

The input scheduling module 1620 can be configured to exchange signals with (e.g., send signals to and receive signals from) the output scheduling module 1630 to coordinate the transmission of the cell group GA to the output port P1 via the switch fabric 1600, and to coordinate the transmission of the group GA via the switch fabric 1600. The cell group GC is transmitted to the output port P1. Since the cell group GA will be sent to the output port P1, this output port P1 can be called the destination port of the cell group GA. Similarly, the output port P1 can be called the destination port of the cell group GB. As shown in FIG. 16A, the cell group GA can be transmitted via a transmission path 4112, which is different from the transmission path 4114 for transmitting the cell group GC.

Cell group GA and cell group GB are defined by the input scheduling module 1620 based on the cells 4110 queued at the input queue IQ1. In particular, a cell group GA can be defined based on each cell from the cell group GA having a common destination port and having a specific position within the input queue IQ1. Similarly, a cell group GC can be defined based on each cell from the cell group GC having a common destination port and having a specific position within the input queue IQK-1. Although not shown, in some embodiments, for example, cell 4110 can comprise switching core 1690 from one or more peripheral processing devices (e.g., personal computers, servers, routers, personal digital assistants (PDAs)) via one or more Content (eg, data packets) received by a plurality of networks (eg, local area network (LAN), wide area network (WAN), virtual network), which may be wired and/or wireless. Further details concerning the definition of cell groups, such as cell groups GA, cell groups GB and/or cell groups GC, are discussed in conjunction with FIGS. 17 and 18 .

Figure 16B is a signaling flow diagram illustrating signaling related to cell group GA transmission, according to one embodiment. As shown in Figure 16B, the time increases in the downstream direction. After the cell group GA has been defined (as shown in FIG. 16A ), the input scheduling module 1620 can be configured to send a request to schedule the cell group GA for transmission via the switch fabric 1600 ; this request is shown as a transmission request 22 . The transfer request 22 can be defined as a request to send the cell group GA to the destination port of the cell group GA, ie output port P1. In some embodiments, the destination port of the cell group GA can also be referred to as the target of the transfer request 22 (also referred to as the target destination port). In some embodiments, the transmission request 22 may include a request to send the cell group GA through the switch fabric 1600 via a specific transmission path (eg, transmission path 4112 shown in FIG. 16A ), or at a specific time. The input scheduling module 1620 can be configured to send the transmission request 22 to the output scheduling module 1630 after the transmission request 22 has been defined in the input scheduling module 1620 .

In some embodiments, transfer requests 22 can be queued on the input side of switch fabric 1600 before being sent to the output side of switch fabric 1600 . In some embodiments, the transfer request 22 can be queued until the input scheduling module 1620 triggers sending the transfer request 22 to the output side of the switch fabric 1600 . In some embodiments, because the capacity for transfer requests sent from the input side of the switch fabric 1600 is above a threshold, the input scheduling module 1620 can be configured to hold (or trigger holding) the transfer requests 22 in, for example, the input transfer request queue ( not shown). The threshold can be defined based on transmission latency via switch fabric 1600 .

In some embodiments, transfer requests 22 can be queued in an output queue (not shown) on the output side of switch fabric 1600 . In some embodiments, output queues can be included in line cards (not shown) located inside or outside switch fabric 1600 , or outside switch core 1690 . Although not shown, in some embodiments, a transfer request 22 can be queued at an output queue or a portion of an output queue associated with a particular input queue (eg, input queue IQ1). In some embodiments, each output port 1640 can be associated with an output queue that is associated with (eg, corresponds to) the priority value of the input queue 1610 . For example, output port P1 can be associated with an output queue (or portion of an output queue) associated with input queue IQ1 (which has a particular priority value) and an output queue associated with input queue IQK (which has a particular priority value). queue (or part of an output queue) to associate with. Thus, a transfer request 22 queued at input queue IQ1 can be queued at the output queue associated with input queue IQ1. In other words, the transfer request 22 can be queued (on the output side of the switch fabric 1600 ) on an output queue associated with the priority value of at least one input queue 1610 . Similarly, transfer requests 22 can be queued in an input transfer request queue (not shown) or a portion of an input transfer queue associated with at least one input queue 1610 priority value.

If the output scheduling module 1630 determines that the destination port of the cell group GA (ie, the output port P1 shown in FIG. 16A ) is available for receiving the cell group GA, the output scheduling module 1630 can be configured to send a transmission response to the input scheduling module 1620 twenty four. Transport response 24 may be, for example, an authorization for cell group GA to be sent (eg, sent from input queue send IQ1 shown in FIG. 16A ) to the destination port of cell group GA. An authorization to send a group of cells can be called a transmission authorization. In some embodiments, cell group GA and/or input queue IQ1 can be referred to as the target of transmission response 24 . In some embodiments, authorization for a cell group GA to be sent can be granted when transmission through switch fabric 1600 is substantially authorized, eg, because the destination port is available.

In response to the transmission response 24 , the input scheduling module 1620 can be configured to send the cell group GA via the switch fabric 1600 from the input side of the switch fabric 1600 to the output side of the switch fabric 1600 . In some embodiments, transport response 24 can include an instruction to send cell group GA through switch fabric 1600 via a particular transport path (eg, transport path 4112 shown in FIG. 16A ), or at a particular time. In some embodiments, the instructions can be defined based on routing policies, for example.

As shown in Figure 16B, the transfer request 22 includes a cell quantity value 30, a destination identifier (ID) 32, a queue identifier (ID) 34, a queue sequence value (SV) 36 (which may collectively be referred to as a request tag) . The number of cells value 30 can represent the number of cells included in the cell group GA. For example, in this embodiment, the cell group GA includes seven (7) cells (shown in FIG. 16A ). The destination identifier 32 can indicate the destination port of the cell group GA so that the destination of the transmission request 22 can be determined by the output scheduling module 1630 .

The cell quantity value 30 and the destination identifier 32 can be used by the output scheduling module 1630 to schedule the transmission of the cell group GA via the switch fabric 1600 to the output port P1 (shown in FIG. 16A ). As shown in FIG. 16B, in this embodiment, since the number of cells included in the cell group GA can be processed at the destination section port (for example, the output port P1 shown in FIG. 16A ) of the cell group GA ( For example, can be received), the output scheduling module 1630 can be configured to define and send the transmission response 24.

In some embodiments, if the number of cells included in the cell group GA cannot be included in the number of cells in the cell group GA because the destination port of the cell group GA is unavailable (for example, in an unavailable state, in a congested state), If the destination port (eg, output port P1 shown in FIG. 16A ) is processed (eg, cannot be received), the output scheduling module 1630 can be configured to be unavailable for communication to the input scheduling module 1620 . In some embodiments, for example, the output scheduling module 1630 can be configured to deny a request (not shown) to send a cell group GA via the switch fabric 1600 when the destination port of the cell group GA is unavailable. A denial of a transfer request 22 can be referred to as a transfer denial. In some embodiments, a transfer rejection can include a response tag.

In some embodiments, availability or unavailability of, for example, output port P1 (shown in FIG. 16A ) can be determined by output scheduling module 1630 based on satisfied conditions. For example, the conditions can relate to exceeding the memory limit of a queue (not shown in FIG. shown in FIG. 16A) the number of cells transmitted and so on. In some embodiments, when output port P1 is disabled, output port P1 is unavailable for receiving cells via switch fabric 1600 .

As shown in FIG. 16B , the queue identifier 34 and the queue sequence value 36 are sent in the transmission request 22 to the output scheduling module 1630 . The queue identifier 34 can represent and/or can be used to identify (eg, individually identify) the input queue IQ1 (shown in FIG. 16A ) in which the cell group GA is queued. The queue sequence value 36 can indicate the position of the cell group GA relative to other cell groups in the input queue IQ1. For example, cell group GA can be associated with queue sequence value x and cell group GB (queued at input queue IQ1 as shown in FIG. 16A) can be associated with queue sequence value Y. The queue sequence value x can indicate that the cell group GA is to be sent from the input queue IQ1 before the cell group GB associated with the queue sequence value Y.

In some embodiments, queue sequence value 36 is selected from the range of queue sequence values associated with input queue IQ1 (shown in FIG. 16A ). Ranges of queue sequence values can be defined such that sequence values from within the range of queue sequence values are not repeated for input queue IQ1 within a specified time period. For example, ranges of queue sequence values can be defined such that queue sequence values from within the range of queue sequence values are not repeated for at least a period of time that requires clearing of some of the queue sequence values by switch core 1690 (shown in FIG. 16A ). The number of cell periods (eg, cells 160) that are queued in input queue IQ1. In some embodiments, a queue sequence value can be incremented (within a queue sequence value range) and associated with each cell group defined by the input scheduling module 1620 based on cells 4110 queued at the input queue IQ1.

In some embodiments, the range of queue sequence values associated with input queue IQ1 can overlap with the range of queue sequence values associated with another of input queues 1610 (shown in FIG. 16A ). Therefore, the queue sequence value 36, even from a non-unique range of queue sequence values, can be included (e.g., included) the queue identifier 34 (which can be unique) to uniquely identify the cell group GA (at least in during a specific period of time). In some embodiments, the queue sequence value 36 is unique or a globally unique value (GUID) (eg, a universally unique identifier (UUID)) within the switch fabric 1600 .

In some embodiments, the input scheduling module 1620 can be configured to wait to define a transmission request (not shown) associated with a cell group GB. For example, the input scheduling module 1620 can be configured to wait until a transfer request 22 is sent or to wait until a response in response to a transfer request 22 (e.g., a transfer response 24, a transfer reject) is received prior to defining a transfer request associated with a cell group GB. take over.

As shown in Figure 16B, the output scheduling module 1630 can be configured to include a queue identifier 34 and a queue sequence value 36 (which can collectively be referred to as a response tag) in the transmission response 24. When a transport response 24 is received at the input scheduling module 1620 , the queue identifier 34 and the queue sequence value 36 can be included in the transport response 24 so that the transport response 24 can be associated with the cell group GA at the input scheduling module 1620 . In particular, queue identifier 34 and queue sequence value 36 together can be used to identify cell group GA as authorized for transmission via switch fabric 1600 .

In some embodiments, output scheduling module 1630 can be configured to delay sending transmission responses 24 corresponding to transmission requests 22 . In some embodiments, output scheduling module 1630 can be configured to delay the response if, for example, the destination port of cell group GA (ie, output port P1 shown in FIG. 16A ) is unavailable (eg, temporarily unavailable). In some embodiments, the output scheduling module 1630 can be configured to send the transmission response 24 in response to the output port P1 changing from the unavailable state to the available state.

In some embodiments, the output scheduling module 1630 can be configured to delay sending the transport response 24 because the destination port of the cell group GA (ie, the output port P1 shown in FIG. 16A ) receives data from another input queue 1610. . For example, because output port P1 receives a different group of cells (not shown) from, for example, input queue IQK (shown in FIG. 16A ), output port P1 is not available to receive data from input queue IQ1. In some embodiments, groups of cells from input queue IQ1 can have a higher priority value than groups of cells from input queue IQK based on the priority values associated with input queue IQ1 and input queue IQK. The output scheduling module 1630 can be configured to delay sending the transmission response 24 for a time period calculated based on, for example, the size of the different groups of cells received at the output port P1. For example, output scheduling module 1630 can be configured to delay sending transmission response 24 targeted for cell group GA for a desired period of time in order to complete processing at output port P1 for a different cell group. In other words, the output scheduling module 1630 can be configured to delay sending the transmission response 24 targeted to the cell group GA based on a predetermined time for the output port P1 to change from the unavailable state to the available state.

In some embodiments, the output scheduling module 1630 can be configured to delay sending transmissions because at least a portion of the transmission path (e.g., transmission path 4112 shown in FIG. 16A ) over which the cell group GA is sent is unavailable (e.g., congested). Response 24. The output scheduling module 1630 can be configured to delay sending the transmission response 24 until the portion of the transmission path is no longer congested, or a predetermined time based on the portion of the transmission path is no longer congested.

As shown in FIG. 16B , the cell group GA can be sent to the destination port of the cell group GA based on (eg, in response to) the transport response 24 . In some embodiments, cell group GA can be sent based on one or more instructions included in transmission response 24 . For example, in some embodiments, cell group GA can be based on instructions included in transmission response 24 via transmission path 4112 (shown in FIG. 16A ), or based on one or more cell group Rules for transmission (eg, rules for cell group transmission via the reconfigurable switch fabric) are sent. Although not shown, in some embodiments, after cell group GA has been received at output port P1 (shown in FIG. A network (eg, LAN, WAN, virtual network), which may be wired and/or wireless, is sent to one or more network entities (eg, personal computer, server, router, PDA).

Referring back to FIG. 16A , in some embodiments, cell group GA is sent via transmission path 4112 and received at an output queue (not shown) that is relatively small compared to, for example, input queue 1610 . In some embodiments, an output queue (or a portion of an output queue) can be associated with a priority value. A priority value can be associated with one or more input queues 1610 . The output scheduling module 1630 can be configured to fetch the group of cells GA from the output queue and can be configured to send the group of cells GA to the output port P1.

In some embodiments, when the cell group GA is sent to the output side of the switch fabric 1600, the cell group GA can be extracted by the input scheduling module 1620 and sent to the output side along with the response identifier included in the cell group GA. Port P1. A response identifier can be defined at the output scheduling module 1630 and included in the transmission response 24 . In some embodiments, if the cell group GA is queued in an output queue (not shown) associated with the cell group GA's destination port, the response identifier can be used to extract the information from the cell group GA's destination port. A tuple GA, and thus a cell group GA, can be sent from the switch fabric 1600 via the destination port of the cell group GA. The response identifier can be associated with a position in the output queue that has been reserved by the output scheduling module 1630 for the queuing of the cell group GA.

In some embodiments, cell groups queued in input queue 1610 can be moved to memory 1622 when a transfer request (eg, transfer request 22 shown in FIG. 16B ) associated with the cell group is defined. For example, cell groups GD queued at input queue IQK can be moved to memory 1622 in response to a transfer request associated with cell groups GD being defined. In some embodiments, the cell group GD can be moved to the memory 1622 before the transmission request associated with the cell group GD is sent from the input scheduling module 1620 to the output scheduling module 1630 . The cell group GD can be stored in the memory 1622 until the cell group GD is sent from the input side of the switch fabric 1600 to the output side of the switch fabric 1600 . In some embodiments, groups of cells can be moved to memory 1622, thereby reducing congestion at the input queue IQK (eg, head-of-line (HOL) blocking).

In some embodiments, input scheduling module 1620 can be configured to retrieve groups of cells stored in memory 1622 based on queue identifiers and/or queue sequence values associated with the groups of cells. In some embodiments, the cell group location of a cell within memory 1622 can be determined based on a lookup table and/or index values. Groups of cells can be extracted before groups of cells are sent from the input side of switch fabric 1600 to the output side of switch fabric 1600 . For example, a cell group GD can be associated with a queue identifier and/or a queue sequence value. The location at which a cell group GD is stored within memory 1622 can be associated with a queue identifier and/or a queue sequence value. Transmission requests defined by the input scheduling module 1620 and sent to the output scheduling module 1630 can include queue identifiers and/or queue sequence values. The transmission response received from the output scheduling module 1630 can include a queue identifier and/or a queue sequence value. In response to the transmission response, input scheduling module 1620 can be configured to fetch cell group GD from memory 1622 at a location based on the queue identifier and/or queue sequence value, and input scheduling module 1620 can trigger transmission of cell group GD.

In some embodiments, some number of cells included in a cell group can be defined based on the amount of space available in memory 1622 . For example, the input scheduling module 1620 can be configured to determine the number of cells to include in the cell group GD based on the amount of available storage space included in the memory 1622 when the cell group GD was defined. In some embodiments, if the amount of available storage space included in the memory 1622 is increased, the number of cells included in the cell group GD can be increased. In some embodiments, the number of cells included in a cell group GD can be increased by the input scheduling module 1620 before and/or after the cell group GD is moved to the memory 1622 for storage.

In some embodiments, the number of certain cells included in a cell group can be defined based on latency of transmission through switch fabric 1600, for example. In particular, the input scheduling module 1620 can be configured to define the size of a cell group to drive traffic through the switch fabric 1600 in view of the latency associated with the switch fabric 1600 . For example, the input scheduling module 1620 can be configured to close a cell group (eg, define the size of the cell group) because the cell group has reached a threshold size defined based on the latency of the switch fabric 1600 . In some embodiments, the input scheduling module 1620 can be configured to send data packets in a cell group immediately rather than waiting for additional data packets to define a larger cell group because of the low latency through the switch fabric 1600 .

In some embodiments, the input scheduling module 1620 can be configured to limit the number of transmission requests sent from the input side of the switch fabric 1600 to the output side of the switch fabric 1600 . In some embodiments, the limits can be defined based on policies stored in the input scheduling module 1620 . In some embodiments, the limit can be defined based on priority values associated with one or more input queues 1610 . For example, input scheduling module 1620 can be configured to allow (based on a threshold limit) more transmission requests associated with input queue IQ1 than from input queue IQK because input queue IQ1 has a higher priority value than input queue IQK .

In some embodiments, one or more portions of input scheduling module 1620 and/or output scheduling module 1630 may be hardware-based modules (e.g., DSP, FPGA) and/or software-based modules (e.g., computer code modules, A set of processor-readable instructions that can be executed on a processor). In some embodiments, one or more functions associated with the input scheduling module 1620 and/or the output scheduling module 1630 can be included in different modules and/or combined into one or more modules. For example, cell group GA can be defined by a first sub-module within input scheduling module 1620 and transmission request 22 (shown in FIG. 16B ) can be defined by a second sub-module within input scheduling module 1620 .

In some embodiments, switch fabric 1600 has more or fewer stages than shown in Figure 16A. In some embodiments, switch fabric 1600 may be a reconfigurable (eg, reconfigurable) switch fabric and/or a time division multiplexing switch fabric. In some embodiments, the switch fabric 1600 can be defined based on a Clos network architecture (eg, strictly non-blocking Clos network, Benes network).

Figure 17 is a schematic block diagram illustrating two groups of cells queued at input queue 1720 located on the input side of switch fabric 1700, according to one embodiment. Cell groups are defined by input scheduling module 1740 on the input side of switch fabric 1700, which may be, for example, associated with and/or included in a switch core such as that shown in FIG. 16A. Input queue 1720 is also on the input side of switch fabric 1700 . In some embodiments, input queue 1720 can be included in an input line card (not shown) associated with switch fabric 1700 . Although not shown, in some embodiments, one or more cell groups can include multiple cells (eg, 25 cells, 10 cells, 100 cells) or only one cell.

As shown in FIG. 17 , input queue 1720 includes cells 1 through T (ie, cells 1 through T), which can collectively be referred to as queued cells 1710 . The input queue 1720 is a FIFO type queue with cell 1 at the front end 1724 (or transmission end) of the queue and cell T at the back end 1722 (or entry end) of the queue. As shown in FIG. 17 , queued cells 1710 at input queue 1720 include a first group 1712 of cells and a second group 1716 of cells. In some embodiments, each cell from queued cells 1710 has an equal length (eg, 32 bit length, 64 bit length). In some embodiments, two or more of the queued cells 1710 can have different lengths.

Each cell from queued cells 1710 has four output ports indicated by an output port label (e.g., letter "E", letter "F") on each cell from queued cells 1710 One of 1770 - output port E, output port F, output port G, or output port H transfers the queued content. The output port 1770 to which the cell is sent can be referred to as a destination port. Each of the queued cells 1710 can be sent via the switch fabric 1700 to its corresponding destination port. In some embodiments, input scheduling module 1740 can be configured to determine a destination port for each cell from queued cells 1710 based on, for example, a look-up table (LUT) such as a routing table. In some embodiments, the destination port of each cell from queued cells 1710 can be determined based on the destination of the content (eg, data) included within the cell. In some embodiments, one or more output ports 1770 can be associated with an output queue in which cells can be queued until sent via the output port 1770 .

The first cell group 1712 and the second cell group 1716 can be defined by the input scheduling module 1740 based on the destination port of the queued cells 1710 . As shown in FIG. 17, each cell included in the first cell group 1712 has the same destination port (ie, output port E) indicated by the output port label "E". Similarly, each cell included in the second cell group 1716 has the same destination port indicated by the output port label "F" (ie, output port F).

Groups of cells (eg, first group of cells 1712 ) can be defined based on destination ports because groups of cells are sent via switch fabric 1700 as groups. For example, if Cell 1 is included in First Cell Group 1712, then First Cell Group 1712 cannot be sent to a separate destination port because Cell 1 has the same E") a different destination port (output port "F"). As such, the first group of cells 1712 is not transmitted via the switch fabric 1700 as a group.

A cell group is defined as a contiguous block of cells because the cell group is sent as a group via the switch fabric 1700 and because the input queue 1720 is a FIFO-type queue. For example, Cell 12, and Cell 2 through Cell 7 cannot be defined as a group of cells because Cell 12 cannot be sent together with a cell block of Cell 2 through Cell 7. Cells 8 through 11 are intermediate cells that must be sent from input queue 1720 after cells 2 through 7 are sent from input queue 1720, but before cell 12 is sent from input queue 1720. send. In some embodiments, if the input queue 1720 is not a FIFO type queue, one or more queued cells 1710 may be sent out of order and groups may span intervening cells.

Although not shown, each cell from queued cells 1710 may have a sequence value that can be referred to as a cell sequence value. The cell sequence value can indicate, for example, the order of cell 2 with respect to cell 3. The cell sequence value can be used to rearrange cells at, for example, one or more output ports 1770 before content associated with the cells is transmitted from the output ports 1770 . For example, in some embodiments, group of cells 1712 can be received at an output queue (not shown) associated with output port E and rearranged based on the cell sequence value. In some embodiments, the output queue may be relatively small compared to the input queue 1720 (eg, a shallow output queue).

Furthermore, the data (eg, data packets) included in cells can also have a sequence value called a data sequence value. For example, a data sequence value can represent, for example, the relative order of a first data packet with respect to a second data packet. The data sequence values can be used to rearrange data packets at, for example, one or more output ports 1770 before the data packets are sent from the output ports 1770 .

FIG. 18 is a schematic block diagram illustrating two groups of cells queued at input queue 1820 located on the input side of switch fabric 1800 according to another embodiment. Cell groups are defined by input scheduling module 1840 on the input side of switch fabric 1800, which may be, for example, associated with and/or included in a switch core as shown in FIG. 16A. Input queue 1820 is also on the input side of switch fabric 1800 . In some embodiments, input queue 1820 can be included in an input line card (not shown) associated with switch fabric 1800 . Although not shown, in some embodiments one or more cell groups can include only one cell.

As shown in FIG. 18 , input queue 1820 includes cells 1 through Z (ie, cell 1 through cell Z), which are collectively referred to as queued cells 1810 . The input queue 1820 is a FIFO type queue with cell 1 at the front 1824 (or transmit end) of the queue and cell Z at the back 1822 (or ingress end) of the queue. As shown in FIG. 18 , queued cells 1810 at input queue 1820 include a first group 1812 of cells and a second group 1816 of cells. In some embodiments, each cell from queued cells 1810 has an equal length (eg, 32 bit length, 64 bit length). In some embodiments, two or more queued cells 1810 have different lengths. In this embodiment, input queue 1820 is mapped to output port F2 such that all cells 1810 are scheduled by input scheduling module 1840 for transmission to output port F2 via switch fabric 1800 .

Each cell from queued cells 1810 has content associated with one or more data packets (eg, Ethernet data packets). The data packets are represented by the letters "Q" to "Y". For example, as shown in FIG. 18, a data packet R is segmented into three different cells, Cell2, Cell3 and Cell4.

Cell groups (eg, first cell group 1812) are defined such that portions of data packets are not associated to different cell groups. In other words, cell groups are defined such that all data packets are associated to a single cell group. Cell group boundaries are defined based on the boundaries of data packets queued at input queue 1820 such that data packets are not included in different cell groups. Splitting data packets into different groups of cells may lead to undesired results, such as buffering on the output side of switch fabric 1800 . For example, if a first portion of data packet T (e.g., cell 6) is included in first cell group 1812 and a second portion of data packet T (e.g., cell 7) is included in second cell group 1816, then The first portion of the data packet T must be buffered in at least a portion of one or more output queues (not shown) on the output side of the switch fabric 1800 until the second portion of the data packet T is sent to the output side of the switch fabric 1800, thereby All data packets T are sent from the switch fabric 1800 via the output port E2.

In some embodiments, data packets included within queued cells 1810 can also have sequence values, referred to as data sequence values. The data sequence value can represent, for example, the relative order of the data packet R with respect to the data packet S. The data sequence value can be used to reassemble the data packet at, for example, one or more output ports 1870 before the data packet is sent from the output port 1870 .

Figure 19 is a flowchart illustrating a method for scheduling cell group transmission via a switch fabric, according to one embodiment. As shown in Figure 19, at 1900, an indicator that a cell is queued for transmission at an input queue is received via a switch fabric. In some embodiments, the switch fabric can be based on a Clos architecture and can have multiple levels. In some embodiments, a switch fabric can be associated with (eg, within) a switch core. In some embodiments, an indicator can be received when a new cell is received in an input queue, or when a cell is ready (or soon ready) to be sent via the switch fabric.

At 1910, groups of cells having a common destination are defined from cells queued at the input queue. The destination of each cell from the cell group is determined based on the look-up table. In some embodiments, the destination is determined policy-based and/or based on a packet classification algorithm. In some embodiments, the common destination may be a common destination port associated with the input portion of the switch fabric.

At 1920, request tags are associated with groups of cells. A request tag may include, for example, one or more cell quantity values, destination identifiers, queue identifiers, queue sequence values, and the like. Request tags may be associated with groups of cells before the groups of cells are sent to the input side of the switch fabric.

At 1930, the transfer request including the request tag is sent to the output scheduling module. In some embodiments, a transmission request includes a request to be sent at a specific time or via a specific transmission path. In some embodiments, the transfer request can be sent after the group of cells has been stored in memory associated with the switch fabric input stage. In some embodiments, groups of cells can be moved to memory to reduce the possibility of congestion at the input queue. In other words, the cell group can be moved to memory so that other cells queued after the cell group can be prepared for transmission (or transmission) from the input queue without waiting for the cell group to be transferred from the input queue send. In some embodiments, a transfer request may be a request to send to a specific output port (eg, a specific destination port).

At 1950, when a transmission via the switch fabric is not authorized at 1940 in response to the transmission request, a transmission rejection including a response tag is sent to the input scheduling module. In some embodiments, the transfer request may be denied because the switch fabric is congested, the destination port is unavailable, and the like. In some embodiments, transfer requests can be denied for a specified period of time. In some embodiments, a response tag may include one or more identifiers that can be used to associate a transmission rejection with a cell group.

If at 1940 transmission via the switch fabric is authorized, then at 1960 a transmission response including a response tag to the input scheduling module is sent. In some embodiments, the transfer response may be a transfer grant. In some embodiments, the transport response may be sent after the destination of the cell group is ready (or immediately ready) to receive the cell group.

At 1970, cell groups are extracted based on response tags. If the cell group has been moved to memory, the cell group can be retrieved from memory. If a cell group is queued at the input queue, the cell group can be extracted from the input queue. Cell groups may be extracted based on the queue identifier and/or queue sequence value included in the response tag. The queue identifier and/or queue sequence value can be derived from the queue label.

At 1980, groups of cells can be sent via the switch fabric. Groups of cells may be sent via the switch fabric according to instructions included in the transport response. In some embodiments, groups of cells may be sent at specific times and/or via specific transmission paths. In some embodiments, groups of cells may be sent via a switch fabric to a destination, such as an output port. In some embodiments, after being sent via the switch fabric, the cell group can be queued at an output queue associated with the cell group's destination (eg, destination port).

Figure 20 is a signaling flow diagram illustrating request sequence value processing associated with a transfer request, according to one embodiment. As shown in FIG. 20, the transmission request 52 is sent from the input scheduling module 2020 on the input side of the switch fabric to the output scheduling module 2030 on the output side of the switch fabric. Transmission request 56 is sent from input scheduling module 2020 to output scheduling module 2030 after transmission request 52 is sent. As shown in FIG. 20 , transmission requests 54 are sent from the input scheduling module 2020 but are not received by the output scheduling module 2030 . Transfer request 52, transfer request 54, and transfer request 56 are each associated with the same input queue IQ1, as indicated by their respective queue identifiers, and with the same destination port EP1, as indicated by their corresponding destination identifiers. instruct. Transfer request 52 , transfer request 54 , and transfer request 56 may collectively be referred to as transfer request 58 . As shown in Figure 20, the time increases in the downstream direction.

As shown in FIG. 20, each transfer request 58 may include a request sequence value (SV). A request sequence value may represent the sequence of a transfer request relative to other transfer requests. In this embodiment, the request sequence value may be from a range of request sequence values associated with the destination port EP1 and incremented in numerical order in full integers. In some embodiments, request sequence values can be, for example, strings, and can be incremented in a different order (eg, reverse numerical order). Transfer request 52 includes request sequence value 5200 , transfer request 54 includes request sequence value 5201 , and transfer request 56 includes request sequence value 5202 . In this example, request sequence value 5200 indicates that transfer request 52 was defined and sent prior to transfer request 54 , which has request sequence value 5201 .

The output scheduling module 2030 can determine that the transmission of the transmission request from the input scheduling module 2020 may have failed based on the request sequence value. In particular, output scheduling module 2030 can determine that the transmission request associated with request sequence value 5201 was not received prior to transmission request 56 , which is associated with request sequence value 5202 . In some embodiments, the output scheduling module 2030 can perform actions regarding missing transmission requests 54 when the time period between receipt of transmission requests 52 and transmission requests 56 (shown as time period 2040 ) exceeds a threshold time period. In some embodiments, output scheduling module 2030 can request input scheduling module 2020 to retransmit transmission request 54 . The output scheduling module 2030 can include a missing request sequence value so that the input scheduling module 2020 can recognize that the transmission request 54 was not received. In some embodiments, output scheduling module 2030 can deny a request to transmit a group of cells included in transmission request 56 . In some embodiments, output scheduling module 2030 can be configured to process and/or respond to transfer requests (eg, transfer request 58 ) based on queue sequence values in a manner substantially similar to the methods described with request sequence values.

Figure 21 is a signaling flow diagram illustrating response sequence values associated with transmission responses, according to one embodiment. As shown in FIG. 21 , the transport response 62 is sent from the output scheduling module 2130 on the output side of the switch fabric to the input scheduling module 2120 on the input side of the switch fabric. Transmission response 66 is sent from output scheduling module 2130 to input scheduling module 2120 after transmission response 62 is sent. As shown in FIG. 21 , transmission responses 64 are sent from the outgoing scheduling module 2130 , but are not received by the incoming scheduling module 2120 . Transmission response 62, transmission response 64, and transmission response 66 are associated with the same input queue IQ2 indicated by its respective queue identifier. Transmission response 62 , transmission response 64 , and transmission response 66 may collectively be referred to as transmission response 68 . As shown in Figure 21, the time increases in the downstream direction.

As shown in FIG. 21, each transmission response 68 may include a response sequence value (SV). A ResponseSequence value may indicate the sequence of a transport response relative to other transport responses. In this embodiment, the response sequence value may come from a range of response sequence values associated with the input queue IQ2, and increase in numerical order in the form of full integers. In some embodiments, the response sequence value may be, for example, a string, and can be incremented in a different order (eg, reverse numerical order). Transmit response 62 may include response sequence value 5300 , transmit response 64 include response sequence value 5301 , and outgoing response 66 include response sequence value 5302 . In this example, a response sequence value of 5300 indicates that a transport response 62 is defined and sent before a transport response 64 with a corresponding sequence value of 5301 .

The input scheduling module 2120 can determine based on the response sequence value that the transmission of the transmission response from the output scheduling module 2130 may have failed. In particular, input scheduling module 2120 can determine that the transmission response associated with response sequence value 5301 was not received before transmission response 66 , which is associated with response sequence value 5302 . In some embodiments, input scheduling module 2120 can perform actions regarding missing transmission responses 64 when the time period (shown as time period 2140 ) between receipt of transmission responses 62 and 66 exceeds a threshold time period. In some embodiments, the input scheduling module 2120 can request the output scheduling module 2130 to retransmit the transmission response 64 . The input scheduling module 2120 can include a missing response sequence value so that the output scheduling module 2130 can identify that the transmission response 64 was not received. In some embodiments, the input scheduling module 2120 can discard groups of cells when a transmission response associated with a transmission request is not received within a specified time period.

Figure 22 is a schematic block diagram illustrating multiple levels of flow-controllable queues, according to one embodiment. As shown in FIG. 22 , the sending side of the first-stage queue 2210 and the sending side of the second-stage queue 2220 are included in the source entity 2230 on the sending side of the physical link 2200 . The receiving side of the first-level queue 2210 and the receiving side of the second-level queue 2220 are included in the destination entity 2240 on the receiving side of the physical link 2200 . Source entity 2230 and/or destination entity 2240 may be any type of computing device (eg, part of a switch core, peripheral processing device) that may be configured to receive and/or transmit data via physical link 2200 . In some embodiments, source entity 2230 and/or destination entity 2240 may be associated with a data center.

As shown in FIG. 22 , the first-level queues 2210 include sending queues A1 to A4 (called first-level sending queues 2234) on the sending side of the physical link 2200 and receiving queues D1 to A4 on the receiving side of the physical link 2200. D4 (referred to as the first-level receive queue 2244). Second-level queues 2220 include send queues B1 and B2 (referred to as second-stage send queues 2232) on the send side of physical link 2200 and receive queues C1 and C2 (referred to as second-stage send queues 2232) on the receive side of physical link 2200. receive queue 2242).

Data flow via physical link 2200 can be controlled (eg, modified, suspended) based on flow control signaling associated with a flow control ring between source entity 2230 and destination entity 2240 . For example, data sent from the source entity 2230 on the sending side of the physical link 2200 can be received at the destination entity 2240 on the receiving side of the physical link 2200 . A flow control signal can be defined at the destination entity 2240 and/or can be sent from the destination entity 2240 to the source entity 2230 when the destination entity 2240 is not available to receive data from the source entity 2230 via the physical link 2200 . The flow control signal can be configured to trigger the source entity 2230 to modify the flow of data from the source entity 2230 to the destination entity 2240 .

For example, if receive queue D2 is not available to process data sent from send queue A1, destination entity 2240 can be configured to send a flow control signal associated with the flow control ring to source entity 2230; the flow control signal can be configured to trigger Suspension of data transmission from send queue A1 to receive queue D2 via a transmission path comprising at least a portion of second stage queue 2220 and physical link 2200 . In some embodiments, receive queue D2 may not be available, for example, when receive queue D2 is too full to accept received data. In some embodiments, receive queue D2 can change from an available state to an unavailable state (eg, a congested state) in response to data previously received from transmit queue A1 . In some embodiments, transmit queue A1 can be referred to as the target of the flow control signal. Send queue A1 can be identified within the flow control signal based on a queue identifier associated with send queue A1. In some embodiments, the flow control signal can be referred to as a feedback signal.

In this embodiment, the flow control ring is associated with the physical link 2200 (called the physical link control ring), the flow control ring is associated with the first-level queue 2210 (called the first-level control ring), and the flow control A ring is associated with the second-level queue 2220 (referred to as the second-level control ring). In particular, the physical link control loop is associated with a transmission path that includes the physical link 2200 and does not include the first-level queue 2210 and the second-level queue 2200 . Data flow via the physical link 2200 can be switched on and off based on flow control signaling associated with the physical link control loop.

The first-level control loop may be based on data transmission from at least one send queue 2234 within the second-level queue 2210 and flow control defined based on availability (e.g., an indicator of availability) of at least one receive queue 2244 within the first-level queue 2210 Signal. As such, a first level control loop can be said to be associated with a first level queue 2210 . A first level control loop can be associated with a transmission path that includes physical link 2200 , at least a portion of second level queue 2220 and at least a portion of first level queue 2210 . Flow control signaling associated with the first stage control loop can trigger control of data flow from the send queue 2234 associated with the first stage queue 2210 .

A second-level control loop can be associated with a transmission path that includes physical link 2200 and that includes at least a portion of second-level queues 2220 , but excludes first-level queues 2210 . The second level control loop can be based on data transmission from flow control signals defined from the at least one transmit queue 2232 within the second level queues 2220 and based on the availability (eg, an indicator of availability) of the at least one receive queue 2242 within the second level queues 2220. As such, the second level control loop can be said to be associated with the second level queue 2220 . Flow control signaling associated with the second stage control loop can trigger control of data flow from the send queue 2232 associated with the second stage queue 2220 .

In this embodiment, the flow control loop associated with the second level queue 2220 is a priority based flow control loop. In particular, each transmit queue from the second-level transmit queue 2232 is paired with a receive queue from the second-level receive queue 2242; and each queue pair is associated with a service level (also referred to as service level or quality of service) . In this embodiment, the second-level send queue B1 and the second-level send queue C1 define a queue pair and are associated with service class x. Second-level send queue B2 and second-level send queue C2 define a queue pair and are associated with service level Y. In some embodiments, different types of network traffic may be associated with different service levels (ie, different priorities). For example, storage traffic (eg, read and write traffic), inter-processor communication, media signaling, session layer signaling, etc. can be associated with at least one level of service. In some embodiments, the second level control loop may be based on, for example, the Institute of Electrical and Electronics Engineers (IEEE) 802.1qbb protocol, which defines a priority-based flow control policy.

Data flow via transmission path 74, as shown in FIG. 22, can be controlled using at least one control loop. The transmission path 74 includes a first-level sending queue A2, a second-level sending queue B1, a physical link 2200, a second-level receiving queue C1, and a first-level receiving queue D3. However, a queue in one stage of transmission path 74 via another stage of transmission path 74 can affect data flow through another stage of transmission path 74 based on a change in data flow of the flow control loop associated with that stage. Flow control at one level can affect data flow at another level because queues within source entity 2230 (e.g., send queue 2232, send queue 2234) and queues within destination entity 2240 (e.g., receive queue 2242, receive Queues 2244) are segmented. In other words, flow control based on a flow control loop can have an effect on data flow via factors associated with different flow control loops.

For example, the data flow from the first-stage transmit queue A1 to the first-stage receive queue D3 via the transmission path 74 can be based on one or more control loops—a first-stage control loop, a second-stage control loop, and/or a physical link control The ring is modified. The suspension of data flow to the first stage receive queue D3 may be triggered due to the first stage receive queue D3 changing from an available state to an unavailable state (eg, a congested state).

If data flow to first-level receive queue D3 is associated with service class x, then data flow via second-level send queue B1 and second-level receive queue C1 (which defines the queue pair associated with service level x) can Pausing based on flow control signaling associated with the second level control loop (which is a priority based control loop). But the suspension of data transmission via the queue pair associated with service class x can cause the suspension of data transmission from the send queue input to the second-stage send queue B1. In particular, the suspension of data transmission via the queue pair associated with service class x can result in a suspension of data transmission not only from the first-stage send queue A2, but also from the first-stage send queue A1. In other words, the data flow from the first-stage send queue A1 is affected indirectly or in parallel. In some embodiments, the data received at send queue A1 and the data received at send queue A2 can be associated with the same service level X, but the data received at send queue A1 and the data received at send queue A2 Data may come from, for example, different (eg, separate) network devices (not shown), such as peripheral processing devices, which may be associated with different service levels.

Data flow to the first-stage receive queue D3 can also be suspended based on flow control signaling related to the first-stage control loop, in particular by the suspension of data transmission from the first-stage send queue A2. By directly suspending the data transmission from the first-level sending queue A2, the data transmission from the first-level sending queue A1 may not be interrupted. In other words, the flow control of the first-stage send queue A2 can be directly controlled based on the flow control signal associated with the first-stage control loop, without the need for a flow control from other first-stage send queues such as the first-stage send queue A1 Data transfer paused.

Data flow to the first stage receive queue D3 can also be controlled by pausing data transmission via the physical link 220 based on flow control signaling associated with the physical link control loop. But the suspension of data transmission via physical link 2200 can cause all data transmission via physical link 2200 to be suspended.

The queue on the send side of the physical link 2200 can be referred to as a send queue 2236 and the queue on the receive side of the physical link can be referred to as a receive queue 2246 . In some embodiments, send queue 2236 can also be referred to as a source queue, and receive queue 2246 can also be referred to as a destination queue. Although not shown, in some embodiments, one or more transmit queues 2236 can be included in one or more interface cards associated with source entity 2230, and one or more receive queues 2246 can be included in In one or more interface cards associated with the destination entity 2240.

When the source entity 2230 transmits data via the physical link 2200 , the source entity 2230 can be referred to as a transmitter located on the sending side of the physical link 2200 . The destination entity 2240 can be configured to receive data and is referred to as a receiver on the receiving side of the physical link 2200 . Although not shown, in some embodiments, source entity 2230 (and associated elements (e.g., transmit queue 2236)) can be configured to work as a destination entity (e.g., receiver) and destination entity 2240 (and associated An element (eg, receive queue 2246)) can be configured to work as a source entity (eg, sender). Additionally, physical link 2200 can operate as a bidirectional link.

In some embodiments, physical link 2200 may be a physical link, such as an optical link (eg, fiber optic cable, plastic fiber optic cable), a cable link (eg, copper-based wire), a twisted pair link ( For example, category 5 cable) and so on. In some embodiments, physical link 2200 may be a wireless link. Data transmission via physical link 2200 can be defined based on protocols such as Ethernet protocols, wireless protocols, Ethernet protocols, Fiber Channel protocols, Fiber Channel over Ethernet protocols, protocols involving InfiniBand, and/or the like.

In some embodiments, the second-level control loop can be said to be nested within the first-level control loop because the second-level queue 2220 associated with the second-level control loop is located in the Inside the first level queue 2210. Similarly, a physical link control loop can be said to be nested within a second level control loop. In some embodiments, the second level control loop can be referred to as an inner control loop and the first level control loop can be referred to as an outer control loop.

Figure 23 is a schematic block diagram illustrating multiple stages of flow-controllable queues, according to one embodiment. As shown in FIG. 23 , the sending side of the first-level queue 2310 and the sending side of the second-level queue 2320 are included in a source entity 2330 located on the sending side of the physical link 2300 . The receive side of the first stage queue 2310 and the receive side of the second stage queue 2320 are included in a destination entity 2340 located on the receive side of the physical link 2300 . The queues on the transmit side of the physical link 2300 may be collectively referred to as transmit queues 2336 and the queues on the receive side of the physical link may be collectively referred to as receive queues 2346 . Although not shown, in some embodiments, source entity 2330 can be configured to operate as a destination entity, and destination entity 2340 can be configured to operate as a source entity (eg, a sender). Additionally, physical link 2300 can operate as a bidirectional link.

As shown in FIG. 23 , a source entity 2330 communicates with a destination entity 2340 via a physical link 2300 . The source entity 2330 has a queue QP1 configured to buffer data (if needed) before it is sent via the physical link 2300, and the destination entity 2340 has a queue QP2 configured to buffer the data before the destination entity 2340 is allocated via Data received by physical link 2300 (if required). In some embodiments, data flow via physical link 2300 can be processed without buffering queues QP1 and QP2.

Each of the transmission queues QA1 to QAN included in the first-level queue 2310 can be referred to as a first-level transmission queue and can be collectively referred to as a transmission queue 2334 (or queues 2334). The transmission queues QB1 to QBM included in the second-level queue 2320 can each be referred to as a second-level transmission queue and can be collectively referred to as a transmission queue 2332 (or queues 2332). Each of the receive queues QD1 to QDR included in the first-stage queue 2310 can be referred to as a first-stage receive queue and can be collectively referred to as a receive queue 2344 (or queues 2344). The reception queues QC1 to QCM included in the second-stage queue 2320 can each be referred to as a second-stage reception queue and can be collectively referred to as a reception queue 2342 (or queues 2342).

As shown in FIG. 23 , each queue from the second-level queues 2320 is located within a transmission path between the physical link 2300 and at least one queue from the first-level queues 2310 . For example, a portion of the transmission path can be defined by the first-stage receive queue QD4 , the second-stage receive queue QC1 , and the physical link 2300 . The second-stage receive queue QC1 is located within the transmission path between the first-stage receive queue QD4 and the physical link 2300 .

In this embodiment, a physical link control ring is associated with physical link 2300 , a first level control ring is associated with first level queue 2310 , and a second level control ring is associated with second level queue 2320 . In some embodiments, the second level control loop may be a priority based control loop. In some embodiments, the physical link control ring includes physical link 2300, queue QP1, and queue QP2.

Flow control signals can be defined and/or sent between a source control module 2370 at the source entity 2330 and a destination control module 2380 at the destination entity 2340 . In some embodiments, the source control module 2370 can be referred to as a source flow control module, and the destination control module 2380 can be referred to as a destination flow control module. For example, the destination control module 2380 can be configured to send a flow control signal to the source control module 2370 when one or more receive queues 2346 (eg, receive queue QD2 ) at the destination entity 2340 are not available to accept data. The flow control signal can be configured to trigger the source control module 2370 to pause the flow of data from the one or more receive queues 2330 to the one or more receive queues 2346, for example.

Source control module 2370 associates queue identifiers with data queued at send queues from send queues 2336 before the data is sent. A queue identifier can represent and/or be used to identify a transmit queue to which data is queued. For example, when a data packet is queued in the first-level transmit queue QA4, a queue identifier that uniquely identifies the first-level transmit queue QA4 can be added to the data packet or included in a field (e.g., header, tail, etc.) of the data packet. , payload). In some embodiments, the queue identifier may be related to data at the source control module 2370 or triggered by the source control module 2370 . In some embodiments, a queue identifier can be associated with data only before the data is sent, or after the data has been sent from one of the send queues 2336 .

A queue identifier can be associated with data sent from the sending side of the physical link 2300 to the receiving side of the physical link 2300 so that the source of the data (eg, the source queue) can be identified. Accordingly, a flow control signal can be defined to suspend transmission for one or more transmit queues 2336 based on the queue identifier. For example, the queue identifier associated with the first-stage transmit queue QAN can be included in data packets sent from the first-stage transmit queue QAN to the first-stage receive queue QD3. If, after receiving a data packet, the first-stage receiving queue QD3 cannot receive another data packet from the first-stage sending queue QAN, request the first-stage sending queue QAN to suspend the transmission of additional data packets to the first-stage receiving queue QD3 The flow control signal for can be defined based on the queue identifier associated with the first level transmit queue QAN. The queue identifier can be parsed from the data packet by the destination control module 2380 and used by the destination control module 2380 to define flow control signals.

In some embodiments, data transfers from several transmit queues 2336 (e.g., first-stage transmit queues 2334) to first-stage receive queues QDR can be performed in response to first-stage receive queues QDR changing from an available state to an unavailable state. pause. Each of several transmit queues 2336 can be identified within the flow control signal based on its corresponding queue identifier.

In some embodiments, one or more transmit queues 2336 and/or one or more receive queues 2346 may be virtual queues (eg, logically defined groups of queues). Accordingly, a queue identifier can be associated (eg, embodied) with a virtual queue. In some embodiments, a queue identifier may be associated with a queue from a set of queues defining a virtual queue. In some embodiments, each queue identifier from the set of queue identifiers associated with physical link 2300 may be unique. For example, each transmit queue 2336 associated with a physical link 2300 (eg, associated with a hop) can be associated with a unique queue identifier.

In some embodiments, source control module 2370 can be configured to associate queue identifiers with only a particular subset of send queues 2336 and/or with only a subset of data queued at one of send queues 2336 . For example, if data is sent from the first-stage send queue QA2 to the first-stage receive queue QD1 without a queue identifier, the flow control signal configured to request the suspension of data transmission from the first-stage send queue QA2 may not be defined , because the source data is unknown. Thus, the send queue from send queue 2336 can be exempted from flow control by not associating (eg, omitting) the queue identifier with the data when data is sent from the send queue.

In some embodiments, the unavailability of one or more receive queues 2346 at the destination entity 2340 is defined based on conditions being met. The condition can relate to the storage limit of the queue, the rate of access to the queue, the rate of data traffic input to the queue, and the like. For example, the flow control signal can be at the destination control module 2380 in response to a state of one or more receive queues 2346, such as a second stage receive queue QC2 changing from an available state to an unavailable state (e.g., a congested state) based on a threshold storage limit being exceeded. ) is defined. When in the unavailable state, the second-stage receive queue QC2 is unavailable for receiving data because, for example, the second-stage receive queue QC2 is considered too full (as indicated by the exceeding of a threshold storage limit). In some embodiments, when disabled, one or more receive queues 2346 can be in an unavailable state. In some embodiments, a flow control signal can be defined based on a request to suspend data transmissions to receive queues from receive queues 2346 when receive queues are not available to receive data. In some embodiments, the state of one or more receive queues 2346 can change from an available state to a congested state (via a destination control module) in response to a particular subset of receive queues 2346 (e.g., receive queues within a particular stage) being in a congested state. 2380).

In some embodiments, a flow control signal can be defined at the destination control module 2380 to indicate that one of the receive queues 2346 has changed from an unavailable state to an available state. For example, initially, the destination control module 2380 can be configured to define and send a first flow control signal to the source control module 2370 in response to the first-stage receive queue QD3 changing from an available state to an unavailable state. The first-stage receive queue QD3 can change from an available state to an unavailable state in response to data transmitted from the first-stage transmit queue QA2. Thus, the first flow control signal may be targeted to the first stage transmit queue QA2 (indicated based on the queue identifier). When the first stage receive queue QD3 changes back to the available state from the unavailable state, the destination control module 2380 can be configured to define and send to the source control module 2370 a second flow control signal indicating the change back from the unavailable state to the available state. In some embodiments, the source control module 2370 can be configured to trigger the transmission of data from the one or more send queues 2336 to the first stage receive queue QD3 in response to the second flow control signal.

In some embodiments, the flow control signal may have one or more parameter values that are used by the source control module 2370 to modify transmissions from one of the transmit queues 2336 (identified within the flow control signal by a queue identifier) . For example, the flow control signal may include a parameter value that triggers the source control module 2370 to suspend transmissions from one of the send queues 2336 for a specified period of time (eg, 10 milliseconds (ms)). In other words, the flow control signal may include a pause period parameter value. In some embodiments, the pause period may be indeterminate. In some embodiments, a flow control signal can define a request to send data from one or more transmit queues 2336 at a particular rate (eg, a particular number of frames per second, a particular number of bits per second).

In some embodiments, a flow control signal (eg, a pause period within the flow control signal) can be defined based on a flow control algorithm. A pause period can be defined based on the time period that elapses while a receive queue from receive queue 2346 (eg, first stage receive queue QD4 ) is in an unavailable state. In some embodiments, a pause period can be defined based on more than one first level receive queue 2344 being unavailable. For example, in some embodiments, the pause period increases when approximately a specified number of first-level receive queues 2344 are congested. In some embodiments, this type of determination can be determined at the purpose control module 2380 . The period of time that the receive queue is unavailable may be a planned (eg, projected) period of time calculated by the destination control module 2380 based on, for example, traffic rates (eg, historical traffic rates, previous traffic rates) from receive queue data.

In some embodiments, source control module 2370 can deny or alter requests to modify data streams from one or more transmit queues 2336 . For example, in some embodiments, the source control module 2370 can be configured to decrease or increase the pause period. In some embodiments, rather than suspending data transmission in response to a flow control signal, source control module 2370 may be configured to modify a transmission path associated with one of transmission queues 2336 . For example, if the first-level sending queue QA2 has received a request to suspend transmission based on a change in the status of the first-level receiving queue QD2, the source control module 2370 can be configured to trigger a transfer from the first-level sending queue QA2 to, for example, the first-level receiving queue. QD3 data transfer, instead of following the request to suspend the transfer.

As shown in FIG. 23 , the queues within the second-level queue 2320 fan into or fan out the physical link 2300 . For example, transmit queue 2332 (eg, QB1 to QBM) on the transmit side of physical link 2300 fans into queue QP1 on the transmit side of physical link 2300 . Thus, data queued at any send queue 2332 can be sent to queue QP1 of physical link 2300 . On the receive side of physical link 2300, data sent from physical link 2300 via queue QP2 can be broadcast to receive queues 2342 (ie, queues QC1 to QCM).

Likewise, as shown in FIG. 23 , the send queue 2334 in the first-level queue 2310 fans into the send queue 2332 in the second-level queue 2320 . For example, data queued anywhere in the first-stage transmission queues QA1, QA4, and QAN-2 can be transmitted to the second-stage transmission queue QB2. On the receive side of the physical link 2300, data transmitted from, for example, the second-stage receive queue QCM can be broadcast to the first-stage receive queues QDR-1 and QDR.

Since many flow control loops (eg, the first control loop) are associated with different fan-in, fan-out architectures, the flow control loops have different impacts on the data flow via the physical link 2300 . For example, when data transmission from the second-stage send queue QB1 is suspended based on the second-stage control loop, from the first-stage send queues QA1, QA2, QA3, and QAN-1 to one or more Data transfers to receive queue 2346 are also suspended. In this case, data transmission from one or more upstream queues (e.g., first-stage send queue QA1) can be is suspended. On the contrary, if the data transmission from the first-stage transmit queue QA1 along the transmission path including at least the downstream second-stage transmit queue QB1 is suspended based on the first-stage control loop, the traffic data rate from the second-stage transmit queue QB1 can be The reduction does not require all data transmissions from the second-level sending queue QB1 to be suspended; for example, the first-level sending queue QA1 can still send data through the second-level sending queue QB1.

In some embodiments, the fan-in and fan-out architecture may differ from that shown in FIG. 23 . For example, in some embodiments, some queues within first stage queues 2310 can be configured to fan into physical link 2300 detouring second stage queues 2320 .

Flow control signaling associated with send queue 2336 is handled by source control module 2370 and flow control signaling associated with receive queue 2346 is handled by destination control module 2380 . Although not shown, in some embodiments flow control signaling can be handled by one or more control modules (or control sub-modules) which may be separate and/or integrated into a single control module. For example, the flow control signaling associated with the first stage receive queue 2344 may be processed by a control module that is separate from the control module configured to process the flow control signaling associated with the second stage receive queue 2342 . Similarly, flow control signaling associated with the first stage transmit queue 2334 may be handled by a control module independent of the control module configured to process flow control signaling associated with the second stage transmit queue 2332 . In some embodiments, one or more portions of source control module 2370 and/or destination control module 2380 may be hardware-based modules (e.g., DSP, FPGA) and/or software-based modules (e.g., compute node modules, A set of processor-readable instructions that can be executed on a processor).

24 is a schematic block diagram illustrating a destination control module 2450 configured to define flow control signals 6428 associated with a plurality of receive queues, according to one embodiment. The queue levels include first-level queues 2410 and second-level queues 2420 . As shown in FIG. 24 , source control module 2460 is associated with the sending side of first-level queue 2410 and destination control module 2450 is associated with the receiving side of first-level queue 2410 . The queues on the transmit side of physical link 2400 can collectively be referred to as transmit queues 2470 . The queues on the receive side of physical link 2400 can collectively be referred to as receive queues 2480 .

Destination control module 2450 is configured to send flow control signal 6428 to source control module 2460 in response to one or more receive queues within first level queues 2410 being unavailable to receive data from a single source queue at first level queues 2410 . The source control module 2460 is configured to suspend data transmission from the source queue at the first stage queue 2410 to the plurality of receive queues at the first stage queue 2410 based on the flow control signal 6428 .

Flow control signal 6428 can be defined by destination control module 2450 based on information associated with each unavailable receive queue within primary queue 2410 . Destination control module 2450 can be configured to collect information associated with unavailable receive queues and to define flow control signals 6428 such that potentially conflicting flow control signals (not shown) are not sent to primary queues 2410 A single source queue for . In some embodiments, the flow control signal 6428 defined based on the collected information can be referred to as an aggregate flow control signal.

In particular, in this example, destination control module 2450 is configured to respond to two receive queues—receive queue 2442 and receive queue 2446—at the receive side of primary queue 2410 being unavailable from the transmit side of primary queue 2410 The send queue 2412 on receives data to define the flow control signal 6428. In this embodiment, receive queue 2442 and receive queue 2446 change from an available state to an unavailable state in response to data packets transmitted from transmit queue 2412 via transmit path 6422 and transmit path 6424, respectively. As shown in FIG. 24 , the transmission path 6422 includes a sending queue 2412 , a sending queue 2422 in the second-level queue 2420 , a physical link 2400 , and a receiving queue 2432 and a receiving queue 2442 in the second-level queue 2420 . Transmission path 6424 includes transmit queue 2412 , transmit queue 2422 , physical link 2400 , receive queue 2432 and receive queue 2446 .

In some embodiments, a flow control algorithm can be used to define flow control signal 6428 based on information related to receive queue 2442 unavailability and/or information related to receive queue 2446 unavailability. For example, if destination control module 2450 determines that receive queue 2442 and receive queue 2446 are unavailable for different time periods, destination control module 2450 may be configured to define flow control signal 6428 based on the different time periods. For example, destination control module 2450 can request via flow control signal 6428 that data transmission from send queue 2412 be suspended for a time period based on a different time period (e.g., a time period equal to the average of different time periods, a time period equal to different time periods The time period of the larger value) calculation. In some embodiments, the flow control signal 6428 can be defined based on separate pause requests from the receive side of the primary queue 2410 (eg, a pause request associated with receive queue 2442 and a pause request associated with receive queue 2446 ).

In some embodiments, the flow control signal 6428 can be defined based on a maximum or minimum allowable time period. In some embodiments, the flow control signal 6428 can be calculated based on the aggregate data flow rate from, for example, the transmit queue 2412 . For example, the pause period can be based on aggregate data traffic rate measurements from the transmit queue 2412 . In some embodiments, for example, the pause period can be increased if the rate of data traffic from the transmit queue 2412 is above a threshold, and the pause period can be increased if the rate of data traffic from the transmit queue 2412 is below a threshold reduce.

In some embodiments, the flow control algorithm can be configured to wait a certain period of time before defining and/or sending a flow control signal 6428 . A wait period can be defined such that multiple pause requests involving the transmit queue 2412 and which can be received at different times within the wait period can be used to define the flow control signal 6428 . In some embodiments, the wait period is triggered in response to at least one pause request involving the send queue 2412 being received.

In some embodiments, the flow control signal 6428 can be defined by a flow control algorithm based on a priority value associated with each receive queue within the first level queue 2410 . For example, if receive queue 2442 has a higher priority value than the priority value associated with receive queue 2446, destination control module 2450 can be configured to define traffic based on information associated with receive queue 2442 rather than receive queue 2446 Control signal 6428. For example, flow control signal 6428 can be defined based on a pause period associated with receive queue 2442 rather than a pause period associated with receive queue 2446 because receive queue 2442 has a higher priority value than receive queue 2446 priority value.

In some embodiments, the flow control signal 6428 can be defined by a flow control algorithm based on attributes associated with each receive queue within the first level queue 2410 . For example, flow control signal 6428 can be defined based on receive queue 2442 and/or receive queue 2446 being a particular type of queue (eg, last-in-first-out (LIFO) queue, first-in-first-out (FIFO) queue). In some embodiments, flow control signals 6428 can be defined based on receive queues 2442 and/or receive queues 2446 configured to receive particular types of data (eg, control data/signal queues, media data/signal queues).

Although not shown, one or more control modules associated with a queue level (eg, first level queue 2410) can be configured to send information to a different control module, where the information is used to define flow control signals. Different control modules are associated with different queue levels. For example, a pause request associated with receive queue 2442 and a pause request associated with receive queue 2446 can be defined at destination control module 2450 . The pause request can be sent to a destination control module (not shown) associated with the receiving side of the secondary queue 2420 . A flow control signal (not shown) can be defined at the destination control module associated with the receiving side of the second stage queue 2420 based on the pause request and based on the flow control algorithm.

The flow control signal 6428 can be defined based on a flow control loop associated with the first stage queue 2410 (eg, a first stage control loop). One or more flow control signals (not shown) can also be defined based on the flow control ring associated with second stage queue 2420 and/or the flow control ring associated with physical link 2400 .

Data transfers associated with send queues (except send queue 2412 ) within first level queues 2410 are not substantially restricted by flow control signal 6428 because data flow to receive queues 2442 and 2446 is controlled based on the first level flow control loop. For example, even if data transmission from send queue 2412 is suspended, send queue 2414 can continue to send data via send queue 2422 . For example, send queue 2414 can be configured to send data to receive queue 2448 via transmission path 6426 that includes send queue 2422 even if data transmission from send queue 2412 via send queue 2422 has been suspended. In some embodiments, send queue 2422 can be configured to continue sending data from, for example, send queue 2416 to receive queue 2442 even if data transfer from queue 2412 via transfer path 6422 has been suspended based on flow control signal 6428 .

Conversely, if data transmission to receive queues 2442 and 2446 is suspended by controlling the flow of data via transmit queue 2422 based on a flow control signal (not shown) associated with a second-level control loop, then (except from transmit queue 2412 In addition to data transfers) data transfers from send queue 2414 and send queue 2416 via send queue 2422 will also be restricted. Data transmission from send queue 2422 will be suspended because it is associated with a particular service level, and data causing congestion, for example, at receive queues 2442 and 2446 may be associated with a particular service level.

One or more parameter values defined within flow control signal 6428 can be stored in memory 2452 of destination control module 2450 . In some embodiments, parameter values can be stored at memory 2452 of destination control module 2450 after one or more parameter values are defined and/or when flow control signal 6428 is sent to source control module 2460 . Parameter values defined within the flow control signal 6428 can be used to track the status of the transmit queue 2412, for example. For example, an entry within memory 2452 can indicate that transmit queue 2412 is in a suspended state (eg, a non-transmit state). Entries can be defined based on the Pause Period parameter value defined within Flow Control Signal 6428. When the pause period has expired, this entry can be updated to indicate that the state of the send queue 2412 has changed to, for example, an active state (eg, a send state). Although not shown, in some embodiments, one or more parameter values can be stored in memory external to the destination control module 2450 (eg, remote memory).

In some embodiments, one or more parameter values stored in memory 2452 of destination control module 2450 (e.g., status information defined based on one or more parameter values) can be used by destination control module 2450 to determine additional traffic flow Whether a control signal (not shown) should be defined. In some embodiments, one or more parameter values can define one or more additional flow control signals by the purpose control module 2450 .

For example, if the receive queue 2442 changes from an available state to an unavailable state (e.g., a congested state) in response to a first data packet received from the transmit queue 2412, a request to suspend data transmission from the transmit queue 2412 can be transmitted via the flow control signal 6428 is sent. The flow control signal 6428 can indicate that the send queue 2412 is the target of the request based on the queue indicator and can specify a pause period. When flow control signal 6428 is sent to source control module 2460 , the pause period and queue identifier associated with send queue 2412 can be stored in memory 2452 of destination control module 2450 . After the flow control signal 6428 is sent, the receive queue 2444 can change from an available state to a congested state in response to receiving a second data packet from the transmit queue 2412 (transmission path not shown in FIG. 24 ). Before data transmission from the send queue 2412 is suspended, a second data packet can be sent from the send queue 2412 based on the flow control signal 6428 . Destination control module 2450 has access to information stored in memory 2452 and can, in response to a change of state associated with receive queue 2444, determine that additional flow control signals destined for transmit queue 2412 should not be defined and sent to source control module 2460 , because flow control signal 6428 has been sent.

In some embodiments, source control module 2460 can be configured to suspend transmissions from send queue 2412 based on the most recent flow control signal parameter value. For example, a later flow control signal (not shown) destined for transmit queue 2412 can be received at source control module 2460 after flow control signal 6428 destined for transmit queue 2412 has been sent to source control module 2460 . Source control module 2460 can be configured to enforce one or more parameter values associated with subsequent flow control signals other than the parameter values associated with flow control signal 6428 . In some embodiments, the later flow control signal can trigger the send queue 2412 to remain in the paused state for a longer or shorter period of time than indicated in the flow control signal 6428 .

In some embodiments, when the priority value associated with the one or more parameter values is higher (or lower) than the priority value associated with the one or more parameter values associated with the flow control signal 6428, The source control module 2460 optionally enforces one or more parameter values associated with the later flow control signal. In some embodiments, each priority value can be defined at the destination control module 2450 , and each priority value can be defined based on priority values associated with one or more receive queues 2480 .

In some embodiments, both the flow control signal 6428 and the later flow control signal (both targeted to the transmit queue 2412 ) are defined in response to the unavailability of the same receive queue from the receive queue 2480 . For example, the later flow control signal can include updated parameter values defined by the destination control module 2450 based on the receive queue 2442 remaining in the unavailable state for a longer period of time than previously calculated. In some embodiments, a flow control signal 6428 targeted to the send queue 2412 can be defined in response to one of the receive queues 2480 changing state (e.g., from an available state to an unavailable state), and a flow control signal 6428 targeted to the send queue 2412 A later flow control signal can be defined in response to another change of state in receive queue 2480 (eg, changing from an available state to an unavailable state).

In some embodiments, multiple flow control signals can be defined at the destination control module 2450 to suspend transmissions from the multiple send queues of the first level queue 2410 . In some embodiments, multiple send queues may send data to a single receive queue, such as receive queue 2444 . In some embodiments, the history of flow control signals to the multiple transmit queues from the first stage queue 2410 can be stored in the memory 2452 of the destination control module 2450 . In some embodiments, later flow control signals associated with individual receive queues can be calculated based on the history of flow control signals.

In some embodiments, pause periods associated with multiple transmit queues can be grouped and included in flow control packets. For example, a pause period associated with transmit queue 2412 and a pause period associated with transmit queue 2414 can be included in a flow control packet (also referred to as a flow control packet). More details related to flow control packets will be described in connection with FIG. 25 .

Figure 25 is a schematic diagram illustrating flow control packets, according to one embodiment. The flow control packet includes a header 2510, a trailer 2520, and a payload including pause period parameter values (shown in column 2512) for several transmit queues represented by queue identifiers (IDs) (shown in column 2514) 2530. As shown in FIG. 25, the transmit queues represented by queue IDs 1 to V (ie, queue ID1 to queue IDV) are each associated with pause period parameter values 1 to V (ie, pause time period 1 to pause time period V). Pause period parameter value 2514 indicates the time period for which the send queue represented by queue 2512 should be suspended (eg, inhibited) from sending data.

In some embodiments, flow control packets can be defined at, for example, a destination control module such as destination control module 2450 shown in FIG. 24 . In some embodiments, the destination control module can be configured to define flow control packets at regular intervals. For example, the destination control module can be configured to define a flow control packet every 10 ms. In some embodiments, the destination control module can be configured to define flow control packets at random times when pause period parameter values have been calculated, and/or when a specified number of pause period parameter values have been calculated. In some embodiments, the destination control module can determine that at least a portion of the flow control packets should not be defined and/or sent based on, for example, one or more parameter values and/or state information accessed by the destination control module.

Although not shown, in some embodiments, multiple queue IDs can be associated with individual pause time period parameter values. In some embodiments, at least one queue ID can be associated with a parameter value other than a pause period parameter value. For example, a queue ID can be associated with a traffic rate parameter value. The traffic rate parameter value can indicate the traffic rate (eg, the maximum traffic rate) at which the sending queue (represented by the queue ID) should send data. In some embodiments, a flow control packet can have one or more means configured to indicate whether a particular receive queue is available to receive data.

Flow control packets can be sent from a destination control module to a source control module (eg, source control module 2460 shown in FIG. 24 ) via a flow control signal (eg, flow control signal 6428 shown in FIG. 24 ). In some embodiments, flow control packets can be defined based on a layer 2 (eg, layer 2 of the OSI model) protocol. In other words, flow control packets can be defined at layer 2 of the network system and used therein. In some embodiments, flow control packets can be sent between devices associated with layer 2 (eg, MAC devices).

Referring back to FIG. 25 , one or more parameter values associated with flow control signal 6428 (eg, state information defined based on the parameter values) can be stored in memory 2562 of source control module 2560 . In some embodiments, one or more parameter values can be stored in memory 2562 of source control module 2560 when flow control signal 6428 is received at source control module 2560 . The parameter values defined in flow control signal 6428 can be used to track the status of one or more receive queues 2580 (eg, receive 2542). For example, an entry in memory 2562 can indicate that receive queue 2542 is unavailable to receive data. This entry can be defined based on the pause time period parameter value defined in the flow control signal 6428 and is associated with an identifier of the receive queue 2542 (eg, a queue identifier). When the pause period expires, this entry can be updated to indicate that the status of the receive queue 2542 has changed to, for example, active. Although not shown, in some embodiments, one or more parameter values can be stored in memory external to source control module 2560 (eg, remote memory).

In some embodiments, one or more parameter values (and/or status information) stored at memory 2562 of source control module 2560 can be used by source control module 2560 to determine whether data should be sent to one or more receiving Queue 2580. For example, source control module 2560 can be configured to send data from send queue 2516 to receive queue 2544 instead of receive queue 2542 based on state information related to receive queue 2544 and receive queue 2542 .

In some embodiments, source control module 2560 can analyze data transfer patterns to determine whether data should be sent from one or more source queues 2570 to one or more receive queues 2580 . For example, source control module 2560 can determine based on parameter values stored at memory 2562 of source control module 2560 that send queue 2514 is sending a relatively high amount of data to receive queue 2546 . Based on this determination, source control module 2560 can trigger queue 2516 to send data to receive queue 2548 instead of receive queue 2546 because receive queue 2546 receives a high volume of data from send queue 2514 . By analyzing transmission patterns associated with transmit queues 2570, the onset of congestion at one or more receive queues 2580 can be substantially avoided.

In some embodiments, source control module 2560 can analyze parameter values (and/or status information) stored at memory 2562 of source control module 2560 to determine whether data should be sent to one or more receive queues 2580 . By analyzing stored parameter values (and/or state information), the onset of congestion at one or more transmit queues 2580 can be substantially avoided. For example, source control module 2560 can trigger data to be sent to receive queue 2540 instead of receive queue 2542 based on the historical availability of receive queue 2540 compared to (eg, better, worse) than the historical availability of receive queue 2542 . In some embodiments, for example, source control module 2560 can send data to receive queue 2542 instead of receive queue 2544 based on receive queue 2542 historical performance regarding data burst patterns compared to receive queue 2544 historical performance. In some embodiments, analysis of parameter values involving one or more receive queues 2580 may be based on particular time windows, particular types of network processing (eg, interprocessor communications), particular service levels, and the like.

In some embodiments, destination control module 2550 can send status information about receive queues 2580 (e.g., current status information), which can be used by source control module 2560 to determine whether data should be sent from one or more source queues 2570 . For example, source control module 2560 can trigger queue 2514 to send data to queue 2544 instead of queue 2546 because queue 2546 has more capacity available than queue 2544 as indicated by destination control module 2550 . In some embodiments, any combination of current state information, transmission pattern analysis, and historical data analysis can be used to substantially prevent or reduce the likelihood of congestion onset for one or more receive queues 2580 .

In some embodiments, the flow control signal 6428 can be sent from the destination control module 2550 to the source control module 2560 via an out-of-band transmission path. For example, flow control signal 6428 can be sent via a dedicated link involving flow control signaling communications. In some embodiments, flow control signals 6428 can be sent via queues associated with second-level queues 2520 , queues associated with first-level queues 2510 , and/or physical link 2500 .

Some embodiments described herein relate to a computer storage product having a computer-readable medium (also referred to as a processor-readable medium) having thereon instructions or computer code for performing various computer-executable operations. The media and computer code (also referred to as code) may be those designed and constructed for a specific purpose. Examples of computer-readable media include, but are not limited to: magnetic storage media such as hard disks, floppy disks, and magnetic tape; Optical storage media for devices; magneto-optical storage media such as optical discs; carrier signal processing modules; and hardware devices specially configured to store and execute program codes, such as ASICs, programmable logic devices (PLDs), and read-only memories ( ROM) and RAM devices.

Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions, such as those produced by assemblers to produce World Wide Web services, and files containing high-level instructions for execution by a computer using a translator . For example, embodiments can be implemented using Java, C++, or other programming languages (eg, object-oriented programming languages) and development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.

While various embodiments have been described above, it should be understood that this has been presented by way of example only, rather than limitation, and various changes in form and detail may be made. Any portion of the devices and/or methods described herein may be combined in any manner, except mutually exclusive combinations. The embodiments described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different embodiments described.

Claims (47)

1. a kind of communication equipment, including：

Exchcange core, the exchcange core defines single logic entity and with the multistage being physically distributed across multiple frames Switching fabric, the multilevel interchange frame has multiple input ports and multiple output ports, and the exchcange core is configured as Multiple peripheral processors are couple to via the multiple input port and the multiple output port,

The exchcange core is configured as in the first peripheral processor for arranging to have the first frame and is arranged in the second frame The second peripheral processor between provide clog-free connectivity with line rate, the exchcange core be configured as receiving with it is described The first associated packet of first peripheral processor, the exchcange core is configured as based on associated with the described first packet Cell, sequentially to second peripheral processor send second packet and to the 3rd peripheral processor send the 3rd Packet, the multilevel interchange frame is configured as from the input port in the multiple input port into the output port Output port sends the cell.

2. communication equipment as claimed in claim 1, has virtually wherein the multiple peripheral processor includes at least one Change the peripheral processor and at least one peripheral processor without virtual resources of resource.

3. communication equipment as claimed in claim 1, wherein the number of the multiple input port and the multiple output port More than 1000, each output port of each input port and the multiple output port in the multiple input port It is both configured to operate with the speed for being not less than 10Gb/s.

4. communication equipment as claimed in claim 1, wherein：

First peripheral processor and second peripheral processor be memory node device, calculate node device, One in service node device or router.

5. communication equipment as claimed in claim 1, wherein the exchcange core is configured as filling in second peripheral processes Put between the 3rd peripheral processor with the clog-free connectivity of line rate offer.

6. communication equipment as claimed in claim 5, wherein：

First peripheral processor and the 3rd peripheral processor be memory node device, calculate node device, One in service node device or router；And

Second peripheral processor is at least one in firewall device, intersecting detection means or load balance device.

7. a kind of communication equipment, including：

Exchcange core, the exchcange core has the multilevel interchange frame being physically distributed across multiple frames, the multistage friendship Changing structure has multiple input ports and multiple output ports, and the exchcange core is configured as via the multiple input port Multiple peripheral processors are couple to the multiple output port,

The exchcange core be configured as using line rate as the multiple peripheral processor in each peripheral processor The connectedness of each remaining processing unit in the multiple peripheral processor is provided, so that the multiple output end Each output port in mouthful can be by each peripheral processor in the multiple peripheral processor via described An input port in multiple input ports is coequally accessed, the number of the multiple input port and the multiple output port Mesh is more than each output end of each input port and the multiple output port in 1000, the multiple input port Mouth is both configured to operate with the speed for being not less than 10Gb/s.

8. communication equipment as claimed in claim 7, wherein the multiple peripheral processor includes at least one via ether The peripheral processor that net connection is couple to the exchcange core is couple to the friendship with least one via non-Ethernet connection Change the peripheral processor of core.

9. communication equipment as claimed in claim 7, wherein the multiple peripheral processor, which includes at least one, uses the 3rd layer The peripheral processor of route and at least one the 4th layer of peripheral processor to the 7th layer of device.

10. a kind of communication equipment, including：

Exchcange core, the exchcange core defines single logic entity and with multilevel interchange frame, and multistage exchange is tied Structure has multiple levels being physically distributed across multiple frames, and the multiple level has multiple input ports and multiple outputs jointly Port, the exchcange core is configured as being couple to multiple peripheries via the multiple input port and the multiple output port Processing unit,

The transmission that the exchcange core is configured as the multiple cells associated with packet can be essentially ensures that without logical When crossing the loss of the multilevel interchange frame, it is allowed to the input port that the multiple cell enters in the multiple input port.

11. communication equipment as claimed in claim 10, wherein the multiple peripheral processor includes being configured as and optical fiber Channel agreement communication the first peripheral processor and be configured as with fiber channel covering Ethernet protocol communicate second Peripheral processor.

12. communication equipment as claimed in claim 10, wherein being configured to determine that property of the multilevel interchange frame network.

13. communication equipment as claimed in claim 10, wherein being configured to determine that property of the multilevel interchange frame network, so that When the multiple cell can be sent to an output port in the multiple output port in the scheduled time, the multistage Switching fabric allows the packet to enter input port.

14. communication equipment as claimed in claim 10, wherein：

The exchcange core is configured as the first output port and into the multiple output port from the input port Two output ports send multiple cells associated with the packet, without in multiple levels of the multilevel interchange frame At least one-level at perform packet loss processing.

15. communication equipment as claimed in claim 10, wherein：

The exchcange core is couple to the multistage including multiple via the multiple input port and the multiple output port The edge device of switching fabric, the multiple edge device is couple to the multiple peripheral processor, and the multiple edge is set Each edge device in standby is configured as receiving described be grouped and based on the multiple cell of the packet definition.

16. communication equipment as claimed in claim 10, wherein：

The exchcange core is configured as multiple levels via the multilevel interchange frame from the input port to the multiple An output port in output port sends multiple cells associated with the packet, without in the multiple level At least one-level at perform packet loss processing.

17. a kind of communication equipment, including：

Exchcange core, the exchcange core defines single logic entity and with multilevel interchange frame, and multistage exchange is tied Structure has multiple levels being physically distributed across multiple frames, and the multilevel interchange frame has multiple input ports and multiple defeated Exit port, the exchcange core is configured as being couple to outside multiple via the multiple input port and the multiple output port Enclose processing unit,

The exchcange core is configured as receiving packet, the exchcange core quilt from the input port in the multiple input port It is configured to send multiple and institute via output port of the multiple level from the input port into the multiple output port The associated cell of packet is stated, is damaged without performing packet at least one-level in multiple levels of the multilevel interchange frame Consumption processing.

18. communication equipment as claimed in claim 17, wherein being configured to determine that property of the multilevel interchange frame network, so that Only when the transmission for the multiple cells associated with packet that can be essentially ensures that in the switching fabric is lossless, Just allow the packet of the input port in the multiple input port.

19. communication equipment as claimed in claim 17, wherein：

The output port is the first output port,

The exchcange core is configured as first output port into the multiple output port from the input port Sent and the multiple cell associated with the packet with the second output port.

20. communication equipment as claimed in claim 17, wherein：

The exchcange core is couple to the multistage including multiple via the multiple input port and the multiple output port The edge device of switching fabric, the multiple edge device is couple to the multiple peripheral processor, and the multiple edge is set Each edge device in standby is configured as receiving described be grouped and based on the multiple cell of the packet definition.

21. a kind of communication equipment, including：

Exchcange core, the exchcange core defines single logic entity and the multistage exchange with being configured to determine that property network Structure, the multilevel interchange frame has a multiple input ports and multiple output ports, the exchcange core be configured as via The multiple input port and the multiple output port are couple to multiple peripheral processors,

The exchcange core is configured as receiving packet, the exchcange core quilt from the input port in the multiple input port It is configured to the output port from the input port into the multiple output port and sends multiple associated with the packet Cell.

22. communication equipment as claimed in claim 21, wherein the multilevel interchange frame is physically distributed across multiple frames.

23. communication equipment as claimed in claim 21, wherein being configured to determine that property of the multilevel interchange frame network, so that Only when the transmission for the multiple cells associated with packet that can be essentially ensures that in the multilevel interchange frame is lossless It is time-consuming, just allow the packet of the input port in the multiple input port.

24. communication equipment as claimed in claim 21, wherein being configured to determine that property of the multilevel interchange frame network, so that An output in the multiple output port can be sent in the scheduled time when the multiple cell associated with packet During port, it is allowed to the packet of the input port in the multiple input port.

25. communication equipment as claimed in claim 21, wherein：

The exchcange core is couple to the multistage including multiple via the multiple input port and the multiple output port The edge device of switching fabric, the multiple edge device is couple to the multiple peripheral processor, and the multiple edge is set Each edge device in standby is configured as receiving described be grouped and based on the multiple cell of the packet definition.

26. communication equipment as claimed in claim 21, wherein：

The exchcange core is configured as multiple levels via the multilevel interchange frame from the input port to the output Port sends multiple cells associated with the packet, without performing packet at least one-level in the multiple level Loss is handled.

27. a kind of communication equipment, including：

Exchcange core, the exchcange core has the multilevel interchange frame being physically distributed between multiple frames, described many Level switching fabric has multiple input buffers and multiple output ports, and the exchcange core is configured to couple to multiple edges Equipment；With

The control for not needing software during operation and realizing and needed during configuration and monitoring software to realize with hardware Device, the controller is couple to the multiple input buffer and the multiple output port, and the controller is configured to work as When the congestion at an output port in multiple output ports is foreseen and it occurs for the congestion in the exchcange core Before, an input buffer transmitted traffic control signal into the multiple input buffer.

28. communication equipment as claimed in claim 27, wherein the controller is configured as independently of for the exchange core Flow is controlled in the structure of the multilevel interchange frame of the heart, the input buffer and the output port is performed end-to-end Flow is controlled.

29. communication equipment as claimed in claim 27, wherein the controller is configured as independently of for the multiple side The flow control of edge equipment, End-to-end flow control is performed to the input buffer and the output port.

30. communication equipment as claimed in claim 27, further comprises：

Multiple peripheral processors for being configured to couple to the multiple edge device,

The controller is configured as independently of the flow control for the multiple edge device, to the input buffer and The output port performs End-to-end flow control.

31. communication equipment as claimed in claim 27, wherein the controller is configured as performing End-to-end flow control, from And cell is buffered at the input buffer for a period of time being sent to before the output port, the time and institute Stating End-to-end flow control is associated.

32. communication equipment as claimed in claim 27, wherein the controller is configured as independently of in the multistage exchange At one level of structure cache cell section and independently of at an edge device in the multiple edge device cache Packet, at the input buffer cache cell perform End-to-end flow control.

33. communication equipment as claimed in claim 27, wherein the controller is configured as independently of associated with Ethernet Flow control mechanism, at the input buffer cache cell perform End-to-end flow control.

34. a kind of communication equipment, including：

Exchcange core, the exchcange core has the multilevel interchange frame being physically distributed between multiple frames, described many Level switching fabric is configured as receiving multiple cells associated with packet and is configured as being based on the multiple cell switching Multiple cell sections；

An edge device in multiple edge devices for being couple to the exchcange core, the edge device is configured as receiving The packet, the edge device is configured to send the multiple cell to the multilevel interchange frame；With

The controller of the multilevel interchange frame is couple to, the controller is configured as setting independently of for the multiple edge Standby flow is controlled and controlled for flow in the structure of the multilevel interchange frame, to the multiple cell traffic control System.

35. communication equipment as claimed in claim 34, wherein：

The controller is not needed software and is realized with hardware and need software real during configuration and monitoring during operation It is existing.

36. communication equipment as claimed in claim 34, wherein：

The multilevel interchange frame has multiple input buffers and multiple output ports,

When the congestion that the controller is configured as at an output port in the multiple output port is foreseen with And before the congestion in the exchcange core occurs, an input buffer into the multiple input buffer sends stream Measure control signal.

37. communication equipment as claimed in claim 34, wherein：

The multilevel interchange frame has multiple input buffers and multiple output ports,

The controller is configured as independently of the flow control mechanism associated with Ethernet, to being buffered in the multiple input The cell cached at an input buffer in device performs End-to-end flow control.

38. a kind of communication equipment, including：

Exchcange core, the exchcange core has multilevel interchange frame；

More than first peripheral processor, the multilevel interchange frame, described are couple to by multiple connections with agreement Each peripheral processor in more than one peripheral processor is the memory node with virtual resources, more than described first The virtual storage resource that the virtual resources common definition of individual peripheral processor is interconnected by the exchcange core；With

More than second peripheral processor, the multilevel interchange frame, described are couple to by multiple connections with agreement Each peripheral processor in more than two peripheral processor is the memory node with virtual resources, more than described second The virtual computing resource that the virtual resources common definition of individual peripheral processor is interconnected by the exchcange core.

39. communication equipment as claimed in claim 38, wherein：

Each peripheral processor in more than first peripheral processor has virtual resources, more than described first Each peripheral processor in peripheral processor is configured such that its virtual resources can be by from described first The virtual resource of remaining peripheral processor in multiple peripheral processors is substituted；And

Each peripheral processor in more than second peripheral processor has virtual resources, more than described second Each peripheral processor in peripheral processor is configured such that its virtual resources can be by from described second The virtual resource of remaining peripheral processor in multiple peripheral processors is substituted.

40. communication equipment as claimed in claim 38, wherein：

More than first peripheral processor is associated and associated with security protocol with based on packet communication protocol；And

More than second peripheral processor is associated and associated with security protocol with based on packet communication protocol.

41. a kind of communication equipment, including：

Exchcange core, the exchcange core has multilevel interchange frame, and the exchcange core is configured as being logically divided into One virtual switch core and the second virtual switch core；

Multiple peripheral processors for being couple to the multilevel interchange frame, the multiple peripheral processor has operationally It is couple to the first peripheral processor subset of the first virtual switch core and to be operably coupled to described second virtual Second peripheral processor subset of exchcange core.

42. communication equipment as claimed in claim 41, wherein：

The exchcange core be configured such that the first virtual switch core and the second virtual switch core independently of Manage to being managed property each other.

43. communication equipment as claimed in claim 41, wherein：

The exchcange core is configured such that the first virtual switch core has independently of the second virtual switch core The bandwidth of the bandwidth of the heart.

44. communication equipment as claimed in claim 41, wherein：

The exchcange core is configured such that the first virtual switch core has and the second virtual switch core Bandwidth and the independent bandwidth of managerial management and managerial management.

45. communication equipment as claimed in claim 41, wherein：

The exchcange core is configured such that the first virtual switch core is operated using l2 protocol, and described second Virtual switch core is operated using l2 protocol and layer-3 protocol.

46. communication equipment as claimed in claim 41, wherein：

The first peripheral processor subset has virtual resource, and the second peripheral processor subset has virtual money Source.

47. communication equipment as claimed in claim 41, wherein：

The first peripheral processor subset is included in being calculate node, memory node, service node device and router The peripheral processor of one, and including being remaining in calculate node, memory node, service node device and router The peripheral processor of one；And

The second peripheral processor subset is included in being calculate node, memory node, service node device and router The peripheral processor of one, and including being remaining in calculate node, memory node, service node device and router The peripheral processor of one.