Data centre spine-leaf switching: 100G port planning
A spine-leaf fabric with 100G uplinks pairs leaf switches carrying 10G/25G server downlinks with spines built on QSFP28 ports: Juniper QFX5110-48S leaves and a QFX10002-60C spine, Huawei CloudEngine CE6820 or CE6855 leaves, or an Arista 7280R3 spine, each run matched to SR4, CWDM4 or LR4 optics. The port arithmetic follows from the mesh rule, and the three worked designs below show what it produces at three sizes.
Which switches support 100G uplinks in a spine-leaf design?
Switches qualify for a 100G spine-leaf role when their uplink ports speak 40G/100G over QSFP28 (or QSFP-DD) and their downlink side terminates dense 10G or 25G server ports. Every model below is stocked in the YT CIG equipment catalogue, and every port figure comes from the vendor datasheet or the catalogue record.
Leaf candidates. The Juniper QFX5110-48S-AFO provides 48 × 1/10GbE SFP+ downlinks plus 4 × 40/100GbE QSFP28 uplinks, with 1760 Gbps bidirectional throughput recorded in its catalogue entry (Juniper QFX5110 datasheet). Huawei’s CE6820-48S6CQ-B carries 48 × 10GE SFP+ and 6 × 100GE QSFP28 ports; Huawei’s documentation notes each QSFP28 can also run at 40 Gbit/s but cannot be split into 4 × 25GE (Huawei CE6820 support documentation). Where a design keeps 40G leaf uplinks, the CE6855-48S6Q-HI offers 48 × 10GE SFP+ and 6 × 40GE QSFP+, with M-LAG, VXLAN and BGP-EVPN features listed in the CloudEngine 6800 datasheet.
Spine candidates. The Juniper QFX10002-60C is a 2U fixed switch with 60 universal QSFP28/QSFP+ ports, each configurable as 40GbE or 100GbE (Juniper QFX10002 datasheet), enough to mesh with 15 leaves at four 100G links each. The Arista DCS-7280CR3K-96-F doubles that density at 96 × 100GbE-class ports and 9.6 Tbps full-duplex throughput (Arista 7280R3 datasheet). For fabrics planning a 400G next step, the Juniper QFX5130-32CD provides 32 QSFP-DD ports that also accept QSFP28 optics (Juniper QFX5130 datasheet). YT CIG additionally lists a dedicated Data Centre Spine Switch 32×100G product page for 32-port 100G spine builds.
Practically, these four vendor families cover the common telecom data-centre sizes: a single QFX10002-60C absorbs a four-leaf pilot; two of them mesh an eight-leaf regional OSS/BSS site without exhausting ports; and a CR3K-96 or a chassis-based QFX10008 absorbs fabrics beyond 16 leaves.
Port planning under the mesh rule
Spine-leaf (a two-stage CLOS topology) connects every leaf to every spine, so any server packet crosses at most two switches and no link sits on a single path. This replaces the three-tier access–aggregation–core model, where bandwidth concentrates at the aggregation layer and spanning-tree or stacking decisions constrain where uplinks go. Related topology context for carrier switches is covered in the site’s IP fabric switches and routers guide.
Port planning follows from the mesh rule:
- Links in the mesh = leaves × spines × links-per-leaf-per-spine. A 4-leaf, 1-spine design with four 100G links per leaf consumes 16 spine ports; an 8-leaf, 2-spine design at three links per leaf-spine pair consumes 48.
- Each leaf reserves S × U uplink ports, where S is spine count and U links per spine. A two-spine design at three links each consumes 6 of the CE6820’s six QSFP28 uplinks (the entire budget) leaving none for a further spine.
- Each spine reserves L × U ports. Eight leaves at three links each occupy 24 of the QFX10002-60C’s 60 ports, leaving 36 for a third leaf group or WAN uplinks.
- Reserve spine headroom for north–south exits (WAN routers, core firewalls, storage arrays), typically 20–30% of spine ports in greenfield plans, a planning convention, not a standard requirement.
Equal-cost multi-path (ECMP) forwarding spreads traffic across all parallel leaf-spine links, so links-per-leaf-per-spine should stay uniform across the fabric; a leaf with an odd link count creates asymmetric capacity that hashing cannot fully even out. The 10G/25G server side is deliberately slower than the mesh side: the oversubscription math in the next section shows why the mesh should always be the wider tier.
Sizing the 100G port count
Server count drives the arithmetic: count 10G server ports first, divide by the leaf downlink density to get leaf count, then size the mesh. The three worked designs below use only port figures from the datasheets cited above.
Design A, small telecom PoP (pilot). Four QFX5110-48S leaves serve up to 192 × 10G servers. Each leaf sends all four QSFP28 uplinks (400 G) to a single QFX10002-60C spine, which uses 16 of its 60 ports. Downlink capacity 480 G versus 400 G of uplink gives a 1.2:1 oversubscription, acceptable for a management-and-monitoring PoP.
Design B, regional OSS/BSS site. Eight CE6820-48S6CQ-B leaves (384 × 10GE servers) mesh with two QFX10002-60C spines at three 100G links per leaf-spine pair. Each spine carries 24 of its 60 ports; each leaf delivers 600 G of uplink against 480 G of downlink, a non-oversubscribed 0.8:1 fabric that suits storage and backup traffic.
Design C, 40G-uplink leaf on a 100G spine. Sixteen CE6855-48S6Q-HI leaves contribute six 40GE uplinks each. On two QFX10002-60C spines running their ports in 40G mode, each spine carries 48 of 60 ports (16 leaves × 3 links), and each leaf has 240 G of uplink against 480 G of downlink, a 2:1 oversubscription that still fits the same chassis. This is the reuse path when existing CE6855 stock must join a new 100G fabric.
Port planning table (datasheet figures)
| Design | Leaf switch | Server downlinks | Uplinks per leaf | Spine switch | Spine ports used | Oversubscription |
|---|---|---|---|---|---|---|
| A — PoP pilot | Juniper QFX5110-48S-AFO | 48 × 1/10GbE SFP+ | 4 × 100G | 1 × Juniper QFX10002-60C | 16 of 60 | 1.2:1 |
| B — regional DC | Huawei CE6820-48S6CQ-B | 48 × 10GE SFP+ | 6 × 100GE | 2 × Juniper QFX10002-60C | 24 of 60 per spine | 0.8:1 |
| C — 40G reuse | Huawei CE6855-48S6Q-HI | 48 × 10GE SFP+ | 6 × 40GE | 2 × QFX10002-60C (40G mode) | 48 of 60 per spine | 2:1 |
| Spine alternative | Arista DCS-7280CR3K-96-F | — (spine role) | 96 × 100G ports | replaces both QFX10002-60C | sized per leaf count | per leaf config |
Oversubscription is the sum of leaf downlink capacity divided by the sum of leaf uplink capacity; the mesh links themselves are never “shared” beyond the ratio the design declares. Leave at least two spine ports per leaf unbonded in Design B-style builds if a third spine might be added later, because adding a spine means re-cabling every leaf.
Which 100G optics pair with each link type?
Every mesh link in the designs above terminates in a QSFP28 cage (hardware spec SFF-8665, the 28 Gb/s 4-lane pluggable standard maintained by the SFF TA TWG of SNIA). The optic choice follows distance, not throughput: inside the same rack the answer is copper, across the data-centre floor multimode SR4, and beyond the building one of the single-mode variants. YT CIG’s 100G QSFP28 transceiver product page and its 25G/100G optical modules guide group the options by part type.
- Same rack / adjacent rack: QSFP28 direct-attach copper, such as the QSFP28-100G-CU1M 1 m DAC in the catalogue. Passive DACs avoid optic cost and power but stop at roughly rack scale.
- In-building, ≤100 m: 100GBASE-SR4 reaches 70 m over OM3 and 100 m over OM4 multimode fibre through an MPO-12 connector, with four 25.78125 Gb/s lanes per IEEE 802.3bm.
- Campus, ≤2 km: 100GBASE-CWDM4 sends four lanes on the 20 nm ITU-T G.694.2 CWDM grid (1271–1331 nm) over G.652 single-mode fibre to 2 km per the CWDM4 MSA; the catalogue’s QSFP-100G-CWDM4 record lists duplex LC, 2 km over duplex SMF, and 100G/40G dual-rate operation. The single-lambda 100GBASE-FR (1310 nm, 2 km, 100G Lambda MSA) is the newer alternative, the QSFP-100G-FR record lists a maximum 4.5 W draw, one class above the CWDM4’s 3.5 W.
- Metro, 10–40 km: 100GBASE-LR4 covers 10 km and 100GBASE-ER4 covers 40 km over G.652 single-mode fibre, both defined in IEEE 802.3bm; ZR4 modules (1550 nm) in the catalogue extend reach further under vendor-specific specifications, so confirm reach from the vendor datasheet before ordering.
- 25G breakout runs: where a QFX5110-48S uplink splits to 4 × 25G (supported per the Juniper QFX5110 datasheet), the 25G PHYs come from IEEE 802.3by-2016, the ratified 25GbE standard.
100G optic comparison (standard and catalogue figures)
| Optic class | Standard / MSA | Max reach | Fibre | Connector | Lane layout |
|---|---|---|---|---|---|
| QSFP28 DAC | Vendor spec (QSFP28-100G-CU1M) | 1 m listed | Copper | QSFP28 plug | 4 × 25G passive |
| 100GBASE-SR4 | IEEE 802.3bm | 70 m OM3 / 100 m OM4 | Multimode | MPO-12 | 4 × 25.78125 Gb/s |
| 100GBASE-CWDM4 | CWDM4 MSA | 2 km | SMF G.652 | Duplex LC | 4λ on ITU-T G.694.2 grid |
| 100GBASE-FR | 100G Lambda MSA | 2 km | SMF G.652 | Duplex LC | Single lambda 1310 nm |
| 100GBASE-LR4 | IEEE 802.3bm | 10 km | SMF G.652 | Duplex LC | 4λ CWDM 1271–1331 nm |
| 100GBASE-ER4 | IEEE 802.3bm | 40 km | SMF G.652 | Duplex LC | 4λ CWDM |
Two rules keep the optics inventory honest. First, match both ends: a CWDM4 optic and an FR optic both do 2 km over duplex LC but use different multiplexing, so a mixed pair will not link. Second, MPO-12 (SR4) and duplex LC (CWDM4/LR4/FR) cabling plants do not convert without a harness, so decide the intra-building fibre plan before ordering modules in quantity.
What oversubscription ratio fits telecom workloads?
Oversubscription is arithmetic: divide the total leaf downlink capacity by the total leaf uplink capacity, per tier. The ratios in the planning table above follow from the port counts in the vendor datasheets, so treat them as budget inputs.
Telecom OSS/BSS traffic shapes the choice. Operations support systems are dominated by north–south flows: performance-counter collection, SNMP/NetFlow polling, backup jobs and screen traffic to virtual desktops all concentrate toward the storage and WAN exit ports. Those workloads tolerate 2:1 to 3:1 oversubscription far better than a database replication fabric does. Billing systems and storage replication, by contrast, behave east–west and burst: nightly CDR batches between database and archive servers can saturate a thin mesh for hours. For those designs the 0.8:1 pattern in Design B (more uplink than downlink capacity) is the deliberate target, and 1:1 is the common upper bound for storage-heavy fabrics.
The practical budget rule: size the mesh for the busiest 30-minute window (typically the nightly batch), then check that the daily average sits comfortably below it. Where the budget forces 2:1 or thinner, keep spine-leaf links at full port speed (no 40G runs on a 100G mesh tier) so ECMP has uniform paths to hash across.
What storage suits telecom OSS/BSS servers?
OSS/BSS servers split into two storage profiles, and a telecom fabric usually needs both. Billing and customer-management databases (the “billing” leg of the Fulfilment–Assurance–Billing triad that TM Forum’s eTOM framework defines as core Level-1 operations processes) need low-latency blocks: SSD or NVMe media on the database hosts, with NVMe devices riding PCIe lanes at 16 GT/s per lane under PCI-SIG PCIe 4.0. The archive tier takes the other profile: CDR records, performance-counter history, software images and backups need cheap sequential capacity on rotating media.
For the capacity tier, the catalogue’s Huawei OceanStor 5300 V5 record shows the typical shape: a 2U dual-controller array on Huawei’s Kunpeng platform with 12 × 3.5-inch LFF bays (8 × 10 TB populated in the listed configuration), 10 TB 7.2K NL-SAS drives and Fibre Channel / iSCSI host interfaces. NL-SAS is SAS media at 12 Gb/s per lane under the T10 SAS-3 specification (INCITS 519-2014) attached in nearline fashion, capacity-class drives that keep the SAS command set for reliability at disk cost. For the database tier, SSD replacement drives such as the Huawei SSD part in the spare-parts catalogue slot into the same server chassis as the 10G SFP+ NICs feeding the fabric.
RAID choice follows the same split. Dual-parity RAID 6 suits wide NL-SAS sets, because a 10 TB drive rebuild takes many hours and a second failure during rebuild must not cost the volume. RAID 10 (mirrored pairs) suits the write-heavy billing database, trading capacity for rebuild speed. Separating the two profiles (database on local SSD or a small flash array, archive on a 12-bay NL-SAS shelf) matches the cost curve better than one large hybrid array, and matches the 0.8:1 fabric pattern in Design B, where storage and backup hosts get unconstrained mesh bandwidth.
How should you plan headroom beyond 100G?
The 100G fabric you build today should survive one vendor generation, and the standards make that cheap to plan for. IEEE 802.3bs-2019 defines the 400G PHYs, 400GBASE-SR8 for roughly 100 m over OM4, 400GBASE-DR4 for 500 m, and 400GBASE-FR4 (2 km) and 400GBASE-LR4 (10 km) for single-mode runs, packaged in the QSFP-DD form factor, which carries eight 50G electrical lanes per the QSFP-DD MSA.
The catalogue already contains both upgrade paths. The Juniper QFX5130-32CD’s 32 QSFP-DD ports accept today’s QSFP28 optics and tomorrow’s 400G modules, so a spine bought for 100G keeps its slot layout across the migration. The Arista DCS-7280DR3A-36-F goes further with 36 × 400GbE ports at 14.4 Tbps full-duplex (catalogue record). Cabling decisions now determine migration friction later: duplex LC plants map directly onto 400GBASE-FR4/LR4, while MPO-12 SR4 plants map onto DR4/SR8, so the optic table above is also the 400G cabling plan.
FAQs on 100G spine-leaf port planning
Can a 40G leaf switch join a 100G spine fabric?
Yes, when the spine port supports 40G mode. The Juniper QFX10002-60C’s 60 universal QSFP28/QSFP+ ports each run 40GbE or 100GbE per the Juniper QFX10002 datasheet, and Huawei’s CE6855-48S6Q-HI leaf contributes six 40GE QSFP+ uplinks per the CloudEngine 6800 datasheet. Design C above budgets 48 of 60 spine ports for exactly this pattern.
A leaf needs 25G server ports. Do these models fit?
Check the downlink side first. The QFX5110-48S-AFO and the CE6820-48S6CQ-B both present 48 × 1/10GbE SFP+ server ports, so a 25G server fleet needs a different leaf class; breakout does not help here. Breakout adds 4 × 25G only on the uplink side, and only where the platform supports it: the QFX5110-48S does, while the CE6820 documentation states its QSFP28 ports cannot be split into 4 × 25GE.
We already run CWDM4 for 2 km campus links. Can those modules serve a new FR-planned span?
Only as a matched pair, and only at 2 km. CWDM4 (CWDM4 MSA) puts four lanes on the 20 nm ITU-T G.694.2 grid over G.652, while 100GBASE-FR (100G Lambda MSA) carries a single 1310 nm lambda. Both reach 2 km over duplex LC, but a CWDM4 at one end and an FR at the other will not link, so keep the two inventories separate and standardise per span.
Design B leaves 36 spine ports free. What should they be held for?
A third leaf group of twelve at three links per spine takes 24 of them, which is the growth path the design anticipates. The remainder covers WAN and storage exits, or a third spine if the fabric outgrows two. Port counts come from the Juniper QFX10002 datasheet and Huawei CE6820 documentation.
How do I size the archive tier against the database tier?
Keep them apart. Database hosts want SSD or NVMe on local media, with NVMe riding PCIe lanes at 16 GT/s per lane under PCI-SIG PCIe 4.0. CDR records and performance-counter history want cheap sequential capacity: NL-SAS drives in a wide shelf, RAID 6 over the set because a rebuild on a 10 TB drive takes hours, and FC or iSCSI host interfaces into the fabric. A single hybrid array serving both profiles usually loses on cost per usable terabyte.
Is there a standard oversubscription ratio to design to?
No. The ratio falls out of the port counts you can afford and the traffic shape you expect: OSS/BSS north–south flows tolerate 2:1 to 3:1, while billing and storage replication want the 0.8:1 pattern of Design B or better. Treat the ratios in the planning table as budget inputs derived from datasheet port counts, and revisit them when the traffic profile changes.
Need a specific part?
Send the part number and quantity — we confirm condition, availability and price before you commit.