Why an optical link keeps flapping and how to isolate it
A flapping optical link after an upgrade almost always means the two ends no longer agree on something the upgrade changed: FEC mode, speed or autonegotiation, transceiver coding, or a port profile default. Confirm the physical link is actually dropping, read the DDM values at the moment of the drop, then diff the port configuration against the pre-upgrade baseline.
What does a flapping optical link look like in the logs?
Two different faults get called flapping, and they send you in opposite directions. One is the physical link bouncing: carrier down and up, every protocol above it restarts. The other is a protocol session restarting while the carrier stays up: LACP times out, BFD drops, BGP resets. The first question concerns which fault you have.
On Cisco IOS, %LINK-3-UPDOWN tracks carrier state and %LINEPROTO-5-UPDOWN tracks the line protocol; Huawei VRP writes comparable entries into display logbuffer. A carrier transition counter that stays at zero through an incident means light never stopped arriving.
Timestamps carry more information than the message text. LACP slow mode times out after 90 seconds (3 × 30 s) and fast mode after 3 seconds (3 × 1 s); BFD expires at 3× the negotiated interval. A drop every few minutes with the carrier bouncing is physical, and the period often matches an err-disable recovery timer.
Error counters split the physical cases. A port that accumulates CRC, symbol or FEC-corrected errors before each drop is losing margin: light arrives, but the eye is closing. A port that drops with zero errors lost the signal outright, through a laser turning off, a connector pulled or a policy shutdown.
| Flap period in the log | Carrier transitions | Likely mechanism |
|---|---|---|
| Random, seconds to minutes | Yes | Marginal optical budget, dirty endface, DDM alarm |
| Exactly 90 s or 3 s | No | LACP slow or fast timeout; carrier never dropped |
| Fixed multi-minute period | Yes | Err-disable recovery timer returning the port |
| Same time daily | Yes | Thermal cycle, HVAC failure, scheduled config job |
| Only under full traffic | Yes | FEC/BER margin, module heating, supply rail sag |
| At config save or reload | Yes | Configuration regression reinstalled by a template |
Why does the flap start after an upgrade and not before?
An upgrade is a controlled experiment somebody already ran. Four things change at once, and each maps to a different layer.
Line rate and lane count. Moving a port from 10GBASE-LR to 25GBASE-LR takes the lane from 10.3125 Gb/s to 25.78125 Gb/s, and IEEE 802.3by makes RS(528,514) FEC part of 25GBASE-R in Clause 108. A link that ran without FEC at 10G now depends on it to hold the bit error rate. If a media converter, OEO regenerator or mux transponder does not terminate FEC the same way, the port trains, carries traffic briefly, then drops on uncorrectable blocks. At 100G the effect is sharper: 100GBASE-LR4 carries four lanes at 25.78125 Gb/s under IEEE 802.3ba, so one lane 6 dB below its neighbours gives a link that comes up and later fails as a whole. The 25G/100G module guide covers the lane-rate differences.
Defaults that were reset. FEC mode, speed negotiation and link-flap detection are port properties whose defaults belong to the release. A new baseline that enables link-flap err-disable converts one transient into a disabled port, and with recovery enabled, into a port that returns on a timer and drops again.
Coding enforcement. Some platforms re-read module EEPROM more strictly after an upgrade and reject a recoded module the old release tolerated. Usually the port refuses outright; a marginal module can pass identity checks and fail later on DDM alarms.
Heat and power. A 100G QSFP28 dissipates several times the power of a 10G SFP+ in the same cage footprint, and a new line card can add hundreds of watts to a chassis whose airflow was balanced for the old configuration.
The upgrade window is also when people re-patch, so the software often gets blamed for a connector disturbed in the same hour.
Which layer should you test first: module, fibre path, configuration or environment?
Test in order of discrimination per minute of work. A config diff costs five minutes; a module swap costs ten and needs a spare. Configuration comes first on any link an upgrade touched, and the fibre check comes first where somebody re-patched.
Read the carrier transition counter and the syslog period before either. A protocol timeout with a steady carrier takes optics out of the picture.
| Layer | What an upgrade changes | Prior before evidence | Evidence that raises it |
|---|---|---|---|
| Configuration | FEC, speed, autoneg, err-disable and loop-detect defaults | High | Ends disagree on FEC or speed; flap begins at first save or reload |
| Optical path | Re-patched connectors, added patch panel, bend radius, contamination | High if cabling was touched, medium otherwise | RX low or moving; cleaning shifts RX by over 0.5 dB |
| Module | Coding, EEPROM identity, laser ageing, thermal behaviour | Medium | DDM alarm at the drop; TX bias drifting upward |
| Environment | Rack temperature, airflow, 3.3 V rail, vibration | Low to medium | Flap follows time of day or load instead of config |
Those priors are conditional on what the upgrade touched. No vendor publishes flap statistics by cause. The ordering is the defensible output; percentages would be invented.
A workable sequence for 30 minutes:
- Counters and syslog. Carrier transitions, CRC and symbol errors, FEC corrected and uncorrected, interval between drops.
- Configuration diff. Running config on both ends against the pre-upgrade backup: FEC, speed, autoneg, MTU, err-disable causes, loop detection.
- DDM read at the flap. Pull the alarm history as well as the current value.
- Endface inspection and clean. IEC 61300-3-35 defines the pass/fail zones; re-read RX afterward.
- Swap with a known-good module or spare patch cord. The side that keeps the fault is the faulty side.
What do the DDM readings tell you that the link state does not?
DDM shows whether light was marginal before the drop or disappeared outright. SFF-8472 puts five real-time values in the A2h page at bytes 96 to 105, two bytes each, and the alarm and warning thresholds at bytes 0 to 39. Raw units are fixed: temperature in 1/256 °C, voltage in 100 µV, bias in 2 µA, and both optical powers in 0.1 µW. A raw power reading of 10000 counts is 1 mW, or 0 dBm.
| DDM parameter (A2h) | Raw unit | Reference, SFP-10G-LR | Flap signature | What it rules out |
|---|---|---|---|---|
| Temperature, 96–97 | 1/256 °C, signed | 0 to 70 °C commercial, per datasheet | Climbs under traffic, falls when traffic stops | Cooling when it tracks load |
| Voltage, 98–99 | 100 µV | 3.3 V ±5%, or 3.135 to 3.465 V | Sags at insertion or full load | Module fault when every module sags together |
| TX bias, 100–101 | 2 µA | The bias figure in the part datasheet | Rises while TX power holds | The fibre; this is a laser-side signature |
| TX power, 102–103 | 0.1 µW | −8.2 to +0.5 dBm, per datasheet | Falls while bias holds or rises | Slope-efficiency loss in laser or driver |
| RX power, 104–105 | 0.1 µW | −14.4 to +0.5 dBm, per datasheet | Wanders after rack work, or sits at low alarm | The path when steady; the module when static |
TX bias and TX power together trace the laser’s L-I curve. Output power is roughly the product of bias above threshold and slope efficiency. Bias climbing while output holds means threshold current rose, the ordinary ageing path for a DFB laser. Output falling while bias holds means slope efficiency dropped, which points at the facet or the driver.
RX power is the other half of the argument. Compare it against the far end’s TX power minus the calculated path loss. A receiver 3 dB below that estimate has lost light between the two cages, and the cause sits in connectors, splices, bend radius or a damaged strand.
Two reading habits prevent wasted swaps. Pull the alarm history instead of the snapshot: a platform polling DDM on a slow cycle can miss a two-second event while the threshold-crossing counters keep it. The module’s own thresholds are stored in EEPROM, so a badly calibrated module can alarm inside its datasheet range.
On a four-lane module the per-lane spread matters more than any single value. A 100G QSFP28 with one lane 5 dB below the other three trains, passes light, and fails under load once FEC runs out of correction capacity.
How much optical margin does the link need, and how do you compute it?
Enough margin to survive the worst-case span plus every connector, with something left for ageing. The 10GBASE-LR numbers make it concrete. The Cisco SFP-10G-LR datasheet gives a transmitter floor of −8.2 dBm and a receiver sensitivity of −14.4 dBm, so the power budget is 6.2 dB. Ten kilometres of ITU-T G.652.D at its maximum 0.35 dB/km at 1310 nm consumes 3.5 dB. That leaves 2.7 dB for every mated pair, splice, patch panel and year of ageing on the span.
A link whose measured RX sits 1 dB above −14.4 dBm has 1 dB of headroom in total. One dirty endface or one extra patch panel exceeds it.
| Reach class | Standard | Lanes × line rate | Fibre and reach | Wavelength plan |
|---|---|---|---|---|
| 10GBASE-LR | IEEE 802.3ae | 1 × 10.3125 Gb/s | G.652 SMF, 10 km | 1310 nm band, 1260–1355 nm |
| 10GBASE-SR | IEEE 802.3ae | 1 × 10.3125 Gb/s | OM3 300 m, OM4 400 m | 850 nm, 840–860 nm |
| 25GBASE-LR | IEEE 802.3by | 1 × 25.78125 Gb/s | G.652 SMF, 10 km | 1310 nm |
| 100GBASE-LR4 | IEEE 802.3ba | 4 × 25.78125 Gb/s | G.652 SMF, 10 km | 1295.56 to 1309.14 nm |
| 100GBASE-CWDM4 | CWDM4 MSA, 2014 | 4 × 25.78125 Gb/s | G.652 SMF, 2 km | 1271, 1291, 1311, 1331 nm |
| 100GBASE-SR4 | IEEE 802.3bm | 4 × 25.78125 Gb/s | OM3 70 m, OM4 100 m | 850 nm |
A 10GBASE-LR port and a 100GBASE-CWDM4 port can share a duplex LC connector and the same fibre, but the 100G module reaches 2 km with four wavelengths on a 20 nm grid, while LR4 uses four narrower-spaced wavelengths around 1310 nm. Replacing one end without the other leaves the link dark or intermittently erroring. The 100G QSFP28 guide and the CWDM versus DWDM comparison cover the plans.
Too much light is as real a fault as too little. The SFP-10G-LR receiver saturates above +0.5 dBm, so a 40 km or 80 km module redeployed onto a 2 km span can overdrive the receiver and produce errors that look like a bad laser.
IEC 61300-3-35 defines visual pass/fail zones for endfaces, and one contaminated ferrule can add more loss than the entire 2.7 dB allowance. IEC 61754-20 defines the LC interface, where an 8-degree APC ferrule mated to a flat UPC ferrule adds loss and a reflection that destabilises the laser. IEC 61754-7 covers MPO, where a Type A versus Type B polarity error swaps transmit and receive pairs so some lanes carry light and others do not.
Which configuration mismatches survive a successful link-up?
The mismatches that waste days are the ones the link state hides. A wrong wavelength gives a dead port in seconds; a mismatch that lets the port come up fails hours later under traffic.
FEC sits at the top because most optical ports configure it per end and never negotiate it. RS(544,514) in IEEE 802.3 Clause 91 corrects 15 ten-bit symbols per codeword; RS(528,514) in Clause 108 corrects 7. A link running the wrong one accumulates uncorrectable blocks at a rate the corrected counter never explains.
Speed and autonegotiation come next. 1000BASE-X autonegotiation is defined in IEEE 802.3 Clause 37; 10GBASE-R optical ports have no equivalent, so a port left on speed auto against a fixed 10G peer can train at the wrong rate or refuse to settle. On backplane and DAC links, link training (Clause 72 for 10GBASE-KR, Clause 93A for 25GBASE-KR and CR) adds another per-end setting.
Policy settings produce flapping that looks optical and is not. Link-flap err-disable counts carrier transitions and disables the port when the count is exceeded; with recovery enabled, the port returns on a timer and trips again, giving a log period that matches the recovery interval exactly. Loop detection and storm control do the same on access ports.
When the module, fibre and config all test clean, what else flaps a link?
Three physical mechanisms remain once the module, fibre and configuration pass: heat, power and mechanical seating.
Heat produces a schedule. A module’s temperature follows rack airflow, so a flap that starts at 14:00 and stops at 20:00 tracks the building’s cooling. Missing blank panels above a new line card let exhaust recirculate, and a fan tray reinstalled the wrong way does the same. A 10 °C rise after an upgrade is enough to move a marginal module across its threshold.
Power faults are rarer and harder to see. SFF-8472 specifies a 3.3 V ±5% supply, and the module samples that rail inside the cage. If every module on a card dips at once, the card supply or chassis power budget is the suspect.
Mechanical faults come from the upgrade window. An LC connector that clicked but did not latch, a patch cord hanging from its boot, a cage with a bent spring finger or a module seated at an angle produce a link that survives until vibration moves it. Re-seating the module and re-latching every connector takes two minutes.
Software belongs last. A release that changed FEC handling or link training can produce a flap no optical measurement explains, and the tell is that the same hardware behaves correctly on the previous release.
What is the exclusion checklist before replacing hardware?
Run the steps in this order: each removes a layer at a known cost.
| Step | Action | Pass criterion | What it rules out |
|---|---|---|---|
| 1 | Count carrier transitions during the incident | Zero transitions while service dropped | Optics; look at LACP, BFD, routing timers |
| 2 | Diff running config against the pre-upgrade backup | FEC, speed, autoneg, err-disable causes identical | Configuration drift |
| 3 | Pull DDM and alarm history at the drop | No threshold crossing; RX and TX stable | Module alarm, path loss, thermal trip |
| 4 | Inspect and clean endfaces to IEC 61300-3-35 | Cleaning changes RX by under 0.5 dB | Contamination and endface damage |
| 5 | Swap in a known-good module of the same code | Fault follows the module | Confirms the module |
| 6 | Swap patch cord or move to a spare strand | Fault stays on the original path | Confirms the fibre path |
| 7 | Loop the port electrically at the host | Link holds under loopback for an hour | Host serdes and port hardware |
| 8 | Soak 24 hours with production-like traffic | No transitions, no uncorrectable FEC blocks | Thermal and load-dependent faults |
The outcome of steps 5 and 6 is the decision. A fault that follows the module justifies an RMA; the 10G SFP product family and 100G QSFP28 family list coding and condition per unit, so the replacement can be matched to the port. A fault that stays on the path calls for re-termination, a re-clean, or an attenuator if the receiver is overdriven. A fault that stays on the port after both swaps points at the line card or its serdes settings.
Two shortcuts make things worse. Replacing every module on a span at once destroys the evidence and may introduce a second fault. Disabling FEC to stabilise a 25G or 100G link moves the failure from corrected errors to uncorrected ones, because the code is part of the 25GBASE-R specification in IEEE 802.3by and the budget was calculated with it in place.
Frequently asked questions
Can a dirty connector cause flapping, or only extra loss?
Both. Contamination absorbs and scatters light, so it adds loss and moves the receiver toward its sensitivity floor. On a link with 2.7 dB of headroom, 1 dB of dirt is a third of the margin. The loss also fluctuates as the connector expands with temperature, turning static attenuation into a link that drops and recovers. IEC 61300-3-35 defines the visual zones.
Will replacing the transceiver stop the flap?
Only when the fault travels with the module. Fit a known-good module of the same part code and watch RX. Same receive power means the light arriving has not changed, so the path or the far-end transmitter is responsible. If the fault follows the old module, the module is the cause. The transceiver verification guide covers the EEPROM checks.
Why does the link drop only under full traffic?
Idle traffic understates every margin. FEC corrects errors up to its limit, RS(544,514) corrects 15 symbols per codeword, and a link near that limit passes cleanly at low load and exceeds it when the error rate rises. Traffic also raises module temperature and supply current. A flap within minutes of a traffic ramp is a margin fault.
What DDM temperature is too hot for an SFP?
Judge it against the module’s grade and its own thresholds. The Cisco SFP-10G-LR datasheet rates the commercial part for 0 to 70 °C. A reading within 5 °C of the high alarm is a finding even when the alarm has not tripped. The SFP-10G-LR catalogue entry states condition and coding so the rating can be checked.
How long should a soak test run before the link is declared fixed?
Twenty-four hours with production-like traffic, long enough to cover a full diurnal temperature cycle. Thirty minutes proves the link comes up, not that it survives the afternoon. Watch carrier transitions and uncorrectable FEC blocks, and compare RX at the start and end.
Does a software upgrade change FEC defaults?
On many platforms it does, because FEC is a per-port property and its default belongs to the release. An upgrade can switch a port from no FEC to RS-FEC without touching the saved configuration. Check both ends after every upgrade and record the setting in the change ticket.
What is the cheapest first test when a link flaps?
Read the carrier transition counter and the syslog period. It costs nothing, needs no spare hardware, and splits physical from protocol before anyone touches a fibre.
Need a specific part?
Send the part number and quantity — we confirm condition, availability and price before you commit.