OSPF troubleshooting

OSPF TROUBLESHOOTING

OSPF runs on top of IP and uses protocol number 89

OSPF doesn't use any transport protocol, such as TCP, for reliability. The protocol itself has a reliable mechanism of transportation.

Debugs in OSPF normally are not very CPU-intensive unless the problem is impacting the entire OSPF network. For example, if OSPF neighbors are not coming up, turning on debug ip ospf adj is not CPU-intensive unless 300 neighbors are having problems at the same time.

Troubleshooting OSPF neighbor relationships

Troubleshooting OSPF route advertisement

Troubleshooting OSPF route installation

Troubleshooting redistribution problems in OSPF

Troubleshooting route summarization in OSPF

Troubleshooting CPUHOG problems

Troubleshooting dial-on-demand routing (DDR) issues in OSPF

Troubleshooting SPF calculation and route flapping

Common OSPF error messages

1)Troubleshooting OSPF neighbor relationships

-      OSPF neighbor relationship problems can be of any of these types:

The OSPF neighbor list is empty.

An OSPF neighbor is stuck in ATTEMPT.

An OSPF neighbor is stuck in INIT.

An OSPF neighbor is stuck in 2-WAY.

An OSPF neighbor is stuck in EXSTART/EXCHANGE.

An OSPF neighbor is stuck in LOADING.

1.      Problem: The OSPF neighbor list is empty

·         OSPF is not enabled on the interface.

·         Layer 1/2 is down.

·         The interface is defined as passive under OSPF.

-      When an interface is defined as passive under router OSPF, it suppresses OSPF Hellos. This means that OSPF does not send or receive any Hellos on such interfaces. Therefore, no adjacency is formed.

-      passive-interface: the command is entered so that the router cannot take part in any OSPF process on that segment. This is the case when you don't want to form any neighbor relationship on an interface but you do want to advertise that interface.

-      In OSPF, a passive interface means "do not send or receive OSPF Hellos on this interface." So, making an interface passive under OSPF with the intention of preventing the router from sending any routes on that interface but receiving all the routes is wrong.

·         An access list is blocking OSPF Hellos on both sides.

-      OSPF sends its Hello on a multicast address of 224.0.0.5. This address should be permitted.

·         A subnet number/mask has been mismatched over a broadcast link.

·         The Hello/dead interval has been mismatched.

·         The authentication type (plain text versus MD5) has been mismatched.

·         An authentication key has been mismatched.

·         An area ID has been mismatched.

·         Stub/transit/NSSA area options have been mismatched.

·         An OSPF adjacency exists with secondary IP addressing.

ü  show ip ospf neighbor - the output displays the OSPF neighbor status

ü  show ip ospf interface - to verify ospf interfaces are up/down, to verify if interface is defined as passive in the output check :(No Hellos (Passive interface))

ü  debug ip ospf adj - Look out for outputs mismatch

2.      An OSPF neighbor is stuck in ATTEMPT.

This problem is valid only for NMBA networks in which neighbor statements are defined. Stuck in ATTEMPT means that a router is trying to contact a neighbor by sending its Hello but hasn't received any response.

Causes:

·         Misconfigured neighbor statement

·         Unicast Connectivity Is Broken on NBMA, cause for this broken connectivity can be an access list is blocking the unicast.

3.      An OSPF neighbor is stuck in INIT.

When a router receives an OSPF Hello from a neighbor, it sends the Hello packet by including that neighbor's router ID in the Hello packet. If it doesn't include the neighbor's router ID, the neighbor will be stuck in INIT

Causes:

·         An access list on one side is blocking OSPF Hellos.

·         Authentication is enabled on only one side (virtual link example).

·         Hellos are getting lost on one side at Layer 2.

4.      An OSPF neighbor is stuck in 2-WAY.

Cause Priority 0 Is Configured on All Routers

It is normal in broadcast media to have a 2-WAY state because not every router becomes adjacent on broadcast media. Every router enters into FULL state with the DR and the BDR.In this example, there are only two routers on Ethernet; both are configured with priority 0. Priority 0 means that this router will not take part in DR/BDR election process. This configuration is useful when there are "low-end" routers on the segment and the desire is not to make those low-end routers DRs. For this purpose, you should configure priority 0. By default, the priority is set to 1. A

router with the highest priority on a segment wins a DR election. If all priorities are kept to the default, the router with the highest router ID becomes the DR

If all the routers on an Ethernet segment are configured with priority 0, no routers on the segment will be in FULL state with any other router. This creates problems. At least one router on the segment must have a priority that is not set to 0.

Solution:

To fix this problem, remove the priority 0 command on at least one router so that router becomes a

DR and forms a FULL adjacency

5.      An OSPF neighbor is stuck in EXSTART/EXCHANGE.

In this state, the router elects a master and a slave and the initial sequence number. The whole database also is exchanged during this state. If a neighbor is stuck in EXSTART/EXCHANGE for a long time, it is an indication of a problem

The most common possible causes of this problem are as follows:

·         Mismatched interface MTU

Solution check output of #debug ip ospf adj

Shows o/p as: OSPF: Nbr 131.108.1.2 has larger interface MTU

·         Duplicate router IDs on neighbors

·         Inability to ping across with more than certain MTU size

·         Broken unicast connectivity because of the following:

- Access list blocking the unicast

- NAT translating the unicast

If NAT is misconfigured, it will start translating the unicast packet coming toward it, which will break the unicast connectivity. R1 is configured with NAT. The outside inter-face of R1 is Serial 0.2, which connects to R2

When R2 sends a unicast packet to R1, R1 tries to translate that packet and R2 never receives the ping reply. The main thing to watch for is the access list in NAT. If the access list is permitting everything, this problem will occur To solve this problem, change access list 1 and permit only those IP address that require translation.The access list could be different from network to network. The whole idea is that the access list permit statement should not cover the neighbor's IP address. Include only the inside network 10.0.0.0/8 is permitted.

6.      An OSPF neighbor is stuck in LOADING.

When a neighbor is stuck in the LOADING state, the local router has sent a link-state request packet to the neighbor requesting an outdated or missing LSA and is waiting for an update from its neighbor. If a neighbor doesn't reply or a neighbors' reply never reaches the local router, the router will be stuck in the LOADING state.

·         The most common possible causes of this problem are as follows:

-       Mismatched MTU

-       Corrupted link-state request packet

o   When a link-state request packet is corrupted, the neighbor discards the packet and the local router never receives the response from the neighbor.  This causes the OSPF neighbor to be stuck in the LOADING state.

Link-state request packets usually become corrupted because of the following reasons:

                                                                                                       I.            A device between the neighbors, such as a switch, is corrupting the packet.

                                                                                                    II.            The sending router's packet is invalid. In this case, either the sending router's interface is bad or the error is caused by a software bug.

                                                                                                 III.            The receiving router is calculating the wrong checksum. In this case, either the receiving router's interface is bad or the error is caused by a software bug. This is the least likely cause of this error message.

Solution

Most of the time, this problem is fixed by replacing hardware. This could be a simple bad port on the

switch or a bad interface card on the sending/receiving router

2)Troubleshooting OSPF route advertisement

OSPF is a link-state protocol. When it forms neighbor relationships, it exchanges the entire link-state database with its neighbor(s).

The most common reasons for OSPF to not share the database information about a specific link are as follows:

-       The OSPF neighbor is not advertising routes.

-       The OSPF neighbor (ABR) is not advertising the summary route.

-       The OSPF neighbor is not advertising external routes.

-       The OSPF neighbor is not advertising the default route.

1.      OSPF Neighbor Is Not Advertising Routes

When a neighbor doesn't advertise a route, that route will not show up in the local router's routing table. This means that the neighbor has not included the route in its database; otherwise, the local router must have received it.

The most common possible causes of this problem are as follows:

·         OSPF is not enabled on the interface that is supposed to be advertised.

·         The advertising interface is down.

·         The secondary interface is in a different area than the primary interface.

2. OSPF Neighbor (ABR) Not Advertising the Summary Route

The ABR generates the summary LSA for one area and sends it to another area. When the ABR fails to generate the summary LSA, the areas become isolated from each other.

The most common possible causes of this problem are as follows:

·         An area is configured as a totally stubby area.

·         An ABR is not connected to area 0.

·         A discontiguous area 0 exists.

3. OSPF Neighbor Is Not Advertising External Routes

Whenever there is a redistribution in OSPF, it generates an external LSA (Type 5) that is flooded throughout the OSPF network. External LSAs are not leaked into stub, totally stubby, and NSSA areas.

The most common possible causes of this problem are as follows:

·         The area is configured as a stub or NSSA.

·         The NSSA ABR is not translating Type 7 into Type 5 LSA.

4. OSPF Neighbor Not Advertising Default Routes

The most common possible causes for an OSPF router not to advertise the default route are as follows:

·         The default-information originate command is missing.

·         The default route is missing from the neighbor's routing table.

·         A neighbor is trying to originate a default into a stub area.

·         The NSSA ABR/ASBR is not originating the Type 7 default.

4) Troubleshooting OSPF Route Installation

It happens that OSPF routers have fully synchronized their databases with those of their neighbors but are not installing routes in the routing table.

After the route is in the database, there can be several reasons that the route is not installed in the database

The most common reasons for OSPF failing to install routes in the routing table are as follows:

·         OSPF is not installing any routes in the routing table.

·         OSPF is not installing external routes in the routing table.

1. OSPF is not installing any routes in the routing table.

This is common problem in OSPF to find routes in the database but not in the routing table.

When OSPF finds any kind of discrepancy in the database, it does not install any routes in the routing table.

·         The most common possible causes of this problem are as follows:

·         The network type is mismatched.

·         IP addresses are flipped in dual serial-connected routers or a subnet/mask mismatch has occurred.

·         One side is a numbered and the other side is an unnumbered point-to-point link.

·         A distribute list is blocking the routes' installation.

4) Troubleshooting Redistribution Problems in OSPF

When a router in OSPF does the redistribution, it becomes an ASBR. The routes that are redistributed into OSPF could be directly connected routes, static routes, or dynamically learned routes from another routing protocol or another OSPF process.

5) Troubleshooting Route Summarization in OSPF

The idea is that if there are contiguous ranges of addresses, instead of advertising every network, you can form a group of contiguous networks and summarize those networks in one, two, or fewer blocks and advertise those blocks. This feature helps reduce the size of the routing table. Reducing the routing table size decreases the convergence time and increases OSPF performance. Thus, summarization needs to be configured manually on the router.

OSPF can use two types of summarization:

·         Interarea summarization that can be done on the ABR

·         External summarization that can be done on the ASBR

Two common problems related to summarization in OSPF are as follows:

·         A router is not summarizing interarea routes.

Cause: area range Command Is Not Configured on ABR

ensure that the area range command is configured on the correct router. Area range

summarization can be done only on the ABR. In summarization, instead of originating separate LSAs for each network, the ABR originates summary LSAs to cover those ranges of addresses.

When configuring the area range command, make sure that the summarization mask is in the form of a prefix mask rather than a wildcard mask

·         A router is not summarizing external routes

Cause: summary-address Command Is NotConfigured on ASBR

6) Troubleshooting CPUHOG Problems

The CPUHOG messages usually appear in two significant stages:

·         Neighbor formation process

·         LSA refresh process

Problem: CPUHOG Messages During Adjacency Formation—Cause: Router Is Not Running Packet-Pacing Code

Problem: CPUHOG Messages During LSA Refresh Period—Cause: Router Is Not Running LSA Group-Pacing Code

7) Troubleshooting SPF Calculation and Route Flapping

Whenever there is a change in topology, OSPF runs the SPF algorithm to compute the shortest path first tree again. Unstable links existing within the OSPF network could cause constant SPF calculation. This section discusses the problem of SPF running constantly in the network for the following reasons:

·         Interface flap within the network

·         Neighbor flap within the network

·         Duplicate router ID

1. SPF Running Constantly—Cause: Interface Flap Within the Network

Whenever there is a link flap in an area, OSPF runs SPF. So, if a network has unstable links, it can cause constant SPF run. SPF itself is not a problem because OSPF is just adjusting the change in database through calculating SPF. The real prob-lem occurs if there are small routers in the network and a constant SPF run might cause a CPU spike in a router. A link flap is shown in Figure. Because R1 also is included in area 0, any link flap in area 0 causes all routers in area 0 to run SPF.

Determining How Often SPF Is Running use command  show ip ospf and check for the output SPF algorithm executed x times

to find out which particular LSA is flapping is to turn on debug ip ospf monitor. This

debug shows exactly which LSA is flapping.

R1# debug ip ospf monitor

OSPF: Schedule SPF in area 0.0.0.0

Change in LS ID 192.168.1.129, LSA type R,

OSPF: schedule SPF: spf_time 1620348064ms wait_interval 10s

next step is to go on that router whose router LSA is flapping and check the log for any interface flap.

Actually two solutions exist in this case:

·         Fix the link flap.

·         Redefine the area boundaries.

Actually two solutions exist in this case:

l Fix the link flap.

l Redefine the area boundaries.

2. SPF Running Constantly—Cause: Neighbor Flap Within the Network

When a neighbor goes down, it causes a change in topology, so SPF runs

There is a way to track the neighbor changes in OSPF. Configure ospf log-adjacency-changes under router ospf to track all the neighbor changes.

router ospf 1

ospf log-adjacency-changes

When this command is configured, it saves all the neighbor state changes in the router's sys log.

3. SPF Running Constantly—Cause: Duplicate Router ID

When two routers have identical router IDs, confusion

results in the OSPF topology database, and the route keeps getting added and deleted. The most common symptom of this problem is that the LS Age field always has a small value.

This problem usually is generated by a cut and paste of a router configuration into another router. This results in two routers with identical router IDs

Common OSPF Error Messages

1)"OSPF: Could not allocate router id"

This message appears in two situations:

l No up/up interface with a valid IP address

l Not enough up interfaces with a valid IP address for multiple OSPF processes

OSPF requires a valid IP address that is up/up so that it can allocate a router ID for the OSPF

process. The IP address must be assigned on an up/up interface. If a router fails to allocate router

IDs, OSPF will not function. This problem can be corrected by using loopback addresses.

The loopback interface solution works for both situations. Just configure a loopback interface for one

process. If you are trying to run more than one process, you might need more than one loopback

interface.

2)"%OSPF-4-BADLSATYPE"

"%OSPF-4-BADLSATYPE: Invalid lsa: Bad LSA type" Type 6

Error Message

This is normal if the neighboring router is sending the multicast OSPF (MOSPF) packet. For more

information on MOSPF, refer to RFC 1584. Cisco routers do not support MOSPF, so they simply ignore

it. To get rid of these messages, simply type the following:

router ospf 1

ignore lsa mospf

If the type is something other than 6, it's probably a bug or a memory corruption error

3)"%OSPF-4-ERRRCV"

This message means that OSPF received an invalid packet.

Three common types of this message can occur:

a)      Mismatch area ID

b)      Bad checksum

c)      OSPF not enabled on the receiving interface

a)      Mismatched Area ID

This message looks like this:

%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from backbone area must be virtual-link but not found from 170.170.3.3, Ethernet0

This means that the neighbor's interface connecting to this interface is in area 0 but that this interface  is not in area 0. In this situation, the router will not form an OSPF adjacency with the neighbor that this packet comes from. This also happens if one side's virtual link is misconfigured.To avoid these messages, make sure that both sides have the same area ID by checking the network statement under OSPF in the router configuration. For example, if the link 10.10.10.0/24 between two routers should be in area 1, make sure that the network statement on both routers includes this particular link in area 1.

The network command would look like this:

router ospf 1

network 10.10.10.0 0.0.0.255 area 1

If a virtual link is configured, double-check the configuration for virtual link.

b)      Bad Checksum

The message looks like this:

%OSPF-4-ERRRCV: Received invalid packet: Bad Checksum from 144.100.21.141, TokenRing0/0

This means that OSPF encountered an error in a packet that was received. This is because the OSPF checksum does not match the OSPF packet that was received by this router.

This problem has three causes:

1.      A device between the neighbors, such as a switch, is corrupting the packet.

2.      The sending router's packet is invalid. In this case, either the sending router's interface is bad or a software bug is causing the error.

3.       The receiving router is calculating the wrong checksum. In this case, either the receiving router's interface is bad or a software bug is causing the error. This is the least likely cause of this error message.

This problem can be difficult to troubleshoot, but you can start with the following solution, which is effective in 90 percent of cases. It's important that you follow the steps in order:

Step 1. Change the cable between the routers. For the example given in this section, this

would be the router that is sending the bad packet (144.100.21.141) and the router that is

complaining about these bad packets.

Step 2. If Step 1 doesn't fix the problem, use a different port on the switch between the

routers.

Step 3. If Step 2 doesn't fix the problem, connect the routers directly using a cross-over

cable. If you receive no further messages, the switch most likely is corrupting the packet.

If none of these steps solves the problem, contact the Cisco Technical Assistance Center (TAC) and work with an engineer to look for a bug in Cisco IOS Software or to obtain a possible Return Material Authorization (RMA) for partial or full parts replacement.

c)      OSPF Not Enabled on the Receiving Interface

The message looks like this:

%OSPF-4-ERRRCV: Received invalid packet: OSPF not enabled on interface from

141.108.16.4, Serial0.100

The router generating this message received a packet from 141.108.16.4 on Serial0.100, but OSPF isnot enabled on the Serial0.100 interface. This message is generated only once for a non-OSPF interface.

View/Debug Commands

show ip ospf interface
show ip ospf database
show ip ospf database network (lsa type 2) 
show ip ospf database router (lsa type 1) 
show ip ospf database summary (lsa type 3)
show ip ospf database asbr-summary (lsa type 4)
show ip ospf database external (lsa type 5)
show ip ospf database nssa-external (lsa type 7)
show ip ospf virtual-links
show ip ospf border-routers
show ip ospf statistics
debug ip ospf hello
debug ip ospf adj