One of the most critical network links in any organization is the link to the internet. When the internet is up (and responding quickly), users are happy. When the internet is down, you’ll very quickly hear about the problem.
Many organizations have identified the internet as a critical resource that should never be down. As a result, a redundant Internet connection is configured to help ensure there’s at least one connection to the internet up at all times.
In this post, we’ll consider the topology for a network with a redundant internet connection that uses Border Gateway Protocol (BGP) routing. Here, BGP redunancy design will do two things: learn the internet routing table, and announce the local, publicly accessible network to the internet.
To keep things as simple as possible for those of you who’ve never worked with BGP redundancy before, we’ll do this with a single internet router.
But before we look at the design, let’s think through 4 important points that are relevant anytime you’re dealing with a redundant internet connection.
1. Think about the physical cable path for the second connection.
If you can, run the secondary internet connection into the building at a different location and route it to your data center through a different path than the existing connection. This ensures that a cable cut won’t take out both links at the same time.
Also ask your local telco if they can route the secondary connection to a different Central Office using a different fiber path in town. Although the answer might be no, it’s worth asking the question.
2. Avoid getting both of your connections from the same ISP if you can help it.
If your ISP is having a bad day, you don’t want that issue to affect both your connections. Ideally you want internet service from two completely different carriers. One carrier having a problem shouldn’t affect the other if they’re truly different carriers with unique infrastructure.
3. Consider whether you need provider independent (PI) IP address space.
Most companies with a single internet connection will use a block of IP addresses assigned to them by their ISP. It’s possible to ask your regional registry for a PI block of IP addresses, using dual-homing as a justification, but my advice is not to bother. Some markets are completely out of IPv4 addresses. As of this writing, the North American market governed by ARIN has only 933 /24 IP blocks left. (Check here to see the current state of ARIN IPv4 availability.) Besides, you can announce a /24 or larger block assigned to you by your ISP via some other ISP.
4. Get a BGP autonomous system number (ASN) that identifies your network.
BGP redundancy design sees an ASN as a collection of routers grouped together. When you announce a network into the global internet BGP routing table, your originating ASN is included. ARIN can assign you an ASN for a nominal cost.
Note that if you’re going to get a secondary internet connection from the same carrier as your primary connection, you won’t need to use a “real” ASN to announce your routers to your carrier. You’ll be able to use whatever private ASN you like, as defined in RFC 6996, from the reserved range of 64512 – 65534 for 16-bit ASNs or 4200000000 – 4294967294 for 32-bit ASNs.
With that preliminary discussion done, let’s move into setting up the BGP feeds and secondary connection.
Dual ISP to single router with BGP
We’ll start with a reference diagram. In this lab setup, an internal network is separated from the internet by a firewall. The firewall connects to an internet edge router that’s been assigned autonomous system number 54321. (Don’t use ASN 54321, or any of the other AS numbers in this post. They all belong to real companies. I’ve just used them here for illustration purposes.)
The internet edge router has redundant connections. One connection goes to an ISP in AS 7018 who’s announcing networks starting with 12.12.x.x. The other connection goes to an ISP in AS 4323, who’s announcing networks starting with 102.102.x.x. And somewhere on the internet, we have another imaginary ISP using AS 12345, announcing the 1.23.45.1/32 host route.
If BGP peering with isp and autonomous systems are new to you, this might seem a bit confusing. Don’t routers communicate with other routers through a link with actual IP addresses assigned? Why bother showing AS numbers?
Well, remember the point I made earlier. From BGP’s perspective, yes, it knows about specific IP neighbors, which we’ll illustrate in a moment. But from a global perspective, BGP knows about AS numbers.
To understand how BGP redundancy makes a “best path” decision, a network administrator must visualize BGP relationships from an AS point of view. BGP sees traffic as flowing internally (between routers in the same AS) and externally (from one AS to another AS) but it doesn’t know the details of the routers in the other autonomous systems. That’s why I drew the reference diagram as I did. We’re focused today on external BGP.
Setting up external BGP neighbors: The absolute basics
Let’s take a look at the Cisco IOS code used on the internet edge router to connect to the two ISP connections. I’ll explain each command along the way.
INT-EDGE-RTR#show run | section router bgp ! Start the BGP process, and place the router in autonomous system 54321. router bgp 54321 ! When a BGP neighbor goes up or down, log that change. This is helpful in diagnosing BGP failover problems. bgp log-neighbor-changes ! BGP does not dynamically discover neighbors as is possible with interior gateway routing protocols like EIGRP or OSPF. Neighbors must be specifically defined. In addition, both BGP routers on either end of the connection must be explicitly configured for the connection to come up. It’s not enough for one side of the connection to have a neighbor statement. ! This neighbor statement tells the router to attempt to bring up a BGP session with a router using IP address 10.12.12.1 and located in AS 7018. neighbor 10.12.12.1 remote-as 7018 ! This neighbor statement tells the router to attempt to bring up a BGP session with a router using IP address 10.102.102.1 and located in AS 4323. neighbor 10.102.102.1 remote-as 4323 INT-EDGE-RTR#
Assuming the ISPs have configured their side of the BGP connection, the BGP routers will become peers. When this is complete, the BGP peer status should report an “established” status. Any status other than “established” means the BGP session is not up. The session might be on the way up, or it might be experiencing a problem.
! Here, we’re asking our internet edge router to show us our BGP neighbors, but to only include lines with the words “neighbor” or “state” in them. That reduces the output to just what we’re looking for — in this case, whether or not our neighbors are in an established state.
INT-EDGE-RTR#show ip bgp neighbors | include neighbor|state
BGP neighbor is 10.12.12.1, remote AS 7018, external link
BGP state = Established, up for 16:39:43
BGP table version 101, neighbor version 101/0
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
BGP neighbor is 10.102.102.1, remote AS 4323, external link
BGP state = Established, up for 16:25:40
BGP table version 101, neighbor version 101/0
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
INT-EDGE-RTR#
Another quick check of the status of BGP neighbors is the “show ip bgp summary” command. Near the bottom of the output of this command, the status of the BGP neighbors is reported.
INT-EDGE-RTR#show ip bgp summary ... output skipped ... ! Here, we see that each neighbor has been up for over 16 hours. Up (established) is implied because there’s a number of prefixes received — 12. If the BGP neighbor was down, the connection state would be reported instead of number of prefixes received. Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.12.12.1 4 7018 2224 2202 101 0 0 16:43:27 12 10.102.102.1 4 4323 2198 2167 101 0 0 16:29:24 12 INT-EDGE-RTR# ! Now, let’s compare output if one of the BGP neighbors is down. INT-EDGE-RTR#show ip bgp summary ... output skipped ... Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.12.12.1 4 7018 2234 2213 111 0 0 16:47:31 21 10.102.102.1 4 4323 0 0 1 0 0 00:00:12 Idle INT-EDGE-RTR#
Examining BGP routes and attributes
What do routes in the routing table look like when they’re learned from BGP? They look like any other route, but they’re designated with a “B” to indicate they were learned from the BGP routing process.
What follows is an edited output of the “show ip route” command, so you can see what the BGP routes look like to the internet edge router once they’ve learned our imaginary internet routes. Notice that routes starting with 12.12 come from the BGP neighbor in AS 7018, while the routes stating with 102.102 come from the BGP neighbor in AS 4323.
INT-EDGE-RTR# show ip route
... output skipped ...
1.0.0.0/32 is subnetted, 1 subnets
B 1.23.45.1 [20/0] via 10.12.12.1, 16:54:11
12.0.0.0/24 is subnetted, 8 subnets
B 12.12.1.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.2.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.3.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.4.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.5.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.6.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.7.0 [20/0] via 10.12.12.1, 16:58:22
B 12.12.8.0 [20/0] via 10.12.12.1, 16:58:22
102.0.0.0/24 is subnetted, 8 subnets
B 102.102.1.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.2.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.3.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.4.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.5.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.6.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.7.0 [20/0] via 10.102.102.1, 00:02:48
B 102.102.8.0 [20/0] via 10.102.102.1, 00:02:48
INT-EDGE-RTR#
The information exchanged between BGP neighbors is more than just IP prefix information, though. In addition to prefixes (what you’re probably thinking of as routes), a number of BGP attributes about that prefix are carried.
For our purposes today, the most interesting one is the AS path. To see these BGP attributes, perform a “show ip bgp.” To see BGP attributes for one specific prefix, perform a “show ip bgp [prefix],” which is what we’ll do in the example below.
INT-EDGE-RTR#show ip bgp 12.12.1.0 BGP routing table entry for 12.12.1.0/24, version 50 ! There are two paths by which 12.12.1.0/24 is reachable, according to BGP. In this output, path #2 is the best path. Why? Path #2 is the fewest number of AS hops away. Paths: (2 available, best #2, table default) Advertised to update-groups: 2 Refresh Epoch 1 ! There are three autonomous systems to traverse if this path is followed: 4323, 12345, and 7018. 4323 12345 7018 10.102.102.1 from 10.102.102.1 (102.102.1.1) Origin IGP, localpref 100, valid, external rx pathid: 0, tx pathid: 0 Refresh Epoch 1 ! There’s only one autonomous system to traverse if this path is followed: 7018. Therefore, this is chosen as the better path. The other path can be used if this path is no longer available to reach 12.12.1.0/24. 7018 10.12.12.1 from 10.12.12.1 (12.12.1.1) Origin IGP, localpref 100, valid, external, best rx pathid: 0, tx pathid: 0x0 INT-EDGE-RTR#
At this point, we’ve got an internet edge router with two connections to the internet through separate ISPs. The internet edge router is learning the internet routing table, which we’ve simulated here with just a few routes from the 12.x and 102.x network so we can get a feel for what the router looks like.
Your Guide to Selling Managed Network Services
Get templates for network assessment reports, presentations, pricing & more—designed just for MSPs.
Note that if you accept the entire routing table from both of your ISPs, you’ll need a good deal of RAM in your router to handle the load. The current IPv4 internet routing table is approximately 509,000 IPv4 routes (and 20,000 IPv6 routes, if you’re running IPv6). When you load more than one full BGP feed, router memory requirements increase proportionately. But for what it’s worth, I’ve fed dual-internet, full BGP feeds into a Cisco 2911 with 2.5GB of RAM, and had plenty of memory headroom.
Announcing routes to the public internet
So far, we’ve handled getting out from our site to the internet. But what about the internet getting back to our site? As with any routing protocol, we need to announce to BGP the public networks we’re responsible for. And, as with many routing protocols, this is accomplished with a network statement.
Network statements by themselves are not magical. Your router must actually know how to deliver traffic for the route listed in the network statement, or else the router’s BGP process won’t announce the route. The route can be known because it’s directly connected, it was learned through a static route, or it was learned through some other routing protocol.
Here’s a second look at our “router bgp” paragraph, this time with the “network” statement added.
INT-EDGE-RTR#show run | section router bgp router bgp 54321 bgp log-neighbor-changes network 5.43.21.1 mask 255.255.255.255 neighbor 10.12.12.1 remote-as 7018 neighbor 10.102.102.1 remote-as 4323 INT-EDGE-RTR#
While BGP can be used to announce a host route, like in our example here, understand that internet backbone routers won’t accept networks into their BGP tables smaller than a /24. Why? Because of the size of the global internet routing table. At over 500,000 routes, it’s already huge! Keep this limitation in mind as you create your internet redundancy network design.
Don’t become a transit BGP router!
To verify which routes are being advertised to a BGP neighbor, we can use the “show ip bgp neighbors advertised-routes” command. Take a close look at the edited list of routes we’re announcing below. We have a problem that’s important to understand and fix.
INT-EDGE-RTR#show ip bgp neighbors 10.12.12.1 advertised-routes
... output skipped ...
Network Next Hop Metric LocPrf Weight Path
*> 1.23.45.1/32 10.12.12.1 0 7018 12345 i
*> 5.43.21.1/32 0.0.0.0 0 32768 i
*> 12.12.1.0/24 10.12.12.1 0 7018 i
*> 12.12.2.0/24 10.12.12.1 0 7018 i
*> 12.12.3.0/24 10.12.12.1 0 7018 i
*> 12.12.4.0/24 10.12.12.1 0 7018 i
*> 12.12.5.0/24 10.12.12.1 0 7018 i
*> 12.12.6.0/24 10.12.12.1 0 7018 i
*> 12.12.7.0/24 10.12.12.1 0 7018 i
*> 12.12.8.0/24 10.12.12.1 0 7018 i
*> 102.102.1.0/24 10.102.102.1 0 4323 i
*> 102.102.2.0/24 10.102.102.1 0 4323 i
*> 102.102.3.0/24 10.102.102.1 0 4323 i
*> 102.102.4.0/24 10.102.102.1 0 4323 i
*> 102.102.5.0/24 10.102.102.1 0 4323 i
*> 102.102.6.0/24 10.102.102.1 0 4323 i
*> 102.102.7.0/24 10.102.102.1 0 4323 i
*> 102.102.8.0/24 10.102.102.1 0 4323 i
The issue here is that we should be announcing exactly one route to our BGP peers. That route is the single one we’re responsible for: 5.43.21.1/32 in our example. But we can see that we’re actually advertising all the BGP routes we’ve learned back to our peers. Whoops!
Advertising all the BGP routes would be appropriate if your internet edge router was a transit internet router, acting as a peering point between your two ISPs. But you don’t want to be a transit router. Keep your internet pipes for yourself and your organization, not as a pathway for any old traffic that might be traveling between the two ISPs.
Your ISPs don’t want you acting as a transit either, as it limits their ability to control their internet traffic. In fact, your ISP will filter the BGP advertisements you send them to strictly enforce their network design, but you should also be filtering what you send them.
The easiest way to filter is to create a prefix list with the routes you wish to announce, and apply that prefix list to your BGP neighbor statement.
! Create a prefix-list that contains only the routes we should be announcing into BGP. INT-EDGE-RTR#show run | section ip prefix ip prefix-list ANNOUNCE-ONLY-THESE seq 5 permit 5.43.21.1/32 ! Apply the prefix-list as a filter to the outbound announcements to our BGP neighbors. You can set up different filters for different neighbors. In this case, we’re using the same filter on both. Note that when applying these filters, you might need to “clear ip bgp neighbor ” to cause the filter to take effect. It depends on your version of IOS. Even if you don’t need to clear the BGP session, the filter might take a minute or two to take effect. In other words, this isn’t an instant change, and might actually be disruptive when applying or updating the filter. INT-EDGE-RTR#show run | section router bgp router bgp 54321 bgp log-neighbor-changes network 5.43.21.1 mask 255.255.255.255 neighbor 10.12.12.1 remote-as 7018 neighbor 10.12.12.1 prefix-list ANNOUNCE-ONLY-THESE out neighbor 10.102.102.1 remote-as 4323 neighbor 10.102.102.1 prefix-list ANNOUNCE-ONLY-THESE out INT-EDGE-RTR#
Now that the filter has been applied (and we’ve waited a couple of minutes), we see only the route we should be announcing being advertised.
INT-EDGE-RTR#show ip bgp neighbors 10.12.12.1 advertised-routes
... output skipped ...
Network Next Hop Metric LocPrf Weight Path
*> 5.43.21.1/32 0.0.0.0 0 32768 i
Total number of prefixes 1
INT-EDGE-RTR#
Good news! The internet edge router is no longer trying to be a transit BGP router. And with that step completed, we now have a working redundant Internet connection with BGP configured.
If you’re interested in further reading on this topic, I’ve found these resources helpful.
Hi Ethan,
If I connect to 2 ISP with bgp and have full internet routing table from both ISP. It is possible to influence so my Internal-NerworkA go to internet with ISPA and internal-NetworkB go to internet with ISPBA. Can you give me a keyword what bgp parameter to achieve this.
Thx
What you are trying to do would require BGP static-route from Network A to ISPA and vice versa for Network B. If you wanted to load balance I would not do it at the BGP level , assuming you are using Routers for Net A and Net B. I would configure a router farm that using EIGRP you load balance in EIGRP . I would avoid trying to load balance in BGP, main reason is its a headache you do not need. Once the Load balancing is accomplished in EIGRP then NAT to your Public IP running BGP. If you load balance or distribute at the IGP level then EBGP can work on managing each route and IBGP link between BGP routers gives reliability and resiliency to the ISP routes.
hello Thanks for the explanations, i have one question which is have one block /24 public IP and i am now connected to up streaming company and block advertised on their router, but i want also to connect another carrier from sattelite for redudency but i still have the /24 block which already advertised on the other side, so if i also give that block to the next up stream it will conflict. so i need a way that i can connect two router and use only one pulibc ip block.
Thanks very much, Ethan for this great writing on BGP, is a good refresher for me.
Kind Regards
Ben