Configuring BGP Peering between pfSense and Cisco.
How-To
January 11, 2023
Overview
Border Gateway protocol is the open standard routing protocol used over Internet. It can also be used for internal networks. This is an example of a BGP peering between two different vendors. This information might also apply, to some degree, between other vendors.
pfSense is an Open-Source firewall/network appliance that can either run as a VM for free, or you can buy an appliance box from Netgate. The software is based on FreeBSD.
Cisco is one of the largest players within network infrastructure. They need no introduction.
Reasons to run BGP on the Firewall edge
Some people might argue that static routing is the simplest way to configure routing between a firewall and the core network. While the initial configuration might be simple, it’s not very flexible. Also, managing dozens of routes can be very difficult (I can tell from experience).
BGP is certainly not the easiest way to do it, but it’s extremely flexible. If you know BGP, then it’s probably easier than static routing, and you have full control of your routing table.
Redundancy might also be a consideration to run BGP. With static routing you would usually have to run a router redundancy protocol like HSRP or VRRP, but those are prone to Layer 2 issues that can take down the whole network (also learnt from experience).
Assumptions
These things have to be in order before BGP configuration can take place.
Interface configuration is already in place.
You have basic knowledge about Border Gateway Protocol.
The Topology
BGP Peerings:
Explanations
As you can see, due to the fact that the firewall and routers are in different autonomus systems, there is eBGP between the firewall (icon with the pause symbol) and the routers.
The firewall is a pfSense SG-1100 appliance.
All the router icons represents the same physical Cisco1111 ISR router. The icons in different colors represent different VRFs = Virtual Routing instances.
The firewall is the gateway between the VRFs.
The firewalls interfaces are in AS65000
The routers interfaces are in AS65001.
I’m running dual-stack and using IPv4 as the carrier of the IPv6 address-family. The IPv4 linknet IPs will be the neighbor IDs. This will be explained more in detail later.
Configuring BGP
The Goal
Every VRF on the router should only have a default route in it’s BGP table.
The firewall should only have the superscopes for every network, no specific subnets.
IPv4 BGP neighbors will route traffic for both IPv4 and IPv6.
pfSense Configuration
Step 1: Download FRR package
pfSense doesn’t have BGP support out of the box. You need to download the FRR Routing (Free Range Routing) package from the package manager:
Go to System > Package Manager > Available Packages and search for frr.
once installed it should be visible under Installed Packages.
Step 2: Activate FRR and FRR Global Settings
The BGP daemon won’t start before you activate the FRR Routing first.
Go to Services > FRR Zebra/Global
Check Enable FRR
Enter a Default Router ID and Master Password
Also, add the default routes and the IPv6 prefix:
Step 3: Configure a Route-Map
FRR is following the RFC 8212 standard, which require a policy to be in place before allowing routes to be imported or exported. Since I’m controlling the importing and exporting of routes, a Simple Route-Map with a “Permit Any” statement will do just fine.
Go to the Route Maps tab and create a new Route-Map that looks like this, it will work for both IPv4 and IPv6:
Notes:
Cisco has not implemented this standard on the IOS-XE platform, but has on the IOS-XR platform.
If you don’t have control over both sides of the connection, a filter with only the expected prefixes is recommended.
You could alternatively disable this requirement under [BGP] > Advanced and scroll down to the eBGP section and check Disable eBGP Require Policy.
Step 4: Configure BGP Global Settings
Head on over to the [BGP] tab and you will land on the Global Settings tab.
Check Enable BGP
Check Log Adjacency Changes (not required but recommended)
Enter Local AS, Router ID and Timers
Notes:
The Default timers of BGP is Keep Alive: 60, Hold Time: 180. That is a bit to long for my preference, but I don’t want to flood the appliances with BGP updates either.
FRR BGP also supports BFD = Bidirectional Forwarding Detection. With that you can configure neighbor establishment down to milliseconds, if convergence is critical. Note that BFD won’t flood updates any faster. It’s only used for fault detection.
You also need to define the networks that are going to be advertised to neighbors:
Note: the 2001:DB8:1234::/48 prefix is the provider assigned IPv6 scope.
Step 5: Configure BGP Neighbors
Go to the Neighbors tab
Under General Options, Enter:
Neighbor IPv4 address.
Description (optional but recommended).
Note: Neighbor IP could be the IPv6 Address as well, but then you would use IPv6 to transfer updates for IPv4 prefixes.
Under the Basic Options, Enter:
Remote AS
Update Source Ipv4
Check Allow neighbor to advertise and receive routes for both IPv4 and IPv6
Check Per-neighbor Inbound Soft Reconfiguration (optional but recommended).
Under Peer Filtering, select the Route-Map configured earlier for both inbound and outbound Route Map Filters.
Just repeat above steps for all your neighbors.
That’s it! No additional configuration is required, on the pfSense side at least.
Note: FRR BGP supports all kinds of complex configuration that you can find on any other high-end router, like Communities or AS-Path filters.
Configure the Cisco Router
Short explanation of VRF = Virtual Routing and forwarding
The VRF configuration won’t be covered here. If it’s a new concept for you, just imagine that the router have multiple routing tables that don’t mix, unless you want them to. It’s much more convenient than setting up 6 physical routers.
pfSense can be configured in a similar fashion, but it’s not necessary for this purpose.
Step 1: Configure Route-Maps
A Cisco IOS or IOS-XE router doesn’t have the policy requirement of RFC 8212. However, it has another requirement:
If you are using IPv4 to transport updates about IPv6 prefixes, the IPv6 next-hop of the router has to be advertised, otherwise the Cisco router will advertise the IPv4 address as the next-hop for the IPv6 prefixes.
route-map GLOBAL permit 100
set ipv6 next-hop 2001:DB8:1234:E::2
route-map MGMT permit 100
set ipv6 next-hop 2001:DB8:1234:A::2
route-map INTERNET permit 100
set ipv6 next-hop 2001:DB8:1234:9::2
route-map USERS permit 100
set ipv6 next-hop 2001:DB8:1234:C::2
route-map GUEST permit 100
set ipv6 next-hop 2001:DB8:1234:D::2
route-map SERVERS permit 100
set ipv6 next-hop 2001:DB8:1234:B::2Note: The ipv6 next-hop addresses is the ones of the routers interfaces.
Step 2: Configure BGP
A brief explanation of the commands:
Aggregate Address … Summary-Only is used to summarize the network scopes and hide specific routes.
As you can see, IPv4 neighbor commands are also used within the IPv6 Address-family. One extra line because of the route-map requirement
The network and redistribute commands are not a part of this scope, but in essence, the routes have to come from somewhere.
router bgp 65001
bgp router-id 10.1.1.1
bgp log-neighbor-changes
neighbor 10.14.255.0 remote-as 65000
neighbor 10.14.255.0 description GLOBAL
!
address-family ipv4
network 10.1.1.1 mask 255.255.255.255
aggregate-address 10.1.0.0 255.255.0.0 summary-only
neighbor 10.14.255.0 activate
exit-address-family
!
address-family ipv6
aggregate-address FD00::/8 summary-only
neighbor 10.14.255.0 activate
neighbor 10.14.255.0 route-map GLOBAL out
exit-address-family
!
address-family ipv4 vrf GUEST
aggregate-address 10.13.0.0 255.255.0.0 summary-only
redistribute ospfv3 1
neighbor 10.13.255.0 remote-as 65000
neighbor 10.13.255.0 activate
exit-address-family
!
address-family ipv6 vrf GUEST
redistribute ospf 1
aggregate-address 2001:DB8:1234:D000::/52 summary-only
neighbor 10.13.255.0 remote-as 65000
neighbor 10.13.255.0 activate
neighbor 10.13.255.0 route-map GUEST out
exit-address-family
!
address-family ipv4 vrf INTERNET
aggregate-address 10.9.0.0 255.255.0.0 summary-only
redistribute ospfv3 1
neighbor 10.9.255.0 remote-as 65000
neighbor 10.9.255.0 activate
exit-address-family
!
address-family ipv6 vrf INTERNET
redistribute connected
redistribute ospf 1
aggregate-address 2001:DB8:1234:9000::/52 summary-only
neighbor 10.9.255.0 remote-as 65000
neighbor 10.9.255.0 activate
neighbor 10.9.255.0 route-map INTERNET out
exit-address-family
!
address-family ipv4 vrf MGMT
aggregate-address 10.0.0.0 255.255.0.0 summary-only
redistribute ospfv3 1
neighbor 10.0.255.0 remote-as 65000
neighbor 10.0.255.0 activate
exit-address-family
!
address-family ipv6 vrf MGMT
redistribute ospf 1
aggregate-address FC00::/8 summary-only
aggregate-address 2001:DB8:1234:A000::/52 summary-only
neighbor 10.0.255.0 remote-as 65000
neighbor 10.0.255.0 activate
neighbor 10.0.255.0 route-map MGMT out
exit-address-family
!
address-family ipv4 vrf SERVERS
aggregate-address 10.11.0.0 255.255.0.0 summary-only
redistribute static
redistribute ospfv3 1
neighbor 10.11.255.0 remote-as 65000
neighbor 10.11.255.0 activate
exit-address-family
!
address-family ipv6 vrf SERVERS
redistribute ospf 1
aggregate-address 2001:DB8:1234:B000::/52 summary-only
neighbor 10.11.255.0 remote-as 65000
neighbor 10.11.255.0 activate
neighbor 10.11.255.0 route-map SERVERS out
exit-address-family
!
address-family ipv4 vrf USERS
aggregate-address 10.12.0.0 255.255.0.0 summary-only
redistribute ospfv3 1
neighbor 10.12.255.0 remote-as 65000
neighbor 10.12.255.0 activate
exit-address-family
!
address-family ipv6 vrf USERS
redistribute ospf 1
aggregate-address 2001:DB8:1234:C000::/52 summary-only
neighbor 10.12.255.0 remote-as 65000
neighbor 10.12.255.0 activate
neighbor 10.12.255.0 route-map USERS out
exit-address-familyThat’s it. We’re done.
Verification
PfSense Verification
Go to Status > FRR > BGP. As we can see, All neighbors are up and only the summarized routes are received.
/usr/local/lib/libfrr.so.0: Unable to relocate undefined weak TLS variable
BGP table version is 19, local router ID is 10.0.1.3, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0/0 0.0.0.0 0 32768 i
*> 10.0.0.0/16 10.0.255.1 0 0 65001 i
*> 10.1.0.0/16 10.14.255.1 0 0 65001 i
*> 10.9.0.0/16 10.9.255.1 0 0 65001 i
*> 10.11.0.0/16 10.11.255.1 0 0 65001 i
*> 10.12.0.0/16 10.12.255.1 0 0 65001 i
*> 10.13.0.0/16 10.13.255.1 0 0 65001 i
Displayed 7 routes and 7 total paths
/usr/local/lib/libfrr.so.0: Unable to relocate undefined weak TLS variable
BGP table version is 27, local router ID is 10.0.1.3, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> ::/0 :: 0 32768 i
*> 2001:db8:1234::/48
:: 0 32768 i
*> 2001:db8:1234:1::/64
fe80::9:2 0 0 65001 ?
*> 2001:db8:1234:9::/64
fe80::9:2 0 0 65001 ?
*> 2001:db8:1234:9000::/52
fe80::9:2 0 0 65001 i
*> 2001:db8:1234:a000::/52
fe80::a:2 0 0 65001 i
*> 2001:db8:1234:b000::/52
fe80::b:2 0 0 65001 i
*> 2001:db8:1234:c000::/52
fe80::c:2 0 0 65001 i
*> 2001:db8:1234:d000::/52
fe80::d:2 0 0 65001 i
*> fc00::/8 fe80::a:2 0 0 65001 i
*> fd00::/8 fe80::e:2 0 0 65001 i
/usr/local/lib/libfrr.so.0: Unable to relocate undefined weak TLS variable
IPv4 Unicast Summary:
BGP router identifier 10.0.1.3, local AS number 65000 vrf-id 0
BGP table version 19
RIB entries 12, using 2304 bytes of memory
Peers 6, using 86 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.0.255.1 4 65001 44331 40995 0 0 0 01w2d11h 1 7
10.9.255.1 4 65001 44339 40995 0 0 0 01w2d11h 1 7
10.11.255.1 4 65001 44316 40995 0 0 0 01w2d11h 1 7
10.12.255.1 4 65001 44314 40995 0 0 0 01w2d11h 1 7
10.13.255.1 4 65001 44314 40995 0 0 0 01w2d11h 1 7
10.14.255.1 4 65001 44315 40995 0 0 0 01w2d11h 1 7
Total number of neighbors 6
IPv6 Unicast Summary:
BGP router identifier 10.0.1.3, local AS number 65000 vrf-id 0
BGP table version 27
RIB entries 17, using 3264 bytes of memory
Peers 6, using 86 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.0.255.1 4 65001 44331 40995 0 0 0 01w2d11h 2 11
10.9.255.1 4 65001 44339 40995 0 0 0 01w2d11h 3 11
10.11.255.1 4 65001 44316 40995 0 0 0 01w2d11h 1 11
10.12.255.1 4 65001 44314 40995 0 0 0 01w2d11h 1 11
10.13.255.1 4 65001 44314 40995 0 0 0 01w2d11h 1 11
10.14.255.1 4 65001 44315 40995 0 0 0 01w2d11h 1 11Cisco Verification
All neighbors are up and only 1 prefix received:
WIL-RO1#show bgp ipv4 unicast summary
BGP router identifier 10.1.1.1, local AS number 65001
BGP table version is 13, main routing table version 13
3 network entries using 744 bytes of memory
3 path entries using 408 bytes of memory
3/3 BGP path/bestpath attribute entries using 864 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
15 BGP extended community entries using 520 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2560 total bytes of memory
BGP activity 261/206 prefixes, 276/221 paths, scan interval 60 secs
3 networks peaked at 21:03:24 Dec 30 2022 CET (1w2d ago)
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.14.255.0 4 65000 41025 44347 13 0 0 1w2d 1
WIL-RO1#show bgp vpnv4 unicast all summary
...
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.255.0 4 65000 41019 44357 266 0 0 1w2d 1
10.9.255.0 4 65000 41019 44365 266 0 0 1w2d 1
10.11.255.0 4 65000 41019 44342 266 0 0 1w2d 1
10.12.255.0 4 65000 41019 44340 266 0 0 1w2d 1
10.13.255.0 4 65000 41019 44340 266 0 0 1w2d 1IPv6:
WIL-RO1#show bgp ipv6 unicast summary
...
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.14.255.0 4 65000 41025 44347 22 0 0 1w2d 2
WIL-RO1#show bgp vpnv6 unicast all summary
...
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.255.0 4 65000 41033 44372 304 0 0 1w2d 2
10.9.255.0 4 65000 41033 44379 304 0 0 1w2d 2
10.11.255.0 4 65000 41033 44357 304 0 0 1w2d 2
10.12.255.0 4 65000 41033 44354 304 0 0 1w2d 2
10.13.255.0 4 65000 41033 44355 304 0 0 1w2d 2The BGP tables for VRF MGMT and USERS are listed below. The only routes learnt from BGP is the default routes and the Provider Assigned prefix for IPv6.
IPv4:
WIL-RO1#show bgp vpnv4 unicast vrf USERS
...
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:2 (default for vrf USERS)
*> 0.0.0.0 10.12.255.0 0 0 65000 i
*> 10.12.0.0/16 0.0.0.0 32768 i
s> 10.12.1.0/25 10.12.255.3 2 32768 ?
s> 10.12.1.128/25 10.12.255.3 2 32768 ?
s> 10.12.255.2/31 0.0.0.0 0 32768 ?
WIL-RO1#show bgp vpnv4 unicast vrf MGMT
...
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:10 (default for vrf MGMT)
*> 0.0.0.0 10.0.255.0 0 0 65000 i
*> 10.0.0.0/16 0.0.0.0 32768 i
s> 10.0.1.0/24 10.0.255.3 2 32768 ?
s> 10.0.255.2/31 0.0.0.0 0 32768 ?Note: “s” means suppressed, and won’t be advertised to peers. those routes also have a weight of 32768 assigned to them, which indicates that those routes are learnt from some other protocol than BGP.
IPv6:
WIL-RO1#show bgp vpnv6 unicast vrf USERS | exclude s>
...
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:2 (default for vrf USERS)
*> ::/0 2001:DB8:1234:C::1
0 0 65000 i
*> 2001:DB8:1234::/48
2001:DB8:1234:C::1
0 0 65000 i
*> 2001:DB8:1234:C000::/52
:: 32768 i
WIL-RO1#show bgp vpnv6 unicast vrf MGMT | exclude s>
...
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:10 (default for vrf MGMT)
*> ::/0 2001:DB8:1234:A::1
0 0 65000 i
*> 2001:DB8:1234::/48
2001:DB8:1234:A::1
0 0 65000 i
*> 2001:DB8:1234:A000::/52
:: 32768 i
*> FC00::/8 :: 32768 i
Note: Specific routes from other VRFs would probably not be installed in any case, since they come from the same AS number.
End-to-End Reachability
And ofcourse, we need to test reachability from one zone to another. This is a ping from a default GW VRF USERS, to a default GW in VRF MGMT:
WIL-SW1#ping vrf USERS 10.0.1.1 source vlan 121
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.12.1.129
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms
WIL-SW1#ping vrf USERS 2001:DB8:1234:A001::1 source vlan 121
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A001::1, timeout is 2 seconds:
Packet sent with a source address of 2001:DB8:1234:C011::1%USERS
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/3/4 msSpecial Mention
This post is dedicated to Herman Hermansen and his IPv6 course. Without that IPv6 next-hop detail I learned from you, I would still be puzzled why the IPv6 prefixes wouldn’t work. You are the one colleague I miss the most from my former employer.