Limitations of NetFlow on a 6500 in HYBRID mode w/Sup 720
Ok, I'm trying to setup NFA to work on a Catalyst 6500/Sup720 running in Hybrid mode. I've skimmed these forums, and it seems to be a default to provide links to configure NetFlow for Native mode, which doesn't help me any.
I've given up on configurating NetFlow on the MSFC, because all I see is inbound packets. This is inline with how MLS on the MSFC in hybrid mode works, so I don't expect to see output traffic, so I relegated to trying to get Netflow and NDE working properly on the Supervisor.
Here's my show mls output from the Supervisor:
Total packets switched = 21659698097
Total bytes switched = 5040698813473
Total routes = 212710
Total number of Netflow entries = 261795
IP statistics flows aging time = 256 seconds
Long-duration flows aging time = 320 seconds
IP statistics flows fast aging time = 0 seconds, packet threshold = 0
IP Current flow mask for NDE is Full flow
Netflow Per Interface disabled
Netflow Data Export version: 7
Netflow Data Export enabled
Netflow Data Export configured for port 9996 on host x.x.x.x
Secondary Netflow Data Export port/host is not configured.
Total packets exported = 168846
Total packets exported to secondary = 0
Total NDE collectors configured = 1
Destination Ifindex export is enabled
Source Ifindex export is enabled
Netflow Per Vlan is enabled on vlan(s) 2,13,62,70,80,100-102,110-111,120-121,130-134,200-203,300,400,500-502,600-603,666,700-703,800-801,810-817.
Netflow Per Vlan is disabled on vlan(s) 1
Cannot get netflow per vlan status information of vlan(s) 1002, 1005
Rate limiting is turned off, packets are bridged to router
Load balancing hash is based on source and destination IP addresses and universal id 95c8db9b
Per-prefix Stats for ALL FIB entries is Enabled
So as you can see, I'm trying to configure per vlan netflow on the Supervisor. So when I go and check NFA, I see traffic for various ifindexes, but the ifindexes seem to mostly correspond to ports/interfaces, not VLANs as I'm expecting (see attached image) even though I've disabled per interface netflow. The port/interface ifindexes are double digits, the vlan ifindexes are triple digits. Let's take the one the ONE vlan ifindex NFA thinks it sees: 306. What does the Supervisor think of that ifindex:
switch> (enable) show vlan trunk | incl 306
switch> (enable)
Hmm..
So here's some nde debugging from the Supervisor:
###### Flow Mask ####
Dest - 1 , DestSrc - 2, Full - 3
ROUTER1
IP - 0 MPLS - 0 L2 - 0
ROUTER2
IP - 0 MPLS - 0 L2 - 0
CLI
IP - 0 MPLS - 0 L2 - 0
QOS
IP - 0 MPLS - 0 L2 - 0
IP Current flow mask for NDE is Full flow
Flowmask saved in Nvram is Full flow
###### NDE ######
NDE related info:
NDE in runtime : TRUE
NDE in nvram : TRUE
Current Export Version : 7
Flows in nde buffer : 10
Nde flow limit : 27
Flow sequence : 5298985
Unused flows : 49134
Non Ip Sc : 0
Filter mismatch : 0
Collector IP : x.x.x.x UDP Port : 9996
Packets sent : 181062
Secondary Collector IP : x.x.x.x UDP Port : 0
Packets to Secondary : 0
#########Pattern Aging########
Pattern 1 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 1, Ntbl Ptr = 0
Pattern 2 :
====================
Aging Timeout = 10, Stats Timeout = 0, Attribute = 1, Ntbl Ptr = 0
Pattern 3 :
====================
Aging Timeout = 300, Stats Timeout = 100, Attribute = 1, Ntbl Ptr = 262143
Pattern 4 :
====================
Aging Timeout = 300, Stats Timeout = 1333, Attribute = 1, Ntbl Ptr = 262143
Pattern 5 :
====================
Aging Timeout = 300, Stats Timeout = 0, Attribute = 3, Ntbl Ptr = 0
Pattern 6 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 1, Ntbl Ptr = 0
Pattern 7 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 8 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 9 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 10 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 11 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 12 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 13 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 14 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 15 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
######## Msg Stats #########
# of IP Range Purge msgs = 20427268
# of IPX Range Purge msgs = 0
# of Add sw shortcut msgs = 0
# of Del sw shortcut msgs = 0
# of fm update to router = 0
# of rx swscAgeTime msgs = 0
# of swscStats msgs = 0
# of ip intf purge msgs = 64
# of next hop purge msgs = 17393
# of FTEP Adj req msg = 4
MSFC IP Address = x.x.x.x
# of Max Hop msgs = 0
# of IP flowmask req msgs = 8 # of IPX flowmask req msgs = 0
# of flowmask updates = 0 # of response msgs = 6
# of NDD IP Add msgs = 0 # of NDD IP Del msgs = 0
# of global IP en = 1 dis = 0
# of global IPX en = 0 dis = 0
# of swsc installed = 0
FIB MSG STATS
=============
#IP Add = 27196489
#IP2Tag Add = 0
#IP Delete = 4117134
#IP Init = 398
#IP Add Default = 2
#IP Delete Default = 4
#IP Addr Change = 0
#IP Standby Addr change = 0
#ADJ Add = 1784642
#ADJ Del = 12176
#IPX Add = 0
#IPX Del = 0
#ADJ Drop = 0
#FIB IP Reload Req = 3
#FIB IPX Reload Req = 0
#FIB IDB Reload Req = 2
#FIB IP Reload Complete = 4
#FIB IPX Reload Complete = 0
#RPF Config = 0
#of fibidb init2 msgs = 89
#RPF Mpath Mode = 1
#Load Sharing Mode = 5
#Tunnel Update = 0
#Tunnel Delete = 0
#Tunnel Feature = 0
#Rate Limit Set = 69
#Rate Limit Rsv = 0
#FIB Version RP Req = 1
#FIB Version SP Ack = 1
#FIB Version RP Ack = 1
#FIB VRF description = 0
#FIB Reload Failed = 0
#Invalid Messages = 0
#Invalid Length = 0
#Invalid Mask = 0
#FIB Version SP Ack Fail = 0
API Stats
==============================
ClearEarlL3EntriesByFlow = 78614762
ClearEarlL3AllEntries = 2
GetEarlL3EntriesByFlow = 950025
CreateEarlL3Entry = 0
MLS HA
===========================
# global sync = 1
FM update = 0
prot Filt sync = 8
IPX Max Hop sync = 0
L3 Age = 0
Fib Sequence Num sync = 5712630
Router Mac sync = 0
swsc Age Time sync = 0
# swsc synced = 0
# purge all fib entries = 1
# wanIfIndices synced = 0
# of ARP sent = 0
HAstdbySyncSeqNum = -1
ERROR counters
===============================
# of IP length errors = 0
# of IP too short = 0
# of IP chksum errors = 0
# of DBUS cksum errors = 0
# of DBUS len errors = 0
# of IB len errors = 0
# of CPU parity errors = 0
# of ACL drop packets = 3556602
# of Netflow full errs = 16777215
So I suppose my questions are:
1) Is vlan based netflow supported? If so, why does NFA not seem to get it?
2) Am I right that collecting netflow stats from the MSFC in hybrid mode is useless for the most part? When would this be useful?
3) The ACL drop packets and Netflow full error counters from the debug above. Do they have any relevance as to why this doesn't seem to be doing what it's supposed to?
I've given up on configurating NetFlow on the MSFC, because all I see is inbound packets. This is inline with how MLS on the MSFC in hybrid mode works, so I don't expect to see output traffic, so I relegated to trying to get Netflow and NDE working properly on the Supervisor.
Here's my show mls output from the Supervisor:
Total packets switched = 21659698097
Total bytes switched = 5040698813473
Total routes = 212710
Total number of Netflow entries = 261795
IP statistics flows aging time = 256 seconds
Long-duration flows aging time = 320 seconds
IP statistics flows fast aging time = 0 seconds, packet threshold = 0
IP Current flow mask for NDE is Full flow
Netflow Per Interface disabled
Netflow Data Export version: 7
Netflow Data Export enabled
Netflow Data Export configured for port 9996 on host x.x.x.x
Secondary Netflow Data Export port/host is not configured.
Total packets exported = 168846
Total packets exported to secondary = 0
Total NDE collectors configured = 1
Destination Ifindex export is enabled
Source Ifindex export is enabled
Netflow Per Vlan is enabled on vlan(s) 2,13,62,70,80,100-102,110-111,120-121,130-134,200-203,300,400,500-502,600-603,666,700-703,800-801,810-817.
Netflow Per Vlan is disabled on vlan(s) 1
Cannot get netflow per vlan status information of vlan(s) 1002, 1005
Rate limiting is turned off, packets are bridged to router
Load balancing hash is based on source and destination IP addresses and universal id 95c8db9b
Per-prefix Stats for ALL FIB entries is Enabled
So as you can see, I'm trying to configure per vlan netflow on the Supervisor. So when I go and check NFA, I see traffic for various ifindexes, but the ifindexes seem to mostly correspond to ports/interfaces, not VLANs as I'm expecting (see attached image) even though I've disabled per interface netflow. The port/interface ifindexes are double digits, the vlan ifindexes are triple digits. Let's take the one the ONE vlan ifindex NFA thinks it sees: 306. What does the Supervisor think of that ifindex:
switch> (enable) show vlan trunk | incl 306
switch> (enable)
Hmm..
So here's some nde debugging from the Supervisor:
###### Flow Mask ####
Dest - 1 , DestSrc - 2, Full - 3
ROUTER1
IP - 0 MPLS - 0 L2 - 0
ROUTER2
IP - 0 MPLS - 0 L2 - 0
CLI
IP - 0 MPLS - 0 L2 - 0
QOS
IP - 0 MPLS - 0 L2 - 0
IP Current flow mask for NDE is Full flow
Flowmask saved in Nvram is Full flow
###### NDE ######
NDE related info:
NDE in runtime : TRUE
NDE in nvram : TRUE
Current Export Version : 7
Flows in nde buffer : 10
Nde flow limit : 27
Flow sequence : 5298985
Unused flows : 49134
Non Ip Sc : 0
Filter mismatch : 0
Collector IP : x.x.x.x UDP Port : 9996
Packets sent : 181062
Secondary Collector IP : x.x.x.x UDP Port : 0
Packets to Secondary : 0
#########Pattern Aging########
Pattern 1 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 1, Ntbl Ptr = 0
Pattern 2 :
====================
Aging Timeout = 10, Stats Timeout = 0, Attribute = 1, Ntbl Ptr = 0
Pattern 3 :
====================
Aging Timeout = 300, Stats Timeout = 100, Attribute = 1, Ntbl Ptr = 262143
Pattern 4 :
====================
Aging Timeout = 300, Stats Timeout = 1333, Attribute = 1, Ntbl Ptr = 262143
Pattern 5 :
====================
Aging Timeout = 300, Stats Timeout = 0, Attribute = 3, Ntbl Ptr = 0
Pattern 6 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 1, Ntbl Ptr = 0
Pattern 7 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 8 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 9 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 10 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 11 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 12 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 13 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 14 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
Pattern 15 :
====================
Aging Timeout = 0, Stats Timeout = 0, Attribute = 0, Ntbl Ptr = 0
######## Msg Stats #########
# of IP Range Purge msgs = 20427268
# of IPX Range Purge msgs = 0
# of Add sw shortcut msgs = 0
# of Del sw shortcut msgs = 0
# of fm update to router = 0
# of rx swscAgeTime msgs = 0
# of swscStats msgs = 0
# of ip intf purge msgs = 64
# of next hop purge msgs = 17393
# of FTEP Adj req msg = 4
MSFC IP Address = x.x.x.x
# of Max Hop msgs = 0
# of IP flowmask req msgs = 8 # of IPX flowmask req msgs = 0
# of flowmask updates = 0 # of response msgs = 6
# of NDD IP Add msgs = 0 # of NDD IP Del msgs = 0
# of global IP en = 1 dis = 0
# of global IPX en = 0 dis = 0
# of swsc installed = 0
FIB MSG STATS
=============
#IP Add = 27196489
#IP2Tag Add = 0
#IP Delete = 4117134
#IP Init = 398
#IP Add Default = 2
#IP Delete Default = 4
#IP Addr Change = 0
#IP Standby Addr change = 0
#ADJ Add = 1784642
#ADJ Del = 12176
#IPX Add = 0
#IPX Del = 0
#ADJ Drop = 0
#FIB IP Reload Req = 3
#FIB IPX Reload Req = 0
#FIB IDB Reload Req = 2
#FIB IP Reload Complete = 4
#FIB IPX Reload Complete = 0
#RPF Config = 0
#of fibidb init2 msgs = 89
#RPF Mpath Mode = 1
#Load Sharing Mode = 5
#Tunnel Update = 0
#Tunnel Delete = 0
#Tunnel Feature = 0
#Rate Limit Set = 69
#Rate Limit Rsv = 0
#FIB Version RP Req = 1
#FIB Version SP Ack = 1
#FIB Version RP Ack = 1
#FIB VRF description = 0
#FIB Reload Failed = 0
#Invalid Messages = 0
#Invalid Length = 0
#Invalid Mask = 0
#FIB Version SP Ack Fail = 0
API Stats
==============================
ClearEarlL3EntriesByFlow = 78614762
ClearEarlL3AllEntries = 2
GetEarlL3EntriesByFlow = 950025
CreateEarlL3Entry = 0
MLS HA
===========================
# global sync = 1
FM update = 0
prot Filt sync = 8
IPX Max Hop sync = 0
L3 Age = 0
Fib Sequence Num sync = 5712630
Router Mac sync = 0
swsc Age Time sync = 0
# swsc synced = 0
# purge all fib entries = 1
# wanIfIndices synced = 0
# of ARP sent = 0
HAstdbySyncSeqNum = -1
ERROR counters
===============================
# of IP length errors = 0
# of IP too short = 0
# of IP chksum errors = 0
# of DBUS cksum errors = 0
# of DBUS len errors = 0
# of IB len errors = 0
# of CPU parity errors = 0
# of ACL drop packets = 3556602
# of Netflow full errs = 16777215
So I suppose my questions are:
1) Is vlan based netflow supported? If so, why does NFA not seem to get it?
2) Am I right that collecting netflow stats from the MSFC in hybrid mode is useless for the most part? When would this be useful?
3) The ACL drop packets and Netflow full error counters from the debug above. Do they have any relevance as to why this doesn't seem to be doing what it's supposed to?