Cisco IOS Events to Splunk – Track IOS Command Execution History

Cisco IOS event details can be send to an external system via “syslog”. Splunk server itself and Splunk Universal Forwarder both can act as a syslog server to accept logs from Cisco IOS devices.

To add more cream to Splunk log consolidation solution for Cisco IOS devices – there are few Splunk plugins already available on Splunk App store! These plugins display IOS events on nice colorful dashboards with graphs & charts.

Let’s talk about how we can get this solution in place.

Technical dependencies to get this solution are following –

1. Cisco IOS devices (routers, switches, wlc, asa) configured to send IOS event to Splunk via “syslog”
2. Splunk Indexer (actually this is the Splunk server)
3. (optional) to get nice dashboards it needs two Splunk Apps – (i)Cisco Networks Add-on (TA-cisco_ios) (ii)Cisco Networks (cisco_ios)

Regarding the solution design, there are two options as following –

1. Send logs to Splunk via Splunk Universal Forwarder; this design suits very well in a large infrastructure. Splunk Universal Forwarder can act as local “syslog” for IOS devices; picture below-


2. Send logs directly to the Splunk server –


Installation technical procedures are following –

Step 1: Configure Cisco IOS to Send Logs to Splunk “syslog”

Following is an example configuration on a Cisco router –

router# config t
router(config)# logging trap notifications
router(config)# logging   ;IPAddr of Splunk syslog – if syslog is running other than UDP 514 – this needs to be specify here

The following commands will send Cisco IOS command execution history to syslog –

router(config)# archive
router(config-archive)# log config
router(config-archive-log-cfg)# logging enable
router(config-archive-log-cfg)# logging size 1000
router(config-archive-log-cfg)# hidekeys ;this will not send passwords to syslog
router(config-archive-log-cfg)# notify syslog

Step 2: Configure Splunk or Splunk Universal Forwarder to Accept Logs on UDP://514

There are multiple ways to ways to do this. Adding new listener & sourcetype to “inputs.conf” works for both universal forwarder and Splunk server running on any platform.

On Linux/Unix the default location of this file is – $SPLUNK_INSTALLATION_DIR/etc/system/local/

On Windows the default location of this file is – x:\Program Files\SplunkUniversalForwarder\etc\system\local\

Add the following to the “inputs.conf” file –

sourcetype = cisco:ios

Restart “splunk” service or “SplunkUniversalForwarder” service to get this change take effect.

If you add “sourcetype = syslog” – this will also work. The “Cisco Network Add-on (TA_cisco-ios)” transforms Cisco syslog to “cisco:ios” sourcetype automatically.

At this stage you should start getting logs coming on to Splunk. Execute some random commands on Cisco IOS and search for sourcetype=”cisco:ios” on Splunk search tab – you should be able to see logs like similar to following –


Step 3 (optional): Install Splunk Cisco Apps to Display IOS Events on Dashboards

Install the following two Apps from “Apps > Find More Apps > search Cisco” –

  1. Cisco Network Add-on (TA-cisco_ios)
  2. Cisco Networks (cisco_ios)

Installation is very straight forward – just click on the icon to install it.

If you still not seeing any logs on the Dashboard of Cisco Networks – this might be incorrect “sourcetype” issue and “TA-cisco_ios” is not doing the source type transformation – in this case change your source type to “cisco:ios” manually or you can log a support case with Splunk support to get the TA-cisco_ios fixed for you.

You should be able to see the following on Dashboards –

(the main dashboard)

(command execution history – who has done what?)

There are lot more you can find here on this dashboard – explore it.

Cisco New Aironet 1700 Series & WLC Software Compatibility Matrix

Cisco recently released Aironet 1700 series access points which support high speed WiFi IEEE 802.11ac. One of the key specifications of this 1700 series – they “only” work with Cisco Unified Wireless Network Software Release 8.0 or later.

It’s now time to upgrade your WLC to version 8.x to get this work for you.

Here is the latest APs & WLCs software compatibility matrix (as of December 15, 2014) –


Details @

Cisco AIP SSM Email Alert – Cisco IPS Manager Express (IME)

Cisco AIP SSM are pluggable hardware modules for advanced intrusion prevention security services (IDS/IPS) to Cisco ASA 5500 series firewalls.

Although lots of AIP-SSM configuration parameters can be set via ASDM > IDM (Cisco IPS Device Manager) or via CLI – however there is no such thing on IDM or CLI to send security events or security reports via email.

So, how do I know > what is happening on the IDS/IPS? Is the IPS device capable of detecting threats? Is IPS is blocking attacker IP address?

The answer is – there is a separate piece of software called Cisco IPS Manager Express (IME) to manage, configure and send email alert notifications for AIP-SSM modules. This software needs to be installed on a Windows machine. As of today the latest version is 7.2.7. Supported windows platforms are – Windows Vista Business+/XP Pro/Windows 8+/Windows 2003 R2/Windows 2008 and above.

Apart from all ASA AIP-SSM modules; this IME software does support following Cisco IPS hardware platforms – 4240, 4255, 4260, 4270-20, 4345, 4360, 4510 and 4520.

Here is the download URL (you need valid Cisco login),

Installation is very straight forward; start the installation > follow next, next and finish.

Once IME installation is finished; add all of your AIP-SSMs or IPS devices to IME console via IP address. Make sure IME Windows machine is able to communicate to AIP-SSM or IPS device’s management interface IP address. You can have bunch of IPS devices under one IME.

Setting Up Email Notification

This is very easy task. Open IME console > click on “Tools” > click on “Preferences”; enter your SMTP server details under “Email Setup” tab; screenshot–


You should send test email to confirm – IME is OK sending email.

Click on the next tab “Notifications” for IDS/IPS security events – configure your preferred notification parameters here; screenshot-


Lastly you might want to see consolidated security events in a report – such as what happened in last 24 hours or last 7 days or last 30 days; go to the next tab called “Reports” – all the report parameters are here; this will send PDF report with colorful presentation of data with graphs or charts; screenshot–


Questions again –

i. You have done configuration of all the email notification parameters, do you need to keep IME running on desktop? Should you close the IME console and logoff?

Answer: Yes – you close IME console and logoff from the Windows computer; IME is still running on the background as a Windows service.

ii. You have added 4 IPS devices on your IME – is email alert notification working on ALL of them?

Answer: Yes – email notification is a global setting within IME that applied to ALL IPS devices those been added to the console. There is no option here to configure email notifications on individual IPS device within the same IME console.

Cisco IOS Site-to-site IPSec VPN with VRF-lite

Just few years ago if there was a requirement of connecting to destinations with same IP networks address or for a low-level network segregation – the solution was to get separate network devices. These days the same can be done on a single hardware platform using VRF (VRF-lite).

On server platform – it’s virtualization everywhere these days; why not VRF-lite on networking then! I have seen lots of routers they never use above 50% of its capacity! This saves us the following –

i. Buying new router hardware
ii. Less power consumption, less power outlet
iii. Less number of switch ports required
iv. Overall high gain total cost of ownership

That’s why I have started implementing VRF-lite on all my new implementations! Why “all” – because if there any new requirements comes into the picture I still can use the same device; no need to reconfigure the existing platform or buy new devices.

So far I experienced – Cisco IOS does support all IP features with VRF-lite; such as static routing, dynamic routing, BGP, site-to-site vpn, nat and packet filtering firewall. On HP Comware5 platform (A-Series, 5xxx) – VRF-lite doesn’t support Layer-3 packet filtering – other than this they support most of IP services.

Let’s talk about IPSec site-to-site VPN with VRF-lite. Following are the key configurable components of a site to site IPsec VPN –

  1. Remote peer with secret keys
  2. IKE Phase 1 security details
  3. IKE Phase 2 security details
  4. Crypto map
  5. NAT
  6. Access List

On a VRF environment – the whole VPN concept and commands remain same except the following only where we specify network addresses –

1. Remote peer & keys – remote peer is reachable via which VRF domain; instead of global “key” we need to configure “keyring” with specific vrf domain name here.
5. NAT – internal source address belongs to which VRF domain; we need to specify vrf domain name in the NAT rules.
6. Although “access-list” contain of IP addresses – no VRF name need to be specify here.

Following are the command syntaxes for remote peer with VRF details –

(config)#crypto keyring tunnelkey vrf my-vrf-A
  (config-keyring)#pre-shared-key address key 6 mysecretkey
 (config)#crypto keyring tunnelkey vrf my-vrf-B
  (config-keyring)#pre-shared-key address key 6 mysecretkey

Without VRF the syntax is (this is called “global key”)-
(config)#crypto isakmp key mysecretkey address

Following are the command syntaxes for NAT rules with VRF domain name –
(config)#ip nat inside source static my_src_ip  my_nat_inside_global_ip vrf my-vrf-A

(config)#ip nat inside source static vrf my-vrf-A

Show configuration commands for the above are –

#show crypto isakmp key
#show ip nat translations vrf my-vrf-A

If the above are not specified correctly – you might receive the following error on the router log file;

No pre-shared key with 10.x.x.x!
Encryption algorithm offered does not match policy!
atts are not acceptable. Next payload is 3
phase 1 SA policy not acceptable!
deleting SA reason “Phase1 SA policy proposal not accepted” state (R) MM_NO_STATE (peer 10.x.x.x)


Cisco ASA 5500 series recommended IOS upgrade path

You might encounter problem while attempt to upgrade Cisco Adaptive Security Appliance (ASA) to Version 8.4.7 or later or to Version 9.1.3 or later. Here is below the recommended upgrade path – that resolves the problem.


ASA version pre-8.3 NAT rule syntax are different than version 8.4 and later. This upgrade path will automatically convert pre-8.3 version NAT rule syntax to version 8.4 and later.

Few well known errors are “”No Cfg structure found in downloaded image file” and “no NAT rule found after the upgrade”.

HP 5820 LACP (802.3ad) with non-HP (Cisco, F5 and others)

HP 5820-24XG-SFP+ is a 24 port 10GB SFP+ Layer 3 enterprise and carrier grade Ethernet switch. This is running Comware5 network operating system. This switch is actually manufactured by H3C.

HP 5820 series LACP (802.3ad) link aggregation is called “Bridge-Aggregation” interface. The configuration is pretty much similar to Cisco EtherChannel.

I will discuss two things here–

(a). how to configure link-aggregation between HP and non-HP (Cisco, F5, Juniper) devices.

(b). how to configure link-aggregation between HP and HP (HP ProCurve ProVision, Comware5/7, HP SAN).

Following are configuration commands to create LACP between HP and non-HP devices –

(a). Enter the following command on the HP switch to create bridge-aggregation interface

(trunk/tagged vlan interface example)

interface Bridge-Aggregation1
 description “LACP Trunk goes to a non-HP device”
 port link-type trunk
 port trunk permit vlan all ;this can be limited to user define VLANs only 
 link-aggregation mode dynamic  ;for non-HP mode dynamic is require

(access port example)

interface Bridge-Aggregation2
 description “LACP access goes to a non-HP device”
 port access vlan 100
 link-aggregation mode dynamic  ;for non-HP mode dynamic is require

After define bridge aggregation interface – you need to add physical interfaces to it. Physical interface should inherit all the parameters configured on the bridge-aggregation interface.

Make sure physical interfaces are configured with “default” settings only before put them to a bridge-aggregation group to get settings auto inherited. Otherwise you need to specify “link-type” and “permit vlan” parameters once again on all the member physical interfaces.

interface Ten-GigabitEthernet1/0/22
   port link-mode bridge
   description “this interface is a member of bridge-aggregation 1”
   port link-type trunk
   port trunk permit vlan all
   port link-aggregation group 1

interface Ten-GigabitEthernet2/0/22
   port link-mode bridge
   description “this interface is a member of bridge-aggregation 1”
   port link-type trunk
   port trunk permit vlan all
   port link-aggregation group 1

Once done – you should be able to see aggregated bandwidth on the “Bridge-Interface”.

Do a “#display interface Bridge-Aggregation 1

Screenshot of a two 10Gbps LACP interface (20Gbps aggregated) –


(b). For LACP between HP and HP devices enter all the above commands except “link-aggregation mode dynamic”.

Inter-VRF routing on the same Router (VRF-lite route leak) – Cisco IOS

I was trying to implement inter-VRFs routing in a multi VRF-lite environment – there was a requirement to implement routing between two VRF domains on the same router. I couldn’t make this working through typical static routing or IGP. Later on I found Cisco recommendation – this has to be done through (i)route-target export/import and (ii)BGP.

“You can not configure two static routes to advertise each prefix between the VRFs, because this method is not supported—packets will not be routed by the router. To achieve route leaking between VRFs, you must use the import functionality of route-target and enable Border Gateway Protocol (BGP) on the router. No BGP neighbor is required.

Here is my topology diagram–


Routing/connectivity requirements are –

Within “same router” inter-VRFs routing:

Source Network Destination Network Site Number
A-1-network-webserver A-1-network-appserver Site 1
A-2-network-webserver A-2-network-appserver Site 2

Inter site (site 1 & site 2) VRFs routing:

  1. Multiple site-1 sources to > multiple site-2 destinations;
Source Network Destination Network
A-1-network-webserver A-2-network-webserver
A-1-network-webserver A-2-network-appserver
A-1-network-appserver A-2-network-appserver
A-1-network-appserver A-2-network-webserver
  1. One site-1 source to > one site-2 destination;
Source Network Destination Network
A-1-network-iscsi A-2-network-scsi
B-1-network-appserver B-2-network-app

Based on above mentioned scenario –

  1. VRFs routing between Site 1 and Site 2 – static route or any dynamic routing protocol such as EIGRP, OSPF are suitable.
  2. VRFs routing within the same router at each site (routing for web & app on the same site) need to be done through multiprotocol BGP and route-target import – which is a recommendation by Cisco.

I will show here how to do inter VRFs routing within the same router using BGP and route-target export-import.

Following are configurations on SITE-1-Router-01,

(Step 1) Define VRFs and route-target export & import as following:

ip vrf a-1-webserver
rd 65111:101
route-target export 65111:101
route-target import 65111:101
route-target import 65111:102  ;import “a-1-network-appserver”
ip vrf a-1-appserver
rd 65111:102
route-target export 65111:102
route-target import 65111:102
route-target import 65111:101  ;import “a-1-network-webserver”
ip vrf a-1-iscsi
rd 65111:103      ;no network export-import here
ip vrf b-1-appserver
rd 65111:104      ;no network export-import here

(Step 2) Apply the VRFs to proper interfaces – assign IP address to interfaces as well.

(Step 3) Configure BGP without neighbour with the VPN instances name as following:

(we need routing between webserver & appserver on the same router)

router bgp 65111
bgp log-neighbor-changes
address-family ipv4 vrf a-1-webserver
redistribute connected
address-family ipv4 vrf a-1-appserver
redistribute connected

Once the BGP is done – do a “#show ip route vrf a-1-webserver”; it should display both a-1-appserver & a-1-webserver networks. Same result should display for “#show ip route vrf a-1-appserver”. At this stage a-1-webservers should be able to talk to a-1-appservers. Configure the same on SITE-2-Router-02 router.

####rest of the configuration are for inter site (site-1 & site-2) communication####

(Step 4) For routing between SITE-1 and SITE-2 following is an example with static routing:

In this example –

Site-1 (source) networks are-

-Customer A webserver network is –; default route is
-Customer A appserver network is –; default route is
-Customer A iscsi network is –
-Customer B appserver network is – 192.168.104/24

Site-2 (destination) networks are-

-Customer A webserver network is –
-Customer A appserver network is –
-Customer A iscsi network is –
-Customer B appserver network is – 192.168.204/24

SITE-1-ROUTER-01 inter site routing commands are following –

ip route vrf a-1-webserver
ip route vrf a-1-webserver; (A-1 web to A-2 web)
ip route vrf a-1-webserver; (A-1 web to A-2 app)
ip route vrf a-1-appserver
ip route vrf a-1-appserver; (A-1 app to A-2 web)
ip route vrf a-1-appserver; (A-1 app to A-2 app)
ip route vrf a-1-iscsi; (A-1 iscsi to A-2 iscsi)
ip route vrf b-1-iscsi; (B-1 app to B-2 app)

Configure the SITE-2-Router-02 same way (change the source and destination networks).

Do “#show ip vrf vrfname” to check your routes; also do ping test “#ping vrf vrfname ip ipAddr”.