Cisco IOS Events to Splunk – Track IOS Command Execution History

Cisco IOS event details can be send to an external system via “syslog”. Splunk server itself and Splunk Universal Forwarder both can act as a syslog server to accept logs from Cisco IOS devices.

To add more cream to Splunk log consolidation solution for Cisco IOS devices – there are few Splunk plugins already available on Splunk App store! These plugins display IOS events on nice colorful dashboards with graphs & charts.

Let’s talk about how we can get this solution in place.

Technical dependencies to get this solution are following –

1. Cisco IOS devices (routers, switches, wlc, asa) configured to send IOS event to Splunk via “syslog”
2. Splunk Indexer (actually this is the Splunk server)
3. (optional) to get nice dashboards it needs two Splunk Apps – (i)Cisco Networks Add-on (TA-cisco_ios) (ii)Cisco Networks (cisco_ios)

Regarding the solution design, there are two options as following –

1. Send logs to Splunk via Splunk Universal Forwarder; this design suits very well in a large infrastructure. Splunk Universal Forwarder can act as local “syslog” for IOS devices; picture below-

splunk-uf-pic-1

2. Send logs directly to the Splunk server –

splunk-server-pic-1

Installation technical procedures are following –

Step 1: Configure Cisco IOS to Send Logs to Splunk “syslog”

Following is an example configuration on a Cisco router –

router# config t
router(config)# logging trap notifications
router(config)# logging 1.1.1.1   ;IPAddr of Splunk syslog – if syslog is running other than UDP 514 – this needs to be specify here

The following commands will send Cisco IOS command execution history to syslog –

router(config)# archive
router(config-archive)# log config
router(config-archive-log-cfg)# logging enable
router(config-archive-log-cfg)# logging size 1000
router(config-archive-log-cfg)# hidekeys ;this will not send passwords to syslog
router(config-archive-log-cfg)# notify syslog
router(config-archive-log-cfg)#exit

Step 2: Configure Splunk or Splunk Universal Forwarder to Accept Logs on UDP://514

There are multiple ways to ways to do this. Adding new listener & sourcetype to “inputs.conf” works for both universal forwarder and Splunk server running on any platform.

On Linux/Unix the default location of this file is – $SPLUNK_INSTALLATION_DIR/etc/system/local/

On Windows the default location of this file is – x:\Program Files\SplunkUniversalForwarder\etc\system\local\

Add the following to the “inputs.conf” file –

[udp://514]
sourcetype = cisco:ios

Restart “splunk” service or “SplunkUniversalForwarder” service to get this change take effect.

If you add “sourcetype = syslog” – this will also work. The “Cisco Network Add-on (TA_cisco-ios)” transforms Cisco syslog to “cisco:ios” sourcetype automatically.

At this stage you should start getting logs coming on to Splunk. Execute some random commands on Cisco IOS and search for sourcetype=”cisco:ios” on Splunk search tab – you should be able to see logs like similar to following –

splunk-search-ciscoios-2

Step 3 (optional): Install Splunk Cisco Apps to Display IOS Events on Dashboards

Install the following two Apps from “Apps > Find More Apps > search Cisco” –

  1. Cisco Network Add-on (TA-cisco_ios)
  2. Cisco Networks (cisco_ios)

Installation is very straight forward – just click on the icon to install it.

If you still not seeing any logs on the Dashboard of Cisco Networks – this might be incorrect “sourcetype” issue and “TA-cisco_ios” is not doing the source type transformation – in this case change your source type to “cisco:ios” manually or you can log a support case with Splunk support to get the TA-cisco_ios fixed for you.

You should be able to see the following on Dashboards –

(the main dashboard)
splunk-cisco-dashboard

(command execution history – who has done what?)
splunk-cisco-exechistory

There are lot more you can find here on this dashboard – explore it.

Splunk user authentication with Windows Active Directory

Recently I have implemented Windows Active Directory authentication for Splunk. My Windows AD DS is a mix of Windows 2008 and Windows 2012; Splunk version is 6.x.

Here is below the technical steps-

1. Login to Splunk using admin account. Go to “Settings > Access Controls > Authentication method”. You should be able to see the available authentication methods here. Select “LDAP” and click on “Configure Splunk to use LDAP and map groups”. Screenshot –

Splunk-AD-1

2. Click on the “New” button. On the new page – this is the main LDAP strategy configuration settings page. Following are the main AD items that you need to enter here –

a. LDAP connection settings – based on connection settings Splunk will talk to AD.

LDAP strategy name: just a name.

You can have multiple LDAP strategies such as –  (i)strategy one for ready only access through an AD Group mapping to Splunk roles (user & power user), (ii)strategy two for full access through another AD Group mapping to other Splunk roles (Admin, Splunk-system-role) or similar.

Default Splunk roles are – admin, can_delete, power, splunk-system-role, user.

Port number: 389 (this is AD LDAP default)

Connection order: default

Bind DN: cn= AcctName Splunk,ou=yourSvcAcctOU,dc=yourDCName,dc=yourDCExtension

This is distinguished name of your Splunk account that you created in AD. It is recommended you should not use default AD administrator account or your own AD login here. You should create a dedicate account for Splunk – no AD administrative privilege required on this account.

Bind DN Password: enter the password of AD Splunk account

Splunk-AD-2

b. User Settings – Splunk will look for users in AD based on this

User base DN: dc=yourDCName,dc=yourDCextension

User base filter: leave this blank or you can enter specific AD search filter here

User name attribute: samaccountname

Real name attribute: displayname

Group mapping attribute: dn

Screenshot –

Splunk-AD-3

c. Group settings – Splunk will look for AD groups in AD based on this

Group base DN: cn=Group_Splunk_Access_Admins,ou=youGroupOUName,dc=yourDCName,dc=DCextension

This is the AD group that been created to grant access in Splunk.

Static member attribute: member

Screenshot-

Splunk-AD-4

d. Dynamic group settings – optional

e. Advanced settings – default is ok; however you can increase search request size limit.

Screenshot –

Splunk-AD-5

Click on the “Save”. If entered parameters are not correct – you won’t be able to save.

Now you should be able to see your LDAP strategy. Make sure it is enabled.

To see your AD group in Splunk, click on “Map groups”.

To map Splunk role(s) to an AD group – click on “Map groups > AD Group Name > available and selected roles”; screenshots –

Splunk-AD-6

Splunk-AD-8

Also you should be able to see AD users at “Settings > Access controls >Users”. Make sure AD users are member of the Splunk group that been created on AD.