Cisco IOS Events to Splunk – Track IOS Command Execution History

Cisco IOS event details can be send to an external system via “syslog”. Splunk server itself and Splunk Universal Forwarder both can act as a syslog server to accept logs from Cisco IOS devices.

To add more cream to Splunk log consolidation solution for Cisco IOS devices – there are few Splunk plugins already available on Splunk App store! These plugins display IOS events on nice colorful dashboards with graphs & charts.

Let’s talk about how we can get this solution in place.

Technical dependencies to get this solution are following –

1. Cisco IOS devices (routers, switches, wlc, asa) configured to send IOS event to Splunk via “syslog”
2. Splunk Indexer (actually this is the Splunk server)
3. (optional) to get nice dashboards it needs two Splunk Apps – (i)Cisco Networks Add-on (TA-cisco_ios) (ii)Cisco Networks (cisco_ios)

Regarding the solution design, there are two options as following –

1. Send logs to Splunk via Splunk Universal Forwarder; this design suits very well in a large infrastructure. Splunk Universal Forwarder can act as local “syslog” for IOS devices; picture below-

splunk-uf-pic-1

2. Send logs directly to the Splunk server –

splunk-server-pic-1

Installation technical procedures are following –

Step 1: Configure Cisco IOS to Send Logs to Splunk “syslog”

Following is an example configuration on a Cisco router –

router# config t
router(config)# logging trap notifications
router(config)# logging 1.1.1.1   ;IPAddr of Splunk syslog – if syslog is running other than UDP 514 – this needs to be specify here

The following commands will send Cisco IOS command execution history to syslog –

router(config)# archive
router(config-archive)# log config
router(config-archive-log-cfg)# logging enable
router(config-archive-log-cfg)# logging size 1000
router(config-archive-log-cfg)# hidekeys ;this will not send passwords to syslog
router(config-archive-log-cfg)# notify syslog
router(config-archive-log-cfg)#exit

Step 2: Configure Splunk or Splunk Universal Forwarder to Accept Logs on UDP://514

There are multiple ways to ways to do this. Adding new listener & sourcetype to “inputs.conf” works for both universal forwarder and Splunk server running on any platform.

On Linux/Unix the default location of this file is – $SPLUNK_INSTALLATION_DIR/etc/system/local/

On Windows the default location of this file is – x:\Program Files\SplunkUniversalForwarder\etc\system\local\

Add the following to the “inputs.conf” file –

[udp://514]
sourcetype = cisco:ios

Restart “splunk” service or “SplunkUniversalForwarder” service to get this change take effect.

If you add “sourcetype = syslog” – this will also work. The “Cisco Network Add-on (TA_cisco-ios)” transforms Cisco syslog to “cisco:ios” sourcetype automatically.

At this stage you should start getting logs coming on to Splunk. Execute some random commands on Cisco IOS and search for sourcetype=”cisco:ios” on Splunk search tab – you should be able to see logs like similar to following –

splunk-search-ciscoios-2

Step 3 (optional): Install Splunk Cisco Apps to Display IOS Events on Dashboards

Install the following two Apps from “Apps > Find More Apps > search Cisco” –

  1. Cisco Network Add-on (TA-cisco_ios)
  2. Cisco Networks (cisco_ios)

Installation is very straight forward – just click on the icon to install it.

If you still not seeing any logs on the Dashboard of Cisco Networks – this might be incorrect “sourcetype” issue and “TA-cisco_ios” is not doing the source type transformation – in this case change your source type to “cisco:ios” manually or you can log a support case with Splunk support to get the TA-cisco_ios fixed for you.

You should be able to see the following on Dashboards –

(the main dashboard)
splunk-cisco-dashboard

(command execution history – who has done what?)
splunk-cisco-exechistory

There are lot more you can find here on this dashboard – explore it.

Advertisements

Cisco New Aironet 1700 Series & WLC Software Compatibility Matrix

Cisco recently released Aironet 1700 series access points which support high speed WiFi IEEE 802.11ac. One of the key specifications of this 1700 series – they “only” work with Cisco Unified Wireless Network Software Release 8.0 or later.

It’s now time to upgrade your WLC to version 8.x to get this work for you.

Here is the latest APs & WLCs software compatibility matrix (as of December 15, 2014) –

cisco-aironet-wlc-software

Details @ http://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html

Cisco AIP SSM Email Alert – Cisco IPS Manager Express (IME)

Cisco AIP SSM are pluggable hardware modules for advanced intrusion prevention security services (IDS/IPS) to Cisco ASA 5500 series firewalls.

Although lots of AIP-SSM configuration parameters can be set via ASDM > IDM (Cisco IPS Device Manager) or via CLI – however there is no such thing on IDM or CLI to send security events or security reports via email.

So, how do I know > what is happening on the IDS/IPS? Is the IPS device capable of detecting threats? Is IPS is blocking attacker IP address?

The answer is – there is a separate piece of software called Cisco IPS Manager Express (IME) to manage, configure and send email alert notifications for AIP-SSM modules. This software needs to be installed on a Windows machine. As of today the latest version is 7.2.7. Supported windows platforms are – Windows Vista Business+/XP Pro/Windows 8+/Windows 2003 R2/Windows 2008 and above.

Apart from all ASA AIP-SSM modules; this IME software does support following Cisco IPS hardware platforms – 4240, 4255, 4260, 4270-20, 4345, 4360, 4510 and 4520.

Here is the download URL (you need valid Cisco login),

https://software.cisco.com/download/type.html?mdfid=282052550&catid=null

Installation is very straight forward; start the installation > follow next, next and finish.

Once IME installation is finished; add all of your AIP-SSMs or IPS devices to IME console via IP address. Make sure IME Windows machine is able to communicate to AIP-SSM or IPS device’s management interface IP address. You can have bunch of IPS devices under one IME.

Setting Up Email Notification

This is very easy task. Open IME console > click on “Tools” > click on “Preferences”; enter your SMTP server details under “Email Setup” tab; screenshot–

IME-EMailSetup

You should send test email to confirm – IME is OK sending email.

Click on the next tab “Notifications” for IDS/IPS security events – configure your preferred notification parameters here; screenshot-

IME-Notifications

Lastly you might want to see consolidated security events in a report – such as what happened in last 24 hours or last 7 days or last 30 days; go to the next tab called “Reports” – all the report parameters are here; this will send PDF report with colorful presentation of data with graphs or charts; screenshot–

IME-Reports

Questions again –

i. You have done configuration of all the email notification parameters, do you need to keep IME running on desktop? Should you close the IME console and logoff?

Answer: Yes – you close IME console and logoff from the Windows computer; IME is still running on the background as a Windows service.

ii. You have added 4 IPS devices on your IME – is email alert notification working on ALL of them?

Answer: Yes – email notification is a global setting within IME that applied to ALL IPS devices those been added to the console. There is no option here to configure email notifications on individual IPS device within the same IME console.

VMware ESXi Host Memory Management, Monitoring, Alert Notification – Part 2

I have described memory monitoring and alert notification gauge on the previous article (Part 1) – let’s do the configuration.

There are so many ways to monitor and get alert notification of VMWare ESXi host memory usage status – most of well-known monitoring solutions come with VMware monitoring plugins pre-installed. vCenter server can also send alerts based on given conditions as well.

Here I will discuss how to configure Nagios Core to monitor memory usage and alert notification; NagiosXI (the commercial edition has a built-in nice easy web UI to do the same). Before moving forward, make sure Nagios server up and running – we need install the following software/tools on the Nagios server –

i. VMware vSphere Perl-SDK; the version should match to the vCenter/ESXi host version – version 5.5 can be download at https://developercenter.vmware.com/web/sdk/55/vsphere-perl
ii. Download and install check_vmware_esx.pl (this is a fork of check_vmware_api.pl) from https://www.monitoringexchange.org/inventory/Check-Plugins/Virtualization/VMWare-%2528ESX%2529/check_vmware_esx.pl—a-fork-of-check_vmware_api.pl-%2528check_esx3-pl%2529 or from https://github.com/BaldMansMojo/check_vmware_esx/blob/master/check_vmware_esx.pl
iii. Install the required Perl modules.

(Step 1 – install VMware vSphere Perl-SDK)

#tar zxvf  VMware-vSphere-Perl-SDK-5.5.0-1384587.x86_64.tar.gz
#cd vmware-vsphere-cli-distrib
#./vmware-install.pl

Accept the license agreement and install with default settings.

If the installation detect missing or old Perl modules – install them; easiest way is install them via CPAN.

(Step 2 – install & configure check_vmware_esx.pl Nagios check script)

Download this from the above mentioned web sites. Copy the “chech_vmware_esx.pl” script to Nagios libexec directory “/usr/local/nagios/libexec/”; make sure it is owned by “nagios” user/group with executable permission.

If you download the “check_vmware_esx_0.9.19.tgz” file – the installation process is following –

#tar zxvf check_vmware_esx_0.9.19.tgz
#cd check_vmware_esx_0.9.19
#cp check_vmware_esx.pl /usr/local/nagios/libexec
#chown nagios.nagios check_vmware_esx.pl
#chmod 751 check_vmware_esx.pl

Copy the perl modules within “check_vmware_esx_0.9.19/modules” to a directory – this can be inside “/usr/local/nagios/libexec” directory –

#mkdir /usr/local/nagios/libexec/vmware_modules
#cp –R /tmp/check_vmware_esx_0.9.19/modules /usr/local/nagios/libexec/vmware_modules/ 
#chown –R nagios.nagios /usr/local/nagios/libexec/vmware_modules

Also change following parameter in the check_vmware_esx.pl file –

use lib “modules”;
to
use lib /usr/local/nagios/libexec/vmware_modules/modules;

Again if the script execution complain about missing Perl module – install them via CPAN.

You should use a “session lock file” to minimize auth log entries on vCenter or ESXi host; every time nagios execute service check with this script – this will create auth log entries in vCenter/ESXi host – it’s huge! The default script will ask you to create session lock file in “/var/nagios_plugin_cache/” directory – create this directory and make sure it is owned by Nagios.

#mkdir /var/nagios_plugin_cache
#chown –R nagios.nagios /var/nagios_plugin_cache

You need to create an user account for this nagios script on your vCenter or on ESXi hosts you want to monitor. You should use “authfile”; this file contains Nagios monitoring user account/password created on vCenter or ESXi host.

#vi /usr/local/nagios/libexec/vmware_plugin/authfile

Enter the following –

username=nagios_userName_on_esxi
password=password_nagios

#chown nagios.nagios /usr/local/nagios/libexec/vmware_plugin/authfile

At this stage the script should be ready to execute! If not – it must be missing Perl modules :(.

(Step 3 – configure Nagios commands and service check)

This script is capable of monitoring lots of other vCenter objects such as cpu, network, datastore, virtual machines etc. Follow standard Nagios guidelines to create your check commands and service checks.

Usage:

To see all memory parameters of an esxi host–
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem

mem usage=42.73% - consumed memory=24501.48 MB - swap used=35.87 MB - overhead=650.41 MB - memctl=0.00 MB: |'mem_usage'=42.73%;;;; 'consumed_memory'=24501.48MB;;;; 'mem_swap'=35.87MB;;;; 'mem_overhead'=650.41MB;;;; 'mem_memctl'=0.00MB;;;;

Set alert notification based on % of memory usage of an esxi host-
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s usage

mem usage=42.73%|'mem_usage'=42.73%;;;;

./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s usage -w 40% -c 60%

Warning! mem usage=42.69%|'mem_usage'=42.69%;40;60;;

Set alert notification based on MB of total memory usage of an esxi host–
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s consumed

consumed memory=24501.29 MB|'consumed_memory'=24501.29MB;;;;

./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s consumed -w 24000 -c 26000

Warning! consumed memory=24475.05 MB|'consumed_memory'=24475.05MB;24000;28000;;

To see swap memory usage only of an esxi host–
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s swapused

swap used=35.87 MB|'mem_swap'=35.87MB;;;;

Screenshot of mem usage on Nagios web UI –

nagios-esxi-memcheck

This script also generate Nagios perfdata which is useful for graphing; if you have pnp4nagios graph installed you should be able to get graph like the following –

nagios-mem-graph

Ruby program as Windows Service – the windows way

Recently I come across Ruby for the first time while installing few cloud-based network monitoring applications on Windows Servers; these are Ruby applications called Ruby “gem”.

After finished installation and configuration – I found I am able to run the Ruby gem without any problem on the command line (very easy – open CMD > go to Ruby bin directory > execute >application_name run). But if I close CMD – the application immediately stop working. I need this application running as Windows Service.

There are couple of ways to make a Ruby windows service –
i. the Ruby way – there are couple of Ruby utilities & gems already available; one of them is called “win32-services”.
ii. the Windows way – by using the OLD “sc.exe” & “SrvAny.exe”; this works OK on Windows 2008 & Windows 2012. This is the easiest one!

I am no Ruby expert – I will describe here how to create Ruby windows service using SC & SrvAny.

Configuration details are following –

i. Get the “SrvAny.exe” and place it in a directory; this can be even inside Ruby directory “C:\Ruby21\mywinservice\srvany.exe”.

ii. Open CMD with admin priv; execute the following sc command to create a windows service –
>sc create MyRubyService binPath= “C:\Ruby21\mywinservice\srvany.exe” DisplayName= “My Ruby Application”

This will create the windows service “MyRubyService” and registry key with the same name. The registry key & entries should look like following–

RubySrvAny

iii. Open regedit; go to “HKLM\SYSTEM\CurrentControlSet\Services\MyRubyService”. Create a new key name “Parameters”. Enter the following entries (String value) under “HKLM\SYSTEM\CurrentControlSet\Services\MyRubyService\Parameters” –

AppDirectory     -this is the Ruby bin directory
Application         -this is the “ruby.exe” file location
AppParameters -this is the ruby gem application “run” command

RubyAppSrv

You might need to stop or disable “Interactive Services Detection” on Windows 2008; by default this is not enabled on Windows 2012.

Cisco IOS Site-to-site IPSec VPN with VRF-lite

Just few years ago if there was a requirement of connecting to destinations with same IP networks address or for a low-level network segregation – the solution was to get separate network devices. These days the same can be done on a single hardware platform using VRF (VRF-lite).

On server platform – it’s virtualization everywhere these days; why not VRF-lite on networking then! I have seen lots of routers they never use above 50% of its capacity! This saves us the following –

i. Buying new router hardware
ii. Less power consumption, less power outlet
iii. Less number of switch ports required
iv. Overall high gain total cost of ownership

That’s why I have started implementing VRF-lite on all my new implementations! Why “all” – because if there any new requirements comes into the picture I still can use the same device; no need to reconfigure the existing platform or buy new devices.

So far I experienced – Cisco IOS does support all IP features with VRF-lite; such as static routing, dynamic routing, BGP, site-to-site vpn, nat and packet filtering firewall. On HP Comware5 platform (A-Series, 5xxx) – VRF-lite doesn’t support Layer-3 packet filtering – other than this they support most of IP services.

Let’s talk about IPSec site-to-site VPN with VRF-lite. Following are the key configurable components of a site to site IPsec VPN –

  1. Remote peer with secret keys
  2. IKE Phase 1 security details
  3. IKE Phase 2 security details
  4. Crypto map
  5. NAT
  6. Access List

On a VRF environment – the whole VPN concept and commands remain same except the following only where we specify network addresses –

1. Remote peer & keys – remote peer is reachable via which VRF domain; instead of global “key” we need to configure “keyring” with specific vrf domain name here.
5. NAT – internal source address belongs to which VRF domain; we need to specify vrf domain name in the NAT rules.
6. Although “access-list” contain of IP addresses – no VRF name need to be specify here.

Following are the command syntaxes for remote peer with VRF details –

(config)#crypto keyring tunnelkey vrf my-vrf-A
  (config-keyring)#pre-shared-key address 10.100.200.1 key 6 mysecretkey
 
 (config)#crypto keyring tunnelkey vrf my-vrf-B
  (config-keyring)#pre-shared-key address 20.100.200.1 key 6 mysecretkey

Without VRF the syntax is (this is called “global key”)-
(config)#crypto isakmp key mysecretkey address 10.100.200.1

Following are the command syntaxes for NAT rules with VRF domain name –
(config)#ip nat inside source static my_src_ip  my_nat_inside_global_ip vrf my-vrf-A

(config)#ip nat inside source static 192.168.1.10 10.100.200.10 vrf my-vrf-A

Show configuration commands for the above are –

#show crypto isakmp key
#show ip nat translations vrf my-vrf-A

If the above are not specified correctly – you might receive the following error on the router log file;

No pre-shared key with 10.x.x.x!
Encryption algorithm offered does not match policy!
atts are not acceptable. Next payload is 3
phase 1 SA policy not acceptable!
deleting SA reason “Phase1 SA policy proposal not accepted” state (R) MM_NO_STATE (peer 10.x.x.x)

 

VMware ESXi Host Memory Management, Monitoring, Alert Notification – Part 1

When it comes VMware memory monitoring – two items to monitor (i)ESXi host memory (ii)VM memory. There are bunch of memory related terminologies and calculations here in this space. I am discussing host memory monitoring here –

-understand physical memory usage monitoring
-what is the right memory counter to monitor & alert notification for esxi host
-what is the right gauge of memory monitoring & alert notification for esxi host

Will also setup Nagios check plugin to monitor the above with performance data for graph (Part 2).

Before moving forward; let’s have a look into Mem.MinFreePct function. This function manage how much host memory should be kept free and when the hypervisor should kick-off advanced memory reclamation techniques such as ballooning, compression, swapping.

(Configuration> Advanced Settings>Mem)
memminfreepct

Based on free host memory & reclamation techniques – there are four (04) different states of host memory utilization;

State Name Mem Reclamation Technique Good or Bad Note
High At this state “Transparent Page Sharing” is will be always running. This is default behaviour. Good – this is normal This is defined by Mem.MinFreePct function. Don’t disable TPS – not recommended.
Soft At this state host will activate memory ballooning. Not good enough This is 64% of Mem.MinFreePct. This means physical memory near to max out.  If host unable to go back to previous state itself – take necessary action to free up more mem.
Hard At this state host will start doing memory compression and hypervisor level swapping. Bad – memory under stress This is 32% of Mem.MinFreePct. Need to free up memory by migrating VMs to other hosts or upgrade memory.
Low At this state host will no more serve any page to VMs. Very Bad – fix it ASAP This is 16% of Mem.MinFreePct. This protects host VMkernel layer from Purple Screen of Death.

Prior to ESXi-5.x this (high state) was set to 6% by default – this means host system will always keep 6% of total physical memory free before activate advanced memory reclamation technique; let’s say an ESXi-4.x host with 64GB memory will be required at least 3.84GB free to be in the High state (normal).

Starting from ESXi-5.x this calculation is no more 6% by default – because high memory servers (512GB/768GB) are becoming common these days; 6% of 512GB is 30.72GB its huge free memory.

The new calculation is following –

Free Memory Threshold Range Calculation Note
6% First 0GB to 4 GB 6% of 4GB
4% Starting from 4GB to 12GB (12-4=8) 4% of 8GB
2% Starting from 12GB to 28GB (28-12=16) 4% of 16GB
1% Remaining memory i.e. 36GB if total size is 64GB (64-28=36)
i.e. 68GB if total size is 96GB (96-28=68)

Based on above – on a system with 128GB memory, the min free memory required to be in “high state” calculation is following –

i. 6% of first 4GB – this is 245.76MB (first 0-4GB)
ii. 4% of 8GB – this is 327.68MB (0-4GB|4-12GB)
iii. 2% of 16GB – this is 327.68MB (0-4GB|4-12GB|12-28GB)
iv. 1% of 100GB – this is 1024MB (0-4|4-12|12-28|28-128GB)
v. Total is 1925.12GB (245.76+327.68+327.68+1024).

esxmemfree

Based on the above we can setup monitoring & alert notification for a 128GB host as following –

Mem State Min Free Mem Monitoring Action Calculation
High 1925.12MB No action required Based on above
Soft 1232.0768MB Warning alert 64% of Mem.MinFreePct
Hard 616.384MB Critical alert 32% of Mem.MinFreePct
Low 308.0192MB Critical alert 16% of Mem.MinFreePct

Also at “Hard” state – memory performance measurement counter “Swap used” will be greater than 0. This condition also should trigger alarm.

vmware-perf-mem

esxtop-mem
(esxtop – memory high state)

References:
http://blogs.vmware.com/vsphere/2012/05/memminfreepct-sliding-scale-function.html