Shadow IDS

SHADOW (Secondary Heuristic Analysis for Defensive Online Warfare) is a very simple combination of tcpdump and Perl that actually works as a rudimentary NIDS but can made more powerful by using snort and/or custom Perl processing scripts. Due to flexibility and power of Perl the architecture used by Shadow remains one of the best NIDS architectures and in my opinion is architecturally superior to Snort.  

The original Shadow code was developed by the US Navy as the result of the "Cooperative Intrusion Detection Evaluation and Response" (CIDER) project. Currently SHADOW codebase is unmaintained abandonware and the code base is largely obsolete. The latest published version (2003) was developed by NSWC.

Solaris is an excellent platform to run any NIDS and Shadow is not an exception. But most steps actually are OS independent. 

Shadow basically consists of two parts that can run on different boxes (although it is not strictly necessary):

• Sensor. The sensor station is the server that running tcpdump that records traffic to a file. If necessary any suitable transport mechanism can be used to transfer file to central "analysis station" (ssh sync, etc). With the current CPUs and disks speed one sensor can run several instanced of TCPdump without breaking sweet and thus can listen to several traffic interception points. 
   
• Analysis Station. The analysis station is optional and serves as the main processing power for recorded traffic (that implies that files needs to be transferred to this station prior to analysis or NFS would be used. That may require creation of a separate segment as it due to volume of traffic it can negatively affect other network users)
   
  • Recorded traffic is runs through a series of Perl filters. All filters are tuned to tcpdump format. As Perl is much more powerful then ad-hoc capabilities of Snort you can get more useful alerts. You can also use Snort for processing tcpdump file to do the dirty work and produce "first level alerts" and database file for further processing.
  • If you are smart the next step can be integration of multiple steams of alerts via Tivoli TEC or similar monitoring engine.  

While the idea is transparent and very good, the code supplied is not very transparent and is of limited usefulness for any knowledgeable Perl programmer. There are things that definitely can be simplified.

The key files are:

cleanup.pl
condense.pl
fetchem.pl
find_scan.pl
obfuscate.pl

There is also pretty convoluted file (sensor_driver.pl) to launch tcpdump and it probably can be discarded and replaced with a simple ksh script. 

Actually you need is the following simple loop that can be invoked from cron each selected sensor using preselected interval, for example 15 min:

1. Write file with tcpdump to the archive (possibly compress it if you nee to work with huge pipe).

2. Optionally transfer it to the "analysis station".

3. Reprocess this file with TCPdump working in "listing mode", feeding the output to Perl analyzer that deciphers TCP dump format and produced actual alerts.  Or better use Snort snort instead of TCPdump for doing this and reprocessed generated alerts with Perl (using Snort database format)

NEWS CONTENTS

• 20050427 : NewsForge Build an IDS with Snort, Shadow, and ACID by Mark Bain. ( NewsForge Build an IDS with Snort, Shadow, and ACID, Apr 27, 2005 )

Old News ;-)

[Apr 27, 2005] NewsForge Build an IDS with Snort, Shadow, and ACID by Mark Bain.

It looks like the author made it all wrong.

Title		Build an IDS with Snort, Shadow, and ACID
Date		2005.04.27 11:00
Author		Mark Bain
Topic

http://software.newsforge.com/article.pl?sid=05/04/14/1535223

Every organization understands the importance of using a firewall to protect its assets. But what happens if someone finds a hole in the wall? What if the infiltrator is actually someone from within your organization who wants to access information that he shouldn't? To cover those contingencies, you need an intrusion detection system (IDS) to complement your firewall. Fortunately, with a minimum amount of time and money you can set up an IDS with open source tools such as Snort, Shadow, and ACID.

An IDS comprises sensors to do the actual monitoring and a central console to store and analyse the data collected on each sensor. A sensor can be a basic PC with an extra network card (to do the actual monitoring), whilst the central console should have a bit more processing power and a larger disk.

If Linux isn't already on the PCs you plan to use then you'll have to install it. You'll need only a minimal install because these boxes will be used only for the IDS. I recommend installing Bastille to make the IDS more secure. Another key thing to do is to put the monitoring network card on the sensor without an IP address; this makes it effectively invisible.

You can now start installing the software to do the actual intrusion detection. You can use Snort to monitor patterns of behaviour (or signatures) and Shadow to monitor packet headers. You'll need ACID to analyse the data collected by Snort (Shadow has its own analyser).

Let's begin with the sensor. The first piece of software to install is the client and develop RPMs for MySQL drivers, which are necessary because Snort store its output in a database on the central console. Once these drivers are in place, download the tar file for Snort, decompress it, and install it with the commands:

./configure --with-mysql
make
make install

The installation and setup instructions for Snort are clear and concise. Snort uses a set of rules to identify signatures (or patterns of behaviour) to identify illegal activity. You can download a default set of customisable rules from Snort's site, along with some excellent documentation. If you have a Windows client, you can manage the signatures using Activeworx's IDS Policy Manager.

Shadow

With Snort on the workstation, the next step is to install Shadow. The software comprises a set of Perl scripts that use tcpdump to collect data from packet headers on the network in order to identify any unusual activity. Shadow's Web site contains documentation that covers package dependencies (including Perl and tcpdump; the latter is supplied with Shadow), installation, and setup. To configure Shadow you need to modify three files -- sensor_init.sh, std.ph, and std.filter -- to reflect your local setup, such as the location of tcpdump. You also need to set up crontab to run sensor_driver.pl (the main script of the sensor) once an hour. The package supplies a suitable cron file -- /usr/local/SHADOW/sensor/sensor_crontab -- that also will synchronize the sensor time with a reliable Network Time Protocol source.

Finally, you must make sure that Shadow runs at system startup time by copying /usr/local/SHADOW/sensor_init.sh to your /etc/rc.d directory. Once Shadow is running, your sensor PC is ready to be connected to the network.

Shadow comes with its own analyser, which requires tcpdump, OpenSSH, and Apache. The Shadow documentation will lead you through the configuration process, tell you how to set up the Shadow CGI scripts, and explain how to set up crontab for automatic operation.

The console

With the sensor set up you can turn to the central console, where the actual analysis of the data will be done.

Snort has no analysis software itself -- it simply generates data on the sensor and then stores it in a MySQL database on the central console. You can use ACID (Analysis Console for Intrusion Databases) to analyse the data. ACID's Web site gives full details on software installation and the creation of the MySQL database for storing the Snort alerts.

At this point your IDS should be up and running with Snort, Shadow, and ACID. On the sensor Snort will be monitoring the traffic patterns and raising alerts to the database. Shadow will be reading packet headers and creating log files. On the central console you will be able to view the Web pages created by Shadow, and to use ACID to analyse the alerts created by Snort.

Now the real work begins. You must learn how to understand the traffic on your own network so that you can filter the false alarms from the real ones. To start learning, have a look at the Reference and Security Links sections of the Shadow Web site. Then congratulate yourself for giving your organization a real chance to catch an intruder before he can do any damage.

Links

1. "Bastille" - http://www.bastille-linux.org/
2. "Snort" - http://www.snort.org/
3. "Shadow" - http://www.nswc.navy.mil/ISSEC/CID
4. "ACID" - http://www.andrew.cmu.edu/user/rdanyliw/snort/snortacid.html
5. "IDS Policy Manager" - http://www.activeworx.org/programs/idspm/index.htm
6. "tcpdump" - http://www.tcpdump.org/
7. "Shadow Web site" - http://www.nswc.navy.mil/

Recommended Links

Softpanorama Recommended

Top articles

Sites

• Current Shadow software - version 1.8, released April 2003
  
• Shadow Screen Shots
  
• Current Shadow documentation (PDF format)
  
• Current Shadow documentation (HTML format)
  
• SHADOW articles of interest
  
• INTRUSION DETECTION: Shadow Style
  
• Whitehats.ca - Shadow Sensor Configuration Policy some not very coherent attempt to use shadow along with SGUIL - The Analyst Console for Network Security Monitoring.

SHADOW Technical Analysis

• Coordinated Attacks and Probes
• Stealth and coordinated powerpoint
• Stealth Coordinated Attacks

Recommended Papers

ShadowCon '00 Presentations

ShadowCon '99 Presentations

Proposed 1999 DARPA Off-line Intrusion Detection Evaluation Plans

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last updated: March 12, 2019