Profiling and Tuning TEC rules

The following list describes suggested techniques for tuning rules.

• Write rules to drop unneeded events immediately (as one of the first rules encountered so that events do not have to process through all rules in the rule base).
• Use the commit_action, commit_rule, and commit_set rule language predicates frequently.
• Monitor the number of rule timers used.
• When writing rules, limit the use of the following predicates:
  • all_instances
  • all_duplicates
  • – generate_event predicates

SG246614

Rule profiling

Rule profiling generates a report that holds rule execution information. Such a report contains the following information for each rule being profiled:

• The amount of time (in seconds) spent by the rule to process the last event that triggered the rule.
• The number of events processed by the rule.
• The amount of time (in seconds) all events spent in the rule for processing.
• The throughput of events for the rule, expressed as the number of events per second.

Profiling can only be activated via CLI commands.

chatham> wrb -lscurrb

The currently used rule base was loaded from the rule base named 'spooler_rb'.

chatham> wrb -comprules -profile spooler_rb

...

chatham> wrb -loadrb -use spooler_rb

...

chatham> wstopesvr

chatham> wstartesvr

The Tivoli Enterprise Console Server is initializing...

The Tivoli Enterprise Console Server is running.

After restarting the Event Server, the profiling starts. A report is written to the $DBDIR/tec/profile file. For each new event processed, a record is appended to this file.

Tip: On restart of the Event Server, the old file contents are still kept. This can cause a large file if the profiling runs for some time, especially if many events are processed. For this reason, we recommend checking periodically for the file size if profiling is used to avoid filling the $DBDIR file system.

Example 14-34 Profile report

===========================================

Timing Summary                

--------------------------------------------

prefix:

Time for last Event: 1.000000000000001e-02

Event Count:         13

Total Time:          1.200000000000001e-01

Events per second:   1.083333333333332e+02

--------------------------------------------

prefix_msgg_rl_of_set_prefix:

Time for last Event: 0.000000000000000e+00

Event Count:         13

Total Time:          8.000000000000007e-02

Events per second:   1.624999999999999e+02

--------------------------------------------

--------------------------------------------

============================================

SG246614

14.5.2 Event Activity Reports

Event Activity Reports represent a textual listing of event activities in the rule engine. The following items can be specified to report and can be dynamically changed during reporting:

How often to write the report

The file and path name for the report

Events to exclude from the report

Attribute criteria to include in the report

Counter threshold (for example, "do not include counts of less than 5 in the report")

The relevant parts of the im.rls file are shown Example 14-35. The im_configure rule includes the initial configuration settings to create an Event Activity Report (the editable parts are shown in bold):

Example 14-35 Event Activity Report

----------------------------------------------------------------

% NAME - IM Configuration.

% This rule is used to configure the IM rule behavior.

% Event activity reporting, event filtering, maintenance mode,

% heartbeat, thresholding and event correlation can be activated.

%---------------------------------------------------------------- 

rule:  im_configure : (

event: _event of_class  'TEC_Start'

where [],

reception_action: im_parameters: (

...

      %----------------------------------------------

% Activate rule functionality (active/inactive)

%----------------------------------------------

record(event_activity,inactive),

...

      %------------------------------------------------------------------

% Event activity

% This section allows the event activity report to be customized.

% The name of the report file, the reporting frequency, 

      % event reporting threshold, events to exclude from the report and 

      % attribute criteria to include in the report can be specified.

%------------------------------------------------------------------

( recorded(event_activity,active) ->

_reporting_frequency is 20,                     % Write statistics this often (in seconds)

init_event_activity('/tmp/event_activity',      % Store report in this file  

                              ['TEC_Heartbeat',          % Do not report these events

'TEC_Maintenance'],

[ source,                  % Attribute reporting criteria

hostname,                % Can be count of single attribute

severity,

status,

[hostname,severity],     % Or multiple attributes

[class,hostname]         % use class key word to report by class name

],

5                          % Do not report counts less than this 

                              ),

first_instance(event:_ev of_class 'TEC_Tick' where [ ]),

set_timer(_ev, _reporting_frequency, 'Event Activity Report')

;

true

),

...

).

To enable Event Activity Reporting, change the line:

record(event_activity,inactive),

to:

record(event_activity,active),

In the Event activity section, all settings for what should be reported can be applied. The _reporting_frequency can be specified as well as the initial event activity settings, such as:

init_event_activity(	output_file_name,

event_exclusion_list,

attribute_reporting_criteria,

reporting_threshold)

The Event Activity rules provided with the im.rls file in the default rule base are shown Example 14-36.

Example 14-36 Event Activity rules

%-----------------------------------------------------------------------------------------

% Collect event activity statistics:

% This rule is used to collect event activity statistics. It is activated when statistics are turned on in the initialization rule.

%-----------------------------------------------------------------------------------------

rule: update_event_activity: (

event: _event of_class _class

where [],

reception_action: update_activity: ( 

     recorded(event_activity,active),   

     update_event_activity(_event)

)

). 

%-----------------------------------------------------------------------------------------

% Print Event Activity Report:

% This periodic call prints the event activity report. Report properties are set in the initialization rule. After the report is printed, all statistics are reset.

%-----------------------------------------------------------------------------------------

timer_rule: reset_event_activity: (

event: _event of_class _class

where [],

timer_info: equals 'Event Activity Report',

timer_duration: _rep_freq,

action: reset_activity: ( 

     recorded(event_activity,active),

print_event_activity,

reset_event_activity,     

     set_timer(_event,_rep_freq,'Event Activity Report')

)

).

Speed TEC vs. Netcool - Who processes events faster Gulf Breeze Software Partners Tivoli Consulting Training We are Tivoli experts

Posted by: justin_eagleson on May 05, 2006 - 05:43 PM

BlogArticle  OK, now I'm treading on sacred ground...
Obviously hardware matters since both are CPU driven applications, but assuming we're dealing with like hardware, the answer is it depends...

TEC

IBM has published benchmarking speeds of the TEC rule engine (a while ago with TEC 3.8 FP whatever running on AIX) at ~65 events per second.

How did they come to derive this number? They start a timer when events are flowing at TEC and stop the timer when the CPU on TEC goes below 5% used.

Is that a good measurement? Maybe, but IMHO not a good way to tell whether the rule engine is really done processing an event, that's for sure. It would be better to actually measure whether the event is PROCESSED in the reception log to really know when the rule engine is finished processing an event. I've also seen events per second increase with a TEC database installed locally rather than remotely.

So documented by IBM, TEC is around ~65 events per second. Of course that is without running any real rules against the incoming events. Another factor to consider is I.V., who as we all know has achieved 300+ events per second using his non-tme rule engine.

Netcool Omnibus

Based on what I've heard in the past from MicroMuse when asked how fast Omnibus processes events, I've heard "thousands of events per second". When we drilled deeper and asked more questions, we were told those were "raw" events and it's more like "hundreds of events per second". What is unclear is whether these events were acted upon by any rules (i.e. is this merely event reception speed). Unfortunately, I don't have any hard benchmarking numbers to share at this point until I get my hands on Netcool code.

So based on this, it really depends on a number of factors such as what kind of rules are you running against your events, how you tell when an event is "processed" by the rule engine, and whether or not you are I.V. ;-)

Re: Speed: TEC vs. Netcool - Who processes events faster?

(Score: 1)
by ivb on May 06, 2006 - 02:43 AM
(User information http://) Firstly, it not my non-tme rule engine, it is IBM's shipped totaly unsupported with TEC as a tar file in $BINDIR/TME/TEC directory. Just wanted to get that straight ;-). Beides myself, Loren Cain is the only other guy I know of brave enought to use it in a production environment.

I found the IBM benchmark flawed as well.. it's trivial to write Prolog that will track processing time, but it is hard to get more than one second granularity. What I have done in the past is use strace/truss with milisecond timestamps to get a higher resolution of processing time. You simply have to get the tec_rule process to make trackable system call such as opening or closing a file. They also go into no detail about how complex the rulebase is. How large is the event cache. In my experience TEC running good rules that perform only one cache search to do duplicate/cause/effect/clearing processing on an 10K cache should handle 40 to 50 events per second. Also, just because you have room for 10K events does not mean you have to have 10K events in your cache, keep your cache lean.

What I found to be a limiting factor with the non-tme rule engine was the network. Think about it, hundreds of connections negotiated, event data transmitted, and the connections torn down a second. It would be very easy to saturate a 100mb network or system TCP/IP stack. I think the same will hold true for Netcool.

I think what we TEC guys think of as "deep" correlation and "enrichment" will be handled by the Netcool "Impact" product. At the conference this year there was mention of "entitlements" when IBM converges the correlation environment. Customers should contact thier sales reps to find out what they will be entitled to down the road. Impact may cost more... or may not... we will have to wait and see.

So what makes TEC so slow? IIHTG, it's the dependency on a database (accessed by two separate processes) through and absraction pipe of the RIM coupled with the overhead of IPC calls to get event data from tec_reception to tec_rule to tec_dispatch.

I would like to see some performance numbers on the Netcool products with Impact running

 

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019