TEC Troubleshooting

If tec console event viewer is not working you can export display and run tec_console & to get it from TMR server

Troubleshooting of TEC server involves several steps:

• Step 1: TEC processes present and possible anomalies

• Step 2: Send a message using postemsg
• Step 3: Check event log using  wtdumprl
• Step 4: Check database
• Step 5: Check rulebase

Note: Networkin issues can be trobleshooted with WireShark, see Utilization of software tool WireShark to Trouble shoot TEC issues

Audience: Level 2 Support, Services, GTS, Business Partners, Customers Abstract: This STE will cover: - Download of Wireshark - Demo of some of the various functions of this tool - Diagnosis of failed connection(s) Presented by: Calvin Moody and Jerry Swan Date: August 25, 2009 11:00 AM Eastern US

Step 1: TEC processes present and possible anomalies

The event server architecture consists of five processes:

You can get the list by using, for example, the command ps -ef | grep tec_

# ps -ef | grep tec_
root 356374 102820 0 Feb 12 - 0:23 tec_gateway
root 839906 1126590 0 18:41:07 - 0:03 tec_dispatch -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
nobody 913592 102820 0 Feb 12 - 1:01 tec_ui_server
root 1105976 1126590 0 18:41:07 - 0:00 tec_reception -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
root 1126590 102820 0 18:23:05 - 0:00 tec_server
root 65910 1126590 0 18:41:07 - 0:02 tec_rule -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
root 495892 1126590 0 18:41:07 - 0:03 tec_task -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
root 672168 782540 0 20:57:03 pts/2 0:00 grep tec_

An additional process, the  tec_ui_server process, plays an integral part in the overall Tivoli Enterprise Console architecture. The UI server is not a part of the event server, although it communicates with the event server. Each process has its own log file.

The first check is to see all those process are present and healthy (consume CPU time), for example there are couple of strange things in the following listing: 

    root 1560692 1839236   0 18:15:20      -  1:40 tec_reception -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
    root 1699860 1839236   0 18:15:20      -  4:59 tec_dispatch -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
    root 1839236 1777810   0 18:14:59      -  0:00 tec_server
    root 1855638 1777810   0 18:15:19      -  0:00 tec_gateway

If the tec_server event server process is no longer responding, restart TEC server using the wstopesvr && wstartesvr command. To check the status of the event server, use the wstatesvr command.

# wstatesvr
The Tivoli Enterprise Console Server is running.

When the event server is restarted, event consoles that were connected to it are automatically reconnected. For details about using the wstartesvr or wstatesvr commands, refer to the IBM Tivoli Enterprise Console Command and Task Reference.

root@nti2171/etc/Tivoli # wstatesvr
The Tivoli Enterprise Console Server is running.

You can also waive dead chicken and restart the TEC server as the first step in troubleshooing (just in case).  This is a quick operation which does not hurt and might even help...

# wstopesvr && wstartesvr

The Tivoli Enterprise Console Server is initializing...
The Tivoli Enterprise Console Server is running.

Step  2: Send a message using postemsg

If TEC server started OK but the problem persist you need to try to send messages to the TEC server and see if they are processed.

Do it first from TMR using wpostemsg and and then from endpoint using both wpostemsg and postemsg and see if they are received. Use the name of the server as the text of the massage to distibuwish beween them

wpostemgs -m "tmr_server wpostemgs" EVENT EVENT

wtdumprl -o DESC | more

### EVENT ###
EVENT;source=EVENT;severity=HARMLESS;msg='tmr_server wpostemgs';origin=10.201.13.227;END

Step 3: Check event log using wtdumprl

Description

If wtdumprl is run from a node other than the Tivoli Enterprise Console server, it uses the time from the local system to determine which events to display, and this might cause unexpected behavior.

For example, if the time on the node is 9:00 and the Tivoli Enterprise Console server is 9:30, a wtdumprl run from the node displays every event in the database except for those occurring during the 30 minutes specified. The same command run on the Tivoli Enterprise Console server would display the entire database.

Authorization

Options

-e end_time

Lists events that were received prior to the specified date and time. The end_time parameter must be a date in the format of "Mon dd hh:mm:ss yyyy". If this flag is omitted, the command uses the current time as the end time.

-f file

Writes output to the specified file.

-m number

Specifies the maximum number of events to record in the report. If the number of events in the log exceeds the specified value, the command omits entries from the end of the report. For example, if the report is displayed in ascending order, the most recent database entries are not included in the report.

-o ASC | DESC

Sets the order in which events are listed. The possible values are:

ASC

Lists the oldest log entry first. This is the default value.

DESC

Lists the most recent log entry first.

-t start_time

Lists events that were received after the specified date and time. The start_time parameter must be a date in the format of "Mon dd hh:mm:ss yyyy".

Examples

wtdumprl -t "MAR 09 19:00:00 1999" -e "MAR 12 09:00:00 1999"

Step 4: Check database and RIM

Use wrimtes to verifies an RDBMS Interface Module (RIM) object's connectivity and functionality.

wrimtest [-l RIM_object_label]

for example

wrimtest -l tec

A prompt enables you to enter one of the following command options:

c: RIM_commit 

Commits a transaction.

d: RIM_delete

Deletes a row from the database.

e: RIM_execute_sql

Executes a Structured Query Language (SQL) statement.

g: RIM_retrieve_rows

Retrieves rows from the database.

i: RIM_insert_rows

Inserts a row into the database.

r: RIM_rollback

Cancels a transaction.

u: RIM_update_rows

Updates rows in the database.

?: Help List

Prints a list of command options.

x: Exit

Exits the wrimtest utility.

It might be that you run out of space in your database. This is a typical Sunday problem when most messages are not closed.  To check this try:

# wtdbspace

You can clean event database with the command:

 # wtdbclear -left 0

There is also a useful command

wtdbmaint reorg  | stat

• For troubleshooting possible database and RIM problems, see Problems with the RIM database.

Step 5: Check rulebase

If you suspect problems with the rule base:

• Determine when rulebase was last modified, by whom and what were the changes
• load and run the default rule base or a known working rule base as a test. If this confirms a problem with the rule base, see Problems with rules and the IBM Tivoli Enterprise Console Rule Developer's Guide for additional troubleshooting information.
• enable trace

NEWS CONTENTS

• 20091006 : Utilization of software tool WireShark to Trouble shoot TEC issues ( Utilization of software tool WireShark to Trouble shoot TEC issues, Oct 6, 2009 )
• 20090308 : IBM - FRWTE0017E; system problem; T-EC Rule exit code 211 SIGSEGV ( IBM - FRWTE0017E; system problem; T-EC Rule exit code 211 SIGSEGV, Mar 8, 2009 )
• 20090308 : msg00495 ( msg00495, )
• 20090308 : developerWorks Tivoli PACO - Performance, Availability, Configuration, and Operations Windows Log file adapter ... ( developerWorks Tivoli PACO - Performance, Availability, Configuration, and Operations Windows Log file adapter ..., )

Old News

[Oct 6, 2009] Utilization of software tool WireShark to Trouble shoot TEC issues

Audience: Level 2 Support, Services, GTS, Business Partners, Customers Abstract: This STE will cover: - Download of Wireshark - Demo of some of the various functions of this tool - Diagnosis of failed connection(s) Presented by: Calvin Moody and Jerry Swan Date: August 25, 2009 11:00 AM Eastern US

[Mar 8, 2009] IBM - FRWTE0017E; system problem; T-EC Rule exit code 211 SIGSEGV

Technote (troubleshooting)

Problem(Abstract)

After enabling a rules trace, the TEC Server either will startup briefly without an error and die within 2-3 minutes, or will not start up and crash. FRWTE0017E (17): system problem: `ERROR: Process T/EC Rule exits with exit code 211: signal SIGSEGV'

Cause

This can be caused by the attempted migration of an older (TEC 3.6.x) rulebase to TEC 3.8 or 3.9 without re-writing the rulebase to change any predicates that are no longer supported in later versions of Tivoli Enterprise Console. (In effect, TEC 3.8/3.9 already has a get_time() predicate where TEC 3.6.x did not and the defining of two such predicates is what is causing the problem when rules tracing is enabled.)
After enabling a rules trace on the TEC rulebase, an error occurs on startup of the TEC Server where the TEC Server briefly starts, but is unable to stay up or the following error occurs : Error::FRWTE0017E Tue Jun 14 15:31:01 CDT 2005 (17): system problem: `ERROR: Process T/EC Rule exits with exit code 211: Process terminated by signal SIGSEGV'

> wstartesvr
The Tivoli Enterprise Console Server is initializing...
Error::FRWTE0017E Tue Jun 14 15:31:01 CDT 2005 (17): system problem:
`ERROR: Process T/EC Rule exits with exit code 211: Process terminated
by signal SIGSEGV'

Resolving the problem

The following information should be helpful in understanding this behavior:
Problem: (IY72851) tec_rule process crashes when compiled with trace.
The problem was that a custom get_time predicate was conflicting with the built-in get_time predicate.

Solution: When creating a custom predicate a good coding practice is to verify that the custom predicate name is not the same as an existing predicate either custom or built-in. This can be done by reviewing the documented built-in predicates in the Tivoli Enterprise Console Rule Developer's Guide and using the flisting predicate to query the Prolog database of the current rule base to see if a given predicate name exists. To use the flisting predicate, add an action to a rule which evaluates a 'TEC_Start' event:

rule: check:
(
event: _event of_class ['TEC_Start'],

reception_action: check_predicates: (
fopen(_fp, '/tmp/check_predicate.pro',w),
flisting(_fp, 'my_predicate')
fclose(_fp)
)
).

If the output file contains an entry for 'my_predicate', then a predicate with that name exists in the current rule base.

msg00495

-----Original Message-----
From: owner-tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx [mailto:owner-tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx]On Behalf Of Jeff.Richards-K67OQRCp/lRBDgjK7y7TUQ@xxxxxxxxxxxxxxxx
Sent: Friday, January 28, 2005 9:47 AM
To: tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx
Subject: [tme10] TEC Reception Cache Full


I guess this is a configuration/performance issue.

Here's what I saw in an event dump:

1~3183448~65537~1106906597(Jan 28 05:03:17 2005)
### EVENT ###
TEC_Notice;source=TEC;msg="Reception Cache Full: NON-TME Events: Reception Log E
nabled. Events saved in WAITING state";severity=CRITICAL;END
.
### END EVENT ###
PROCESSED

1~3184146~65537~1106909724(Jan 28 05:55:24 2005)
### EVENT ###
TEC_Notice;source=TEC;msg="Reception Cache Full: NON-TME Events: Reception Log E
nabled. Events saved in WAITING state";severity=CRITICAL;END
.
### END EVENT ###
PROCESSED

Here's what I saw in /tmp/tec_rule

Jan 28 05:55:22.348559 tec_rule[48154] ERR tec_msg.c:45: No message type: 1, T
Jan 28 05:55:22.354526 tec_rule[48154] ERR tec_pool.c:406: Could not receive msg
: msg does not contain msgtype

Jan 28 06:19:31.097860 tec_rule[48154] ERR tec_msg.c:45: No message type: 1, T
Jan 28 06:19:31.098340 tec_rule[48154] ERR tec_pool.c:406: Could not receive msg
: msg does not contain msgtype

Any help would be appreciated.

Thanks.

developerWorks Tivoli PACO - Performance, Availability, Configuration, and Operations Windows Log file adapter ...

Windows Log file adapter

Posted: Apr 30, 2004 11:24:49 AM

I've been struggling for the past couble of days trying to get my newly created events to show up in TEC and was hoping someone out there could lend a hand.

Here's the situation, I found the location of my current rule base using the following:

wrb -lscrurrb ===> returns "Default"
wrb -lsrb -path Default ===> returns "70w2ks:C:/PROGRA~1/Tivoli/bin/w32-ix86/TME/TEC/default_rb"

I go to 70w2ks:C:/PROGRA~1/Tivoli/bin/w32-ix86/TME/TEC/default_rb/TEC_CLASSES and add
a file called xpoint.baroc containing the following:

TEC_CLASS :
RREBackupStart ISA NT_Base
DEFINES {
message: STRING;
clientName: STRING;
};
END

I add xpoint.baroc to .load_classes after tecad_nt.baroc. Recompile my rule base using the following:

wrb -comprules Default
wrb -loadrb Default

Recycle my TEC server using wstopesvr and wstartesvr.

Next I created an ACP profile and used ACF to create the tecad_win adapter, including the log file format editor.
The format of my event in tecad_win.fmt file looks like the following:

FORMAT RREBackupStart FOLLOWS NT_Base
%t %s %s %s %s %s %s RRE: Backup Starting on client: %s
clientName $8
message PRINTF("RRE: Backup Starting on client: %s", clientName)
END

I push the profile to my endpoint using the distribute menu option, start tecad_win -d, and append the following statement to
my log file:

Mar 7 13:06:19 2004 0 HARMLESS N/A LogFile 8000 RRE: Backup Starting on client: DELL-1100-01

The following is printed by tecad_win:

matched RREBackupStart
$1 is `Mar 7 13:06:19'
$2 is `2004'
$3 is `0'
$4 is `HARMLESS'
$5 is `N/A'
$6 is `LogFile'
$7 is `8000'
$8 is `DELL-1100-01'

All looks good. I now stop tecad_win and start the service using "net start TECWinAdapter" and add the exact same line to my log file.

I now go to the TEC server and use "wtdumprl -o DESC -m 5" to view the reception log. Nothing prints. To verify that events are
flowing to TEC from the logfile I create an event in proper format for a NT_Audit_Policy_Change event. This event is displayed properly
by both tecad_win and by wtdumprl, thus verifying that the log file adapter is configured and working fine. What is not working is
my newly created events.

I have the following questions:
(a) Are there some other tools or log files I can monitor to determine whether the event is being sent from endpoint and received by the TEC server?
(b) According to my interpretation of the documentation, the tecad_win should still forward the events from the endpoint to the TEC server.
But this does not appear to be the case based on my personal observation. Do I have something setup wrong, is it broken, or am I misreading
the documentation?
(c) Did I miss a step somewhere?

Thanks in advance,

Ken

Recommended Links

Debugging Tivoli Enterprise Console 3.9 logfile adapter problems by Sandi Comsudi

Plus Module User's Guide - good description of integration of TWS with TEC using logfile adapter.

Adding a new message to a Logfile Adapters Provides an explanation of how to add new messages to be processed by the Unix Logfile adapter or the NT Event Log Adapter.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019