TCP Troubleshooting Case studies

• Duplicate IP Address
• Duplicate MAC Address
• Faulty Cable
• Multi-Homed System Acts as Rogue
• TCP Treason Uncloaked
• Broadcom NetXtreme Ethernet card random disconnects on Linux
• Possible SYN flooding on port 25. Sending cookies
• Which Process Listens to Particular Port

Apache

See also Apache troubleshooting

mod_status

mod_status is an Apache module that can show an HTML page representing various information about the internal status of Apache. This includes number of httpds, their current status, network connections, amount of traffic, etc.

Very useful when trying to track down performance related issues.

module debugging

Some Apache httpd modules include options to enable extra debugging info. Unfortunately, this seems to depend on the module.

log files

Log files, the httpd error logs in particular (typically in /var/log/httpd/error_log), are often the best place to look when troubleshooting. It's also where any module debugging information will log to.

Testing the configuration file for syntax errors

Apache comes with an executable called apachectl(8). This program can run a configuration check on Apache's configuration files by issuing the command

  apachectl configtest

Some distros (like RedHat/Fedora) also include this command in Apache's init script and invoke apachectl in the background.

-X debug mode

One of the biggest problems with trying to track down problems with apache httpd is the multiprocess nature of it. It makes it difficult to strace or to attach gdb.

To force httpd to run in a single process mode start it with:

       httpd -X

Note that on Red Hat linux boxes you probably need to include the commandline arguments that the init scripts start httpd with. The easiest way to do this is to start httpd normally, then run `ps auxwwww` and cut and paste one of the httpd commandline lines.

PHP

The following assumes that you know PHP coding.

The most informative (but also most disruptive in a visual sense) thing to do is set

  error_reporting = E_ALL

in your php.ini (under debian: /etc/php/<calling entity>/php.ini). Remember to restart your webserver/calling entity after changing this setting. If you come from the C corner of things, you'll know that good programming style dictates that you treat warnings and notices as errors. So off you go, clean up that code!

Back and still not working? Ok, now it gets ugly. PHP doesn't come with a debugger like `gdb`. Such things exist, but usually they will be embedded in an IDE that also emulates a web server and costs $$$. So basically you get to do stuff just as in regular shell scripts: debug echos. Echo early, echo often. Hand in hand with echo statements comes the print_r function, which will print arrays/hashes (same thing in PHP) recursively. Drawback here: print_r formats in plain ASCII, not HTML. So you'll either have to look at the page source to see a clean version of the output, or do something ugly like

 echo join( "<br>", print_r($myarray) );

 FIXME: can you turn on warnings about variables only used once, like in perl? One of my most
 frequent errors....

iptables

I have a Windows VPN Client behind an Linux Gateway doing NAT and I can't connect to the server

First things first, you'll want to know what kind of Windows VPN tunnel you're building. The following will assume the standard PPTP tunnel.

 FIXME: What about l2tp tunnels?

First things first, you need rules that allow the forwarding of the used connections and rules for NATing. The tricky part here is that the PPTP tunnel uses two connections: one going to tcp/1723 on the server, and one GRE tunnel (meaning you can only have one PPTP NATting session active on the gateway at a time). So you'll need the following rules to allow the forwarding:

 iptables -A FORWARD -p tcp --dport 1723 -d vpn-server-address -j ACCEPT
 iptables -A FORWARD -p gre -d vpn-server-address -j ACCEPT

and the NATting is handled by these rules:

 iptables -t nat -A PREROUTING -p tcp --sport 1723 -s vpn-server-address -j DNAT --to-dest vpn-client-ip:1723
 iptables -t nat -A PREROUTING -p gre -s vpn-server-address -j DNAT --to-dest vpn-client-ip
 iptables -t nat -A POSTROUTING -p tcp --dport 1723 -d vpn-server-address -j SNAT --to-source gateway-public-ip
 iptables -t nat -A POSTROUTING -p gre -d vpn-server-address -j SNAT --to-source gateway-public-ip

If you're still having trouble connecting, and Windows is giving you an error 721 (or, if you're looking at the data flow with tcpdump and you're seeing the 1723/tcp connection working fine, but the GRE tunnel connection not working because for some reason the source IP of the GRE tunnel is the private ip of the machine running the vpn client), you will need to build the PPTP connection tracking module for the linux kernel (as of 2.6.x?) and insert the following to modules:

 modprobe ip_conntrack_pptp
 modprobe ip_nat_pptp

Now everything should be working as expected.

SSH

Most problems occur here when you're trying to set up logins via RSA/DSA keys (and probably without passwords too...). It's usually down to basics: Make sure that your ~/.ssh is owned by your user and set to mode 600. ~/.ssh/authorized_keys has to be set to 0600. If these basic conditions aren't met, sshd will refuse to even look at your authorized_keys file and drop you back to password logins.

Another word about the format of the authorized_keys file: it's one key per row. Make sure that your added keys are in a single row! vi is notorious for adding linebreaks if you have 'tw' set in your ~/.vimrc and use copy and paste to add a new key to the file. Use cat or ssh-copy-id instead.

You can run

  ssh -v fred@godot

to see what SSH is up to and where things start hickupping. You can go all the way up to

  ssh -vvv fred@godot

if you really want to know about how modulo groups are being prodded. Usually -vv suffices.

I just updated my openssh packages and now I can't login

If the error message is something like "Upsupported Protocol - Remote host closed the connection", it's probably due to an incompatibility between OpenSSH 4.2 and anything pre-4.2. If you have the server under your control, the solution is easy: Update the server to the 4.2 version as well (recommended as there are some nasty zlib buffer overruns in pre-4.2 anyway).  Try

sshd -d -D 

Kerberos

When something goes wrong with Kerberos, it's usually down to a few things: - Something in the network topology changed, mandating that you re-check your /etc/krb5.conf - Your Kerberos server is unreachable - You entered a wrong password while generating a keytab file or the associated user/service name is not known to the server.

Unfortunately, tools like kinit(1) do have a -v option for verbose output, but this only starts outputting useful information after they aquire a TGT from the KDC. It's more useful to watch the logs of the KDC and see what (if anything) actually happens there.

/etc/krb5.conf

This configuration file is read and used by the Kerberos libraries, so any settings here affect everything on your system that uses Kerberos. The most important setting is

 [realms]
 <YOUR DEFAULT REALM> = {
   kdc = <IP of your KDC>
 }

There may be several realm definitions within the [realms] section. Be sure that you set the correct IP here. Otherwise your Kerberos requests will just hang there and time out after a while.

The second most important setting is

 [libdefaults]
 default_realm = <YOUR DEFAULT REALM>

This specifies what realm Kerberos tools will use if no explicit realm is given for a request.

Finally, if you're fooling around with a KDC that resides on a Windows2003 server, be sure that you've enabled arcfour-hmac-md5 and des-cbc-crc as cryto algorithms for the settings default_tgs_enctypes, default_tkt_enctypes and permitted enctypes in the [libdefaults] section. Otherwise your keytab files will be unreadable.

sendmail

The first question asked might be, “How do we count the total number of messages that flow through the system?” This question isn’t as simple to answer as it might seem. The answer depends on whether one wants to count the number of SMTP connections, the number of unique messages as sent by a sender, or the number of messages (often sent to multiple recipients) that end up in someone’s mailbox somewhere. Each of these metrics is a valid choice, but in my judgment the bulk of the work is done for each successful message recipient, so I generally choose to count the number of syslog entries with both the to= pattern and stat=Sentin them. I call that measure the number of messages that the system has successfully processed, mindful that it is merely one statistic that is much more nebulous than it appears at first glance.

If we consider the format of these log entries to contain a set of fields delimited by whitespace, the first three fields contain information about the date and time when the log entry was made. This information can be parsed to track the busiest time of day for the server. In the from entry, the eighth field contains the size of the message in bytes, which we can use to find out the average message size handled by the system. On the same entry, the tenth field lists the number of recipients per message, another interesting statistic to track. In the toentry, the information in the delay and xdelay fields are of particular interest. The delay field measures the total amount of elapsed time between the receipt of the message and this particular delivery attempt. The xdelayfield, which stands for transaction delay, measures the amount of time consumed on this particular delivery attempt, which should reveal something about the current connectivity to a particular site.

A great deal more information available in the logs can be extracted for various purposes, but at this point the next step will be left to the imagination of the reader.

A similar set of information can be extracted from the logs left by any of the POP or IMAP daemons discussed in this book. Combined with other statistical information gathered with the tools described here, one can plot number of processes versus load average, connection rates versus disk activity, and so on to obtain a thorough understanding of any email server’s performance. These checks can be easily automated, and at least the most basic ones should be part of an email administrator’s baselining effort.

Sendmail Disgnostic Interpretation

Sendmail Log Formats

Using syslog for Sendmail troubleshooting

Mail Relay Testing

• http://www.tech-recipes.com/sendmail_tips246.html
• http://docs.hp.com/en/B2355-90685/ch04s11.html

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019