In short: When mail sticks in the queue, work the path in order: service → config → queue → DNS → port 587 → auth/TLS → logs.
Prerequisites
Before beginning this guide, you should have completed the previous articles in this series and have a Sendmail installation configured as an SMTP client.
- Article 1 – Understanding Email Architecture in AIX
- Article 2 – Understanding Sendmail in AIX
- Article 3 – Installing Sendmail on IBM AIX
- Article 4 – Configuring Sendmail as an SMTP Client
- Article 5 – Configuring Amazon SES SMTP Authentication
Although this article uses Amazon SES as an example, the troubleshooting techniques described here apply to most SMTP relay services, including Microsoft Exchange Online, Postfix, SendGrid, and other enterprise mail gateways.
1. A Systematic Troubleshooting Approach
When email delivery fails, it's tempting to immediately edit configuration files or restart services. In practice, this often makes troubleshooting more difficult because it introduces new variables before the original problem has been identified.
A better approach is to troubleshoot the same path every message follows (see Article 1): application → mail/mailx → Sendmail → queue → SMTP → relay → mailbox. Check each stage before changing config.
If an email reaches the mail queue but never leaves the server, the problem is likely related to SMTP connectivity or authentication. If the message never reaches the queue, the issue may be with the application or the local Sendmail configuration.
Never troubleshoot Sendmail by guessing. Always determine the exact stage at which email delivery stops, then focus your investigation on that specific component.
2. Verify the Sendmail Service
Before checking SMTP authentication or network connectivity, confirm that the Sendmail daemon is running correctly.
IBM AIX manages Sendmail through the System Resource Controller (SRC).
Display the current service status using:
lssrc -s sendmail
A healthy service typically reports a status similar to:
Subsystem Group PID Status
sendmail mail 123456 active
If the daemon is not active, start it using:
startsrc -s sendmail
After modifying the Sendmail configuration, reload the service so the changes take effect.
refresh -s sendmail
If a complete restart is required, stop and start the daemon.
stopsrc -s sendmail
startsrc -s sendmail
Always verify that Sendmail starts successfully after making configuration changes. A syntax error in sendmail.cf or an invalid configuration directive can prevent the daemon from functioning correctly.
3. Validate the Configuration Files
Once the service is confirmed to be running, verify that the required configuration files are present and contain the expected settings.
| Configuration File | Purpose |
|---|---|
| /etc/mail/sendmail.cf | Main Sendmail configuration. |
| /etc/mail/authinfo | SMTP authentication credentials. |
| /etc/mail/authinfo.db | Hashed authentication database. |
| /etc/mail/submit.cf | Local mail submission configuration. |
| /etc/mail/aliases | Mail alias definitions. |
One of the most common mistakes is updating the authinfo file without rebuilding the authentication database.
makemap hash /etc/mail/authinfo.db \
< /etc/mail/authinfo
Remember that Sendmail authenticates using authinfo.db, not the plain-text authinfo file. If the database is not regenerated after making changes, Sendmail continues using the previous credentials.
Whenever SMTP credentials are changed, verify both the contents of authinfo and the timestamp of authinfo.db to ensure the authentication database has been rebuilt.
4. Inspect the Mail Queue
The mail queue is one of the most valuable diagnostic tools available to a Sendmail administrator. Every message that cannot be delivered immediately is placed into the queue for later processing.
Display the queue using either of the following commands.
mailq
or
sendmail -bp
Interpreting the queue correctly often provides the first indication of where the delivery process is failing.
| Queue Status | Meaning |
|---|---|
| Queue Empty | No pending messages. Delivery is completing successfully. |
| Queued | Message is waiting for the next delivery attempt. |
| Deferred | Temporary error encountered during delivery. |
| Frozen | Message requires administrative attention before retrying. |
If the number of queued messages continues to increase, the underlying issue is typically related to one of the following:
- SMTP authentication failure.
- SMTP relay unavailable.
- Network connectivity problems.
- DNS resolution failure.
- Firewall restrictions.
Queued mail is either delivered, deferred for retry, or frozen for admin review — use mailq and queue IDs to see which path a message took.
Checking the mail queue should be one of the first troubleshooting steps whenever users report missing email notifications. The queue often contains valuable information about why delivery has been delayed or deferred.
5. Troubleshooting SMTP Authentication Errors
Once Sendmail has successfully connected to the SMTP relay, the next stage is authentication. This is one of the most common points of failure when integrating Sendmail with Amazon SES or any authenticated SMTP relay.
Authentication problems usually occur because of incorrect SMTP credentials, an outdated authentication database, or an incorrect Sendmail configuration.
Common SMTP authentication errors include:
| SMTP Error | Description | Recommended Action |
|---|---|---|
| 530 Authentication Required | The SMTP server requires authentication before accepting mail. | Verify that SMTP authentication is configured correctly. |
| 535 Authentication Credentials Invalid | The supplied username or password is incorrect. | Regenerate Amazon SES SMTP credentials and rebuild the authentication database. |
| 535 Authentication Failed | Authentication was attempted but rejected. | Verify the SMTP username, password, and AuthInfo configuration. |
| 554 Transaction Failed | The SMTP server rejected the message. | Review the relay configuration and sender restrictions. |
Whenever SMTP credentials are changed, regenerate the authentication database using makemap before restarting Sendmail.
makemap hash /etc/mail/authinfo.db \
< /etc/mail/authinfo
refresh -s sendmail
6. Troubleshooting DNS Resolution
Before Sendmail can communicate with an SMTP relay, it must resolve the relay hostname to an IP address. If DNS resolution fails, Sendmail cannot establish a connection regardless of whether the configuration is correct.
Verify DNS resolution using one of the following commands.
host email-smtp...amazonaws.com
or
nslookup email-smtp...amazonaws.com
A successful lookup should return one or more IP addresses.
Name:
email-smtp...amazonaws.com
Address:
xx.xx.xx.xx
If the hostname cannot be resolved, verify the following:
- DNS servers configured in /etc/resolv.conf.
- Network connectivity to the configured DNS servers.
- Correct SMTP relay hostname.
- Firewall rules preventing DNS traffic.
Many SMTP problems that appear to be Sendmail issues are actually caused by incorrect DNS configuration.
7. Verifying Network Connectivity
After confirming DNS resolution, verify that the AIX server can reach the SMTP relay on the required port. For Amazon SES and many cloud relays, ICMP (ping) is often blocked, so prefer testing the SMTP port directly.
telnet email-smtp...amazonaws.com 587
If telnet is unavailable, use nc (Netcat) or OpenSSL:
nc -vz email-smtp...amazonaws.com 587
openssl s_client -connect email-smtp...amazonaws.com:587 -starttls smtp
Optional: ping can confirm basic IP reachability when ICMP is allowed, but a failed ping does not prove the SMTP port is blocked.
ping email-smtp...amazonaws.com
A successful TCP connection on port 587 indicates the relay is reachable and that routing and firewall policy allow SMTP traffic.
| Result | Meaning |
|---|---|
| Connected | SMTP relay is reachable. |
| Connection Refused | SMTP service unavailable or incorrect port. |
| Timed Out | Firewall, routing, or network issue. |
8. Troubleshooting STARTTLS and TLS Issues
Modern SMTP relays, including Amazon SES, require encrypted communication before authentication. Problems during the TLS negotiation stage prevent Sendmail from authenticating successfully.
Use OpenSSL to test the SMTP STARTTLS handshake.
openssl s_client \
-connect email-smtp.ap-south-1.amazonaws.com:587 \
-starttls smtp
A successful connection displays certificate information followed by SMTP server responses.
Common TLS-related errors include:
| Error | Possible Cause |
|---|---|
| 454 TLS Not Available | STARTTLS is not supported or incorrectly configured. |
| Certificate Verify Failed | Missing or untrusted Certificate Authority (CA). |
| Handshake Failure | TLS version or cipher mismatch. |
Always verify the TLS handshake independently using openssl s_client before modifying Sendmail configuration. This helps determine whether the issue lies with Sendmail or the underlying network and certificate infrastructure.
9. Reviewing Sendmail Logs
When troubleshooting becomes more complex, log files provide the most detailed information about the mail delivery process.
Depending on your AIX configuration, Sendmail activity may be recorded through syslog, syslog-ng, or another system logging service.
Useful locations include:
| Location | Description |
|---|---|
| /var/adm/syslog | Default AIX syslog file when mail.* is directed here (confirm in /etc/syslog.conf). |
| /var/log/maillog | Optional: only if you explicitly configured a Linux-style mail log path. |
| errpt | AIX operating system and device errors that may affect mail delivery. |
Always verify the active mail facility destination:
grep -E '^[^#]*mail\.' /etc/syslog.conf
tail -f /var/adm/syslogThe errpt command is particularly useful for identifying operating system or network-related problems that may indirectly affect Sendmail.
errpt -a
When reviewing logs, pay particular attention to:
- SMTP authentication failures.
- DNS lookup failures.
- Connection timeout messages.
- TLS negotiation errors.
- Repeated delivery retries.
Do not rely on a single error message when troubleshooting. Compare Sendmail logs, mail queue status, DNS results, network connectivity, and operating system logs together. Looking at the complete picture usually identifies the root cause much faster than investigating each component in isolation.
10. Real-World Troubleshooting Scenarios
Even with a correct Sendmail configuration, production environments can present unexpected challenges. The following scenarios are based on common issues encountered when integrating Sendmail with enterprise SMTP relay services such as Amazon SES.
Scenario 1 – Email Remains in the Mail Queue
Users report that email notifications are not being received, but no errors are displayed when using the mail command.
The first step is to inspect the mail queue.
mailq
If messages remain queued, verify the following:
- Sendmail service is running.
- SMTP relay hostname resolves correctly.
- Port 587 is reachable (telnet/nc/openssl; do not rely on ping alone for SES).
- SMTP authentication is configured correctly.
- The SMTP relay is accepting connections.
Brief variants
Auth failed (535): Check username/password, regenerate SES credentials if needed, rebuild authinfo.db with makemap, then refresh -s sendmail.
Connection timed out: Test port 587 with nc -vz or openssl s_client (SES often blocks ICMP). Check firewall, routing, and the SMTP endpoint hostname.
DNS failure: host email-smtp.ap-south-1.amazonaws.com; verify /etc/resolv.conf and reachability to DNS servers.
STARTTLS failure: openssl s_client -connect …:587 -starttls smtp; check CA certs, TLS version, and Sendmail TLS settings.
Isolate one problem at a time. Resolve DNS, network, and TLS before deep-diving into authentication.
11. Sendmail Troubleshooting Flow
The following workflow can be used whenever email delivery fails.
Email Failed
|
v
Is Sendmail Running?
|
Yes / No
|
v
Check Mail Queue
|
v
Queued?
|
v
Check DNS
|
v
Check SMTP Connectivity
|
v
Check STARTTLS
|
v
Check Authentication
|
v
Review Logs
|
v
Resolved
Following the same troubleshooting sequence every time reduces unnecessary configuration changes and helps identify the root cause more efficiently.
12. Production Troubleshooting Checklist
Before escalating an email delivery issue, complete the following checklist.
- Sendmail daemon is active.
- SMTP relay hostname resolves correctly.
- SMTP port is reachable.
- Mail queue inspected.
- Authentication database rebuilt.
- TLS handshake verified.
- System logs reviewed.
- Test email sent.
Maintain a documented troubleshooting checklist within your operations team. Following a standard process ensures consistency during production incidents and reduces recovery time.
13. Troubleshooting Best Practices
- Always begin with the Sendmail service status.
- Verify DNS before investigating SMTP authentication.
- Inspect the mail queue before modifying configuration files.
- Confirm network connectivity to the SMTP relay.
- Use OpenSSL to verify the STARTTLS handshake independently.
- Review both Sendmail logs and AIX system logs.
- Keep backup copies of Sendmail configuration files before making changes.
- Test configuration changes on a development system whenever possible.
14. Summary
Troubleshooting Sendmail is easier when you follow the delivery path and verify each stage—service, queue, DNS, network, auth, TLS, and logs—rather than changing configuration at random.
- Verified Sendmail service
- Validated configuration files
- Inspected the mail queue
- Verified DNS resolution
- Tested SMTP connectivity
- Diagnosed authentication failures
- Verified STARTTLS communication
- Reviewed production troubleshooting practices
Key Takeaways
- Always troubleshoot using the complete email delivery path.
- The mail queue is one of the most valuable diagnostic tools.
- DNS, network connectivity, authentication, and TLS should be verified in that order.
- Use log files and command-line tools together to identify the root cause.
- A structured troubleshooting process significantly reduces production recovery time.
15. Frequently Asked Questions
Why are emails staying in the Sendmail queue?
Messages usually remain queued because Sendmail cannot contact the SMTP relay, authentication has failed, DNS resolution is unavailable, or a firewall is blocking the connection.
How can I verify that Sendmail is running?
Use the following command:
lssrc -s sendmail
A status of active indicates that the daemon is running.
What is the first thing I should check when email delivery fails?
Start by verifying the Sendmail service, then inspect the mail queue. These two checks usually identify whether the problem is local or related to the SMTP relay.
How can I test the SMTP relay without sending an email?
Use openssl s_client to verify the STARTTLS handshake or use nc (Netcat) to confirm that the SMTP port is reachable.