What You'll Learn
- How to troubleshoot Sendmail using a systematic approach.
- How to verify that the Sendmail service is running correctly.
- How to validate Sendmail configuration files.
- How to inspect and interpret the mail queue.
- How to identify where email delivery is failing.
- Best practices for troubleshooting Sendmail in production environments.
Prerequisites
Before beginning this guide, you should have completed the previous articles in this series and have a Sendmail installation configured as an SMTP client.
- Article 1 – Understanding Email Architecture in AIX
- Article 2 – Understanding Sendmail in AIX
- Article 3 – Installing Sendmail on IBM AIX
- Article 4 – Configuring Sendmail as an SMTP Client
- Article 5 – Configuring Amazon SES SMTP Authentication
Although this article uses Amazon SES as an example, the troubleshooting techniques described here apply to most SMTP relay services, including Microsoft Exchange Online, Postfix, SendGrid, and other enterprise mail gateways.
1. A Systematic Troubleshooting Approach
When email delivery fails, it's tempting to immediately edit configuration files or restart services. In practice, this often makes troubleshooting more difficult because it introduces new variables before the original problem has been identified.
A better approach is to troubleshoot the same path followed by every email message. By checking each stage individually, you can quickly determine where the delivery process is failing.
Application / Script
|
|
v
mail / mailx
|
|
v
Sendmail
|
|
v
Mail Queue
|
|
v
SMTP Connection
|
|
v
SMTP Relay
(Amazon SES)
|
|
v
Recipient Mail Server
|
|
v
Recipient Mailbox
If an email reaches the mail queue but never leaves the server, the problem is likely related to SMTP connectivity or authentication. If the message never reaches the queue, the issue may be with the application or the local Sendmail configuration.
Never troubleshoot Sendmail by guessing. Always determine the exact stage at which email delivery stops, then focus your investigation on that specific component.
2. Verify the Sendmail Service
Before checking SMTP authentication or network connectivity, confirm that the Sendmail daemon is running correctly.
IBM AIX manages Sendmail through the System Resource Controller (SRC).
Display the current service status using:
lssrc -s sendmail
A healthy service typically reports a status similar to:
Subsystem Group PID Status
sendmail mail 123456 active
If the daemon is not active, start it using:
startsrc -s sendmail
After modifying the Sendmail configuration, reload the service so the changes take effect.
refresh -s sendmail
If a complete restart is required, stop and start the daemon.
stopsrc -s sendmail
startsrc -s sendmail
Always verify that Sendmail starts successfully after making configuration changes. A syntax error in sendmail.cf or an invalid configuration directive can prevent the daemon from functioning correctly.
3. Validate the Configuration Files
Once the service is confirmed to be running, verify that the required configuration files are present and contain the expected settings.
| Configuration File | Purpose |
|---|---|
| /etc/mail/sendmail.cf | Main Sendmail configuration. |
| /etc/mail/authinfo | SMTP authentication credentials. |
| /etc/mail/authinfo.db | Hashed authentication database. |
| /etc/mail/submit.cf | Local mail submission configuration. |
| /etc/mail/aliases | Mail alias definitions. |
One of the most common mistakes is updating the authinfo file without rebuilding the authentication database.
makemap hash /etc/mail/authinfo.db \
< /etc/mail/authinfo
Remember that Sendmail authenticates using authinfo.db, not the plain-text authinfo file. If the database is not regenerated after making changes, Sendmail continues using the previous credentials.
Whenever SMTP credentials are changed, verify both the contents of authinfo and the timestamp of authinfo.db to ensure the authentication database has been rebuilt.
4. Inspect the Mail Queue
The mail queue is one of the most valuable diagnostic tools available to a Sendmail administrator. Every message that cannot be delivered immediately is placed into the queue for later processing.
Display the queue using either of the following commands.
mailq
or
sendmail -bp
Interpreting the queue correctly often provides the first indication of where the delivery process is failing.
| Queue Status | Meaning |
|---|---|
| Queue Empty | No pending messages. Delivery is completing successfully. |
| Queued | Message is waiting for the next delivery attempt. |
| Deferred | Temporary error encountered during delivery. |
| Frozen | Message requires administrative attention before retrying. |
If the number of queued messages continues to increase, the underlying issue is typically related to one of the following:
- SMTP authentication failure.
- SMTP relay unavailable.
- Network connectivity problems.
- DNS resolution failure.
- Firewall restrictions.
Application
|
|
v
Sendmail
|
|
v
Mail Queue
|
+-------------> Delivered
|
+-------------> Deferred
|
+-------------> Retry
|
+-------------> Frozen
Checking the mail queue should be one of the first troubleshooting steps whenever users report missing email notifications. The queue often contains valuable information about why delivery has been delayed or deferred.
5. Troubleshooting SMTP Authentication Errors
Once Sendmail has successfully connected to the SMTP relay, the next stage is authentication. This is one of the most common points of failure when integrating Sendmail with Amazon SES or any authenticated SMTP relay.
Authentication problems usually occur because of incorrect SMTP credentials, an outdated authentication database, or an incorrect Sendmail configuration.
Common SMTP authentication errors include:
| SMTP Error | Description | Recommended Action |
|---|---|---|
| 530 Authentication Required | The SMTP server requires authentication before accepting mail. | Verify that SMTP authentication is configured correctly. |
| 535 Authentication Credentials Invalid | The supplied username or password is incorrect. | Regenerate Amazon SES SMTP credentials and rebuild the authentication database. |
| 535 Authentication Failed | Authentication was attempted but rejected. | Verify the SMTP username, password, and AuthInfo configuration. |
| 554 Transaction Failed | The SMTP server rejected the message. | Review the relay configuration and sender restrictions. |
Whenever SMTP credentials are changed, regenerate the authentication database using makemap before restarting Sendmail.
makemap hash /etc/mail/authinfo.db \
< /etc/mail/authinfo
refresh -s sendmail
6. Troubleshooting DNS Resolution
Before Sendmail can communicate with an SMTP relay, it must resolve the relay hostname to an IP address. If DNS resolution fails, Sendmail cannot establish a connection regardless of whether the configuration is correct.
Verify DNS resolution using one of the following commands.
host email-smtp...amazonaws.com
or
nslookup email-smtp...amazonaws.com
A successful lookup should return one or more IP addresses.
Name:
email-smtp...amazonaws.com
Address:
xx.xx.xx.xx
If the hostname cannot be resolved, verify the following:
- DNS servers configured in /etc/resolv.conf.
- Network connectivity to the configured DNS servers.
- Correct SMTP relay hostname.
- Firewall rules preventing DNS traffic.
Many SMTP problems that appear to be Sendmail issues are actually caused by incorrect DNS configuration.
7. Verifying Network Connectivity
After confirming DNS resolution, verify that the AIX server can reach the SMTP relay over the required network port.
Begin by testing basic IP connectivity.
ping email-smtp...amazonaws.com
Next, verify that the SMTP port is reachable.
telnet email-smtp...amazonaws.com 587
If the telnet utility is unavailable, use nc (Netcat).
nc -vz email-smtp...amazonaws.com 587
A successful connection indicates that the relay is reachable and that network routing and firewall policies are allowing SMTP traffic.
| Result | Meaning |
|---|---|
| Connected | SMTP relay is reachable. |
| Connection Refused | SMTP service unavailable or incorrect port. |
| Timed Out | Firewall, routing, or network issue. |
8. Troubleshooting STARTTLS and TLS Issues
Modern SMTP relays, including Amazon SES, require encrypted communication before authentication. Problems during the TLS negotiation stage prevent Sendmail from authenticating successfully.
Use OpenSSL to test the SMTP STARTTLS handshake.
openssl s_client \
-connect email-smtp.ap-south-1.amazonaws.com:587 \
-starttls smtp
A successful connection displays certificate information followed by SMTP server responses.
Common TLS-related errors include:
| Error | Possible Cause |
|---|---|
| 454 TLS Not Available | STARTTLS is not supported or incorrectly configured. |
| Certificate Verify Failed | Missing or untrusted Certificate Authority (CA). |
| Handshake Failure | TLS version or cipher mismatch. |
Always verify the TLS handshake independently using openssl s_client before modifying Sendmail configuration. This helps determine whether the issue lies with Sendmail or the underlying network and certificate infrastructure.
9. Reviewing Sendmail Logs
When troubleshooting becomes more complex, log files provide the most detailed information about the mail delivery process.
Depending on your AIX configuration, Sendmail activity may be recorded through syslog, syslog-ng, or another system logging service.
Useful locations include:
| Location | Description |
|---|---|
| /var/log/maillog | Sendmail activity and SMTP transactions. |
| /var/log/messages | General operating system events. |
| errpt | AIX operating system errors. |
The errpt command is particularly useful for identifying operating system or network-related problems that may indirectly affect Sendmail.
errpt -a
When reviewing logs, pay particular attention to:
- SMTP authentication failures.
- DNS lookup failures.
- Connection timeout messages.
- TLS negotiation errors.
- Repeated delivery retries.
Do not rely on a single error message when troubleshooting. Compare Sendmail logs, mail queue status, DNS results, network connectivity, and operating system logs together. Looking at the complete picture usually identifies the root cause much faster than investigating each component in isolation.
10. Real-World Troubleshooting Scenarios
Even with a correct Sendmail configuration, production environments can present unexpected challenges. The following scenarios are based on common issues encountered when integrating Sendmail with enterprise SMTP relay services such as Amazon SES.
Scenario 1 – Email Remains in the Mail Queue
Users report that email notifications are not being received, but no errors are displayed when using the mail command.
The first step is to inspect the mail queue.
mailq
If messages remain queued, verify the following:
- Sendmail service is running.
- SMTP relay hostname resolves correctly.
- Port 587 is reachable.
- SMTP authentication is configured correctly.
- The SMTP relay is accepting connections.
Scenario 2 – SMTP Authentication Failed
Authentication errors usually appear as:
535 Authentication Credentials Invalid
Possible causes include:
- Incorrect SMTP username.
- Incorrect SMTP password.
- Expired or regenerated SMTP credentials.
- The authentication database was not rebuilt.
Recommended verification:
cat /etc/mail/authinfo
ls -l /etc/mail/authinfo.db
makemap hash /etc/mail/authinfo.db \
< /etc/mail/authinfo
refresh -s sendmail
Scenario 3 – Connection Timed Out
If Sendmail cannot establish a connection to the SMTP relay, verify basic network connectivity.
ping email-smtp.ap-south-1.amazonaws.com
nc -vz email-smtp.ap-south-1.amazonaws.com 587
Typical causes include:
- Firewall restrictions.
- Incorrect routing.
- Network outage.
- Incorrect SMTP endpoint.
Scenario 4 – DNS Resolution Failure
SMTP communication cannot begin if the relay hostname cannot be resolved.
host email-smtp.ap-south-1.amazonaws.com
If the hostname cannot be resolved:
- Verify /etc/resolv.conf.
- Check configured DNS servers.
- Verify network connectivity to the DNS infrastructure.
Scenario 5 – STARTTLS Negotiation Failure
TLS errors usually occur before SMTP authentication begins.
openssl s_client \
-connect email-smtp.ap-south-1.amazonaws.com:587 \
-starttls smtp
Common causes include:
- Missing CA certificates.
- TLS version incompatibility.
- Firewall devices inspecting encrypted SMTP traffic.
- Incorrect Sendmail TLS configuration.
Whenever possible, isolate one problem at a time. Resolving DNS, network connectivity, and TLS issues before investigating authentication usually results in much faster troubleshooting.
11. Sendmail Troubleshooting Flow
The following workflow can be used whenever email delivery fails.
Email Failed
|
v
Is Sendmail Running?
|
Yes / No
|
v
Check Mail Queue
|
v
Queued?
|
v
Check DNS
|
v
Check SMTP Connectivity
|
v
Check STARTTLS
|
v
Check Authentication
|
v
Review Logs
|
v
Resolved
Following the same troubleshooting sequence every time reduces unnecessary configuration changes and helps identify the root cause more efficiently.
12. Production Troubleshooting Checklist
Before escalating an email delivery issue, complete the following checklist.
| Verification | Status |
|---|---|
| Sendmail daemon is active. | ✔ |
| SMTP relay hostname resolves correctly. | ✔ |
| SMTP port is reachable. | ✔ |
| Mail queue inspected. | ✔ |
| Authentication database rebuilt. | ✔ |
| TLS handshake verified. | ✔ |
| System logs reviewed. | ✔ |
| Test email sent. | ✔ |
Maintain a documented troubleshooting checklist within your operations team. Following a standard process ensures consistency during production incidents and reduces recovery time.
13. Troubleshooting Best Practices
- Always begin with the Sendmail service status.
- Verify DNS before investigating SMTP authentication.
- Inspect the mail queue before modifying configuration files.
- Confirm network connectivity to the SMTP relay.
- Use OpenSSL to verify the STARTTLS handshake independently.
- Review both Sendmail logs and AIX system logs.
- Keep backup copies of Sendmail configuration files before making changes.
- Test configuration changes on a development system whenever possible.
14. Summary
Troubleshooting Sendmail becomes much easier when approached methodically. Rather than changing configuration files at random, administrators should follow the complete email delivery path—from the application to the recipient's mailbox—and verify each stage individually.
Throughout this article, we've examined how to validate the Sendmail service, inspect the mail queue, verify SMTP connectivity, diagnose authentication failures, troubleshoot TLS negotiation, and review system logs. These techniques form the foundation of effective Sendmail administration in production environments.
| Troubleshooting Area | Completed |
|---|---|
| Verified Sendmail service | ✔ |
| Validated configuration files | ✔ |
| Inspected the mail queue | ✔ |
| Verified DNS resolution | ✔ |
| Tested SMTP connectivity | ✔ |
| Diagnosed authentication failures | ✔ |
| Verified STARTTLS communication | ✔ |
| Reviewed production troubleshooting practices | ✔ |
Key Takeaways
- Always troubleshoot using the complete email delivery path.
- The mail queue is one of the most valuable diagnostic tools.
- DNS, network connectivity, authentication, and TLS should be verified in that order.
- Use log files and command-line tools together to identify the root cause.
- A structured troubleshooting process significantly reduces production recovery time.
15. Frequently Asked Questions
Why are emails staying in the Sendmail queue?
Messages usually remain queued because Sendmail cannot contact the SMTP relay, authentication has failed, DNS resolution is unavailable, or a firewall is blocking the connection.
How can I verify that Sendmail is running?
Use the following command:
lssrc -s sendmail
A status of active indicates that the daemon is running.
What is the first thing I should check when email delivery fails?
Start by verifying the Sendmail service, then inspect the mail queue. These two checks usually identify whether the problem is local or related to the SMTP relay.
How can I test the SMTP relay without sending an email?
Use openssl s_client to verify the STARTTLS handshake or use nc (Netcat) to confirm that the SMTP port is reachable.
16. Next Article
Article 7 – Automating Email Notifications on IBM AIX Using Sendmail
With Sendmail fully configured and production-ready, the next article focuses on automation. You'll learn how to build reusable shell scripts, send HTML emails, notify multiple recipients, include attachments, handle errors gracefully, and create production-ready email notification frameworks for backups, monitoring, Oracle databases, filesystem alerts, and other administrative tasks.