Configuring Sendmail as an SMTP Client on IBM AIX

1. What Happens After Installing Sendmail?

Many administrators believe that installing Sendmail automatically enables email delivery. In reality, installation is only the first step.

After installation, Sendmail is capable of sending email, but it still doesn't know where messages should be delivered.

Several important questions remain unanswered:

• Which SMTP server should Sendmail connect to?
• Should mail be delivered directly to the Internet?
• Should all mail be forwarded to an enterprise SMTP relay?
• Will authentication be required?
• Should communication use TLS encryption?

Until these questions are answered through configuration, Sendmail cannot function as a modern SMTP client.

Install Sendmail
        v
Sendmail Binary
        v
No SMTP Configuration
        v
Cannot Relay Mail
        v
Configure SMTP Relay
        v
Ready for Enterprise Email

2. Understanding sendmail.cf

The heart of every Sendmail installation is the sendmail.cf configuration file.

This file defines nearly every aspect of Sendmail's behaviour, including mail routing, SMTP relay settings, daemon options, trusted users, delivery modes, timeout values, and protocol behaviour.

Unlike modern configuration files, sendmail.cf is intentionally compact and highly optimized. Many administrators find it difficult to read because it was historically generated rather than written manually.

sendmail.mc
      v
     m4
      v
sendmail.cf

On many UNIX and Linux systems, administrators edit sendmail.mc and then regenerate sendmail.cf using the m4 macro processor.

However, many AIX environments—particularly those using the AIX Toolbox packages—work directly with the generated sendmail.cf file. Understanding its structure is therefore an important administrative skill.

cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.bkp

3. Important Configuration Files

Although sendmail.cf is the primary configuration file, Sendmail relies on several additional files that work together to control mail processing.

At this stage, we will focus only on understanding these files. Detailed modifications—particularly those related to SMTP authentication—will be covered in the next articles.

4. Understanding SMART_HOST

One of the most important concepts in modern Sendmail configuration is the SMART_HOST. Before configuring SMTP authentication, every administrator should understand why this feature exists and how it changes the way Sendmail delivers email.

By default, Sendmail behaves like a traditional Mail Transfer Agent. When an application generates an email, Sendmail performs a DNS lookup for the recipient's domain, retrieves its Mail Exchange (MX) record, and attempts to establish a direct SMTP connection with the destination mail server.

Application
      v
Sendmail
      v
DNS MX Lookup
      v
Recipient Mail Server
      v
Recipient Mailbox

Although this approach works in theory, it is rarely used in modern enterprise environments. Most organizations prohibit production servers from sending email directly to Internet mail servers.

Instead, all outgoing email is forwarded to a trusted SMTP relay server. This relay becomes responsible for authentication, encryption, spam filtering, logging, and final delivery.

Application
      v
Sendmail
      v
Enterprise SMTP Relay
      v
Internet
      v
Recipient Mail Server
      v
Recipient Mailbox

This relay server is known in Sendmail terminology as the SMART_HOST.

5. Why Use a SMART_HOST?

Using a SMART_HOST offers significant advantages over direct Internet mail delivery. Rather than every server communicating independently with external mail systems, all outgoing messages pass through a centralized relay.

This architecture provides greater security, easier administration, and improved email delivery reliability.

For these reasons, nearly every enterprise configures Sendmail to forward all outgoing mail to a SMART_HOST instead of attempting direct delivery.

6. Configuring the SMTP Relay

In Sendmail, the SMART_HOST is defined using the DS directive within the sendmail.cf configuration file.

The directive specifies the hostname of the SMTP server that will receive all outgoing email.

DSsmtp.example.com

In this example:

• DS indicates the Default Smart Host.
• smtp.example.com is the SMTP relay server.

Once this value is configured, Sendmail no longer performs DNS MX lookups for each outgoing message. Instead, every email is forwarded directly to the configured relay server.

=> Without SMART_HOST

Sendmail
      v
DNS MX Lookup
      v
Recipient Mail Server
  
=> With SMART_HOST

Sendmail
      v
SMTP Relay
      v
Recipient Mail Server

7. Applying Configuration Changes

After modifying the Sendmail configuration, the daemon must reload the updated settings before the changes become active.

On IBM AIX, Sendmail is managed through the System Resource Controller (SRC).

To reload the configuration without completely stopping the service, use:

refresh -s sendmail

If a full restart is required, stop and start the service using the following commands.

stopsrc -s sendmail

startsrc -s sendmail

Finally, verify that the daemon is running correctly.

lssrc -s sendmail

A successful response confirms that Sendmail is active and ready to process outgoing mail.

8. Verifying the Configuration

Before configuring SMTP authentication, perform a few basic validation checks.

• Verify that the Sendmail daemon is running.
• Confirm that the SMART_HOST hostname resolves correctly.
• Ensure that the configured SMTP port is reachable.
• Review the mail queue for unexpected pending messages.
• Check the system logs for configuration-related errors.

These simple checks help identify networking or configuration issues before authentication and encryption are introduced.

9. Testing the SMTP Client Configuration

After configuring Sendmail as an SMTP client, the next step is to verify that the configuration behaves as expected. At this stage, the goal is not to authenticate with an SMTP relay but to ensure that Sendmail is processing messages correctly and attempting delivery to the configured SMART_HOST.

A simple test email can be generated using the mail utility.

echo "This is a test mail." | mail -s "SMTP Client Test" admin@example.com

After sending the message, verify that no immediate errors are reported. If the SMTP relay is not yet configured for authentication, the message may remain in the queue, which is expected at this stage.

10. Understanding Mail Queue Behaviour

One of Sendmail's strengths is its ability to queue messages whenever immediate delivery is not possible. Rather than discarding email because of a temporary network or configuration problem, Sendmail safely stores the message and retries delivery later.

Display the current mail queue using either of the following commands.

mailq

or

sendmail -bp

Typical queue states include:

A growing mail queue is often an indication of connectivity, DNS, relay configuration, or authentication problems. Monitoring the queue regularly provides valuable insight into the health of your email infrastructure.

Application
      v
Sendmail
      v
Mail Queue
      |
      +-------> SMTP Available --------> Delivered
      |
      +-------> Temporary Failure -----> Retry

11. Verifying SMTP Connectivity

Before enabling SMTP authentication, confirm that the AIX server can establish a network connection to the configured SMTP relay.

Depending on the tools available on your server, connectivity can be verified using one of the following commands.

telnet smtp.example.com 25

or

nc -vz smtp.example.com 25

For encrypted SMTP services, OpenSSL can be used to verify the TLS handshake.

openssl s_client -connect smtp.example.com:587 -starttls smtp

A successful connection confirms that the server can reach the SMTP relay and that firewalls or routing issues are unlikely to prevent email delivery.

12. Common Configuration Mistakes

Many Sendmail configuration issues are caused by small mistakes rather than software defects. Reviewing the following checklist can often resolve problems quickly.

Always verify basic connectivity and configuration before assuming there is a problem with Sendmail itself.

13. Best Practices

• Always create a backup of sendmail.cf before making changes.
• Use a SMART_HOST instead of direct Internet mail delivery.
• Verify DNS resolution before troubleshooting Sendmail.
• Monitor the mail queue regularly.
• Document every configuration change.
• Restart or refresh Sendmail after modifying configuration files.
• Validate configuration changes on a test LPAR whenever possible.

14. Summary

In this article, we transformed Sendmail from a freshly installed Mail Transfer Agent into a properly configured SMTP client capable of forwarding messages to an enterprise relay server. We explored the purpose of sendmail.cf, understood the role of the SMART_HOST, learned how to manage the Sendmail daemon, verified connectivity, and reviewed common configuration pitfalls.

With Sendmail now functioning as an SMTP client, the next step is to enable authenticated communication with Amazon Simple Email Service (AWS SES). We'll configure SMTP credentials, create the authentication database, generate the required hash maps, enable SASL authentication, configure TLS encryption, and send the first authenticated email from an IBM AIX server.