You are here
Mail In a Box on EC2 with S3 backup script data networks and data 

Mail In a Box on EC2 with S3 backup script

I wanted to break free and set up my own email server on Amazon’s AWS cloud. After signing up for a one year free trial, creating my mailinabox server on a T2 Micro instance with Ubunut 14.04LTS, getting the externally hosted DNS settings up and running, and testing OK, it was now time to focus my attention on backing up our email. Amazon’s free tier provides you with 5GB of free S3 storage, which is more than enough to keep and restore the emails on the server i wish to retain. Here’s the procedure I used.


1) Log into amazon S3 dashboard
https://console.aws.amazon.com/s3/home?region=us-east-1

2) Create a bucket. The free tier allows for 5GB. ChooseUS Standard. S3 Buckets appear to all zones.

Don’t set up logging and just hit create.

 

Mail In a Box on EC2 with S3 backup script - Create a bucket

Mail In a Box on EC2 with S3 backup script - S3 bucket has been created.

3) Get the google tools on your mail-in-a-box
http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html

 

ubuntu@ip-172-31-45-109:~$ wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
 --2015-06-17 16:29:10-- http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
 Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.80.203
 Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.80.203|:80... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 16675870 (16M) [binary/octet-stream]
 Saving to: ‘ec2-api-tools.zip’
100%[=====================================================================>] 16,675,870 75.4MB/s in 0.2s
2015-06-17 16:29:10 (75.4 MB/s) - ‘ec2-api-tools.zip’ saved [16675870/16675870]
ubuntu@ip-172-31-45-109:~$ sudo mkdir /usr/local/ec2
 sudo: unable to resolve host ip-172-31-45-109

4) Ran into this problem after working around the CLI when mailinabox had already been installed.
Added the following to /etc/hosts:
ubuntu@ip-172-31-45-109:~$ sudo vi /etc/hosts
127.0.0.1 localhost ip-172-31-45-109
Then :wq.

5) Expand the files.
ubuntu@ip-172-31-45-109:~$ sudo unzip ec2-api-tools.zip -d /usr/local/ec2
Archive: ec2-api-tools.zip
creating: /usr/local/ec2/ec2-api-tools-1.7.4.0/
inflating: /usr/local/ec2/ec2-api-tools-1.7.4.0/THIRDPARTYLICENSE.TXT

inflating: /usr/local/ec2/ec2-api-tools-1.7.4.0/notice.txt

6) Tell the system where the tools live.
ubuntu@ip-172-31-45-109:/usr/local/ec2/ec2-api-tools-1.7.4.0/bin$ export EC2_HOME=/usr/local/ec2/ec2-api-tools-1.7.4.0
ubuntu@ip-172-31-45-109:/usr/local/ec2/ec2-api-tools-1.7.4.0/bin$ export PATH=$PATH:$EC2_HOME/bin
ubuntu@ip-172-31-45-109:/usr/local/ec2/ec2-api-tools-1.7.4.0/bin$
7) Install java
ubuntu@ip-172-31-45-109:/usr/local/ec2/ec2-api-tools-1.7.4.0/bin$ sudo apt-get install default-jre
ubuntu@ip-172-31-45-109:~$ export JAVA_HOME=”/usr/lib/jvm/java-7-openjdk-amd64/jre”
ubuntu@ip-172-31-45-109:~$ $JAVA_HOME/bin/java -version
java version “1.7.0_79”
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
ubuntu@ip-172-31-45-109:~$

8) Grab your security credentials. You’re gonna need access key and secret key.
https://console.aws.amazon.com/iam/home?#security_credential

 

Mail In a Box on EC2 with S3 backup script - Your Security Credentials on AWS

Choose to show the access key. Copy and paste into bottom of bashrc file:

 

Mail In a Box on EC2 with S3 backup script - Create Access Key on AWS

 

ubuntu@ip-172-31-45-109:~$ vi ~/.bashrc
export AWS_ACCESS_KEY=XXXXXXXXXXXXXXX
export AWS_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Write the file and :
ubuntu@ip-172-31-45-109:~$ source ~/.bashrc
ubuntu@ip-172-31-45-109:~$

 

 

9) Quick test of the environment.
ubuntu@ip-172-31-45-109:~$ $JAVA_HOME/bin/java -version
java version “1.7.0_79”
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

ubuntu@ip-172-31-45-109:~$ $EC2_HOME/bin/ec2-describe-regions
REGION eu-central-1 ec2.eu-central-1.amazonaws.com
REGION sa-east-1 ec2.sa-east-1.amazonaws.com
REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com
REGION eu-west-1 ec2.eu-west-1.amazonaws.com
REGION us-east-1 ec2.us-east-1.amazonaws.com
REGION us-west-1 ec2.us-west-1.amazonaws.com
REGION us-west-2 ec2.us-west-2.amazonaws.com
REGION ap-southeast-2 ec2.ap-southeast-2.amazonaws.com
REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com

The above output indicates things are working OK.

10) Install the S3 tools:
https://marcqualie.com/2013/07/server-backups-with-s3cmd

ubuntu@ip-172-31-45-109:~$ wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add –
OK
ubuntu@ip-172-31-45-109:~$ wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list
/etc/apt/sources.list.d/s3tools.list: Permission denied
ubuntu@ip-172-31-45-109:~$
ubuntu@ip-172-31-45-109:~$ sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list
–2015-06-17 17:26:25– http://s3tools.org/repo/deb-all/stable/s3tools.list
Resolving s3tools.org (s3tools.org)… 66.39.80.91
Connecting to s3tools.org (s3tools.org)|66.39.80.91|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 44
Saving to: ‘/etc/apt/sources.list.d/s3tools.list’

100%[===============================================================================================================>] 44 –.-K/s in 0s

2015-06-17 17:26:25 (12.3 MB/s) – ‘/etc/apt/sources.list.d/s3tools.list’ saved [44/44]

ubuntu@ip-172-31-45-109:~$

ubuntu@ip-172-31-45-109:~$ sudo apt-get update
Ign http://us-east-1.ec2.archive.ubuntu.com trusty InRelease
Ign http://us-east-1.ec2.archive.ubuntu.com trusty-updates InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty Release.gpg
Get:1 http://us-east-1.ec2.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Ign http://s3tools.org stable/ InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty Release
Get:2 http://us-east-1.ec2.archive.ubuntu.com trusty-updates Release [63.5 kB]
Get:3 http://s3tools.org stable/ Release.gpg [287 B]
Ign http://security.ubuntu.com trusty-security InRelease
Get:4 http://s3tools.org stable/ Release [1,504 B]
Get:5 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main Sources
Get:6 http://security.ubuntu.com trusty-security Release [63.5 kB]
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main Translation-en
Get:7 http://s3tools.org stable/ Packages [572 B]
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe Translation-en
Get:8 http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main Sources [208 kB]
Get:9 http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe Sources [121 kB]
Get:10 http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main amd64 Packages [542 kB]
Get:11 http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages [287 kB]
Get:12 http://security.ubuntu.com trusty-security/main Sources [85.8 kB]
Get:13 http://security.ubuntu.com trusty-security/universe Sources [25.7 kB]
Ign http://s3tools.org stable/ Translation-en_US
Get:14 http://security.ubuntu.com trusty-security/main amd64 Packages [299 kB]
Get:15 http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main Translation-en [262 kB]
Ign http://s3tools.org stable/ Translation-en
Get:16 http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe Translation-en [150 kB]
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US
Get:17 http://security.ubuntu.com trusty-security/universe amd64 Packages [108 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Fetched 2,219 kB in 2s (893 kB/s)
Reading package lists… Done
ubuntu@ip-172-31-45-109:~$ sudo apt-get install -y s3cmd
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
s3cmd
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 67.6 kB of archives.
After this operation, 295 kB of additional disk space will be used.
Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe s3cmd all 1.1.0~beta3-2 [67.6 kB]
Fetched 67.6 kB in 0s (0 B/s)
Selecting previously unselected package s3cmd.
(Reading database … 95758 files and directories currently installed.)
Preparing to unpack …/s3cmd_1.1.0~beta3-2_all.deb …
Unpacking s3cmd (1.1.0~beta3-2) …
Processing triggers for man-db (2.6.7.1-1ubuntu1) …
Setting up s3cmd (1.1.0~beta3-2) …
ubuntu@ip-172-31-45-109:~$
11) Configure the s3tools
Setting up s3cmd (1.1.0~beta3-2) …
ubuntu@ip-172-31-45-109:~$ s3cmd –configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: asdfadsfadsfasdf
Secret Key: fadsfadsfadsfadsfadsfadsfadsfadsfddasfafadsf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can’t be used if you’re behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: afddsfasfd
Secret Key: fdasfadsfdssdfaadsfasfdasfadsfadsfasf
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets…
Success. Your access key and secret key worked fine 🙂

Now verifying that encryption works…
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to ‘/home/ubuntu/.s3cfg’
ubuntu@ip-172-31-45-109:~$

12) Test the backup
ubuntu@ip-172-31-45-109:~$ sudo s3cmd sync /home/user-data/backup s3://mailserverbackup/mail.rippul.com/
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
/home/user-data/backup/cache/77bb3fb2afb5975d26aa819871b4bd44/duplicity-full-signatures.20150617T064207Z.sigtar.gz -> s3://mailserverbackup/mail.rippul.com/backup/cache/77bb3fb2afb5975d26aa819871b4bd44/duplicity-full-signatures.20150617T064207Z.sigtar.gz [1 of 6]
71521 of 71521 100% in 0s 820.06 kB/s done
/home/user-data/backup/cache/77bb3fb2afb5975d26aa819871b4bd44/duplicity-full.20150617T064207Z.manifest -> s3://mailserverbackup/mail.rippul.com/backup/cache/77bb3fb2afb5975d26aa819871b4bd44/duplicity-full.20150617T064207Z.manifest [2 of 6]
183 of 183 100% in 0s 1079.10 B/s done
/home/user-data/backup/encrypted/duplicity-full-signatures.20150617T064207Z.sigtar.gpg -> s3://mailserverbackup/mail.rippul.com/backup/encrypted/duplicity-full-signatures.20150617T064207Z.sigtar.gpg [3 of 6]
71990 of 71990 100% in 0s 750.58 kB/s done
/home/user-data/backup/encrypted/duplicity-full.20150617T064207Z.manifest.gpg -> s3://mailserverbackup/mail.rippul.com/backup/encrypted/duplicity-full.20150617T064207Z.manifest.gpg [4 of 6]
226 of 226 100% in 0s 2.81 kB/s done
/home/user-data/backup/encrypted/duplicity-full.20150617T064207Z.vol1.difftar.gpg -> s3://mailserverbackup/mail.rippul.com/backup/encrypted/duplicity-full.20150617T064207Z.vol1.difftar.gpg [5 of 6]
2492789 of 2492789 100% in 0s 14.69 MB/s done
/home/user-data/backup/secret_key.txt -> s3://mailserverbackup/mail.rippul.com/backup/secret_key.txt [6 of 6]
2775 of 2775 100% in 0s 34.75 kB/s done
Done. Uploaded 2639484 bytes in 0.7 seconds, 3.73 MB/s
ubuntu@ip-172-31-45-109:~$

 

Mail In a Box on EC2 with S3 backup script - Verify the File Transfer Worked.

13) Implement the cron job as the root user. You can see what time the backup completed. So sometime after that should be good.
ubuntu@ip-172-31-45-109:/home/user-data/backup/encrypted$ ls -la
total 2520
drwxr-xr-x 2 user-data root 4096 Jun 17 06:42 .
drwxr-xr-x 4 root root 4096 Jun 17 06:42 ..
-rw——- 1 user-data root 226 Jun 17 06:42 duplicity-full.20150617T064207Z.manifest.gpg
-rw——- 1 user-data root 2492789 Jun 17 06:42 duplicity-full.20150617T064207Z.vol1.difftar.gpg
-rw——- 1 user-data root 71990 Jun 17 06:42 duplicity-full-signatures.20150617T064207Z.sigtar.gpg
ubuntu@ip-172-31-45-109:/home/user-data/backup/encrypted$ sudo crontab -l
# m h dom mon dow command
0 8 * * * s3cmd sync /home/user-data/backup s3://mailserverbackup/mail.rippul.com/

 

Setting up a mail notification of S3 bucket size:
0 9 * * 1 echo “Subject: Weekly S3 Size Report” | echo `sudo s3cmd du s3://mailserverbackup` | sendmail person@domain.com

Related posts

3 thoughts on “Mail In a Box on EC2 with S3 backup script

  1. Too Complex

    Man, that is way too complex for me. I know I’m not some techie but neither am I a know-nothing goof.

    good for you for figuring it out, but way too complex for me. Hello… Yahoo.com? … HA!

    1. You know, I figured this out so you wouldn’t have figure it out! Yahoo? Gmail? Hotmail? There are still benefits to running your own mailserver on your own domain. For starters, your email remains quite personal. No search crawlers looking for hints that you will be purchasing a new BBQ anytime soon and projecting pervasive advertising to you. If the NSA wants to look at your mail, not that there’s anything to find there, but they’d have to purposely target your email, instead of casting a dragnet on a nice juicy big-data collection in a single spot.

  2. Captain D

    Good points on the security of emails. I wonder where all of this is leading us.

    SneakerNet? That’s what it is coming down to.

    What is the new sneaker net? How can we get around the pervasive, invasive all-knowing big brother?

    I’m not doing anything wrong, but I want to regain my freedom of thought and expression.

    Perhaps ‘off the grid’ a la Terminator movies is the solution?

    Alternative identities might be the solution. Use ’em and lose ’em once a year.

Leave a Comment