During mayor upgrades on several systems I came along the bad situation with duplicate packages..

This helped me solve the issue of having duplicate packages installed. I only needed to add another flag to have:

rpm -e --justdb --nodeps <package-old_version>

Below you will find the steps to semi automate this, but beware, Your millage may vary, make sure you check the output that are put into the files.

Bottom of output from:

yum --assumeno upgrade
Warning: RPMDB altered outside of yum.
** Found 247 pre-existing rpmdb problem(s), 'yum check' output follows:
chkconfig-1.7.4-1.el7.x86_64 is a duplicate with chkconfig-1.7.2-1.el7.x86_64
1:gmp-6.0.0-15.el7.x86_64 is a duplicate with 1:gmp-6.0.0-12.el7_1.x86_64
gobject-introspection-1.50.0-1.el7.x86_64 is a duplicate with gobject-introspection-1.42.0-1.el7.x86_64
grep-2.20-3.el7.x86_64 is a duplicate with grep-2.20-2.el7.x86_64
gzip-1.5-9.el7.x86_64 is a duplicate with gzip-1.5-8.el7.x86_64
pcre-8.32-17.el7.x86_64 is a duplicate with pcre-8.32-15.el7_2.1.x86_64
4:perl-5.16.3-292.el7.x86_64 is a duplicate with 4:perl-5.16.3-291.el7.x86_64
4:perl-libs-5.16.3-292.el7.x86_64 is a duplicate with 4:perl-libs-5.16.3-291.el7.x86_64
4:perl-macros-5.16.3-292.el7.x86_64 is a duplicate with 4:perl-macros-5.16.3-291.el7.x86_64
shared-mime-info-1.8-3.el7.x86_64 is a duplicate with shared-mime-info-1.1-9.el7.x86_64
2:tar-1.26-32.el7.x86_64 is a duplicate with 2:tar-1.26-31.el7.x86_64

Put above package names (starting from packages listed) into:

vi yum-fix.txt

Suggested to run from yum, but doesn’t seem to help. maybe it does a bit..

yum-complete-transaction --cleanup-only
cat yum-fix.txt|grep "is a duplicate with" > yum-fix_dupe.txt
cat yum-fix.txt|grep -v "is a duplicate with" > yum-fix_manually.txt

Parse output of some packages that start with a number and colon:

cat yum-fix_dupe.txt |awk '{print $6}'|awk -F":" '{if ((length($1) != 1) && (length($1) != 2)) print $1}' > yum-fix_dupe2.txt
cat yum-fix_dupe.txt |awk '{print $6}'|awk -F":" '{if ((length($1) == 1) || (length($1) == 2)) print $2}' >> yum-fix_dupe2.txt

Remove old package version from rpm db:

for i in $(cat yum-fix_dupe2.txt);do rpm -e --justdb --nodeps $i;done

Remove last 2 columns from output, and put on same line:

cat yum-fix_dupe2.txt | awk -F"-[0-9]" '{print $1}' |tr '\n' ' ' > yum-fix_dupe2-reinstall.txt

Reinstall packages:

yum reinstall $(cat yum-fix_dupe2-reinstall.txt)

Don’t forget to go back and manually fix the non-duplicates above could have fixed them.. try yum upgrade to see if any more errors):

cat yum-fix_manually.txt
yum --assumeno upgrade

If you’ve ever rebuilt a server that you have connected to in the past, chances are you’ve received an error when trying to ssh back into it for the first time since the rebuild. If you’re getting a screen that says “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!”:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is b7:f5:48:4c:d0:1d:76:6a:50:4a:88:12:c7:80:f1:e5.
Please contact your system administrator. Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending key in /home/user/.ssh/known_hosts:2 RSA host key for myhost has changed and you have requested strict checking.
Host key verification failed.

Since we just rebuilt our server, this error is expected as the it is sending a new RSA key different than the one stored in /home/user/.ssh/known_hosts and now your computer’s ssh system is saying, “Hold on, this server you’re connecting to via this IP is not giving me the same ID. It could be that you just rebuilt the server, or it could be a man-in-the-middle attack!”

The workaround is actually quite simple, we have to remove the stored RSA key so we can add safely the new one. To do so, take a look at the last integer of the line that says “Offending ECDSA key,” because this integer is actually the line number inside your known_hosts file that’s throwing the error.


Then type in the following command, changing the number 2 to whatever matches your own error.

sed -i '2d' ~/.ssh/known_hosts

The offending RSA key can be also be removed with:

ssh-keygen -f "/home/user/.ssh/known_hosts" -R <server ip>

Here we go again… For using ipv6 dnsbl, we need postfix version => 2.6 as the author of postfix state in postfix-users list.

How ipv6 dnsbl keep AAAA record in their zone? this is how it done. for example we got ipv6: 2001:1af8:4400:a00d:1::1 (my webserver)

RBL query lookup would be like this:

$ dig aaaa
$ dig txt

So, we need configure our private BIND9 RBL like this: first create dnsbl.example.tld zone in /etc/named.conf

zone "dnsbl.example.tld" {
      type master;
      file "dnsbl.example.tld";

second, we have to create dnsbl.example.tld zone file.

$TTL 86400
@       IN      SOA     ns1.dnsbl.example.tld.     hostmaster.dnsbl.example.tld. (
                        2011082200      ; serial number YYMMDDNN
                        28800           ; Refresh
                        7200            ; Retry
                        864000          ; Expire
                        86400           ; Min TTL

                NS      ns1.dnsbl.example.tld.
                NS      ns2.dnsbl.example.tld.

$ORIGIN dnsbl.example.tld.
blackhole       IN      A
                IN      AAAA    ::2
                IN      TXT     "Blocked by dnsbl.example.tld for SPAM Sources"         IN      CNAME   blackhole                                                               IN      CNAME   blackhole                                                           IN      CNAME   blackhole                                                          IN      CNAME   blackhole         IN      CNAME   blackhole

why do i using CNAME instead of direct AAAA record? it’s just for efficiency, to avoid repetitions when adding ipv6 address on the blacklist. beside, postfix resolver can follow CNAME until found AAAA and TXT record.

IN postfix configuration, main.cf add this line:

smtpd_recipient_restrictions =
        reject_rbl_client dnsbl.example.tld,

now test all the things we’ve configured. (with my own ipv6 in the temporary list)

$ telnet mx.example.tld 25
220 mx.example.tld ESMTP Postfix (Ubuntu)
ehlo s1.example.tld
250-SIZE 10240000
250 DSN
mail from:<pietje@example.tld>
250 2.1.0 Ok
rcpt to:<jantje@example.tld>
554 5.7.1 Service unavailable; Client host [2001:1af8:4400:a00d:1::1] blocked using dnsbl.example.tld
Blocked by dnsbl.example.tld for SPAM Sources
221 2.0.0 Bye
Connection to host lost.

In Postfix log, we will see rejection like this:

Aug 22 16:21:14 mx postfix/smtpd[25932]: NOQUEUE: reject: RCPT from s1[2001:1af8:4400:a00d:1::1]: 554 5.7.1 Service unavailable; Client host [2001:1af8:4400:a00d:1::1] blocked using dnsbl.example.tld; from=<pietje@example.tld> to=<jantje@example.tld> proto=ESMTP helo=<s1.example.tld>

That’s all :)

Munin does have some limitations. It does not scale well (to hundreds of servers) and I find it particularly painful to create aggregated graphs (for example aggregated network graph of two or more hosts). But I know these issues are being worked on.

Okay, enough talk – let’s monitor Bind:

First we need enable logging. Create a log directory and add log directives to the Bind configuration file (here on Ubuntu):

# mkdir /var/log/bind9
# chown bind:bind /var/log/bind9
# cat /etc/bind/named.conf.options
logging {
        channel b_log {
                file "/var/log/bind9/bind.log" versions 30 size 1m;
                print-time yes;
                print-category yes;
                print-severity yes;
                severity info;

        channel b_debug {
                file "/var/log/bind9/debug.log" versions 2 size 1m;
                print-time yes;
                print-category yes;
                print-severity yes;
                severity dynamic;

        channel b_query {
                file "/var/log/bind9/query.log" versions 2 size 1m;
                print-time yes;
                severity info;

        category default { b_log; b_debug; };
        category config { b_log; b_debug; };
        category queries { b_query; };

Restart bind:

# /etc/init.d/bind9 restart
  Stopping domain name service: named.
  Starting domain name service: named.

You can now see log files are being populated under /var/log/bind9/

Next, configure Munin:

Make sure the munin-user (“munin”) can read you bind log files.

We need two additional plugins: “bind” and “bind_rndc”. If you can’t find them in your default install, head over here.

The “bind” plugin should work right away. “bind9_rndc” however need to read the “rndc.key” file, which only are readable by the user “bind”. You have two options, either run the plugin as root or add the user “munin” to the group “bind” and enable the group “bind” to read the rndc.file. For the sake of simplicity, I run the plugin as root here. So you need to add:

# cat /etc/munin/plugin-conf.d/munin-node
  user root
  env.querystats /var/log/bind9/named.stats

Next restart Munin:

# /etc/init.d/munin-node restart
  Stopping munin-node: done.
  Starting munin-node: done.

Munin run every five minutes, so go take a coffee and… Wait.

In order to have automatic and unattended security updates in Ubuntu, one needs to install the according package:

sudo aptitude install unattended-upgrades

The file /etc/apt/apt.conf.d/10periodic needs to be created with the following content:

APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "5";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::RandomSleep "1800";

Also, change the first few lines of /etc/apt/apt.conf.d/50unattended-upgrades as follows so that only security updates are considered:

Unattended-Upgrade::Allowed-Origins {
        "Ubuntu lucid-security";
        "Ubuntu lucid-updates";

Unattended-Upgrade::Package-Blacklist {

Unattended-Upgrade::Mail "root@localhost";

Unattended-Upgrade::Remove-Unused-Dependencies "false";
Unattended-Upgrade::Automatic-Reboot "false";

It is vital to redo these setting after a global upgrade to a new distro release.

If configured correctly the following command should produce this output:

apt-config shell UnattendedUpgradeInterval APT::Periodic::Unattended-Upgrade