Ronny Roethof Reading time ~1 minute

During mayor upgrades on several systems I came along the bad situation with duplicate packages..

This helped me solve the issue of having duplicate packages installed. I only needed to add another flag to have:

rpm -e --justdb --nodeps <package-old_version>

Below you will find the steps to semi automate this, but beware, Your millage may vary, make sure you check the output that are put into the files.

Bottom of output from:

yum --assumeno upgrade

Warning: RPMDB altered outside of yum.
** Found 247 pre-existing rpmdb problem(s), 'yum check' output follows:
#(truncated)
chkconfig-1.7.4-1.el7.x86_64 is a duplicate with chkconfig-1.7.2-1.el7.x86_64
1:gmp-6.0.0-15.el7.x86_64 is a duplicate with 1:gmp-6.0.0-12.el7_1.x86_64
gobject-introspection-1.50.0-1.el7.x86_64 is a duplicate with gobject-introspection-1.42.0-1.el7.x86_64
grep-2.20-3.el7.x86_64 is a duplicate with grep-2.20-2.el7.x86_64
gzip-1.5-9.el7.x86_64 is a duplicate with gzip-1.5-8.el7.x86_64
pcre-8.32-17.el7.x86_64 is a duplicate with pcre-8.32-15.el7_2.1.x86_64
4:perl-5.16.3-292.el7.x86_64 is a duplicate with 4:perl-5.16.3-291.el7.x86_64
4:perl-libs-5.16.3-292.el7.x86_64 is a duplicate with 4:perl-libs-5.16.3-291.el7.x86_64
4:perl-macros-5.16.3-292.el7.x86_64 is a duplicate with 4:perl-macros-5.16.3-291.el7.x86_64
shared-mime-info-1.8-3.el7.x86_64 is a duplicate with shared-mime-info-1.1-9.el7.x86_64
2:tar-1.26-32.el7.x86_64 is a duplicate with 2:tar-1.26-31.el7.x86_64

Put above package names (starting from packages listed) into:

vi yum-fix.txt

Suggested to run from yum, but doesn’t seem to help. maybe it does a bit..

yum-complete-transaction --cleanup-only

cat yum-fix.txt|grep "is a duplicate with" > yum-fix_dupe.txt
cat yum-fix.txt|grep -v "is a duplicate with" > yum-fix_manually.txt

Parse output of some packages that start with a number and colon:

cat yum-fix_dupe.txt |awk '{print $6}'|awk -F":" '{if ((length($1) != 1) && (length($1) != 2)) print $1}' > yum-fix_dupe2.txt
cat yum-fix_dupe.txt |awk '{print $6}'|awk -F":" '{if ((length($1) == 1) || (length($1) == 2)) print $2}' >> yum-fix_dupe2.txt

Remove old package version from rpm db:

for i in $(cat yum-fix_dupe2.txt);do rpm -e --justdb --nodeps $i;done

Remove last 2 columns from output, and put on same line:

cat yum-fix_dupe2.txt | awk -F"-[0-9]" '{print $1}' |tr '\n' ' ' > yum-fix_dupe2-reinstall.txt

Reinstall packages:

yum reinstall $(cat yum-fix_dupe2-reinstall.txt)

Don’t forget to go back and manually fix the non-duplicates above could have fixed them.. try yum upgrade to see if any more errors):

cat yum-fix_manually.txt

yum --assumeno upgrade
Ronny Roethof Reading time ~1 minute

The following order is the correct PEM file order for HA-PROXY

certificate.key

certificate.crt

certificate.bundle.crt

This bash script will do it for you with Let’s Encrypt

cat /usr/local/bin/renew.sh

#!/bin/sh

# move to the correct let's encrypt directory
cd /etc/letsencrypt/live/override.nl-0001/
cat fullchain.pem privkey.pem > /etc/haproxy/certs/override.nl.pem

cd /etc/letsencrypt/live/cloud.override.nl/
cat fullchain.pem privkey.pem > /etc/haproxy/certs/cloud.override.nl.pem

# reload haproxy
service haproxy reload
Ronny Roethof Reading time ~1 minute

Remove all stopped containers

docker rm -v $(docker ps -a | grep "Exited" | awk "{print $1}")

Remove all stopped containers v2

docker rm $(docker ps -a -q)
Ronny Roethof Reading time ~1 minute

If you’ve ever rebuilt a server that you have connected to in the past, chances are you’ve received an error when trying to ssh back into it for the first time since the rebuild. If you’re getting a screen that says “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!”:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is b7:f5:48:4c:d0:1d:76:6a:50:4a:88:12:c7:80:f1:e5.
Please contact your system administrator. Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending key in /home/user/.ssh/known_hosts:2 RSA host key for myhost has changed and you have requested strict checking.
Host key verification failed.

Since we just rebuilt our server, this error is expected as the it is sending a new RSA key different than the one stored in /home/user/.ssh/known_hosts and now your computer’s ssh system is saying, “Hold on, this server you’re connecting to via this IP is not giving me the same ID. It could be that you just rebuilt the server, or it could be a man-in-the-middle attack!”

The workaround is actually quite simple, we have to remove the stored RSA key so we can add safely the new one. To do so, take a look at the last integer of the line that says “Offending ECDSA key,” because this integer is actually the line number inside your known_hosts file that’s throwing the error.

/home/user/.ssh/known_hosts:2

Then type in the following command, changing the number 2 to whatever matches your own error.

sed -i '2d' ~/.ssh/known_hosts

The offending RSA key can be also be removed with:

ssh-keygen -f "/home/user/.ssh/known_hosts" -R <server ip>
Ronny Roethof Reading time ~1 minute

Here we go again… For using ipv6 dnsbl, we need postfix version => 2.6 as the author of postfix state in postfix-users list.

How ipv6 dnsbl keep AAAA record in their zone? this is how it done. for example we got ipv6: 2001:1af8:4400:a00d:1::1 (my webserver)

RBL query lookup would be like this:

$ dig aaaa 1.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.d.0.0.a.0.0.4.4.8.f.a.1.1.0.0.2.dnsbl.example.tld.
$ dig txt 1.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.d.0.0.a.0.0.4.4.8.f.a.1.1.0.0.2.dnsbl.example.tld.

So, we need configure our private BIND9 RBL like this: first create dnsbl.example.tld zone in /etc/named.conf

zone "dnsbl.example.tld" {
      type master;
      file "dnsbl.example.tld";
};

second, we have to create dnsbl.example.tld zone file.

$TTL 86400
@       IN      SOA     ns1.dnsbl.example.tld.     hostmaster.dnsbl.example.tld. (
                        2011082200      ; serial number YYMMDDNN
                        28800           ; Refresh
                        7200            ; Retry
                        864000          ; Expire
                        86400           ; Min TTL
                        )

                NS      ns1.dnsbl.example.tld.
                NS      ns2.dnsbl.example.tld.

$ORIGIN dnsbl.example.tld.
blackhole       IN      A       127.0.0.2
                IN      AAAA    ::2
                IN      TXT     "Blocked by dnsbl.example.tld for SPAM Sources"

2.0.0.0.0.0.f.7.f.f.f.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0         IN      CNAME   blackhole
2.0.0.127                                                               IN      CNAME   blackhole
118.62.140.11                                                           IN      CNAME   blackhole
238.128.73.115                                                          IN      CNAME   blackhole
1.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.d.0.0.a.0.0.4.4.8.f.a.1.1.0.0.2         IN      CNAME   blackhole

why do i using CNAME instead of direct AAAA record? it’s just for efficiency, to avoid repetitions when adding ipv6 address on the blacklist. beside, postfix resolver can follow CNAME until found AAAA and TXT record.

IN postfix configuration, main.cf add this line:

smtpd_recipient_restrictions =
    ...
        reject_unauth_destination,
        reject_rbl_client dnsbl.example.tld,
    ...

now test all the things we’ve configured. (with my own ipv6 in the temporary list)

$ telnet mx.example.tld 25
220 mx.example.tld ESMTP Postfix (Ubuntu)
ehlo s1.example.tld
250-mx.example.tld
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from:<pietje@example.tld>
250 2.1.0 Ok
rcpt to:<jantje@example.tld>
554 5.7.1 Service unavailable; Client host [2001:1af8:4400:a00d:1::1] blocked using dnsbl.example.tld
Blocked by dnsbl.example.tld for SPAM Sources
quit
221 2.0.0 Bye
Connection to host lost.

In Postfix log, we will see rejection like this:

Aug 22 16:21:14 mx postfix/smtpd[25932]: NOQUEUE: reject: RCPT from s1[2001:1af8:4400:a00d:1::1]: 554 5.7.1 Service unavailable; Client host [2001:1af8:4400:a00d:1::1] blocked using dnsbl.example.tld; from=<pietje@example.tld> to=<jantje@example.tld> proto=ESMTP helo=<s1.example.tld>

That’s all :)