Category Archives: IBM MQ

My iKeycmd cheat sheet

Posted by on November 18, 2019 No comments 
# Add a client certificate to the Queue manager keystore (using stashed password)
ikeycmd -cert -add -db "key.kdb" -label ibmwebspheremqclient01 -file ibmwebspheremqclient01.crt -format ascii -stashed

# Remove certificate from Queue manager keystore (using stashed password)
ikeycmd -cert -delete -label ibmwebspheremqclient01 -db "key.kdb" -stashed

# Set default certificate (using stashed password)
ikeycmd -cert -setdefault -db key.kdb -stashed -label "mydefaultcertificate"

# List certificates
ikeycmd -cert -list personal -db "key.kdb" -pw changeit
ikeycmd -cert -list ca -db "key.kdb" -pw changeit

* List default signers
ikeycmd -cert -listsigners

Tested on IBM MQ 7.0.1.0 and Red Hat Linux 7

Replace a CA certified Client Certificate in IBM MQ using iKeyCmd

Posted by on March 18, 2019 No comments

Every now and then a client certificate expires and need to be replaced in the Queue Manager keystore. This is an example of such a change, using the iKeyCmd program (comes with IBM MQ v8 and above) and PEM formatted certificates

1. First we check that we have all the files necessary and determine the order in which they will be added.
For this example we use 4 files:

certrumroot.crt
ssl_com_root_certification_authyority_rsa.crt
ssl_com_rsa_ssl_sub_ca.crt
star_client_se.crt

Now, to determine the order in which they should be added we need to look inside each file for “Issuer” and “Subject”. For this I use openssl command:

openssl x509 -in <certificate file name> -text -noout

Example output (focusing on the Common Name (CN)):
certrumroot.crt
Issuer: …, CN=Certum Trusted Network CA
Subject: …, CN=Certum Trusted Network CA

ssl_com_root_certification_authyority_rsa.crt
Issuer: …, CN=Certum Trusted Network CA
Subject: …, CN=SSL.com Root Certification Authority RSA

ssl_com_rsa_ssl_sub_ca.crt
Issuer: …, CN=SSL.com Root Certification Authority RSA
Subject: …, CN=SSL.com RSA SSL subCA

star_client_se.crt
Issuer: …, CN=SSL.com RSA SSL subCA
Subject: CN=*.client.se

This needs a little explaining:
– Let us start with the top one. Here the “Issuer” and “Subject” is the same. This means that this is the root certificate. This should always be added first.

– The next certificate is issued by the first so that should be added as number 2

– The certificate after that is issued by the second one and should be added as number 3

– Lastly we have the client certificate and that should be added last, so now we have the order

2. Now we need to clear the queue manager key store from the old certificate chain. Let’s look at the current chain using the iKeyCmd program:

ikeycmd -cert -list ca -db key.kdb -stashed

Example output:

...
ibmwebspheremqclient43
ibmwebspheremqclient44
client44intermediate
client44root
client44ca
intermidiateca
rootca
...

For this example we are only interested in the client44 chain, so let us remove the current one:

ikeycmd -cert -delete -label clinent44ca -db key.kdb -stashed
ikeycmd -cert -delete -label client44root -db key.kdb -stashed
ikeycmd -cert -delete -label client44intermediate -db key.kdb -stashed
ikeycmd -cert -delete -label ibmwebspheremqclient44 -db key.kdb -stashed

A few things to note here about the parameters for iKeyCmd:

  • -cert – handle certificates
  • -delete – operation “delete”. Can also be “add” as we see further down
  • -label – label on the certificate you want to performe the operation on
  • -db – points to the file pointed out as the queue manager key store. Can be found in the SSLKEYR property on the queue manager
  • -stashed – use stashed password. Resides in the *.sth file – if any

Now we check that they have been removed

ikeycmd -cert -list ca -db key.kdb -stashed

ibmwebspheremqclient43
intermidiateca
rootca

3. Looks good. Now lets add the new ones, in order:

ikeycmd -cert -add -db key.kdb -label client44certumroot -filecertrumroot.crt -format ascii -stashed
ikeycmd -cert -add -db key.kdb -label client44root -file ssl_com_root_certification_authyority_rsa.crt -format ascii -stashed
ikeycmd -cert -add -db key.kdb -label client44subca -file ssl_com_rsa_ssl_sub_ca.crt -format ascii -stashed
ikeycmd -cert -add -db key.kdb -label ibmwebspheremqclient44 -file star_client_se.crt -format ascii -stashed

A note on the label for the client certificate. Here I use the default name pattern which is “ibmwebspheremq” + “username”, where userid is the username on OS level

If we now run the command:

ikeycmd -cert -list ca -db key.kdb -stashed

We see that they are all in place in the keystore.

ibmwebspheremqclient43
ibmwebspheremqclient44
client44certumroot
client44root
client44subca
intermidiateca
rootca

and now to the CRUCIAL PART! Whenever you make changes to the queue manager keystore you need to REFRESH SECURITY on the queue manager. This can be done by using the runmqsc console and issuing:

REFRESH SECURITY TYPE(SSL)

If you fail to this last part no changes will take place

Thats it!

Tested on Red Hat 7 and IBM MQ v9

My IBM MQ Cluster Ping Nagios script

Posted by on February 8, 2019 No comments

Monitoring cluster health can be quite tricky. I have created a small Nagios script to help in the task. It uses the sample programs amqsput and amqsget, delivered with the IBM MQ installation

Preparations:
In this example I’m only going to use two queue managers QMBASE (full repository) and QMEXTERNAL (partial repository). Commands below are for runmqsc
QMEXTERNAL

DEFINE QA(QA.CLUSTER.PING) TARGET(TOP.CLUSTER.PING) TARGTYPE(TOPIC)
DEFINE QL(QL.CLUSTER.PING) CLUSTER(EXTERNAL)

QMBASE

DEFINE TOP(TOP.CLUSTER.PING) TOPICSTR('ping') CLUSTER(EXTERNAL) 
DEFINE SUB(SUB.CLUSTER.PING) TOPICOBJ(TOP.CLUSTER.PING) TOPICSTR('#') DEST(QL.CLUSTER.PING)

How it works:
It puts a message on a queue alias, using amqsput, that is connected to a cluster topic on the base queue manager
A subscription to the topic picks up the message and put it on the cluster queue on the external queue manager
The message is then picked up by and checked. I also check the length of the message to check for more than one message

And here is the script:

#!/bin/bash

if [ $# -lt 1 ]; then
  echo "********************"
  echo "Cluster Ping"
  echo "********************"
  echo "NOTE!"
  echo "This script needs that all the MQ objects are in place (see below)"
  echo ""
  echo "QMEXTENAL                        QMBASE (REPOSITORY)"
  echo "QA.CLUSTER.PING            ->    TOP.CLUSTER.PING (CLUSTER)"
  echo "QL.CLUSTER.PING (CLUSTER)  <-    SUB.CLUSTER.PING(TOP.CLUSTER.PING)"
  echo ""
  echo "Usage: $0 <external queue manager>"
  echo ""
  echo "External Queue Manager: ex. QMEXTERNAL"
  echo ""
  echo "Ex. $0 QMEXTERNAL"
  exit 1
fi

# Define variables
qmanager=$1
inqueue=QA.CLUSTER.PING
outqueue=QL.CLUSTER.PING
timestamp=$(date +%s)
match=false
normal=true

#Send message to QMBASE
printf "%s\n\n" ${timestamp} | amqsput ${inqueue} ${qmanager} > /dev/null

msg=$(amqsget ${outqueue} ${qmanager})

if [[ ${msg} == *"${timestamp}"* ]]; then
  match=true
fi

if [ ${#msg} -gt 80 ]; then
  normal=false
fi

if [[ ${match} == "true" && ${normal} == "true" ]]; then
  echo "OK - Message received from cluster"
  exit 0
fi

if [[ ${match} == "true" && ${normal} == "false" ]]; then
  echo "WARNING - More then one message was received"
  exit 1
fi

if [[ ${match} == "false" ]]; then
  echo "ERROR - No message received!"
  exit 2
fi

echo "UNKNOWN - Script has not run correctly"
exit 3

Now, this will tell us that the cluster can transport messages to and from the base queue manager, but we need more to be able to feel safe (at least I do) so here are a few other things to look at:
* Keep track of the transmission queues so that they process messages. I have, so far, only worked with small clusters so monitoring queue depth of the SYSTEM.CLUSTER.TRANSMIT.QUEUE has been enough, but if you have more transmission queues you need to monitor them all.
* Keep track of the command queue SYSTEM.CLUSTER.COMMAND.QUEUE. This queue should also process messages and not only grow
* Look for error messages in the queue manager error log (AMQERR01.LOG). Here I look for the following codes (short description in parentheses):

- AMQ9465 (Failed republishing of cluster information)
- AMQ5534E (User ID 'anyuser' authentication failed)
- AMQ5542I (The failed authentication check was caused by the queue manager)
- AMQ9202E (Remote host 'anyhost (anyip) (anyport)' not available, retry)
- AMQ9259E (Connection timed out from host 'anyhost')
- AMQ9469W: (Update not received for CLUSRCVR channel any channel)
- AMQ9492E (The TCP/IP responder program encountered an error)
- AMQ9558E (The remote channel 'any channel' encountered a problem)
- AMQ9633E (Bad SSL certificate for channel 'any channel')
- AMQ9637E (Channel is lacking a certificate)
- AMQ9716E (Remote SSL certificate revocation status check failed for channel)

After all of the tests above I usually feel pretty confident that we have a working environment.

Tested on IBM MQ 9 and Red Hat 7