How to resolve Replication Issues

So many times in real world you will come across ‘Database replication Issues’. Replication Issues are also very much possible during your CCIE lab so one must be prepare to deal with such Issues.

How can you find if your cluster has proper replication? Here are few tools to find out replication Issues:

RTMT:

RTMT (Real-time monitoring Tool) is a very useful tool for not only checking replication Issues but for lot of other stuff. I will discuss RTMT in detail in some other section. To find out replication status using RTMT, you will need to click the ‘Call manager’ tab on bottom left and then double click ‘Database Summary’. You will get the following stats for a healthy replication.

As long as ‘replication status’ shows same number across the cluster – that means replication is good. If it’s not same then that means replication is not good and you better take some action really fast!

CUCM Unified Reporting:

Go to Unified Reporting > System Reports > Unified CM Database Status > Generate new Report

CUCM OS Admin CLI:

SSH onto Publisher box and Issue the following command:

admin:utils dbreplication status
********************************************************************************                                                                              ********************
This command reads and writes database information from all machines and will take quite some time…please be patient.
********************************************************************************                                                                              ********************

——————– utils dbreplication status ——————–

Output is in file cm/trace/dbl/sdi/ReplicationStatus.2010_06_28_18_24_51.out

Please use “file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_06_28_18_24_51.out ” command to see the output

Control-C pressed

admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_06_28_18_24_51.out

SERVER                 ID STATE    STATUS     QUEUE  CONNECTION CHANGED
———————————————————————–
g_ucmpub_ccm7_0_1_11000_2    2 Active   Local           0
g_ucmsub_ccm7_0_1_11000_2    3 Active   Connected       0 Jun 28 17:27:04

————————————————-
No Errors or Mismatches found.
Replication status is good on all available servers.

utils dbreplication status output

To determine if replication is suspect, look for the following:
(1) Number of rows in a table do not match on all nodes.
(2) Non-zero values occur in any of the other output columns for a table

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_91_typedberrors with 899 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_91_typedberrors ——
Node                  Rows     Extra   Missing  Mismatch Processed

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 – 20 of 2901) :
—————- ——— ——— ——— ——— ———
g_ucmpub_ccm7_0_1_11000_2       899         0         0         0         0
g_ucmsub_ccm7_0_1_11000_2       899         0         0         0         0

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_41_typeadminerror with 57 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_41_typeadminerror ——
Node                  Rows     Extra   Missing  Mismatch Processed
—————- ——— ——— ——— ——— ———
g_ucmpub_ccm7_0_1_11000_2        57         0         0         0         0
g_ucmsub_ccm7_0_1_11000_2        57         0         0         0         0

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_25_mohaudiosource with 51 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_25_mohaudiosource ——
Node                  Rows     Extra   Missing  Mismatch Processed
—————- ——— ——— ——— ——— ———
g_ucmpub_ccm7_0_1_11000_2        51         0         0         0         0
g_ucmsub_ccm7_0_1_11000_2        51         0         0         0         0

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_44_typeannouncements with 19 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_44_typeannouncements ——

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 – 40 of 2901) :

Replication Indicators:

1. Even after configuring everything perfectly fine, things won’t work
2. If Sub is your Primary Call manager for Phone registration then due to replication Issues phones will not register to Sub and will keep on looping with an error on screen ‘DB-Error – Registration Rejected’.
3. You configure something on Publisher, it won’t come up on Sub

Solution:

This is what I came across as the most simplest solution out of many.

• Take off Subscriber Server from CUCM group
• Shut down all services at Sub
• SSH onto Publisher and issue the command

admin:utils dbreplication
utils dbreplication clusterreset
utils dbreplication dropadmindb
utils dbreplication forcedatasyncsub
utils dbreplication repair
utils dbreplication reset
utils dbreplication setrepltimeout
utils dbreplication status
utils dbreplication stop

admin:utils dbreplication rep
admin:utils dbreplication repair ?
Syntax:
utils dbreplication repair [nodename]|all
nodename = which node to repair replication on.
all = fix replication on all nodes.

admin:utils dbreplication repair all

Give it sometime to repair and when it is done then restart all services at Subscriber and Put it back in CUCM group. If the replication issue has been resolved you will see phones quickly getting registered back to Sub from Pub.

You can then run the same replication checks as described at the start of this post for verification.