Archive for June, 2010

MOH Issues and Resolution

Posted: June 29, 2010 in Media Resources

There are quite a few issues which you may come across while configuring Music on Hold.

As of SRST/ITS 3.0 an IOS branch gateway has the capability to function as a multicast Music on Hold (MoH) resource. This allows for a distributed MoH design where MoH at remote sites is provided by the local gateway, which can save valuable bandwidth by not having to stream MoH across the WAN.

The MoH capabilities of IOS are very simple. IOS can be configured to permanently multicast RTP packets from an audio file stored locally on flash. This multicasting/streaming is not under the control of CCM, in fact CCM is not even aware of the fact that an IOS MoH resource exists in the network. By configuring IOS MoH to multicast to the same multicast IP address and port as that used by CCM, we can simply block multicast packets originated by CCM, and phones and gateways at a remote site will instead pickup the identical packets being multicast by the local branch gateway.

Note that IOS MoH works for multicast MoH only. Unicast MoH is not supported.

CUCM Checks:

  • The Communications Manager configuration must be designed appropriately.  This involves enabling multicast MoH, and configuring Media Resource Groups (MRG) and Media Resource Group Lists (MRGL) to control which devices get multicast and which get unicast MoH. It also involves assigning Regions so that g711 is used whenever an IOS MoH resource is being invoked.
  • On the “Music On Hold (MOH) Server Configuration” tick the Enable Multicast Audio Sources on this MOH Server box. Note that this MoH server can still unicast MoH, we are just expanding it’s capabilities to include multicast MoH. Specify a base for the IP multicast address and port numbers. For example IP address 239.1.1.1 and port 16384.
Inc. Multicast on IP Address Inc. Multicast on Port Number
Audio Stream Codec Dst. IP Address Dst. Port Dst. IP Address Dst. Port
1 G.711 ulaw 239.1.1.1 16384 239.1.1.1 16384
1 G.711 Alaw 239.1.1.2 16384 239.1.1.1 16385
1 G.729 239.1.1.3 16384 239.1.1.1 16386
1 Wideband 239.1.1.4 16384 239.1.1.1 16387
2 G.711 ulaw 239.1.1.5 16384 239.1.1.1 16388
2 G.711 Alaw 239.1.1.6 16384 239.1.1.1 16389
2 G.729 239.1.1.7 16384 239.1.1.1 16390
2 Wideband 239.1.1.8 16384 239.1.1.1 16391
  • On the “Music On Hold (MOH) Audio Source Configuration” select audio source number 1. Tick the Allow Multicasting box. This will allow this audio source to be used for multicast MoH. Note that it can still be used simultaneously for unicast MoH.
  • It is important to realize that only a single audio source can be used throughout the network when IOS MoH is used.. Even if you have 500 sites and only one site is using IOS MoH you are subject to this restriction.
  • Place the MRGL with Multicast MOH Group in Device Pool or Remote Gateway
  • IOS MoH supports G.711 only. So it is critical that CCM be configured to open up a G711 MoH connection when a device at the remote branch is placed on hold.
  • First test the MOH from CUCM. Even with an IOS MoH resource present the CCM MoH configuration that we have just put together must work. If the WAN is multicast enabled then placing a remote IPhone and/or gateway on hold is the ultimate test to validate the configuration.  If the WAN is not multicast enabled then it will not be possible for the CCM MoH server to stream MoH to the remote device. In this case the second best test is to place an IPhone on the same subnet as the CCM MoH server and verify that Moh can be heard. Since the phone and MoH server are on the same subnet no multicast routing capabilities in the network are required.
  • Use perfmon to verify that the MoH you are hearing is provided via multicast and not unicast. Since unicast MoH is enabled by default it is easy to mistakenly conclude that multicast MoH is working when in fact it is not. Monitor the MOHMulticastResourceActive and MOHUnicastResourceActive counters under the Cisco MOH Device performance object. The multicast counter must be the one incrementing in order for IOS MoH to later work.

IOS & Gateway Checks:

  • The IOS MoH gateway must be running 12.2(15)ZJ2, or 12.3(2nd)T and later when available.
  • Copy the desired MoH audio file to flash on the gateway. The MOH file can be in .wav or .au file format, but must contain 8-bit 8 kHz data, such as a-law or u-law data format.  A known working MoH audio file (music-on-hold.au) is included in its-3.0.1.zip and srst-3.0.zip, which can be downloaded from http://www.cisco.com/cgi-bin/tablebuild.pl/ip-key
  • Configure the music on hold commands as described in previous MOH section.

Troubleshoot IOS MoH to PSTN:

  • If the PSTN caller on hold hears tone on hold (ToH) instead of MoH, then one can conclude that a configuration exists on the CCM side. CCM has failed to activate MoH and has used ToH as a last resort. In this case use the trouble shooting guidelines described in the CCM section.
  • If the PSTN caller on hold hears complete silence, then one can conclude that the problem lies on the IOS or network side. In this case CCM has successfully completed the multicast MoH handshaking with the GW, but the gateway is failing to pick up the locally generated multicast RTP stream. In this case use the ‘show ccm-manager music-on-hold’ command to troubleshoot as described below.

HQ#show ccm-manager music-on-hold

Current active multicast sessions : 1

Multicast Address RTP Port Number Packets In/Out Codec Incoming Interface
239.1.1.1 16387 326/326 42 G711ulaw L
  • If no MoH streams are shown by this command then CCM has failed to provide the gateway with MoH. In this case you’ll normally hear tone on hold instead. The typical cause for this is CCM not having an appropriate MoH resource available. This could be because the required codec is has not been enabled on CCM (check the service parameters). Or it could be because no MRGL has been assigned to the gateway, or if one has been assigned then it has insufficient resources. Check the Event Viewer for error message
  • The IP address and port number above is the MoH source that IOS has been directed to listen to by CCM. Make sure that these match those configured in IOS MoH with the multicast moh command.
  • The ‘show ccm-manager music-on-hold’ command shows only PSTN connections on hold. It does not show multicast streams going to IPhones on hold. When an IPhone is placed on hold, and the MoH is sourced from an IOS gateway, the MoH signaling is directly between CCM and IPhone. In this scenario the IOS gateway plays no role other then to blindly stream multicast RTP packets to the IPhone.
  • Keep in mind that hearing MoH does not necessarily mean that multicast IOS MoH is working as expected. If the PSTN caller hears MoH, but ‘show ccm-manager music-on-hold’ shows no active multicast streams, then the MoH is coming from a unicast. This can be confirmed by checking the MoH perfmon counters as discussed earlier.
  • Check that the remote IP address is MOH Multicast IP

HQ#show call active voice | include RemoteMedia
RemoteMediaIPAddress=239.1.1.1
RemoteMediaPort=16384

Troubleshoot IOS MoH to IP Phones:

  • When an IPhone is placed on hold, and the MoH is sourced from an IOS gateway, the MoH signaling takes place directly between CCM and IPhone.
  • The IOS gateway places no other role than to blindly stream multicast RTP packets to the IPhone.  So other than verifying that the IOS gateway is streaming multicast RTP packets out the Ethernet interface where the IPhones reside, there is no other troubleshooting that can be performed on the gateway.
  • In order to verify that CCM is correctly signaling the IPhone to listen for multicast MoH, place the IPhone on hold. Then check the MOHMulticastResourceActive and MOHUnicastResourceActive counters under the Cisco MOH Device performance object. The multicast counter must be the one incrementing in order for IOS MoH to later work.
  • If MoH signaling looks OK, but no MoH is heard, connect a sniffer to the PC port on the back of the port. If the IPhone and IOS MoH gateway are connected to the same subnet then multicast RTP packets should be seen here at all times, even when the IPhone is not placed on hold. If IPhone and IOS MoH gateway are not connected to the same subnet then, then multicast RTP packets are only seen when the IPhone is placed on hold and sends an IGMP join to the closest router.

So many times in real world you will come across ‘Database replication Issues’. Replication Issues are also very much possible during your CCIE lab so one must be prepare to deal with such Issues.

How can you find if your cluster has proper replication? Here are few tools to find out replication Issues:

RTMT:

RTMT (Real-time monitoring Tool) is a very useful tool for not only checking replication Issues but for lot of other stuff. I will discuss RTMT in detail in some other section. To find out replication status using RTMT, you will need to click the ‘Call manager’ tab on bottom left and then double click ‘Database Summary’. You will get the following stats for a healthy replication.

As long as ‘replication status’ shows same number across the cluster – that means replication is good. If it’s not same then that means replication is not good and you better take some action really fast!

CUCM Unified Reporting:

Go to Unified Reporting > System Reports > Unified CM Database Status > Generate new Report

CUCM OS Admin CLI:

SSH onto Publisher box and Issue the following command:

admin:utils dbreplication status
********************************************************************************                                                                              ********************
This command reads and writes database information from all machines and will take quite some time…please be patient.
********************************************************************************                                                                              ********************

——————– utils dbreplication status ——————–

Output is in file cm/trace/dbl/sdi/ReplicationStatus.2010_06_28_18_24_51.out

Please use “file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_06_28_18_24_51.out ” command to see the output

Control-C pressed

admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_06_28_18_24_51.out

SERVER                 ID STATE    STATUS     QUEUE  CONNECTION CHANGED
———————————————————————–
g_ucmpub_ccm7_0_1_11000_2    2 Active   Local           0
g_ucmsub_ccm7_0_1_11000_2    3 Active   Connected       0 Jun 28 17:27:04

————————————————-
No Errors or Mismatches found.
Replication status is good on all available servers.

utils dbreplication status output

To determine if replication is suspect, look for the following:
(1) Number of rows in a table do not match on all nodes.
(2) Non-zero values occur in any of the other output columns for a table

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_91_typedberrors with 899 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_91_typedberrors ——
Node                  Rows     Extra   Missing  Mismatch Processed

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 – 20 of 2901) :
—————- ——— ——— ——— ——— ———
g_ucmpub_ccm7_0_1_11000_2       899         0         0         0         0
g_ucmsub_ccm7_0_1_11000_2       899         0         0         0         0

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_41_typeadminerror with 57 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_41_typeadminerror ——
Node                  Rows     Extra   Missing  Mismatch Processed
—————- ——— ——— ——— ——— ———
g_ucmpub_ccm7_0_1_11000_2        57         0         0         0         0
g_ucmsub_ccm7_0_1_11000_2        57         0         0         0         0

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_25_mohaudiosource with 51 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_25_mohaudiosource ——
Node                  Rows     Extra   Missing  Mismatch Processed
—————- ——— ——— ——— ——— ———
g_ucmpub_ccm7_0_1_11000_2        51         0         0         0         0
g_ucmsub_ccm7_0_1_11000_2        51         0         0         0         0

Processing ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_44_typeannouncements with 19 rows group 1
——   Statistics for ccmdbtemplate_ucmpub_ccm7_0_1_11000_2_1_44_typeannouncements ——

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 – 40 of 2901) :

Replication Indicators:

  1. Even after configuring everything perfectly fine, things won’t work
  2. If Sub is your Primary Call manager for Phone registration then due to replication Issues phones will not register to Sub and will keep on looping with an error on screen ‘DB-Error – Registration Rejected’.
  3. You configure something on Publisher, it won’t come up on Sub

Solution:

This is what I came across as the most simplest solution out of many.

  • Take off Subscriber Server from CUCM group
  • Shut down all services at Sub
  • SSH onto Publisher and issue the command

admin:utils dbreplication
utils dbreplication clusterreset
utils dbreplication dropadmindb
utils dbreplication forcedatasyncsub
utils dbreplication repair
utils dbreplication reset
utils dbreplication setrepltimeout
utils dbreplication status
utils dbreplication stop

admin:utils dbreplication rep
admin:utils dbreplication repair ?
Syntax:
utils dbreplication repair [nodename]|all
nodename = which node to repair replication on.
all = fix replication on all nodes.

admin:utils dbreplication repair all

Give it sometime to repair and when it is done then restart all services at Subscriber and Put it back in CUCM group. If the replication issue has been resolved you will see phones quickly getting registered back to Sub from Pub.

You can then run the same replication checks as described at the start of this post for verification.

I have come across so many customers who have the same issue – Extension mobility won’t work when you try to login. It will log you in but then quickly logs out the users. The only workaround is as follows:

  • Restart the TFTP service on all servers
  • Restart the Extension mobility service on Publisher
  • Restart the Call manager service across the cluster

This bug has been found in 6.1.1.

Music on hold is a very useful feature in Call manager which is used for streaming music when you put a PSTN phone or an internal IP phone on hold.

I will discuss briefly how it is configured and how you may use CUCM to stream music from remote gateways.

  1. Go to Media Resources > Music on Hold Source
  2. Check the box ‘Allow Multicasting’ 
  3. Go to Media Resources > Music on Hold Server
  4. Select the device pool – DP-MOH (This device pool should have a region REG-MOH which has G711 codec with all other regions
  5. Create a MRG and select the MOH Server in it – don’t forget to check the box ‘Use Multicast for MOH Audio’
  6. Select the MRG in an MRGL
  7. Assign the MRGL to a device pool and place the gateway in the same device pool
  8. Check IP Voice Media Streaming App (IPVMA) is activated
  9. At the gateway, configure the following

ip multicast routing

Int Loopback 0
ip address 10.10.110.2
ip pim sparse-dense-mode

Int vlan 300
ip address 10.10.201.1
ip pim sparse-dense-mode

(Vlan 300 is voice vlan)

ccm-manager music-on-hold

telephony-service
max-ephone 2
max-dn 2
ip source-address 10.10.210.1 port 2000
moh music-on-hold.au
multicast moh 239.1.1.1 port 16384 route 10.10.110.2 10.10.210.1
create cnf-files

Note: Telephony-service command is very important for the router flash to stream MOH. Even if you are not using SRST, there must be telephony-service or call-manager-fallback command for MOH. Also, the ‘ip source-address’ and ‘max-eph’ ‘max-dn’ is essential.

Verification:

R2#sh flash
-#- –length– —–date/time—— path
1     58084804 Jan 21 2009 12:15:48 c2800nm-adventerprisek9_ivs-mz.124-20.T1.bin
2          720 Feb 11 2009 19:58:34 VLAN.DAT
3          720 Jun 12 2010 09:09:10 vlan.dat
4       496521 Feb 12 2010 18:19:06 music-on-hold.au

4665344 bytes available (59346944 bytes used)

Check that ‘music-on-hold.au’ is on flash.

Do a show ephone summary and check if MOH file is active:

R2#sh ephone summary

hairpin_block:
Max 2, Registered 0, Unregistered 0, Deceased 0 High Water Mark 3, Sockets 0
ephone_send_packet process switched 0

Max Conferences 8 with 0 active (8 allowed)
Skinny Music On Hold Status
Active MOH clients 0 (max 210), Media Clients 0, B-ACD Clients 0
File music-on-hold.au type AU Media_Payload_G711Ulaw64k  160 bytes
Moh multicast on 239.1.1.1 port 16384 via 10.10.110.2
via 10.10.201.1

Do a debug ephone moh and check if router is streaming MOH:

R2#deb ephone moh

Jun 28 23:07:43.299: MoH route If Loopback0 46 10.10.110.2 via 10.10.110.2
Jun 28 23:07:43.299: MoH route If Vlan240 ETHERNET 10.10.201.1 via ARP
R2#
Jun 28 23:07:48.747: MoH route If Loopback0 46 10.10.110.2 via 10.10.110.2
Jun 28 23:07:48.747: MoH route If Vlan240 ETHERNET 10.10.201.1 via ARP
R2#
Jun 28 23:07:50.551: ifs_read flash:music-on-hold.au end of file at 490557 read 5964 = 496521
Jun 28 23:07:50.555: moh tail fill from 24 at 0x4A73816C length 2036

During a HELD call, issue command ‘show ccm-manager music-on-hold’ … this will show the ports/interfaces used for Music on hold streaming.

R2#sh ccm-manager music-on-hold
Current active multicast sessions : 1
Multicast       RTP port   Packets       Call   Codec    Incoming
Address         number     in/out        id              Interface
===================================================================
239.1.1.1         16384   508/508          1    g711ulaw  Lo0

Also you can check the multicast Interface 239.1.1.1 status by using following command:

R2#sh ip mroute
IP Multicast Routing Table
Flags: D – Dense, S – Sparse, B – Bidir Group, s – SSM Group, C – Connected,
L – Local, P – Pruned, R – RP-bit set, F – Register flag,
T – SPT-bit set, J – Join SPT, M – MSDP created entry,
X – Proxy Join Timer Running, A – Candidate for MSDP Advertisement,
U – URD, I – Received Source Specific Host Report,
Z – Multicast Tunnel, z – MDT-data group sender,
Y – Joined MDT-data group, y – Sending to MDT-data group,
V – RD & Vector, v – Vector
Outgoing interface flags: H – Hardware switched, A – Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.1.1.1), 00:01:01/00:01:59, RP 0.0.0.0, flags: DCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Vlan240, Forward/Sparse-Dense, 00:01:01/00:00:00

(*, 224.0.1.40), 02:14:08/00:02:52, RP 0.0.0.0, flags: DCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Loopback0, Forward/Sparse-Dense, 02:14:08/00:00:00

Also when the call is on hold:

R2#sh call active voice | i RemoteMedia
RemoteMediaIPAddress=239.1.1.1
RemoteMediaPort=16384

If the remote media IP address shows up as 0.0.0.0 then there is a good chance that you have encountered CSCdz00697.

I will discuss MOH issues in another section.

Background Image on Gen 2 and Gen 3 Cisco IP phones can be changed by using the following procedure.

I will be showing you how to do for a Cisco 7965 IP phone but the same procedure can be used for other IP phones (like 7941. 7961, 7945 etc). The only difference between different family of phones will be the Image sizes.

1) Resize the Images as per 7965 Phone.
2) Full size image— 320 pixels (width) X 212 pixels (height) and the thumbnail image— 80 pixels (width) X 53 pixels (height).
3) Save the full size file as ‘large.png’ and the thumbnail as ‘small.png’ – (you can give any name as long as it match exactly in List.xml)
4) Create a notepad file called “List.xml” – do change the extension to .xml.
5) Add the following lines in List.xml

<CiscoIPPhoneImageList>
<ImageItem Image=”TFTP:Desktops/320x212x16/small.png”
URL=”TFTP:Desktops/320x212x16/large.png”/>
</CiscoIPPhoneImageList>

6) Save the file and double click it to open it, the file should open in a web browser
7) Load the Image files and List.xml on CUCUM TFTP


8 )Repeat the same process for the two Image files
9) Upload all the files on both Publisher and Subscriber
10 Restart TFTP service

Go to the phones > settings > user settings > Background Image and select the image from there.

The information about customization can be accessed from Cisco Support link.

Problem description:

One of my customer reported an issue with their VG224. All Polycom conference phones connected to the Vg224 were facing an strange issue of DTMF tones not recognized by PSTN. Users trying to get into a MeetMe conference were having  tough time as their pin was not getting recognised.

Analysis & Observations:

This issue was passed onto me after some of my colleagues already spent few days trying all sort of things for resolution. I found out from the customer that the phones never worked properly ever since they hooked them in VG224. One thing was clear from this statement that nothing changed on Vg224 or on their network as this has been a known issue since they started using VG224.

I checked the gateway which was a SCCP configured VG224. For your information let me share how a SCCP VG224 is configured:

Configuration:

service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!

!
no aaa new-model
clock timezone GMT 0
clock summer-time BST recurring
!
!
!
stcapp ccm-group 10
stcapp
!
voice-card 0
!
!
!
voice service voip
fax protocol none
modem passthrough nse codec g711ulaw
!
!
!
!
!
!
!
!
!
!

archive
log config
hidekeys
!
!
!
class-map match-any Voice-RTP
match ip dscp ef
match ip rtp 16384 16383
class-map match-any Voice-Cntl
match ip dscp af31
!
!
policy-map QoS-LAN-Policy
class Voice-RTP
set cos 5
class Voice-Cntl
set cos 3
!
!
!
interface FastEthernet0/0
description ### Trunk to EIG LAN ###
no ip address
duplex auto
speed auto
!
interface FastEthernet0/0.1
description ### Management VLAN ###
encapsulation dot1Q 1 native
ip address 10.1.2.175 255.255.0.0
!
interface FastEthernet0/0.205
description ### Voice Server VLAN ###
encapsulation dot1Q 205
ip address 10.205.2.175 255.255.0.0
service-policy output QoS-LAN-Policy
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 10.205.1.1
!
no ip http server
!
access-list 30 permit 10.1.5.1
access-list 30 permit 10.200.4.216
snmp-server community E1GpUb73 RO 30
snmp-server community 3I6pR186e RW 30
snmp-server trap-source FastEthernet0/0.1
!
!
control-plane
!
!
!
voice-port 2/0
input gain 14
cptone GB
timeouts ringing infinity
impedance 900r
!
voice-port 2/1
cptone GB
timeouts ringing infinity
!
voice-port 2/2
cptone GB
timeouts ringing infinity
!
voice-port 2/3
cptone GB
timeouts ringing infinity
!
voice-port 2/4
cptone GB
timeouts ringing infinity
!
voice-port 2/5
cptone GB
timeouts ringing infinity
!
voice-port 2/6
cptone GB
timeouts ringing infinity
!
voice-port 2/7
cptone GB
timeouts ringing infinity
!
voice-port 2/8
cptone GB
timeouts ringing infinity
!
voice-port 2/9
cptone GB
timeouts ringing infinity
!
voice-port 2/10
cptone GB
timeouts ringing infinity
!
voice-port 2/11
cptone GB
timeouts ringing infinity
!
voice-port 2/12
cptone GB
timeouts ringing infinity
!
voice-port 2/13
cptone GB
timeouts ringing infinity
!
voice-port 2/14
cptone GB
timeouts ringing infinity
!
voice-port 2/15
cptone GB
timeouts ringing infinity
!
voice-port 2/16
cptone GB
timeouts ringing infinity
!
voice-port 2/17
cptone GB
timeouts ringing infinity
!
voice-port 2/18
cptone GB
timeouts ringing infinity
!
voice-port 2/19
cptone GB
timeouts ringing infinity
!
voice-port 2/20
cptone GB
timeouts ringing infinity
!
voice-port 2/21
cptone GB
timeouts ringing infinity
!
voice-port 2/22
cptone GB
timeouts ringing infinity
!
voice-port 2/23
cptone GB
timeouts ringing infinity
!
!
!
sccp local FastEthernet0/0.205
sccp ccm 10.210.2.1 identifier 1 version 4.1
sccp ccm 10.210.4.1 identifier 2 version 4.1
sccp ccm 10.205.250.1 identifier 3 version 4.1
sccp
!
sccp ccm group 10
associate ccm 2 priority 1
associate ccm 3 priority 2
associate ccm 1 priority 3
!
!
dial-peer voice 200 pots
description ### Dial Peer for Port 2/0 ##
service stcapp
port 2/0
!
dial-peer voice 201 pots
description ### Dial Peer for Port 2/1 ##
service stcapp
port 2/1
!
dial-peer voice 202 pots
description ### Dial Peer for Port 2/2 ##
service stcapp
port 2/2
!
dial-peer voice 203 pots
description ### Dial Peer for Port 2/3 ##
service stcapp
port 2/3
!
dial-peer voice 204 pots
description ### Dial Peer for Port 2/4 ##
service stcapp
port 2/4
!
dial-peer voice 205 pots
description ### Dial Peer for Port 2/5 ##
service stcapp
port 2/5
!
dial-peer voice 206 pots
description ### Dial Peer for Port 2/6 ##
service stcapp
port 2/6
!
dial-peer voice 207 pots
description ### Dial Peer for Port 2/7 ##
service stcapp
port 2/7
!
dial-peer voice 208 pots
description ### Dial Peer for Port 2/8 ##
service stcapp
port 2/8
!
dial-peer voice 209 pots
description ### Dial Peer for Port 2/9 ##
service stcapp
port 2/9
!
dial-peer voice 210 pots
description ### Dial Peer for Port 2/10 ##
service stcapp
port 2/10
!
dial-peer voice 211 pots
description ### Dial Peer for Port 2/11 ##
service stcapp
port 2/11
!
dial-peer voice 212 pots
description ### Dial Peer for Port 2/12 ##
service stcapp
port 2/12
!
dial-peer voice 213 pots
description ### Dial Peer for Port 2/13 ##
service stcapp
port 2/13
!
dial-peer voice 214 pots
description ### Dial Peer for Port 2/14 ##
service stcapp
port 2/14
!
dial-peer voice 215 pots
description ### Dial Peer for Port 2/15 ##
service stcapp
port 2/15
!
dial-peer voice 216 pots
description ### Dial Peer for Port 2/16 ##
service stcapp
port 2/16
!
dial-peer voice 217 pots
description ### Dial Peer for Port 2/17 ##
service stcapp
port 2/17
!
dial-peer voice 218 pots
description ### Dial Peer for Port 2/18 ##
service stcapp
port 2/18
!
dial-peer voice 219 pots
description ### Dial Peer for Port 2/19 ##
service stcapp
port 2/19
!
dial-peer voice 220 pots
description ### Dial Peer for Port 2/20 ##
service stcapp
port 2/20
!
dial-peer voice 221 pots
description ### Dial Peer for Port 2/21 ##
service stcapp
port 2/21
!
dial-peer voice 222 pots
description ### Dial Peer for Port 2/22 ##
service stcapp
port 2/22
!
dial-peer voice 223 pots
description ### Dial Peer for Port 2/23 ##
service stcapp
port 2/23
!
!
!
line con 0
login local
line aux 0
line vty 0 4
login local
!
ntp clock-period 17179905
ntp server 10.1.1.2
end

I found nothing wrong in the config. I checked the IOS which was running on the gateway since they started using it. The IOS was 12.4(15)T5. I decided to do a bug scrub before I further dive into it. A little use of my  googling skills and I found the issue. This particular IOS for VG224 has a well known bug CSCsg85602. The bug description is “DTMF tones not recognised on calls to PSTN from VG224 due to tone leakage”.

This bug was introduced in DSPware 9.4.5 and seems to be causing the problem. I found out that One possible solution was by downgrading the DSPware on the VG224 (while leaving the IOS at 12.4(15)T5 from 9.4.5 to 9.4.4. Then I read about it and realized this has not always resolved the issue.

After going through several release notes I decided to upgrade the IOS to 12.4(15)T13 which was a stable MD release.

Upgarded the IOS and the issue was gone!

H323 gateway is a non-survivable endpoint. This means an active call will drop if any call manager goes down. This default behavior of an H323 gateway can be changed by playing with few settings at the CUCM and gateway.

At gateway:

voice service voip
h323
call preserve

At CUCM:

Allow TCP Keepalive for H323  = False
Allow Peer to Preserve Call  = True
Restart CCM service of PUB

Verification:

Make an inbound call from PSTN to the phone.
During the call, stop the CCM service at Sub but your call will not drop!

This is what you will see at the gateway:
Issue debug on H323 Gateway – debug h225 events

Jun  3 21:09:07.052: h323chan_chn_process_read_socket: fd=3 of type CONNECTED has data
Jun  3 21:09:07.052: h323chan_recvdata: Connection lost fd=3
Jun  3 21:09:07.052: h323chan_close: TCP connection from fd=3 closed
Jun  3 21:09:07.052: h323chan_chn_process_read_socket: fd=2 of type CONNECTED has data
Jun  3 21:09:07.052: h323chan_recvdata: Connection lost fd=2
Jun  3 21:09:07.056: %CCH323-6-CALL_PRESERVED: cch323_h225_handle_conn_loss: H.323 call preserved due to socket closure or error, Call Id = 4589, fd = 2
h323chan_chn_close: Calls[1] Exist on socketfd=2 Owner[2]
Jun  3 21:09:07.056: h323chan_close: TCP connection from fd=2 closed

MGCP gateway is a Survivable endpoint unlike H323 which is not survivable. What does it mean? It means that an active call through an MGCP gateway will not tear-down and the call will be preserved.

As discussed already in another section that an MGCP gateway terminates Layer 2 signalling at the gateway while Backhauling Layer 3 Q931 Signalling to CUCM through TCP port 2428. Terminating Layer 2 Signal (Q921) at gateway means the D-channel will remain up even if one of the call manager goes down. There are three steps involved in MGCP call preservation which are transparent to user:

  1. CUCM send AUEP (Audit Endpoint) to the gateway to find the state of each B-channel
  2. CUCM send AUCX for any Endpoint for which the gateway reports a preserved call
  3. Send Q.931 Status Enquiry messages to the PRI device attached to the gateway to confirm the status of any calls CUCM believes are preserved.

Verification:

  • Open the debug mgcp packet and debug isdn q931
  • Make an active call
  • Packets flow can be seen between gateway and Primary call manager – Subscriber (10.10.210.11)
  • Shut the Subscriber (deactivate ccm service at Sub)
  • On the gateway you will notice change of IP from Sub to Pub (10.10.210.10)
  • CUCM Pub will now send AUEP to gateway
  • Gateway will send back the status of each channel (with calls, without calls)
  • CUCM will send AUCX to gateway
  • Gateway will send connection details something like “D0000892839238”
  • Pub will take over L3 backhauling

Traces:
Calling 911 from Internal externsion:

Jun  1 20:54:38.905: ISDN Se0/0/0:23 Q931: TX -> SETUP pd = 8  callref = 0x0001
Bearer Capability i = 0x8090A2
Standard = CCITT
Transfer Capability = Speech
Transfer Mode = Circuit
Transfer Rate = 64 kbit/s
Channel ID i = 0xA98381
Exclusive, Channel 1
Display i = ‘Joe Bloggs’
Calling Party Number i = 0x0081, ‘+12025676001’
Plan:Unknown, Type:Unknown
Called Party Number i = 0x81, ‘911’
Plan:ISDN, Type:Unknown
Jun  1 20:54:38.941: ISDN Se0/0/0:23 Q931: RX <- CALL_PROC pd = 8  callref = 0x8001
Channel ID i = 0xA98381
Exclusive, Channel 1
Jun  1 20:54:38.953: ISDN Se0/0/0:23 Q931: RX <- ALERTING pd = 8  callref = 0x8001
Progress Ind i = 0x8188 – In-band info or appropriate now available
Jun  1 20:54:43.089: ISDN Se0/0/0:23 Q931: RX <- CONNECT pd = 8  callref = 0x8001
Jun  1 20:54:43.113: ISDN Se0/0/0:23 Q931: TX -> CONNECT_ACK pd = 8  callref = 0x0001

Communication between Sub and gateway:

Jun  1 20:54:38.889: MGCP Packet sent to 10.10.210.11:2427—>
200 208 OK
I: 2

v=0
c=IN IP4 10.10.200.3
m=audio 16668 RTP/AVP 0 100
a=rtpmap:100 X-NSE/8000
a=fmtp:100 192-194,200-202
a=X-sqn:0
a=X-cap: 1 audio RTP/AVP 100
a=X-cpar: a=rtpmap:100 X-NSE/8000
a=X-cpar: a=fmtp:100 192-194,200-202
a=X-cap: 2 image udptl t38
<—

Connection established:

Jun  1 20:54:39.001: MGCP Packet received from 10.10.210.11:2427—>
MDCX 209 S0/SU0/DS1-0/1@R12.cisco.com MGCP 0.1
C: D00000000213927c000000F500000001
I: 2
X: 1
L: p:20, a:PCMU, s:off, t:b8, fxr/fx:t38
M: recvonly
R: D/[0-9ABCD*#]
Q: process,loop
<—

Jun  1 20:54:39.189: MGCP Packet received from 10.10.210.11:2427—>

MDCX 210 S0/SU0/DS1-0/1@R12.cisco.com MGCP 0.1
C: D00000000213927c000000F500000001
I: 2
X: 1
L: p:20, a:PCMU, s:off, t:b8, fxr/fx:t38
M: sendrecv
R: D/[0-9ABCD*#], FXR/t38
S:
Q: process,loop

v=0
o=- 2 0 IN EPN S0/SU0/DS1-0/1@R12.cisco.com
s=Cisco SDP 0
t=0 0
m=audio 21962 RTP/AVP 0
c=IN IP4 192.168.10.13
a=X-sqn:0
a=X-cap:1 image udptl t38
<—

Jun  1 20:58:20.055: MGCP Packet sent to 10.10.210.11:2427—>
NTFY 521011089 *@R12.cisco.com MGCP 0.1
X: 0
O:
<—

Jun  1 20:58:20.055: MGCP Packet received from 10.10.210.11:2427—>
200 521011089
<—

Subscriber went down – see how Publisher took over:

Jun  1 21:36:12.548: MGCP Packet sent to 10.10.210.10:2427—>
200 28
I:
X: 0
L: p:10-20, a:PCMU;PCMA;G.nX64, b:64, e:on, es-cci, gc:1, s:on, t:10, r:g, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
L: p:10-220, a:G.729;G.729a;G.729b, b:8, e:on, es-cci, gc:1, s:on, t:10, r:g, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
L: p:10-110, a:G.726-16;G.728, b:16, e:on, es-cci, gc:1, s:on, t:10, r:g, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
L: p:10-70, a:G.726-24, b:24, e:on, es-cci, gc:1, s:on, t:10, r:g, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
L: p:10-50, a:G.726-32, b:32, e:on, es-cci, gc:1, s:on, t:10, r:g, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
L: p:30-270, a:G.723.1-H;G.723;G.723.1a-H, b:6, e:on, es-cci, gc:1, s:on, t:10, r:g
R1#, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
L: p:30-330, a:G.723.1-L;G.723.1a-L, b:5, e:on, es-cci, gc:1, s:on, t:10, r:g, nt:IN;ATM;LOCAL, v:T;G;D;L;H;R;ATM;SST;FXR;PRE
M: sendonly, recvonly, sendrecv, inactive, loopback, conttest, data, netwloop, netwtest
<—

EVENT 1 = Audit endpoint

Jun  1 21:36:12.548: MGCP Packet received from 10.10.210.10:2427—>
AUEP 29 S0/SU0/DS1-0/4@R12.cisco.com MGCP 0.1
F: X, A, I
<—

Jun  1 21:36:12.548: MGCP Packet sent to 10.10.210.10:2427—>
500 29 Endpt Unknown
<—

Jun  1 21:36:12.552: MGCP Pa
R1#cket received from 10.10.210.10:2427—>
AUEP 30 S0/SU0/DS1-0/5@R12.cisco.com MGCP 0.1
F: X, A, I
<—

Jun  1 21:36:12.552: MGCP Packet sent to 10.10.210.10:2427—>
500 30 Endpt Unknown
<—

Jun  1 21:36:12.552: MGCP Packet received from 10.10.210.10:2427—>
AUEP 31 S0/SU0/DS1-0/6@R12.cisco.com MGCP 0.1
F: X, A, I
<—

Jun  1 21:36:12.552: MGCP Packet sent to 10.10.210.10:2427—>
500 31 Endpt Unknown

(output omitted)

EVENT 2 = Audit connection

Jun  1 21:36:12.572: MGCP Packet received from 10.10.210.10:2427—>
AUCX 49 S0/SU0/DS1-0/1@R12.cisco.com MGCP 0.1
I: 3
F: C, M
<—

Jun  1 21:36:12.576: MGCP Packet sent to 10.10.210.10:2427—>
200 49
C: D00000000213927e000000F500000002
M: sendrecv
<—

Jun  1 21:36:12.576: MGCP Packet received from 10.10.210.10:2427—>
RQNT 50 S0/SU0/DS1-0/1@R12.cisco.com MGCP 0.1
X: 1
R: R/iu, FXR/t38
Q: process,loop
<—

Jun  1 21:36:12.580: MGCP Packet sent to 10.10.210.10:2427—>
200 50 OK
<—

EVENT 3 = Status enquiry
Jun  1 21:36:12.592: ISDN Se0/0/0:23 Q931: TX -> STATUS_ENQ pd = 8  callref = 0x0002
Jun  1 21:36:12.600: ISDN Se0/0/0:23 Q931: RX <- STATUS pd = 8  callref = 0x8002
Cause i = 0x809E – Response to STATUS ENQUIRY or number unassigned
Call State i = 0x0A
Jun  1 21:36:13.732: MGCP Packet sent to 10.10.210.10:2427—>
NTFY 521011249 *@R12.cisco.com MGCP 0.1
X: 0
O:
<—

Jun  1 21:36:13.732: MGCP Packet received from 10.10.210.10:2427—>
200 521011249
<—

Jun  1 21:36:25.184: MGCP Packet sent to 10.10.210.10:2427—>
NTFY 521011252 *@R12.cisco.com MGCP 0.1
X: 0
O:
<—

To make use of DSP resources for a Conference Call, we will have to enable Conference Bridge at the gateway.

Configure the following at the gateway:

voice-card 0
dspfarm
dsp services dspfarm

sccp local Loopback0
sccp ccm 10.10.210.10 identifier 2 priority 2 version 5.0.1
sccp ccm 10.10.210.11 identifier 1 priority 1 version 5.0.1
sccp
!
sccp ccm group 1
bind interface Loopback0
associate ccm 1 priority 1
associate ccm 2 priority 2
associate profile 1 register HQ-conf  << Name must match at CUCM
!
dspfarm profile 1 conference
codec g729r8
maximum sessions 2
associate application SCCP
no shut  << Very Important

At CUCM under Media Resources  > Conference Bridge

Conference Bridge

After adding the Conference Bridge, add it under Media Resource Group and then to MRGL.

Add the MRGL under particular device pool or Gateway.

Verification:

Show sccp conn

show dspfarm all

Transcoding is required to enable communication between different Codecs. For example, CUE is G.711 only and will not be reachable from a Branch side if Inter-region codec has been set as G.729. Similarly a UCCX Pilot point is G.711 only and cannot be reached from another site using G.729 codec. To make this communication possible, a transcoder sitting at terminating gateway do media conversion from one codec to another.

Setting up a IOS transcoder is same as Conference Bridge.

At the gateway use the following commands:

voice-card 0
dspfarm
dsp services dspfarm

sccp local Loopback0
sccp ccm 10.10.210.10 identifier 2 priority 2 version 5.0.1
sccp ccm 10.10.210.11 identifier 1 priority 1 version 5.0.1
sccp
!
sccp ccm group 1
bind interface Loopback0
associate ccm 1 priority 1
associate ccm 2 priority 2
associate profile 1 register HQXcode
!
dspfarm profile 1 transcode
codec g729r8
maximum sessions 2
associate application SCCP
no shut

!

At the Call manager under Media Resources > Transcoder

Transcoder

Verfication:

Show sccp connection

show dspfarm session

show dspfarm all