Issue
Alarms are being logged but are not being delivered to all Cyberstations
Environment
CyberStation, MSMQ
Cause
MSMQ issues
Resolution
When enhanced alarm delivery is selected and Cyberstations are taken offline, they will not receive alarms until Cyberstation is restarted. Occasionally you can get into a state where even though Cyberstation has been restarted you no longer receive alarms and there are a lot of messages building up in the MSMQ queues.
Event Notifications
Eliminate Cyberstations that no longer exist or are often offline and are setup as recipients on event notifications. These Cyberstations should be removed as recipients to avoid alarm messages from building up in the MSMQ queues.
Three Reasons MSMQ does not deliver messages and some analysis tools
According to MS there are several reasons why MSMQ messages do not get sent/received and in order of precedence are Connectivity, Security and Resource depletion
-
Connectivity
-
In Control Panel>Administrative Tools>Computer Management>Services & Applications>Message Queuing>Private Messaging>accalarmchannel, columns can be added to the queue messages line by right clicking, select View, add/remove columns Add Recipient queue and Response queue
This gives you visibility into what workstations are receiving messages - Ping - The amount of time that a ping test takes to respond can indicate a problem, as can the fact that such a test succeeds only intermittently. (The latter would indicate such issues as name resolution failing and the computer falling back to doing broadcasts for name resolution or network saturation). ping machinename -t this will run until stopped by pressing Ctrl-C, some baselines on each of the machines to the other machines would be good. Once a machine starts building up messages in queues, perform this ping command again to see if there's a difference.
-
Ethereal (now Wireshark)
Use Ethereal on one of the machines that is building up messages in a queue, it would be helpful to get an ethereal capture to look at the MSMQ traffic.
-
In Control Panel>Administrative Tools>Computer Management>Services & Applications>Message Queuing>Private Messaging>accalarmchannel, columns can be added to the queue messages line by right clicking, select View, add/remove columns Add Recipient queue and Response queue
-
Security
- A Firewall can affect MSMQ delivery. Some firewalls allow configuration down to which specific machines are allowed for the MSMQ service. Here's a Microsoft link to what ports need to be opened for MSMQ in addition to what we require for Continuum. http://support.microsoft.com/kb/183293
-
Resources
-
MSMQ has a size limitation for the total amount of space used by queuing messages. If this total size has reached 1.4 Gig then more than likely the storage capacity of MSMQ has been exceeded and the MSMQ service may not start and therefore not deliver messages.
Determine if the MSMQ capacity has been exceeded
On each of the WS that should receive messages, go to My Computer c:\windows or c:\WINNT\system32\msmq\storage, right click on the storage folder and go to properties, note the Size.- Perfmon- is a utility that can be run to take a look at resources on a machine and the MSMQ Service attribute Total Message can be added.
- Go to Start> Settings> Administrative Tools> Double click on Performance
- Keep the defaults that are listed, there should be 3 items listed for memory, Physical Disk and Processor
- Click on the "+" button to add a counter
- In the performance object drop down, select MSMQ service and under Select counters, select "Total messages in all queues"
- Click Add, then Click Close
- Right click on Total Messaging and select properties, select a color and width that will stand out, click Ok
- Keep an eye on the resources that are being used and adjust the machine or add resources accordingly.
- Logging - if it is determined that you need to log some of this info, there are some additional instructions from Microsoft on doing that at http://support.microsoft.com/kb/150934
-
MSMQ has a size limitation for the total amount of space used by queuing messages. If this total size has reached 1.4 Gig then more than likely the storage capacity of MSMQ has been exceeded and the MSMQ service may not start and therefore not deliver messages.