Netcontroller is running slowly (or going offline)

Issue

Netcontroller is running slowly or occasionally goes offline (bCX NC2 ACX CX)

Product Line

Andover Continuum

Environment

Continuum

Netcontroller, Net Controller, Infinet, MSTP, Bacnet, Xdriver, x-driver

Cause

Operation of one or more Netcontrollers and or attached controllers is slow and/or goes offline occasionally.

Resolution

General troubleshooting techniques should be used.

  1. Define the exact problem.
    • What exactly is slow? eg. Sending to the controller or its connected devices, opening or refreshing graphics, opening or refreshing points, updating Xdriver points, saving programs, creating new points.
    • Is the controller slow from all cyberstations?
    • Is the controller always slow? If not then try to find what makes it go slow, it is time related? Can it be forced it to go slow?
    • Get a screen dump of any errors seen.
  2. Gather site information.
    1. Continuum version, Continuum hotfixes and service packs applied, number of Cyberstation. Cyberstation operating system, details of Ethernet network and subnets used, with BBMD information of Bacnet sites (usually there should be a spreadsheet available with this sort of information (if not it would be good practice to create one))
    2. Continuum Analyzer report, site controller map or spreadsheet or text dump (network with "Dump child objects, but do not dump children's children" selected, and slow controller with "Dump child objects, but do not dump children's children")
    3. Check the PCs running Cyberstation, are they correct to the installation manual? (RAM, Processor etc)
  3. Gather controller information
    1. What is the Scan rate? Is the controller time sync. correct? How many objects are on the controller, how many programs are running? What is the firmware version of the netcontroller and b3 or i2 controllers?
    2. Ethernet connection - Is the controller on the same subnet as the Cyberstations? A wireshark trace may help to establish the quantity of traffic on the Ethernet (See Lessons Learned Article #5133)
    3. Comms Ports - XDriver
      What is on the comms ports? Xdriver (how much communications, points etc)
    4. Comms Ports - Bacnet MSTP - How many b3 controllers? How much extended logging? How many alarms being sent? How many 3rd party devices? Look at bacnet settings, how many values are being sent across the network? Check COV settings. Try removing the MSTP cable from the bCX (there could be an issue with an MSTP device causing the bCX to go offline/go slow)
    5. Comms Port - Infinet - How many infinet controllers? How much extended logging? How many alarms being sent? Look at infinet settings, how many values are being sent across the network? Check port reconfigs and errors. Try removing the infinet cable from the NetController (there could be an issue with an infinet device causing the NetController to go offline/go slow)
    6. Comms Port - other. Again try removing the comms cable from the NetController (there could be an issue with a device on this network causing the NetController to go offline/go slow)
    7. Run the Controller error log tool to gather the controller errors See Lessons Learned Article #2786
    8. If the controller goes offline, take a copy of the Continuum alarm Log, does tie up with the error log?
    9. Check the CPU LED flash rate on all connected devices (i1, i2, b3 etc) as http://buildingskb.schneider-electric.com/view.php?AID=5414, also the RD and TD LEDs should be checked for anything unusual
  4. This guide is primarily to help the on site engineer to find the faults by helping them to assess the site are were the problem is likely to be. If there is still a need to contact Product Support Services, please review and record items above in a .zip file to speed the fault finding process. Much of the above can be generated by creating listviews.
  5. Further tests can be carried out including an ACCtrace to determine if the restriction could be at the Cyberstation, See Lessons Learned Article #5220
    NOTE: Ensure all Cyberstations, Controllers and other laptops used for logging are in time synchronisation before recording any logs.
  6. If it is believed there is an issue with the Infinet or MSTP network see Lessons Learned Article #5820
  7. If it is believed the problem is with extended logging see Lessons Learned Article #3275
  8. It is possible that something connected to the controller causes the controller to go offline, try disconnecting all other cables and connections to the controller and then re-testing
  9. Where a controller is offline when connected directly to a PC/Laptop, try resetting the IP address, otherwise the controller will probably need replacing.

NOTE
Disconnecting all networks from the NetController is often a good starting point (reconnected IP/Ethernet cable)