I am working in a full production environment that has a range of PLCS around our production mill, each of these PLC's talk back through a 'DataHighway +' network back to a special PC on our LAN Network called the MicroLinks PC. This has the ROCKWELL OPC RSLinx Classic server software on it.
So, recently I have put together a piece of .NET software in c# using the OPC .NET API to read to ROCKWELL OPC server on the Microlinks PC and sync data back into our MYSQL database that is sat on our WINDOWS R2 server PC
Ever since turning on the .net software, the engineers on site have experienced a massive slow down in developing new PLC scripts and fault finding.
Some of the reports are even as bad as 10 second lags.
Consequently, we have had to turn of the .NET software to sync the data to allow the Engineers to do their work swiftly without issues.
So i am looking for some advice on where or what i should look for, any resources to read for this type of problem etc. As PLC and networks are way out of my depth, I am just the .NET programmer.
Here is the structure of our network:
I'm not sure which type of rockwell PLCs you are using. I'm most familiar with the ControlLogix platform so I'll talk about that.
The ethernet card in a controllogix PLC connects at 100Mb/s but the card can't actually handle 100Mb/s continuously. A 1756-ENBt card can handle about 5000 packets per second, the EN2T roughly double that. There are formulas in the rockwell docs explain how to calculate packets per second but another option when you have a running system is to connect the 'Logix5000 task monitor' that comes with RS Logix and verify the CPU usage of the Ethernet card I think Rockwell recommends you keep it under 60%. If you are requesting too many packets then this CPU won't keep up
The PLC itself can starve communications. Controllogix has a "overhead time slice" setting which is the percent of time the PLC spends servicing communication tasks as opposed to running its own logic. Increasing this percentage can improve comms a bit.
It sounds like your program is putting a large burden on the PLC. Does it get better if you slow down your app so that it is not pulling as much data as fast?
One easy way to reduce the number of packets required to retrieve a block of data without slowing down the update rate is to put it all in one array. RSLinx will then be able to optimize the request instead of pulling individual tags
I have had plenty of troubles using Rockwell RSLinx on my local PC trying to find the IP address of a PLC plugged directly into my ethernet port. Using the "Autobrowse" option, it completely locks up my PC trying to scan the ports and IP addresses for targets.
It might just be poorly optimized Rockwell software causing issues. You also may be exchanging a whole lot of data and your server PC is struggling to keep up.
I would contact Rockwell/Allen Bradley support for help with this. They will probably want some cash to help you.
You're almost deinifitely over-polling the PLC. Try polling less and less frequently until find a value that doesn't slow down the network For example, if you're requesting data every 100ms now, change that once per second. Then once per minute. Then once every 15 minutes. At each step check the comms speed for the programming terminals.
Related
I'm developing a gaming TCP server and I'm planning that it should withstand up to about 500 connections. Data transfer occurs constantly (movement of the character, attack, etc.).
I started develop with the TCPListener, but I got a performance problem.
I used await XXXSync methods to accept clients, receive data, and sending messages.
I created a program that simulates the actions of players. It allowed to create only 30-40 connections and this caused 100% of the CPU. To test my server, I used two machines: one was a simulator of players, on the other - a server.
These 30-40 clients load the CPU on the server by 10-20% and very much (at 100%) while simultaneously disconnecting all TCP clients.
I was looking for information about optimizing the TCPListener, but nothing found. I've seen topics where people have written that they own a server with 1,000+ connections and this works well.
I thought that the reason could be in the logic of my server, but the profiling tools showed that the main part of the CPU load is getting and sending data (await client.ReadAsync / client.WriteAsync).
Picture 1 | Picture 2
I started looking for possible alternatives and found SocketAsyncEventArgs. At the forum, people wrote that this method can increase productivity by 2 times. I found the source codes of some servers written in C # and took from there the implementation of SocketAsyncEventArgs.
This really greatly increased the productivity by 10-15 times. For 30 simulations, this causes now only 0.5-2%.
I rewrote the emulator of the players at SocketAsyncEventArgs and this allowed me to create 100 simulations with 20% CPU.
After 70 players the load on the CPU on the server grew and is 7-13%.
Profiling tools show that the main use of the CPU is the sending of messages.
For some reason, the most productive option for me was BeginWrite. Write was not suitable because it blocks the stream. WriteAsync was slower, or I just misused it. Picture 3 | Picture 4
I can not find information about optimizing Sockets on c #. Everyone everywhere writes that they are all well even on the TCPListener, which for me is simply unbearable.
Is there any other way to improve performance?
P.S. I tested the server on a computer with i7 seventh generation.
I've read a lot about this topic, but still am not sure what to do.
First, the situation: I have software written in C# using .NET 4.5 that polls up to 64 devices on a CAN network that I communicate with via USB using a third party API from the device manufacturer. The purpose is to provide the user with realtime updates of temperature, pressure, and other values like that from some sensors.
Currently I create a System.Threading.Thread for every device which runs a while loop that queries the device for the relevant info, saves updates to SQL Server via Entity Framework, then sleeps for 1.25 seconds.
This runs ok on smaller systems with ~20 or fewer devices, but on a large install with 50+ devices it runs very slowly. I think that my problem is the overhead of creating so many threads. And it doesn't help that I'm stuck with a crappy Atom processor, although at least this one is quad core unlike the previous system I used that was dual core.
So, I've been trying to make the process more efficient. Everything I read seems to point to Task.Run() being the more effective way of doing something like this, but this software could potentially be running for weeks or months at a time, which I THINK means I would need to run it with TaskCreationOptions.LongRunning. But I've read conflicting things on this, so I'm not sure. But if that is the case, then my understanding is that TPL will just start up a new dedicated thread anyways, so it seems like that would still have the overhead I'm trying to avoid.
So, as you can see, I'm pretty lost on this topic. I don't know if I should just give Task.Run() a try, and see what happens, or if there's a whole different way I should do this.
Any help would be immensely appreciated.
Thank you.
I'm creating a program (it's a game) which will have multiplayer. It will need both client-to-server and server-to-client, with the server being run on one of the player's machines, not on a separate machine. I'll be using C# sockets, and I need to know the maximum amount of data that they can handle. For instance, my data will be between 256b and 128kb, I can assume. Am I going to have any trouble sending that through the sockets? (There will never be more than 7 clients connected to the server-handler machine).
EDIT
After reading some other posts, I don't think it will be a problem. Others have said that sending data upwards of 1MB doesn't cause a problem, so I don't think I'll have an issue.
P.S.
If this is not the case, please let me know. Thanks!
We are currently investigating the most efficient way of communicating between 120-140 embedded hardware devices running on the .NET Micro framework and a server.
Each embedded device needs to send to, and request information from the server on a fairly regular basis all in real time through TCP.
My question is this: Would it be better to initialise 140 TCP connections to the server, and then hang on to these connections, or initialise a new connection for each requests to and from the devices? Would holding on to and managing 140 TCP connections put a lot of strain on the server?
When the server detects new data in the database it needs to send this new info to 1..* devices (information is targeted to specific devices), if I held on to the 140 connections I would need to do a lookup for the correct connection each time I needed to send information instead of just sending to an IP:PORT associated with the new data.
I guess another possibly stupid question would be is it actually possibly to hang on to 140 TCP connections on a single port?
Any suggestions/comments are appreciated!
In general you are better maintaining the connections for as long as possible. If you have each device opening a connection each time it sends a message you can end up effectively DoS'ing the server as it ends up with lots of sockets in the TIME_WAIT state taking up space in it's tables.
I worked on a system where there were a bunch of clients talking to a server and while they could be turned on and off regularly, it was still better to maintain the connection (and re-establish it when it had dropped and a new message needed to be sent). You may end up needing to write slightly more complex code, but I've found it to be well worth the effort for the reduced load on the server.
Modern operating systems may have bigger buffers than the ones I actually encountered the DoS effect on, but it's fundamentally not the best idea to be using lots of connections like that.
Things can get relatively complicated on the client side, especially when the device tends to go to sleep transparently to the application because that means connections will time out while the app thinks they are still open. When we did this we ended up with relatively complex network code because we needed to deal with the fact that the sockets could (and would) fail as a matter of course and we simply needed to setup a new connection and re-attempt sending the message. You just tuck this code away into your libraries and forget about it once it's done though.
In actual fact in practice our initial application had even more complex code because it was dealing with a network library that was semi-aware of the stop start nature of the devices and tried to resend failed messages, sometimes meaning that the same message got sent twice. We ended up doing an extra layer of communication on top in order to ensure duplicates got rejected. If you're using C# or regular BSD style sockets you shouldn't have that problem though I'm guessing. This was a proprietary library that managed the reconnects but caused headaches with the resends and it's inappropriate default time-outs.
You usually can connect much more than 140 "clients" to a server (that is with decent network / HW / RAM)...
I recommend always to test this sort of thing with real scenarios (load etc.) to decide since there are aspects like network (performance, stability...), HW (server RAM etc.) and SW (what does the server exactly do?) that can only be checked by you.
Depending on the protocol you could/should even put some timeout/reconnect mechanism in there.
The lookup you mean would be really fast - just use ConcurrentDictionary to hold the needed information with IP:PORT as the key (assuming the server runs on a full .NET 4).
For some references see:
http://msdn.microsoft.com/en-us/library/dd287191.aspx
http://geekswithblogs.net/BlackRabbitCoder/archive/2011/02/17/c.net-little-wonders-the-concurrentdictionary.aspx
EDIT - as per comments:
Holding on to a TCP/IP connection doesn't take much processing client-side... it costs a bit of memory. I would recommend to do a small test (1-2 clients) to check this assumption for your specific case.
If you are talking about a system with hardware devices then I suggest to go with closing the connection every time the client finishes sending data.
To make sure the client gets some update from the server, the client can wait for a 5 second period for any data to arrive from the server. If the data is received within/before this timeframe, then close the connection and process the data. If not, close the connection and wait after sending next set of data.
This way scaling becomes much easier. Keeping the connections open always leads to strain on the resources and in my opinion is not necessary unless it is some life-saving device like heart rate monitor, oxygen supply monitor etc.,
I have a series of systems on a LAN running a synchronized display routine. For example, think of a chorus line. The program they ran is fixed. I have each "client" download the entire routine, and then contact the central "server" at fixed points in the routine for synchronization. The routine itself is mundane with, perhaps, 20 possible instructions.
Each client runs the same routine, but they can be doing completely different things at any one time. One part of the chorus line can be kicking left, another part kicking right, but all in time with each other. Clients can join and drop out at any time, but they're all assigned a part. If no-one is there to run the part, it just doesn't get run.
This is all coded in C# .Net.
The client display is a Windows Forms application. The server accepts TCP connections, and then services them round-robin fashion, keeping a master clock of what's going on. The clients send a signal that says "I've reached sync-point 32" (or 19, or 5, or whatever) and waits for the server to acknowledge and then moves on. Or the server can say "No, you need to start at sync-point 15".
This all works great. There is a minor bit of delay between the first and last clients to hit a sync-point, but it's hardly noticeable. Ran for months.
Then the Specification changed.
Now the clients need to respond to near real-time instructions from the server -- it's no longer a pre-set dance program. The server is going to be sending instructions out and the dance program is made up on the fly. I get the fun job of re-designing the protocol, the servicing loops, and the programming instructions.
My toolkit includes anything in a standard .Net 3.5 toolbox. Installing new software is a pain in the arse, since so many systems (clients) can be involved.
I'm looking for suggestions on keeping the clients synced (some sort of latching system? UDP? Broadcast?), distribution of the "dance program", anything that might make this easier than a traditional Client/Server TCP arrangement.
Keep in mind that there are time/speed limitations going on as well. I could put the dance program in a network database, but I'd have to shove instructions in fairly quickly and there'd be a lot of readers using a rather thick protocol (DBI, SqlClient, etc..) to get a small bit of text. That seems overly complex. And I still need something to keep them all displaying in sync.
Suggestions? Opinions? Wild-ass speculation? Code examples?
PS: Answers may not get marked as "correct" (since this isn't a "correct" answer), but +1 votes for good suggestions for sure.
I did something similar (quite a while back) with synchronizing a bank of 4 displays, each run by a single system, receiving messages from a central server.
The architecture we finally settled on after a fair amount of testing involved having one "master" machine. In your case, this would be having one of your 20 clients that acts as the master, and have it connect to the server via TCP.
The server then would send the entire series of commands for the series through to that one machine.
That machine then used UDP to broadcast real-time instructions to each of the other machines (the 19 other clients on its LAN) to keep their displays up to date. We used UDP for a couple reasons here - there was lower overhead involved, which helped keep the total resource usage down. Also, since you're updating in real-time, if one or two "frames" was out of sync, it was never noticable, at least not noticeable enough for our purposes (having a human sitting and interacting with the system).
The key point to this working smoothly, though, is having an intelligent communication means between the main server and the "master" machine - you want to keep the bandwidth as low as possible. In a case like yours, I'd probably come up with a single binary blob that had the current instruction set for the 20 machines, in its smallest form. (Maybe something like 20 bytes, or 40 bytes if you need it, etc). The "master" machine would then worry about translating this out to the other 19 machines and itself.
There are some nice things about this - the server has a much easier time transmitting to one machine in the cluster instead of every machine in the cluster. This let us, for example, have one single, centralized server "drive" multiple clusters efficiently, without having ridiculous hardware requirements anywhere. It also keeps the client code very, very simple. It just has to listen for a UDP datagram and do whatever it says - in your case, it sounds like it would have one of 20 commands, so the client becomes very, very simple.
The "master" server is the trickiest. In our implementation, we actually had the same client code on it as the other 19 (as a separate proces) and one "translation" process that took the blob, broke it into 20 pieces, and transmitted them. It was fairly simple to write, and worked very well.