I have a c# winapp that calls a java app by tcp and sends everything back by tcp to the c#.
The reason why I'm using java is because I received a java api. So I wrote a small java app that calls that api and get all the data I need.
So I was wondering if there is another solution for this. Because it's going slow, especially with a lot of data.
Thanks
TCP over the local machine should be pretty fast (named pipes might be a bit faster, but may be harder to do at both ends).
The biggest bottleneck is likely to be serialization and deserialization of the data. What format are you currently using to represent the data?
Sounds to me like you need to profile this. Are you sure it's the network aspect that is slow ? Or the serialisation/deserialisation, or the actual client/server processing beyond the data transmission.
Before you address profiling solutions, you should identify the particular problem point.
You received a java api and you wrote a small java api ... so you know java. Why not write the rest of the application in java as well?
Sometimes is easier to rewrite the "uncompatible" parts dependeing on how much work it is.
I'd recommend trying to use two good Web Service stacks. It would be very interesting to hear your results. It sounds counter-intuitive but these stacks are optimized even though they have the burden of converting to text. Also, it sholdn't be much work to take the stacks in. At least its easy in C# to parallelize calling web services if that's appropriate for your situation.
There are also commercial solutions that allow invoking Java from .NET (and vice-versa). I used JNBridge a few years ago, and it's great.
You can download a 30-day free trial at http://www.jnbridge.com/bin/downloads.php?pr=1&id=0 (no, I don't work for them).
I did some profiling in the java and the method of the api I received are taking the most of the time I noticed.
If I receive around the 1200 records I have a waiting time of 5 minutes. Once I receive it in c#, it only needs a few seconds to add it to objects and show me a view of the results.
So I think the problem isn't really at my end or can it be better if I used something like the JNBridge or webservice?
Related
I have a desktop application that's essentially a glorified Monte Carlo simulation. The simulation is written in C++ (<10,000 lines of code) and the GUI, which calls the .exe is written in python (don't ask, I had (bad) reasons). I'm looking to convert this to a web application on Azure, which I know is going to be a less than trivial task, but I'm hoping to get advice on how to best do this. From how I see this, I have three options, modifying the C++ code and somehow converting this to a web app, rewriting entirely in the .NET framework using C#, or rewriting entirely with Django/Python. Regardless of what I do, I'll have a ton to learn, and I'll need to completely rewrite the front end.
Modifying the C++ code seemed like the best option at first. However, after doing some research, I'm thinking my modifications will be heavy, and C++ really isn't an ideal fit for web apps to begin with. I'm worried that between the likely heavy modifications and the finagling I'll have to do with C++, that will greatly outweigh me getting to reuse a good amount of code.
The next option would be to to completely rewrite the program using .NET and C#. I don't know C#, but I've read where it's not a tough transition to learn C# after knowing C++. I also feel like "translating" certain chunks of C++ code will be manageable, as the languages are so similar.
The final option I see is Python and Django. Obviously, knowing Python should help, but I feel like I'll have to do even more rewriting, as it's less similar to C++. I'm also worried about performance, as I'm assuming python will be slower to complete the simulation than C#.
Obviously this is going to be a pretty large undertaking for myself regardless of the option I choose, but minimizing the difficulty is certainly a top priority here. In addition, performance is a huge consideration, as the user is under a time crunch, and the more iterations the simulation can run, the better the result.
For longer term considerations, I would like this to be something that can be worked on in the future. Additionally, I do hope to continue writing web apps after I complete this first project. These two considerations make me even more focused on doing this "right" and not just quick.
I'm an amateur programmer working on a pet project of mine and I would like some pointers on how to make a C# server application. Here's the general idea:
A client connects to the server application, which in turn fetches the necessary information from a mysql database and sends it back to the client to be displayed and wait for the next action.
I got the idea of making something like this after seeing a somewhat old IBM AS400 mainframe running a warehouse management system, and I though: "Hey, I could try developing a small version of this with a nice UI that doesn't look like it stepped out of a time machine!"
I searched around and used the tcplistener class to communicate between the server and client and managed to send some calls and responses using one thread per client. However I've read that this is not scalable for a large number of clients...
Am I looking at this problem the wrong way and I should try something else? Any input will be appreciated
You don't need to deal with TCP directly for this - WCF (Windows Communication Foundation) was written to abstract all the low level stuff from you.
Check this link out for a good example of how to create a client/server application, it has an entry level explanation, some code and a downloadable project...
http://www.codeproject.com/Articles/16765/WCF-Windows-Communication-Foundation-Example
You can find plenty of information about WCF elsewhere on the internet and here on SO.
It is a very large topic, but the situation you have described is pretty simple - so I doubt you will have problems following the example.
I'm Currenty writing a server architecture and came across this as well: faily easy to solve.
You're right, using 1 thread per client is not effiecient and is a huge waste or resources! The way around that is Thread Pooling. There are loads of different ways to do this but the way I chose to implement it on my server was to add every connection to a queue and then have x number of threads (which you can easily increase or decrease to handle demand) simply dequeue a connection, process it, then enqueue it again.
Of course using WCF will make life easier for you and will speed things up drastically but where's the challange in that!
I've got a for loop I want to parallelize with something like PLINQ's Parallel.ForEach().
The key here is that the C++ library i'm calling to do the computation is decidedly not thread safe, therefore, any plans to parallelize this need to do so across multiple processes.
I was thinking about using WCF to create a "distributor" process to which the "client" and multiple "calculators" could both connect and add/remove items to/from a queue and then the "calculator" sends the results directly back to the client which could update the gui as it receives them. This architecture would allow me to bring as many "calculators" online as I have processors and as I see it even bring them up across multiple computers creating a potential farm of processing power to which all the clients could share.
I'm just wondering if anyone has had any experience doing this and if there are existing application blocks or frameworks that I can use to build this for me. PLINQ does it within the process. is there like a DPLINQ (distributed) or something?
Also if that doesn't exist, does anybody want to give an opinion on my proposed architecture? Any obvious pitfalls? Does anyone think it will work!?!?!?
Sounds like you could be looking for Dryad. It's a Microsoft research project right now, but they do have an "academic release" available. My understanding is that they are also in the process of better productizing it (probably some kind of integration with Azure) for RTM sometime near the end of 2011. Mary Jo Foley covers more about this here.
A long time standard for controlling/dispatching distributed work is MPI. I've only ever used it from C++, but implementations from many languages exist. A quick google suggests that MPI.Net could be a good implementation for .Net!
I've been badly let-down and received an application that in certain situations is at least 100 times too slow, which I have to release to our customers very soon (a matter of weeks).
Through some very simple profiling I have discovered that the bottleneck is its use of .NET Remoting to transfer data between a Windows service and the graphical front-end - both running on the same machine.
Microsoft guidelines say "Minimize round trips and avoid chatty interfaces": write
MyComponent.SaveCustomer("bob", "smith");
rather than
MyComponent.Firstname = "bob";
MyComponent.LastName = "smith";
MyComponent.SaveCustomer();
I think this is the root of the problem in our application. Unfortunately calls to MyComponent.* (the profiler shows that 99.999% of the time is spent in such statements) are scattered liberally throughout the source code and I don't see any hope of redesigning the interface in accordance with the guidelines above.
Edit: In fact, most of the time the front-end reads properties from MyComponent rather than writes to it. But I suspect that MyComponent can change at any time in the back-end.
I looked to see if I can read all properties from MyComponent in one go and then cache them locally (ignoring the change-at-any-time issue above), but that would involve altering hundreds of lines of code.
My question is: Are they any 'quick-win' things I can try to improve performance?
I need at least a 100-times speed-up. I am a C/C++/Delphi programmer and am pretty-much unfamiliar with C#/.NET/Remoting other than what I have read up on in the last couple of days. I'm looking for things that can be completed in a few days - a major restructuring of the code is not an option.
Just for starters, I have already confirmed that it is using BinaryFormatter.
(Sorry, this is probably a terrible question along the lines of 'How can I feasibly fix X if I rule out all of the feasible options'… but I'm desperate!)
Edit 2
In response to Richard's comment below: I think my question boils down to:
Is there any setting I can change to reduce the cost of a .NET Remoting round-trip when both ends of the connection are on the same machine?
Is there any setting I can change to reduce the number of round-trips - so that each invocation of a remote object property doesn't result in a separate round-trip? And might this break anything?
Under .Net Remoting you have 3 ways of communicating by HTTP, TCP and IPC. If the commnuicatin is on the same pc I sugest using IPC channels it will speed up your calls.
In short, no there are no quick wins here. Personally I would not make MyComponent (as a DTO) a MarshalByRefObject (which is presumably the problem), as those round trips are going to cripple you. I would keep it as a regular class, and just move a few key methods to pump them around (i.e. have a MarshalByRef manager/repository/etc class).
That should reduce round-trips; if you still have problems then it will probably be bandwidth related; this is easier to fix; for example by changing the serializer. protobuf-net allows you to do this easily by simply implementing ISerializable and forwarding the two methods (one from the interface, plus the ctor) to ProtoBuf.Serializer - it then does all the work for you, and works with remoting. I can provide examples of this if you like.
Actually, protobuf-net may help with CPU usage too, as it is a much more CPU-efficient serializer.
Could you make MyComponent a class that will cache the values and only submit them when SaveCustomer() is called?
You can try compressing traffic. If not 100-times increase, you'll still gain some performance benefit
If you need the latest data (always see the real value), and the cost of getting the data each time dominates the runtime then you need to be radical.
How about changing polling to push. Rather than calling the remote side each time you need a value, have the remote push all changes and cache the latest values locally.
Local lookups (after the initial get) are always up to date with all remoting overhead being done in the background (on another thread). Just be careful about thread safety for non-atomic types.
I'm looking to write a small web service to run on a small Linux box. I prefer to code in C#, so I'm looking to use Mono.
I don't want the overhead of running a full web server or Mono's version of ASP.NET. I'm thinking of having a single process with a thread dealing with each client connection. Shared memory between threads instead of a database.
I've read a little on Microsoft's version of HttpListener and how it works with the Http.sys driver. Alas, Mono's documentation on this class is just the automated class interface with no discussion of how it works under the hood. (Linux doesn't have Http.sys, so I imagine it's implemented substantially differently.)
Could anyone point me towards some resources discussing this module please?
Many thanks, Bill, billpg.com
(A little background to my question for the interested.)
Some time ago, I asked this question, interested in keeping a long conversation open with lots of back-and-forth. I had settled on designing my own ad-hoc protocol, but people I spoke to really wanted a REST interface, even at the cost of the "Okay, send your command now" signal.
So, I wondered about running ASP.NET on a Linux/Mono server, but stumbled upon HttpListener. This seemed ideal, as each "conversation" could run in a separate thread. The thread that calls HttpListener in a loop can look for which thread each incomming connection is for and pass the reference to that thread.
The alternative for an ASP.NET driven service, would be to have the ASPX code pick up the state from a database, and write back the new state when it finishes. Yes, it would work, but that's a lot of overhead.
Greetings,
The HttpListener class in Mono works without much of a problem. I think that the most significant difference between its usage in a MS environment and a Linux environment is that port 80 cannot be bound to without a root/su/sudo security. Other ports do not have this restriction. For instance if you specify the prefix: http://localhost:1234/ the HttpListener works as expected. However if you add the prefix http://localhost/, which you would expect to listen on port 80, it fails silently. If you explicitly attempt to bind to port 80 (http://localhost:80/) then you throw an exception. If you invoke your application as a super user or root, you can explicitly bind to port 80 (http://localhost:80/).
I have not yet explored the rest of the HttpListener members in enough detail to make any useful comments about how well it operates in a linux environment. However, if there is interest, I will continue to post my observations.
chickenSandwich
I am not sure why you want to look so deep into the hood. Even on Microsoft side, the documents about http.sys may not provide you really valuable information if you are using the .NET Framework.
To know if something works on Mono good enough, you are always supposed to download its VMware or VPC image, and test your applications on it.
http://www.go-mono.com/mono-downloads/download.html
Though Mono is much more mature than a few years ago, we cannot say it has been tested by enough real-world applications like Microsoft.NET. So please test out your applications and submit issues you find to Mono team.
Based on my experience, minor issues are fixed within only a few days, while for major issues it takes a longer time. But with Mono source code available, you can fix on your own or find out good workarounds most of the times.