Good afternoon,
I am looking to have a web service that deals other web services (and databases) to collect data that is then stored within its own database. This is all quite straight-forward as I can regularly poll these to gather changes.
now communicating with the clients is the problem here. The web service can be created in any language (although C# or Java are the main candidates). But they will need to push data to clients written in Silverlight and Android (and others to be decided).
Polling is of course possible but on mobile devices we don't want constant polling, and besides in as real time as possible comms is important here
If this was totally in C# then I would imagine I could use duplex channels but I need something that is more general in order to push. So the question is what is available to do this? Also it needs to be free in an ideal world.
Seems a simple thing, a way to push from a web service toa client but I suspect it is far from simple (but maybe I can be proved wrong, I hope!)
It seems that there is no platform independent way to do what I want and so the way forward has been decided as the following for now:
Silverlight clients will use a polling mechanism to get the data and Android clients will use XMPP to get the messages.
This does have disadvantages in the polling aspect however there are no readily available free libraries to do it that work with Silverlight but in the future we will move to using XMPP throughout when we have the cash
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have to write an api-client system that connects to multiple api-servers, does a job and disconnects. It does two simple things, but needs to do it at scale (ie: aiming for 200-500m outbound API client calls per day):
(1) Simple client connects to an API-server (http/rest), sends a query, receives a response (text based), saves the response for later, and moves on to the next server/query.
Once responses start coming in, a separate process will:
(2) parse the text in the responses and add them to a large file/queue for reporting
I currently have a test system in C#, running 20 console applications on a machine, with 20 threaded clients in each console application carrying out the work. I need to be able to scale this up on demand. What is the best approach to do this? ... I am sure a solid pattern exists to this simple problem?
My thoughts so far are:
-> design a management system that depending on the volume of API-servers to be queried in a given hour, orchestrates the provisioning of virtual machines (not trying to redesign the wheel - will hook into any existing framework like chef/puppet etc if suitable)
-> have a central system for collection of data from the api-clients (perhaps a node instance passing the data off to RabbitMQ for later pickup/processing)
-> have a separate management system that orchestrates the text parsing of data received from the API clients.
-> As project is network latency bound, I believe development language is not really relevant so long as it has good network support.
My main questions then are around:
(1) What would be a most appropriate language/framework to implement this in to enable a lean/cost-effective system? ... ie: no point in spinning up multiple Windows VMs for example if they have a bigger footprint/overhead/cost than doing the same thing in linux? (so in this case I could use the mono framework - get the benefit of C# that my team knows, but the lower cost of linux VMs...)
(2) Is my thinking about having to spin multiple VMs up to do this correct (albeit small VMs running X client applications each)?
(3) Another approach I thought of is to write the clients in Javascript - the reason being that the bottleneck for the api-client is network and api-server response time, not client-side, so it might be well suited to async work? .... in this case I could have one Node server running 100x more api-clients than I could ever get in even a bunch of micro-windows VMs ?
(4) Finally, am I reinventing the wheel? ... is there anything out there on Amazon or Azure already that I can plug into that would provide a ready framework for what I need?
All comments and suggestions and guidance most welcome.
Many thanks.
I am not a specialist in what Amazon provides. Here is what you can use on Azure depending on your needs:
Worker role - this is pretty much a scalable virtual machine. You can scale out or autoscale by condition.
AppFabric and Microservices - for more complex deployment and more granulated development infrastructures.
Azure Functions - an interesting scalable and cost effective processing option. Check it out.
In terms of choosing the language, I would use Node.js if your application is not too complex and it's not going to in the near future. C# is better for more solid systems with complex architecture. Both platforms are supported on Azure.
Have a central system for collection of data from the api-clients
(perhaps a node instance passing the data off to RabbitMQ for later
pickup/processing)
If you need a really big throughput, RabbitMQ may not be enough. On Azure you can use EventHub. More info here.
"Finally, am I reinventing the wheel?" Its a good question - you might be. From your description, you have a lot of proprietary management of servers going on - and a lot of VMs. Depending on your workload, you may not need need manage any traditional VMs at all. Avoid that if you can to keep things lean. There are some great technologies that make server management (patching, security, server administration, etc) a thing of the past for many work loads: event-driven computing frameworks such as AWS Lambda.
Consider a server-less implementation using the API gateway pattern, and microservice architecure pattern, using the following AWS services:
AWS Lambda is a compute service where you can upload your code to AWS Lambda and the service can run the code on your behalf using AWS infrastructure. After you upload your code and create what we call a Lambda function, AWS Lambda takes care of provisioning and managing the servers that you use to run the code. Very light weight. The first 1 million requests per month are free
"Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale." $3.50 per million calls. Scaling, security and management all built in. Lambda supports the specification of HTTP endpoints via the API Gateway to trigger Lambda functions.
AWS Lambda provides an easy way to build back ends without managing
servers. API Gateway and Lambda together can be powerful to create and
deploy serverless Web applications. In this walkthrough, you learn how
to create Lambda functions and build an API Gateway API to enable a
Web client to call the Lambda functions synchronously.
You can also integrate DataPipeline for data transformation, and Simple Queueing Service for queuing/messaging, if needed you your workloads.
If you're doing anything stateful and at scale Service Fabric might be the better choice over Azure Functions/Lambda or Worker Roles.
https://azure.microsoft.com/en-us/services/service-fabric/
Good afternoon,
I'm trying to find a way to synchronize an ObservableCollection between applications. Basically, one would be the server that has a collection of strings, and the other would be the client which would have that same list and display it in a GUI. When a string is modified on the server, I want the modification to be reflected on the client.
I'm not very familiar with server-client architectures or frameworks and would like to know where to start. So far I have looked at the WCF architecture and it looks like it might be a bit overkill for what I'm trying to accomplish, but I could be wrong.
Any guidance will be greatly appreciated.
WCF is the standard way of performing inter-process and inter-machine communications for .NET applications and it works very well. You need your client application to connect to the server and get the current state of the collection. Then each time the collection changes the client needs to be notified so it can apply the same change locally. There are many resource on the Web and at MSDN describing how to create a WCF connection.
I'm writing a web server app for the first time, and I'm not really sure that I know what I'm doing.
Basically I have some server side C# code and a native iOS app. I need to be able to push updates from the server to the app. The method which we have decided to use is Long Polling, and I can see three ways of doing this:
1) Writing my own web server in C# - not neccesarily tempting as it requires re-inventing the wheel
2) Using WCF - I've seen a few articles about how to implement long polling over WCF, but the tutorials that I've seen all seem to use clients which are implemented in .NET WCF which is not applicable for me as I need to use an iOS app.
3) Something else, possibly using IIS - I don't really know where to begin with this option.
Can anyone recommend a good tutorial, or exemplar project which uses standard HTTP to implement long polling with a C# server? So long as it's using standard HTTP, I'm confident with the iOS side of things.
Obviously if there's an even neater way of doing things then I'm all ears as well.
I would highly recommend that you investigate SignalR which allows you to achieve exactly what you are after. There are many iOS tutorials as well as HTML / JavaScript and of course C#.
One of the benefits of SignalR is that it tries to use the best technology available on the various devices and down-grades until it works. So, will start with Web Sockets for example and fail down to long-polling if nothing better is available.
What "technology" would you suggest to exchange some kind of messages between a Java server and several clients written in C#, Javascript and Java?
The background story:
In our current project we're trying to build a generic UI backend in Java (running on the server) which is then "bridged" into several UI frontends by means of different UI adaptors (running on the client, the server, or both). While our server technology will always be Java, there will be C# (Silverlight), JavaScript and Java clients. Maybe even more in the future (different Smartphones, Tablets).
The UI backend and UI frontends communicate through a bunch of more or less simple messages (mostly name/value-pairs) each of which encapsulates a specific property/state/data change on the client or server respectively. Within a single request cycle, several such simple messages are aggregated into one big message which is then passed from backend to frontend or vice versa. At the moment sending and receiving messages is done at a single entry point on the client as well as on the server. So there are no server methods exposed as WebService etc. - simply because this would most definitely be to slow in our case.
Our current prototype consists solely of a Java server, a Java Desktop Client (Swing) and a Java Web Client (Vaadin). The message exchanged between backend and frontend is effectively a list of POJOs (each representing a specific "change") serialized/deserialized to/from XML. So far, so good.
Now C# and Javascript come to the table. Since we want to work with some kind of object in each technology, we thought it would be a good idea to specify the messages/changes/pojos in some kind of abstract language and then generate objects for each target language. At some point these objects could then be serialized/deserialized and sent over the wire (probably via http/s). For this purpose we thought of Google's protocol buffers or Thrift. What do you think?
For the moment our synchronous request-response-cycle is enough but we will need asynchronous request-response or server-push respectively pretty soon. That's why we thought of using something like ActiveMQ straight away. What do you think? Too much? If not, how can we accomplish the object generation mentioned above (xsd, jaxb, ? for js)? Are there better ways? I've never used ActiveMQ but according to the website it should be possible with Java, C# (Spring.NET) and somehow with Javascript (STOMP) too. However, this seems pretty complex to me...
Any tips, hints, experiences or comments about this or related topics would be really helpful.
Thank you in advance.
I would recommend using webservices. The WSDL language defines objects and messages of your protocol in abstract form. Most modern languages like Java and C# have tools for converting WSDL to native types and libraries for handling I/O.
For the last two years i have been involved in building a similar system: backend for our project is c#, java and bunch of other languages, frontend is phone clients for ios, android, symbian, all the common webbrowsers and even windows desktop apps.
For all these services we use JSON, since it seems to be most widely supported format across all languages and platforms, it decodes fairly fast on the clients compared to xml based solution (esp. webbrowsers/javascript) and it has a pretty low overhead that compresses extremely well which is great for clients that lack bandwidth.
So my company stores alot of data in a foxpro database and trying to get around the performance hit of touching it directly I was thinking of messaging anything that can be done asynchronously for a snappier user experience. I started looking at ActiveMQ but don't know how well C# will hook with it. Wanting to hear what all of you guys think.
edit : It is going to be a web application. Anything touching this foxpro is kinda slow (probably because the person who set it up 10 years ago messed it all to hell, some of the table files are incredibly large). We replicate the foxpro to sql nightly and most of our data reads are ok being a day old so we are focusing on the writes. plus the write affects a critical part of the user experience (purchasing), we store it in sql and then just message to have it put into foxpro when it can. I wish we could just get rid of the foxpro, unfortunately the company doesn't want to get rid of a very old piece of software they bought that depends on it.
ActiveMQ works well with C# using the Spring.NET integrations and NMS. A post with some links to get you started in that direction is here. Also consider using MSMQ (The System.Messaging namespace) or a .NET based asynchronous messaging solution, with some options here.
MSMQ (Microsoft Message Queueing) may be a great choice. It is part of the OS and present as an optional component (can be installed via Add/Remove Programs / Windows Components), meaning it's free (as long you already paid for Windows, of course). MSMQ provides Win32/COM and System.Messaging APIs. More modern Windows Communication Foundation (aka Indigo) queued channels also use MSMQ.
Note that MSMQ is not supported on Home SKUs of Windows (XP Home and Vista Home)
Its worth mentioning that the ActiveMQ open source project defines a C# API for messaging called NMS which allows you to develop against a single C# / .Net API that can then use various messaging back ends such as
ActiveMQ
MSMQ
TibCo's EMS
any STOMP provider
any JMS provider via StompConnect
You may want to look at MSMQ. It can be used by .NET and VFP, but you'll need to rewrite to use them. Here's an article that tells you how to use MSMQ from VFP. https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/ms917361(v=msdn.10)
Sorry if this isn't what you are asking for...
Have you considered some sort of cache behind the scenes that acts a bit like the "bucket system" when using asynchronous sockets in c/c++ using winsock? Basicly, it works by accepting requests, and sends an immediate response back to the web app, and when it finally gets around to finding your record, it updates it on the app via AJAX or any other technology of your choice. Since I'm not a C# programmer I can't provide any specific example. Hope this helps!
Does the Fox app use .CDX indexes? If so, you might be able to improve performance by adding indexes without needing to change any program code. If it uses .IDX indexes, though, the change would have to be done in the actual app.
As the problem is with writes, I would look more towards >removing< any unneeded indexes on the tables. As is common in RDBMS, every index on a FoxPro table slows down a write operation as the indexes need to be updated, and as you aren't reading directly from (or presumably directly querying) the table you shouldn't need very many indexes. You might also want to look at any triggers or field rules on the tables as they may be slowing down the write operation. Be sure your referential integrity is still preserved, though..