Develop Reference

Cloud architecture pattern for multiple simple api clients (200-500m api calls/day)? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have to write an api-client system that connects to multiple api-servers, does a job and disconnects. It does two simple things, but needs to do it at scale (ie: aiming for 200-500m outbound API client calls per day):
(1) Simple client connects to an API-server (http/rest), sends a query, receives a response (text based), saves the response for later, and moves on to the next server/query.
Once responses start coming in, a separate process will:
(2) parse the text in the responses and add them to a large file/queue for reporting
I currently have a test system in C#, running 20 console applications on a machine, with 20 threaded clients in each console application carrying out the work. I need to be able to scale this up on demand. What is the best approach to do this? ... I am sure a solid pattern exists to this simple problem?
My thoughts so far are:
-> design a management system that depending on the volume of API-servers to be queried in a given hour, orchestrates the provisioning of virtual machines (not trying to redesign the wheel - will hook into any existing framework like chef/puppet etc if suitable)
-> have a central system for collection of data from the api-clients (perhaps a node instance passing the data off to RabbitMQ for later pickup/processing)
-> have a separate management system that orchestrates the text parsing of data received from the API clients.
-> As project is network latency bound, I believe development language is not really relevant so long as it has good network support.
My main questions then are around:
(1) What would be a most appropriate language/framework to implement this in to enable a lean/cost-effective system? ... ie: no point in spinning up multiple Windows VMs for example if they have a bigger footprint/overhead/cost than doing the same thing in linux? (so in this case I could use the mono framework - get the benefit of C# that my team knows, but the lower cost of linux VMs...)
(2) Is my thinking about having to spin multiple VMs up to do this correct (albeit small VMs running X client applications each)?
(3) Another approach I thought of is to write the clients in Javascript - the reason being that the bottleneck for the api-client is network and api-server response time, not client-side, so it might be well suited to async work? .... in this case I could have one Node server running 100x more api-clients than I could ever get in even a bunch of micro-windows VMs ?
(4) Finally, am I reinventing the wheel? ... is there anything out there on Amazon or Azure already that I can plug into that would provide a ready framework for what I need?
All comments and suggestions and guidance most welcome.
Many thanks.

I am not a specialist in what Amazon provides. Here is what you can use on Azure depending on your needs:
Worker role - this is pretty much a scalable virtual machine. You can scale out or autoscale by condition.
AppFabric and Microservices - for more complex deployment and more granulated development infrastructures.
Azure Functions - an interesting scalable and cost effective processing option. Check it out.
In terms of choosing the language, I would use Node.js if your application is not too complex and it's not going to in the near future. C# is better for more solid systems with complex architecture. Both platforms are supported on Azure.
Have a central system for collection of data from the api-clients
(perhaps a node instance passing the data off to RabbitMQ for later
pickup/processing)
If you need a really big throughput, RabbitMQ may not be enough. On Azure you can use EventHub. More info here.

"Finally, am I reinventing the wheel?" Its a good question - you might be. From your description, you have a lot of proprietary management of servers going on - and a lot of VMs. Depending on your workload, you may not need need manage any traditional VMs at all. Avoid that if you can to keep things lean. There are some great technologies that make server management (patching, security, server administration, etc) a thing of the past for many work loads: event-driven computing frameworks such as AWS Lambda.
Consider a server-less implementation using the API gateway pattern, and microservice architecure pattern, using the following AWS services:
AWS Lambda is a compute service where you can upload your code to AWS Lambda and the service can run the code on your behalf using AWS infrastructure. After you upload your code and create what we call a Lambda function, AWS Lambda takes care of provisioning and managing the servers that you use to run the code. Very light weight. The first 1 million requests per month are free
"Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale." $3.50 per million calls. Scaling, security and management all built in. Lambda supports the specification of HTTP endpoints via the API Gateway to trigger Lambda functions.
AWS Lambda provides an easy way to build back ends without managing
servers. API Gateway and Lambda together can be powerful to create and
deploy serverless Web applications. In this walkthrough, you learn how
to create Lambda functions and build an API Gateway API to enable a
Web client to call the Lambda functions synchronously.
You can also integrate DataPipeline for data transformation, and Simple Queueing Service for queuing/messaging, if needed you your workloads.

If you're doing anything stateful and at scale Service Fabric might be the better choice over Azure Functions/Lambda or Worker Roles.
https://azure.microsoft.com/en-us/services/service-fabric/

Does Windows Azure AppFabric Caching support High Availability?

Back when AppFabric Caching was "Velocity", High Availability was "out of the box" option, and one of the few major features that made it stand out over other caching systems, namely memcached. However along the way "someone" decided it best to charge customers extra for HA by making it only available to Windows Server 2008 Enterprise Edition (or higher)... I am hopeful that Windows Azure AppFabric Caching does provide HA. I'm betting it must, if for no other reason because there is no "upgrade" option. Anyone know for certain?
As a secondary question, if anyone knows of the ETA for the production release of Windows Azure AppFabric Caching other than "2011", much appreciated. Roadmaps would be mighty handy... Instead most of us rely on Google to determine guesswork roadmaps :). Always been a big fan of MS products/services, but they could really use some additional "focus" in the area of Azure.
I understand AppFabric Caching is currently only in "alpha", and is only available via the Azure Labs portal.

Found the answer...
As we move towards commercial launch, we'll look to add many of the features that make Windows Server AppFabric Caching extremely popular, such as High Availability, the ability to emit notifications to clients when they need to refresh their local cache, and more.
http://blogs.msdn.com/b/windowsazureappfabric/archive/2010/10/28/introduction-to-windows-azure-appfabric-caching-ctp.aspx
So no go for the time being, but looks like it is planned. Good enough for me, since it is not yet released anyway. But any ETA on release date (or quarter) would be very helpful for configuring my roadmap...

commercial availability for the current Azure AppFabric labs features is looking to be Q2 of 2011. I'm basing this on the various PDC10 sessions. Only time will tell if we see HA included in that feature set or not.

Web Services for Remote Portlets in C# / .NET -- Options?

I recently had my mind expanded by a new concept: Web Services for Remote Portlets, or WSRP. I learned of it during a presentation on a Java-based web portal we are considering purchasing at work; we are a .NET shop and WSRP would be the means by which we would extend this portal.
Although I cannot control the end decision as to whether or not we purchase the product, I can provide input as to how difficult it would be to build WSRP-compliant portlets. Unfortunately, my recent queries into the subject have turned up almost nill.
So I ask you, the SO community, the following: what libraries or frameworks are out there for building WSRP-compliant portlets in C#/.NET? What are some of the pros and cons of using WSRP in general?
Because there is no correct answer here, I will make this a community wiki post.
So far, I have only found the following:
WSRP Toolkit for Sharepoint by Microsoft (but requiring Sharepoint).
WSRP Portal and WSRP .NET Framework By NetUnity.
Given that WSRP is on top of SOAP, this seems like a perfect candidate for a WCF binding and channel, and yet I see nothing on the subject, anywhere.

WSRP is very contrarian. By now the world has seen that tight coupling between the data model and the presentation model is suboptimal. The success of RSS, REST, MVC, and web services in general shows this. Despite the WS in the name, WSRP stands against the core principles of Web services. The WSRP spec ignores the sound advice to keep data and presentation separate, and couples them tightly.
WSRP promises integration, at the UI level. This seems like the wrong problem to be solving.
It baffles me that this thing has lived as long as it has.
The problem it attempts to solve is often not the problem that should be solved.

If you read the WSRP spec carefully, you'll find it is a remote version of the Java Portlet Specification (if I'm spelling that right). That means that it's useful for integrating Java Portlets. Anything else will have to look like a Java Portlet, which is not very generic.

I think its popularity / adoption can be inferred by the fact that the last release from NetUnit was "This latest release adds support for Visual Studio 2005 and .NET 2.0."

I would have to agree with Cheeso. Integrating the UI with the Data only serves the portlet consumers and adds a big, unnecessary, risky layer to portlet producers. Our .NET shop has been recently forced to consider WSRP and I have found a lack of support and experience. The best MS-centric approach I have seen discussed is here. But I have not found any specific WCF implementation/support. Any leads greatly appreciated!

WSRP is essentially a portal-to-portlet web service standard. What is the primary data exchanged between portal and portlet? It's markup and largely because most portals use a web UI. This whole idea that it is not pure data versus UI is moot point. It's meant to be a web service for portlet discovery, meta data, markup, interactions, caching, portlet-to-portlet communication, etc. That's what a portal does even if not WSRP. WSRP however is an open, cross platform standard.
What is a portal that only integrates portlets from its own products and/or platform? Got Java-based PeopleSoft HR and would like to provide access to their portlets from SharePoint to your employees? Good luck. Why can't this be an achievable scenario for most enterprise software? And yes, I realize it is integration related to UI. That is one of the primary reason for why I'm using a portal. It's not like I'm expecting to get PeopleSoft integrated with SharePoint at the "pure" data level and somehow an Employee Benefits Web Part magically pops up in SharePoint ready for use. However, that is what I expect if the portlet-to-portlet integration is based on WSRP.
WSRP, although not perfect, is a superior solution in my opinion. Besides easy integration of portlet within a portal, it separates the portal from the application. No deploying of binaries to the portal server or even running on the same server. This makes sense. Never run applications on the same server as the portal server: neither will ever be upgraded. I've come to the conclusion that it's insane to put application binaries on the same server as the portal server. "Please deploy this application to the portal server and have it affect security, stability, performance, and everything in between and I would like create as many dependencies as possible and bring down the whole portal sever whenever I upgrade the application". It's a dependency nightmare. Better get a couple of portal vendor consultants to hold hands with when upgrading and to have someone to blame.
Do you need to load balance an entire portal platform when only a select number of portlets are hit the most? Portal vendors would like you to think so. A lot of the time, the portal is doing nothing more than waiting on portlets to finish processing. With WSRP, you have the flexibility to load balance portlets independently of the portal platform. It always breaks down to a few portlets that are hit the most. Why not load balance just those portlets? So instead of unnecessarily load balancing the portal on 80 CPU, you could load balance those few portlets on 10 CPUs. WSRP is also absolutely perfect for cloud computing.
WSRP is a portal-to-portlet standard. If you want to write a portlet that works in multiple portals and potentially across platforms, WSRP is it. If you are remotely contemplating integrating third party portlets, WSRP is it. It's the only standard. However, it also has some significant benefits over other proprietary local portal-to-portlet interfaces and should be considered for those benefits as well.

Choosing a Platform C#/MsSql or Php/Mysql or JSP or what?

It seems like there are an inordinate number of really intelligent people here, so I thought it was the best place to ask a couple involved questions (shameless flattery). Your feedback on any item would be so appreciated.
I am about to develop a very large web based operations, inventory, sales management system. I had intended on C#/MsSql/Reporting, but am now leaning towards Php/MySql.
Multiple facility support. I
would love to have it all run from a
single place, but I need for each
facility to be able to run
autonomously from each other in case
of internet connectivity loss.
Servers could reside in each
facility and replicate to a central
one. MS Replication will hang the
DB for large databases. MS
recommendation to restructure the
database into smaller
segments…thanks. MySql seems to
have very solid replicability
features. Thoughts on this issue?
Distribution: Eventually this could be marketable
to other companies in the industry.
Using the method above I would have
to distribute the system to my
server in the customer’s facility.
With C# I can distribute compiled
code. With PHP any intellectual
property is vulnerable.
Reporting. Will I be able to print labels and
really specifically configured
reports with PHP? Pdf export is a must. MsSql has the
reporting service, but it has issues
with PDF font embedment, the
flexibility is very time consuming
and I am thinking that building it
all by hand might be faster. MS
print control also seems problematic.
Cost. Load would probably not ever exceed 100
simultaneous hits. Full Ms server
licensing is cost prohibitive.
Would sql express support this load?
MySql cost structure is very
appealing.
Thanks in advance for your insight. Sorry for the book.

As with most software, the biggest cost factor is almost certain to be developer time.
That in mind, you should go with whatever your team can get up and running in the fastest. Both of the stacks you are suggesting can get the job done.
(although I'm sure someone will be along in a minute to slate PHP and suggest you use C#/the MS Stack)
To answer the other points from a PHP perspective, which is what I know best:
Replication in mysql will do what you want, but make sure you set up something to let you know if there are problems.
You can use Zend_Guard to compile PHP, it costs though. PHP 5.3 is due for release soon and includes Phar archives - a shameless copy of java's jar archives so that you can distribute an app as a single file.
PHP can do PDF via fPDF. Never used the MS options, so not sure how it compares here.
I think sql server express could do that load, and mysql could too.

Adding point 2 with point 4 if you're intending to sell, you should think that, although in your company it won't exceed 100 simultaneous hits, in the companies your going to sell to it can exceed 10000 hits. So you should think about scalability now
As it is an application with massive size for what you say, you should expect in the future to have someone wanting to communicate with it, excluding php, for jsp or c# so you can make web services if needed.
PHP won't also allow so much distribution as C# or JSP.
3 - Reporting You can do that with any of those languages.
I think in the end is pretty much up to what you know to handle better. I would go for a language i already know instead of hitting a new one. That way you'll minimize deployment time, which is probably very important also.
For database i think you can get mysql or posgresql as they are both free, and will lower your costs.
Hope it helps :)

"I am about to develop a very large web based operations, inventory, sales management system." - this is a very ambitious undertaking. Are you doing this for fun, education, or profit? Do you intend to sell or open source this?
You realize, of course, that there are a lot of alternatives already in the marketplace. What advantage will yours have over the others?
Sounds like a classic buy versus build choice. Have you considered the buy before building?

It sounds like you prefer C# and MySQL. MySQL has several articles on using .NET with MySQL and the Connector/NET driver.
Unfortunately, I know very little about reporting with MySQL.

What to use for Messaging with C#

So my company stores alot of data in a foxpro database and trying to get around the performance hit of touching it directly I was thinking of messaging anything that can be done asynchronously for a snappier user experience. I started looking at ActiveMQ but don't know how well C# will hook with it. Wanting to hear what all of you guys think.
edit : It is going to be a web application. Anything touching this foxpro is kinda slow (probably because the person who set it up 10 years ago messed it all to hell, some of the table files are incredibly large). We replicate the foxpro to sql nightly and most of our data reads are ok being a day old so we are focusing on the writes. plus the write affects a critical part of the user experience (purchasing), we store it in sql and then just message to have it put into foxpro when it can. I wish we could just get rid of the foxpro, unfortunately the company doesn't want to get rid of a very old piece of software they bought that depends on it.

ActiveMQ works well with C# using the Spring.NET integrations and NMS. A post with some links to get you started in that direction is here. Also consider using MSMQ (The System.Messaging namespace) or a .NET based asynchronous messaging solution, with some options here.

MSMQ (Microsoft Message Queueing) may be a great choice. It is part of the OS and present as an optional component (can be installed via Add/Remove Programs / Windows Components), meaning it's free (as long you already paid for Windows, of course). MSMQ provides Win32/COM and System.Messaging APIs. More modern Windows Communication Foundation (aka Indigo) queued channels also use MSMQ.
Note that MSMQ is not supported on Home SKUs of Windows (XP Home and Vista Home)

Its worth mentioning that the ActiveMQ open source project defines a C# API for messaging called NMS which allows you to develop against a single C# / .Net API that can then use various messaging back ends such as
ActiveMQ
MSMQ
TibCo's EMS
any STOMP provider
any JMS provider via StompConnect

You may want to look at MSMQ. It can be used by .NET and VFP, but you'll need to rewrite to use them. Here's an article that tells you how to use MSMQ from VFP. https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/ms917361(v=msdn.10)

Sorry if this isn't what you are asking for...
Have you considered some sort of cache behind the scenes that acts a bit like the "bucket system" when using asynchronous sockets in c/c++ using winsock? Basicly, it works by accepting requests, and sends an immediate response back to the web app, and when it finally gets around to finding your record, it updates it on the app via AJAX or any other technology of your choice. Since I'm not a C# programmer I can't provide any specific example. Hope this helps!

Does the Fox app use .CDX indexes? If so, you might be able to improve performance by adding indexes without needing to change any program code. If it uses .IDX indexes, though, the change would have to be done in the actual app.

As the problem is with writes, I would look more towards >removing< any unneeded indexes on the tables. As is common in RDBMS, every index on a FoxPro table slows down a write operation as the indexes need to be updated, and as you aren't reading directly from (or presumably directly querying) the table you shouldn't need very many indexes. You might also want to look at any triggers or field rules on the tables as they may be slowing down the write operation. Be sure your referential integrity is still preserved, though..