Develop Reference

Alternative to start thread inside of an mvc .net core application?

There is currently multiple instruments inside of our manufacture plant that are inserting data into multiples tables inside of a database at different speeds.
There is a computer on each production line connected to a web page where the operator enter the assigned job number and some related information is displayed.
Our goal is to display indications base on the data inserted by the plant devices. Status are related to raw material availability, warehouse storage availability, temperature range, etc.
My initial idea was to modify the current MVC application by spawning a thread per production line that is scanning the inserted information each 10 seconds and may push data trough signarlR to advice operators. I read that starting threads inside of an MVC application is a bad practice that may disturb how IIS manage threads.
I was wondering how to host fast-recurrent-independent processes in MVC if it is not by using a separate thread?
Thank you for your time!

Yes, starting polling threads might not be the best approach here. An alternative solution that I might suggest is to modify your so called instruments (that are currently inserting data) to be SignalR clients and broadcast a message to the server each time they insert some data. The SignalR server could then simply broadcast this message to the javascript SignalR clients connected to it. This way you could achieve direct communication (through the SignalR server) between the instruments that are producing data and the browser clients that can display this data in real time.

How can I handle multiple queries with threads in c#

I've a problem with a multithreading app in C# and I would appreciate some help, since I'm new to multithreading.
This is the scenario: I'll have a mobile app that will do a lot of queries/requests in my database(Mysql), my goal is to make a server side application that handles multiple queries using C# in a Linux machine(mono to the rescue). My company is doing the database side of the application, there's another company making the mobile app. I'll send the data to the cloud and the cloud server will send it to the client.
I'm reading the threading chapters of CLR via C# and C# 4.0 in a nutshell, but until now I have only a little clue of what I can do, I believe that asynchronous methods would work, since it doesn't use a lot of resources but I'm a little confused about how to handle thread concurrency(priority, state).
So here are my questions:
What is the best way to solve this problem?
Which class from .NET framework suits best for this job?
How should I handle the query queue?
How can I handle thousands of threads/queries fast enough, so my mobile app user can have the query result in a estimated time of 5 minutes?
Some observations:
I know that the data and time to finish a query will be exponentially equal to the size of the user's data in my database, but I need to handle(few and large data) it as fast as I can.
I'm sending the data to a cloud database(Amazon EC2) and from there i'll send it to the client. I'll not handle this, this will be handled by another company, so my job is to get the queries done quickly and make them avaliable to the cloud database.
I'm aware that to send the information to my client I depend on my IT infrastructure, but the point here is: how I can solve this problem quickly in a way that I'll have only to worry about my application infrastructure.
I cannot put the queries on a big string and throw it on the database, because I need to handle each query result separately before sending the result to the user.
The storage engine is MyISAM, so no transactions are allowed.

I would create a REST web service, either on top servicestack or WebAPI, to abstract access to your data via a service. Either of these services would be able to handle simultaneous requests from your mobile client, as they are designed to do so. In addition, I would create a class that can mediate access and provide a unit-of-work to your database (ie repository). The connection provider for MySQL should be able to handle simultaneous requests from your web service, so you should not have to worry about threading and request management. If a single instance is not enough, you can add more web servers running the same code and use a load-balancer to distribute the requests to each of your instances, where the service/data code is the same.
Some resources for mono based web-api/servicestack:
http://www.piotrwalat.net/running-asp-net-web-api-services-under-linux-and-os-x/
What is the best way to run ServiceStack on Linux / Mono?

Best transport mechanism for transferring large amounts of data between ASP.NET MVC websites

I have a scenario that requires me to export a large mailing list (> 1m customers) to an external email system. I have source code and control over both applications.
I need a mechanism for transferring the data from one system to another that is:
Robust
Fast
Secure
So far, I have set up a standard MVC controller method that responds to a request (over https), performs some custom security checks, and then pulls the data from the DB.
As data is retrieved from the DB, the controller method iterates over the results, and writes response in a plain text format, flushing the response every 100 records or so. The receiver reads each row of the response and performs storeage and processing logic.
I have chosen this method because it does not require persisting user data to a permanent file, and a client built in any language will be able to implement receiver logic without a dependency on any proprietary technology (e.g. WCF).
I am aware of other transport mechanisms that I can use with .NET, but none with an overall advantage, given the requirements listed above.
Any insight into which technologies might be better than my request / response solution?

Two suggestions come to mind, we had something similar to this happen at our company a little while ago (acquired website with over 1 million monthly active users and associated data needed a complete datacenter change, including 180gb db that was still active).
We ended up setting up a pull replication to it over SSH (SQL Server 2005), this is black magic at best and took us about a month to set up properly between research and failed configurations. There are various blog posts about it, but the key parts are:
1) set up a named server alias in SQL Server configuration manager on the subscriber db machine, specifying localhost:1234 (choose a better number).
2) set up putty to make a ssh tunnel between your subscriber's localhost:1234 from step 1 and publish db's port 9876 (again, choose a better number). Also make sure you have ssh server enabled on the publisher. Also keep the port a secret and figure out a secure password for the ssh permissions.
3) add a server alias on publisher for port 9876 for the replicated db.
4) if your data set is small enough, create the publications and try starting up the subscriber using snapshot initialize. If not, you need to create a publication with "initialize from backup" enabled, and restore a partial backup at the subscriber using ftp to transfer the backup file over. This method is much faster than snapshot initialization for larger datasets.
Pros: You don't need to worry about authentication for the sql server, "just" the ssh tunnel. Publication can be easily modified in case you realize you need more columns or schema changes. You save time writing an api that may be only temporary and might have more security issues.
Cons: It's weird, there's not much official documentation and ssh on windows is finicky. If you have a linux based load balancer, it may be easier. There are a lot of steps.
Second suggestion: use ServiceStack and protobuf.NET to create a very quick webservice and expose it over https. If you know how to use ServiceStack, it should be very quick. If you don't, it would take a little time because it operates on a different design philosophy from Web API and WCF. Protobuf.NET is the most compact and fastest serialization/deserialization wire format widely available currently. Links:
ServiceStack.NET
Protobuf.NET
Pros: You can handle security however you like. This is also a downside since you then have to worry about it. It's much better documented. You get to use or learn a great framework that will speed up the rest of your webservice-related projects for the rest of your life (or until something better comes along).
Cons: You have to write it. You have to test it. You have to support it.

How to effectively communicate between database bound applications?

We have a number of different old school client-server C# WinForm client-side apps that are essentially front-ends for the database. Then there is a C# server-side windows service that waits on the client apps to submit orders and then it processes them.
The way the server-side service finds out whether there is work to do is that it polls the database. Over the years the logic of polling for waiting orders has gotten a lot more complicated due to the myriad of business rules. So because of this, the polling stored proc itself uses quite a bit of SQL Server resources even if there is nothing to do. Add to this the requirement that the orders be processed the moment they are submitted and you got yourself a performance problem, as the database is being polled constantly.
The setup actually works fine right now, but the load is about to go through the roof and, it is obvious, that it won't hold up.
What are some effective ways to communicate between a bunch of different client-side apps and a server-side windows service, that will be more future-proof than the current method?
The database server is SQL Server 2005. I can probably get the powers that be to pony up for latest SQL Server if it really comes to that, but I'd rather not fight that battle.

There are numerous options ways you can notify the clients.
You can use a ready-made solution like NServiceBus, to publish information from the server to the clients or other servers. NServiceBus uses MSMQ to publish one message to multiple subscribers in a very easy and durable way.
You can use MSMQ or another queuing product to publish messages from the server that will be delivered to the clients.
You can host a WCF service on the Windows service and connect to it from each client using a Duplex channel. Each time there is a change the service will notify the appropriate clients or even all of them. This is more complex to code but also much more flexible. You could probably send enough information back to the clients that they wouldn't need to poll the database at all.
You can have the service broadcast a UDP packet to all clients to notify them there are changes they need to pull. You can probably add enough information in the packet to allow the clients to decide whether they need to pull data from the server or not. This is a very lightweight for the server and the network, but it assumes that all clients are in the same LAN.

Perhaps you can leverage SqlDependency to receive notifications only when the data actually changes.

You can use any messaging middleware like MSMQ, JMS or TIBCO to communicate between your client and the service.

By far the easiest, and most likely the cheapest, answer is to simply buy a bigger server.
Barring that, you are in for a development effort that has a high probability of early failure. By failure I don't mean that you end up scraping whatever it is you end up building. Rather, I mean you launch the changes and orders will be screwed up while you are debugging your myriad of business rules.
Quite frankly, I wouldn't consider approaching a communications change under pressure; presuming your statement about load going "through the roof" in the near term.
If your risk exposure is such that it has to be 100% functional day one (which is normal when you are expecting a large increase in orders), with no hiccups then just upsize the DB server. Heck, I wouldn't even install the latest sql server on it. Instead, just buy a larger machine, install the exact same OS and DB server (and patch levels) and move your database.
Then look at your architecture to determine what needs to go away and what can be salvaged.

If everybody connects to SQL Server then there is also the option of Service Broker. Unlike other messaging/queueing solution recommended so far it is entirely contained in your database (no separate product to deploy, administer and configure), it offers a single story vis-a-vis your backup/recovery and high availability needs ( no separate backup for message store, no separate DR/HA, whatever is your DB solution is also your messaging solution) and overs a uniform programming API (SQL).
Even when everything is within one single SQL Server instance (ie. there is no need to communicate over network between multiple SQL Service instances) Service Broker still has an ace that no one can match: activation. With activation you eliminate completely the need to poll because the system itself will launch your processing code (will 'activate') when there are events to process. The processing code can be internal (T-SQL procedure or SQLCLR .Net procedure) or external (see external activator).

How should server push data to rich client

I'm writing a simple accounting program consists of several C# winform clients and a java server app that read/write data into a database. One of the requirement is that all C# clients should receive updates from the server. For example, if user a create a new invoice from his C# client, other users should see this new invoice from their client.
My experience is mainly on web development and I don't know what's the best way to fulfill this requirement with C#s client and Java servlet server.
My initial though is to run ActiveMQ with Glassfish and use messaging pub/sub method so that updates can be pushed to C# client. I will create different topics like newInvoice, cancelInvoice, etc in order to differentiate the message type. Each message will simply contains the object encoded in JSON.
But it seems to me that this involves quite a lot of work. Given that my user base is very small ( just 3 or 4 concurrent user), it seems to me that there should be some simpler solutions. (I'm not familiar socket programming :) )
I know this is a client-server programming 101 questions but would be great if any experienced programmer can point me to some simple solutions.

The simplest approach here is often to simply use a poll - i.e. have the clients query for data every (your time interval). That avoids a whole family of issues (firewalls, security, line-of-sight, resolution, client-tracking, etc).
With WCF, you can have callbacks on duplex channels (allowing the server to actively send a message to clients), but this is more complex. I value simplicity, so I usually just poll.
Tricks that help here are designing the system to have an inbuilt mechanism for querying "changes since x" - for example, an audit table, perhaps fed by database triggers. The exact details vary per project, of course.
Another option that you might want to look at is ADO.NET Sync Services; this does much of what you ask for, for keeping a local copy of the database up to date with the server - but has a few complexities of its own. This is available (IIRC) in the "Local Database Cache" VS template.

Rather than pushing information from the server to 1:N Clients, would it not be easier to have the clients Poll the server for updates every so often ? Or when the client launches and creates a connection to the server, the server could dynamically generate a new Message Queue for that Client Connection, which the client could then poll for updates?

There are several push technologies available to you, like ActiveMQ (as you mentioned), or XMPP. But if you only have 3 or 4 clients to concern yourself with, polling would be the simplest solution. It doesn't scale well, but that isn't really a concern in your case, unless your server is an 8086 or something 8-)

You may want to take a look at StreamHub Push Server - its a popular Comet server written in Java that has a .NET Client SDK for receiving updates from the server in C#. It also has a Java Client SDK and the usual Ajax/Comet web browser support giving you more flexibility in the future to push data to web, Java and C# clients.