Develop Reference

Submit a Spark job from C# and get results

As per title, I would like to request a calculation to a Spark cluster (local/HDInsight in Azure) and get the results back from a C# application.
I acknowledged the existence of Livy which I understand is a REST API application sitting on top of Spark to query it, and I have not found a standard C# API package. Is this the right tool for the job? Is it just missing a well known C# API?
The Spark cluster needs to access Azure Cosmos DB, therefore I need to be able to submit a job including the connector jar library (or its path on the cluster driver) in order for Spark to read data from Cosmos.

As a .NET Spark connector to query data did not seem to exist I wrote one
https://github.com/UnoSD/SparkSharp
It is just a quick implementation, but it does have also a way of querying Cosmos DB using Spark SQL
It's just a C# client for Livy but it should be more than enough.
using (var client = new HdInsightClient("clusterName", "admin", "password"))
using (var session = await client.CreateSessionAsync(config))
{
var sum = await session.ExecuteStatementAsync<int>("val res = 1 + 1\nprintln(res)");
const string sql = "SELECT id, SUM(json.total) AS total FROM cosmos GROUP BY id";
var cosmos = await session.ExecuteCosmosDbSparkSqlQueryAsync<IEnumerable<Result>>
(
"cosmosName",
"cosmosKey",
"cosmosDatabase",
"cosmosCollection",
"cosmosPreferredRegions",
sql
);
}

If your just looking for a way to query your spark cluster using SparkSql then this is a way to do it from C#:
https://github.com/Azure-Samples/hdinsight-dotnet-odbc-spark-sql/blob/master/Program.cs
The console app requires an ODBC driver installed. You can find that here:
https://www.microsoft.com/en-us/download/details.aspx?id=49883
Also the console app has a bug: add this line to the code after the part where the connection string is generated.
Immediately after this line:
connectionString = GetDefaultConnectionString();
Add this line
connectionString = connectionString + "DSN=Sample Microsoft Spark DSN";
If you change the name of the DSN when you install the spark ODBC Driver you will need to change the name in the above line then.
Since you need to access data from Cosmos DB, you could open a Jupyter Notebook on your cluster and ingest data into spark (create a permanent table of your data there) and then use this console app/your c# app to query that data.
If you have a spark job written in scala/python and need to submit it from a C# app then I guess LIVY is the best way to go. I am unsure if Mobius supports that.

Microsoft just released a dataframe based .NET support for Apache Spark via the .NET Foundation OSS. See http://dot.net/spark and http://github.com/dotnet/spark for more details. It is now available in HDInsight per default if you select the correct HDP/Spark version (currently 3.6 and 2.3, soon others as well).

UPDATE:
Long ago I said a clear no to this question.
However times has changed and Microsoft made an effort.
Pleas check out https://dotnet.microsoft.com/apps/data/spark
https://github.com/dotnet/spark
// Create a Spark session
var spark = SparkSession
.Builder()
.AppName("word_count_sample")
.GetOrCreate();
Writing spark applications in C# now is that easy!
OUTDATED:
No, C# is not the tool you should choose if you would like to work with Spark! However if you really want to do the job with it try as mentioned above Mobius
https://github.com/Microsoft/Mobius
Spark has 4 main languages and API-s for them: Scala, Java, Python, R.
If you are looking for a language in production I would not suggest the R API. The Other 3 work well.
For Cosmo DB connection I would suggest: https://github.com/Azure/azure-cosmosdb-spark

Get Biztalk's message count through C# .NET

How can I get total message sent / received at any given point of time by the BizTalk Server through C# .NET?
Any Ideas to achieve this?

Message tracking in BizTalk is done using BAM. Of course, you could write your own database and data access API, but why would you?!? See Using Business Activity Monitoring for a primer. Don't be put off by the learning curve - once you get over it, you'll love it!
This book has a very good section on BAM, don't be put off by the version - the basic concepts still apply to 2010.
And finally this tool will speed up your development.

A simple approach would be to use the existing BizTalk Perfmon Counters to monitor messages (albeit since the last Host instance restart)
Assuming that you've got a BizTalk Server called BizTalkServer with a SendHost and ReceiveHost configured, the below message counters should give an indication of the messages sent / received:
PerformanceCounter msgsReceivedCounter = new
PerformanceCounter("BizTalk:Messaging", "Documents received", "ReceiveHost", "BizTalkServer");
msgsReceivedCounter.ReadOnly = true;
PerformanceCounter msgsSentCounter = new
PerformanceCounter("BizTalk:Messaging", "Documents processed", "SendHost", "BizTalkServer");
msgsSentCounter.ReadOnly = true;
lblSent.Text = string.Format("{0}", msgsSentCounter.NextValue());
lblReceived.Text = string.Format("{0}", msgsReceivedCounter.NextValue());
Note that if you have more than one BizTalk host in your group, you will need to aggregate across all hosts.
More on using the PerformanceCounter class in C# here.
More on the BizTalk PerfMon Counters here.

You can get that information from querying Tracking Database MessageInOutEvents table. This is one single table where all the activities are stored.
The only requirements is you should have switched off the global tracking property in your BizTalk environment.

Jabber-net integration

I'd like to ask your help regarding having a Google Talk Bot that will communicate with my code on my server.
I have downloaded Jabber-Net from code.google.com, but the examples there are not enough... I am new to these technologies, and have no clue about:
How will client arrive to my server? where should I change [if any] DNS to my server?
Which server side library should I use?
From the examples I understood that I need to have a Desktop-app running in the background constantly, which doesn't make sense to me.
Does anyone has an example of some better references to understand this better?
[Sorry for my ignorance...]

I'm not sure if I understand what you ask correctly. If you're asking how to connect to chosen server, console sample shows how to do it simply, you basically fill out JID class.
Sample from Jabber-Net
JabberClient jc = new JabberClient();
JID j = new JID(jid);
jc.User = j.User;
jc.Server = j.Server;
jc.NetworkHost = networkHost;
jc.Port = port;
jc.Resource = "Jabber.Net Console Client";
jc.Password = pass;
jc.AutoStartTLS = TLS;
jc.AutoPresence = initialPresence;
If you want to create your own server, there's a library (also running under .NET) called agsxmpp, it allows to create both, server and client, it's open source on MIT/GPL license afair. I don't know if jabber-net enables this feature. On the other hand, there are plenty of free jabber-server if you don't want to just use one of "public" ones, so it may be worth to consider just using something that is ready to be launched.
There's a console sample in the project, you don't need desktop-app (if this is what you were asking?), so you can write service, console app or anything else.

Here's a recent post that shows an example of replying to incoming messages on Gtalk using .NET

Local database, I need some examples

I'm making an app that will be installed and run on multiple computers, my target is to make an empty local database file that is installed with the app and when user uses the app his database to be filled with the data from the app .
can you provide me with the following examples :
what do I need to do so my app can connect to its local database
how to execute a query with variables from the app for example how would you add to the database the following thing
String abc = "ABC";
String BBB = "Something longer than abc";
and etc
Edit ::
I am using a "local database" created from " add > new item > Local database" so how would i connect to that ? Sorry for the dumb question .. i have never used databases in .net

Depending on the needs you could also consider Sql CE. I'm sure that if you specified the database you're thinking of using, or your requirements if you're usure you would get proper and real examples of connection strings etc.
Edit: Here's code for SqlCe / Sql Compact
public void ConnectListAndSaveSQLCompactExample()
{
// Create a connection to the file datafile.sdf in the program folder
string dbfile = new System.IO.FileInfo(System.Reflection.Assembly.GetExecutingAssembly().Location).DirectoryName + "\\datafile.sdf";
SqlCeConnection connection = new SqlCeConnection("datasource=" + dbfile);
// Read all rows from the table test_table into a dataset (note, the adapter automatically opens the connection)
SqlCeDataAdapter adapter = new SqlCeDataAdapter("select * from test_table", connection);
DataSet data = new DataSet();
adapter.Fill(data);
// Add a row to the test_table (assume that table consists of a text column)
data.Tables[0].Rows.Add(new object[] { "New row added by code" });
// Save data back to the databasefile
adapter.Update(data);
// Close
connection.Close();
}
Remember to add a reference to System.Data.SqlServerCe

I'm not seeing anybody suggesting SQL Compact; it's similar to SQLite in that it doesn't require installation and tailors to the low-end database. It grew out of SQL Mobile and as such has a small footprint and limited feature-set, but if you're familiar with Microsoft's SQL offerings it should have some familiarity.
SQL Express is another option, but be aware that it requires a standalone installation and is a bit beefier than you might need for an applciation's local cache. That said it's also quite a bit more powerful than SQL Compact or SQLite.

Seems like you're:
-Making a C# app that will be installed and run on multiple
computers
-That needs a local database (I'm assuming an RDBMS)
-You need to generate a blank database at installation
-You then need to be able to connect to the database and populate it when
the app runs.
In general, it seems that you need to read up on using a small database engine for applications. I'd start by checking out SQLite, especially if you need multi-OS capability (eg., your C# program will run on Microsoft's .NET Framework and Novell's Mono). There are C# wrappers for accessing the SQLite database.

I believe this question is about the "Local Database" item template in Visual Studio:

What are you considering as a database? From what little you've provided in your question, I'd suggest SQLite.
You can get sample code from their site Sqlite.NET

Not sure I fully understand what you're asking but Sqlite is a good option for lightweight, locally deployed database persistence. Have a look here:
http://www.sqlite.org/
and here for an ADO.NET provider..
http://sqlite.phxsoftware.com/

For 1)
The easiest way to provide this functionality is through SQL Server Express User Instances. SQL Server Express is free, so your user does not have to pay additional license for SQL Server, and the User Instance is file-based, which suits your requirement.
For 2)
This is a big topic. You may want to go through some of the tutorials from Microsoft to get the feeling of how to connect to DB, etc.

Connecting to a database from the beginning

I have been working in a business writing advanced software apps, and obviously im provided with access to our SQL server and all the connection strings needed.This is fine for my job now - but what if i wanted to do this for a new (very small) business... If i wanted to purchase a small database server and set up a piece of software that talks to the databases on this server, how would i go about a) Talking and connecting to the server in code (c#) and b)What would i need regarding things like internet/phone connections etc to make this possible.
Edit: the reason it would need a server is because it would need to be accessed from 2 or 3 different computers in different locations?

Actually there are quite a few ways to create a database connection, but I would say one of the easiest ways is to utilize the methods and classes found in System.Data.SQLClient. A basic connection would look something like the following:
using System.Data.SQLClient;
namespace YourNamespace
{
public class DatabaseConnect
{
public DataType getData()
{
DataType dataObj = new DataType();
SqlConnection testConn = new SqlConnection("connection string here");
SqlCommand testCommand = new SqlCommand("select * from dataTable", testConn);
testConn.Open()
using (SqlDataReader reader = testCommand.ExecuteReader())
{
while (reader.Read())
{
//Get data from reader and set into DataType object
}
}
return dataObj;
}
}
}
Keep in mind, this is a very, very simple version of a connection for reading data, but it should give you an idea of what you need to do. Make sure to use a "using" or "try/catch" statement to ensure that the connection is closed and resources are freed after each use (whether it successfully gets data or not).
As for your other question about what equipment you may require. In the beginning I would suggest just creating the database on your local machine and running tests from there. Once you are confident with trading data back and forth, feel free to move the database to another box or an online server. Any internet connection type should suffice, though I can't vouch for dial-up, haven't used it in years.
One final note, if you do happen to decide to move to an online server system, make sure that the service you use allows for outside connections. Certain services use shared server systems, and force users to use their internal database interfaces to manage and write to the database.
--- EDIT ---
As for the server system itself, build up a separate box on your local network that you can see, and load up the database software of your choice. Since you are using C#, it would probably be easiest to go with Microsoft SQL Server 2005 / 2008. The installation is rather straightforward, and it will prompt you to automatically create your first database while installing.
After installation it will be up to you to add in the tables, stored procedures, custom functions, etc... Once your base structure is created, go ahead and use the above code to make some simple connections. Since you are familiar with the above practices then I'm sure you know that all you really need to do is target the server machine and database in the connection string to be on your way.

In case your application is small (by small I mean the usage of resources like CPU and memory) then your SQL Server can reside on the same box.
Else you need to have a separate server box for your database and connect to that from your application. In this case, preferably your database box and application box would be on the local area network.
Check this link for having a connection to SQL Server from C# code - http://www.codeproject.com/KB/database/sql_in_csharp.aspx
cheers

You should probably expose your database with an xml web services layer, so that your architecture will be scalable. The general idea is host your sql server and webservices, using Native SQL Server XML Web Services you can make this available to your remote clients. When in your clients you simply add a service reference in Visual Studio and your data will now be available in your client app.
Hope this helps some.
Cheers

You may find the connectionstrings website useful - worth bookmarking.