I am trying to implement the Aspera cloud object storage using ATS API in .NET.
Is there any way to retrieve the cloud API object to retrieve using C#?
While going with the IBM documentation there is only support for Java and python.
I would suggest the fllowing:
look at: https://www.rubydoc.info/gems/asperalm#ibm-cloud-object-storage
This will allow you to transfer to COS using command line and Aspera
analyze the code, it is relatively sumple (cos.rb)
use IBM Aspera "Transfer SDK" you can find here:
https://developer.ibm.com/aspera
write your own code for the REST calls part, and use the Transfer SDK once you have built the "transfer spec" to start the transfer.
We have an emerging need to modify the schema in our Agile Data Warehouse including adding new tables. We've been able to manually connect to the ADS database using Squirrel SQL and CloudConnect.
However, we would like a way to automate this process so that we can ensure that the schema remains consistent between our development, test and production ADS instances.
We're a .NET shop and most of our code is in C#. Has anyone had any success connecting directly to ADS using .NET (C# or VB)?
I've looked at trying to use the GoodData JDBC driver but it looks like referencing a JDBC driver from .NET is not particularly straight-forward and there is not a GoodData ODBC or ADO.NET driver available.
I'd rather use something like the Vertica driver for ADO.NET available at https://my.vertica.com/vertica-client-drivers/ however, I'm not sure what to use for all of the connection properties. I've attempted to connect using the Host and Port that are returned from the DW connection endpoints API but receive the following error: "SSL Startup Failed." when using the Vertica ADO.NET driver.
Is there a way to connect to the GoodData ADS daatabase using .NET or any better approaches to modifying the ADS schema using a CloudConnect graph or REST API?
Any advice would be appreciated.
GoodData currently provides only a custom JDBC driver for connecting to ADS. Standard Vertica drivers cannot be used. See https://help.gooddata.com/display/doc/Data+Warehouse+Technology
It is not possible to use the JDBC driver in .NET for obvious reasons. Theoretically it could be possible to use a ODBC-JDBC gateway, but I have not tested this solution. Also there is JVM implementation for .NET, but I have not tested it also. It is http://www.ikvm.net/.
So the easiest way is really to use CloudConnect or a SQL client which supports JDBC drivers. For automating a process the easiest ways are probably Java or JRuby.
I am trying to connect to a PostgreSQL database using SSL. The connection requires that SSLMode is verify-full, but the driver I use (from devart.com), supports Require as the highest level.
Is there a way to accomplish this using the latest driver from devart? If not, is there any other driver that supports this?
I am able to connect using a native ODBC driver, so the connection works for native programs, but need a .Net driver to avoid importing ODBC libraries into my .NET application.
Use nPgSQL with SSL=True;Sslmode=Require; to enforce SSL. The enum doesn't seem to provide Verify, though (like in the JDBC driver) it might always verify unless overridden. I don't have it around to test with.
If you need to you can provide a suitable CertificateValidationCallback to do a certificate verification using C# APIs yourself.
I am working on a solution where I will have a Hadoop cluster with Hive running and I want to send jobs and hive queries from a .NET application to be processed and get notified when they are done. I can't find any solutions for interfacing with Hadoop other than directly from a Java app, is there an API I can access that I am just not finding?
Apparently it is possible to connect to Hadoop with non-Java solutions - see Do I have to write my application in Java?
With Hadoop: there is no straight way to connect from C# because Hadoop communication tier is working with java only and is not cross platform. It is probably possible but in very non-trivial ways.
I know there is a patch to add Protocol Buffers support for Hadoop but at the moment of writing (Aug 2011) is is not released yet.
With Hive situation is better because Hive has Thrift interface which supports C#. You can download Hive Thrift interfaces and generate C# client on your own but beware that it requires some hacking of generated code. Instead I would recommend you downloading dll from https://bitbucket.org/vadim/hive-sharp/downloads/hive-sharp-lib.dll or use Nuget package manager, search for "hive": http://nuget.org/List/Packages/Hive.Sharp.Lib
Disclaimer: I'm the author.
There is Hortonworks ODBC driver. I havn't used it personally, but it shall let you work with hive as with any other ODBC datasource. You can use OdbcConnection class to connect to Hive once ODBC driver is installed.
As noted in other answers - you can use Thrift api. For that you need to generate C# classes from interface definition files, which you can download from Hive source repository. This approach works for me.
You can use IKVM, to convert hadoop client java libraries into .Net assemblies which you can use from C#. I havn't used IKVM with Hive client, but I've IKVMed some other hadoop client library and surprisingly it worked.
EDIT:
There's also Apache templeton, which allows submitting Hive jobs (Pig and MR also) using Rest interface. The problem with it is that it spawns another map task to submit Hive job, which makes it slower.
It is possible to access Hive utilizing C# by making use of Microsoft's ODBC connector. Download the Nuget package for "Microsoft.Hadoop.Hive" and follow the example provided at http://msdn.microsoft.com/en-us/library/dn749834.aspx
The trick lies in building the connection string to connect with it. The best way I came up with was to download the Microsoft Hive ODBC Driver (http://www.microsoft.com/en-us/download/details.aspx?id=40886), install it, then use the Server Explorer inside Visual Studio to add a new connection, then build the connection string for me. To do this, I used the following steps:
Change the data source to "Microsoft ODBC Data Source" and ensure you're using the ".NET Framework Data Provider for ODBC" as the data provider.
Under the "Data source specification" portion, check the "Use connection string" then click the "Build" button.
Under the "Machine Data Source" tab, select the "Sample Microsoft Hive DSN" data source name, then click the "OK" button.
A window titled "Microsoft Hive ODBC Driver Connection Dialog" will open. Enter an optional description, then type in the path to your Hive server, the port you will be using, and what database it should connect to. Indicate the Hive Server Type, and specify an authentication mechanism to use, then fill out the appropriate fields.
Finally, click the "Test" button in the bottom to ensure that you're able to successfully connect. If successful, click the "OK" button, then you'll be back in the "Modify Connection" window. Enter the login information for your Hive service here.
Either utilize this data source or copy the connection string that it's built for you and use it within your application.
Thrift API is also another way for other language to access hdfs and hive
See if this helps. I have tried to connect to Hadoop via C#
How to communicate to Hadoop via Hive using .NET/C#
Use Hbase.Net library from https://hbasenet.codeplex.com/
Then you can connect to hbase/hive as shown below:
Client c = new Client("10.20.14.179", 9090, 1000000);
var cli = c.TotalClients;
var tableList = c.GetTableNames();
FYI, we are using hortonworks sandbox and it connects fine.
In above example, 10.20.14.179 is host and 9090 is port.
Also, below might help from https://community.hortonworks.com/questions/25101/is-there-a-way-to-connect-to-hbase-using-c.html
There is no native C# HBase client. however, there are several options for interacting with HBase from C#.
C# HBase Thrift client - Thrift allows for defining service endpoints
and data models in a common format and using code generators to
create language specific bindings. HBase provides a Thirft server and
definitions. There are many examples online for creating a C# HBase
Thrift Client.
Marlin - Marlin is a C# client for interacting with Stargate (HBase
REST API) that ultimately became hbase-sdk-for-net. I have not
personally tested this against HBase 1.x+, but considering it uses
Stargate, I expect it should work. If you are planning to use
Stargate and implement your own client, which I would recommend over
Thrift, make sure to use protobufs to avoid the JSON serialization
overhead. Using a HTTP based approach also makes it much easier to
load balance requests over multiple gateways.
Phoenix Query Server - Phoenix is a SQL skin on HBase. Phoenix Query
Server is a REST API for submitting SQL queries to Phoenix. Here is
some example code, however, I have not yet tested it.
Simba HBase ODBC Driver - Using ODBC to connect to HBase. I've heard
positive feedback on this approach, especially from tools like
Tableau. This is not open source and requires purchasing a license.
I'm in the initial phase of designing an application that will have a backend implemented in C# that will provide data for other platforms using WCF web services hosted on IIS. One of the platforms will the the iPhone.
Since it's a personal project, I want to use it to learn MongoDB. I already know that there are community developed drivers for MongoDB and C#, so I could handle the persistence on the server side using MongoDB.
Without even knowing the replications models offered by MongoDB, I was thinking about some kind of simple synchronization model to keep data local if the iPhone is not connected or has a poor connection.
Here's the question: Can MongoDB be used in the iPhone using the MongoDB C drivers? Has anybody already tried that?
The typical iPhone architecture is to have your application call out to a web service. Even if it is possible to use a MongoDB driver directly from a mobile client I would not recommend it. For a few reasons.
You are basically talking about doing client server architecture where your client application talks directly to the datastore (MongoDB.) What about security? When any authenticated client talks directly to the datastore all sorts of bad things can happen.
Tightly coupling your client application directly to any given data access technology is dangerous in that it would require you to rewrite your client if for some reason you needed to change your data access solution.
It is more common these days to have your client applications go through a data access tier and when the Internet is involved this tier often involves a web service of some sort unless you want to get elbows deep writing server code.
Think about writing a RESTful api exposing your datastore to your iPhone client. I've heard good things about Open Rasta (C# REST library)
Edit - More about hosting MongoDB on the iPhone
Sorry I didn't understand that you wish to run MongoDB locally on iPhone. MongoDB is a server. I do not believe that it is embeddable as an in-process datastore. And it is not possible to run more than one process on the iPhone.
If you are familiar with C# you might want to check out MonoTouch. It allows you to run C# applications on iPhone. There is a nice library for using SqlLite which is supported by iPhone.