I am trying to implement the Aspera cloud object storage using ATS API in .NET.
Is there any way to retrieve the cloud API object to retrieve using C#?
While going with the IBM documentation there is only support for Java and python.
I would suggest the fllowing:
look at: https://www.rubydoc.info/gems/asperalm#ibm-cloud-object-storage
This will allow you to transfer to COS using command line and Aspera
analyze the code, it is relatively sumple (cos.rb)
use IBM Aspera "Transfer SDK" you can find here:
https://developer.ibm.com/aspera
write your own code for the REST calls part, and use the Transfer SDK once you have built the "transfer spec" to start the transfer.
We have an emerging need to modify the schema in our Agile Data Warehouse including adding new tables. We've been able to manually connect to the ADS database using Squirrel SQL and CloudConnect.
However, we would like a way to automate this process so that we can ensure that the schema remains consistent between our development, test and production ADS instances.
We're a .NET shop and most of our code is in C#. Has anyone had any success connecting directly to ADS using .NET (C# or VB)?
I've looked at trying to use the GoodData JDBC driver but it looks like referencing a JDBC driver from .NET is not particularly straight-forward and there is not a GoodData ODBC or ADO.NET driver available.
I'd rather use something like the Vertica driver for ADO.NET available at https://my.vertica.com/vertica-client-drivers/ however, I'm not sure what to use for all of the connection properties. I've attempted to connect using the Host and Port that are returned from the DW connection endpoints API but receive the following error: "SSL Startup Failed." when using the Vertica ADO.NET driver.
Is there a way to connect to the GoodData ADS daatabase using .NET or any better approaches to modifying the ADS schema using a CloudConnect graph or REST API?
Any advice would be appreciated.
GoodData currently provides only a custom JDBC driver for connecting to ADS. Standard Vertica drivers cannot be used. See https://help.gooddata.com/display/doc/Data+Warehouse+Technology
It is not possible to use the JDBC driver in .NET for obvious reasons. Theoretically it could be possible to use a ODBC-JDBC gateway, but I have not tested this solution. Also there is JVM implementation for .NET, but I have not tested it also. It is http://www.ikvm.net/.
So the easiest way is really to use CloudConnect or a SQL client which supports JDBC drivers. For automating a process the easiest ways are probably Java or JRuby.
I', using
Google .NET Client Library
Stable Release '1.6.0-beta' (Wednesday, October 23, 2013)
But the samples it contains, any one refers to bigquery.
the examples I found, refer to the Google.Apis.Bigquery.v2.dll which I found somewhere else
and other dlls, that are not in the client library... I have them all already! But I still cant
make the sample work!...
both examples in this question
Google BigQuery with .NET documentation/ samples
have methods that are not valid now. like "Fetch", or some classes not compatible or different!
¿Can any of you, share me a small project, just to make a query to a table?
I already have the table on the paid service, I can query it from the webpage!... I just need to do it from a c# simple console connecting server to server, in order to start using it in my web application.
thanks in advance.
I don't have C# to test it, but I have some pointers:
You can find the latest BigQuery dll at https://developers.google.com/api-client-library/dotnet/apis/bigquery/v2.
For an example that uses OAuth2, follow https://code.google.com/p/google-api-dotnet-client/wiki/GettingStarted
Once you are done installing the dlls and OAuth2 it should be easy to run and review the BigQuery code, even if I can't run it.
Is there any SDK compatible with C# to get live data from a network fingerprint reader like EP300?
I made an app on C#/SQL Server only compatible with usb fingerprint readers (with Griaule SDK), but now i need made it compatible with network fingerprint readers, getting the live data from the machine
I just found this, but it's incomplete
http://www.codeproject.com/Questions/125530/FingerPrint-machine.aspx
Yes, download the C# sdk version from the following link
I am working on a solution where I will have a Hadoop cluster with Hive running and I want to send jobs and hive queries from a .NET application to be processed and get notified when they are done. I can't find any solutions for interfacing with Hadoop other than directly from a Java app, is there an API I can access that I am just not finding?
Apparently it is possible to connect to Hadoop with non-Java solutions - see Do I have to write my application in Java?
With Hadoop: there is no straight way to connect from C# because Hadoop communication tier is working with java only and is not cross platform. It is probably possible but in very non-trivial ways.
I know there is a patch to add Protocol Buffers support for Hadoop but at the moment of writing (Aug 2011) is is not released yet.
With Hive situation is better because Hive has Thrift interface which supports C#. You can download Hive Thrift interfaces and generate C# client on your own but beware that it requires some hacking of generated code. Instead I would recommend you downloading dll from https://bitbucket.org/vadim/hive-sharp/downloads/hive-sharp-lib.dll or use Nuget package manager, search for "hive": http://nuget.org/List/Packages/Hive.Sharp.Lib
Disclaimer: I'm the author.
There is Hortonworks ODBC driver. I havn't used it personally, but it shall let you work with hive as with any other ODBC datasource. You can use OdbcConnection class to connect to Hive once ODBC driver is installed.
As noted in other answers - you can use Thrift api. For that you need to generate C# classes from interface definition files, which you can download from Hive source repository. This approach works for me.
You can use IKVM, to convert hadoop client java libraries into .Net assemblies which you can use from C#. I havn't used IKVM with Hive client, but I've IKVMed some other hadoop client library and surprisingly it worked.
EDIT:
There's also Apache templeton, which allows submitting Hive jobs (Pig and MR also) using Rest interface. The problem with it is that it spawns another map task to submit Hive job, which makes it slower.
It is possible to access Hive utilizing C# by making use of Microsoft's ODBC connector. Download the Nuget package for "Microsoft.Hadoop.Hive" and follow the example provided at http://msdn.microsoft.com/en-us/library/dn749834.aspx
The trick lies in building the connection string to connect with it. The best way I came up with was to download the Microsoft Hive ODBC Driver (http://www.microsoft.com/en-us/download/details.aspx?id=40886), install it, then use the Server Explorer inside Visual Studio to add a new connection, then build the connection string for me. To do this, I used the following steps:
Change the data source to "Microsoft ODBC Data Source" and ensure you're using the ".NET Framework Data Provider for ODBC" as the data provider.
Under the "Data source specification" portion, check the "Use connection string" then click the "Build" button.
Under the "Machine Data Source" tab, select the "Sample Microsoft Hive DSN" data source name, then click the "OK" button.
A window titled "Microsoft Hive ODBC Driver Connection Dialog" will open. Enter an optional description, then type in the path to your Hive server, the port you will be using, and what database it should connect to. Indicate the Hive Server Type, and specify an authentication mechanism to use, then fill out the appropriate fields.
Finally, click the "Test" button in the bottom to ensure that you're able to successfully connect. If successful, click the "OK" button, then you'll be back in the "Modify Connection" window. Enter the login information for your Hive service here.
Either utilize this data source or copy the connection string that it's built for you and use it within your application.
Thrift API is also another way for other language to access hdfs and hive
See if this helps. I have tried to connect to Hadoop via C#
How to communicate to Hadoop via Hive using .NET/C#
Use Hbase.Net library from https://hbasenet.codeplex.com/
Then you can connect to hbase/hive as shown below:
Client c = new Client("10.20.14.179", 9090, 1000000);
var cli = c.TotalClients;
var tableList = c.GetTableNames();
FYI, we are using hortonworks sandbox and it connects fine.
In above example, 10.20.14.179 is host and 9090 is port.
Also, below might help from https://community.hortonworks.com/questions/25101/is-there-a-way-to-connect-to-hbase-using-c.html
There is no native C# HBase client. however, there are several options for interacting with HBase from C#.
C# HBase Thrift client - Thrift allows for defining service endpoints
and data models in a common format and using code generators to
create language specific bindings. HBase provides a Thirft server and
definitions. There are many examples online for creating a C# HBase
Thrift Client.
Marlin - Marlin is a C# client for interacting with Stargate (HBase
REST API) that ultimately became hbase-sdk-for-net. I have not
personally tested this against HBase 1.x+, but considering it uses
Stargate, I expect it should work. If you are planning to use
Stargate and implement your own client, which I would recommend over
Thrift, make sure to use protobufs to avoid the JSON serialization
overhead. Using a HTTP based approach also makes it much easier to
load balance requests over multiple gateways.
Phoenix Query Server - Phoenix is a SQL skin on HBase. Phoenix Query
Server is a REST API for submitting SQL queries to Phoenix. Here is
some example code, however, I have not yet tested it.
Simba HBase ODBC Driver - Using ODBC to connect to HBase. I've heard
positive feedback on this approach, especially from tools like
Tableau. This is not open source and requires purchasing a license.