I'm trying to code a very basic C# console app to read from a bucket in AWS. The file I'm trying to read is in the avro format.
At this point I have a console program with the nuget packages for AWSSDK.s3, AWSSDK.Core, and the avro package from Apache.
I know how to get a list of files in the bucket. So I can connect to AWS. I guess what I need to do now is figure out how to deserialize the data.
The final goal is to load the data into an SQL Server table. The files I'm working with are not very large.
We are working with another company on this project and they are sending us this data in the avro format.
I'm completely new to AWS programming and never heard of avro until about a week ago. Finding information on the internet has been kind of hard.
Any help would be great.
Thanks.
First, you need to read object data from an Amazon S3 bucket using the AWS SDK for .NET. You can use the client.GetObjectAsync method to read the data. See this example in Github:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/dotnetv3/S3/GetObjectExample/GetObjectExample/GetObject.cs
For your use case, you would need to use a lib like https://www.nuget.org/packages/Apache.Avro/ to handle the Avro requirements. As this is a very specific use case, I doubt you will find this specific example on the Internet.
Related
i'm trying to use Azure Cognitive Services Speech to Text and i am hitting a roadblock in .net Core
i have native support for a WAV file using the audioConfig.FromWafFileInput(); which is great.
however i need to also support MP3's
I have found compressed audio support
https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-use-codec-compressed-audio-input-streams?tabs=debian&pivots=programming-language-csharp
however this is referencing PushAudio Streams.
this is where i'm getting lost....
i have found this example for stream codec compressed audio
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/cpp/linux/compressed-audio-input/compressed-audio-input.cpp
however this is not C# .net core and conversion is not really my strong suit.
so yeah at a bit of a loss.
any assistance would be greatly appreciated (y)
This sample: https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/csharp/sharedcontent/console/speech_recognition_samples.cs has compressed audio specific methods here and here. The latter pull stream sample seems pretty straightforward, just plug in your key, region, and filepath.
If you have files, especially if you have multiple of them, you can benefit from using batch transcription. It natively supports files in WAV, MP3 and OGG format.
The documentation links to the API documentation, that also includes model customization. Here you can select the region you are interested in and export a swagger file. The swagger file you can use to generate a client in the programming language of your choice.
For your scenario you will only need 4 APIs and you could use the standard HttpClient to execute the requests. You would want to
Create a batch transcription.
Get your transcriptions to check the state. If it is complete, you get the URL you will need next. If it is failed, you get a message about the problem.
Get the results after the batch transcription succeeded. The object with the kind TranscriptionReport contains a list of files that got transcribed, if the transcription was successful and if not, why. The other objects contain the result of the successful transcriptions.
(here you need to iterate over the contentUrls, to download the files.)
Delete the transcription(s), after you got the results.
Being a novice in Python and C#, and having reached a point with no significant progress, I'd like to ask for some suggestions.
Here is the issue I find:
I am trying to read the data from a Digital Multimeter through its C# library and targeting to export it into Python (being the only supported language of the target application).
The meter and communication interface libraries come from Lcrrsearch.
Referring to the instructions provided, I have run the C# project file and have succeeded in getting the readings (through FTD2XX_NET.dll, LCR_CIL.dll).
However, separately, when trying to use the .dll files in Python, I have failed to do so.
The solution paths I sought after were:
Calling the LCR_CIL.dll in Python via Ctypes
Trying IronPython via Visual Studio for .NET
Using FTDI library to read the data from memory
Therefore, I would like to have some hints or ideas as to which path is the most straightforward one to import the data in Python and/or any similar example you could propose I would gladly welcome.
I am developping a mobile app using Unity3D framework which is a C#/Javscript 3D engine, working on desktop and mobile plateform.
I need to recover the content of an XLS file on a cloud-based storage (Dropbox or GoogleDrive), then process it on the mobile plateform to transform it into a local SQLite database.
The mobile app will regulary check for modification on the remote xls file, and push it locally if needed to process it.
The framework I use (Unity3D) allow me to work with both C# and Javascript technologies.
What would be the best strategy to implement such a fonctionnality.
I'm totally newbie with web API, and I've seen in Dropbox documentation that there is several possibilities to interact with a cloud folder.
Is there a possiblity to fetch the content of an xls document into the mobile device memory (with writing to its local drive)? What would be the easiest/more elegant way to achieve it?
Thanx in advance
Unity3D supports using managed DLLs in your project. So you can look at using one of the following libraries:
http://sergeytihon.wordpress.com/2013/09/29/dropbox-for-net-developers/
https://developers.google.com/drive/web/quickstart/quickstart-cs
and choose one.
There's also code for reading excel files from Unity3D on the wiki: http://wiki.unity3d.com/index.php?title=Load_Data_from_Excel_2003
And code for using SQLite: http://wiki.unity3d.com/index.php/SQLite.
From there putting it together is up to you, the documentation from each link should be more than enough to implement what you want. I'd recommend abstracting the differences in platforms here (Google Drive vs Dropbox) from your main game (It doesn't sound like you've made up your mind on one), in case you want to change them later on. In my experience, operations between the two are similar enough to make changing over reasonably straight forward down the road as long as you're using the proper abstraction techniques.
I've been tasked with a feature that can generate PowerPoint files on the server using C#. I'd basically start with a template, and programmatically replace some text with live data from the database. I've been doing some research into this area for the past day and here's what I've found:
PowerPoint has this sort of thing built in, meaning it can connect to external data sources and pull in data. Most examples of this, I've found, either use PowerPoint automation done on the server (I've been advised against this) or assume a SQL Server backend. Our company uses Oracle for our RDMS needs. Oracle has a solution for this called Oracle BI, but it requires a whole new web server setup to run various Java EE components and what not. I didn't look at the price, but knowing Oracle it's not cheap. It also requires new software to be installed on the end user's machine, which we really want to avoid.
Generating PowerPoint files on the fly is possible. The company that is basically the go-to guys for this problem (every help forum points to them, and they get all the rave reviews) is Aspose. They have .NET components for dealing with just about any Office format you can think of. The problem is, they are astronomically expensive. Just the PowerPoint component (a site license for up to 10 developers) would cost $3,995.
The third possibility is generating a solution in-house. After all, a PPTX file is just xml, right? Well, looking closer, a PPTX appears to be a gzip archive. It contains many folders, each containing many XML files. Modifying a PPTX file would, correct me if I'm wrong, entail unzipping the file to a temporary directory, reading the XML file and modifying the contents, then packaging up everything again and write the file out to the response stream. Perhaps there are libraries that can work with gzip streams on the fly without extracting everything.
My Question: Are there easier ways to work with a PPTX file using .NET that don't require working with compressed XML files or buying very expensive software? Basically, we need to modify a PowerPoint file, change some text, and allow the user to download that generated file from a web server.
OpenXML is Microsoft's .Net library that lets you manipulate Office documents. It lets you open a PPTX file and provides an object model that wraps the XML contents.
Here's the link to the OpenXML SDK and the MSDN documentation.
I've used OpenXML to let a ASP.Net page dynamically generate Word documents from a database.
Don't use Office Interop on a web server. It's an all-around bad idea.
If you are only replacing text placeholders for files that will not change, the home grown solution that finds the placeholders in the xml files in the gzip archive should be doable. .Net has had zip support for some time, and it is greatly improved if you are able to use .Net 4.5, so you shouldn't need to extract the archive to a temporary location at all.
PowerPoint should also support connecting directly to Oracle in the same way it supports connecting to Sql Server (just play around with the connection options), without needing the special Oracle BI stuff. However, I'd still prefer the home-grown solution, as this will only work while the powerpoint file is able to reach your database directly, which is typically only possible in your local LAN environment or with an active VPN.
If you want anything fancier than a simple text replacement, perhaps looks for an Aspose competitor.
I am trying to make tool for backup/restore of Documents from Google account.
Backup is easy and I have no problems with it. But I have two unsolved questions for restore:
1) Is it possible to upload new version of existing document? When I upload document, it appears as separate copy.
I found it was discussed already here Upload and replace file in given folder on Google Docs using .net api, but it seems it was suggested just to remove old version before uploading new, the Id of document will be changed. Is this correct?
2) Google Docs have limit for size of documents able to be converted into internal format. http://docs.google.com/support/bin/answer.py?hl=en&answer=37603. So it is possible to create large document, save it to local computer and then Google Docs will refuse to convert it because the document's size is over limit. In such case it is possible to upload the document without convert, but it becomes un-editable via web site. Is there some workaround for this situation?
Unable to upload large files to Google Docs - Here is advice to break document into small pieces before uploading and link them together after. But maybe there some other ideas?
1. Is it possible to upload new version of existing document? When I upload document, it appears as separate copy.
Yes, this is possible. We call it "upload & replace" as you've noticed. No need to remove the existing version first. The following link describes how to do this in the protocol:
http://code.google.com/apis/documents/docs/3.0/developers_guide_protocol.html#UpdatingMetadataAndContent
From the .NET client library, what you need to do is attach a an input stream to the Update() request. The method header for what you need is here:
http://code.google.com/p/google-gdata/source/browse/trunk/clients/cs/src/core/service.cs#554
Create a stream containing your new file content, and just pass that in. That should be it!
2. Google Docs have limit for size... Is there some workaround for this situation?
Unfortunately there is not a way currently to circumvent the size limitations of converted documents. They must be uploaded as unconverted files, and thus, are not editable in the Google Docs user interface.