Hololens video stream with spatial data - c#

I'm using the hololens and I'm trying to save a video stream with the world/projection matrices avaiable.
I've been trying to just take a sequence of pictures and save the data, but I can't find a way to save the image and the matrices.
When saving to disk, there is no option to get the photocaptureframe (which contains the matrix data), when saving to memory, I seem to not be able to save the image to disk.
I tried using the following methode, but this seemed to crash my unity program:
List<byte> imageBufferList = new List<byte>();
photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBufferList);
byte[] myArrayImage = imageBufferList.ToArray();
And then use this to convert the byte array:
using (MemoryStream mStream = new MemoryStream(byteArrayIn))
return Image.FromStream(mStream);
After which I save the result.
When I remove the memorystream thing, the program doesn't crash (but it doesn't save my image either).
I've been looking all over the internet but there are a lot of vague statements about it
a) not beeing possible
b) using the memorystream (but that crashes)
Any suggestions?
If anyone knows a way to save all the the matrix (projection and world) data per frame and the corresponding frame for a video stream, it would be a great help.
Edit: I also tried to look into https://github.com/VulcanTechnologies/HoloLensCameraStream but this seems to give problems with newer Unity versions. Any remarks about this?
To clarify my end goal:
When filming, the program should save all frames and the corresponding matrices, for example:
Frame_01, Frame_02, Frame_03, ... (.jpg/png/...)
World_matrix_01, World_matrix_02, ... (.txt)
Projection_matrix_01, Projection_matrix_02,... (.txt)

Edit: I also tried to look into https://github.com/VulcanTechnologies/HoloLensCameraStream but this seems to give problems with newer Unity versions. Any remarks about this?
I used it and it worked very well in Unity 2018.3.13f1. You can only test in Hololens out of debug mode.

Related

How can I download a video using VideoLib when it's Async?

Alright, here is my dilemma:
I wanted to learn how to use NuGet, so I tested it out using a system called VideoLibrary(https://www.nuget.org/packages/VideoLibrary/). I successfully got it installed to my computer, and was finally got it working inside of code. I was able to successfully download a video, which is the purpose of the extension, however it was in the wrong format. So, having read through the questions and answers section, I got this code:
var youTube = YouTube.Default;
var allVideos = await youTube.GetAllVideosAsync(URL);
var mp4Videos = allVideos.Where(_ => _.Format == VideoFormat.Mp4);
Now, from what my understanding is, that first downloads all the videos, and then sets the variable to only the MP4 video.
However, after that is done, I don't know how to save it to a byte array. Typically, to save a video using VideoLib, you have to assign it to a byte array and then save the byte array to a file. So I used this code previously with a video that wasn't async:
byte[] bytes = video.GetBytes();
var stream = video.Stream();
File.WriteAllBytes(#"C:\" + fullName, bytes);
Now, this method doesn't work with my videos now because they're async. THe developer mentioned later that you could assign the video to a byte array by using:
byte[] bytes = allVideos.GetBytesAsync();
However, that doesn't work for some reason.
I'm assuming I'm using it wrong and that's why, but I don't know.
The code is underlined in red, and it gives:
'IEnumerable<YouTubeVideo>' does not contain a definition to 'GetBytesAsync'
and no extension method 'GetBytesAsync accepting a first argument of type
'IEnumerable<YouTubeVideo>'coulb be found
Any help would be appreciated.
Okay, so, I've discovered the answer to my question:
The problem is that allVideos is multiple videos. When you use the method GetAllVideosAsync it retrieves a whole list of videos. You have to narrow it down.
When you use mp4Videos you've narrowed it down to one video, however the computer still doesn't know that because it's a variable of multiple videos.
So, simply put the code through this:
var SingleVideo = mp4Videos.FirstOrDefault();
Then, using the SingleVideo variable, you can save it to a byte array like a conventional video, but using the async method:
Byte[] Byte = await SingleVideo.GetBytesAsync();
Once the contents are saved to the byte array, you can save it any way you please.
I hope this helps anyone who needs it!

Sending bitmaps over ethernet and then back again (c# and c)

I'm looking to send bitmap (or image) over ethernet in order to perform some image processing, then send them back.
The 'client' is running c# code, extracting frames from a video using emgucv. The 'server' will be running c/c++ on an arm cpu, although at the moment is x86 on my laptop using elementary os. So I need to avoid using things like opencv for the image processing itself, but that's another point.
I looked into sockets etc and can send some data to/from the server, just text typed into a console at the moment.
From initial research, it seems like I'll need to convert the bitmap into a byte array in order to then send it, which I've done (I think) using the following code:
Stream stm = tcpclnt.GetStream();
int l;
using (MemoryStream ms = new MemoryStream())
{
bmpFrame.Save(ms, ImageFormat.Bmp);
byte[] byteFrame = ms.ToArray();
l = byteFrame.Length;
stm.Write(byteFrame, 0, byteFrame.Length);
stm.Flush();
}
Then on the server side trying to read it using:
char buff[10000];
int n;
n = read(conn_desc, buff, sizeof(buff)-1);
if ( n > 0)
{
//MemoryStream ms = new MemoryStream(receivedBytes);
//Bitmap bmpReceived = new Bitmap(ms, System.Drawing.Imaging.ImageFormat.Bmp);
printf("Received %s\n", buff);
}
else
{
printf("Failed receiving\n");
}
You can see the commented code where I thought I'd be able to change it back into a bitmap, but I'm not sure if I want/need to anymore, if I can just edit the images by accessing the bytes directly, and also I don't know how/if bitmaps work in C rather than C#.
Am I going along the right lines? Ideally I want to send all the data for a single frame, then do stuff to it, then send it back, but I've no idea if that's what I'm doing. Finding it more difficult than usual as on the server I'm just writing in scratch/gedit and using gcc to compile, having never coded on linux before so I'm missing things like intellisense/debugging.
Any help/recommendations would be appreciated
Yes, in short, I would say your approach for doing what you say you want to do is correct.
I would ask though, why are you sending it to the server? I'm guessing it has to do with parallel processing or something? Unless you're doing very heavy processing, working with a single frame is probably faster than transferring it to the server (as far as I know, EmguCV in C# isn't considerably slower than OpenCV in C/C++).
The System.Drawing.Bitmap class is part of the .Net framework, and you might not find a direct correspondance in C. You might want to consider first converting the image to some known format, such as a .png file. Then you send the raw bytes of that file. On the server side, you receive the raw byte array, and use whatever C struct to load that image as a .png file. The point is to remove some abstraction and have more precise knowledge of exactly what is being sent.
Perhaps it'd be easier to use something like the System.Net.WebRequest class? Again, it depends a bit on exactly what problem you're trying to solve.
I hope this helps at all - this response is a bit vague (your question is, too :P), so feel free to ask for clarification on specific parts :)

Storing an SVG as bytes in DB?

Wee bit of background to set the scene : we've told a client he can provide us with images of any type and we'll put them on his reports. I've just had a quick run through of doing this and found that the reports (and other things between me and them) are all designed to only use SVGs.
I thought I'd struck gold when I found online that you can convert an image from a jpg or PNG into an SVG using free tools but alas I've not yet succeeded in getting an SVG stored as bytes in the DB in a format that allows me to use it again after reading it back out.
Here's a quick timeline of what followed leading up to my problem.
I use an online tool to generate an SVG from a PNG (e.g., MobileFish)
I can open and view it in Firefox and it looks ok
I ultimately need the SVG to be stored as bytes in the DB, from where the report will pull it via a webpage that serves it up as SVG. So I write it as bytes into a SQL data script. The code I use to write these bytes is included below.
I visit said webpage and I get an error that there is an "XML parsing error on Line 1 Column 1" and it shows some of my bytes. They begin "3C73"
I revisit the DB and compare the bytes I've just written there with some pre-existing (and working ones). While my new ones begin "3C73", the others begin "0xFFFE".
I think I've probably just pointed out something really fundamental but it hasn't clicked.
Can someone tell me what I've done that means my bytes aren't stored in the correct encoding/format?
When I open my new SVG in Notepad++ I can see the content includes the following which could be relevant :
<image width="900" height="401" xLink:href="data:image/png;base64,
(base 64 encoded data follows for 600+ lines)
Here's the brains of the code that turns my SVG into the bytes to be stored in DB :
var bytes = File.ReadAllBytes(file);
using (var fs = new StreamWriter(file + ".txt"))
{
foreach (var item in bytes)
{
fs.Write(String.Format("{0:X2}",item));
}
}
Any help appreciated.
Cheers
Two things:
SVGs are vector images, not bitmap files. All that online tool is doing is taking a JPEG and creating a SVG file with a JPEG embedded in it. You aren't really getting the benefit of a true SVG image. If you realise and understand that, then no worries.
SVG files are just text. In theory there is no reason you can't just store them as strings in your db. As long as the column is big enough. However normally if you are storing unstructured files in a db, the preferred column type to use is a "Blob".
http://technet.microsoft.com/en-us/library/bb895234.aspx
Converting your SVG file to hex is just making things slower and doubling the size of your files. Also when you convert back, you have to be very careful about the string encoding you are using. Which, in fact, sounds like the problem you are having.
I am suspecting you are doing it incorrectly. SVG is simply and XML based vector image format. I guess your application might be using SVG image element and you need to convert your png image to base64 encoded string .

Getting mp3 file length

I am currently trying to write an Audio Player in C#. I am using BASS library to deal with playing music but now i have small problem with getting length of the song.
Well i have read BASS doc and found a way:
"All" i need to do is
int stream = Bass.BASS_StreamCreateFile(filepath,....);
int length = Bass.BASS_ChannelBytes2Seconds(stream, Bass.BASS_ChannelGetLength(stream));
And in most of cases i get valid length of song. And here the problem starts. As far as i know the stream creation operation is quite expensive (correct me if i am mistaken) and creating a stream only to get length of the song looks a little silly.
So my question is: Is there any other way to get it without creating steam file (not being so expensive). I will be thinking later about reading id3 tags. Is creating that stream "evil that must be done no matter what" and even if i would try to get it with other library it would do exactly the same thing?
You can use the Microsoft.WindowsAPICodePack.Shell:
using Microsoft.WindowsAPICodePack.Shell;
Then code like so:
string file = "myfile.mp3"
ShellFile so = ShellFile.FromFilePath(file);
double 100nanoseconds;
double.TryParse(so.Properties.System.Media.Duration.Value.ToString(), out 100nanoseconds);
There is a code project that could help you as well

C# PdfImage LibTiff iTextSharp G3 / G4 compression

I have a service that takes a pdf document, resizes all the images, and replaces it in the pdf. The problem that I'm getting at, is the compression.
Some documents are scanned and saved with a Compression.CCITTFAX3 compression and some are saved with a Compression.CCITTFAX4 compression. I am using iTextSharp and convert the stream bytes to a Tiff, otherwise the image becomes funky because of stride or something.
Below is the code I'm currently making use of to check for the correct filter, and then convert to tiff image.
if (filter == "/CCITTFaxDecode")
{
byte[] data = PdfReader.GetStreamBytesRaw((PRStream)stream);
using (MemoryStream ms = new MemoryStream())
{
using (Tiff myTiff = Tiff.ClientOpen("in-memory", "w", ms, new TiffStream()))
{
myTiff.SetField(TiffTag.IMAGEWIDTH, UInt32.Parse(dict.Get(PdfName.WIDTH).ToString()));
myTiff.SetField(TiffTag.IMAGELENGTH, UInt32.Parse(dict.Get(PdfName.HEIGHT).ToString()));
myTiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX3);
myTiff.SetField(TiffTag.BITSPERSAMPLE, UInt32.Parse(dict.Get(PdfName.BITSPERCOMPONENT).ToString()));
myTiff.SetField(TiffTag.SAMPLESPERPIXEL, 1);
myTiff.WriteRawStrip(0, data, data.Length);
myTiff.Flush();
using (System.Drawing.Image img = new Bitmap(ms))
{
if (img == null) continue;
ReduceResolution(stream, img, quality);
}
myTiff.Close();
}
}
}
Just to make sure that you understand my question...
I want to find out how I know when to use G3 compression and when to use G4 compression.
Keep in mind that I've tried every code sample I could find.
This is quite important, as we interface with banking systems, and the files uploaded are sent to them as FICA documents.
Please help...
You need to go low level and inspect the image dictionary. The /DecodeParms entry is a dictionary that contains several keys related to CCITT compression. The /K key specifies the compression type: -1 is G4, 0 is G3 1D and 1 is G3 2D.
Update: to be more exact a negative value, usually -1, is G4, 0 is G3 1D and a positive value, usually 1, is G3 2D. To answer your question in the comment, the /K entry is optional and if it is missing the default value is considered to be 0.
I would not advise inserting the data direct. I base this assertion on many years of practical experience of PDFs and TIFF in products like ABCpdf .NET (on which I work).
While in theory you should be able to move the data over direct, minor differences between the formats of the compressed data are likely to lead to occasional mismatches.
The fact that some Fax TIFFs contain data which will display correctly in a TIFF viewer but not in a PDF one leads me to suspect that the same kind of problem is likely to operate in the other direction too.
I'm not going to say this kind of problem is common but it is the kind of thing I wouldn't rely on if I was in a bank. Unless you are very sure your data source will be uniform I would suggest it is much safer to decompress and recompress.
I would also note that sometimes images are held inline in the content stream rather than in a separate XObject. Again this is something you will need to cope with unless your data source produces a standard format which you can be sure will not contain this kind of structure.
Thank you for the replies above. The solution from Mihai seems viable if you do have all the information from the stream. I found that iTextSharp does not do this properly, so I ended up buying pdf4net. Much simpler than trying to figure out whats the better solution, besides, it ended up cheaper than my time I spent on this.
OnceUponATime.... Thank you for the information given above.
PDF4Net has a built in method that you get all the images per page... This sorted my issues, whereas I tried to do this myself using iTextSharp and the examples that were given to me.

Categories

Resources