Unity Hololens Asset streaming optimization - c#

I'm working on a Hololens app that displays a PNG image with info for the user. The images are loaded from a StreamingAssets folder in a coroutine. The issue lies in the speed at which these assets are loaded. If the user cycles to another page, the app momentarily drops to about 1-3 FPS on a PC.
What I hope some of you can help me with is think of ways to optimize the storage and streaming of these images. Is there a way to for example, load the image in a lower resolution to save on time and memory (with the hardware's very limited memory) and load in the additional detail when it actually needs to be displayed? Would multi-threading make the framerate better while loading the image?

So Programmer's suggestion in the comments helped eliminate the performance issues completely. The code down below is the coroutine that is used to stream in the required image(s) upon startup.
IEnumerator LoadImages()
{
int oldImgIndx = imageIndex;
imageIndex = 1;
bool thereAreImages = true;
while (thereAreImages && imageIndex < 1000)
{
if (System.IO.File.Exists(CreateFilePath(imageIndex)))
{
string url = "File:///" + CreateFilePath(imageIndex);
Texture2D tex = new Texture2D(4, 4);
WWW www = new WWW(url);
yield return www;
www.LoadImageIntoTexture(tex);
spriteList.Add(Sprite.Create(tex, new Rect(0, 0, tex.width, tex.height), new Vector2(0.5f, 0.5f)));
imageIndex++;
}
else
{
thereAreImages = false;
}
}
finished = true;
imageIndex = oldImgIndx;
}
The issue on the Hololens lies in the www.LoadImageIntoTexture(tex); line. This code is required when displaying multiple images on the PC side of things, however on the Hololens, where only one image is displayed at a time it can be left out.

Related

How can i update an image on runtime in unity?

I'm buidling a smooth transtion from a level scene to a menu scene.
The idea is to make a screenshot of the level just before the next scene(menu) loads.
then take this screenshot which overlays the entire screen and will fade out to reveal the menu.
At the end of a level I take a screen shot.
ScreenCapture.CaptureScreenshot("Resources/DestroyedBoss.png");
When I load the next scene this screenshot should load and overlay the entire scene.
public Texture2D myTexture;
void Start()
{
// load texture from resource folder
myTexture = Resources.Load("DestroyedBoss") as Texture2D;
GameObject rawImage = GameObject.Find("RawImage");
rawImage.GetComponent<RawImage>().texture = myTexture;
}
The screenshot will fade and the object containing it will be destroyed.
public RawImage imageToFade;
void Update()
{
imageToFade.color -= new Color(0, 0, 0, 0.2f * Time.deltaTime);
if (imageToFade.color.a < 0.1)
Destroy(this.gameObject);
}
This all works fine, except for loading of the screenshot itself.
It would seem that unity does not update the image while running the game. And always uses the screenshot from the previous time I ran the game.
I have been searching/googling and trying for 2 days now and I am at a loss.
How can I make this work? Is there a way to update the png file while running my game? Maybe create a temp container for the screenshot and set this to the RawImage?
Like derHugo commented. you can use Application.persistentDataPath to save your image.
ScreenCapture.CaptureScreenshot(Application.persistentDataPath+"DestroyedBoss.png");
For loading the image, you can use UnityWebRequestTexture.GetTexture.
IEnumerator SetImage()
{
yield return new WaitForSeconds(1);
using (UnityWebRequest uwr =
UnityWebRequestTexture.GetTexture(Application.persistentDataPath + "DestroyedBoss.png"))
{
yield return uwr.SendWebRequest();
if (uwr.isNetworkError || uwr.isHttpError)
{
Debug.Log(uwr.error);
}
else
{
// Get downloaded asset bundle
myTexture = DownloadHandlerTexture.GetContent(uwr);
GameObject rawImage = GameObject.Find("RawImage");
rawImage.GetComponent<RawImage>().texture = myTexture;
FadeOut.instance.StartFade();
}
}
}
It works ok, but it takes a few seconds for the screenshot to appear in the folder. So for me... it does not work great.
I'll be using another technique.
Instead of fading the screenshot while loading the menu. I'll stop the game, (Time.timescale = 0) and fade in a screenshot of the menu, then load the actual menu scene.
Thanks again derHugo.

Microsoft media encoder screen capture, save a specific part of the video being recorded

I'm coding a screen capture program in C# using windows media encoder. While the screen is being recorded i want to save the last 30sec of the current video being recorded in a separate video when a btn gets clicked. Is there a function in encoder that would allow me to do that?
I've attached the code that I'm currently using. Couldn't find any documentation on media encoder...
void startRecording() {
System.Drawing.Size workingArea = SystemInformation.WorkingArea.Size;
Rectangle captureRec = new Rectangle(0, 0, workingArea.Width -(workingArea.Width % 4), workingArea.Height - (workingArea.Height % 4));
job.CaptureRectangle = captureRec;
job.ShowFlashingBoundary = true;
job.ShowCountdown = true;
job.CaptureMouseCursor = true;
job.AddAudioDeviceSource(AudioDevices());
job.OutputPath = #"C:\Users\Moe36\Desktop\Screen Recorder";
job.Start();
}
void saveLast30Sec() {
//Save last 30sec of the currently recorded video as a separate video file.
}

Reversed bitmap when creating AVI container

I have a list of images in my program, and I am generating an AVI video from them. For that purpose I use avifilewrapper_src library that handles the creation of video.
The process of creating is:
Bitmap bitmap;
//load the first image
bitmap = (Bitmap)imageSequence[0];
//create a new AVI file
AviManager aviManager = new AviManager(paths.outputVideo, false);
//add a new video stream and one frame to the new file
VideoStream aviStream =
aviManager.AddVideoStream(true, (double)nud_picturePerSec.Value, bitmap);
if(chb_audio.Checked)
aviManager.AddAudioStream(paths.sampleAudio, 0);
int count = 0;
for (int n = 0; n < imageSequence.Count; n++) {
bitmap = (Bitmap)imageSequence[n];
aviStream.AddFrame(bitmap);
bitmap.Dispose();
count++;
}
aviManager.Close();
If I keep giving different images, it works fine. If I however, put two similar images, than the video shows second image upside down (left/right side is correct). By two similar images I mean creating second image and copying it from the first one.
I have a feeling that this is somehow related to streams, but I can't find why the images are inverted.
Well I didn't managed to find the cause of that behavior. But fliping it between each use does the correction well.
bitmap.RotateFlip(RotateFlipType.RotateNoneFlipY);

Monogame for Windows Phone 8: black textures after being deactivated

I've faced a strange issue in my monogame (3.1) windows phone 8 app. When app is deactivated and then activated all textures become black. This happened also after the lock screen (UserIdleDetectionMode is enabled).
I've checked GraphicsDevice.IsDisposed, GraphicsDevice.IsContentLost, GraphicsDevice.ResourcesLost but everything looks ok. I've implemented reload of all my textures on Activated and Unobscured events, but full texture reload takes too much time. In the same time on Marketplace I see monogame apps easily handling desactivate-activate. Moreover, the same app for windows phone 7 written on xna, restores very quickly. What do I do wrong with monogame?
My app is based on monogame WP8 template.
Update:
Just have found out that all textures which loaded via Content.Load(...) are restored very quickly. But all my textures are written by a hand: I load a file from TileContainer, unpack it, read its data with ImageTools, create Texture2D and set its pixels with loaded data. Jpeg files also are rendered to RenderTarget2D as BGR565 to consume space.
Moreover I widely use RenderTarget2D for rendering text labels with shadows, sprite runtime compositions and so on. So it looks like that Monogame just don't want to restore images loaded not by Content.Load.
Continue investigating...
I just got a response from Tom Spillman in the Monogame forums and apparently the Content.Load stuff is restored normally and other data needs to be reinitialized by the program. What you can do is hook up to GraphicsDevice.DeviceResetting event to get notified when this reset is taking place.
According to monogame devs lost textures is a normal situation.
I've made full texture reload in GraphicsDevice.DeviceReset event. To make it work fast I've implemented load from xnb uncompressed files. It's pretty simple as long as this format just have pixel values in it. This is the only solution.
Here's how to read from uncompressed xnb:
private static Texture2D TextureFromUncompressedXnbStream(GraphicsDevice graphicsDevice, Stream stream)
{
BinaryReader xnbReader = new BinaryReader(stream);
byte cx = xnbReader.ReadByte();
byte cn = xnbReader.ReadByte();
byte cb = xnbReader.ReadByte();
byte platform = xnbReader.ReadByte();
if (cx != 'X' || cn != 'N' || cb != 'B')
return null;
byte version = xnbReader.ReadByte();
byte flags = xnbReader.ReadByte();
bool compressed = (flags & 0x80) != 0;
if (version != 5 && version != 4)
return null;
int xnbLength = xnbReader.ReadInt32();
xnbReader.ReadBytes(0x9D);//skipping processor string
SurfaceFormat surfaceFormat = (SurfaceFormat)xnbReader.ReadInt32();
int width = (xnbReader.ReadInt32());
int height = (xnbReader.ReadInt32());
int levelCount = (xnbReader.ReadInt32());
int levelCountOutput = levelCount;
Texture2D texture = texture = new Texture2D(graphicsDevice, width, height, false, SurfaceFormat.Color);
for (int level = 0; level < levelCount; level++)
{
int levelDataSizeInBytes = (xnbReader.ReadInt32());
byte[] levelData = xnbReader.ReadBytes(levelDataSizeInBytes);
if (level >= levelCountOutput)
continue;
texture.SetData(level, null, levelData, 0, levelData.Length);
}
xnbReader.Dispose();
return texture;
}

Texture downloaded from server appears black on iOS

I am building an application in Unity3d, and I need to download textures from my server and apply them to prefabs.
I have two types of prefabs;
The first is a simple plane that I use to display 2d images, and the second is a prefab to play videos and have a thumbnail texture that is displayed before the video is played in full screen.
I am having problems with the video prefab. If I create a public texture in my script and apply it to the prefab, everything works fine. However, if I download the texture from my server and apply it to the prefab it appears black. This only happens in iOS, in the Unity Player everything appears fine.
Here is my code:
Instantiate the prefab:
newVideo = (GameObject)Instantiate(arvideo, new Vector3(15*i, 0, 0), Quaternion.identity);
newVideo.GetComponent<VideoPlaybackBehaviour>().m_path = ((Assets)Data.Assets[i]).AssetContent; // SET THE URL FOR THE VIDEO
string url = ((Assets)Data.Assets[i]).AssetThumbnail;
StartCoroutine(DownloadImage(url, newVideo, ((Assets)Data.Assets[i]).AssetFilename, "VIDEO"));
newVideo.transform.rotation = Quaternion.Euler(0, -180, 0);
Download IEnumerator:
public IEnumerator DownloadImage(string url, GameObject tex, string filename, string type)
{
WWW www = new WWW(url);
yield return www;
/* EDIT: */
if (!string.IsNullOrEmpty(www.error)){
Debug.LogWarning("LOCAL FILE ERROR: "+www.error);
} else if(www.texture == null) {
Debug.LogWarning("LOCAL FILE ERROR: TEXTURE NULL");
} else {
/* EOF EDIT */
tex.GetComponent<VideoPlaybackBehaviour>().KeyframeTexture = www.texture;
Color color = tex.renderer.material.color;
color.a = 1f;
tex.renderer.material.color = color;
}
}
I struggled with same issue. And after several hours search, for my case i solved this problem with disabling "Metal write-only BackBuffer" in iOS player settings. I'm using unity 2020.1.8f1 and actually i have no any idea about why back buffer setting is causing black texture problem in iOS devices.

Categories

Resources