Using ARCore for Unity, trying to save Frame.CameraImage.AcquireCameraImageBytes() as image and scanning the image for QR code. But the converted image is not in actual scale and it is repeating, so not able to deduct the QR code correctly.
Here is my code
void Update()
{
using (var image = Frame.CameraImage.AcquireCameraImageBytes())
{
if (image.IsAvailable)
{
byte[] m_EdgeImage = null;
Color32[] pixels = null;
IParser Parser = new ZXingParser();
if (_texture == null || m_EdgeImage == null || _texture.width != image.Width || _texture.height != image.Height)
{
_texture = new Texture2D(image.Width, image.Height, TextureFormat.RGBA32, false, false);
m_EdgeImage = new byte[image.Width * image.Height*4];
}
System.Runtime.InteropServices.Marshal.Copy(image.Y, m_EdgeImage, 0, image.Width * image.Height);
_texture.LoadRawTextureData(m_EdgeImage);
_texture.Apply();
ParserResult Result = Parser.Decode(pixels, _texture.width, _texture.height);
if (Result != null)
{
Debug.Log("QRCODE");
}
else
{
var encodedJpg = _texture.EncodeToJPG();
var path = Application.persistentDataPath;
File.WriteAllBytes(path + "/test.jpg", encodedJpg);
Debug.Log("NOQRCODE");
Application.Quit();
}
}
}
}
Here is the converted image
What is wrong here
An ARCore camera image with its data accessible from the CPU in YUV-420-888 formatCheck this. The buffer size is width*height*1.5 for YUV. You may need to convert YUV to RGB format.
Related
I made a script to download some images from the web on click page, and when I moved to the second page, a second set of images was downloaded, but the problem is that the image size is relatively large, and when there are more than one image, it consumes a lot of RAM, how can I save as much memory as possible by compressing the texture
this is my Code
IEnumerator D_Image(string url_)
{
Destroy(Icon.texture);
Icon.texture = LoadT;
using (UnityWebRequest uwr = UnityWebRequestTexture.GetTexture(url_))
{
yield return uwr.SendWebRequest();
if (uwr.result != UnityWebRequest.Result.Success)
{
Icon.texture = ErrorT;
}
else
{
Icon.texture = DownloadHandlerTexture.GetContent(uwr);
}
uwr.Dispose();
}
}
Try compressing the images, it makes a big difference
You can use this code
I found it in this TextureOps add-on
public static Texture2D Scale( Texture sourceTex, int targetWidth, int targetHeight, TextureFormat format = TextureFormat.RGBA32, Options options = new Options() )
{
if( sourceTex == null )
throw new ArgumentException( "Parameter 'sourceTex' is null!" );
Texture2D result = null;
RenderTexture rt = RenderTexture.GetTemporary( targetWidth, targetHeight );
RenderTexture activeRT = RenderTexture.active;
try
{
Graphics.Blit( sourceTex, rt );
RenderTexture.active = rt;
result = new Texture2D( targetWidth, targetHeight, format, options.generateMipmaps, options.linearColorSpace );
result.ReadPixels( new Rect( 0, 0, targetWidth, targetHeight ), 0, 0, false );
result.Apply( options.generateMipmaps, options.markNonReadable );
}
catch( Exception e )
{
Debug.LogException( e );
Object.Destroy( result );
result = null;
}
finally
{
RenderTexture.active = activeRT;
RenderTexture.ReleaseTemporary( rt );
}
return result;
}
I need to extract images from PDF.
I know that some images are rotated 90 degrees (I checked with online tools).
I'm using this code:
PdfRenderListener:
public class PdfRenderListener : IExtRenderListener
{
// other methods ...
public void RenderImage(ImageRenderInfo renderInfo)
{
try
{
var mtx = renderInfo.GetImageCTM();
var image = renderInfo.GetImage();
var fillColor = renderInfo.GetCurrentFillColor();
var color = Color.FromArgb(fillColor?.RGB ?? Color.Empty.ToArgb());
var fileType = image.GetFileType();
var extension = "." + fileType;
var bytes = image.GetImageAsBytes();
var height = mtx[Matrix.I22];
var width = mtx[Matrix.I11];
// rotated image
if (height == 0 && width == 0)
{
var h = Math.Abs(mtx[Matrix.I12]);
var w = Math.Abs(mtx[Matrix.I21]);
}
// save image
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
}
When I save images with this code the rotated images are saved with distortion.
I have read this post iText 7 ImageRenderInfo Matrix contains negative height on Even number Pages and mkl answer.
In current transfromation matrix (mtx) I have these values:
0
841.9
0
-595.1
0
0
595.1
0
1
I know image rotated 90 degrees. How can I transform an image to get a normal image?
As #mkl mentioned, the true reason was not in the rotation of the image, but with the applied filter.
I analyzed the pdf file with iText RUPS and found that the image was encoded with a CCITTFaxDecode filter:
RUPS screen
Next, I looked for ways to decode this filter and found these questions
Extracting image from PDF with /CCITTFaxDecode filter.
How to use Bit Miracle LibTiff.Net to write the image to a MemoryStream
I used the BitMiracle.LibTiff.NET library
I wrote this method:
private byte[] DecodeInternal(byte[] rawBytes, int width, int height, int k, int bitsPerComponent)
{
var compression = GetCompression(k);
using var ms = new MemoryStream();
var tms = new TiffStream();
using var tiff = Tiff.ClientOpen("in-memory", "w", ms, tms);
tiff.SetField(TiffTag.IMAGEWIDTH, width);
tiff.SetField(TiffTag.IMAGELENGTH, height);
tiff.SetField(TiffTag.COMPRESSION, compression);
tiff.SetField(TiffTag.BITSPERSAMPLE, bitsPerComponent);
tiff.SetField(TiffTag.SAMPLESPERPIXEL, 1);
var writeResult = tiff.WriteRawStrip(0, rawBytes, rawBytes.Length);
if (writeResult == -1)
{
Console.WriteLine("Decoding error");
}
tiff.CheckpointDirectory();
var decodedBytes = ms.ToArray();
tiff.Close();
return decodedBytes;
}
private Compression GetCompression(int k)
{
return k switch
{
< 0 => Compression.CCITTFAX4,
0 => Compression.CCITTFAX3,
_ => throw new NotImplementedException("K > 0"),
};
}
After decoding and rotating the image, I was able to save a normal image. Thanks everyone for the help.
You can try this. I'm using Itext 7 for java. Here you still need to write your own listener:
public class MyImageRenderListener implements IEventListener {
protected String path;
protected String extension;
public MyImageRenderListener (String path) {
this.path = path;
}
public void eventOccurred(IEventData data, EventType type) {
switch (type) {
case RENDER_IMAGE:
try {
String filename;
FileOutputStream os;
ImageRenderInfo renderInfo = (ImageRenderInfo) data;
PdfImageXObject image = renderInfo.getImage();
if (image == null) {
return;
}
byte[] imageByte = image.getImageBytes(true);
extension = image.identifyImageFileExtension();
filename = String.format(path, image.getPdfObject().getIndirectReference().getObjNumber(), extension);
os = new FileOutputStream(filename);
os.write(imageByte);
os.flush();
os.close();
} catch (com.itextpdf.io.exceptions.IOException | IOException e) {
System.out.println(e.getMessage());
}
break;
default:
break;
}
}
public Set<EventType> getSupportedEvents() {
return null;
}
}
I checked for a pdf with a random rotation angle, and 90 degrees, the resulting picture was obtained without distortion
public void manipulatePdf() throws IOException, SQLException, ParserConfigurationException, SAXException {
PdfDocument pdfDoc = new PdfDocument(new PdfReader("path to pdf"), new PdfWriter(new ByteArrayOutputStream()));
MyImageRenderListener listener = new MyImageRenderListener("path to resulting image");
PdfCanvasProcessor parser = new PdfCanvasProcessor(listener);
for (int i = 1; i <= pdfDoc.getNumberOfPages(); i++) {
parser.processPageContent(pdfDoc.getPage(i));
}
pdfDoc.close();
}
I want to upload an image jpg to my server. I have try but it didn't work.
I don't what cause the upload failed.
Below the detail code.
Unity C#
public void TakePicture(int maxSize)
{
if (NativeCamera.IsCameraBusy())
{
Debug.Log("Camera Busy");
return;
}
else
{
NativeCamera.Permission permission = NativeCamera.TakePicture((path) =>
{
Debug.Log("Image path: " + path);
if (path != null)
{
// Create a Texture2D from the captured image
Texture2D texture = NativeCamera.LoadImageAtPath(path, maxSize);
//snap.SetPixels(tex.GetPixels());
byte[] bytes = File.ReadAllBytes(path);
Debug.Log("Bytes:" + bytes);
Destroy(texture);
StartCoroutine(upload_ocr_image(bytes));
if (texture == null)
{
Debug.Log("Couldn't load texture from " + path);
return;
}
// Assign texture to a temporary quad and destroy it after 5 seconds
GameObject quad = GameObject.CreatePrimitive(PrimitiveType.Quad);
quad.transform.position = Camera.main.transform.position + Camera.main.transform.forward * 2.5f;
quad.transform.forward = Camera.main.transform.forward;
quad.transform.localScale = new Vector3(1f, texture.height / (float)texture.width, 1f);
Material material = quad.GetComponent<Renderer>().material;
if (!material.shader.isSupported) // happens when Standard shader is not included in the build
material.shader = Shader.Find("Legacy Shaders/Diffuse");
material.mainTexture = texture;
Destroy(quad, 5f);
// If a procedural texture is not destroyed manually,
// it will only be freed after a scene change
Destroy(texture, 5f);
}
}, maxSize);
Debug.Log("Permission result: " + permission);
}
}
IEnumerator upload_ocr_image(byte[] bytes)
{
// UtilityScript.GetComponent<utility>().Sand_Clock_Loading(true);
// yield return new WaitForSeconds(2.5f);
WWWForm formDate = new WWWForm();
formDate.AddField("frameCount", Time.frameCount.ToString());
formDate.AddBinaryData("ocr_image", bytes, "ocr.jpg", "image/jpg");
formDate.AddField("phone_code", "+61");
formDate.AddField("phone_number", "434599859");
using (UnityWebRequest www = UnityWebRequest.Post(web_url.url_upload_ocr_image, formDate))
{
yield return www.Send();
//UtilityScript.GetComponent<utility>().Sand_Clock_Loading(false);
if (www.isNetworkError)
{
Debug.Log(www.error);
// UtilityScript.GetComponent<utility>().MessageBox_Check_Connection();
}
else
{
Debug.Log(www.downloadHandler.text);
}
}
}
And this the code in server codeigniter php.
function upload_ocr_image()
{
$phone_code = $this->input->post('phone_code', true);
$phone_number = $this->input->post('phone_number', true);
$allowedType = array(IMAGETYPE_GIF,IMAGETYPE_JPEG,IMAGETYPE_PNG);
$imgType = exif_imagetype($_FILES['ocr_image']['tmp_name']);
if(!in_array($imgType,$allowedType))
{
echo "Images Type Error. Images Type Only : GIF , JPEG, PNG";
exit;
}
else
{
//upload original size front end slider
$config['upload_path'] = './assets/ocr_image/';
$config['allowed_types'] = 'gif|jpg|png|jpeg';
$config['file_name'] = $phone_code.$phone_number;
$config['overwrite'] = FALSE;
$config['max_size'] = '8096';
$config['max_width'] = '6000';
$config['max_height'] = '6000';
$this->load->library('upload', $config);
if(!$this->upload->do_upload("ocr_image"))
{
echo "Maximum File Size Only 2 Mb Or Max Width = 2000 , Height = 2000";
exit;
}
else
{
$img_data = $this->upload->data();
// Create Thumbnail
/*
$magicianObj = new imageLib("assets/ocr_image/".$img_data["file_name"].$phone_code.$phone_number);
$magicianObj -> resizeImage(80, 80, 0, true);
$magicianObj -> saveImage("assets/admin/img/photos/".$img_data["raw_name"]."_thumb".$img_data["file_ext"], 100);
$next_id = $next_id.$img_data["file_ext"];
$thumb_name = $img_data["raw_name"]."_thumb".$img_data["file_ext"];
$data = array("photo_name"=>$next_id,
"thumb_photo_name"=>$thumb_name,
"create_date"=>date("Y-m-d H:i:s"));
$this->db->insert("gallery_tbl",$data);
*/
}
}
}
No error found. But it always failed.
This line of code in codeigniter PHP :
if(!$this->upload->do_upload("ocr_image"))
Always return true.
How it work ?, How to upload the picture with proper way ?
Edited :
I use an asset native camera android and ios from unity to take a picture from camera.
Thank You
simply add initialize $this->upload->initialize($config); after
$this->load->library('upload', $config); and check, also check upload directory path is right ? and directory permission
Finally i found the problem.
Just set allowed type extension to except all extension in codeigniter upload file.
change :
$config['allowed_types'] = 'gif|jpg|png|jpeg';
to
$config['allowed_types'] = '*';
My program has a problem with poor coordination between the depth and color images.
The player mask is not in the same place as the person (see the picture below).
void _AllFreamReady(object sender, AllFramesReadyEventArgs e)
{
using (ColorImageFrame _colorFrame = e.OpenColorImageFrame())
{
if (_colorFrame == null) //jezeli pusta ramka nie rob nic
{
return;
}
byte[] _pixels = new byte[_colorFrame.PixelDataLength]; //utworzenie tablicy pixeli dla 1 ramki obrazu o rozmiarach przechwyconej ramki z strumienia
_colorFrame.CopyPixelDataTo(_pixels); //kopiujemy pixele do tablicy
int _stride = _colorFrame.Width * 4; //Kazdy pixel moze miec 4 wartosci Red Green Blue lub pusty
image1.Source =
BitmapSource.Create(_colorFrame.Width, _colorFrame.Height,
96, 96, PixelFormats.Bgr32, null, _pixels, _stride);
if (_closing)
{
return;
}
using (DepthImageFrame _depthFrame = e.OpenDepthImageFrame())
{
if (_depthFrame == null)
{
return;
}
byte[] _pixelsdepth = _GenerateColoredBytes(_depthFrame,_pixels);
int _dstride = _depthFrame.Width * 4;
image3.Source =
BitmapSource.Create(_depthFrame.Width, _depthFrame.Height,
96, 96, PixelFormats.Bgr32, null, _pixelsdepth, _dstride);
}
}
}
private byte[] _GenerateColoredBytes(DepthImageFrame _depthFrame, byte[] _pixels)
{
short[] _rawDepthData = new short[_depthFrame.PixelDataLength];
_depthFrame.CopyPixelDataTo(_rawDepthData);
Byte[] _dpixels = new byte[_depthFrame.Height * _depthFrame.Width * 4];
const int _blueindex = 0;
const int _greenindex = 1;
const int _redindex = 2;
for (int _depthindex = 0, _colorindex = 0;
_depthindex < _rawDepthData.Length && _colorindex < _dpixels.Length;
_depthindex++, _colorindex += 4)
{
int _player = _rawDepthData[_depthindex] & DepthImageFrame.PlayerIndexBitmaskWidth;
if (_player > 0)
{
_dpixels[_colorindex + _redindex] = _pixels[_colorindex + _redindex];
_dpixels[_colorindex + _greenindex] = _pixels[_colorindex + _greenindex];
_dpixels[_colorindex + _blueindex] = _pixels[_colorindex + _blueindex];
};
}
return _dpixels;
}
RGB and depth data are not aligned. This is due to the position of depth sensor and RGB camera in the Kinect case: they are different, so you cannot expect aligned images using different points of view.
However, you problem is quite common, and was solved by the KinectSensor.MapDepthFrameToColorFrame, that was deprecated after SDK 1.6. Now, what you need is the CoordinateMapper.MapDepthFrameToColorFrame method.
The Coordinate Mapping Basics-WPF C# Sample shows how to use this method. You can find some significant parts of the code in the following:
// Intermediate storage for the depth data received from the sensor
private DepthImagePixel[] depthPixels;
// Intermediate storage for the color data received from the camera
private byte[] colorPixels;
// Intermediate storage for the depth to color mapping
private ColorImagePoint[] colorCoordinates;
// Inverse scaling factor between color and depth
private int colorToDepthDivisor;
// Format we will use for the depth stream
private const DepthImageFormat DepthFormat = DepthImageFormat.Resolution320x240Fps30;
// Format we will use for the color stream
private const ColorImageFormat ColorFormat = ColorImageFormat.RgbResolution640x480Fps30;
//...
// Initialization
this.colorCoordinates = new ColorImagePoint[this.sensor.DepthStream.FramePixelDataLength];
this.depthWidth = this.sensor.DepthStream.FrameWidth;
this.depthHeight = this.sensor.DepthStream.FrameHeight;
int colorWidth = this.sensor.ColorStream.FrameWidth;
int colorHeight = this.sensor.ColorStream.FrameHeight;
this.colorToDepthDivisor = colorWidth / this.depthWidth;
this.sensor.AllFramesReady += this.SensorAllFramesReady;
//...
private void SensorAllFramesReady(object sender, AllFramesReadyEventArgs e)
{
// in the middle of shutting down, so nothing to do
if (null == this.sensor)
{
return;
}
bool depthReceived = false;
bool colorReceived = false;
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
if (null != depthFrame)
{
// Copy the pixel data from the image to a temporary array
depthFrame.CopyDepthImagePixelDataTo(this.depthPixels);
depthReceived = true;
}
}
using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
{
if (null != colorFrame)
{
// Copy the pixel data from the image to a temporary array
colorFrame.CopyPixelDataTo(this.colorPixels);
colorReceived = true;
}
}
if (true == depthReceived)
{
this.sensor.CoordinateMapper.MapDepthFrameToColorFrame(
DepthFormat,
this.depthPixels,
ColorFormat,
this.colorCoordinates);
// ...
int depthIndex = x + (y * this.depthWidth);
DepthImagePixel depthPixel = this.depthPixels[depthIndex];
// scale color coordinates to depth resolution
int X = colorImagePoint.X / this.colorToDepthDivisor;
int Y = colorImagePoint.Y / this.colorToDepthDivisor;
// depthPixel is the depth for the (X,Y) pixel in the color frame
}
}
I am working on this problem myself. I agree with VitoShadow that one solution is in the coordinate mapping, but a section not posted where the ratio between the miss matched depth and color screen resolutions(this.colorToDepthDivisor = colorWidth / this.depthWidth;). This is used with a shift of the data (this.playerPixelData[playerPixelIndex - 1] = opaquePixelValue;) to account for the miss match.
Unfortunately, this can create a border around the masked image where the depthframe isn't stretched to the edge of the color frame. I am trying to not use skeleton mapping and am optimizing my code by tracking depthdata with emgu cv to pass a point as the center of the ROI of the colorframe. I am still working on it.
I am importing a picture into my application and rotating the image according to the EXIF metadata. After this I am saving the rotated image to my disk, but as I have still left the rotated image metadata on the image and windows thinks it should rotate it again... basically meaning my image ends up upside down.
So far I have got:
using (Stream sourceStream = File.Open(dlg.FileName, FileMode.Open, FileAccess.Read))
{
BitmapDecoder sourceDecoder = BitmapDecoder.Create(sourceStream, createOptions, BitmapCacheOption.OnLoad);
// Check source is has valid frames
if (sourceDecoder.Frames[0] != null && sourceDecoder.Frames[0].Metadata != null)
{
sourceDecoder.Frames[0].Metadata.Freeze();
// Get a clone copy of the metadata
BitmapMetadata sourceMetadata = sourceDecoder.Frames[0].Metadata.Clone() as BitmapMetadata;
ImportedPhotoMetaData = sourceMetadata;
}
}
and
using (var image = Image.FromFile(dlg.FileName))
{
foreach (var prop in image.PropertyItems)
{
if (prop.Id == 0x112)
{
if (prop.Value[0] == 6)
rotate = 90;
if (prop.Value[0] == 8)
rotate = -90;
if (prop.Value[0] == 3)
rotate = 180;
prop.Value[0] = 1;
}
}
}
but the prop.Value[0] = 1; line does not seem to reset the image metadata. I need to reset the image orientation on the ImportedPhotoMetaData property
Got it...
Replace
prop.Value[0] = 1;
with
ImportedPhotoMetaData.SetQuery("System.Photo.Orientation", 1);