I have an Azure function that has been working all day. About twenty minutes ago I started getting an error when trying to click the run button in Visual Studio. It successfully builds and then when it starts it displays and error the just states an 'illegal character in path - Visual Studio'. I've checked the git logs and no changes have been made so I'm unsure what's wrong.
I've attached a picture of the problem to try illustrate the issue:
I have tried everything I can think of. Here's a list of all the steps I've taken.
Restart machine
Re-cloned the repo
Re-installed Visual Studio 2017 & 2019
Deleted the appdata folder
Cleared Cache files
Browsed the debug menus
System restored
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Zupa.Products.ProductsService.Models.Messages.V1;
namespace Zupa.ProductFileUploadFunction
{
public static class FileUploadProcessor
{
private const string FunctionName = nameof(FileUploadProcessor);
[FunctionName(FunctionName)]
public static void Run(
[BlobTrigger("%IncomingContainerPath%/{blobFileName}", Connection = "AzureWebJobsStorage")]Stream inputBlob,
[ServiceBus("%QueueName%", Connection = "QueueServiceBus")]out string outgoingMessage,
string blobFileName, ILogger log)
{
log.LogInformation($"{FunctionName} Processing blob {Environment.NewLine} Name:{blobFileName + Environment.NewLine} Size: {inputBlob.Length} Bytes");
outgoingMessage = null;
var (parsedOrganisationId, parsedUploadId) = (Guid.Empty, Guid.Empty);
var organisationId = GetPartFromString(blobFileName, "/", 0);
var fileUploadType = GetPartFromString(blobFileName, "/", 1);
var isRecognisedFileUploadType = Enum.GetNames(typeof(FileUploadType)).Select(name => name.ToLowerInvariant()).Contains(fileUploadType.ToLowerInvariant());
var uploadId = GetPartFromString(blobFileName, "/", 2);
var fileName = GetPartFromString(blobFileName, "/", 3);
var propertyValidation = new Dictionary<string, bool>()
{
{
nameof(FileUploadDataEventMessage.OrganisationId),
string.IsNullOrEmpty(organisationId) || !Guid.TryParse(organisationId, out parsedOrganisationId)
},
{
nameof(FileUploadDataEventMessage.FileUploadType),
string.IsNullOrEmpty(fileUploadType) || isRecognisedFileUploadType
},
{
nameof(FileUploadDataEventMessage.UploadId),
string.IsNullOrEmpty(uploadId) || !Guid.TryParse(uploadId, out parsedUploadId)
},
{
nameof(FileUploadDataEventMessage.FileName),
string.IsNullOrEmpty(fileName)
}
};
foreach (KeyValuePair<string, bool> propertyValidationPair in propertyValidation)
LogPropertyInErrorState(log, propertyValidationPair.Value, blobFileName, propertyValidationPair.Key);
var hasInvalidPathParameters = parsedOrganisationId == Guid.Empty || !isRecognisedFileUploadType || parsedUploadId == Guid.Empty || string.IsNullOrEmpty(fileName);
outgoingMessage = !hasInvalidPathParameters ? JsonConvert.SerializeObject(new FileUploadDataEventMessage()
{
OrganisationId = parsedOrganisationId,
UploadId = parsedUploadId,
FileUploadType = Enum.Parse<FileUploadType>(PascalCaseWord(fileUploadType)),
FileName = fileName,
Timestamp = DateTimeOffset.Now
}) : null;
log.LogInformation($"{FunctionName} Processed blob {Environment.NewLine} Name:{blobFileName + Environment.NewLine} Size: {inputBlob.Length} Bytes");
}
private static string GetPartFromString(string inputString, string delimiter, int targetIndex)
{
var inputStringParts = inputString.Split(delimiter);
if (inputStringParts.ElementAtOrDefault(targetIndex) != null)
return inputStringParts[targetIndex];
return null;
}
private static void LogPropertyInErrorState(ILogger log, bool errorValidationCondition, string fileName, string propertyName)
{
var errorMessage = errorValidationCondition ?
$"{FunctionName} Failed to process blob {Environment.NewLine} Name: {fileName + Environment.NewLine} Missing or invalid {propertyName}" : string.Empty;
if (!string.IsNullOrEmpty(errorMessage))
log.LogInformation(errorMessage);
}
private static string PascalCaseWord(string input) =>
input.Substring(0, 1).ToUpperInvariant() + input.Substring(1);
}
}
It doesn't seem to be the issue with the function as the rest of the team doesn't seem to experience the same issue and can successfully run the function. So I'm assuming its a system issue.
For anyone experiencing this issue I have finally found a way to fix it
Thank you so much Tsuyoshi Ushio over on medium, follow the steps below to resolve the issue seems to be when the azure function tools files get in a bad state within the %appdata%/local folder
https://medium.com/#tsuyoshiushio/visual-studio-2017-2019-fails-when-i-create-an-azure-functions-project-89e993ef31f
You need to delete the following folder:
%localappdata%\AzureFunctionsTools
Related
I need to get (not download) the content from 10.000~ manifest files within a project in Azure DevOps, but I don't manage to achieve this. I have found several ways to retrieve the content from one file at a time, but in this context, it is neither an efficient nor sustainable solution. I have managed to retrieve all files of a particular file type by checking if the file path ends with the name of the file, then using the TfvcHttpClientBase.GetItemsBatch method. However, this method does not return the item's content.
Program.cs
using Microsoft.TeamFoundation.SourceControl.WebApi;
AzureRest azureRest = new AzureRest();
var tfvcItems = azureRest.GetTfvcItems();
List<TfvcItemDescriptor> itemDescriptorsList = new List<TfvcItemDescriptor>();
foreach(var item in tfvcItems)
{
//Example manifest file .NET
if (item.Path.EndsWith("packages.config"))
{
var itemDescriptor = new TfvcItemDescriptor()
{
Path = item.Path,
RecursionLevel = VersionControlRecursionType.None,
Version = "",
VersionOption = TfvcVersionOption.None,
VersionType = TfvcVersionType.Latest
};
itemDescriptorsList.Add(itemDescriptor);
}
}
TfvcItemDescriptor[] itemDescriptorsArray = itemDescriptorsList.ToArray();
var itemBatch = azureRest.GetTfvcItemsBatch(itemDescriptorsArray);
foreach(var itemList in itemBatch)
{
foreach(var itemListList in itemList)
{
Console.WriteLine("Content: " + itemListList.Content); //empty/null
Console.WriteLine("ContentMetadata: " + itemListList.ContentMetadata); //not empty/null
}
}
AzureRest.cs
using Microsoft.TeamFoundation.SourceControl.WebApi;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
public class AzureRest
{
const string ORG_URL = "https://org/url/url";
const string PROJECT = "Project";
const string PAT = "PersonalAccessToken";
private string GetTokenConfig()
{
return PAT;
}
private string GetProjectNameConfig()
{
return PROJECT;
}
private VssConnection Authenticate()
{
string token = GetTokenConfig();
string projectName = GetProjectNameConfig();
var credentials = new VssBasicCredential(string.Empty, token);
var connection = new VssConnection(new Uri(ORG_URL), credentials);
return connection;
}
public List<TfvcItem> GetTfvcItems()
{
var connection = Authenticate();
using (TfvcHttpClient tfvcClient = connection.GetClient<TfvcHttpClient>())
{
var tfvcItems = tfvcClient.GetItemsAsync(scopePath: "/Path", recursionLevel: VersionControlRecursionType.Full, true).Result;
return tfvcItems;
}
}
public List<List<TfvcItem>> GetTfvcItemsBatch(TfvcItemDescriptor[] itemDescriptors)
{
TfvcItemRequestData requestData = new TfvcItemRequestData()
{
IncludeContentMetadata = true,
IncludeLinks = true,
ItemDescriptors = itemDescriptors
};
var connection = Authenticate();
using (TfvcHttpClient tfvcClient = connection.GetClient<TfvcHttpClient>())
{
var tfvcItems = tfvcClient.GetItemsBatchAsync(requestData).Result;
return tfvcItems;
}
}
}
}
For reference:
I have tested the codes you shared and when debugging at "itemDescriptorsList" and have found that there is no content specified in it, so that's why you cannot get the txt content.
You should first check and add the content property into the "itemDescriptorsList".
I am trying to save data locally to my device app folder.
When I try to save collected data on an actual Android smartphone, it doesn't work. It is limited by name and filetype, as I cannot change it from test.txt and it is limited in string length, as a maximum of twelve characters get saved.
I have the acquired the following permissions:
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
This is my code:
MainPage.xaml.cs
private async void ExportData(object sender, EventArgs e)
{
var items = await App.Database.GetDataAsync();
DependencyService.Get<IFileService>().CreateFile(items);
}
Interface
using System;
using System.Collections.Generic;
using System.Text;
namespace LocationApp.Interface
{
public interface IFileService
{
void CreateFile(List<LocationData> items);
}
}
Service
using Android.App;
using LocationApp.Droid;
using LocationApp.Interface;
using System.Collections.Generic;
using System.IO;
[assembly:Xamarin.Forms.Dependency(typeof(FileService))]
namespace LocationApp.Droid
{
public class FileService : IFileService
{
public string GetRootPath()
{
return Application.Context.GetExternalFilesDir(null).ToString();
}
public void CreateFile(List<LocationData> items)
{
var fileName = "test-file.txt";
var destination = Path.Combine(GetRootPath(), fileName);
string[] text = new string[items.Count];
for (int i = 0; i < text.Length; i++)
{
text[i] = $"{items[i].Latitude},{items[i].Longitude},{items[i].Day},{items[i].Time}";
}
File.WriteAllLines(destination, text);
}
}
}
I also attempted to see what would happen to an emulator, I used a Pixel 2 with Android 9.0, API 28 where I got the following error:
[ContextImpl] Failed to ensure /storage/120E-0B1B/Android/data/com.companyname.locationapp/files: java.lang.IllegalStateException: Failed to prepare /storage/120E-0B1B/Android/data/com.companyname.locationapp/files/: android.os.ServiceSpecificException: (code -13)
In the end, I only care about putting all my data in a single file. The filename or the error on my emulator I provided in case the error is based on that. If not, I do not care if they are fixed/fixable.
Based on your code, I created a simple demo, and it works on my android emulator(android 11) .
You can test on your side.
The code is:
public void CreateFile(List<LocationData> items)
{
var fileName = "test-file.txt";
var documentsPath = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
var destination = Path.Combine(documentsPath, fileName);
string[] text = new string[items.Count];
for (int i = 0; i < text.Length; i++)
{
text[i] = $"{items[i].Latitude},{items[i].Longitude}";
}
File.WriteAllLines(destination, text);
}
And after I saved the data,I could get the saved data by the following code(the filename is test-file.txt):
public string ReadData(string filename)
{
var documentsPath = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
var filePath = Path.Combine(documentsPath, filename);
return File.ReadAllText(filePath);
}
I have an application that is written in C# so that it can be a stand-alone .exe with parameters passed in via cmd at runtime. I am trying to update a file in google drive without changing the fileid so that any links using that id will not be broken. The following code works until it gets to the third section using Google.Apis.Drive.v3.Data.File updateFile in which the file is always 'null'. The request.ResponseBody has the correct fileId, the correct path and all but always comes up null.
How can I get this to update a file?
Is there some code update/change that no longer works?
Any help would be appreciated-
Section 1- This is the start of the code that calls the next two sections:
// System Library Requirements
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Collections.ObjectModel;
using System.Runtime.Serialization;
using System.Threading;
using System.Threading.Tasks;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Security;
// Microsoft Library Requirements
using Microsoft.Win32;
// Google API Library Requirements
using Google.Apis.Gmail.v1;
using Google.Apis.Gmail.v1.Data;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Drive.v3;
using GoogleV2 = Google.Apis.Drive.v2;
using Google.Apis.Drive.v3.Data;
using GoogleV2data = Google.Apis.Drive.v2.Data;
using Google.Apis.Services;
using Google.Apis.Util.Store;
using Google.Apis.Requests;
using Google.Apis.Auth.OAuth2.Responses;
using Google.Api.Gax;
using Google.Cloud.Channel.V1;
// JSON reading/serializing/De-serializing Library Requirements
using Newtonsoft.Json;
// Mime-Type Library Requirements
using MimeKit;
using Google.Apis.Upload;
// CyberArk Library Requirements
using CyberArk.AIM.NetPasswordSDK;
using CyberArk.AIM.NetPasswordSDK.Exceptions;
static class Globals
{
//cyberark values will change based on the domain and the app and the safe
public static string google_client_secret;
public static string cyberark_client_id = "...xxxx.....apps.googleusercontent.com";
public static string cyberark_object = "...xxx...";
public static string cyberark_app_id = "...xxx...";
public static string cyberark_safe = "...xxx...";
//google app values will change based on the domain and gcp project information e.g. np vs prod
public static string google_app_name = "my_google_app_name";
public static string google_client_id = "my_google_client_id";
public static string google_data_store = "my_google_data_store";
}
class Program
{
static void Main(string[] args)
{
Globals.google_client_secret = NetPasswordSDK.CyberArkProgram.GetMyPassword(NetPasswordSDK.MyCredentials.MY_APP_ID, NetPasswordSDK.MyCredentials.MY_SAFE, NetPasswordSDK.MyCredentials.MY_OBJECT, NetPasswordSDK.MyCredentials.MY_SECURE_STRING);
var paramFolderID = "";
var paramDriveFromFolderName = "";
var paramUserID = "";
var paramTeamID = "";
var paramFileName = "";
var paramFilePath = "";
var paramLocalFolderPath = "";
if (args == null || args.Length == 0)
{
Console.WriteLine("No Arguments passed");
}
else
{
paramFolderID = args[0];
paramDriveFromFolderName = args[1];
paramUserID = args[2];
paramTeamID = args[3];
paramFileName = args[4];
paramFilePath = args[5];
paramLocalFolderPath = args[6];
Console.WriteLine("FolderID " + args[0]);
Console.WriteLine("DriveFromFolderName " + args[1]);
Console.WriteLine("UserID " + args[2]);
Console.WriteLine("TeamID " + args[3]);
Console.WriteLine("FileName " + args[4]);
Console.WriteLine("FilePath " + args[5]);
Console.WriteLine("LocalFolderPath " + args[6]);
}
string googleUpdateText = "";
string googleUpdateFilePath = "";
string googleUpdateLocalFolderPath = "";
string googleUpdateFileName = "";
string googleUpdateParentFolder = "";
string googleUpdateTeamDriveID = "";
string googleUpdateParentFolderID = "";
var proxy = WebRequest.DefaultWebProxy;
proxy.Credentials = CredentialCache.DefaultCredentials;
var httpClientHandler = new HttpClientHandler()
{
Proxy = proxy
};
{
{
//googleUpdateParentFolderID = paramFolderID;
//googleUpdateParentFolder = paramDriveFromFolderName;
//googleUpdateTeamDriveID = paramTeamID;
//googleUpdateFileName = paramFileName;
//googleUpdateFilePath = paramFilePath;
//googleUpdateLocalFolderPath = paramLocalFolderPath;
// HC values for testing
googleUpdateFilePath = "C:\\Path\\to_update_file\\update_file.txt";
googleUpdateLocalFolderPath = "C:\\Path\\to_update_file\\";
googleUpdateFileName = "update_file.txt";
googleUpdateParentFolder = "Folder_on_Google_Drive";
googleUpdateTeamDriveID = "Team_Drive_ID";
googleUpdateParentFolderID = "Parent_Folder_ID";
Console.WriteLine("value of googleUploadFilePath: " + googleUpdateFilePath);
Console.WriteLine("value of googleUploadLocalFolderPath: " + googleUpdateLocalFolderPath);
Console.WriteLine("value of googleUploadFileName: " + googleUpdateFileName);
Console.WriteLine("value of googleUploadParentFolder: " + googleUpdateParentFolder);
Console.WriteLine("value of googleUploadTeamDriveID: " + googleUpdateTeamDriveID);
Console.WriteLine("value of googleUploadParentFolderID: " + googleUpdateParentFolderID);
googleUpdateText = (JsonConvert.SerializeObject(GoogleAPI.GoogleAPI_Drive.GDrive_Update_File(googleUpdateFilePath, googleUpdateLocalFolderPath, googleUpdateFileName, "no description", googleUpdateParentFolder, paramUserID, "", "", googleUpdateTeamDriveID, googleUpdateParentFolderID), Formatting.Indented));
Console.WriteLine("value of googleUpdateText: " + googleUpdateText);
Console.WriteLine("this is the end of the update text");
};
// end
}
// end of Main
}
// end of program class
}
Section 2- This is the middle part of the code that updates files in a list of files which calls each file to update it.
public static String GDrive_Update_File(string pathToFile, string localFilePath, string fileName, string description, string parentFolder = "", string userid = "", string sharedUserIds = "", string sharerole = "", string teamID = "", string parentFolderID = "")
{
FileResponseJSON fileJson = new FileResponseJSON();
List<string> fileArray = new List<string> { };
IList<FileResponseJSON> filesJSON = new List<FileResponseJSON>();
IList<Google.Apis.Drive.v3.Data.File> files = new List<Google.Apis.Drive.v3.Data.File>();
List<string> parents = new List<String> { };
List<string> kids = new List<String> { };
UserCredential credential = GoogleAPI_Functions.GetCredentials(userid);
var service = new DriveService(new BaseClientService.Initializer
{
ApplicationName = APP_NAME,
HttpClientInitializer = credential,
});
if (!String.IsNullOrEmpty(parentFolder))
{
FilesResource.ListRequest listRequest = service.Files.List();
listRequest.Q = String.Format("(name contains '{0}') and (mimeType = 'application/vnd.google-apps.folder')", parentFolder);
listRequest.Fields = "files(*)";
if (teamID.Length > 3)
{
listRequest.SupportsTeamDrives = true; // new
listRequest.Corpora = "teamDrive"; // new
listRequest.IncludeTeamDriveItems = true;//new
listRequest.TeamDriveId = teamID;//new
}
// List files.
IList<Google.Apis.Drive.v3.Data.File> folders = listRequest.Execute().Files;
foreach (var fldr in folders)
{
if (parentFolderID.Length > 3)
{
if (fldr.Id.Equals(parentFolderID))
{
parents.Add(fldr.Id);
}
}
else
{
parents.Add(fldr.Id);
}
Console.WriteLine("value of fldr.Id " + fldr.Id);
}
}
// Upload File MetaData
// mimeType is important and so it the 'trashed' true/false or it won't be found
FilesResource.ListRequest fileRequest = service.Files.List();
string query = "name contains 'update_file.txt' and (mimeType = 'text/plain') and trashed = false";
fileRequest.Q = String.Format(query);
fileRequest.Fields = "files(*)";
fileRequest.SupportsTeamDrives = true; // new
fileRequest.Corpora = "teamDrive"; // new
fileRequest.IncludeTeamDriveItems = true;//new
fileRequest.TeamDriveId = teamID;//new
IList<Google.Apis.Drive.v3.Data.File> myFiles = fileRequest.Execute().Files;
Console.WriteLine("Files:");
if (myFiles != null && myFiles.Count > 0)
{
foreach (var myf in myFiles)
{
myf.Name = localFilePath + myf.Name;
Console.WriteLine("{0} ({1})", myf.Name, myf.Id);
UpdateFile(service, myf.Name, myf.Id);
}
}
else
{
Console.WriteLine("No files found.");
}
Console.Read();
var UpdatedID = "";
return UpdatedID;
}
Section 3- This section is supposed to update each file --> THIS IS THE SECTION WITH THE PROBLEM (The Google.Apis.Drive.v3.Data.File updatedFile always returns null)
public static Google.Apis.Drive.v3.Data.File UpdateFile(DriveService _service, string _uploadFile, string _fileId)
{
Google.Apis.Drive.v3.Data.File file = new Google.Apis.Drive.v3.Data.File();
byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
System.IO.MemoryStream stream = new System.IO.MemoryStream(byteArray);
FilesResource.UpdateMediaUpload request = _service.Files.Update(file, _fileId, stream, GetMimeType(_uploadFile));
void Upload_ProgressChanged(IUploadProgress progress) =>
Console.WriteLine(progress.Status + " " + progress.BytesSent);
void Upload_ResponseReceived(Google.Apis.Drive.v3.Data.File myfile) =>
Console.WriteLine(myfile.Name + " was uploaded successfully");
request.ProgressChanged += Upload_ProgressChanged;
request.ResponseReceived += Upload_ResponseReceived;
request.Upload();
Google.Apis.Drive.v3.Data.File updatedFile = request.ResponseBody;
return updatedFile;
}
update: I used IUploadProgress.Exception and got the following error-
Exception: The service drive has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError
File not found: 1XYZ..... [404]
Errors [
Message[File not found: 1XYZ.......] Location[fileId - parameter] Reason[notFound] Domain[global]
]
at Google.Apis.Upload.ResumableUpload.<HandleResponse>d__78.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Upload.ResumableUpload.<SendNextChunkAsync>d__77.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Upload.ResumableUpload.<UploadCoreAsync>d__74.MoveNext() 0
Update again, I was able to see the uri that was being generated and copy/paste it into the browser, and when I did I got a 405 Method Not Allowed; additionally when I tried it without the https and just used http I got a 403 Forbidden and that SSL is required to perform this operation
This is the uri without the ids
https://www.googleapis.com/upload/drive/v3/files/[file_id]?uploadType=resumable&upload_id=[upload_id]
Update:
During the process the api produces these two urls that both work and allow me access
WebContentLink
"https://drive.google.com/uc?id=[docID]&export=download"
WebViewLink
"https://drive.google.com/file/d/[docID]/view?usp=drivesdk"
Latest update applied fixes from DalmTo but am getting new errors-
// - removed // Google.Apis.Drive.v3.Data.File file = new
// - added //
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Id = _fileId
};
var fsSource = new MemoryStream(Encoding.UTF8.GetBytes(_uploadFile ?? ""));
var request = _service.Files.Update(fileMetadata, fileMetadata.Id, fsSource, GetMimeType(_uploadFile));
New error:
Exception: The service drive has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError
The resource body includes fields which are not directly writable. [403]
Errors [
Message[The resource body includes fields which are not directly writable.] Location[ - ] Reason[fieldNotWritable] Domain[global]
]
Not sure what would cause this.
File id is writeable so you should be able to send it with the file metadata in your request.
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Id = "xxx"
};
var fsSource = new MemoryStream(Encoding.UTF8.GetBytes(uploadString ?? ""));
var request = service.Files.Update(fileMetadata, fileMetadata.Id , fsSource, GetMimeType(uploadFilePath));
Update
From comments you are sending an empty metadata file object with your request.
Google.Apis.Drive.v3.Data.File file = new Google.Apis.Drive.v3.Data.File();
There for Drive is always going to give you a new file id. I am sending the original file id as part of the metadata there for it will use the file id.
I'm using SyslogNet library to send SIEM Syslog message from my app to 3rd party vendors SIEM system in LEEF format (in this it's QRadar).
However, I the message doesn't get sent and nowhere the procedure produce any exception.
I can see and confirm that the string correlates to the correct template/format according to Rfc5424 for LEEF messages (see example below the code).
The only indication that the message was not sent properly is a WireShark trace showing the that the message was consisted of only a few bytes and more importantly is considered a malformed packet:
The code is as follows:
using System.IO;
using System.Text;
using System.Reflection;
using SyslogNet.Client;
using SyslogNet.Client.Serialization;
using SyslogNet.Client.Transport;
using Severity = SyslogNet.Client.Severity;
namespace Providers.Syslog
{
public class SyslogLogger
{
private readonly SyslogSettings _syslogSettings;
private readonly SyslogUdpSender _syslogUdpSender;
public SyslogLogger(SyslogSettings syslogSettings)
{
_syslogSettings = syslogSettings;
_syslogUdpSender = new SyslogUdpSender(syslogSettings.SyslogServerName, syslogSettings.SyslogServerPort);
}
public void Log(string message)
{
var syslogMessage = new SyslogMessage(
null,
(Facility)_syslogSettings.Facility,
Severity.Informational,
null,
_syslogSettings.Identity,
message);
_syslogUdpSender.Send(syslogMessage, new SyslogMessageSerializer());
}
private class SyslogMessageSerializer : SyslogMessageSerializerBase, ISyslogMessageSerializer
{
private const int AppNameMaxLength = 32;
public void Serialize(SyslogMessage message, Stream stream)
{
var priorityValue = CalculatePriorityValue(message.Facility, message.Severity);
string timestamp = null;
if (message.DateTimeOffset.HasValue)
{
var dt = message.DateTimeOffset.Value;
var day = dt.Day < 10 ? " " + dt.Day : dt.Day.ToString();
timestamp = string.Concat(dt.ToString("MMM "), day, dt.ToString(" HH:mm:ss"));
}
var headerBuilder = new StringBuilder();
headerBuilder.Append("<").Append(priorityValue).Append(">");
if (!string.IsNullOrEmpty(timestamp))
{
headerBuilder.Append(timestamp).Append(" ");
}
if (!string.IsNullOrEmpty(message.HostName))
{
headerBuilder.Append(message.HostName).Append(" ");
}
if (!message.Message.IsSiemOrientadTemplate())
{
var appName = message.AppName;
headerBuilder.Append(string.IsNullOrWhiteSpace(appName)
? appName
: (appName.Length > AppNameMaxLength ? appName.Substring(0, AppNameMaxLength) : appName) + ": ");
headerBuilder.Append(message.Message ?? "No Content");
}
var encoding = new UTF8Encoding(false);
var bytes = encoding.GetBytes(headerBuilder.ToString());
stream.Write(bytes, 0, bytes.Length);
}
}
}
}
e.g. of a message (I removed actual data and put on placeholders):
"LEEF:2.0|CompanyNameString|ProduceNameString|VersionString|int|^|
^cat=Test - QRADAR^sev=6^DescriptionString=^Timestamp=Apr 03 2017 14:48:02^userNameString=domain\DisplayName^accountName=samAccountName^proto=Syslog"
Any help will be much appreciated as it's been a while and I couldn't find a solution. Thanks in advance!
Finally found the issue.
I overlooked the fact that the content was in an "if" clause for non-SIEM systems.
Moving this line out of the clause resolve the issue for all syslog provider types.
headerBuilder.Append(message.Message ?? "No Content");
I'm looking to parse the WebCacheV01.dat file using C# to find the last file location for upload in an Internet browser.
%LocalAppData%\Microsoft\Windows\WebCache\WebCacheV01.dat
I using the Managed Esent nuget package.
Esent.Isam
Esent.Interop
When I try and run the below code it fails at:
Api.JetGetDatabaseFileInfo(filePath, out pageSize, JET_DbInfo.PageSize);
Or if I use
Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null);
at
Api.JetAttachDatabase(sesid, filePath, AttachDatabaseGrbit.ReadOnly);
I get the following error:
An unhandled exception of type
'Microsoft.Isam.Esent.Interop.EsentFileAccessDeniedException' occurred
in Esent.Interop.dll
Additional information: Cannot access file, the file is locked or in use
string localAppDataPath = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);
string filePathExtra = #"\Microsoft\Windows\WebCache\WebCacheV01.dat";
string filePath = string.Format("{0}{1}", localAppDataPath, filePathExtra);
JET_INSTANCE instance;
JET_SESID sesid;
JET_DBID dbid;
JET_TABLEID tableid;
String connect = "";
JET_SNP snp;
JET_SNT snt;
object data;
int numInstance = 0;
JET_INSTANCE_INFO [] instances;
int pageSize;
JET_COLUMNDEF columndef = new JET_COLUMNDEF();
JET_COLUMNID columnid;
Api.JetCreateInstance(out instance, "instance");
Api.JetGetDatabaseFileInfo(filePath, out pageSize, JET_DbInfo.PageSize);
Api.JetSetSystemParameter(JET_INSTANCE.Nil, JET_SESID.Nil, JET_param.DatabasePageSize, pageSize, null);
//Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null);
Api.JetInit(ref instance);
Api.JetBeginSession(instance, out sesid, null, null);
//Do stuff in db
Api.JetEndSession(sesid, EndSessionGrbit.None);
Api.JetTerm(instance);
Is it not possible to read this without making modifications?
Viewer
http://www.nirsoft.net/utils/ese_database_view.html
Python
https://jon.glass/attempts-to-parse-webcachev01-dat/
libesedb
impacket
Issue:
The file is probably in use.
Solution:
in order to free the locked file, please stop the Schedule Task -\Microsoft\Windows\Wininet\CacheTask.
The Code
public override IEnumerable<string> GetBrowsingHistoryUrls(FileInfo fileInfo)
{
var fileName = fileInfo.FullName;
var results = new List<string>();
try
{
int pageSize;
Api.JetGetDatabaseFileInfo(fileName, out pageSize, JET_DbInfo.PageSize);
SystemParameters.DatabasePageSize = pageSize;
using (var instance = new Instance("Browsing History"))
{
var param = new InstanceParameters(instance);
param.Recovery = false;
instance.Init();
using (var session = new Session(instance))
{
Api.JetAttachDatabase(session, fileName, AttachDatabaseGrbit.ReadOnly);
JET_DBID dbid;
Api.JetOpenDatabase(session, fileName, null, out dbid, OpenDatabaseGrbit.ReadOnly);
using (var tableContainers = new Table(session, dbid, "Containers", OpenTableGrbit.ReadOnly))
{
IDictionary<string, JET_COLUMNID> containerColumns = Api.GetColumnDictionary(session, tableContainers);
if (Api.TryMoveFirst(session, tableContainers))
{
do
{
var retrieveColumnAsInt32 = Api.RetrieveColumnAsInt32(session, tableContainers, columnIds["ContainerId"]);
if (retrieveColumnAsInt32 != null)
{
var containerId = (int)retrieveColumnAsInt32;
using (var table = new Table(session, dbid, "Container_" + containerId, OpenTableGrbit.ReadOnly))
{
var tableColumns = Api.GetColumnDictionary(session, table);
if (Api.TryMoveFirst(session, table))
{
do
{
var url = Api.RetrieveColumnAsString(
session,
table,
tableColumns["Url"],
Encoding.Unicode);
var downloadedFileName = Api.RetrieveColumnAsString(
session,
table,
columnIds2["Filename"]);
if(string.IsNullOrEmpty(downloadedFileName)) // check for download history only.
continue;
// Order by access Time to find the last uploaded file.
var accessedTime = Api.RetrieveColumnAsInt64(
session,
table,
columnIds2["AccessedTime"]);
var lastVisitTime = accessedTime.HasValue ? DateTime.FromFileTimeUtc(accessedTime.Value) : DateTime.MinValue;
results.Add(url);
}
while (Api.TryMoveNext(session, table.JetTableid));
}
}
}
} while (Api.TryMoveNext(session, tableContainers));
}
}
}
}
}
catch (Exception ex)
{
// log goes here....
}
return results;
}
Utils
Task Scheduler Wrapper
You can use Microsoft.Win32.TaskScheduler.TaskService Wrapper to stop it using c#, just add this Nuget package [nuget]:https://taskscheduler.codeplex.com/
Usage
public static FileInfo CopyLockedFileRtl(DirectoryInfo directory, FileInfo fileInfo, string remoteEndPoint)
{
FileInfo copiedFileInfo = null;
using (var ts = new TaskService(string.Format(#"\\{0}", remoteEndPoint)))
{
var task = ts.GetTask(#"\Microsoft\Windows\Wininet\CacheTask");
task.Stop();
task.Enabled = false;
var byteArray = FileHelper.ReadOnlyAllBytes(fileInfo);
var filePath = Path.Combine(directory.FullName, "unlockedfile.dat");
File.WriteAllBytes(filePath, byteArray);
copiedFileInfo = new FileInfo(filePath);
task.Enabled = true;
task.Run();
task.Dispose();
}
return copiedFileInfo;
}
I was not able to get Adam's answer to work. What worked for me was making a copy with AlphaVSS (a .NET class library that has a managed API for the Volume Shadow Copy Service). The file was in "Dirty Shutdown" state, so I additionally wrote this to handle the exception it threw when I opened it:
catch (EsentErrorException ex)
{ // Usually after the database is copied, it's in Dirty Shutdown state
// This can be verified by running "esentutl.exe /Mh WebCacheV01.dat"
logger.Info(ex.Message);
switch (ex.Error)
{
case JET_err.SecondaryIndexCorrupted:
logger.Info("Secondary Index Corrupted detected, exiting...");
Api.JetTerm2(instance, TermGrbit.Complete);
return false;
case JET_err.DatabaseDirtyShutdown:
logger.Info("Dirty shutdown detected, attempting to recover...");
try
{
Api.JetTerm2(instance, TermGrbit.Complete);
Process.Start("esentutl.exe", "/p /o " + newPath);
Thread.Sleep(5000);
Api.JetInit(ref instance);
Api.JetBeginSession(instance, out sessionId, null, null);
Api.JetAttachDatabase(sessionId, newPath, AttachDatabaseGrbit.None);
}
catch (Exception e2)
{
logger.Info("Could not recover database " + newPath + ", will try opening it one last time. If that doesn't work, try using other esentutl commands", e2);
}
break;
}
}
I'm thinking about using the 'Recent Items' folder as when you select a file to upload an entry is written here:
C:\Users\USER\AppData\Roaming\Microsoft\Windows\Recent
string recent = (Environment.GetFolderPath(Environment.SpecialFolder.Recent));