Checking Elasticsearch Heap Size - c#

How can I check the heap size that is assigned to Elasticsearch engine, is there a way to check it using URL API ? And can I use NEST to check it?
Thanks

In a clustered environment, heap settings can be queried as :
curl -sS "localhost:9200/_cat/nodes?h=heap*&v"
Eg:
curl -sS "localhost:9200/_cat/nodes?h=heap*&v"
heap.current heap.percent heap.max
321.1mb 32 989.8mb
424.1mb 42 989.8mb
280.3mb 28 989.8mb
This can also be checked from the ps results, though it will only shed light on the min-max values.
~#ps aux | grep --color=auto -i Xms
elastic 6020 0.0 0.0 12788 936 pts/4 S+ 04:24 0:00 grep --color=auto -i Xms elastic+ 7218
0.6 9.5 5001220 1565112 ? Ssl Jun24 5:14 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.WoiU4NhH -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Xms1g -Xmx1g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
where -Xms is the minheap and -Xmx is maxheap configured.

use GET _nodes/stats
Then look at the following in the return /jvm/mem/heap_committed_in_bytes

This is a new account so I couldn't comment on jhilden's answer. His answer is correct but to answer your comment I use Postman which is a chrome app. It's very easy to use and has been ideal for me whilst working with Elasticsearch.
Putting the following in postman and hitting send does the job!
http://localhost:9200/_nodes/stats
review the results and look for the "jvm" block. Here is what is returned when I run it on a default elasticsearch setup:
"jvm": {
"timestamp": 1477474766408,
"uptime_in_millis": 1309586,
"mem": {
**"heap_used_in_bytes": 87134248**,
"heap_used_percent": 8,
"heap_committed_in_bytes": 259522560,
"heap_max_in_bytes": 1038876672,
"non_heap_used_in_bytes": 56166992,
"non_heap_committed_in_bytes": 57348096,

As per the online documentation, the default heap size is 1 Gb unless you explicitly mention in the Environment Variable [ES_HEAP_SIZE]
The default installation of Elasticsearch is configured with a 1 GB
heap. For just about every deployment, this number is far too small.
If you are using the default heap values, your cluster is probably
configured incorrectly.
There are two ways to change the heap size in Elasticsearch. The
easiest is to set an environment variable called ES_HEAP_SIZE. When
the server process starts, it will read this environment variable and
set the heap accordingly. As an example, you can set it via the
command line as follows:
export ES_HEAP_SIZE=10g Alternatively, you can pass in the heap size
via a command-line argument when starting the process, if that is
easier for your setup:
./bin/elasticsearch -Xmx10g -Xms10g
Ensure that the min (Xms) and max (Xmx) sizes are the same to prevent
the heap from resizing at runtime, a very costly process.
Generally, setting the ES_HEAP_SIZE environment variable is preferred
over setting explicit -Xmx and -Xms values.

Related

How to use environment variables DOTNET_RUNNING_IN_CONTAINERS and DOTNET_RUNNING_IN_CONTAINER to determine where is .net application running?

The official MS page suggests existence of "well known" environment variables, that are supposed to be set by 'official .NET images'.
Frankly, this seems to be true, but to my experience they are always set to 'false', even if I run app in a container.
I googled some example of usage and this is what I found:
When I followed the same in a regular .net 5.0 app, I always get 'false' from this:
private bool InDocker { get { return Environment.GetEnvironmentVariable("DOTNET_RUNNING_IN_CONTAINER") == "true";} }
Same result, if I try the plural version: DOTNET_RUNNING_IN_CONTAINERS
I have also tried to set the variable manualy in my Dockerfile like so:
ENV DOTNET_RUNNING_IN_CONTAINER = true
What could I be doing wrong ?
My problem was, that I also set my own ENV variable with the same name within my Dockerfile, which caused kind of overwritting the original variable.
In the image below, I have renamed my own ENV variable and set it to true like so:
ENV DOTNET_RUNNING_IN_CONTAINER_PH = true
Perhaps I should have set it as string:
ENV DOTNET_RUNNING_IN_CONTAINER_PH = "true"
Many thanks to #Ralf for pointing out very good hints.
If you bother to compose an aswer in this manner, I will make it THE answer :o)

How to change language of powershell script? [duplicate]

I wrote some software that uses the output of system (powershell) commands, but did not foresee that the output would be different for languages other than English.
Is there a way to temporarily change the language in Powershell to English for just that one, single powershell session?
Notes
In case it is of significance, the particular powershell code I wish to run is netstat -n -a
I have come across some ways to change powershell language (e.g. here, here). But I want to be careful not to change it permanently! (that would be bad)
(a) For external programs such as netstat.exe, there is unfortunately no way (that I know of) to change the UI language in-session:
On Windows Server 2012 / Windows 8 and above, the Set-WinUILanguageOverride cmdlet allows you to (persistently) change the system-wide UI language for the current user, but that only takes effect in future logon sessions - that is, logging off and back on or a reboot are required.
As an aside: On Windows Server 2012 / Windows 8 and above, there is also the Set-Culture cmdlet, but its purpose is not to change the UI culture (display language), but only culture-specific settings such as date, number, and currency formats. It too changes the setting persistently for the current user, but only requires a new session (process) for the change to take effect.
(b) For PowerShell commands and .NET types, there is an in-session (non-persistent) solution - assuming the commands are culture-aware and come with localized strings:
Set [cultureinfo]::CurrentUICulture (temporarily) to the desired culture name (use [cultureinfo]::GetCultures('SpecificCultures') to see all predefined ones) ; e.g., [cultureinfo]::CurrentUICulture = 'en-US'
Complementarily, you may want to set [cultureinfo]::CurrentCulture (note the missing UI part) as well, which determines the culture-specific number, date, ... formatting.
In older versions of PowerShell / .NET, you'll have to set these properties on [System.Threading.Thread]::CurrentThread instead; e.g.,
[System.Threading.Thread]::CurrentThread.CurrentUICulture = 'en-US'
See the bottom section for helper function Use-Culture that wraps this functionality for execution of code with a different culture temporarily in effect; here's an example call
with the culture-sensitive Get-LocalGroupMember cmdlet:
# Try with values other than "en-US", e.g. "fr-FR" to see localized
# values in the "ObjectClass" output column.
Use-Culture en-US { Get-LocalGroupMember Administrators }
An ad hoc example, if you don't want to define a helper function (only the UI culture is changed here):
& {
$prev=[cultureinfo]::CurrentUICulture
[cultureinfo]::CurrentUICulture='en-US'
Get-LocalGroupMember Administrators
[cultureinfo]::CurrentUICulture=$prev
}
Caveats:
PowerShell [Core] itself is not localized yet, as of v7.2.x; progress is being tracked in GitHub issue #666; however, the solution below does work with third-party modules that ship with localized messages and help content, as well as select Windows-specific modules that talk to platform APIs, such as the Microsoft.PowerShell.LocalAccounts module, whose Get-LocalGroupMember cmdlet was used in the example above.
Due to a bug in Windows PowerShell (PowerShell [Core] v6+ is not affected), in-session changes to [cultureinfo]::CurrentUICulture and [cultureinfo]::CurrentCulture are automatically reset at the command prompt, whenever a command finishes executing; however, for a given script the changes remain in effect for the entire script and its callees - see this answer.
Taking a step back:
I wrote some software that uses the output of system (powershell) commands, but did not foresee that the output would be different for languages other than English.
This is precisely why it's generally worth looking for PowerShell-native solutions as opposed to calling external programs:
Instead of having to parse - possibly localized - text, as with netstat.exe, for instance, PowerShell commands return objects whose properties you can robustly access in a culture-independent fashion.
Specifically, Mathias R. Jessen suggests looking at Get-NetTCPConnection as a PowerShell alternative to netstat.exe (available on Windows Server 2012 / Windows 8 and above).
Function Use-Culture's source code:
Note: The code was gratefully adapted from this venerable blog post; it is designed
# Runs a script block in the context of the specified culture, without changing
# the session's culture persistently.
# Handy for quickly testing the behavior of a command in the context of a different culture.
# Example:
# Use-Culture fr-FR { Get-Date }
function Use-Culture
{
param(
[Parameter(Mandatory)] [cultureinfo] $Culture,
[Parameter(Mandatory)] [scriptblock] $ScriptBlock
)
# Note: In Windows 10, a culture-info object can be created from *any* string.
# However, an identifier that does't refer to a *predefined* culture is
# reflected in .LCID containing 4096 (0x1000)
if ($Culture.LCID -eq 4096) { Throw "Unrecognized culture: $($Culture.DisplayName)" }
# Save the current culture / UI culture values.
$PrevCultures = [Threading.Thread]::CurrentThread.CurrentCulture, [Threading.Thread]::CurrentThread.CurrentUICulture
try {
# (Temporarily) set the culture and UI culture for the current thread.
[Threading.Thread]::CurrentThread.CurrentCulture = [Threading.Thread]::CurrentThread.CurrentUICulture = $Culture
# Now invoke the given code.
& $ScriptBlock
}
finally {
# Restore the previous culture / UI culture values.
[Threading.Thread]::CurrentThread.CurrentCulture = $PrevCultures[0]
[Threading.Thread]::CurrentThread.CurrentUICulture = $PrevCultures[1]
}
}
Original author of this code is #Scepticalist.
Run this from powershell console. It will change the culture to en-US for current session.
function Set-CultureWin([System.Globalization.CultureInfo] $culture) { [System.Threading.Thread]::CurrentThread.CurrentUICulture = $culture ; [System.Threading.Thread]::CurrentThread.CurrentCulture = $culture } ; Set-CultureWin en-US ; [system.threading.thread]::currentthread.currentculture
Then you have to use the command Get-NetTCPConnection Instead of netstat. For its usage see https://learn.microsoft.com/en-us/powershell/module/nettcpip/get-nettcpconnection?view=win10-ps

Encrypting BIG emails in Outlook cause Low Ressources Exception in C#

I am developing a tool, that encrypts emails with S/MIME in bulk within Outlook 2013. It works so far, but when I am trying to encrypt a REALLY BIG email (in the test case it was about 60MB raw). I get a COMException stating unsufficient ressources.
I can go around this, by working direktly with EWS and MimeKit (which works like a charm! Thank you #jstedfast), but I'd like to find a way to work in Outlook, for network traffic considerations. I know these changes will be synched to Exchange eventually, but during the process itself, it is independent of bandwidth.
I am also looking at MapiEx, but if there is an easier solution, than having yet another dependency (and with MFC too), I'd be happy! Maybe there are some settings, I'd have to make before.
A bit of code. The Exception it caught somewhere else.
public void String SetEncryption(MailItem mailItem)
{
PropertyAccessor pa = null;
try
{
pa = mailItem.PropertyAccessor;
Int32 prop = (int)pa.GetProperty(_PR_SECURITY_FLAGS);
Int32 newprop = prop | 1;
pa.SetProperty(_PR_SECURITY_FLAGS, newprop);
}
finally
{
Marshal.FinalReleaseComObject(pa);
pa = null;
}
}
Edit: The Exception is not coming, when the encryption is set, but when the result is saved, after the encryption is set.
SetEncryption(mailItem);
mailItem.Save();
I solved it myself.
Since I had the problems in Outlook itself, I was trying MAPIEx to access the raw S/MIME Attachment in the email and de-/encrypt it using MimeKit/BouncyCastle.
The same problem occoured, but with a different error message, which lead me to the following site: ASN.1 value too large.
To sum it up: The Crypto API has two signatures. One which takes a byte array and one, which takes a stream. The first one has an arbitrary imposed (!!!) limit of 100 Million Bytes. Since the enveloped CMS has double base64 the ratio of this 100 MB is 9/16, which is round about 56 MB.
I assume, that Outlook uses the same API-Call and therefore has this limit.
I know it is not a fact, but the evidence strongly supports this theory. :)
Check if KB 2480994 is installed: http://support.microsoft.com/kb/2480994

Adjust the screen brightness in a Windows 8 App

I am developing a Windows 8 application. Is there a way I can access the device's brightness settings so the user can adjust the brightness of the device from inside the app? Thanks
I was using a batch file to change things but it also looks like Windows provided an API you can directly from C# (well p/Invoke and then call) as well... so all the above is still true and will still work, but you can also call
PowerGetActiveScheme
PowerWriteACValueIndex
PowerWriteDCValueIndex
Those links are nice too, because they have all the Sub guids listed out for you. So just P/Invoke those bad boys and call them directly from your app, no batch file needed :)
The built in windows Utility PowerCfg can change the brightness on the fly...
Typically PowerCfg is called from the cmd line or in batch files, but you can also call it from your app using Process.Start.
When using PowerCfg you needs to know several things:
The scheme guid: which is the guid associated with the power scheme you want to change.
The Sub Guid: Which is the guid associated with the power setting group you want to change.
The Setting Guid: Which is the guid associated with the actual setting in that power settings group you want to change.
The range or index value: Finally you need to know what value you want to change that setting to... Some settings have a range that you can set it to anything inside of that range, some settings have a list of values to choose and you need to know the index of the value you want.
The Scheme guid is actually the hardest one to get, since the user may have defined new schemes, and thus the guid needs to be found by calling PowerCfg -getactivescheme
The other guids are all constants and can be found by running PowerCfg - query
Once you have all the guids lined up you can set the ac (power plugged in) and dc (on battery) value for each setting. with:
POWERCFG -SETACVALUEINDEX <SCHEME_GUID> <SUB_GUID> <SETTING_GUID> <SettingIndex>
and
POWERCFG -SETDCVALUEINDEX <SCHEME_GUID> <SUB_GUID> <SETTING_GUID> <SettingIndex>
As an example... here is a little batch file that I use to turn off the Adaptive Brightness feature:
Echo Disable Adaptive Display Brightness Setting
for /f "tokens=2 delims=:" %%G in ('powercfg -getactivescheme') do set guid=%%G
for /f %%G in ("%guid%") do set guid=%%G
powercfg -setacvalueindex %guid% 7516b95f-f776-4464-8c53-06167f40cc99 fbd9aa66-9553-4097-ba44-ed6e9d65eab8 000
powercfg -setdcvalueindex %guid% 7516b95f-f776-4464-8c53-06167f40cc99 fbd9aa66-9553-4097-ba44-ed6e9d65eab8 000
The first couple of lines are getting the scheme guid, then the next two are setting the actual values
You could do something very similar for the Display Brightness settings... Which is this sub group and setting guid (same sub group as the Adaptive Brightness):
Subgroup GUID: 7516b95f-f776-4464-8c53-06167f40cc99 (Display)
Power Setting GUID: aded5e82-b909-4619-9949-f5d71dac0bcb (Display brightness)
Minimum Possible Setting: 0x00000000
Maximum Possible Setting: 0x00000064
Possible Settings increment: 0x00000001
Possible Settings units: %
To call it from your C# app you could build a cmd file on the fly and run it with Process.Start
Hope that helps!

Run powershell from c# most efficient way to create multiple mailcontacts in Exchange Online

I wish to create multiple mailcontacts (external Contacts) in the GAL in Microsoft Online by running Powershell command from C#. The code below works, but is very slow and takes about 15-20 min to run for 400 mailcontacts.
foreach(EmailAdressVM emailAddressVM in emailList.emailAddresses1)
{
//Create New MailContact.
Pipeline pplNewMailContact = runspace.CreatePipeline();
Command cmdNewMailContact = new Command("New-MailContact");
cmdNewMailContact.Parameters.Add("Name", emailAddressVM.sExternalEmailAddress);
cmdNewMailContact.Parameters.Add("Displayname", emailAddressVM.sFullName.Trim());
cmdNewMailContact.Parameters.Add("Lastname", emailAddressVM.sLastName.Trim());
cmdNewMailContact.Parameters.Add("Firstname", emailAddressVM.sFirstName.Trim());
cmdNewMailContact.Parameters.Add("ExternalEmailAddress", emailAddressVM.sExternalEmailAddress.Trim());
pplNewMailContact.Commands.Add(cmdNewMailContact);
pplNewMailContact.Invoke();
pplNewMailContact.Stop();
pplNewMailContact.Dispose();
}
I am guessing that this is slow since I create a new Pipeline for every new mailcontact that is added and there has to be a more eficient way of doing this since running...
import-csv <filename> | ForEach {
new-mailcontact -name $_.emailaddress -displayname $_.FullName -lastname $_.lastname -firstname $_.firstname -externalemailaddress $_.emailaddress -alias $_.alias
}
...is much faster.
I have found some references after many hours of searching the web that you can do something similar to using a CSV when running Powershell commands from C#, i.e. send a list (or array) of values to a command (in this case the "new-mailcontact" command). But, I have not found any good example of how to send more than one value to a command and I need to supply many values (for example: -name $.emailAddress -displayname $.FullName, etc.) to the "new-mailcontact" command.
Is it possible to send a list (or array) in a similar way as the "import-csv" command (when using regular powershell) and will this be faster, or is there an evan better way? Would I get better performance if I use Powershell 3 instead of 1 (as I am using now).
Please provide working sample code i C#!
Please note that I cannot save a CSV file to disk and the execute powershell from CMD since I do not have write access to disk and that I do not think that I can run an entire script remotely (since remote scripting probably is disabled on Exchange Online).
The biggest reason I would think is because for each address you are creating a new Powershell instance and you are not multithreaded.
You code looks something like this from above:
Foreach email address{
Declare a new Powershell process
Add attributes to call later
Start Powershell and pipe stuff in
Close Powershell instance
}
I think you would be better off creating the Powershell instance / pipe once and then sending each object into it. More along the lines of:
Create PS Pipe
Foreach email address{
PS.SendArguments(Email, Name, DN, etc.);
}
I am not in an environment to get something working or tested right now, so hopefully this gives you at least most of what you need...

Categories

Resources