Develop Reference

Inserting large amount of data in Apache Ignite

I am using Apache Ignite .NET thin client 2.8.1 to insert a large amount of data in the Apache Ignite node. Ignite is hosted on Amazon Linux AMI.
I am trying to insert over 500000 records using PutAllAsync method:
await cacheClient.PutAllAsync(entities); // ICacheClient<int, T>
After that I see the following exception in client logs:
2021-02-03 07:23:37.9917 - NO_TRACE - ****************** - Error: Could not get data from cache
Exception has been thrown by the target of an invocation. System.Reflection.TargetInvocationException System.Object InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at Apache.Ignite.Core.Impl.Client.ClientSocket.SocketRead(Byte[] buf, Int32 pos, Int32 len)
at Apache.Ignite.Core.Impl.Client.ClientSocket.ReceiveBytes(Int32 size)
at Apache.Ignite.Core.Impl.Client.ClientSocket.ReceiveMessage()
at Apache.Ignite.Core.Impl.Client.ClientSocket.SendRequest(RequestMessage& reqMsg)
at Apache.Ignite.Core.Impl.Client.ClientSocket.DoOutInOp[T](ClientOp opId, Action`1 writeAction, Func`2 readFunc, Func`3 errorFunc)
at Apache.Ignite.Core.Impl.Client.Cache.CacheClient`2.DoOutInOp[T](ClientOp opId, Action`1 writeAction, Func`2 readFunc)
at Apache.Ignite.Core.Impl.Client.Cache.CacheClient`2.DoOutOp(ClientOp opId, Action`1 writeAction)
And the following error in ignite logs:
[07:44:53,529][WARNING][grid-timeout-worker-#22][ClientListenerNioListener] Unable to perform handshake within timeout [timeout=10000, remoteAddr=/172.31.56.14:52631]
Are there any best practices of how to insert over 5000 records in ignite cache? Insert batches within transaction?

Ignite is hosted on Amazon Linux AMI
Where do you run the thin client? I suspect that the issue can be simply the low connection speed between the server and the client.
Can you test the connection speed?
What is the size of the data, in megabytes?
How much time does PutAll take for 10, 100, 1000 entries?
There is no DataStreamer in the thin client (yet), so my suggestions are:
Split one big PutAll into multiple smaller ones (e.g. 100 entries at a time)
Increase timeouts on server and client
UPDATE: DataStreamer is now available in Ignite.NET thin client:
IIgniteClient.GetDataStreamer
https://ptupitsyn.github.io/Whats-New-In-Ignite-Net-2.11/

Maybe you actually need to use some sort of streaming mode.
If you have persistence, consider disabling WAL beforehand.
Maybe you have hit some specific issue such as IGNITE-14076. Try if you can work around it by using smaller batches.
Try adjusting timeouts for thin client in ClientConnectorConfiguration and client configuration.

Microsoft Azure Service Bus Timeout Exceptions

We have an application that runs on the Microsoft Azure cloud platform. The communication between some components is made using Service Bus. Everything was working fine, until recently we started to get the following type of timeout exceptions:
When calling QueueClient x.Send(...)
Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory `1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result)
When calling NamespaceManager x.GetQueue(...)
PROGRESS queue processing failed. System.TimeoutException: The request
has timed out after 60000 milliseconds. The successful completion of
the request cannot be determined. Additional queries should be made to
determine whether or not the operation has succeeded.
TrackingId:bdffb6bd-5367-4573-aaa3-8ea9a03f5a2b,TimeStamp:5/28/2015
8:39:46 AM ---> System.Net.WebException: The request was aborted: The
request was canceled. at
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at
Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.b__49(GetAsyncResult`1
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result)
When calling NamespaceManager x.SubscriptionExists(...)
Exception doing periodic work: System.TimeoutException: The request
has timed out after 00:10:00 milliseconds. The successful completion
of the request cannot be determined. Additional queries should be made
to determine whether or not the operation has succeeded. Server stack
trace: Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.NamespaceManager.OnEndSubscriptionExists(IAsyncResult
result) at
Microsoft.ServiceBus.NamespaceManager.SubscriptionExists(String
topicPath, String name) ...
When calling QueueClient x.Receive(...)
PROGRESS queue processing failed.
Microsoft.ServiceBus.Messaging.MessagingCommunicationException: Error
during communication with Service Bus. Check the connection
information, then retry. --->
System.ServiceModel.CommunicationObjectFaultedException: Internal
Server Error: The server did not provide a meaningful reply; this
might be caused by a premature session shutdown.
TrackingId:04ba0220-0350-4806-9c65-c2bba9671054, Timestamp:28.05.2015
13:00:55 Server stack trace: Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception
exception) at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result) ….
The exceptions are clearly related to ServiceBus and they are non-deterministic. The functions throwing them, e.g., Send, GetQueue, SubscriptionExists, are called no more than 100-120 times per minute. We changed nothing in the code and increasing the timeout values (even to ridiculously high values, like 10min) did not help. Also, we do not believe it is some network related problem (on our side) since the same error occur when the application is run from different places.
Has anyone else encountered recently these kind of exceptions? Is there a problem on the side of Microsoft or we are missing something?

A few weeks back we had sudden and unexplainable timing issues crop up with our Service Bus app that has been in production for many months. Ours continued to work but every few calls would take 10+ seconds when they are normally 100-200 millisconds. This went on for a couple weeks and I spent most of that trying to figure out what was going on and never did as the problem suddenly vanished.
We did learn that new Service Bus namespaces we created in the same and other data centers for testing while the problem was occurring did not exhibit the same issue. The Service Bus group offered no help and would only say response times are not guaranteed only the SLA is.

I had similar problem when my running code started to generate Timeout exception. Upon research found that firewall blocked the port used for communication. However, port 80 and 443 were still open. So adding following line of code worked for me:
ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Https;

Apache NMS throws an established connection was aborted by the software in your host machine under heavy use

Background:
C# WPF application talking to JAVA server running on linux via ActiveMQ/JSON
Total 5 instances of connection:
Queues: 2
Topics: 3 (1 producer, 2 consumers)
Problem:
Under heavy use (throughput rate of sending/receiving around 200 messages in less than 500ms and memory working set around 1-1.2 GB), throws ‘An established connection was aborted by the software in your host machine’.
Sample stack:
Apache.NMS.NMSException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
at System.IO.BinaryReader.ReadInt32()
at Apache.NMS.Util.EndianBinaryReader.ReadInt32()
at Apache.NMS.ActiveMQ.OpenWire.OpenWireFormat.Unmarshal(BinaryReader dis)
at Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport.ReadLoop()
Tried so far:
Switched off Inactivity Monitoring to reduce traffic across 5 connections. Mostly because application has got its own heartbeat implementation.
Set ConnectionFactory.OptimizeAcknowledge to true to batch the acknowledgement

Problems with Starksoft.Net.Proxy for SOCKS and bittorrent

I'm using C# to make a bittorrent application with MonoTorrent. The application is somewhat mature, but I'm adding SOCKSv5 proxy support. I've found Starksoft.Net.Proxy to handle the proxy part. However, I'm having some trouble.
I got a BTGuard account for testing with. When I connect (providing username and password and such), I get this error:
Starksoft.Net.Proxy.ProxyException: Connection to proxy host 63.142.161.35 on port 1025 failed. ---> Starksoft.Net.Proxy.ProxyException: The the connection is not allowed by proxy destination rule set concerning destination host 195.122.253.23 port number 11523. The destination reported the host as 0.0.0.0 port -16676.
at Starksoft.Net.Proxy.Socks5ProxyClient.HandleProxyCommandError(Byte[] response, String destinationHost, Int32 destinationPort)
at Starksoft.Net.Proxy.Socks5ProxyClient.SendCommand(Byte command, String destinationHost, Int32 destinationPort)
at Starksoft.Net.Proxy.Socks5ProxyClient.CreateConnection(String destinationHost, Int32 destinationPort)
--- End of inner exception stack trace ---
at Starksoft.Net.Proxy.Socks5ProxyClient.CreateConnection(String destinationHost, Int32 destinationPort)
at Starksoft.Net.Proxy.Socks5ProxyClient.CreateConnectionAsync_DoWork(Object sender, DoWorkEventArgs e)
Which is weird. I have no idea what causes this. I read through the Starksoft code and it seems reasonably in-line with the SOCKS RFC.
Here's the code I'm using to connect to the proxy via Starksoft.Net.Proxy. The stack trace above comes from e.Error on line 133.

I found the issue. Here's the part where Starksoft.Net.Proxy prepares and then discards the data for the username/password exchange.
I fixed it by updating to Biko, but it required some tweaking to get working right. Should be simple for anyone who finds this problem later to deal with.
Alternatively, just grab the source of the broken version and stick some code in to send the data and receive the confirmation.

C# An established connection was aborted by the software in your host machine

These errors are getting more and more frequent on my Game Server. They are causing the server to keep closing and restarting...
System.Net.Sockets.SocketException (0x80004005): An established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
at iRP.Game.Sessions.Session.SendData(Byte[] Data)
This is the code from which these errors are generated:
public void SendData(byte[] Data)
{
try
{
if (mSocket == null)
{
//Output.WriteLine("[SND] Socket has a null exception, which means it is now invalid. Remove this socket!", OutputLevel.CriticalError);
}
else
{
mSocket.BeginSend(Data, 0, Data.Length, SocketFlags.None, sendCallback, mSocket);
}
}
catch (Exception e)
{
string WhatToWrite = "Error handled (SESSION): " + e.ToString() + "\n\n" + e.Message + "\n\nStack: " + e.StackTrace + Environment.NewLine + "\n\n";
File.AppendAllText(Environment.CurrentDirectory + "\\data\\fatal.txt", WhatToWrite);
Program.Stop();
}
}
The buffer sizes are correctly set, we are using KeepAlive on the socket and were using Send and Receive Timeouts.
People suggested that disabling the firewall would help, but whenever I do this our Game Server (Dedicated Server) restarts itself as if it's under attack, so the firewall must remain enabled.
Anyone else got any other solutions for this?
PS: We are behind DDoS Mitigation Services which may be limiting the number of connections...

An established connection was aborted by the software in your host machine
That is a boiler-plate error message, it comes out of Windows. The underlying error code is WSAECONNABORTED. Which really doesn't mean more than "connection was aborted". You have to be a bit careful about the "your host machine" part of the phrase. In the vast majority of Windows application programs, it is indeed the host that the desktop app is connected to that aborted the connection. Usually a server somewhere else.
The roles are reversed however when you implement your own server. Now you need to read the error message as "aborted by the application at the other end of the wire". Which is of course not uncommon when you implement a server, client programs that use your server are not unlikely to abort a connection for whatever reason. It can mean that a fire-wall or a proxy terminated the connection but that's not very likely since they typically would not allow the connection to be established in the first place.
You don't really know why a connection was aborted unless you have insight what is going on at the other end of the wire. That's of course hard to come by. If your server is reachable through the Internet then don't discount the possibility that you are being probed by a port scanner. Or your customers, looking for a game cheat.

This problem appear if two software use same port for connecting to the server
try to close the port by cmd according to your operating system
then reboot your Android studio or your Eclipse or your Software.

Could be related to the maximum number of concurrent requests. I was able to fix it with two solutions:
Increase the default limit of the max concurrent connections (by default it is set to 2):
ServicePointManager.DefaultConnectionLimit = 25
Wrap sending requests around a buffer: could use ConcurrentQueue to limit the rate, or implement a simple wait as the following:
while (activeRequests >= maxConcurrentRequests)
Thread.Sleep(1000);
Interlocked.Increment(ref activeRequests);
var response = await _client.GetStreamAsync(endpoint);
Interlocked.Decrement(ref activeRequests);

While the answer from Hans is an excellent high level summary of the error that worked great for getting me started, I ended up finding a page that explained the root cause well enough that I was able to recreate it with a python script.
The page presents a couple different descriptions of this error that are more detailed than the "connection was aborted" paraphrasing from Hans's answer but still pretty cryptic and not very informative.
The post then explains this scenario that would produce the error:
An HTTP POST is to be sent to an HTTP server.
The server begins reading the POST and notices that the HTTP request header is invalid.
It immediately sends an HTTP response (with an error status, perhaps status=400) and closes the connection without trying to continue reading the remainder of the HTTP request that is forthcoming.
Meanwhile, the client is still happily writing the remainder of the HTTP request to the socket. (Remember a TCP/IP socket connection needs to be closed from both sides. In this case, the server has closed its side, but the client is still pumping data into the half-open connection.)
The client finishes writing the HTTP POST to the socket — meaning that data has been buffered to Winsock. The client application then tries to read the HTTP response, but it cannot because the outgoing retransmission (of the buffered data by WinSock) failed and the socket connection was shutdown on the client side (by Winsock). Even though the HTTP server sent the response, it is lost and cannot be retrieved. The error your application will receive when trying to read the HTTP response on the socket is WSAECONNABORTED
Wikipedia also has a page to explain what winsock is, but all you really need to know for this scenario is that its the Windows socket API.

I had "ManageEngine Agent" installed on my computer, which was blocking my connection to the databases, so uninstalling it was the solution.
Check which application could block the connection to your DB.