My memory usage on my website causes a recycle every 10 minutes without fail on my Windows server. Memory normally is 140mb but all of a sudden it jumps up to 600mb or more and the website needs to be recyled. I have the following entries in my web.config file to stop bots from attacking my website. However, I get a syntax error for the filteringRules line although my website runs ok with the updated web.config file. I noticed that some of the bots still appear on my http log and am wondering if my filter is working properly. Does anyone see a problem with this code or is it actually working in spite of the syntax error I get in Visual Studio? Also, can all these bots cause memory spikes?
<requestFiltering>
<filteringRules> !-- this line gives me a syntax error
<filteringRule name="BlockSearchEngines" scanUrl="false"
scanQueryString="false">
<scanHeaders>
<clear />
<add requestHeader="User-Agent" />
</scanHeaders>
<appliesTo>
<clear />
</appliesTo>
<denyStrings>
<clear />
<add string="AhrefsBot" />
<add string="MJ12bot" />
<add string="ExtLinksfBot" />
<add string="Yeti" />
<add string="YandexBot" />
<add string="SemrushBot" />
<add string="DotBot" />
<add string="istellabot" />
<add string="Qwantify" />
<add string="GrapeshotCrawler" />
<add string="archive.org_bot" />
<add string="Applebot" />
<add string="ias_crawler" />
<add string="Uipbot" />
<add string="Cliqzbot" />
<add string="TinEye-bot" />
<add string="YandexImages" />
</denyStrings>
</filteringRule>
</filteringRules>
The syntax of your <filteringRules> entry looks correct (except for the closing tag typo).
The schema for web.config that Visual Studio uses doesn't seem to include filteringRules, for some reason. However, that schema is independent from what IIS knows and does, so it shouldn't matter.
Also, can all these bots cause memory spikes?
Just because a request is coming from a bot won't make it use a lot of memory. Unintentional memory spikes are usually application bugs -- memory leaks and the like. You can look at your app's memory use with a memory profiler. Allocating memory buffers and then storing references to them in a global variable/collection is a common cause. Failing to call Dispose() on all Disposable objects is another potential source of leaks.
I hope this is not considered off topic, but it is a C# .Net problem of sorts:
I am using log4net 1.2.10. It has been stated on the apache log4net forum that from 1.2.9 onwards, different file locking schemas are available. I have configured log4net as per examples given to use a non-default method (minimal-locking or interprocess locking) but the immediate effect is the following error message:
log4net:ERROR XmlHierarchyConfigurator: Cannot find Property [lockingModel] to set object on [log4net.Repository.Hierarchy.DefaultLoggerFactory+LoggerImpl]
log4net:ERROR [RollingFileAppender] Unable to acquire lock on file C:\Program Files (x86)\Autoscribe\Matrix Gemini LIMS\Exe\log\Trace.log. The process cannot access the file 'C:\Program Files (x86)\Autoscribe\Matrix Gemini LIMS\Exe\log\Trace.log' because it is being used by another process.
My log4net.config file is:
<logger name="Matrix.Trace">
<!--this is for routine debug logging to a file-->
<level value="OFF" />
<lockingModel type="log4net.Appender.RollingFileAppender+InterProcessLock" />
<appender-ref ref="RollingFileAppenderTRACE" />
</logger>
<appender name="RollingFileAppenderTRACE" type="log4net.Appender.RollingFileAppender">
<file value=".\log\Trace.log" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%m%newline" />
</layout>
<appendToFile value="true" />
<maximumFileSize value="100KB" />
<maxSizeRollBackups value="2" />
</appender>
etc...
My C# application does this:
private static readonly ILog bllDebugLog = LogManager.GetLogger("Matrix.Trace"); //for debugging
and
log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo("log4net.config"));
If I use the default locking method, logging does work (but I have file-lock problems with multiple processes, which is what I am trying to resolve).
I cannot see from the log4net documentation why this should fail. The error appears to suggest that log4net 1.2.10 does not support non-default locking, but the apache documentation is quite clear that it should. What am I doing wrong?
TIA
Apologies all round - I was being dumb. The lockingmodel bit needs to go in the appender definitions, not the logger definitions, and it appears the locking model should be log4net.Appender.FileAppender+InterProcessLock, not log4net.Appender.RollingFileAppender+InterProcessLock (even though I am using a RollingFileAppender logger type).
I'm having a problem logging to a text file using log4net dll.
I'm using vs 2012 express on a windows server 2008 r2 standard (64).
I have preform the following steps:
Open a new asp.net web forms
Install log4net via nuget manager.
in the global.asax file I have this in Application_Start :
log4net.Config.XmlConfigurator.Configure ();
In the web config I have added to the configSections node this line :
< section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>
In the web config I have added this section :
<log4net debug="true">
<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
<file value="c:\webSite\extraDownloadServerResources\Logs\ALU\log.txt" />
<appendToFile value="true" />
<rollingStyle value="Composite"/>
<datePattern value="yyyyMMdd" />
<maxSizeRollBackups value="5" />
<maximumFileSize value="5MB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%n%-5p %d %5rms %-22.22c{1} %-18.18M %n - %m%n" />
</layout>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="RollingLogFileAppender" />
</root>
</log4net>
In a start up page I have fetch instance of the logger :
private static readonly ILog log = LogManager.GetLogger ( System.Reflection.MethodBase.GetCurrentMethod ().DeclaringType );
In the Page_Load I have wrote this line :
log.Debug ( "test" );
I have add to the log.txt file security the Dedualt App Pool user with read/write permissions.
I have ended up with nothing in my log file.
To test the steps I did I have created a new empty web site and follow these steps one by one.
alas (I like this word) this time the log have been written to the file.
Can any one point me to a possible solution ?
To debug issues with log4net configuration you should enable log4net internal debugging.
Inside appSettings add log4net.Internal.Debug key as following:
<appSettings>
<add key="log4net.Internal.Debug" value="true"/>
</appSettings>
This will enable log4net to output all messages to console and to the Trace. To forward all diagnostics messages to file you can add a trace listener:
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add name="textWriterTraceListener"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="C:\tmp\log4net.txt" />
</listeners>
</trace>
</system.diagnostics>
These snippets were taken directly from the log4net FAQ and here only for quick reference.
This at least will give you understanding of what is going on as you will get error messages.
i have managed to solve my problem.
i have downloaded a tool to monitor all iss trafic.
When monitoring the traffic i saw the user name wasn't defaultAppPool nor AppPoolName.
it was ASP.NET v4.0.
After i granted the permissions and restarted everything , i finally saw some logs in the file...
I've checked the connection string (I got it from the server explorer).
I've checked the commandText in log4net config.
I've checked the database permissions (integrated security is fine and works outside of the log4net class).
I've checked the repository's configured property (it is configured, it finds the config file fine).
I've also checked that the fields defined in the config file match the attributes (field size etc.) of the table in the database.
Any ideas?
When I'm debugging it seems to be hitting all the right methods at all the right times, with no exceptions raised.
<log4net>
<appender name="ADONetAppender" type="log4net.Appender.ADONetAppender">
<bufferSize value="1" />
<connectionType value="System.Data.SqlClient.SqlConnection, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<connectionString value="" />
<commandText value="INSERT INTO dbo.Log4Net ([Date],[Thread],[Level],[Logger],[Message]) VALUES ('01-01-2001', 'test', 'test', 'test', 'test')"/>
<!--<commandText value="INSERT INTO dbo.Log4Net ([Date],[Thread],[Level],[Logger],[Message],[Exception],[MachineName],[CultureId],[SourcePage],[Details],[Method]) VALUES (#log_date, #thread, #log_level, #logger, #message, #exception, #MachineName, #CultureId, #SourcePage, #Details, #Method)" />-->
<parameter>
<parameterName value="#log_date"/>
<dbType value="DateTime"/>
<layout type="log4net.Layout.RawTimeStampLayout"/>
</parameter>
<parameter>
<parameterName value="#thread"/>
<dbType value="String"/>
<size value="255"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%thread"/>
</layout>
</parameter>
<parameter>
<parameterName value="#log_level"/>
<dbType value="String"/>
<size value="50"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%level"/>
</layout>
</parameter>
...more parameters
<securitycontext type="log4net.Util.WindowsSecurityContext">
<credentials value="Process">
</credentials>
</securitycontext>
</appender>
<appender name="FileAppender" type="log4net.Appender.FileAppender">
<param name="File" value="LogTest.txt"/>
<param name="AppendToFile" value="true"/>
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d [%t] %-2p %c [%x] - %m%n"/>
</layout>
</appender>
<root>
<appender-ref ref="ADONetAppender"/>
<appender-ref ref="FileAppender"/>
</root>
</log4net>
It's writing to neither appender.
Right, after hours of pulling my hair out - I've cracked it.
This line:
log4net.Config.XmlConfigurator.Configure();
Needed putting in prior to any logging (well, as early as possible in the app). That's it. That was all it took. This is one of those problems were I'm extremely relieved but frustrated at the same time.
I would recommend turning on Log4Net debugging:
<add key="log4net.Internal.Debug" value="true"/>
That may point you in the right direction if there's an error that's occurring behind the scenes. The output will be directed to the console output in the IDE or in the command line.
Check if log4net.dll is placed in the same folder as your application.
Try to enable log4net self-logging, maybe it'll help to find out:
<configuration>
<appSettings>
<add key="log4net.Internal.Debug" value="true"/>
</appSettings>
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add name="textWriterTraceListener"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="C:\tmp\log4net.txt" />
</listeners>
</trace>
</system.diagnostics>
</configuration>
See also the official log4net FAQ.
I ran into a similar issue yesterday, Log4Net was just not writing to the database. I copied the configuration from another existing appender that successfully writes logs to the database. My solution was to run SQL Server Profiler to try and catch what was happening. Profiler showed that the INSERT statements were being sent by Log4Net, but it was failing on the SQL Server side. Manually running the INSERT statement in SQL Server Management Studio showed me exactly what was wrong with it, in my case, it was inserting NULL into a column that didn't accept NULL.
per the ADONetAppender config example:
<commandText value="INSERT INTO dbo.Log4Net
([Date],[Thread],[Level],[Logger],[Message])
VALUES (#log_date, #thread, #log_level, #logger, #message)"/>
This uses the ADO.NET parameterized query format, so you need to use that syntax. Additionally, you may not want to use integrated security for the db connection (esp. if you are running a web site or a service). For your file appender, I recommend a fully qualified path, and make sure it is writeable by the logger.
I assume you have already created the table in the specified database?
NOTE I recommend setting the trace appender in Debug mode too, to make sure you are actually logging stuff.
Add this line in the AssemblyInfo.cs file
[assembly: log4net.Config.XmlConfigurator(Watch = true)]
As the official configuration manual states "The log4net configuration can be configured using assembly-level attributes rather than specified programmatically", which I found as a more clear approach. Some people might find my answer as a more straightforward.
Source: https://logging.apache.org/log4net/release/manual/configuration.html
for sql server connectivity I'm using my live account to login to azure and sql server and due to that I had to change integrated security to "SSPI" instead of "true" did the trick
only found out it was the con string by adding this
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add
name="textWriterTraceListener"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="C:\Logs\log4net.txt" />
</listeners>
</trace>
I have a rather large program that have some odd behaviour once in a while. After it has been deployed at a customer it's not possible to do debugging. But it is permissible to use log files, so this is what I have created. Something like this:
TextWriter tw = new StreamWriter(#"C:\AS-log.txt", true);
tw.WriteLine("ValidateMetaData");
tw.Close();
3 lines like this has been inserted into the code at many places and do give excellent log information. There are 2 problems with this approach however:
The code looks very messy when there are more lines regarding logging than actual code.
I would like to be able to switch logging on and off via a configuration file.
Any suggestions to a way of logging that can do this and still be simple?
Maybe you could try Enterprise libraries from Microsoft. It has a logging application block which works quite nice
http://msdn.microsoft.com/en-us/library/ff648951.aspx
Log4net is a simple framework that you can utilize.
http://logging.apache.org/log4net/
I would suggets to use log4Net. It has a huge potential, that you probbaly don't need, but give you easy predefined formatting in loog entries.
Before use it yuo should configure it your application's .config file.
This is just an example how to do, you can use others that easily can find on internet:
<log4net debug="true">
<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
<file value="Logs\\TestLog.txt" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="10MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-22.22c{1} %-18.18M - %m%n" />
</layout>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="RollingLogFileAppender" />
</root>
</log4net>
Regards.
You can use a Listener of System.Diagnostics.Debug and System.Diagnostics.Trace.
Normally these are controlled by compile options but you can attach the listener depending on your config option.
System.Diagnosics.Trace.WriteLine("ValidateMetaData");
This also allows you to watch live with DebugView, etc.
A simple solution would be to make the log writing a class. On application startup it opens the file for writing to, and there is a simple method Write. You can then simply use Log.Write("ValidateMetaData") which reduces the amount of code that you use inline, and stops you having to always open and close the file. You can also add checks depending on configuration (the easiest way to do that would be with application settings).
Try log4net or nlog (I prefer nlog)
http://logging.apache.org/log4net/
http://nlog-project.org/
If you want something built-in and configurable from config-file see http://msdn.microsoft.com/en-us/library/ms228993.aspx