I have the following problem:
I have a file which is nearly 500mb big. Its text, all in one line. The text is seperated with a virtual line ending, its called ROW_DEL and is in the text like this:
this is a line ROW_DEL and this is a line
now I need to make the following, I want to split this file into its lines so I get a file like this:
this is a line
and this is a line
the problem, even if I open it with the windows text editor, it breakes because the file is to big.
Is it possible to split this file like I mentioned with C#, Java or Python? Whats would be the best soultion to dont overkill my cpu.
Actually 500mb of text is not that big, it's just that notepad sucks. You probably don't have sed available since you're on windows but at least try naive solution in python, I think it will work fine:
import os
with open('infile.txt') as f_in, open('outfile.txt', 'w') as f_out:
f_out.write(f_in.read().replace('ROW_DEL ', os.linesep))
Read this file in chunks, for example use StreamReader.ReadBlock in c#. You can set the maximum number of characters to read there.
For each readed chunk you can replace ROW_DEL to \r\n and append it to new file.
Just remember to increase current index by the number of character you just read.
Here's my solution.
Easy in the principle (ŁukaszW.pl gave it) but not so easy to code if one wants to take care of peculiar cases (which ŁukaszW.pl did not).
The peculiar cases are when the separator ROW_DEL is splitted in two of the read chunks (as I4V pointed out), and even more subtlely if there are two contiguous ROW_DEL of which the second is splitted in two read chunks.
Since ROW_DEL is longer than any of the possible newlines ('\r', '\n', '\r\n') , it can be replaced in place in the file by the newline used by the OS. That's why I choosed to rewrite the file in itself.
For that I use mode 'r+', it doesn't create a new file.
It's also absolutely mandatory to use a binary mode 'b'.
The principle is to read a chunk (in real life its size will be 262144 for example) and x additional characters, wher x is the length of the separator -1.
And then to examine if the separator is present in the end of the chunk + the x characters.
Accoridng if it is present or not, the chunk is shortened or not before the transformation of the ROW_DEL is performed, and rewritten in place.
The nude code is:
text = ('The hospital roommate of a man infected ROW_DEL'
'with novel coronavirus (NCoV)ROW_DEL'
'—a SARS-related virus first identified ROW_DELROW_DEL'
'last year and already linked to 18 deaths—ROW_DEL'
'has contracted the illness himself, ROW_DEL'
'intensifying concerns about the ROW_DEL'
"virus's ability to spread ROW_DEL"
'from person to person.')
with open('eessaa.txt','w') as f:
f.write(text)
with open('eessaa.txt','rb') as f:
ch = f.read()
print ch.replace('ROW_DEL','ROW_DEL\n')
print '\nlength of the text : %d chars\n' % len(text)
#==========================================
from os.path import getsize
from os import fsync,linesep
def rewrite(whichfile,sep,chunk_length,OSeol=linesep):
if chunk_length<len(sep):
print 'Length of second argument, %d , is '\
'the minimum value for the third argument'\
% len(sep)
return
x = len(sep)-1
x2 = 2*x
file_length = getsize(whichfile)
with open(whichfile,'rb+') as fR,\
open(whichfile,'rb+') as fW:
while True:
chunk = fR.read(chunk_length)
pch = fR.tell()
twelve = chunk[-x:] + fR.read(x)
ptw = fR.tell()
if sep in twelve:
pt = twelve.find(sep)
m = ("\n !! %r is "
"at position %d in twelve !!" % (sep,pt))
y = chunk[0:-x+pt].replace(sep,OSeol)
else:
pt = x
m = ''
y = chunk.replace(sep,OSeol)
pos = fW.tell()
fW.write(y)
fW.flush()
fsync(fW.fileno())
if fR.tell()<file_length:
fR.seek(-x2+pt,1)
else:
fW.truncate()
break
rewrite('eessaa.txt','ROW_DEL',14)
with open('eessaa.txt','rb') as f:
ch = f.read()
print '\n'.join(repr(line)[1:-1] for line in ch.splitlines(1))
print '\nlength of the text : %d chars\n' % len(ch)
To follow the execution, here's another code that prints messages all along:
text = ('The hospital roommate of a man infected ROW_DEL'
'with novel coronavirus (NCoV)ROW_DEL'
'—a SARS-related virus first identified ROW_DELROW_DEL'
'last year and already linked to 18 deaths—ROW_DEL'
'has contracted the illness himself, ROW_DEL'
'intensifying concerns about the ROW_DEL'
"virus's ability to spread ROW_DEL"
'from person to person.')
with open('eessaa.txt','w') as f:
f.write(text)
with open('eessaa.txt','rb') as f:
ch = f.read()
print ch.replace('ROW_DEL','ROW_DEL\n')
print '\nlength of the text : %d chars\n' % len(text)
#==========================================
from os.path import getsize
from os import fsync,linesep
def rewrite(whichfile,sep,chunk_length,OSeol=linesep):
if chunk_length<len(sep):
print 'Length of second argument, %d , is '\
'the minimum value for the third argument'\
% len(sep)
return
x = len(sep)-1
x2 = 2*x
file_length = getsize(whichfile)
with open(whichfile,'rb+') as fR,\
open(whichfile,'rb+') as fW:
while True:
chunk = fR.read(chunk_length)
pch = fR.tell()
twelve = chunk[-x:] + fR.read(x)
ptw = fR.tell()
if sep in twelve:
pt = twelve.find(sep)
m = ("\n !! %r is "
"at position %d in twelve !!" % (sep,pt))
y = chunk[0:-x+pt].replace(sep,OSeol)
else:
pt = x
m = ''
y = chunk.replace(sep,OSeol)
print ('chunk == %r %d chars\n'
' -> fR now at position %d\n'
'twelve == %r %d chars %s\n'
' -> fR now at position %d'
% (chunk ,len(chunk), pch,
twelve,len(twelve),m, ptw) )
pos = fW.tell()
fW.write(y)
fW.flush()
fsync(fW.fileno())
print (' %r %d long\n'
' has been written from position %d\n'
' => fW now at position %d'
% (y,len(y),pos,fW.tell()))
if fR.tell()<file_length:
fR.seek(-x2+pt,1)
print ' -> fR moved %d characters back to position %d'\
% (x2-pt,fR.tell())
else:
print (" => fR is at position %d == file's size\n"
' File has thoroughly been read'
% fR.tell())
fW.truncate()
break
raw_input('\npress any key to continue')
rewrite('eessaa.txt','ROW_DEL',14)
with open('eessaa.txt','rb') as f:
ch = f.read()
print '\n'.join(repr(line)[1:-1] for line in ch.splitlines(1))
print '\nlength of the text : %d chars\n' % len(ch)
There's some subtlety in the treatment of the ends of the chunks in order to detect if ROW_DEL straddles on two chunks and if there are two ROW_DEL contiguous. That's why I took a long time to post my solution: I finally was obliged to write fR.seek(-x2+pt,1) and not only fR.seek(-2*x,1) or fR.seek(-x,1) according if sep is straddling or not (2*x is x2 in the code, with ROW_DEL x and x2 are 6 and 12). Anybody interested by this point will examine it by changing the codes in the sections accoridng if 'ROW_DEL' is in twelve or not.
Related
I trying to use a regex command with loop over regex expression but is slow for files greater than 500 kb
please help me
using System;
using System.IO;
using System.Text.RegularExpressions;
class Program
{
static void Main()
{
string Value3 =#"((#40y#.*?xlrz)(.*?)(#41y#[\s\S\r\n]*?)(\2))";
var match = Regex.Match(File.ReadAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt") , Value3);
while (match.Success)
{
var match2 = Regex.Match(File.ReadAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt") , Value3);
Regex rgx = new Regex(Value3);
match = match2;
string strFile3 = rgx.Replace(File.ReadAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt") , "$1$5$3+", 1);
File.WriteAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt", strFile3);
File.WriteAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\backupregex.txt", string.Concat(File.ReadAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\backupregex.txt") , strFile3.Substring(0, match2.Index +match2.Length)));
File.WriteAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt", strFile3.Substring(match2.Index +match2.Length, strFile3.Length - match2.Index - match2.Length ));
strFile3 = null;
int oldCacheSize = Regex.CacheSize;
Regex.CacheSize= 0;
GC.Collect();
Regex.CacheSize = oldCacheSize;
}
}
}
how can solve please help me
i have a loop each replace of this loop regex get a string of 10mb and make one replacement but this process is too slow , exist any method for solve this problem
when the file is small , regex process each replace fast –
hi my command divide the string in two then use the second string and find a word near at the begin of string then use the index and match lenght for divide the new string and repeat the same process in a loop –
the regex command find the string fast with string small 500 kb
but for string large 1mb this turn slow
word1 word2 word3 replace
word1 word2 word3 replace1
word1 word2 word3 replace2
output
word1 word2 word3 replace
word1 word2 word3 replace+replace1
word1 word2 word3 replace+replace1+replace2
002-0759586-1#39y#REPARTO 01#40y#002-075958655xlrz10,4#41y##42y#-10.20
002-0759586-2#39y#REPARTO 01#40y#002-0759586xlrz54#41y#0#42y#
002-0759586-2#39y#REPARTO 01#40y#002-0759586xlrz56#41y#0#42y#
002-0759586-2#39y#REPARTO 01#40y#002-0759586xlrz57#41y#0#42y#
An attempt at an answer.
Problem 1. Regex
Existing:
( # (1 start)
( \# 40y \# .*? xlrz ) # (2)
( .*? ) # (3)
( \# 41y \# [\s\S\r\n]*? ) # (4)
( \2 ) # (5)
) # (1 end)
What it should be:
( # (1 start)
\# 40y \#
.*?
xlrz
) # (1 end)
( .*? ) # (2)
\# 41y \#
(?s: .*? )
\1
Capture group changes (old -> new):
1 -> 0
2 -> 1
3 -> 2
4 -> N/A
5 -> 1
Benchmark
Regex1: ((\#40y\#.*?xlrz)(.*?)(\#41y\#[\s\S\r\n]*?)(\2))
Options: < none >
Completed iterations: 50 / 50 ( x 1000 )
Matches found per iteration: 1
Elapsed Time: 4.04 s, 4042.77 ms, 4042771 µs
Regex2: ((\#40y\#.*?xlrz)(.*?)\#41y\#(?s:.*?)\2)
Options: < none >
Completed iterations: 50 / 50 ( x 1000 )
Matches found per iteration: 1
Elapsed Time: 1.91 s, 1913.65 ms, 1913650 µs
The general regex problem:
You are delimiting a string with what is captured in Group 1.
The problem is group 1 contains a middle sub-expression .*? which is
a hole that backtracking drives a truck through.
This probably can't be avoided, but since you are only matching once each
time, it might not make a difference.
Problem 2. Regex
Never remake a regex within a loop construct. Make it once outside the loop.
If you feel you need to use the same regex twice within the same loop,
red flags should go up as this is never necessary.
If you do however, make two separate regex var's of the same regex.
I.e. Regex rgx1 = new Regex(Value3); Regex rgx2 = new Regex(Value3);
then access the object instance methods (not class methods) for matching.
Problem 3. Program flow
You are manipulating a single file, C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt
Within a loop, you constantly read it, manipulate it, then write it.
This is problematic at best.
The only reason to ever do this would be to make temporary copies, not what
you're doing with it now.
If you think for some reason you're saving memory resources, you'd be mistaken.
For one, the stack is of limited size as compared to the heap.
Once you have the ORIGINAL file read into a string variable, do all the
operations on that string, then as a last step, write it out to the file.
This part of your code is inappropriate, but used as an example:
string strFile3 = rgx.Replace(File.ReadAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt") , "$1$5$3+", 1);
File.WriteAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt", strFile3);
File.WriteAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\backupregex.txt", string.Concat(File.ReadAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\backupregex.txt") , strFile3.Substring(0, match2.Index +match2.Length)));
File.WriteAllText(#"C:\Users\diego\Desktop\pruebatabladinamica\pruebassregex.txt", strFile3.Substring(match2.Index +match2.Length, strFile3.Length - match2.Index - match2.Length ));
All can be reduced to the manipulation of strFile3 without doing the
intense overhead of reading and writing and thrashing the disk.
In actuality, this is what is causing the performance lag you see on a larger
file.
End of attempt ..
Here is code;
foreach (var file in d.GetFiles("*.xml"))
{
string test = getValuesOneFile(file.ToString());
result.Add(test);
Console.WriteLine(test);
Console.ReadLine();
}
File.WriteAllLines(filepath + #"\MapData.txt", result);
Here is what it looks like in the console;
[30000]
total=5
sp 0 -144 152 999999999
sp 0 -207 123 999999999
sp 0 -173 125 999999999
in00 1 -184 213 999999999
out00 2 1046 94 40000
Here is how it looks like in the text file (when written at end of loop).
[30000]total=5sp 0 -144 152 999999999sp 0 -207 123 999999999sp 0 -173 125 999999999in00 1 -184 213 999999999out00 2 1046 94 40000
I need it to write the lines in the same style as the console output.
WriteAllLines is going to separate each of the values with the environments new line string, however, throughout the history of computers a number of possible different characters have been used to represent new lines. You are looking at the text file using some program that is expecting a different type of new line separator. You should either be using a different program to look at the value of that file; one that either properly handles this type of separator (or can handle any type of separator), you should be configuring your program to expect the given type of separator, or you'll need to replace WriteAllLines with a manual method of writing the strings that uses another new line separator.
Rather than WriteAllLines You'll probably want to just write the text manually:
string textToWrite = "";
foreach (var res in result)
{
textToWrite += res.Replace("\r","").Replace("\n",""); //Ensure there are no line feeds or carriage returns
textToWrite += "\r\n"; //Add the Carriage Return
}
File.WriteAllText(filepath + #"\MapData.txt", textToWrite)
The problem is definitely how you are looking for newlines in your output. Environment.NewLine will get inserted after each string written by WriteAllLines.
I would recommend opening the output file in NotePad++ and turn on View-> ShowSymbol-> Show End of Line to see what end of line characters are in the file. On my machine for instance it is [CR][LF] (Carriage Return / Line Feed) at the end of each line which is standard for windows.
So I'm trying to load some data from a text file when the user types 'Load'. I have it sort of working, but there's a bug with it, it seems.
Let's say you buy an axe, which is 70 coins. You start with 100 coins, so you're left with 30 coins. You decide to save and exit the game. You come back and load your saved game, and you have 49 coins and 48 health points.
It also does the exact same thing when you save straight away with all the default values. I don't have any idea where it's get the value of 49 and 48 from.
const string f = "savedgame.txt";
using (StreamReader r = new StreamReader(f))
{
string line;
while ((line = r.ReadLine()) != null)
{
player.gold = line[0];
player.health = line[1];
Console.WriteLine("Your game has been loaded.");
menu.start(menu, shop, player, fishing, woodcut, mine, monster, quests, save, load);
}
}
This is my text file that I've just saved now.
100
20
1
0
0
5
I've tried examples off Google, but they did the same thing. So I did a bit of research and made one myself.... Did the same thing.
I don't know how much I can show you guys, but I can show more if needed.
Am I doing something wrong?
There are two problems with your current approach -
First, your logic is currently working on the characters of the first line -
The first time through your loop, line is "100", so line[0] is 1. This is a char of '1', not the value 1, so player.gold = line[0] translates into setting player.gold to the numerical equivelent of the character '1', which is 49.
Second, you're starting a loop and reading line by line, and then doing your work instead of reading all of the lines at once.
You can solve these issues by reading all of the lines at once, and working line by line. Then you also need to convert the entire line to a number, not read one character:
const string f = "savedgame.txt";
var lines = File.ReadAllLines(f);
player.gold = Convert.ToInt32(lines[0]);
player.health = Convert.ToInt32(lines[1]);
r is a string, not an array of lines. you're trying to set gold to the first character of the first line, and health to the second character, etc.
Part of a series of educational regex articles, this is a gentle introduction to the concept of nested references.
The first few triangular numbers are:
1 = 1
3 = 1 + 2
6 = 1 + 2 + 3
10 = 1 + 2 + 3 + 4
15 = 1 + 2 + 3 + 4 + 5
There are many ways to check if a number is triangular. There's this interesting technique that uses regular expressions as follows:
Given n, we first create a string of length n filled with the same character
We then match this string against the pattern ^(\1.|^.)+$
n is triangular if and only if this pattern matches the string
Here are some snippets to show that this works in several languages:
PHP (on ideone.com)
$r = '/^(\1.|^.)+$/';
foreach (range(0,50) as $n) {
if (preg_match($r, str_repeat('o', $n))) {
print("$n ");
}
}
Java (on ideone.com)
for (int n = 0; n <= 50; n++) {
String s = new String(new char[n]);
if (s.matches("(\\1.|^.)+")) {
System.out.print(n + " ");
}
}
C# (on ideone.com)
Regex r = new Regex(#"^(\1.|^.)+$");
for (int n = 0; n <= 50; n++) {
if (r.IsMatch("".PadLeft(n))) {
Console.Write("{0} ", n);
}
}
So this regex seems to work, but can someone explain how?
Similar questions
How to determine if a number is a prime with regex?
Explanation
Here's a schematic breakdown of the pattern:
from beginning…
| …to end
| |
^(\1.|^.)+$
\______/|___match
group 1 one-or-more times
The (…) brackets define capturing group 1, and this group is matched repeatedly with +. This subpattern is anchored with ^ and $ to see if it can match the entire string.
Group 1 tries to match this|that alternates:
\1., that is, what group 1 matched (self reference!), plus one of "any" character,
or ^., that is, just "any" one character at the beginning
Note that in group 1, we have a reference to what group 1 matched! This is a nested/self reference, and is the main idea introduced in this example. Keep in mind that when a capturing group is repeated, generally it only keeps the last capture, so the self reference in this case essentially says:
"Try to match what I matched last time, plus one more. That's what I'll match this time."
Similar to a recursion, there has to be a "base case" with self references. At the first iteration of the +, group 1 had not captured anything yet (which is NOT the same as saying that it starts off with an empty string). Hence the second alternation is introduced, as a way to "initialize" group 1, which is that it's allowed to capture one character when it's at the beginning of the string.
So as it is repeated with +, group 1 first tries to match 1 character, then 2, then 3, then 4, etc. The sum of these numbers is a triangular number.
Further explorations
Note that for simplification, we used strings that consists of the same repeating character as our input. Now that we know how this pattern works, we can see that this pattern can also match strings like "1121231234", "aababc", etc.
Note also that if we find that n is a triangular number, i.e. n = 1 + 2 + … + k, the length of the string captured by group 1 at the end will be k.
Both of these points are shown in the following C# snippet (also seen on ideone.com):
Regex r = new Regex(#"^(\1.|^.)+$");
Console.WriteLine(r.IsMatch("aababc")); // True
Console.WriteLine(r.IsMatch("1121231234")); // True
Console.WriteLine(r.IsMatch("iLoveRegEx")); // False
for (int n = 0; n <= 50; n++) {
Match m = r.Match("".PadLeft(n));
if (m.Success) {
Console.WriteLine("{0} = sum(1..{1})", n, m.Groups[1].Length);
}
}
// 1 = sum(1..1)
// 3 = sum(1..2)
// 6 = sum(1..3)
// 10 = sum(1..4)
// 15 = sum(1..5)
// 21 = sum(1..6)
// 28 = sum(1..7)
// 36 = sum(1..8)
// 45 = sum(1..9)
Flavor notes
Not all flavors support nested references. Always familiarize yourself with the quirks of the flavor that you're working with (and consequently, it almost always helps to provide this information whenever you're asking regex-related questions).
In most flavors, the standard regex matching mechanism tries to see if a pattern can match any part of the input string (possibly, but not necessarily, the entire input). This means that you should remember to always anchor your pattern with ^ and $ whenever necessary.
Java is slightly different in that String.matches, Pattern.matches and Matcher.matches attempt to match a pattern against the entire input string. This is why the anchors can be omitted in the above snippet.
Note that in other contexts, you may need to use \A and \Z anchors instead. For example, in multiline mode, ^ and $ match the beginning and end of each line in the input.
One last thing is that in .NET regex, you CAN actually get all the intermediate captures made by a repeated capturing group. In most flavors, you can't: all intermediate captures are lost and you only get to keep the last.
Related questions
(Java) method matches not work well - with examples on how to do prefix/suffix/infix matching
Is there a regex flavor that allows me to count the number of repetitions matched by * and + (.NET!)
Bonus material: Using regex to find power of twos!!!
With very slight modification, you can use the same techniques presented here to find power of twos.
Here's the basic mathematical property that you want to take advantage of:
1 = 1
2 = (1) + 1
4 = (1+2) + 1
8 = (1+2+4) + 1
16 = (1+2+4+8) + 1
32 = (1+2+4+8+16) + 1
The solution is given below (but do try to solve it yourself first!!!!)
(see on ideone.com in PHP, Java, and C#):
^(\1\1|^.)*.$
Is there any way to do integer incremental replacement only with regex.
Here is the problem, I have text file containing 1 000 000 lines all starting with %
I would like to have replace # by integer incrementally using regex.
input:
% line one
% line two
% line three
...
output:
1 line one
2 line two
3 line three
...
n = 1
with open('sourcefile.txt') as input:
with open('destination.txt', 'w') as output:
for line in input:
if line.startswith('%'):
line = str(n) + line[1:]
n += 1
output.write(line)
Here's a way to do it in Python
import re
from itertools import count
s="""
% line one
% line two
% line three"""
def f():
n=count(1)
def inner(m):
return str(next(n))
return inner
new_s = re.sub("%",f(),s)
alternatively you could use a lambda function in there like so:
new_s = re.sub("%",lambda m,n=count(1):str(next(n)),s)
But it's easy and better to skip regexp altogether
from __future__ import print_function # For Python<3
import fileinput
f=fileinput.FileInput("file.txt", inplace=1)
for i,line in enumerate(f):
print ("{0}{1}".format(i, line[1:]), end="")
Since all the lines start with "%" there is no need to even look at that first char
Although this problem would best be solved by reading the file line by line and checking the first character with simple string functions, here is how you would do incremental replacement on a string in java:
Pattern p = Pattern.compile("^%");
Matcher m = p.matcher(text);
StringBuffer sb = new StringBuffer();
int i = 0;
while (m.find()) {
m.appendReplacement(sb, String.valueOf(i++));
}
m.appendTail(sb);
return sb.toString();
in python re.sub accept function as parameter see http://docs.python.org/library/re.html#re.sub
Depending on your choice of language (you've listed a few) PHP's preg_replace_callback() might be an appropriate function to use
$text = "% First Line\n% Second Line\n% Third Line";
function cb_numbers($matches)
{
static $c = 1;
return $c++;
}
$text = preg_replace_callback(
"/(%)/",
"cb_numbers",
$text);
echo $text;
Here's a C# (3.0+) version:
string s = "% line one\n% line two\n% line three";
int n = 1;
s = Regex.Replace(s, #"(?m)^%", m => { return n++.ToString(); });
Console.WriteLine(s);
output:
1 line one
2 line two
3 line three
Of course it requires the whole text to be loaded into memory. If I were doing this for real, I'd probably go with a line-by-line approach.
And a PHP version for good measure:
$input = #fopen('input.txt', 'r');
$output = #fopen("output.txt", "w");
if ($input && $output) {
$i = 0;
while (!feof($input)) {
$line = fgets($input);
fputs($output, ($line[0] === '%') ?
substr_replace($line, ++$i, 0, 1) :
$line
);
}
fclose($input);
fclose($output);
}
And just because you can, a perl one-liner (yes, with a regex):
perl -i.bak -pe 'BEGIN{$i=1} (s/^%/$i/) && $i++' input.txt
import re, itertools
counter= itertools.count(1)
replacer= lambda match: "%d" % counter.next()
text= re.sub("(?m)^%", replacer, text)
counter is… a counter :). replacer is a function returning the counter values as strings. The "(?m)^%" regex is true for every % at the start of a line (note the multi-line flag).