Develop Reference

Removing punctuation from an extremely long string

I'm working on a book encryption program for one of my courses and I've run into a problem. Our professor gave us the example of using say Pride and Prejudice as the book used to encrypt, so I chose that one to test my program. The current function I'm using to remove the punctuation from the string is taking so long that the program is being forced into break mode. This function works for smaller strings even pages long, but when I fed it Pride and Prejudice it takes way to long.
public void removePunctuation(ref string s) {
string result = "";
for (int i = 0; i < s.Length; i++) {
if (Char.IsWhiteSpace(s[i])) {
result += ' ';
} else if (!Char.IsLetter(s[i]) && !Char.IsNumber(s[i])) {
// do nothing
} else {
result += s[i];
}
}
s = result;
}
So I think I need a faster way to remove punctuation from this string if anyone has any suggestions? I know looping through every character is horrible, but I'm stumped and I was never taught Regex in depth.
Edit: I was asked how I was storing the string in the dictionary class! This is the constructor for another class that actually uses the formatted string.
public CodeBook(string book)
{
BookMap = new Dictionary<string, List<int>>();
Key = book.Split(null).ToList(); // split string into words
foreach(string s in Key)
{
if (!BookMap.Keys.Contains(s))
{
BookMap.Add(s, Enumerable.Range(0, Key.Count).Where(i => Key[i] == s).ToList());
// add word and add list of occurrances of word
}
}
}

This is slow because you construct string by concatenations in a loop. You have several approaches that are more performant:
Use StringBuilder - unlike string concatenation which constructs a new object each time you add a character, this approach expands the string under construction by larger chunks, preventing excessive garbage creation.
Use LINQ's filtering with Where - this approach constructs an array of chars in a single shot, then constructs a single string from it.
Use regular expression's Replace - this method is optimized to deal with strings of virtually unlimited sizes.
Roll your own algorithm - create an array of chars that corresponds to the length of the original string. Walk through the string, and add the characters that you wish to keep to the array. Use string's constructor that takes the array, the initial index, and the length to construct the string at once.

Looping through every character once is not that bad. You're doing it all in one pass, that's not trivial to avoid.
The problem lies in the fact that the framework will need to allocate a new copy of the (partial) string whenever you do something like
result += s[i];
You can avoid that by introducing a StringBuilder documented here to append non-punctuation characters as you go.
public string removePunctuation(string s)
{
var result = new StringBuilder();
for (int i = 0; i < s.Length; i++) {
if (Char.IsWhiteSpace(s[i])) {
result.Append(" ");
} else if (!Char.IsLetter(s[i]) && !Char.IsNumber(s[i])) {
// do nothing
} else {
result.Append(s[i]);
}
}
return result.ToString();
}
You could further reduce the number of necessary Append calls with a refined algorithm, for example look ahead to the next punctuation and append larger portions at once, or use an existing string manipulation library like RegEx. But the introduction of StringBuilder above should give you a noticable performance gain already.
I was never taught Regex in depth
Use the search provider of your choice, you may end up with a tested solution which you can just study and use: https://stackoverflow.com/a/5871826/1132334

You can use Regex to remove punctuations as below.
public string removePunctuation(string s)
{
string result = Regex.Replace(s, #"[^\w\s]", "");
return result;
}
^ Means: not these characters (letters, numbers).
\w Means: word characters.
\s Means: space characters.

String[0] vs ToCharArray then Char[0]

Say I want to get the third letter in a string.
string s = "cats";
what is the difference between using s[2] to get the value,
as I see most examples online use:
char[] c = s.ToCharArray();
char x = c[2];

If you need a char at particular index, use "cats"[2]
If you need the whole string as char array, use "cats".ToCharArray()

The s[2] method simply looks up the character in that position and returns it.
The s.ToCharArray(); method allocates a new char[], then c[2] looks it up in the array.
If you need just a single character, use the s[2] method as the ToCharArray() method is wasteful. If you need all the characters in the string as a char[] then the ToCharArray() method is cleaner and probably faster (I haven't done any benchmarking, so don't hold me to that).
EDIT:
Both MS.NET and Mono's implementations of string.ToCharArray() use unsafe code that pins the internal char and a duplicate char[] and operates on the pointers. This is definitely faster than calling the indexer for every character and is probably faster than using an enumerator to iterate over all the characters. Plus I find it a lot cleaner, especially in the case where it's used as a parameter:
DoSomethingWithCharArray(s.ToCharArray());
vs
int i = 0;
char[] arr = new char[s.Length];
foreach (char c in s)
{
arr[i] = c;
i++;
}
DoSomethingWithCharArray(arr);
or
char[] arr = new char[s.Length];
for (int i = 0; i < s.Length; i++)
{
arr[i] = s[i];
}
DoSeomthingWithCharArray(arr);

The difference is exactly that you're creating a character array unnecessarily. Use s[2].

If you're just reading the character the difference is only a slightly larger memory footprint. However, if you also want to change some characters and retain the original string then utilizing ToCharArray is very usefull.

Replace a list of invalid character with their valid version (like tr)

I need to do something like this dreamed .trReplace:
str = str.trReplace("áéíüñ","aeiu&");
It should change this string:
a stríng with inválid charactérs
to:
a string with invalid characters
My current ideas are:
str = str.Replace("á","a").Replace("é","e").Replace("í","ï"...
and:
sb = new StringBuilder(str)
sb.Replace("á","a").
sb.Replace("é","e")
sb.Replace("í","ï"...
But I don't think they are efficient for long strings.

Richard has a good answer, but performance may suffer slightly on longer strings (about 25% slower than straight string replace as shown in question). I felt complelled to look in to this a little further. There are actually several good related answers already on StackOverflow as captured below:
Fastest way to remove chars from string
C# Stripping / converting one or more characters
There is also a good article on the CodeProject covering the different options.
http://www.codeproject.com/KB/string/fastestcscaseinsstringrep.aspx
To explain why the function provided in Richards answer gets slower with longer strings is due to the fact that the replacements are happening one character at a time; thus if you have large sequences of non-mapped characters, you are wasting extra cycles while re-appending together the string . As such, if you want to take a few points from the CodePlex Article you end up with a slightly modified version of Richards answer that looks like:
private static readonly Char[] ReplacementChars = new[] { 'á', 'é', 'í', 'ü', 'ñ' };
private static readonly Dictionary<Char, Char> ReplacementMappings = new Dictionary<Char, Char>
{
{ 'á', 'a'},
{ 'é', 'e'},
{ 'í', 'i'},
{ 'ü', 'u'},
{ 'ñ', '&'}
};
private static string Translate(String source)
{
var startIndex = 0;
var currentIndex = 0;
var result = new StringBuilder(source.Length);
while ((currentIndex = source.IndexOfAny(ReplacementChars, startIndex)) != -1)
{
result.Append(source.Substring(startIndex, currentIndex - startIndex));
result.Append(ReplacementMappings[source[currentIndex]]);
startIndex = currentIndex + 1;
}
if (startIndex == 0)
return source;
result.Append(source.Substring(startIndex));
return result.ToString();
}
NOTE Not all edge cases have been tested.
NOTE Could replace ReplacementChars with ReplacementMappings.Keys.ToArray() for a slight cost.
Assuming that NOT every character is a replacement char, then this will actually run slightly faster than straigt string replacements (again about 20%).
That being said, remember when considering performance cost, what we are actually talking about... in this case... the difference between the optimized solution and original solution is about 1 second over 100,000 iterations on a 1,000 character string.
Either way, just wanted to add some information to the answers for this question.

I did something similar for ICAO Passports. The names had to be 'transliterated'. Basically I had a Dictionary of char to char mappings.
Dictionary<char, char> mappings;
static public string Translate(string s)
{
var t = new StringBuilder(s.Length);
foreach (char c in s)
{
char to;
if (mappings.TryGetValue(c, out to))
t.Append(to);
else
t.Append(c);
}
return t.ToString();
}

What you want is a way to go through the string once and do all the replacements. I am not not sure that regex is the best way to do it if you want efficiency. It could very well be that a case switch (for all the characters that you want to replace) in a for loop to test every character is faster. I would profile the two approaches.

It would be better to use an array of char instead of Stringbuilder.
The indexer is faster than calling the Append method, because:
push all local variables to the stack
move to Append address
return to address
pop all local variables from the stack
The example below is about 20 percent faster (depends on your hardware and input string)
static Dictionary<char, char> mappings;
public static string TranslateV2(string s)
{
var len = s.Length;
var array = new char[len];
char c;
for (var index = 0; index < len; index++)
{
c = s[index];
if (mappings.ContainsKey(c))
array[index] = mappings[c];
else
array[index] = c;
}
return new string(array);
}

ReverseString, a C# interview-question

I had an interview question that asked me for my 'feedback' on a piece of code a junior programmer wrote. They hinted there may be a problem and said it will be used heavily on large strings.
public string ReverseString(string sz)
{
string result = string.Empty;
for(int i = sz.Length-1; i>=0; i--)
{
result += sz[i]
}
return result;
}
I couldn't spot it. I saw no problems whatsoever.
In hindsight I could have said the user should resize but it looks like C# doesn't have a resize (i am a C++ guy).
I ended up writing things like use an iterator if its possible, [x] in containers could not be random access so it may be slow. and misc things. But I definitely said I never had to optimize C# code so my thinking may have not failed me on the interview.
I wanted to know, what is the problem with this code, do you guys see it?
-edit-
I changed this into a wiki because there can be several right answers.
Also i am so glad i explicitly said i never had to optimize a C# program and mentioned the misc other things. Oops. I always thought C# didnt have any performance problems with these type of things. oops.

Most importantly? That will suck performance wise - it has to create lots of strings (one per character). The simplest way is something like:
public static string Reverse(string sz) // ideal for an extension method
{
if (string.IsNullOrEmpty(sz) || sz.Length == 1) return sz;
char[] chars = sz.ToCharArray();
Array.Reverse(chars);
return new string(chars);
}

The problem is that string concatenations are expensive to do as strings are immutable in C#. The example given will create a new string one character longer each iteration which is very inefficient. To avoid this you should use the StringBuilder class instead like so:
public string ReverseString(string sz)
{
var builder = new StringBuilder(sz.Length);
for(int i = sz.Length-1; i>=0; i--)
{
builder.Append(sz[i]);
}
return builder.ToString();
}
The StringBuilder is written specifically for scenarios like this as it gives you the ability to concatenate strings without the drawback of excessive memory allocation.
You will notice I have provided the StringBuilder with an initial capacity which you don't often see. As you know the length of the result to begin with, this removes needless memory allocations.
What normally happens is it allocates an amount of memory to the StringBuilder (default 16 characters). Once the contents attempts to exceed that capacity it doubles (I think) its own capactity and carries on. This is much better than allocating memory each time as would happen with normal strings, but if you can avoid this as well it's even better.

A few comments on the answers given so far:
Every single one of them (so far!) will fail on surrogate pairs and combining characters. Oh the joys of Unicode. Reversing a string isn't the same as reversing a sequence of chars.
I like Marc's optimisation for null, empty, and single character inputs. In particular, not only does this get the right answer quickly, but it also handles null (which none of the other answers do)
I originally thought that ToCharArray followed by Array.Reverse would be the fastest, but it does create one "garbage" copy.
The StringBuilder solution creates a single string (not char array) and manipulates that until you call ToString. There's no extra copying involved... but there's a lot more work maintaining lengths etc.
Which is the more efficient solution? Well, I'd have to benchmark it to have any idea at all - but even so that's not going to tell the whole story. Are you using this in a situation with high memory pressure, where extra garbage is a real pain? How fast is your memory vs your CPU, etc?
As ever, readability is usually king - and it doesn't get much better than Marc's answer on that front. In particular, there's no room for an off-by-one error, whereas I'd have to actually put some thought into validating the other answers. I don't like thinking. It hurts my brain, so I try not to do it very often. Using the built-in Array.Reverse sounds much better to me. (Okay, so it still fails on surrogates etc, but hey...)

Since strings are immutable, each += statement will create a new string by copying the string in the last step, along with the single character to form a new string. Effectively, this will be an O(n2) algorithm instead of O(n).
A faster way would be (O(n)):
// pseudocode:
static string ReverseString(string input) {
char[] buf = new char[input.Length];
for(int i = 0; i < buf.Length; ++i)
buf[i] = input[input.Length - i - 1];
return new string(buf);
}

You can do this in .NET 3.5 instead:
public static string Reverse(this string s)
{
return new String((s.ToCharArray().Reverse()).ToArray());
}

Better way to tackle it would be to use a StringBuilder, since it is not immutable you won't get the terrible object generation behavior that you would get above. In .net all strings are immutable, which means that the += operator there will create a new object each time it is hit. StringBuilder uses an internal buffer, so the reversal could be done in the buffer w/ no extra object allocations.

You should use the StringBuilder class to create your resulting string. A string is immutable so when you append a string in each interation of the loop, a new string has to be created, which isn't very efficient.

I prefer something like this:
using System;
using System.Text;
namespace SpringTest3
{
static class Extentions
{
static private StringBuilder ReverseStringImpl(string s, int pos, StringBuilder sb)
{
return (s.Length <= --pos || pos < 0) ? sb : ReverseStringImpl(s, pos, sb.Append(s[pos]));
}
static public string Reverse(this string s)
{
return ReverseStringImpl(s, s.Length, new StringBuilder()).ToString();
}
}
class Program
{
static void Main(string[] args)
{
Console.WriteLine("abc".Reverse());
}
}
}

x is the string to reverse.
Stack<char> stack = new Stack<char>(x);
string s = new string(stack.ToArray());

This method cuts the number of iterations in half. Rather than starting from the end, it starts from the beginning and swaps characters until it hits center. Had to convert the string to a char array because the indexer on a string has no setter.
public string Reverse(String value)
{
if (String.IsNullOrEmpty(value)) throw new ArgumentNullException("value");
char[] array = value.ToCharArray();
for (int i = 0; i < value.Length / 2; i++)
{
char temp = array[i];
array[i] = array[(array.Length - 1) - i];
array[(array.Length - 1) - i] = temp;
}
return new string(array);
}

Necromancing.
As a public service, this is how you actually CORRECTLY reverse a string (reversing a string is NOT equal to reversing a sequence of chars)
public static class Test
{
private static System.Collections.Generic.List<string> GraphemeClusters(string s)
{
System.Collections.Generic.List<string> ls = new System.Collections.Generic.List<string>();
System.Globalization.TextElementEnumerator enumerator = System.Globalization.StringInfo.GetTextElementEnumerator(s);
while (enumerator.MoveNext())
{
ls.Add((string)enumerator.Current);
}
return ls;
}
// this
private static string ReverseGraphemeClusters(string s)
{
if(string.IsNullOrEmpty(s) || s.Length == 1)
return s;
System.Collections.Generic.List<string> ls = GraphemeClusters(s);
ls.Reverse();
return string.Join("", ls.ToArray());
}
public static void TestMe()
{
string s = "Les Mise\u0301rables";
// s = "noël";
string r = ReverseGraphemeClusters(s);
// This would be wrong:
// char[] a = s.ToCharArray();
// System.Array.Reverse(a);
// string r = new string(a);
System.Console.WriteLine(r);
}
}
See:
https://vimeo.com/7403673
By the way, in Golang, the correct way is this:
package main
import (
"unicode"
"regexp"
)
func main() {
str := "\u0308" + "a\u0308" + "o\u0308" + "u\u0308"
println("u\u0308" + "o\u0308" + "a\u0308" + "\u0308" == ReverseGrapheme(str))
println("u\u0308" + "o\u0308" + "a\u0308" + "\u0308" == ReverseGrapheme2(str))
}
func ReverseGrapheme(str string) string {
buf := []rune("")
checked := false
index := 0
ret := ""
for _, c := range str {
if !unicode.Is(unicode.M, c) {
if len(buf) > 0 {
ret = string(buf) + ret
}
buf = buf[:0]
buf = append(buf, c)
if checked == false {
checked = true
}
} else if checked == false {
ret = string(append([]rune(""), c)) + ret
} else {
buf = append(buf, c)
}
index += 1
}
return string(buf) + ret
}
func ReverseGrapheme2(str string) string {
re := regexp.MustCompile("\\PM\\pM*|.")
slice := re.FindAllString(str, -1)
length := len(slice)
ret := ""
for i := 0; i < length; i += 1 {
ret += slice[length-1-i]
}
return ret
}
And the incorrect way is this (ToCharArray.Reverse):
func Reverse(s string) string {
runes := []rune(s)
for i, j := 0, len(runes)-1; i < j; i, j = i+1, j-1 {
runes[i], runes[j] = runes[j], runes[i]
}
return string(runes)
}
Note that you need to know the difference between
- a character and a glyph
- a byte (8 bit) and a codepoint/rune (32 bit)
- a codepoint and a GraphemeCluster [32+ bit] (aka Grapheme/Glyph)
Reference:
Character is an overloaded term than can mean many things.
A code point is the atomic unit of information. Text is a sequence of
code points. Each code point is a number which is given meaning by the
Unicode standard.
A grapheme is a sequence of one or more code points that are displayed
as a single, graphical unit that a reader recognizes as a single
element of the writing system. For example, both a and ä are
graphemes, but they may consist of multiple code points (e.g. ä may be
two code points, one for the base character a followed by one for the
diaresis; but there's also an alternative, legacy, single code point
representing this grapheme). Some code points are never part of any
grapheme (e.g. the zero-width non-joiner, or directional overrides).
A glyph is an image, usually stored in a font (which is a collection
of glyphs), used to represent graphemes or parts thereof. Fonts may
compose multiple glyphs into a single representation, for example, if
the above ä is a single code point, a font may chose to render that as
two separate, spatially overlaid glyphs. For OTF, the font's GSUB and
GPOS tables contain substitution and positioning information to make
this work. A font may contain multiple alternative glyphs for the same
grapheme, too.

static string reverseString(string text)
{
Char[] a = text.ToCharArray();
string b = "";
for (int q = a.Count() - 1; q >= 0; q--)
{
b = b + a[q].ToString();
}
return b;
}

How to find the index of the first char in a string that is not in a list

I know I can loop over the string or build a regex or invert the set (ASCII isn't that big after all) and search for the first instance of that, but Yuck.
What I'm looking for is a nice one liner.
fewer features is better, LINQ is out (for me, don't ask, it's a long story)
The solution I'm going with (unless I see something better)
static int FirstNotMeta(int i, string str)
{
for(; i < str.Length; i++)
switch(str[i])
{
case '\\':
case '/':
case '.':
continue;
default:
return i;
}
return -1;
}
OK, I cheated, I know in advance what char's I care about.

This works:
public static char FindFirstNotAny(this string value, params char[] charset)
{
return value.TrimStart(charset)[0];
}

If you don't have access to LINQ, I think you may just have to write a static method with a loop (which is probably more efficient than LINQ anyway. Remember the compiler will inline small methods when possible.
The simplest non-LINQ I can come up with is below. I recommend adding braces so scope and the blocks are clear:
public static char? GetFirstChar(string str, char[] list)
{
foreach (char c in str) if (!list.Contains(c)) return c;
return null;
}
With C# 3.0 and LINQ:
char[] list = { 'A', 'B' };
string str = "AABAGAF";
char first = str.ToArray().Where(c => !list.Contains(c)).FirstOrDefault();
In that case, if there is no non-list character, first will equal 0x0000 (or the character null). You could do this:
char? first = str.ToArray().Cast<char?>().Where(
c => !list.Contains(c.Value)).FirstOrDefault();
Then first will be null if there are no matches. This can also be written as:
var query = from char c in str
where !list.Contains(c)
select (char?)c;
char? first = query.FirstOrDefault();

Not all that efficient, but:
char f(string str, IEnumerable<char> list)
{
return str.ToCharArray().First(c => !list.Contains(c))
}

Will this C/C++ example work for you:
char *strToSearch = "This is the one liner you want"
char *skipChars = "Tthise";
size_f numToSkip = strcspn(strToSearch, skipChars);
The strcspn() function scans a string for the complement of the specified set. It returns the number of initial characters that do not include a character in the set.