Develop Reference

Is it possible to have overlapping regex matches?

Take this data as an example:
ID: JK546|Guitar: 0|Piano: 1|Violin: 0|Expiry: Aug14,2021
I was wondering if it's possible to create a regex that will return this set of matches
ID: JK546|Guitar: 0|Expiry: Aug14,2021
ID: JK546|Piano: 1|Expiry: Aug14,2021
ID: JK546|Violin: 0|Expiry: Aug14,2021
I did try creating one below:
ID: (?<id>\w+).*\|(?<instrument>\w+):\s(?<count>\d).*Expiry:\s(?<expiry>[\w\d]+)
but it only returned the one with the violin instrument. I would highly appreciate your insights on this.

I would not use a regular expression. Especially since the string ID: JK546|Guitar: 0|Expiry: Aug14,2021 does not appear in the string ID: JK546|Guitar: 0|Piano: 1|Violin: 0|Expiry: Aug14,2021, so it's not strictly a match, but more of a replacement. But there's no good way to get all replacements from all matches.
So, I'd just split the input string on |.
Then you want to compose a result string that is comprised of the first field, one of the middle fields, and the last field. You'll get one result for each middle field that exists. If it splits into N fields, you'll get N-2 results. e.g.: if it splits into 5 fields, then you'll get 3 results, one for each of the "middle" fields.
string input = "ID: JK546|Guitar: 0|Piano: 1|Violin: 0|Expiry: Aug14,2021";
string[] fields = input.Split('|');
for( int i = 1; i < fields.Length - 1; ++i) {
string result = string.Join("|", fields.First(), fields[i], fields.Last());
Console.WriteLine(result);
}
output:
ID: JK546|Guitar: 0|Expiry: Aug14,2021
ID: JK546|Piano: 1|Expiry: Aug14,2021
ID: JK546|Violin: 0|Expiry: Aug14,2021

A single regular expression to return multiple matches on multiple calls? 
I wonder whether that is possible.
I’m not familiar with how to do regex processing in C#,
but this sed command will do what you want. 
Perhaps you can understand how it works and adapt it to your needs:
sed -n ':loop; h; s/^\([^|]*|[^|]*\).*\(|.*\)$/\1\2/p; g; s/^\([^|]*\)|[^|]*\(|.*\)$/\1\2/; t loop'
For simplicity, let’s pretend that the input string is “A|B|C|D|E”.
What it does:
-n is the option to tell sed not to print anything automatically
(but only print when told to, with a p command).
:loop is a label for, effectively, a “goto”. 
So use a while loop structure.
h saves the pattern space into the hold space. 
In other words, make a copy of your string.
s/^\([^|]*|[^|]*\).*\(|.*\)$/\1\2/p captures the first two segments
and the last one, and prints the result. 
So “A|B|C|D|E” becomes “A|B|E” (i.e., your first desired output).
g restores the saved string from the hold space into the pattern space. 
In other words, retrieve the copy of the string that you saved.
s/^\([^|]*\)|[^|]*\(|.*\)$/\1\2/ captures the first segment,
skips the second, and then captures the rest. 
So “A|B|C|D|E” becomes “A|C|D|E”.
t loop is the “goto” command. 
It says to go back to the beginning of the loop
if the most recent substitution succeeded. 
In other words, this is the end of the loop,
and the specification of the loop condition.
The second iteration of the loop will change “A|C|D|E” to “A|C|E”
and print it. 
And then change “A|C|D|E” to “A|D|E” and iterate. 
The third iteration of the loop will change “A|D|E” to “A|D|E” and print it. 
(Obviously there is no change, because the .* in the middle of the regex
matches the zero-length string between “A|D” and “|E”.) 
The final substitution changes “A|D|E” to “A|E”,
and then there is nothing left to find.

You can make use of the .NET Groups.Captures property to get the values of Guitar, Piano and Violin.
(ID: \w+\|)(\w+: \d+\|)+(Expiry: \w+,\d+)
The pattern matches:
(ID: \w+\|) Capture group 1 match ID: 1+ word chars and |
(\w+: \d+\|)+ Capture group 2 Repeat 1+ times matching 1+ word chars : 1+ digits |
(Expiry: \w+,\d+) Capture group 3 match Expiry: 1+ word chars , and 1+ digits
See a .NET regex demo | C# demo
For example
var str = "ID: JK546|Guitar: 0|Piano: 1|Violin: 0|Expiry: Aug14,2021";
string pattern = #"(ID: \w+\|)(\w+: \d+\|)+(Expiry: \w+,\d+)";
Match m = Regex.Match(str, pattern);
foreach(Capture c in m.Groups[2].Captures) {
Console.WriteLine(m.Groups[1].Value + c.Value + m.Groups[3].Value);
}
Output
ID: JK546|Guitar: 0|Expiry: Aug14,2021
ID: JK546|Piano: 1|Expiry: Aug14,2021
ID: JK546|Violin: 0|Expiry: Aug14,2021

It should be possible with look behind and look ahead:
string foo = #"ID: JK546 | Guitar: 0 | Piano: 1 | Violin: 0 | Expiry: Aug14,2021";
// First look at "Guitar: 0", "Piano: 1" and "Violin: 0". Then look behind "(?<= )" and search for the ID. Then look ahead "(?= )" and search for Expiry.
string pattern = #"(\w+: \d)(?<=(ID: [A-Z0-9]+).*?)(?=.*?(Expiry: \S+))";
foreach (var match in Regex.Matches(foo, pattern))
{
....
}
Fortunately c# is one of the few languages that can handle variable length look behinds.

Regular expression to match optional group

I'm expecting a string that may or may not have '#' followed by some numbers. If there's any numbers after '#' I want to capture it as second group, otherwise just capture everything in first group. Following is example for C#
ABC#99999//match Group1: ABC and Group2: 99999
9ABC#8 //match Group1 9ABC and Group
9ABC //match Group1 9ABC
9ABC# //match Group1 9ABC#
The following Regex works, but for 3rd and 4th string, it captures into group 3 instead of group 1. is there a better way for the above scenario?
(?:(.+)#(\d+))|(.+)
Alternatively, I came up with following Regex, but the problem is since the first group doesn't have a fixed format (like length), it captures whole string from 1st and second string instead of capturing 2 groups
(.+)(?:#(\d+))?

To match a whole string and place the part before an optional # + digits into Group 1 and all those digits into Group 2, you may use
^(.+?)(?:#(\d+))?$
See the .NET regex demo. The \r? is added because it is a multiline input demo, you won't need it if you plan to test separate strings against the pattern.
Details
^ - start of string
(.+?) - Group 1: one or more chars as few as possible (due to +? lazy quantifier) (NOTE that if Group 1 value may be missing, use (.*?) instead)
(?:#(\d+))? - an optional non-capturing group matching 1 or 0 occurrences of
# - a # symbol
(\d+) - Group 2: one or more digits
$ - end of string.

Try
(\w+)(?:#(\d+))?
The # is in an optional non-capturing group, and the following digits in a captured group inside the non-capturing group.
https://regex101.com/r/obKPFw/1

An alternative solution, WITHOUT using regular expressions:
public class Program
{
static void Main(string[] args)
{
List<string> inputs = new List<string>
{
"ABC#99999",
"9ABC#8",
"9ABC",
"9ABC#"
};
var groups = new List<Group>();
foreach (string input in inputs)
{
string[] parts = input.Split("#", StringSplitOptions.RemoveEmptyEntries);
var group = new Group
{
Part1 = input
};
if (parts.Length == 2)
{
group.Part1 = parts[0];
group.Part2 = parts[1];
};
groups.Add(group);
Console.WriteLine($"Input: '{input}': {group}");
}
Console.ReadKey();
}
}
public class Group
{
public string Part1 { get; set; }
public string Part2 { get; set; }
/// <inheritdoc />
public override string ToString()
{
return $"Part1: {Part1 ?? "null"}, Part2: {Part2 ?? "[null]"}";
}
}
Output:
Input: 'ABC#99999': Part1: ABC, Part2: 99999
Input: '9ABC#8': Part1: 9ABC, Part2: 8
Input: '9ABC': Part1: 9ABC, Part2: [null]
Input: '9ABC#': Part1: 9ABC#, Part2: [null]

Regex - Match versus Groups

I am sorry in advance if this would fall under duplicates but I could not see these answered my questions.
Could you please help and explain:
Where is the match or capture only for name held? The initial part of the pattern [A-Za-z0-9_\-\.]+ does not show it between brackets so I understand it won't be a group, how then is name captured and held as a component of Match 0?
If I replace the string t2 to name#domain.com alt#yahoo.net and pattern to ^([A-Za-z0-9_\-\.\ ]+#(([A-Za-z0-9\-])+\.)+([A-Za-z\-])+)+$
I would expect 2 matches: One for each full email address. Output only shows 1 match holding both separated by a space, why?
How should the pattern read to get 2 matches or would the string need to be different for this pattern?
I don't see the consistency in the Group output because it does not show another Group holding capture 0=com and capture 1=net, similarly to Group 2 holding domain. and yahoo. captures, why?
Group 3 captures seem to hold the captures of the Group 2 Capture 0 and 1, is that how hierarchies work, there are captures of captures of groups?
Code
static void Main(string[] args)
{
string t2 = "name#domain.com";
string p2 = #"^[A-Za-z0-9_\-\.\ ]+#(([A-Za-z0-9\-])+\.)+([A-Za-z\-])+$";
MatchCollection matches = Regex.Matches(t2, p2);
GroupCollection gc;
int groupIndex = 0;
int matchIndex = 0;
int captureIndex = 0;
foreach (Match nextMatch in matches)
{
gc = nextMatch.Groups;
Console.WriteLine("Match {0} holds: {1}", matchIndex, nextMatch.Value);
matchIndex++;
foreach (Group g in gc)
{
Console.WriteLine("Group {0} holding: {1}", groupIndex, g.ToString());
groupIndex++;
foreach (Capture capture in g.Captures)
{
Console.WriteLine("\tCapture {0} holds {1}", captureIndex, capture.ToString());
captureIndex++;
}
captureIndex = 0;
}
groupIndex = 0;
}
matchIndex = 0;
}
Output for the above code:
Match 0 holds: name#domain.com
Group 0 holding: name#domain.com
Capture 0 holds name#domain.com
Group 1 holding: domain.
Capture 0 holds domain.
Group 2 holding: n
Capture 0 holds d
Capture 1 holds o
Capture 2 holds m
Capture 3 holds a
Capture 4 holds i
Capture 5 holds n
Group 3 holding: m
Capture 0 holds c
Capture 1 holds o
Capture 2 holds m
Press any key to continue . . .
Output if string t2 = "name#domain.com alt#yahoo.net"; and string p2 = #"^([A-Za-z0-9_\-\.\ ]+#(([A-Za-z0-9\-])+\.)+([A-Za-z\-])+)+$";
Match 0 holds: name#domain.com alt#yahoo.net
Group 0 holding: name#domain.com alt#yahoo.net
Capture 0 holds name#domain.com alt#yahoo.net
Group 1 holding: alt#yahoo.net
Capture 0 holds name#domain.com
Capture 1 holds alt#yahoo.net
Group 2 holding: yahoo.
Capture 0 holds domain.
Capture 1 holds yahoo.
Group 3 holding: o
Capture 0 holds d
Capture 1 holds o
Capture 2 holds m
Capture 3 holds a
Capture 4 holds i
Capture 5 holds n
Capture 6 holds y
Capture 7 holds a
Capture 8 holds h
Capture 9 holds o
Capture 10 holds o
Group 4 holding: t
Capture 0 holds c
Capture 1 holds o
Capture 2 holds m
Capture 3 holds n
Capture 4 holds e
Capture 5 holds t
Press any key to continue . . .

The Match covers the matching of the entire regex. The regex can be applied to the given string.
Groups are part of that Match and Captures are (if you specified multiple occurences of a group like (someRegex)+ ) all Captures of that Group. Try changing ([A-Za-z\-])+ to ([A-Za-z\-]+) and see the difference!
Examples:
\w*(123)\w* on "asdsa123asdf"
Match -> asdsa123asdf
Group -> 123 (== last capture)
Captures -> 123
\w*([123])+\w* on "asdsa123asdf"
Match -> asdsa123asdf
Group -> 3 (== last capture)
Captures -> 1, 2, 3
There are multiple sites to test and show details of your regex, i.e. https://regexr.com or https://regex101.com

Need multiple regular expression matches using C#

So I have this list of flight data and I need to be able to parse through it using regular expressions (this isn't the entire list).
1 AA2401 F7 A4 Y7 B7 M7 H7 K7 /DFW A LAX 4 0715 0836 E0.M80 9 3:21
2 AA2421 F7 A1 Y7 B7 M7 H7 K7 DFWLAX 4 1106 1215 E0.777 7 3:09
3UA:US6352 B9 M9 H9 K0 /DFW 1 LAX 1200 1448 E0.733 1:48
For example, I might need from the first line 1, AA, 2401, and so on and so on. Now, I'm not asking for someone to come up with a regular expression for me because for the most part I'm getting to where I can pretty much handle that myself. My issue has more to do with being able to store the data some where and access it.
So I'm just trying to initially just "match" the first piece of data I need, which is the line number '1'. My "pattern" for just getting the first number is: ".?(\d{1,2}).*" . The reason it's {1,2} is because obviously once you get past 10 it needs to be able to take 2 numbers. The rest of the line is set up so that it will definitely be a space or a letter.
Here's the code:
var assembly = Assembly.GetExecutingAssembly();
var textStreamReader = new StreamReader(
assembly.GetManifestResourceStream("FlightParser.flightdata.txt"));
List<string> lines = new List<string>();
do
{
lines.Add(textStreamReader.ReadLine());
} while (!textStreamReader.EndOfStream);
Regex sPattern = new Regex(#".?(\d{1,2}).*");//whatever the pattern is
foreach (string line in lines)
{
System.Console.Write("{0,24}", line);
MatchCollection mc = sPattern.Matches(line);
if ( sPattern.IsMatch(line))
{
System.Console.WriteLine(" (match for '{0}' found)", sPattern);
}
else
{
System.Console.WriteLine();
}
System.Console.WriteLine(mc[0].Groups[0].Captures);
System.Console.WriteLine(line);
}//end foreach
System.Console.ReadLine();
With the code I'm writing, I'm basically just trying to get '1' into the match collection and somehow access it and write it to the console (for the sake of testing, that's not the ultimate goal).

Your regex pattern includes an asterisk which matches any number of characters - ie. the whole line. Remove the "*" and it will only match the "1". You may find an online RegEx tester such as this useful.

Assuming your file is not actually formatted as you posted and has each of the fields separated by something, you can match the first two-digit number of the line with this regex (ignoring 0 and leading zeros):
^\s*([1-9]\d?)
Since it is grouped, you can access the matched part through the Groups property of the Match object.
var line = "12 foobar blah 123 etc";
var re = new Regex(#"^\s*([1-9]\d?)");
var match = re.Match(line);
if (match.Success)
{
Console.WriteLine(match.Groups[1].Value); // "12"
}
else
{
Console.WriteLine("No match");
}

The following expression matches the first digit, that you wanted to capture, in the group "First".
^\s*(?<First>\d{1})
I find this regular expression tool highly useful when dealing with regex. Give it a try.
Also set RegexOption to Multiline when you are making the match.

How does this regex find triangular numbers?

Part of a series of educational regex articles, this is a gentle introduction to the concept of nested references.
The first few triangular numbers are:
1 = 1
3 = 1 + 2
6 = 1 + 2 + 3
10 = 1 + 2 + 3 + 4
15 = 1 + 2 + 3 + 4 + 5
There are many ways to check if a number is triangular. There's this interesting technique that uses regular expressions as follows:
Given n, we first create a string of length n filled with the same character
We then match this string against the pattern ^(\1.|^.)+$
n is triangular if and only if this pattern matches the string
Here are some snippets to show that this works in several languages:
PHP (on ideone.com)
$r = '/^(\1.|^.)+$/';
foreach (range(0,50) as $n) {
if (preg_match($r, str_repeat('o', $n))) {
print("$n ");
}
}
Java (on ideone.com)
for (int n = 0; n <= 50; n++) {
String s = new String(new char[n]);
if (s.matches("(\\1.|^.)+")) {
System.out.print(n + " ");
}
}
C# (on ideone.com)
Regex r = new Regex(#"^(\1.|^.)+$");
for (int n = 0; n <= 50; n++) {
if (r.IsMatch("".PadLeft(n))) {
Console.Write("{0} ", n);
}
}
So this regex seems to work, but can someone explain how?
Similar questions
How to determine if a number is a prime with regex?

Explanation
Here's a schematic breakdown of the pattern:
from beginning…
| …to end
| |
^(\1.|^.)+$
\______/|___match
group 1 one-or-more times
The (…) brackets define capturing group 1, and this group is matched repeatedly with +. This subpattern is anchored with ^ and $ to see if it can match the entire string.
Group 1 tries to match this|that alternates:
\1., that is, what group 1 matched (self reference!), plus one of "any" character,
or ^., that is, just "any" one character at the beginning
Note that in group 1, we have a reference to what group 1 matched! This is a nested/self reference, and is the main idea introduced in this example. Keep in mind that when a capturing group is repeated, generally it only keeps the last capture, so the self reference in this case essentially says:
"Try to match what I matched last time, plus one more. That's what I'll match this time."
Similar to a recursion, there has to be a "base case" with self references. At the first iteration of the +, group 1 had not captured anything yet (which is NOT the same as saying that it starts off with an empty string). Hence the second alternation is introduced, as a way to "initialize" group 1, which is that it's allowed to capture one character when it's at the beginning of the string.
So as it is repeated with +, group 1 first tries to match 1 character, then 2, then 3, then 4, etc. The sum of these numbers is a triangular number.
Further explorations
Note that for simplification, we used strings that consists of the same repeating character as our input. Now that we know how this pattern works, we can see that this pattern can also match strings like "1121231234", "aababc", etc.
Note also that if we find that n is a triangular number, i.e. n = 1 + 2 + … + k, the length of the string captured by group 1 at the end will be k.
Both of these points are shown in the following C# snippet (also seen on ideone.com):
Regex r = new Regex(#"^(\1.|^.)+$");
Console.WriteLine(r.IsMatch("aababc")); // True
Console.WriteLine(r.IsMatch("1121231234")); // True
Console.WriteLine(r.IsMatch("iLoveRegEx")); // False
for (int n = 0; n <= 50; n++) {
Match m = r.Match("".PadLeft(n));
if (m.Success) {
Console.WriteLine("{0} = sum(1..{1})", n, m.Groups[1].Length);
}
}
// 1 = sum(1..1)
// 3 = sum(1..2)
// 6 = sum(1..3)
// 10 = sum(1..4)
// 15 = sum(1..5)
// 21 = sum(1..6)
// 28 = sum(1..7)
// 36 = sum(1..8)
// 45 = sum(1..9)
Flavor notes
Not all flavors support nested references. Always familiarize yourself with the quirks of the flavor that you're working with (and consequently, it almost always helps to provide this information whenever you're asking regex-related questions).
In most flavors, the standard regex matching mechanism tries to see if a pattern can match any part of the input string (possibly, but not necessarily, the entire input). This means that you should remember to always anchor your pattern with ^ and $ whenever necessary.
Java is slightly different in that String.matches, Pattern.matches and Matcher.matches attempt to match a pattern against the entire input string. This is why the anchors can be omitted in the above snippet.
Note that in other contexts, you may need to use \A and \Z anchors instead. For example, in multiline mode, ^ and $ match the beginning and end of each line in the input.
One last thing is that in .NET regex, you CAN actually get all the intermediate captures made by a repeated capturing group. In most flavors, you can't: all intermediate captures are lost and you only get to keep the last.
Related questions
(Java) method matches not work well - with examples on how to do prefix/suffix/infix matching
Is there a regex flavor that allows me to count the number of repetitions matched by * and + (.NET!)
Bonus material: Using regex to find power of twos!!!
With very slight modification, you can use the same techniques presented here to find power of twos.
Here's the basic mathematical property that you want to take advantage of:
1 = 1
2 = (1) + 1
4 = (1+2) + 1
8 = (1+2+4) + 1
16 = (1+2+4+8) + 1
32 = (1+2+4+8+16) + 1
The solution is given below (but do try to solve it yourself first!!!!)
(see on ideone.com in PHP, Java, and C#):
^(\1\1|^.)*.$