Regular expressions in Python from simple to complex

Content

Regular Expression Examples

Strength and responsibility

Regular expressions, or shortly, regulars is a very powerful tool. But they should be used with intelligence and caution, and only where they really benefit, not harm. First, poorly written regular expressions are slow. Secondly, they are often very difficult to read, especially if the regular was not written by you five minutes ago. Thirdly, very often even a small change in the task (what is required to be found) leads to a significant change in expression. Therefore, it is often said about regulars that this is write only code (code that is only written from scratch, but does not read and does not rule). As well as joking: Some people, when faced with a problem, think, "I know, I will solve it with regular expressions." Now they have two problems. Here is an example of a regular write-only (to check the validity of an e-mail address (do not do this !!!)):

 (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]| 2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\]) 

Documentation and links

• Original documentation: https://docs.python.org/3/library/re.html ;
• Very detailed and detailed material: https://www.regular-expressions.info/ ;
• Various complex tricks and subtleties with examples: http://www.rexegg.com/ ;
• On-line debugging of regulars https://regex101.com (do not forget to put a tick in Python in the FLAVOR section on the left);
• On-line visualization of regulars https://www.debuggex.com/ (do not forget to choose Python);
• Powerful text editor Sublime text 3 , in which a very convenient search for regulars;

Basic syntax

Templates matching one character

Quantifiers (specifying the number of repetitions)

Greed in regulars and the boundaries of the pattern found

As stated above, the default quantifiers are greedy . This approach solves a very important problem - the template boundary problem. Let's say the \d+ pattern captures as many digits as possible. Therefore, you can be sure that before the pattern found there is not a figure, and after that there is no figure. However, if there are non-greedy parts in the pattern (for example, explicit text), then the substring can be found unsuccessfully. For example, if we want to find “words” starting with , followed by numbers, with the regular \d* , then we will find the wrong patterns:

PA SU13 SU12 TO SU6 END IT IS SUCCESSFUL.

Intersection of substrings

Normally, regulars allow you to find only non-overlapping patterns. Together with the word boundary problem, this makes their use in some cases more difficult. For example, if we decide to search for e-mail addresses using the irregular regular \w+@\w+ (or even better, [\w'._+-]+@[\w'._+-]+ ), then in unsuccessful case we find this:

foo @ boo @ goo @ moo @ roo @ zoo

Sandbox Experiments

1. Find all natural numbers (possibly surrounded by letters);
2. Find all the “words” written in caps (i.e., strictly capitalized), possibly inside real words (aaa BBB vvv);
3. Find the words that have the Russian letter, and sometime the number behind it;
4. Find all words beginning with a Russian or Latin capital letter ( \b - word boundary);
5. Find words that begin with a vowel ( \b - word boundary) ;;
6. Find all natural numbers that are not inside or on the boundary of the word;
7. Find the lines that have a * ( . - this is definitely not the end of the line!);
8. Find the lines that have an opening and sometime then closing brackets;
9. Highlight in one fell swoop the entire piece of the table of contents (at the end of the example, along with the tags);
10. Highlight in one fell swoop only the text part of the table of contents, without tags;
11. Find empty lines;

Regulars in python

Functions for working with regulars live in the re module. Main functions:

An example of using all the basic functions

 import re match = re.search(r'\d\d\D\d\d', r' 123-12-12') print(match[0] if match else 'Not found') # -> 23-12 match = re.search(r'\d\d\D\d\d', r' 1231212') print(match[0] if match else 'Not found') # -> Not found match = re.fullmatch(r'\d\d\D\d\d', r'12-12') print('YES' if match else 'NO') # -> YES match = re.fullmatch(r'\d\d\D\d\d', r'. 12-12') print('YES' if match else 'NO') # -> NO print(re.split(r'\W+', ',  ,  ??!')) # -> ['', '', '', '', '', ''] print(re.findall(r'\d\d\.\d\d\.\d{4}', r'   19.01.2018,     01.09.2017')) # -> ['19.01.2018', '01.09.2017'] for m in re.finditer(r'\d\d\.\d\d\.\d{4}', r'   19.01.2018,     01.09.2017'): print('', m[0], '  ', m.start()) # ->  19.01.2018    20 # ->  01.09.2017    45 print(re.sub(r'\d\d\.\d\d\.\d{4}', r'DD.MM.YYYY', r'   19.01.2018,     01.09.2017')) # ->    DD.MM.YYYY,     DD.MM.YYYY 

The subtleties of screening in python ( '\\\\\\\\foo' )

Since the \ character in Python lines must also be escaped, as a result, constructs of the form '\\\\par' may appear in the templates. The first slash means that the character following it must be left "as is". The third also. As a result, from the point of view of python, '\\\\' means just two slashes \\ . Now from the point of view of the regular expression engine, the first slash escapes the second. Thus, as a template for the regular '\\\\par' means simply the text \par . In order to avoid such a pile of slashes, before the opening quote you need to put the symbol r , which will tell the python "do not consider \ as a shielding character (except when the opening quotation is shielded)". Accordingly, it will be possible to write r'\\par' .

Using additional flags in python

 import re print(re.findall(r'\d+', '12 + ٦٧')) # -> ['12', '٦٧'] print(re.findall(r'\w+', 'Hello, !')) # -> ['Hello', ''] print(re.findall(r'\d+', '12 + ٦٧', flags=re.ASCII)) # -> ['12'] print(re.findall(r'\w+', 'Hello, !', flags=re.ASCII)) # -> ['Hello'] print(re.findall(r'[]+', '    ')) # -> ['', ''] print(re.findall(r'[]+', '    ', flags=re.IGNORECASE)) # -> ['', '', '', ''] text = r"""   1 2 """ print(re.findall(r'.', text)) # -> [] print(re.findall(r'.', text, flags=re.DOTALL)) # -> ['\n'] print(re.findall(r'\w+', text, flags=re.MULTILINE)) # -> ['1', '2'] print(re.findall(r'^\w+', text, flags=re.MULTILINE)) # -> ['2'] 

Writing and Testing Regular Expressions

For writing and testing regular expressions, it is convenient to use the https://regex101.com service (do not forget to put the Python checkbox in the FLAVOR section on the left) or the Sublime text text editor 3 .

Tasks - 1

 C227NA777 
 KU22777 
 T22B7477 
 M227K19U9 
  C227NA777 

 Private 
 Taxi 
 Fail 
 Fail 
 Fail 

  He is a gray-brown-crimson radish !! 
 >>>: -> 
 And not a cat. 
 www.kot.ru 

  9 

  Ivan Ivanovich! 
 Need a response to a letter from ivanoff@ivan-chai.ru. 
 Do not forget to put in a copy 
 serge'o-lupin@mail.ru- this is important. 

  ivanoff@ivan-chai.ru 
 serge'o-lupin@mail.ru 

  NO: foo. @ Ya.ru, foo @ .ya.ru 
 PARTLY: boo @ ya_ru, -boo@ya.ru-, foo№boo@ya.ru 

  boo @ ya 
 boo@ya.ru 
 boo@ya.ru 

Bracket groups (?:...) and enumerations |

Enumerations (OR operation)

To check whether a string satisfies at least one of the patterns, you can use the analogue of the or operator, which is written using the symbol | . , A|B , A B . , |[]|| .

( )

. , MAC- , - : . , 01:23:45:67:89:ab . [0-9a-fA-F] , :
[0-9a-fA-F]{2}[:-][0-9a-fA-F]{2}[:-][0-9a-fA-F]{2}[:-][0-9a-fA-F]{2}[:-][0-9a-fA-F]{2}[:-][0-9a-fA-F]{2}

, .
(?:...) . ?: , , . . , REGEXP — , (?:REGEXP) — . , (?:REGEXP) , , . , MAC-, :
[0-9a-fA-F]{2}(?:[:-][0-9a-fA-F]{2}){5}

(?:...) , . , (?:|) (?:|) « », « », « », « », | | | .

— 2

 !    09:00   
,    09:00:01     . 
 Ps.   25:50  ! 

 !    (TBD)   
,    (TBD)     . 
 Ps.   25:50  ! 

 
 1.2 
   one. 
    1.0e-55  
      e-12   
  6.5E 
        1e-12  
  +4.1234567890E-99999           
  7.6e+12.5 
    99 

 
1.2 is legal. 
1. is illegal. 
1.0e-55 is legal. 
e-12 is illegal. 
6.5E is illegal. 
1e-12 is legal. 
+4.1234567890E-99999 is legal. 
7.6e+12.5 is illegal. 
99 is illegal. 

       , 
      

(...) match -

Match-

re.search , re.fullmatch , None , re.finditer . , match -. . , .

(...)

(...) ?: , . match-, re.search , re.fullmatch re.finditer , , . , (...) , . .

 import re pattern = r'\s*([--]+)(\d+)\s*' string = r'--- 45 ---' match = re.search(pattern, string) print(f'  >{match[0]}<   {match.start(0)}  {match.end(0)}') print(f'  >{match[1]}<   {match.start(1)}  {match.end(1)}') print(f'  >{match[2]}<   {match.start(2)}  {match.end(2)}') ### # ->   > 45 <   3  16 # ->   ><   6  11 # ->   >45<   11  13 

.

( ), match- . , '\s*([--])+(\d)+\s*' , :

 # ->   > 45 <   3  16 # ->   ><   10  11 # ->   >5<   12  13 

. .

 import re pattern = r'((\d)(\d))((\d)(\d))' string = r'123456789' match = re.search(pattern, string) print(f'  >{match[0]}<   {match.start(0)}  {match.end(0)}') for i in range(1, 7): print(f' №{i} >{match[i]}<   {match.start(i)}  {match.end(i)}') ### # ->   >1234<   0  4 # ->  №1 >12<   0  2 # ->  №2 >1<   0  1 # ->  №3 >2<   1  2 # ->  №4 >34<   2  4 # ->  №5 >3<   2  3 # ->  №6 >4<   3  4 

re.findall

, , . , (?:...) .

 import re print(re.findall(r'([az]+)(\d*)', r'foo3, im12, go, 24buz42')) # -> [('foo', '3'), ('im', '12'), ('go', ''), ('buz', '42')] 

re.split

, re.split str.split . , .

 import re print(re.split(r'(\s*)([+*/-])(\s*)', r'12 + 13*15 - 6')) # -> ['12', ' ', '+', ' ', '13', '', '*', '', '15', ' ', '-', ' ', '6'] 

 import re print(re.split(r'\s*([+*/-])\s*', r'12 + 13*15 - 6')) # -> ['12', '+', '13', '*', '15', '-', '6'] 

( re.sub , , ) : \1, \2, \3, ... . , // .., :

 import re text = "We arrive on 03/25/2018. So you are welcome after 04/01/2018." print(re.sub(r'(\d\d)/(\d\d)/(\d{4})', r'\2.\1.\3', text)) # -> We arrive on 25.03.2018. So you are welcome after 01.04.2018. 

9, \g<12> .

: re.sub , match- , . , . , «» , «»:

 import re def repl(m): return '>censored(' + str(len(m[0])) + ')<' text = "   : , , ." print(re.sub(r'\b[xX]\w*', repl, text)) # ->  >censored(7)<  : >censored(3)<, >censored(7)<, >censored(15)<. 

\1, \2, \3, ... \g<12> . , xml html.

, xml html ! . . , html , . « html, », . html , :) html . html xml : html xml , , .

lxml beautiful soup .

 import re text = "SPAM <foo>Here we can <boo>find</boo> something interesting</foo> SPAM" print(re.search(r'<(\w+?)>.*?</\1>', text)[0]) # -> <foo>Here we can <boo>find</boo> something interesting</foo> text = "SPAM <foo>Here we can <foo>find</foo> OH, NO MATCH HERE!</foo> SPAM" print(re.search(r'<(\w+?)>.*?</\1>', text)[0]) # -> <foo>Here we can <foo>find</foo> 

— 3

   12   
 410.37 . 

   1728   
 68921000.50653 . 

    , 
    

, ,

, . , , «» .

,

, ( lookaround Co)

, , ,
match-.

. , . , ( AND - ).

lookaround

(?=VI) — , VI

IV, IX, V, VI, VII, VIII,
IX, VI, VII, VIII, X, ..., XVIII,
I, II, III, IV, V, VI

IV, IX, V, VI, VII, VIII,
IX, VI, VII, VIII, X, ..., XVIII,
I, II, III, IV, V, VI

IV, IX, V, VI, VII, VIII,
IX, VI , VI I, VI II, X, ..., XVIII,
I, II, III, IV, V, VI

IV, IX, V, VI , VI I, VI II,
IX, VI, VII, VIII, X, ..., X VI II,
I, II, III, IV, V, VI

, , , , . . , .
Perl — :)

— 4

 MyVar17 = OtherVar + YetAnother2Var 
TheAnswerToLifeTheUniverseAndEverything = 42 

 my_var17 = other_var + yet_another2_var 
the_answer_to_life_the_universe_and_everything = 42 

   ,   ! 

 ,    

 12  
 123 
1234  
12354  
 123456 
 1234567 

 12  
 123 
1,234  
12,354  
 123,456 
 1,234,567 

   
  ...   
  ! ? , !  And this 

  . 

     ... 
    ! 
? 
, ! 
    . 

 +7 123 456-78-90 

 +7 123 456-78-90 

 8(123)456-78-90 

 +7 123 456-78-90 

 7(123) 456-78-90 

 +7 123 456-78-90 

  1234567890 

 +7 123 456-78-90 

 123456789 

 Fail! 

 +9 123 456-78-90 

 Fail! 

 +7 123 456+78=90 

 Fail! 

 +7(123 45678-90 

 +7 123 456-78-90 

 8(123 456-78-90 

 Fail! 

  ! 
     ivanoff@ivan-chai.ru. 
     
serge'o-lupin@mail.ru-  . 

 ivanoff@ivan-chai.ru 
serge'o-lupin@mail.ru 

 NO: foo.@ya.ru, foo@.ya.ru, foo@foo@foo
NO: +foo@ya.ru, foo@ya-ru
NO: foo@ya_ru, -foo@ya.ru-, foo@ya.ru+
NO: foo..foo@ya.ru 
YES: (boo1@ya.ru), boo2@ya.ru!, boo3@ya.ru 

 boo1@ya.ru 
boo2@ya.ru 
boo3@ya.ru 

Post scriptum