Creating and normalizing dictionaries. Choose the best, remove too much

The use of suitable dictionaries during penetration testing largely determines the success of the selection of credentials. In this publication, I will tell you what modern tools can be used to create dictionaries, optimize them for a particular case and how not to waste time trying to find thousands of deliberately false combinations.

Instruments

crunch

Perhaps one of the most famous tools for quickly creating dictionaries. It is included by default in the popular distribution for the Kali Linux pentest.
')
The tool works in several modes:

Creating a dictionary consisting of the listed characters, such as numbers

crunch 4 5 1234567890 -o all_numbers_from_4_to_5.txt 

Creates a dictionary of four to five digits.

Creating a dictionary pattern

 crunch 10 10 qwe RTY 123 \#\@ -t P^@@,ord%% -o Password_template.txt 

First, the password length is indicated - 10 characters. Then the character sets are listed: lowercase letters, uppercase letters, numbers, and special characters. The -t switch sets the pattern, where

• ^ - special characters
• @ - lowercase letters
• , - uppercase letters
• % - numbers

And the third crunch mode is permutations.

 crunch 1 1 -p Alex Company Position 

The dictionary consists of all possible combinations of the words Alex, Company and Position.

You can learn more about the tool through standard man pages, they are quite detailed.

maskprocessor

Sometimes you need to specify not only sets for a specific type of characters, but generally your own set, including letters, numbers, and special characters. In this case, you can use the maskprocessor utility from the hashcat brute force. You can download it from the official githab hashcat .

You can specify up to four custom character sets and use presets.

 ?l = abcdefghijklmnopqrstuvwxyz ?u = ABCDEFGHIJKLMNOPQRSTUVWXYZ ?d = 0123456789 ?s = !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ ?a = ?l?u?d?s ?b = 0x00 - 0xff 

Usage example

 mp64.bin -1 Pp -2 \@\#\$ ?1assw?2r?d 

Or you can set a set of numbers, but add a few special characters to it so

 mp64.bin -1 Qq -2 ?d\@\#\$ ?1werty_12?2 

We get this result



John the ripper

John the Ripper (JTR), a popular bruteformer, also allows you to generate rule-based dictionaries. This is done using the --rules key, and the rules themselves are described in the john.conf file

Here is the standard rule used to crack NTLM hash

 [List.Rules:NT] : -c T0Q -c T1QT[z0] -c T2QT[z0]T[z1] -c T3QT[z0]T[z1]T[z2] -c T4QT[z0]T[z1]T[z2]T[z3] -c T5QT[z0]T[z1]T[z2]T[z3]T[z4] -c T6QT[z0]T[z1]T[z2]T[z3]T[z4]T[z5] -c T7QT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6] -c T8QT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6]T[z7] -c T9QT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6]T[z7]T[z8] -c TAQT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6]T[z7]T[z8]T[z9] -c TBQT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6]T[z7]T[z8]T[z9]T[zA] -c TCQT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6]T[z7]T[z8]T[z9]T[zA]T[zB] -c TDQT[z0]T[z1]T[z2]T[z3]T[z4]T[z5]T[z6]T[z7]T[z8]T[z9]T[zA]T[zB]T[zC] 

The first line says that you need to change the character register at the zero position (T0), the Q character allows you to prevent duplicates in the resulting dictionary. In the second line, the character in the first position changes its register, then the parentheses specify the preprocessor so that passwords are generated with a modified null character, and so on.

Suppose you successfully performed the brutfors LM hash and got the value QWERTY123, since for LM the register is not important.
But for authorization, you need to perform a NTLM hash brutforce, where case is important. Using the rule described above, you can get the following dictionary

 john -w:QWERTY123.dict --stdout --rules:NT 

JTR by default contains many ready-made rules, but you can write your own ones, or you can take as a basis the already written and adjust for the current situation.
Details about the syntax of the rules can be read here .

hashcat-tools

Another useful tool is a set of utilities from the popular hashcat bruteforcer.

You can download them from the official site .

Consider some of them. Descriptions of all utilities in English can be found here .

combinanor.bin - allows you to generate a dictionary of words from two other dictionaries.



combinanor3.bin does the same thing, but accepts three files as input, instead of two.

combipow.bin - creates all possible combinations of the words listed in the file (similar to the -p switch in crunch)



cutb.bin - cuts words in the dictionary to the specified length. You can specify an offset



expander.bin - receives input words, parses them into characters, combines and sends to STDOUT



permute.bin - creates a dictionary that is used by hashcat during an attack of type Permutation attack . Before using the dictionary you need to skip through the prepare utility.

gate.bin - splits the dictionary into several parts for parallel processing by several cores or several machines. In the example below, we break the standard JTR dictionary into two parts. The first part includes the words numbered 0, 2, 4, 6, .... In the second 1, 3, 5, 7, ...



len.bin - leaves in the dictionary only words of a certain length from min to max



mli2.bin - combines two dictionaries.

req-include.bin is an extremely useful tool that removes everything from the dictionary that does not fit the specified rules. For example, you know that the password policy in the password must contain a letter in upper case, a number and a special character.



The number is selected based on the table.



If you normalize the famous rockyou dictionary in this way, you can reduce its size by 270 times! and do not spend resources on deliberately false combinations.



req-exclude.bin does the same thing as req-include, but exactly the opposite.

rli.bin - this utility removes values ​​from the first dictionary, if they are found in the second. Useful if you create one dictionary of several.

When there are no utilities at hand

It may turn out that there is no possibility to use the hashcat-utils set or crunch, and you need to urgently create a dictionary or normalize it. Some algorithms are quite complex to implement, but basic operations can be performed simply on the command line.

A simple dictionary with dates can be created with a series of similar commands.

 echo 0{1..9}0{1..9}19{60..99} | tr ' ' '\n' >> dates 

If you need to split the dictionary into parts for parallel processing, you can use the split command

 split -d -l 1000 password.lst splitted_ 

You can quickly combine two dictionaries

 cat dict1 dict2 > combined_dict 

To capitalize the first or last letter in each word, you need to execute, respectively, the commands

 sed 's/^./\u&/' dict_file sed 's/.$/\u&/' dict_file 

To translate the register in the lower you need to note "u" to "l"

You can add something to the beginning of each word from the dictionary

 sed 's/^./word/' dict_file 

And so you can add a word to the end

 sed 's/.$/word/' dict_file 

The next command can add to the beginning a number from 0 to 99 to each word in the dictionary.

 for i in $(cat dict_file) ; do seq -f %02.0f$i 0 99 ; done > numbers_dict_file 

You can clear the dictionary from values ​​in which at least 2 numbers are not present

 nawk 'gsub("[0-9]","&",$0)==2' password.lst 

Get



These are just some examples. You can write more complex processing in Python and other scripting languages. But you should always remember that the creation of a high-quality dictionary and its normalization under the target protocol is an important stage in conducting penetration testing.