Hex editor reference material 010 Editor: templates
Having written a couple of articles about the work in the 010 Editor ( part I , part II ), I came to the conclusion that it’s better to start translating the certificate from the program in more detail than basic things, since everything is not bad in it but the Russian translation is not. So, in this article we will get acquainted with the peculiarities of writing templates 010 Editor, which the developers themselves are talking about.
Binary templates are one of the most powerful features of the 010 Editor, which allows you to represent any file as an ordered set of variables and structures. Templates help you explore and edit files in a much more convenient way than regular hex editors allow. Each template is a text file with the extension "* .bt", which can be edited in notepad and right in 010 Editor (see the Templates menu). The template runs as an interpreted code, after it works, the execution result is displayed on the Template Results panel. Templates can be configured to automatically download and run when a file with a certain extension is opened in the editor, for example, the standard set of 010 Editor includes templates for working with * .zip, * .bmp and * .wav.
A variable declaration is identical to C, but with one important difference: whenever a variable is declared in a template, it is “transferred” in its presentation to a set of bytes in the file. For example, consider the pattern:
This pattern will create an array of header characters that will “cover” the first 4 bytes of the file being examined, and the integer variable numRecords will cover the next 4 bytes. Both variables will be displayed in the Template Results panel and can be used for editing directly in the file. The main way to group variables together is to declare structures (structs) and unions.
')
Special attributes can be defined after the declared variable inside the 'corner' ('<' and '>') brackets. The following attributes are supported:
The assignment of all attributes is discussed below, with the exception of the read and write functions (they are discussed in the Custom Varuables item), as well as the size (discussed in the On-Demand Structures section).
By default, all variables displayed in Template Results are in decimal format. To switch between hexadecimal, decimal, octal, and binary, you can use the DisplayFormatDecimal, DisplayFormatHex, DisplayFormatBinary, and DisplayFormatOctal functions. An alternative way is to use the following attribute:
After declaring a variable or typedef. For example:
When parsing a file using templates, you can color the hex code in different colors. For example, a file header may be displayed in a color different from the rest of the file. There are two ways to use “colors. If you want to set the color of a single variable, you can use the attribute '<fgcolor = ???>' or '<bgcolor = ???>' to set the color of the characters and the background, respectively. The argument ('???') can be either a built-in color constant (cBlack, cYellow, cWhile, etc.), or a number in the format '0xBBGGRR' (for example, 0xFF0000 is blue). For example:
The second way is to use the SetForeColor, SetBackColor and SetColor functions. All variables used after one of these functions will be assigned a color defined by them. The constant 'cNone' turns off the use of colors. For example:
The way of reading and writing depends on the order of the bytes in the file: big-endian or little-endian. By default, all declared variables have the same byte order as the file being examined, but the modes can be switched using the BigEndian () and LittleEndian () functions. Use this technique if the file contains data with different byte order.
You can add comments to variables using the '<comment =' '>' attribute. For example:
This comment will be displayed in the Comment column of the Template Results panel. An arbitrary function can also be specified as a comment: '<comment = <function_name >>'. Such a function takes a variable as an argument and returns a string that will be displayed in the Comment column. For example:
The parameter "name" can be used to replace the text displayed in the Name column of the Template Results panel. As well as the comment parameter discussed above, the name parameter can be set using '<name = "">' or it can contain an arbitrary function '<name = <function_name >>'. Using functions in the "name" parameter is equivalent to using the parameter “Comment”: this function takes a variable as an argument and returns a string. For example:
This variable will be displayed in the Template Results panel not as “byte _si8”, but as “Signed Byte”. Note that if a variable is part of an array, array indices will be automatically assigned to the name.
After each variable is declared in the template, the current read address is shifted. The current address can be read using the FTell function, and the movements are processed using FSeek and FSkip. This technique allows you to parse the file without a strictly defined order. Note that you can use the ReadByte, ReadShort, ReadInt, etc. functions to read a file without declaring variables.
In some cases, you have to use variables that should not be displayed in the Template Results panel, and which generally may not relate to the internal structure of the file (that is, be ordinary C / C ++ variables). In this case, the declaration of variables should use the 'local' keyword. For example:
In this example, the variable i is not added to the Template Results pane, but, nevertheless, you can turn on the display of local variables by using the Show Local Variables command (right-click on Template Results).
After starting the template, all created variables are displayed in the tree in the Template Results panel. By default, all arrays and structures are "collapsed" and have the ability to "expand" only by clicking on the '+' in the list; nevertheless, it is sometimes useful to have an expanded structure or a complete list of array elements before your eyes, so that they are displayed in this way in Template Results immediately after the template is completed. To do this, you can use the '<open = true>' attribute in the variable parameters, and the '<open = false>' attribute can also be used to display a certain structure or array collapsed (as set by default).
In 010 Editor, there are two syntax options for working with null-terminated strings:
Both imply reading the string before it hits 0. Unicode strings (long strings) can be read using:
When declaring a variable of type enum (enumeration), an arrow is displayed in the Template Results panel with a drop-down list in which the entire contents of the enumeration will be displayed.
The attribute '<hidden = true>' allows you to hide the display of variables in the Template Results panel, '<hidden = false>' to cancel.
In C, the 'struct' keyword is used to declare a hierarchical data structure; at 010, it has the exact same purpose. Structures are declared using C / C ++ syntax, for example:
This code automatically creates a new type of myStruct, the next step is to declare a variable of this type (structure instance):
After you declare a variable of type myStruct in Template Results, the entry 'myStruct s' appears with a '+' sign. By clicking on '+', you can see the drop-down list of variables and their values: a, b, c.
Instances may also be declared in an alternative way:
This code will create two instances of the myStruct type, with s1 occupying the first 12 bytes of the file (3 integer values of 4 bytes each), and s2 - the next 12 bytes.
These structures are more “powerful” than the structures of the C standard. In 010, the use of expressions such as if, for, while is supported. For example:
In this example, after declaring an instance of the structure s, only two variables are created: a and either b or c. Note that patterns are executed as interpretable code, so all strings are executed sequentially. The value of the variable a is read directly from the file.
Structures can also be nested, for example:
An alternative way to declare structures is typedef. For example:
Associations can be declared in the same way as structures, with the only difference being that the keyword 'union' is used instead of 'struct'. In associations, all variables begin with a “zero position”, that is, from the address of the association itself (in fact, they “overlap” each other). The union size is equal to the maximum by the number of bytes of the variable. For example:
All three variables will be read starting from one address and the union size will be 8 bytes, since it contains a double variable.
When declaring structures or unions after the keyword 'struct' or 'union', you can use an argument list. This is a convenient way to work with complex structures, the list of arguments is determined in the same way as for functions. For example:
After declaring this structure, when creating its instance, you must enclose the corresponding arguments in parentheses, for example:
Arguments can also be used in structures and unions declared as typedefs. For example:
In 010 Editor there is a special syntax that allows you to create arrays in a special way. Variable declaration with the same name (duplicates) is supported, for example:
In 010 Editor, a variable declaration with the same name is equivalent to creating an array. In this example, x [0] can be used as a reference to the first use of x, and x [1] as a reference to the second. Duplicates can be declared in cycles, for example:
010 Editor supports optimization by default, which makes it possible to work with arrays of structures with a large number of elements (for example, a million). Optimization works as follows: the size of the first element of the array is calculated, after which it is assumed that all other elements will be the same size. Accordingly, in the case of elements of variable size, the result will be erroneous. To turn off optimization, use the '<optimize = false>' attribute. For example:
If 010 Editor displays a warning in the Output field, but you still want to use optimization, enable it using the '<optimize = true>' attribute, the warning will be hidden. The attribute '<optimize = false>' can also be used for structures declared not through typedef.
Fields in structures can be divided into groups of bits, this feature allows you to store several variables in a single memory block. The syntax for declaring a bit field is:
The type can be char, short, int, int64 (unsigned or signed) or any other equivalent type. If the name of some variable is not specified, it will be skipped. For example:
Thus, in one four-byte field (32 bits) there will be two variables: alpha and beta, the first one takes 5 bits, the second one - 15 bits, and 12 bits between variables are omitted.
Bitwise structures can be combined with joins by adding a colon character and a value defining the number of bits after its declaration. For example, in order to “pack” two unions into a single ushort, you can do the following:
Some binary formats use variables of specific types that are inconvenient for visual presentation. The 010 Editor has a good mechanism for defining a variable in any way. To do this, you can create a function that translates the value of a variable into a string (after performing a series of transformations), this string will be displayed on the Template Results panel as a variable value. Such is the reading mechanism, the writing mechanism is similarly defined.
To assign a read and write function for a variable, use the attribute '<read = <function_name >> and <write = <name_function >>', respectively. The read function takes a variable as an argument and returns a string; the write function takes as its argument a pointer to the variable and its value in the string representation. For example, to define a “fractional” type using 16 bits (8 high bits define the integer part, and 8 low bits - fractional), you can use the function:
In this example, the FIXEDPT variable can be defined without the record attribute, but in this case it will be read-only.
When displaying arrays in the Template Results panel, usually no values are displayed until the array is “expanded”. In order to determine a value at once for the entire array, you can use an arbitrary variable, read and write functions. For example:
Learn that now the functions of reading and writing can not be defined for individual elements of an array or each element.
A large number of variables can be defined in a template, which is very expensive in terms of system memory. In order to take into account this nuance, you can use On-Demand structures. In this case, the variables within the structure or union are not “transferred” to the examined file until it is necessary (for example, until the structure is expanded in the Template Resilts panel). In order to use On-Demand structures or unions, you must manually specify their sizes in advance: '<size = |'. The size of such structures and associations is always fixed. For example:
In this case, the contents of the MyStruct structure (header, value1, and value2) are not determined until it is necessary.
The 010 Editor templates, despite a number of similarities with ANSI C, have a number of significant differences and are not fully compatible. The list of important differences between ANSI C and 010 Editor templates:
If you have questions, ask, always ready to answer.
The translation may be minor inaccuracies, if you find anything, write to the LAN, I will correct it.
Regards, Dageron .
See also:
Binary templates are one of the most powerful features of the 010 Editor, which allows you to represent any file as an ordered set of variables and structures. Templates help you explore and edit files in a much more convenient way than regular hex editors allow. Each template is a text file with the extension "* .bt", which can be edited in notepad and right in 010 Editor (see the Templates menu). The template runs as an interpreted code, after it works, the execution result is displayed on the Template Results panel. Templates can be configured to automatically download and run when a file with a certain extension is opened in the editor, for example, the standard set of 010 Editor includes templates for working with * .zip, * .bmp and * .wav.
I - Declaring Variables
A variable declaration is identical to C, but with one important difference: whenever a variable is declared in a template, it is “transferred” in its presentation to a set of bytes in the file. For example, consider the pattern:
char header[4]; int numRecords; This pattern will create an array of header characters that will “cover” the first 4 bytes of the file being examined, and the integer variable numRecords will cover the next 4 bytes. Both variables will be displayed in the Template Results panel and can be used for editing directly in the file. The main way to group variables together is to declare structures (structs) and unions.
')
Special Attributes
Special attributes can be defined after the declared variable inside the 'corner' ('<' and '>') brackets. The following attributes are supported:
< format=hex|decimal|octal|binary, // : , , , fgcolor=<color>, // bgcolor=<color>, // comment="<string>"|<function_name>, // name="<string>"|<function_name>, // open=true|false|suppress, // hidden=true|false, // read=<function_name>, // write=<function_name> // size=<number>|<function_name> > //
The assignment of all attributes is discussed below, with the exception of the read and write functions (they are discussed in the Custom Varuables item), as well as the size (discussed in the On-Demand Structures section).
Display mode
By default, all variables displayed in Template Results are in decimal format. To switch between hexadecimal, decimal, octal, and binary, you can use the DisplayFormatDecimal, DisplayFormatHex, DisplayFormatBinary, and DisplayFormatOctal functions. An alternative way is to use the following attribute:
'<format=hex|decimal|octal|binary>'
After declaring a variable or typedef. For example:
int id; int crc <format=hex>; int flags <format=binary>;
Colors
When parsing a file using templates, you can color the hex code in different colors. For example, a file header may be displayed in a color different from the rest of the file. There are two ways to use “colors. If you want to set the color of a single variable, you can use the attribute '<fgcolor = ???>' or '<bgcolor = ???>' to set the color of the characters and the background, respectively. The argument ('???') can be either a built-in color constant (cBlack, cYellow, cWhile, etc.), or a number in the format '0xBBGGRR' (for example, 0xFF0000 is blue). For example:
int id <fgcolor=cBlack, bgcolor=0x0000FF>;
The second way is to use the SetForeColor, SetBackColor and SetColor functions. All variables used after one of these functions will be assigned a color defined by them. The constant 'cNone' turns off the use of colors. For example:
SetForeColor( cRed ); // int first; // int second; // SetForeColor( cNone ); // int third; // -
Byte order (endian)
The way of reading and writing depends on the order of the bytes in the file: big-endian or little-endian. By default, all declared variables have the same byte order as the file being examined, but the modes can be switched using the BigEndian () and LittleEndian () functions. Use this technique if the file contains data with different byte order.
Comments
You can add comments to variables using the '<comment =' '>' attribute. For example:
int machineStatus <comment=" , 15.">;
This comment will be displayed in the Comment column of the Template Results panel. An arbitrary function can also be specified as a comment: '<comment = <function_name >>'. Such a function takes a variable as an argument and returns a string that will be displayed in the Comment column. For example:
int machineStatus <comment=MachineStatusComment>; string MachineStatusComment( int status ) { if( status <= 15 ) return "*** - "; else return " "; }
Names
The parameter "name" can be used to replace the text displayed in the Name column of the Template Results panel. As well as the comment parameter discussed above, the name parameter can be set using '<name = "">' or it can contain an arbitrary function '<name = <function_name >>'. Using functions in the "name" parameter is equivalent to using the parameter “Comment”: this function takes a variable as an argument and returns a string. For example:
byte _si8 <name="Signed Byte">;
This variable will be displayed in the Template Results panel not as “byte _si8”, but as “Signed Byte”. Note that if a variable is part of an array, array indices will be automatically assigned to the name.
Variable reading order
After each variable is declared in the template, the current read address is shifted. The current address can be read using the FTell function, and the movements are processed using FSeek and FSkip. This technique allows you to parse the file without a strictly defined order. Note that you can use the ReadByte, ReadShort, ReadInt, etc. functions to read a file without declaring variables.
Local variables
In some cases, you have to use variables that should not be displayed in the Template Results panel, and which generally may not relate to the internal structure of the file (that is, be ordinary C / C ++ variables). In this case, the declaration of variables should use the 'local' keyword. For example:
local int i, total = 0; int recordCounts[5]; for( i = 0; i < 5; i++ ) total += recordCounts[i]; double records[ total ];
In this example, the variable i is not added to the Template Results pane, but, nevertheless, you can turn on the display of local variables by using the Show Local Variables command (right-click on Template Results).
Variable status
After starting the template, all created variables are displayed in the tree in the Template Results panel. By default, all arrays and structures are "collapsed" and have the ability to "expand" only by clicking on the '+' in the list; nevertheless, it is sometimes useful to have an expanded structure or a complete list of array elements before your eyes, so that they are displayed in this way in Template Results immediately after the template is completed. To do this, you can use the '<open = true>' attribute in the variable parameters, and the '<open = false>' attribute can also be used to display a certain structure or array collapsed (as set by default).
Strings
In 010 Editor, there are two syntax options for working with null-terminated strings:
char str[]; or
string str;
Both imply reading the string before it hits 0. Unicode strings (long strings) can be read using:
wchar_t str[]; or
wstring str;
Transfers
When declaring a variable of type enum (enumeration), an arrow is displayed in the Template Results panel with a drop-down list in which the entire contents of the enumeration will be displayed.
Hidden variables
The attribute '<hidden = true>' allows you to hide the display of variables in the Template Results panel, '<hidden = false>' to cancel.
II - Structures and associations
Structures
In C, the 'struct' keyword is used to declare a hierarchical data structure; at 010, it has the exact same purpose. Structures are declared using C / C ++ syntax, for example:
struct myStruct { int a; int b; int c; };
This code automatically creates a new type of myStruct, the next step is to declare a variable of this type (structure instance):
myStruct s;
After you declare a variable of type myStruct in Template Results, the entry 'myStruct s' appears with a '+' sign. By clicking on '+', you can see the drop-down list of variables and their values: a, b, c.
Instances may also be declared in an alternative way:
struct myStruct { int a; int b; int c; } s1, s2;
This code will create two instances of the myStruct type, with s1 occupying the first 12 bytes of the file (3 integer values of 4 bytes each), and s2 - the next 12 bytes.
These structures are more “powerful” than the structures of the C standard. In 010, the use of expressions such as if, for, while is supported. For example:
struct myIfStruct { int a; if( a > 5 ) int b; else int c; } s;
In this example, after declaring an instance of the structure s, only two variables are created: a and either b or c. Note that patterns are executed as interpretable code, so all strings are executed sequentially. The value of the variable a is read directly from the file.
Structures can also be nested, for example:
struct { int width; struct COLOR { uchar r, g, b; } colors[width]; } line1;
An alternative way to declare structures is typedef. For example:
typedef struct { ushort id; int size; } myData;
Associations
Associations can be declared in the same way as structures, with the only difference being that the keyword 'union' is used instead of 'struct'. In associations, all variables begin with a “zero position”, that is, from the address of the association itself (in fact, they “overlap” each other). The union size is equal to the maximum by the number of bytes of the variable. For example:
union myUnion { ushort s; double d; int i; } u;
All three variables will be read starting from one address and the union size will be 8 bytes, since it contains a double variable.
Structures and unions with arguments
When declaring structures or unions after the keyword 'struct' or 'union', you can use an argument list. This is a convenient way to work with complex structures, the list of arguments is determined in the same way as for functions. For example:
struct VarSizeStruct (int arraySize) { int id; int array[arraySize]; };
After declaring this structure, when creating its instance, you must enclose the corresponding arguments in parentheses, for example:
VarSizeStruct s1(5); VarSizeStruct s2(7);
Arguments can also be used in structures and unions declared as typedefs. For example:
typedef struct (int arraySize) { int id; int array[arraySize]; } VarSizeStruct;
III - Arrays, duplicates, optimization
Arrays and Duplicates
In 010 Editor there is a special syntax that allows you to create arrays in a special way. Variable declaration with the same name (duplicates) is supported, for example:
int x; int y; int x;
In 010 Editor, a variable declaration with the same name is equivalent to creating an array. In this example, x [0] can be used as a reference to the first use of x, and x [1] as a reference to the second. Duplicates can be declared in cycles, for example:
local int i; for( i = 0; i < 5; i++ ) int x;
Structure optimization
010 Editor supports optimization by default, which makes it possible to work with arrays of structures with a large number of elements (for example, a million). Optimization works as follows: the size of the first element of the array is calculated, after which it is assumed that all other elements will be the same size. Accordingly, in the case of elements of variable size, the result will be erroneous. To turn off optimization, use the '<optimize = false>' attribute. For example:
typedef struct { int id; int length; uchar data[ length ]; } RECORD; RECORD record[5] <optimize=false>;
If 010 Editor displays a warning in the Output field, but you still want to use optimization, enable it using the '<optimize = true>' attribute, the warning will be hidden. The attribute '<optimize = false>' can also be used for structures declared not through typedef.
IV - Bitwise structures
Fields in structures can be divided into groups of bits, this feature allows you to store several variables in a single memory block. The syntax for declaring a bit field is:
_ <_> : _
The type can be char, short, int, int64 (unsigned or signed) or any other equivalent type. If the name of some variable is not specified, it will be skipped. For example:
int alpha : 5; int : 12; int beta : 15;
Thus, in one four-byte field (32 bits) there will be two variables: alpha and beta, the first one takes 5 bits, the second one - 15 bits, and 12 bits between variables are omitted.
Bitwise structures can be combined with joins by adding a colon character and a value defining the number of bits after its declaration. For example, in order to “pack” two unions into a single ushort, you can do the following:
enum <ushort> ENUM1 { VAL1_1=25, VAL1_2=29, VAL1_3=7 } var1 : 12; enum <ushort> ENUM2 { VAL2_1=5, VAL2_2=6 } var2 : 4;
V - Arbitrary variables (custom variables)
Some binary formats use variables of specific types that are inconvenient for visual presentation. The 010 Editor has a good mechanism for defining a variable in any way. To do this, you can create a function that translates the value of a variable into a string (after performing a series of transformations), this string will be displayed on the Template Results panel as a variable value. Such is the reading mechanism, the writing mechanism is similarly defined.
To assign a read and write function for a variable, use the attribute '<read = <function_name >> and <write = <name_function >>', respectively. The read function takes a variable as an argument and returns a string; the write function takes as its argument a pointer to the variable and its value in the string representation. For example, to define a “fractional” type using 16 bits (8 high bits define the integer part, and 8 low bits - fractional), you can use the function:
typedef ushort FIXEDPT <read=FIXEDPTRead, write=FIXEDPTWrite>; string FIXEDPTRead( FIXEDPT f ) { string s; SPrintf( s, "%lg", f / 256.0 ); return s; } void FIXEDPTWrite( FIXEDPT &f, string s ) { f = (FIXEDPT)( Atof( s ) * 256 ); }
In this example, the FIXEDPT variable can be defined without the record attribute, but in this case it will be read-only.
When displaying arrays in the Template Results panel, usually no values are displayed until the array is “expanded”. In order to determine a value at once for the entire array, you can use an arbitrary variable, read and write functions. For example:
typedef float VEC3F[3] <read=Vec3FRead, write=Vec3FWrite>; string Vec3FRead( VEC3F v ) { string s; SPrintf( s, "(%f %f %f)", v[0], v[1], v[2] ); return s; } void Vec3FWrite( VEC3F &v, string s ) { SScanf( s, "(%f %f %f)", v[0], v[1], v[2] ); }
Learn that now the functions of reading and writing can not be defined for individual elements of an array or each element.
VI - On-Demand Structures
A large number of variables can be defined in a template, which is very expensive in terms of system memory. In order to take into account this nuance, you can use On-Demand structures. In this case, the variables within the structure or union are not “transferred” to the examined file until it is necessary (for example, until the structure is expanded in the Template Resilts panel). In order to use On-Demand structures or unions, you must manually specify their sizes in advance: '<size = |'. The size of such structures and associations is always fixed. For example:
typedef struct { int header; int value1; int value2; } MyStruct <size=12>;
In this case, the contents of the MyStruct structure (header, value1, and value2) are not determined until it is necessary.
VII - Restrictions
The 010 Editor templates, despite a number of similarities with ANSI C, have a number of significant differences and are not fully compatible. The list of important differences between ANSI C and 010 Editor templates:
Pointers
Currently, the use of '* "pointers is not supported. Only' & 'links are supported for function arguments.Directives
Most preprocessor directives are supported, including #define, #ifdef, #ifndef, etc. However, the #if directive is not currently supported, as is a macro declaration using the #define directive.Multidimensional arrays
Currently, multidimensional arrays are not supported, but there is an alternative way to use them. For example, instead of defining 'float matrix [4] [4]', the same array can be defined as: typedef struct { float row[4]; } MATRIX[4]; MATRIX m;
Transfer of control
Goto is not supported.If you have questions, ask, always ready to answer.
The translation may be minor inaccuracies, if you find anything, write to the LAN, I will correct it.
Regards, Dageron .
See also:
Source: https://habr.com/ru/post/213639/
All Articles