Creating stones for games using photogrammetry
Sebastian van Elverding talks about his approach to creating stunning three-dimensional stones and rocks with the help of many photographs. More information about him can be found in the tutorial on gumroad.com/sebvhe .
Hi, my name is Sebastian, I live in Brussels, the birthplace of the best french fries, beer and chocolate! I am currently working as an environmental artist at Starbreeze Studios (Stockholm). I used to work in Playground Games on Forza Horizon 3, which became for me an excellent opportunity to apply photogrammetry in an AAA-class game.
')
Over the past four years I have experimented a lot with photogrammetry. Pretty quickly, I started working on textures and materials that I knew little about. About a year ago I released a tutorial on how to create textures using photogrammetry.
Although my workflow has changed a bit since the writing, the tutorial still applies to what I'm going to tell you. If you find something incomprehensible in this interview, then most likely you will find a detailed explanation in the tutorial.
My very first seamless scan of 2014 (left) and one of the latest scans (right)
Usually, people perceive photogrammetry as a way to transfer static scanned grids (mesh) to a game. When working with scanning environments, they always deal with huge unique textures with a side of 4-16 kilopixels on unique, non-repeating grids. A good example is the UE4 Kite demo. But what is suitable for cinematic clips like Kite will be completely inapplicable for use in real time, especially on large surfaces.
This was important to me when I was working on the UE4 Marketplace Rock Texture Set. Imagine that we are making a small rocky canyon. It will be an error to scan a set of 4-8 large rocks, from which you will then try to build a canyon. As a result, you will most likely get four unique textures with a side of 4 kilopixels. Instead, I strive to create one very good seamless stone texture, writing beautiful shapes to its height map. Then I just create a very simple canyon grid, for which I apply displacement based on the resulting texture. This way I get a whole scene in which only one 4k texture is used for the rocks.
Quickly created “envelope grid” of the canyon with displacement in UE4, one seamless stone texture is used and some snow for beauty.
You can without any problems perform the advance in advance in the 3D-editor based on the height map, and then import the optimized grid instead of using tessellation. The choice depends on the importance of the environment and the resources available.
I believe that this method takes much less memory, takes an incredibly little time (compared to scanning multiple grids) and allows you to perform design iterations much faster. If you want to turn the scene into a desert canyon, then it will be enough just to scan one texture of the desert rock.
Just imagine how many grids would have to be scanned to replace the biome, and in our case only one texture is required (plus snow, sand or moss)
However, it is wise to approach the concealment or removal of seams. My tessellation material allows you to remove 100% of the seams.
In the end, photogrammetry can be very flexible, you only need to think about it outside the traditional "hard" use.
In fact, besides the camera and computer, you will not need anything else. A good analogy: to play the guitar, you do not need Gibson for $ 2000, with a simple guitar for $ 100 you can achieve a lot. Ultimately, mastery means more than equipment. When working with photogrammetry, I use very simple equipment. I would love to “upgrade”, but for now I think that the limitations of bad equipment helped me find clever ways to improve the poor quality of scans.
I spend all my scanning work with the help of the Canon 100D / Rebel SL1, one of the cheapest DSLRs on the market, it costs only about $ 400.
However, it will take some effort to improve the quality of scans.
And now about the equipment that I do not use:
Let's talk about computers: if you have a standard gaming desktop computer, then there will be no problems with processing. I recommend to have at least 16 GB of RAM and a decent video card. I use the GTX670 and 16 GB of RAM, a fairly standard set. Probably the bottleneck will be memory.
Good lighting is very difficult to achieve, because on the street you can not control it and there are no other options than to wait for the right moment. In general, two points should be avoided:
This is one of the worst examples.
Your goal is a cloudy sky, both uniform and very bright. Do not wait too long, the probability of getting it is not very great!
The number of shots strongly depends on the software used (about it a little later), the size of the scanned surface and the desired level of detail. It is always better to take more pictures than you need.
Of course, overlay is crucial. Photogrammetric software compares the features in different images to determine the spatial position of frames. Therefore, several pictures should have the same parts, the more the better.
So, you want to scan the surface found, where to start?
Remember, I said that I can refuse a tripod, because I am changing a narrow diaphragm at a high shutter speed? I can afford a narrow depth of field almost without blurring, because I always shoot from top to bottom (but here you have to be careful not to get into the frame with your shoes!). This not only provides the best results, but also means that the object will be almost at one distance from the camera throughout the frame, reducing the amount of blurring of the depth of field of sharpness. Of course, this refers to scanning relatively flat surfaces.
Source: Agisoft User Guide
And now about the overlay: imagine yourself a Google Earth mapper, first you want to capture the whole Earth, the whole object in the first pictures, and then add different levels of detail. First, take a global picture, at least eight, better more. Perhaps they can not be done from the top down, do not worry, just move through the object, and be careful with the depth of field. Now you have on each snapshot all the characteristic features of the surface to which the following, more accurate snaps will be attached.
Next, you need to create the first zoom level: take pictures from top to bottom, about 1/4 the size of the surface. Make sure they overlap each other by more than 50%. Repeat again for 1/8 surface, 1/16, etc ... It all depends on the level of detail you need.
Such an algorithm of work ensures that close pictures do not mix with their neighbors, for binding they will be able to rely on the previous level of detail.
Of course, in reality, you are doing everything by the eye, no need to carry a tape measure. Moreover, this principle does not always work, for example, if you scan sand, then it cannot be stepped on during the scanning process. But it's a good idea to always remember this principle.
In this example, the general images are highlighted in blue, the first level of images from top to bottom is shown in green, and the other detailed view from top to bottom - in red.
There are two main competitors, Agisoft Photoscan and a fairly new Reality Capture package. I mainly used Photoscan, but recently I started using Reality Capture. Not to say that one is much better than the other in some ways, but lately the industry seems to be leaning towards Reality Capture.
Here are my pros and cons of each package. Do not forget, I am still rather inexperienced in Reality Capture and I can be mistaken about something.
I think it is worth advising beginners to start with Reality Capture, it is more tolerant of bad pictures, besides, you can compensate for poor-quality pictures with good image quality, which Agisoft may not provide on your computer.
However, if you plan to work with photogrammetry only from time to time, then it will probably be cheaper to choose a standalone license from Agisoft.
A stunning, ultra-detailed scan may not look beautiful in the game unless you take care of a good artistic vision in advance.
The most important and difficult is tiling. He should always be remembered, especially when searching for surfaces to scan. Many people ask me: “Is there a faster way to place textures and is it possible to automate it, for example, using Substance?”
In 90% of cases there is no such method.
I think the belief in the existence of such a method arose from the belief that paving is the removal of seams. Yes, it probably is, but it’s less than 10% of the texture merging process. The real job is to balance the frequency and characteristic features of your surface, to make the texture look good even with repeated repetitions. In most cases, this cannot yet be automated.
Perhaps this process should not be called “placement”, but “connection”.
Do not focus only on the seams, move the fragments of the texture, remove too noticeable pebbles, etc ...
It is impossible to move from the extended rock (on the left) to the square seamless texture (on the right) simply by “removing the seams”
The way I make textures seamless in Photoshop makes it quite easy to move objects compared to 3D editors, especially when working with grids of 50 million polygons.
I almost never use ZBrush , except for the texture offset for baking all the cards I need. Sometimes I can quickly fix some artifacts, but usually this does not happen, and I do nothing in ZBrush.
Getting rid of lighting is a very big problem when scanning 3D objects. For textures ... it's not too important. Most often, this problem is solved quickly and straightforwardly. The most important thing to have when scanning uniform lighting. If it was possible to achieve it, then the matter remains with using the AO (Ambient occlusion) map to remove the blackout from the diffuse map. Then, you may have to fix some fragments manually.
I almost never spend more than thirty minutes on deleting lighting information.
Normal maps are created quite simply, but you should pay attention to two points:
Very tricky question!
Since PBR is widely used in games, the opinions of artists are divided. Some want the system to be 100% accurate, and therefore state that you should never use reflection maps. Others say that some objects look better with them, although technically incorrect. I can say that both sides are right, but I tend to use reflective maps in certain cases. Moreover, perhaps they are not as inaccurate as you think. Let me explain:
When baking organic textures on a plane, you overly simplify extremely complex shapes, leaves, small pebbles, etc ... Have you ever realized how difficult moss can be?
All these irregularities cannot be imitated by simple roughness and normal maps. The main reason for this is that all these small details cast shadows on each other. The light is “tangled” in the complex moss pattern, and cannot simply be reflected back.
Tessellation can bring these small shadows back, but for small details it is not enough.
Is it possible to solve this problem with an AO card?
Yes, maybe, but the problem with AO cards is that they really only work in shaded parts of the grid, and all the parts illuminated with direct light remain very flat and “plastic”.
Using a reflection map is an easy way to imitate the complex behavior of complex organic forms.
Without a reflection map (left) and with a reflection map (right). Note that dirt and very small stones look less plastic with a reflection map.
At the same time, you should carefully handle the reflection map. Always understand why you are doing it, what logic is behind it. The need for a reflection map also strongly depends on the engine used.
When using a reflection map, I transfer only its part to the input using linear interpolation in order to control the amount of changeable reflection.
I always create reflection maps for my textures, because it only takes a few minutes, but as a result I use them only in special cases, mainly for grass and moss.
My other works can be viewed on ArtStation !
Sebastian van Elverdinge, environmental artist at Starbreeze Studios.
Introduction
Hi, my name is Sebastian, I live in Brussels, the birthplace of the best french fries, beer and chocolate! I am currently working as an environmental artist at Starbreeze Studios (Stockholm). I used to work in Playground Games on Forza Horizon 3, which became for me an excellent opportunity to apply photogrammetry in an AAA-class game.
')
Over the past four years I have experimented a lot with photogrammetry. Pretty quickly, I started working on textures and materials that I knew little about. About a year ago I released a tutorial on how to create textures using photogrammetry.
Although my workflow has changed a bit since the writing, the tutorial still applies to what I'm going to tell you. If you find something incomprehensible in this interview, then most likely you will find a detailed explanation in the tutorial.
My very first seamless scan of 2014 (left) and one of the latest scans (right)
Photogrammetry
Usually, people perceive photogrammetry as a way to transfer static scanned grids (mesh) to a game. When working with scanning environments, they always deal with huge unique textures with a side of 4-16 kilopixels on unique, non-repeating grids. A good example is the UE4 Kite demo. But what is suitable for cinematic clips like Kite will be completely inapplicable for use in real time, especially on large surfaces.
This was important to me when I was working on the UE4 Marketplace Rock Texture Set. Imagine that we are making a small rocky canyon. It will be an error to scan a set of 4-8 large rocks, from which you will then try to build a canyon. As a result, you will most likely get four unique textures with a side of 4 kilopixels. Instead, I strive to create one very good seamless stone texture, writing beautiful shapes to its height map. Then I just create a very simple canyon grid, for which I apply displacement based on the resulting texture. This way I get a whole scene in which only one 4k texture is used for the rocks.
Quickly created “envelope grid” of the canyon with displacement in UE4, one seamless stone texture is used and some snow for beauty.
You can without any problems perform the advance in advance in the 3D-editor based on the height map, and then import the optimized grid instead of using tessellation. The choice depends on the importance of the environment and the resources available.
I believe that this method takes much less memory, takes an incredibly little time (compared to scanning multiple grids) and allows you to perform design iterations much faster. If you want to turn the scene into a desert canyon, then it will be enough just to scan one texture of the desert rock.
Just imagine how many grids would have to be scanned to replace the biome, and in our case only one texture is required (plus snow, sand or moss)
However, it is wise to approach the concealment or removal of seams. My tessellation material allows you to remove 100% of the seams.
In the end, photogrammetry can be very flexible, you only need to think about it outside the traditional "hard" use.
Equipment
In fact, besides the camera and computer, you will not need anything else. A good analogy: to play the guitar, you do not need Gibson for $ 2000, with a simple guitar for $ 100 you can achieve a lot. Ultimately, mastery means more than equipment. When working with photogrammetry, I use very simple equipment. I would love to “upgrade”, but for now I think that the limitations of bad equipment helped me find clever ways to improve the poor quality of scans.
I spend all my scanning work with the help of the Canon 100D / Rebel SL1, one of the cheapest DSLRs on the market, it costs only about $ 400.
However, it will take some effort to improve the quality of scans.
- Manual shutter speed: although it is technically possible to use a poor camera with automatic settings, in fact, the lack of manual shutter speed is very limiting. It is necessary that all images have the same settings, otherwise the camera will begin to compensate for darker images and it will be much more difficult for the software to link the images to each other, and you will delete the lighting information. Almost all cameras and phones today have a manual shutter function.
- A good SD card / high write speed: this is not so obvious at first, but it can change your world. For a long time I used a slow SD card and I had to wait about 5 seconds between shots (as soon as the buffer memory of the card was filled). When you make hundreds of shots, 5 seconds between shots is a lot.
- RAW files: this is the main reason I recommend DSLR. The format of RAW files contains much more information and dynamic range than a regular JPG (which, moreover, has compression artifacts). It allows you to pre-process images, achieving, among other things, better quality, color balance and eliminating vignettes.
In addition, it allows you to reduce information about the lighting before processing scans, which makes it easy to remove this information after the process. RAW files can be huge, so it’s worth considering when choosing an SD card.
And now about the equipment that I do not use:
- Tripod: quite right, I rarely use a tripod, at least when shooting on the street. This is my personal choice. For me, a higher shutter speed is more important than a large aperture (more on this later). Do not misunderstand me, using a tripod without any doubt improves the quality of scans, but there is a drawback - time is spent on installing a tripod for each shot, a lot of time (and even a few seconds per frame add up very much). When shooting on the street, lighting conditions are rarely perfect, so I usually strive to be as fast as possible (but without fuss) to avoid a gradual change in lighting conditions when scanning. I repeat, if you scan indoors or the lighting conditions are stable, then there is no reason not to use a tripod! Moreover, perhaps, for you a good compromise between time spent and quality can be a monopod.
- Color checker: there is no worthy reason not to use them, except for the high cost — about $ 100 for the X-Rite Passport. The color calibrator provides the correct color calibration. Soon I will buy it myself.
- Chrome balls and all the equipment to remove information about lighting HDR: in my opinion, it's just a waste of time and money, at least for textures, but it can be useful for large scans at 360 degrees. By default, if you scan textures, the removed surface most often looks in one direction and the illumination of the entire scan is almost the same. You can get rid of the remnants of lighting in a few minutes of working in Photoshop and not wasting time fiddling with the removal of information about lighting HDRI.
Let's talk about computers: if you have a standard gaming desktop computer, then there will be no problems with processing. I recommend to have at least 16 GB of RAM and a decent video card. I use the GTX670 and 16 GB of RAM, a fairly standard set. Probably the bottleneck will be memory.
Lighting
Good lighting is very difficult to achieve, because on the street you can not control it and there are no other options than to wait for the right moment. In general, two points should be avoided:
- Rain: when scanning should be dry. Depending on the material, rain can make it more reflective or darker.
- Changes in conditions: if you do not manage to catch a good uniform cloudy sky, at least aspire to something stable, even if it is sunny, in which case you can scan something in the shade. It is necessary to avoid moving clouds at all costs, because they can significantly change the illumination in just seconds.
This is one of the worst examples.
Your goal is a cloudy sky, both uniform and very bright. Do not wait too long, the probability of getting it is not very great!
How many snapshots do you usually need to get good stuff?
The number of shots strongly depends on the software used (about it a little later), the size of the scanned surface and the desired level of detail. It is always better to take more pictures than you need.
Of course, overlay is crucial. Photogrammetric software compares the features in different images to determine the spatial position of frames. Therefore, several pictures should have the same parts, the more the better.
So, you want to scan the surface found, where to start?
Remember, I said that I can refuse a tripod, because I am changing a narrow diaphragm at a high shutter speed? I can afford a narrow depth of field almost without blurring, because I always shoot from top to bottom (but here you have to be careful not to get into the frame with your shoes!). This not only provides the best results, but also means that the object will be almost at one distance from the camera throughout the frame, reducing the amount of blurring of the depth of field of sharpness. Of course, this refers to scanning relatively flat surfaces.
Source: Agisoft User Guide
And now about the overlay: imagine yourself a Google Earth mapper, first you want to capture the whole Earth, the whole object in the first pictures, and then add different levels of detail. First, take a global picture, at least eight, better more. Perhaps they can not be done from the top down, do not worry, just move through the object, and be careful with the depth of field. Now you have on each snapshot all the characteristic features of the surface to which the following, more accurate snaps will be attached.
Next, you need to create the first zoom level: take pictures from top to bottom, about 1/4 the size of the surface. Make sure they overlap each other by more than 50%. Repeat again for 1/8 surface, 1/16, etc ... It all depends on the level of detail you need.
Such an algorithm of work ensures that close pictures do not mix with their neighbors, for binding they will be able to rely on the previous level of detail.
Of course, in reality, you are doing everything by the eye, no need to carry a tape measure. Moreover, this principle does not always work, for example, if you scan sand, then it cannot be stepped on during the scanning process. But it's a good idea to always remember this principle.
In this example, the general images are highlighted in blue, the first level of images from top to bottom is shown in green, and the other detailed view from top to bottom - in red.
Software
There are two main competitors, Agisoft Photoscan and a fairly new Reality Capture package. I mainly used Photoscan, but recently I started using Reality Capture. Not to say that one is much better than the other in some ways, but lately the industry seems to be leaning towards Reality Capture.
Here are my pros and cons of each package. Do not forget, I am still rather inexperienced in Reality Capture and I can be mistaken about something.
Agisoft Photoscan | Reality capture | |
pros | Good documentation and online tutorials | Insanely fast to handle, sometimes you don't even have time to drink coffee! |
| Good filtering tools for removing unwanted points after binding. It doesn't seem to make as many holes as Reality Capture. | Can process thousands of snapshots on a simple computer. | |
| In my opinion, the quality of textures in Agisoft is slightly better. | The program never fell out, it happens so rarely that it is worth mentioning | |
| Standalone's license is quite cheap, $ 179, cheaper in the long term than Reality Capture. | ||
Minuses | Quite slow and uses a lot of RAM, with a mediocre computer, the resolution of the scans can be quite severely reduced | It creates more holes and requires a little more manual binding. However, the problem here may be in my settings, which are not perfect yet. |
| You need to carefully select the pictures, because the computer may not be able to process them all. | Most of the parameters are very mysterious, not enough documentation and examples. | |
| For the same reasons, most likely it will not be possible to achieve ultra-high resolution grids. | Too similar to the one-touch solution approach. Conveniently, when it works, but when it does not work, then a lot of problems arise during the correction. | |
| The subscription system makes it more expensive in the long run (from six months) |
I think it is worth advising beginners to start with Reality Capture, it is more tolerant of bad pictures, besides, you can compensate for poor-quality pictures with good image quality, which Agisoft may not provide on your computer.
However, if you plan to work with photogrammetry only from time to time, then it will probably be cheaper to choose a standalone license from Agisoft.
Special features
A stunning, ultra-detailed scan may not look beautiful in the game unless you take care of a good artistic vision in advance.
The most important and difficult is tiling. He should always be remembered, especially when searching for surfaces to scan. Many people ask me: “Is there a faster way to place textures and is it possible to automate it, for example, using Substance?”
In 90% of cases there is no such method.
I think the belief in the existence of such a method arose from the belief that paving is the removal of seams. Yes, it probably is, but it’s less than 10% of the texture merging process. The real job is to balance the frequency and characteristic features of your surface, to make the texture look good even with repeated repetitions. In most cases, this cannot yet be automated.
Perhaps this process should not be called “placement”, but “connection”.
Do not focus only on the seams, move the fragments of the texture, remove too noticeable pebbles, etc ...
It is impossible to move from the extended rock (on the left) to the square seamless texture (on the right) simply by “removing the seams”
The way I make textures seamless in Photoshop makes it quite easy to move objects compared to 3D editors, especially when working with grids of 50 million polygons.
I almost never use ZBrush , except for the texture offset for baking all the cards I need. Sometimes I can quickly fix some artifacts, but usually this does not happen, and I do nothing in ZBrush.
Deleting lighting information
Getting rid of lighting is a very big problem when scanning 3D objects. For textures ... it's not too important. Most often, this problem is solved quickly and straightforwardly. The most important thing to have when scanning uniform lighting. If it was possible to achieve it, then the matter remains with using the AO (Ambient occlusion) map to remove the blackout from the diffuse map. Then, you may have to fix some fragments manually.
I almost never spend more than thirty minutes on deleting lighting information.
Normal maps
Normal maps are created quite simply, but you should pay attention to two points:
- At any cost, avoid overhanging: you need to remember this even while shooting, figure out which area may look wrong when shifted on a plane, and know how to fix it if you decide to shoot anyway. As I explained in my tutorial, most of the problems can be solved by creating a special low-poly model. However, this is a tricky problem, because you may not notice it until you get very far in the process. On my first scans there was a lot of stretching due to overhangs.
An example of stretching due to overhangs in one of my first scans. The special low poly mesh I now use would fix this.
Do not expect the scan to be detailed enough for a super fine normal map. You will need to regenerate high-frequency parts by converting albedo to a normal map and overlaying small parts on top of the scan normals.
Why do you think the use of reflection maps is important for choosing the right types of materials?
Very tricky question!
Since PBR is widely used in games, the opinions of artists are divided. Some want the system to be 100% accurate, and therefore state that you should never use reflection maps. Others say that some objects look better with them, although technically incorrect. I can say that both sides are right, but I tend to use reflective maps in certain cases. Moreover, perhaps they are not as inaccurate as you think. Let me explain:
When baking organic textures on a plane, you overly simplify extremely complex shapes, leaves, small pebbles, etc ... Have you ever realized how difficult moss can be?
All these irregularities cannot be imitated by simple roughness and normal maps. The main reason for this is that all these small details cast shadows on each other. The light is “tangled” in the complex moss pattern, and cannot simply be reflected back.
Tessellation can bring these small shadows back, but for small details it is not enough.
Is it possible to solve this problem with an AO card?
Yes, maybe, but the problem with AO cards is that they really only work in shaded parts of the grid, and all the parts illuminated with direct light remain very flat and “plastic”.
Using a reflection map is an easy way to imitate the complex behavior of complex organic forms.
Without a reflection map (left) and with a reflection map (right). Note that dirt and very small stones look less plastic with a reflection map.
At the same time, you should carefully handle the reflection map. Always understand why you are doing it, what logic is behind it. The need for a reflection map also strongly depends on the engine used.
When using a reflection map, I transfer only its part to the input using linear interpolation in order to control the amount of changeable reflection.
I always create reflection maps for my textures, because it only takes a few minutes, but as a result I use them only in special cases, mainly for grass and moss.
My other works can be viewed on ArtStation !
Sebastian van Elverdinge, environmental artist at Starbreeze Studios.
Source: https://habr.com/ru/post/333950/
All Articles