Using Pixel Bender filters on the example of creating an application for VKontakte

This article was written in continuation of the topic covered here , at the request of the user namata . The article will discuss the use of ActionScript and Pixel Bender filters in your application, and those pitfalls that can be used in the business. Flash gurus and their ilk can go and drink some tea. They are unlikely to be interested in this article. But for those who are just acquainted with these things, this article will help not to step on the rake once again.







So! Where to start? .. Well, probably better to start with the question: "Why for VKontakte?". The answer is simple: I wanted to deal with their API :) In fact, it does not matter. Just with applications hosted on third-party servers, the situation is somewhat different than with applications hosted on your site. But first things first.

')

In this article, I will not dwell on the questions of writing the filters themselves. We assume that you already have them. Well, at least one ... If all the same is not, then here and there over there are a few examples. What to do with them when you already have a filter file with the .pbj extension in your hand?



There are at least two ways to connect Pixel Bender filters to your Flash / Flex application. The first method involves embedding a filter using meta tags:

[Embed(source="SomeFilter.pbj", mimeType="application/octet-stream")]

private var _someFilter:Class;



In this case, the further use of the filter will occur approximately in the following scenario:

var shader:Shader = new Shader(new _someFilter() as ByteArray);

var shaderFilter:ShaderFilter = new ShaderFilter(shader);



var bitmap:Bitmap = new Bitmap(someBitmapData);

bitmap.filters = [shaderFilter];



The filters property is an array of filters. But simply adding a new filter to the end of the array (for example, using the push ([...]) method) will not entail changing this filter. You must first add a filter to the array, and then reassign this array to the filters property.



The second method involves loading filters at runtime:

var urlLoader:URLLoader = new URLLoader();

urlLoader.dataFormat = URLLoaderDataFormat.BINARY;

urlLoader.addEventListener(Event.COMPLETE, onLoadComplete);

urlLoader.load(new URLRequest("SomeShader.pbj"));

var shader:Shader;



function onLoadComplete(event:Event):void {

shader = new Shader();

shader.byteCode = loader.data;



var shaderFilter:ShaderFilter = new ShaderFilter(shader);

var bitmap:Bitmap = new Bitmap(someBitmapData);

bitmap.filters = [shaderFilter];

}



In general, after loading (or embedding) the filter, it no longer matters what the filter is: written on Pixel Bender or the base language filter. Their further use in general terms is similar.

So what about the stones?

Now let's talk about the features of the application of these filters for your application.



Commercial break :

The application that I developed is a small photo editor that allows you to change image characteristics such as brightness, contrast, saturation and lightening. You can change the values ​​of all the channels of the image, and each channel separately. It is also possible to set the threshold values ​​of the filter.



Select Albums:







Photo editing:







The first pitfall of which I spoke was that we can get the images (in this case, photos of the user) using the API. But we cannot get access to the image data :) What am I talking about? This is what:



Suppose we have a certain variable in which we stuffed a line containing the path to the photo of interest on the server (we received this path as a result of a request to the server). Next, using the Image class from the Flex framework, we write something like the following:

[Bindable]

private var _photoSrc:String = "path/to/image";

<mx:Image id="curPhoto" source="{this._photoSrc}" />



Next, somewhere in the code (in my case it happened in the slider's drag-and-drop event handler), we write the following:

curPhoto.filters = [shaderFilter];



The filter seems to be applied -> the picture on the screen has become a little different -> we are satisfied -> you can upload to the VK server.



But hey, be glad. In this case, exactly the same image will be sent to the server as on the original. Is the filter screwed up? Not at all.



Consider another scenario.



Suppose that you used the Loader class to load an image. When using this method there are two key points:



1. The load () method of the Loader class object should generally have the following signature:

loader.load(new URLRequest(this._photoSrc), new LoaderContext(true));



The second argument of the method must be an object of class LoaderContext whose first argument will be true. As a result, Flash Player will try to load the security policy file before starting to download the image file itself (use Security.allowDomain ("*") option; in this case it does not channel).



2. In the handler of the Event.COMPLETE event, we should have the following:

var bitmapData:BitmapData = new BitmapData((event.currentTarget as LoaderInfo).content.width, (event.currentTarget as LoaderInfo).content.height);

bitmapData.draw(this.loader);



this.bitmap = new Bitmap(bitmapData);

this.curPhoto.source = this.bitmap;

this.bitmap.filters = [shaderFilter];



The important thing is that in the draw () method we must pass not this.loader.content, but the object itself this.loader. Why so, and what is it - a bug or a feature? I honestly did not understand. But it works, and all other options lead to violation of the security policy of an isolated environment.



So, the bitmap object in our pocket, you can add it to the display list, apply filters to it and upload to the server. Stories end? Not really.



The second pitfall is that not all filters are equally useful. Applying filters is not easy. The fact is that when we apply a filter through the filters property, we apply it not to the data of this image, but only to the screen object of this image. Recall that the BitmapData class contains the actual image data, and the Bitmap class contains a display object that can be included in the display list. As you remember, we assigned our filter to the filters property of the Bitmap object, which means that the BitmapData object itself (contained in the bitmapData property of the object this.bitmap) remained unchanged (the same thing happened when we tried to load to the server an instance of the Image object in the case described above).



It would seem that the output is obvious - you need to apply the applyFilter method ([...]) to some intermediate BitmapData object (in this case, the first argument of the method would be a BitmapData object from the this.bitmap.bitmapData property). But in my case, when support for changing several properties is implemented, applying each subsequent filter overwrites all changes to the previous filter. Those. after any manipulations, only changes from the last filter were visible. Unfortunately, it is impossible to use in this method not one filter, but an array of filters.

How to be?

The exit was found somewhat unusual. Remember our object of class Image? Here he was useful to us. Here is an example code:

var bitmapdata:BitmapData = new BitmapData((curPhoto.content as Bitmap).width, (curPhoto.content as Bitmap).height);

bitmapdata.draw(curPhoto.content as Bitmap);



var jpgEncoder:JPGEncoder = new JPGEncoder(85);

this.jpgStream = jpgEncoder.encode(bitmapdata);



// ...



Those. we created an intermediate object of the BitmapData class and rendered in it the contents of our Image object. As we remember, earlier we assigned an instance of this.bitmap of the Bitmap class (to which the filter (s) were applied) to the source property of our Image object. This means that in fact we did not work with the data of the image itself, applying filters, but only with its screen object.



Commercial break :

To show the result of applying filters in my application, I’ll provide a couple of screenshots (before and after applying filters).



Photo before editing:







… and after:

Some thoughts on the topic:

1. In principle, to edit the parameters that are in my application at the moment, one could also use the ActionScript convolution filter (ConvolutionFilter). As far as I can tell, it would be enough. But I just wanted a better deal with Pixel Bender :)). Pixel Bender is more suitable for creating complex filters. It is easier and clearer to work with the image (more precisely, with its pixels). When using the same convolutional filter, you have to work with matrices. In general, this is a personal matter for everyone.



2. It is better to build Pixel Bender filters into the project than to load at runtime. They are lightweight enough (none of my filters weighed more than one kilobyte). Extra code is useless.



3. The VK API is generally good. A little annoying verification phase of the administration. Reading the questions and answers of other developers in the “Questions about creating applications” topic in the “Discussions” section of the FLASH API group, I realized that in order for your application to be approved, it is necessary that it not only meets all the requirements, but also so that there is a necessary phase the moon (and the parade of planets would not hurt the campaign ...). Probably, I was just lucky that the application was approved the first time. Heh, by the way, the application was written from scratch in a week, and they checked it for almost a week ... It would be a little bit more and it would be funny.



4. Not at all pleased with the feedback from the developers. Yes, there is a group of FLASH API, there is a topic in which you can ask your questions about creating applications. BUT! The questions, as far as I understand, are answered by people who, in principle, have no relation to the administration of the site (that is, in most cases other developers who have already made cones). And further. When I was looking for the answer to one question, I realized two things:

1. The questions in this topic of the FLASH API group are repeated at best five times each.
2. In order not to be a dolbozvonchik, I honestly tried to find the answer to my question in this topic. The logic was simple: someone had to deal with this before. But, damn, reaching the "twenty-some" page, I realized that I had already forgotten what I was looking for. If someone close to the site administration people is reading this post now, then you should be ashamed. There is no search at all or at least some kind of structure.
   
   

PS: Well, at the end of the topic I will provide a link to the application itself. All wishes / suggestions for the expansion of the project are welcome. If users like the application, I can implement almost everything in the photo editor on picnik.com , and even more (of course, as part of the VKontakte API;)).