Fast deserialization of really big JSON responses
Under the cat there is a small but useful description of how to quickly and easily turn the received JSON response into a set of objects. No manual parsing. And if you’ve encountered an OutOfMemory problem on older smartphones, there’s a solution that supports Android 2.X versions.
In addition, under the cut there will be a link to the github repository with sample code. But the pictures will not, but there is a place for a small tablet.
So, on the current project, I had to parse the service response, consisting of a bundle of objects nested into each other, inside of which there could be objects, inside of which ... The data was in JSON format, moreover, gzip-compression was used by the server, after all the difference in the size of the transmitted data was significant (4 megabytes versus 300 kilobytes in compressed form - this is not a joke for mobile communication).
As a lazy person, parsing each field and object with my hands was not at all with my hand ... Thus, the Gson library was involved, judging by the test - the fastest deserializer from JSON format. Well, now let's get started and start right away with the code. For simplicity, all the output is in the console, so as not to think about views and so on.
')
Here are the objects that arrive to us from the network:
public class HumorItem { public String text; public String url; } public class HumorItems { List<HumorItem> Items; // , , . } And this is the code that downloads and deserializes it.
The first version of the code
public class LoadData extends AsyncTask<Void, Void, Void> { String _url=""; public LoadData(String url){ _url=url; } @Override protected Void doInBackground(Void... voids) { try { // HttpClient httpclient = new DefaultHttpClient(); HttpPost httppost = new HttpPost(_url); HttpResponse response = httpclient.execute(httppost); HttpEntity httpEntity=response.getEntity(); InputStream stream = AndroidHttpClient.getUngzippedContent(httpEntity); // gzip- BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream)); StringBuilder responseBuilder= new StringBuilder(); char[] buff = new char[1024*512]; int read; while((read = bufferedReader.read(buff)) != -1) { responseBuilder.append(buff, 0, read) ; Log.d(" " + PrepareSize(responseBuilder.length())); } // HumorItems list= Gson.fromJson(responseBuilder.toString(),HumorItems.class); // for (HumorItem message:list.Items){ Log.d(": "+message.text); Log.d(": "+message.url); Log.d("-------------------"); } Log.d(" "+list.Items.size()); } catch (IOException e) { e.printStackTrace(); Log.e(" "+e.getMessage()); } return null; } } Wrapper around Log and method for file size
public class Log { public static final String TAG="hhh"; public static void d(String text){ android.util.Log.d(TAG,text); } public static void e(String text){ android.util.Log.e(TAG,text); } } public String PrepareSize(long size){ if (size<1024){ return size+" ."; }else { return size/1024+" ."; } } And this solution worked great! For the time being. The answer for one of the combination of parameters weighed about 8 megabytes. When testing on the part of the phone - the program fell, where on the fifth downloaded megabyte, where on the third.
Google first suggested a simple solution - to set largeHeap in the AndroidManifest file.
<application [...] android:largeHeap="true">
This parameter allows the application to allocate more RAM for itself. The option is of course lazy and simple, but Android phones below the 3rd version are not supported. And in general, some defeatist approach - “why optimize, if you can buy more hardware?”
Further, after several attempts, the following simple option was chosen:
- Do not fill the file with a variable, no - download the data directly to a USB flash drive (or internal memory, which will turn up at hand).
- We set Gson on this file. There is no problem with parsing and file storage.
No sooner said than done:
The second version of the code - with a temporary file
public class LoadBigDataTmpFile extends AsyncTask<Void, Void, Void> { String _url=""; File cache_dir; public LoadBigDataTmpFile(String url){ _url=url; cache_dir = getExternalCacheDir(); } @Override protected Void doInBackground(Void... voids) { try { // HttpClient httpclient = new DefaultHttpClient(); HttpPost httppost = new HttpPost(_url); HttpResponse response = httpclient.execute(httppost); HttpEntity httpEntity=response.getEntity(); InputStream stream = AndroidHttpClient.getUngzippedContent(httpEntity); // - BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream)); File file = new File(cache_dir, "temp_json_new.json"); if (file.exists()){ // - file.delete(); } file.createNewFile(); FileOutputStream fileOutputStream=new FileOutputStream(file,true); BufferedWriter bufferedWriter=new BufferedWriter(new OutputStreamWriter(fileOutputStream)); char[] buff = new char[1024*1024]; int read; long FullSize=0; while((read = bufferedReader.read(buff)) != -1) { bufferedWriter.write(buff,0,read); // FullSize+=read; Log.d(" " + PrepareSize(FullSize)); } bufferedWriter.flush(); fileOutputStream.close(); // Log.d(" ..."); FileInputStream fileInputStream=new FileInputStream(file); InputStreamReader reader = new InputStreamReader(fileInputStream); HumorItems list= Gson.fromJson(reader,HumorItems.class); Log.d(" ."); / for (HumorItem message:list.Items){ Log.d(": "+message.text); Log.d(": "+message.url); Log.d("-------------------"); } Log.d(" "+list.Items.size()); } catch (IOException e) { e.printStackTrace(); Log.e(" "+e.getMessage()); } return null; } } That's all. The code is tested in combat conditions, it works stably with a bang. However, you can make it even easier and do without a temporary file.
The third version of the code - without a temporary file
public class LoadBigData extends AsyncTask<Void, Void, Void> { String _url=""; public LoadBigData(String url){ _url=url; } @Override protected Void doInBackground(Void... voids) { try { // HttpClient httpclient = new DefaultHttpClient(); HttpPost httppost = new HttpPost(_url); HttpResponse response = httpclient.execute(httppost); HttpEntity httpEntity=response.getEntity(); InputStream stream = AndroidHttpClient.getUngzippedContent(httpEntity); // BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream)); // InputStreamReader reader = new InputStreamReader(stream); HumorItems list= Gson.fromJson(reader,HumorItems.class); // for (HumorItem message:list.Items){ Log.d(": "+message.text); Log.d(": "+message.url); Log.d("-------------------"); } Log.d(" "+list.Items.size()); } catch (IOException e) { e.printStackTrace(); Log.e(" "+e.getMessage()); } return null; } } Minus - you will not be able to control the download process (interrupt it in an adequate way), and it is also unknown how much data has already been downloaded. Beautiful progress bar can not draw.
There is one more option given in the documentation , which allows you to sequentially pull objects out and immediately process them, but it’s difficult to work with it if you have an object of different arrays of objects, and not just an array of the same type. However, if you have a beautiful solution, I will gladly see it in the comments, and will definitely include it in the update article!
As a bonus - some statistics.
| file size | The number of objects inside | Deserialization time on the emulator | Deserialization time on Highscreen Boost |
| 5.79 MB | 4,000 | 35 seconds | 2 seconds |
| 13.3 MB | 9000 | 1 minute 11 seconds | 5 second |
Example of use - on githab , test files in the same place.
Link to the Gson library .
If anyone is interested in the theme of development for android, then at least there will be posts about push notifications (server and client side - there were articles on this topic on Habré, but they are all somewhat outdated), about working with the base and others on Android development .
Update. On the githaba they showed the solution to the problem “Minus - it will not be possible to control the process of downloading (to interrupt it in an adequate way), and it is also unknown how much data has already been downloaded. Beautiful progress bar can not draw. ". Details are in the latest commit in the repository.
Source: https://habr.com/ru/post/200898/
All Articles