When there is no web API at the server, but it is very necessary

Quite often in the life of developers there are situations “when you really want, but you can't.” And very often this issue is solved, like everything in our country - “if you really want, then you can.” Today I want to tell you about my experience in creating an independent API for a web project that this API does not provide. The article will be useful to Java or Solaris developers, as well as to all those who are faced with the problem of integrating various services.

Not so long ago, the SourceJuicer project ( http://jucr.opensolaris.org/ ) was opened for all developers, which allows you to publish your projects for OpenSolaris, build and publish them to open repositories. The project description is done in the form of a spec file ( http://jucr.opensolaris.org/help/spec_file ), which lists the attributes of the project, how to collect it, where the source code is downloaded, under what licenses it is distributed, etc. All the files necessary for the assembly are uploaded to the server and after the review it will be compiled and uploaded to the repository. It seems to be all cool, but what's the catch? And the catch is that you can create a project and upload or update files only through the web interface. That, in principle, is not so fatal for a small project, but already with 10 files this download-update process begins to tire a little.

In addition, such a boot system is absolutely not suitable for Continuous Integration - for which, in essence, SourceJuicer was created. Therefore, then I will tell you how to use this service (or any other) as efficiently as possible and can for someone, it will throw interesting ideas for the future.
')


Where do we start? Wizards will always recognize the tool, and the person who explores someone else's website, by Firebug. Typically, the scenario of the user with any system looks like this

1. We perform authorization.
2. Go to the desired link.
3. Fill in the form fields.
4. We confirm the action.
5. From point 2, repeat as necessary.

In order to create our own web API for the site, we need to describe these user actions as precisely as possible and perform them as necessary. For example, authorization is nothing more than getting the right cookies in response to a form. Therefore, we take Firefox with Firebug ( http://getfirebug.com/ ) and Firecookie ( http://www.softwareishard.com/blog/firecookie/ ) installed, go to the login page and see which cookies were set with the successful login.

As a rule, when authorizing, a label is issued - “the number of sessions” (token), which does not change upon requests. This can be “jsessionid”, “phpsession”, “OSS”, etc. But there are also options with changing the cookie when repeated requests, so it is worth considering. In the case of authorization on SourceJuicer, three cookies are used, so all three will have to be saved and transmitted when queried. Also do not forget that the page with the authorization form is not always the starting point of the journey, often in order to initialize the session you need to go to the start page of the server or user.

We know what we need to do, so now let's see how to describe it in code. There is such a wonderful library HttpUnit ( http://httpunit.sourceforge.net/ ), which allows you to test the web-interface - follow the links, send forms, check the contents of the tables and many, many interesting things. But now we are interested in it as an easy way to send http requests and process responses. In order to use this library you need to connect an html parser and a javascript interpreter. I used nekohtml ( http://nekohtml.sourceforge.net/ ) and Mozilla rhino ( http://www.mozilla.org/rhino/ ) - although neko would not recommend for production because of the hard dependency on xerces.

So we need to go to the page, get cookies and send the form. The code for this is

wc.getResponse(JUICER_HOME_URL); // Creating new jsession
wc.getResponse(JUICER_AUTH_URL); // Go to login

WebResponse response = wc.getResponse(AUTH_URL); // Load login form
WebForm loginForm = null ;

for (WebForm form : response.getForms()) {
if (form.getAction().equals( "/login.action" )) {
loginForm = form;

break ;
}
}

loginForm.setParameter( "userName" , username);
loginForm.setParameter( "password" , password);

loginResult = loginForm.submit(); // Submit user name and password

* This source code was highlighted with Source Code Highlighter .

The next option is when we update the project on SourceJuicer not all files may already be needed to build the project. Therefore, we need to check which files are out of date and delete them. Through the web interface, this is solved as follows - the page of an existing project opens, the first table has a list of files and the Delete button next to each file. When you click on the button, a confirmation form appears and after clicking on “Confirm” the file is deleted.

How does all this look in code?

WebResponse response = conversation.getResponse( String .format(JUICER_SUBMISSION_MASK, submissionId));
WebTable fileList = response.getTableStartingWith( "Summary" ). // Summary
getTableCell(2, 1). //
getTables()[0]; //

Collection< String > forDelete = new ArrayList < String >();

for ( int i = 0; i < fileList.getRowCount(); i++) {
WebLink link = fileList.getTableCell(i, 0).getLinks()[0]; //

String fullName = link.getText().replaceAll( "\\s+" , "" );
String fileName = fullName.replaceAll( "[^/]+/([^/]+)" , "$1" );
SubmitType type = detectType(fullName);

if (existFileInSubmission(files, type, fileName) == null ) {
String fileId = fileList.getTableCell(i, 1).getForms()[0].getParameterValue( "file_id" ); // id

forDelete.add(fileId);

debug( "Need to delete file [%s]" , fullName);
}
}

for ( String fileId : forDelete) {
PostMethodWebRequest post = new PostMethodWebRequest(
String .format(JUICER_DELETE_MASK, submissionId),
true
);

post.setParameter( "delete_file" , "Delete File" );
post.setParameter( "file_id" , fileId);

WebResponse confirmResponse = conversation.getResponse(post); //

WebForm form = confirmResponse.getForms()[0];
SubmitButton confirm = form.getSubmitButtons()[1]; // Confirm ...

WebResponse r = form.submit(confirm); // ...

}

* This source code was highlighted with Source Code Highlighter .

So with the help of simple manipulations with cookies, requests and html parsing, we got a working version of api ( http://bitbucket.org/abashev/pusher/src/ade4e90f01a9/src/main/java/org/bitbucket/pusher/api/ SourceJuicerAPI.java ), which can be used as it is convenient for us, and not the developer of the service.

The full version of the project is here - bitbucket.org/abashev/pusher

I am pleased to hear your questions and suggestions.