How to assemble Ceylon in a container if your container ship is blocked

It so happened that in our team of kind, sweet, and responsive mathematicians ( DMEiM ), Ceylon is used as a general purpose language. Being responsive, we not only use this language, but also participate in its development, mainly bug reports. Pulkwests are worse, and the first reason is that there is no direct Internet access in the office, only through a proxy server. (Zero, of course, lack of time.)







Under the cut there are details about exactly what problems arose when building the Ceylon project from the source code and how they were solved. In the end, just a few words about its ultimate goal.

What is this elephant?

First, a few words about Ceylon, how about the project. (How about a programming language, who is interested, read here .) Two basic subprojects (besides tools such as IDE plugins, own repository) are ceylon and ceylon-sdk . The first one includes the compiler itself and a set of console utilities written in Java. The second is a set of basic libraries written in Ceylon itself. Each project is built using Apache Ant, of course, you also need an installed JDK.

')

The compiler is built with the ant clean dist command, after which it can be copied from the dist / dist directory to / usr / local / share / ceylon or anywhere else to taste and link to the executable file in the directory that is visible in $ PATH. Libraries are collected and copied where necessary with the command ant clean publish .



If you have direct access to the Internet (along with the correct versions of Java, Ant and Ceylon sources), the assembly goes without any difficulty.

Problem on course

Security requirements in our company prescribe the lack of direct Internet access from workstations and most servers, assuming that local mirrors and a proxy server are enough for all occasions. But the opposite cases, however, are. When setting up CI for my own projects, my colleagues had to suffer because of this.



When building a bare iron workstation on an open deck , the compiler gathered quietly (maybe all dependencies are included in the project, maybe the previously made settings helped), but the set of libraries did not want to be assembled. It would seem that there is a gateway proxy, and tanks with libraries (Ceylon Herd, Maven Nexus), but something is missing. Probably, “salty splashes, gusts of a mad wind ...” Barash.





The funny thing is that the build system requires Java dependencies from Maven in Ceylon ( car ) format, which should not be there in principle.

Packing assembly into container

What is a container, the local audience should be aware of. For passing by - link .



So, not having achieved success, I decided: I would assemble Ceylon and Ceylon SDK on the open sea on a computer with direct access to the Internet and in the container I would deliver this assembly to the office network. In theory, the container will contain all the dependencies, after which it will be possible to edit the code for its own needs and pullrequests and run the reassembly.



As an intermediate level, an image with a JDK was prepared.





Dockerfile was borrowed from corporate configurations, for generalization, the creation of the Russian locale was commented out. Its main task is to install the Oracle JDK (through an auxiliary program, by virtue of the Oracle license) and Maven, it is also updated at the beginning, and at the end the data for the APT is cleaned. Ant is also present, apparently, as a dependency.



Next step: make an image with the Ceylon assembly. When creating an image next to the Dockerfile should be the directory ceylon-sources, and in it - the projects ceylon and ceylon-sdk. At first I wanted to shove git clone directly into the creation of an image, but we will edit the source code locally, and there is no sense to clone it twice.



Dockerfile kopilov / ceylon_build: 1.3.4-SNAPSHOT

 FROM kopilov/java8:latest #      ENV CEYLON_VERSION 1.3.4-SNAPSHOT #   APT ( ,    ), # git ( -    ) # netcat (   ) RUN apt-get update -y && \ apt-get install -y git && \ apt-get install netcat-traditional #    . , #   git clone,    ,   . WORKDIR /usr/src/ceylon ADD ceylon-sources /usr/src/ceylon # ,    WORKDIR /usr/src/ceylon/ceylon RUN ant clean dist && \ cp -a dist/dist /usr/local/share/ceylon-${CEYLON_VERSION} && \ ln -s /usr/local/share/ceylon-${CEYLON_VERSION}/bin/ceylon /usr/local/bin #  WORKDIR /usr/src/ceylon/ceylon-sdk RUN ant clean publish #  RUN apt-get clean && rm -rf /var/lib/apt/lists/* 

The assembly of this image will be successful only with direct access to the Internet. Having collected, I posted it on hub.docker.com .

Closed channel reassembly

It is expected that the container with the finished assembly includes all the dependencies, and the Internet will no longer be required. Run the docker run -it kopilov/ceylon_build , then ant clean publish - no matter how.





The same error on the computer where the image was created, if you disable access to the Internet. What else is missing? No shark traffic to figure it out.



With Internet access turned off, the Docker traffic in Wireshark looks like this:







After several unsuccessful attempts to determine the server IP repo1.maven.org, the above error is displayed. But what happens if the connection is restored:







Paradox: the system makes a GET request to get a response with a 404 error, and then calmly continues to build. And if she does not execute this request, the user is given an error that seems to be absolutely orthogonal to that GET request. Below you can notice requests for modules.ceylon-lang.org (aka Herd ) over HTTPS, but first we will try to deal with the first one.



The first thing that was decided to do was to add the line “127.0.0.1 repo1.maven.org” to the / etc / hosts file. Now we need to somehow simulate the answer "404 NOT FOUND". A recent habrasurf showed that netcat ( proof ) can act as the simplest web server. Before launching the assembly (but after launching the container) I type in a parallel terminal

 docker container ls # _ docker exec -it _ bash nc -lp 80 

After that, I run the assembly (ant), I wait for the GET request to appear in the terminal with the netcat, I print in response

HTTP/1.1 404 NOT FOUND

Server: nc





Voila! Build went further! Then the system makes another exactly the same request (when building under JavaScript), and the process is successfully completed.

Automating the above and finalizing the library

The prepared image included the assembly of the original Ceylon SDK library, and the ultimate goal was to assemble the modified one. Therefore, another Dockerfile was made, replacing the sources:

 FROM kopilov/ceylon_build:1.3.4-SNAPSHOT ENV CEYLON_VERSION 1.3.4-SNAPSHOT WORKDIR /usr/src/ceylon/ceylon-sdk RUN rm -rf * ADD ceylon-sources/ceylon-sdk . 

It should have been as simple as possible (after all, the image is re-created for each test rebuild — for almost every editing of the source code), which is why the netcat installation was done in advance. The netcat trick was wrapped in the following script (plug.sh):

 #!/bin/bash IMAGE_NAME="kopilov/ceylon_patch_src" CONTAINER_ID=$(docker container ls | grep "${IMAGE_NAME}" | sed 's/ .*//') docker exec -i $CONTAINER_ID bash << END echo "127.0.0.1 repo1.maven.org" >> /etc/hosts echo "HTTP/1.1 404 NOT FOUND" > /tmp/notfound echo "Server: nc" >> /tmp/notfound echo "" >> /tmp/notfound nc -lp 80 < /tmp/notfound nc -lp 80 < /tmp/notfound END 

Another script to get the build from the container (get_built_ceylon.sh):

 #!/bin/bash CONTAINER_ID=$(docker container ls -a | grep kopilov/ceylon_patch_src | sed 's/ .*//') rm -r ~/.sdkman/candidates/ceylon/1.3.4-SNAPSHOT/ docker cp $CONTAINER_ID:/usr/local/share/ceylon-1.3.4-SNAPSHOT . mv ceylon-1.3.4-SNAPSHOT /home/akopilov/.sdkman/candidates/ceylon/1.3.4-SNAPSHOT rm -r ~/.ceylon/repo/ docker cp $CONTAINER_ID:/root/.ceylon/repo ~/.ceylon 

Further, the drive to rationalize dried up, it remained to act manually. After each source editing, first in one terminal tab, run docker build -t kopilov/ceylon_patch_src . && docker run -it kopilov/ceylon_patch_src docker build -t kopilov/ceylon_patch_src . && docker run -it kopilov/ceylon_patch_src , then in the next ./plug.sh , then again in the first ant clean publish . And, if the build passed without errors (and if you already have something to test) - ./get_built_ceylon.sh .

Results and origins

The main result of the work done “for the future” was the possibility of our (and maybe not only) team to send pre-tested pullrequest to the upstream project. At the moment, I personally sent this, written along the way: github.com/ceylon/ceylon-sdk/pull/688



Images of Docker kopilov / java8 and kopilov / ceylon_build are available on hub.docker.com , if suddenly someone needs it.



It all started with the fact that last week I had extremely little urgent, and even not very urgent tasks, even the head of the department went somewhere. And when he left, he said: “Think about what else you can fasten to your project.” (The project is at the prototype stage.) And I wanted to fasten internationalization, and not with crutches, but with a ready-made solution.



PS

Background source KDPV