I wanted to sleep today, but again failed. In the Telegram there was a message that someone was not going to Java ... and we woke up only after a couple of hours, tired and satisfied.

Who can this post be useful for? Yes, probably, to anyone except those who also collect JDK8 or just like to read horrible horrors. In general, I warned you, close the article urgently.

Three problems:

• Not going ( level one )
  
  Very boring part you can skip. Only needed for those who want to fully restore the history of events;
• Not going ( level two )
  
  More interesting, because there are a couple of common mistakes, necromancy, necrophilia, what BSD is better than GNU / Linux and why it is worth switching to new versions of the JDK.
• Even if going, falls into the crust
  
  More interesting. Yahuuu, JVM fell into the crust, let's kick it with your feet!

Under the cut is shown a detailed course of solving problems, with different side thoughts about life.

There will be a lot of C ++, there will be no Java code at all. In the end, any javist starts writing only in C ++ ...

Not going

Anyone who has collected Java at all, knows that it looks something like this:

hg clone http://hg.openjdk.java.net/jdk8u/jdk8u cd jdk8u sh ./get_source.sh sh ./configure \ --with-debug-level=fastdebug \ --with-target-bits=64 \ --with-native-debug-symbols=internal \ --with-boot-jdk=/home/me/opt/jdk1.8.0_161 make images 

(All my users are simply called “me”, so that the virtual user can at any time give to any person and not create a rejection from using it not by their username)

The problem, of course, is that it does not work. And quite cynical way.

First level of immersion

Let's try to run:

 /home/me/git/jdk8u/hotspot/src/os/linux/vm/os_linux.inline.hpp:127:18: warning: 'int readdir_r(DIR*, dirent*, dirent**)' is deprecated [-Wdeprecated-declarations] if((status = ::readdir_r(dirp, dbuf, &p)) != 0) { ^~~~~~~~~ 

Initially, for you to understand, I have installed this:

 $ g++ --version g++ (Ubuntu 7.3.0-16ubuntu3) 7.3.0 Copyright (C) 2017 Free Software Foundation, Inc. 

The compiler is not the first freshness, not 8.2, but this one should come up.

C ++ developers love to test software only on the version of the compiler that they have installed. Usually the desire to test on different platforms ends somewhere around the difference between gcc and clang in a general sense. Therefore, it is quite normal at first to introduce -Werror (“read warnings as errors”) and then write such code, which in all other versions will be considered to be vornings.

The problem is known, and it is clear how to solve it. You need to set your environment variable CXX_FLAGS, in which to prescribe the correct level of error.

 export CXX_FLAGS=-Wno-error=deprecated-declarations -Wno-error-deprecated-declarations 

And then we see the miraculous:

 Ignoring CXXFLAGS found in environment. Use --with-extra-cxxflags 

Okay, build system, whatever you want! We substitute configure for this:

 hg clone http://hg.openjdk.java.net/jdk8u/jdk8u cd jdk8u sh ./configure \ --with-extra-cflags='-Wno-cpp -Wno-error=deprecated-declarations' \ --with-extra-cxxflags='-Wno-cpp -Wno-error=deprecated-declarations' \ --with-debug-level=fastdebug \ --with-target-bits=64 \ --with-native-debug-symbols=internal \ --with-boot-jdk=/home/me/opt/jdk1.8.0_161 make images 

And the error remains the same!

Go to the heavy artillery: grepe sources.

 grep -rl "Werror" . 

A huge amount of any auto-generated hat falls out, among which there are glimpses of meaningful files:

 ./common/autoconf/flags.m4 ./hotspot/make/bsd/makefiles/gcc.make ./hotspot/make/solaris/makefiles/gcc.make ./hotspot/make/aix/makefiles/xlc.make 

In flags.m4 we easily find the previous message about “Ignoring CXXFLAGS” and the more hard-coded hard-coded flag CCXX_FLGS (yes, two letters C), which immediately acts instead of CFLAGS , and instead of XX_FLAGS . Conveniently! Two facts are interesting:

• This flag is not passed through the configure parameters;
• The default values ​​are meaningful and suspiciously similar to the real parameters:

  # Setup compiler/platform specific flags to CFLAGS_JDK, # CXXFLAGS_JDK and CCXXFLAGS_JDK (common to C and CXX?) if test "x$TOOLCHAIN_TYPE" = xgcc; then # these options are used for both C and C++ compiles CCXXFLAGS_JDK="$CCXXFLAGS $CCXXFLAGS_JDK -Wall -Wno-parentheses -Wextra -Wno-unused -Wno-unused-parameter -Wformat=2 \ -pipe -D_GNU_SOURCE -D_REENTRANT -D_LARGEFILE64_SOURCE" 

This question looks very nice in the comments - is it a common flag? True?

Let's not play democracy and authoritarian zhudkudim -w ("show no errors"):

  CCXXFLAGS_JDK="$CCXXFLAGS $CCXXFLAGS_JDK -w -ffreestanding -fno-builtin -Wno-parentheses -Wno-unused -Wno-unused-parameter -Wformat=2 \ 

And - hooray! - we passed the first mistake. She is no longer reporting, and in general everything is fine. It would seem that.

Second level of immersion

But now it falls in a bunch of other new places!

It turns out that our -w works, but it is not forwarded to all parts of the assembly. Carefully read the makefiles and don’t understand how this parameter can even be forwarded. Have you forgotten about him?

Knowing the right question to google (“why does cxx not get to build ?!”), we quickly get to the bug page with the saying “configure --with-extra-cxxflags doesn't affect hotspot” ( JDK-8156967 ).

Which promise to fix in JDK 12. Maybe. Wonderful - the most important build option is not used in the assembly!

The first idea - well, let's roll up our sleeves and correct mistakes!

Error 1. xn [12]

 dependencies.cpp: In function 'static void Dependencies::write_dependency_to(xmlStream*, Dependencies::DepType, GrowableArray<Dependencies::DepArgument>*, Klass*)': dependencies.cpp:498:6: error: '%d' directive writing between 1 and 10 bytes into a region of size 9 [-Werror=format-overflow=] void Dependencies::write_dependency_to(xmlStream* xtty, ^~~~~~~~~~~~ dependencies.cpp:498:6: note: directive argument in the range [0, 2147483647] 

Well, we probably need to increase the region. One hundred pounds someone calculated the buffer by pressing the "I'm feeling lucky!" Button in Google.

But how to understand how much is necessary? Below is a clarification of another kind:

 stdio2.h:34:43: note: '__builtin___sprintf_chk' output between 3 and 12 bytes into a destination of size 10 __bos (__s), __fmt, __va_arg_pack ()); 

Position 12 looks like something worthwhile, with which you can now break your dirty feet into the source.

We climb in dependencies.cpp and observe the following picture:

 DepArgument arg = args->at(j); if (j == 1) { if (arg.is_oop()) { xtty->object("x", arg.oop_value()); } else { xtty->object("x", arg.metadata_value()); } } else { char xn[10]; sprintf(xn, "x%d", j); if (arg.is_oop()) { xtty->object(xn, arg.oop_value()); } else { xtty->object(xn, arg.metadata_value()); } } 

Pay attention to the problem line:

 char xn[10]; sprintf(xn, "x%d", j); 

We change 10 for 12, we rebuild and ... the assembly has gone!

But am I really the only one so smart and fixed the bug of all times and peoples? Not a question, again we drive our megapatch into Google: char xn[12];

And we see ... yes, that's right. The JDK-8184309 bug , made by Vladimir Ivanov, contains exactly the same fix.

But the point is that it is fixed only in JDK 10 and it is not backported to jdk8u. This is the question of why we need new versions of Java.

Error 2. strcmp

 fprofiler.cpp: In member function 'void ThreadProfiler::vm_update(TickPosition)': /home/me/git/jdk8ut/hotspot/src/share/vm/runtime/fprofiler.cpp:638:56: error: argument 1 null where non-null expected [-Werror=nonnull] bool vm_match(const char* name) const { return strcmp(name, _name) == 0; } 

Taught by previous bitter experience, we immediately go to see what is in this place in JDK 11. And ... this file is not there. The directory structure also underwent some refactoring.

But it’s not so easy to leave us!

Any javist has a bit of a necromancer in his soul, and maybe even a necrophil. Therefore, now there will be NONCOMANTAINMENT IN ACTION!

First you need to appeal to the soul of the dead and find out when he died:

 $ hg log --template "File(s) deleted in rev {rev}: {file_dels % '\n {file}'}\n\n" -r 'removes("**/fprofiler.cpp")' File(s) deleted in rev 47106: hotspot/src/share/vm/runtime/fprofiler.cpp hotspot/src/share/vm/runtime/fprofiler.hpp hotspot/test/runtime/MinimalVM/Xprof.java 

Now you need to find out the cause of his death:

 hg log -r 47106 changeset: 47106:bed18a111b90 parent: 47104:6bdc0c9c44af user: gziemski date: Thu Aug 31 20:26:53 2017 -0500 summary: 8173715: Remove FlatProfiler 

So, we have a killer: gziemski . Let's find out why he nailed this unfortunate file.

To do this, go through the fat in the ticket specified in the summary of the commit. This is JDK-8173715 :

Remove FlatProfiler:

This is a concept for the GC.

For-shi-be. In fact, now we are offered to repair the corpse just so that the build is ready. Which has decomposed so much that even our necromancer colleagues from OpenJDK have abandoned.

Let's revive the dead man and try to ask him what he remembered last. He was already dead in revision 47106, so in revision, one less is “a second before”:

 hg cat "~/git/jdk11/hotspot/src/share/vm/runtime/fprofiler.cpp" -r 47105 > ~/tmp/fprofiler_new.cpp cp ~/git/jdk8u/hotspot/src/share/vm/runtime/fprofiler.cpp ~/tmp/fprofiler_old.cpp cd ~/tmp diff fprofiler_old.cpp fprofiler_new.cpp 

Unfortunately, absolutely nothing about return strcmp(name, _name) == 0; Diff no. Potsienta died of a blow with a blunt sharp object (rm utility), but at the time of death he was incurably ill.

Let's dig into the essence of the error.

This is what the author of the code would like to tell us:

  const char *name() const { return _name; } bool is_compiled() const { return true; } bool vm_match(const char* name) const { return strcmp(name, _name) == 0; } 

Now a little philosophy.

The C11 standard in clause 7.1.4, “Use of library functions”, explicitly says:

The following is a statement of value (such as [...] a null pointer [...]) [...] the behavior is undefined.

That is, now the whole question is whether there is some kind of “explicitly stated otherwise” . In the description of strcmp in section 7.24.4, nothing like this is written, and I have no other sections for you.

That is, we have an undefined behavior here.

Of course, you can take and rewrite this piece of code, surrounding it with a check. But I'm not at all sure that I correctly understand the logic of people who use UB where it should not be. For example, some systems generate SIGSERV zero dereferencing, and a hacker lover can get involved in this, but this behavior is optional and can bang on another platform.

Yes, of course, someone will say that you are your own fool, that you are using GCC 7.3, but in GCC 4 everything would be fine. But undefined behavior! = Unspecified! = Implementation defined. This is for the last two can be laid to work in the old compiler. And UB in the sixth version was UB.

In short, I was very sad over this complex philosophical question (whether to crawl with my assumptions into the code) when I suddenly realized that it was possible and different.

There is another way

As you know, good heroes always go around.

Even apart from our philosophy about UB, there are an incredible amount of problems there. Not the fact that they can be repaired until the morning. Not the fact that I do not nakosyachu my crooked hands. Even less, the fact that it will be accepted upstream: the last patch in jdk8u was 6 weeks ago, and it was the global merge of the new tag.

Just imagine that the code above is actually written correctly. All that stands between us and its execution is a kind of warning that was perceived as an error due to a bug in the build system. But we can build the assembly system.

The Witcher Geralt of Rivia once said:

“Evil is evil, Stregobor,” the Witcher said seriously, getting up. - The smaller, the larger, the average — everything is one, the proportions are conditional, and the boundaries are blurred. I am not a holy hermit, not only one good did in life. But if you have to choose between one evil and another, I prefer not to choose at all.



- Zło to zło, Stregoborze - rzekł poważnie wiedźmin wstając. - Mniejsze, większe, średnie, wszystko jedno, proporcje są umowne a granice zatarte. Nie jestem świątobliwym pustelnikiem, nie samo dobro czyniłem w życiu. Ale jeżeli mam wybierać pomiędzy jednym złem a drugim, to wol nie wybierać wcale.

This is a quote from the book "Last Wish," the story "Lesser Evil." We know that Geralt almost never could play the role of a truly neutral character until the end, and even died due to the regular classical chaotic good behavior.

So let's zashkvarimsya about the lesser evil. Loosen the assembly system.

At the very beginning, we already saw this exhaust:

 grep -rl "Werror" . ./common/autoconf/flags.m4 ./hotspot/make/linux/makefiles/gcc.make ./hotspot/make/bsd/makefiles/gcc.make ./hotspot/make/solaris/makefiles/gcc.make ./hotspot/make/aix/makefiles/xlc.make 

Comparing these two files, I broke the whole face with a facepalm and realized the difference in the culture of the two platforms:

BSD is a story about freedom and choices:

 # Compiler warnings are treated as errors ifneq ($(COMPILER_WARNINGS_FATAL),false) WARNINGS_ARE_ERRORS = -Werror endif 

GNU / Linux is an authoritarian purist mode:

 # Compiler warnings are treated as errors WARNINGS_ARE_ERRORS = -Werror 

Well, it would still be forwarded to linux via XX_FLAGS , this variable is not taken into account when calculating WARNINGS_ARE_ERRORS ! In the GNU / Linux build, we simply have no choice but to follow defaults lowered over.

Well, or you can make it easier and change the value of WARNINGS_ARE_ERRORS for a short, but no less powerful -w . How do you like that, Ilon Musk?

As you might have guessed, this completely solves this build problem.

When the code is collected, you see a bunch of strange, creepy looking problems flying by. Sometimes it was so scary that I really wanted to press ctrl + C and try to figure it out. But no, it is impossible, it is impossible ...

It seems that everything has gathered and has not brought any additional problems. Although I, of course, did not dare to start testing. Still, the night, the eyes begin to stick together, but somehow I don’t want to switch to the last resort - the four banks of energy from the refrigerator.

Falls into the crust

The build has passed, the executables have been generated, we are great.

And here we are at the finish line. Or did not come?

Our assembly is in the following way:

 export JAVA_HOME=~/git/jdk8u/build/linux-x86_64-normal-server-fastdebug/jdk export PATH=$JAVA_HOME/bin:$PATH 

When you try to run the java executable file, it instantly falls into the crust. For those who are not familiar - it looks like this:

At the same time, Alex has Debian 9.5, and I have Ubuntu. Two different versions of GCC, two differently looking crusts. I have innocent pranks with a manual strcmp patch and a few more places, Alex doesn’t. What is the problem?

This story is worthy of a separate story, but here let's go straight to dry conclusions, otherwise I will never finish this post.

The problem is that our favorite C ++ - pogromists again used undefined behavior.

(And where it depends on the implementation of the compiler in an unknown way. However, we must remember that UB is always UB, even on a known version of the compiler, it cannot be laid on it)

In one place, we turn to the field of an undeconstructed class there, and everything breaks down. Do not ask how it happened, everything is difficult.

It is very difficult for a javista to imagine how you can refer to an underconstructed class, except by releasing a link to it directly from the constructor. Fortunately, the wonderful C ++ language can do everything or almost everything. I will write an example with some kind of pseudocode:

 class A { A() { _b.Show(); } private: static B _b; }; A a; BA::_b; int main() { } 

Have a nice debug!

If you look at C ++ 98 [class.cdtor]:

For an object of the non-POD class type ...

Starting with some version of GCC (and I have 7.3), optimization of “lifetime dead store elimination” has appeared, which considers that we only turn to the object during its lifetime, and mounts it outside of lifetime.

The solution is to disable new optimizations and return them as they were in the old GCC:

 CFLAGS += -fno-strict-aliasing -fno-lifetime-dse -fno-delete-null-pointer-checks 

There is a discussion about this here .

For some reason, the panelists decided that this would not be included in the upstream. But you still need to try to send.

Add these options to our ./hotspot/make/linux/makefiles/gcc.make , rebuild everything again and see the cherished lines:

 t$ ~/git/jdk8u/build/linux-x86_64-normal-server-fastdebug/jdk/bin/java -version openjdk version "1.8.0-internal-fastdebug" OpenJDK Runtime Environment (build 1.8.0-internal-fastdebug-me_2018_09_10_08_14-b00) OpenJDK 64-Bit Server VM (build 25.71-b00-fastdebug, mixed mode) 

Conclusion

You probably thought that the output would be the following: “Java is some kind of hell, there is no garbage in the code, there is no support, everything is bad.”

This is not true! On the contrary, the examples above show from what terrible evil our friends, necromancers from OpenJDK, keep us.

And despite the fact that they have to live and use C ++, tremble from each UB and change the compiler version and learn the details of the platforms, the final user code in Java is insanely stable, and on the builds posted on the official websites of companies such as Azul, Red Hat and Oracle, it is hardly possible to run into a crust in the simple case.

The only sad thing is that most likely the errors found are unlikely to be accepted in jdk8u. We took JDK 8 simply because it's easier for us to patch it right here and now, and we will have to deal with JDK 11. Nevertheless, to use JDK 8 in 2018 - IMHO, this is a very bad practice, and we are not doing this from a good life. Perhaps in the future our life will improve, and you will read many more incredible stories from the world of JDK 11 and JDK 12.

Thank you for the attention paid to such a boring text without pictures :-)

Minute advertising. The Joker 2018 conference will take place very soon, featuring many prominent Java and JVM specialists. View the full list of speakers and reports can be on the official site . I'll be there too, it will be possible to meet and grind for life and OpenJDK.