Hidden Classes in Java 15
Article Management Technical Details
1. Introduction
Java can have anonymous classes from the very start.
(Well, actually, they came in version 1.1 only.)
However, anonymous classes were not anonymous.
You did not need to name them, but under the hood, they were named by the Java compiler.
If you are familiar with the command javap, you can "disassemble" a JAR file and see the name of the compiler generated for the anonymous classes.
Java 15 introduced hidden classes, which do not have a name. Almost, as you will see. It is not part of the language but part of the JDK. There is no language element to create hidden classes, but JDK methods and classes come to the rescue.
In this article, we will discuss
-
what hidden classes are, and what is the reason to have them,
-
how you can use hidden classes,
-
how to load hidden classes using the JDK methods and, finally
-
how to easily create and load hidden classes using SourceBuddy.
|
Note
|
I created SourceBuddy, an Apache v2.0 licensed open-source program. While creating the code, I learned a few things about hidden classes I wanted to share with you. You may also look at this article as a SourceBuddy promotion, which is ok if you do that. Nevertheless, I hope to successfully add extra value to this article so that this is not simply a promo. |
2. What are hidden classes?
There is an easy-to-read introductory article about hidden classes on Baeldung. If you are impatient and do not care about some intricate details, go there and read that article. Baeldung articles are always short, focusing on the most important and correct. They give a good starting point, which there would be no reason to repeat.
Hidden classes were proposed in the JEP371, and it reads:
hidden … classes that cannot be used directly by the bytecode of other classes
It is a bit short and may not be easy to understand.
A hidden class is loaded into the JVM. When a class is in source code format or byte code, it cannot be "hidden". This term can refer only to loaded classes. Calling them secretly loaded classes could be more appropriate.
A class gets hidden when it is loaded in a particular way so that it remains secret in front of other code parts. Remaining hidden does not mean that other codes cannot use this class. They can so long as long they "know" about the secret. The big difference is that this class is not "advertised" because you cannot find it using the name.
When you load a class the hidden way creating a hidden class, you will have a reference to this class. Using the reflective methods, you can instantiate the class many times, and then you can invoke methods, set, and get fields. If the class implements an interface or extends a class, you can cast the instance reference to the interface and class and invoke the methods without reflection.
The class is hidden for two reasons:
-
it does not have a name other classes could reference, and
-
there is no reference from the class loader to the class.
When you call getName() or getSimpleName() on a variable referencing a hidden class, you will get some string.
These are names for messages for humans and are irrelevant for the other classes.
When a class refers to another class it needs the canonical name.
getCanonicalName() returns null.
The canonical name is the actual name of the class, which is non-existent in the case of hidden classes.
Since the class cannot be found through the class loader without the canonical name, there is no reason for the loader to keep a reference to the class. Why would it keep a reference when it cannot give the class to anyone? Keeping a reference would have only one side effect: preventing the GC from unloading the class so long as the class loader is alive.
Since there is no reference from the class loader, the GC can unload the class object as soon as it is out of use.
3. What is the use of hidden classes?
The JEP371 describes the reason for hidden classes. It says
Allow frameworks to define classes as non-discoverable implementation details of the framework so that they cannot be linked against other classes nor discovered through reflection.
Many frameworks use dynamically created classes. They are proxy classes in most cases. A proxy class implements an interface or extends another class, and when invoked, it calls an instance of the interface or the original class. It usually does something extra as well, or else there would be no reason for the proxy class and instance.
An example is the Spring framework when your code requires injecting a request bean into a session bean. (Or any other shorter lifecycle bean into a longer one.) Several threads can serve different requests at the same time, all belonging to the same session. All these threads will see the same session bean, but they magically will see their request beans. The magic is a proxy object extending the request bean’s class. When you call a method on the request bean, you invoke the proxy instance. It checks the thread and the request it serves and forwards the call to the appropriate request bean.
Another example is JPA lazy loading. You can have an SQL table where each row references the previous one. When you try to load the last record, it will automatically load the previous one, which indeed will load the one before. It will load the whole table. It happens unless you annotate the field as lazy. It means that the actual data from the database has to be loaded only when it is needed.
When you load the record, you get a proxy object. This proxy object knows which record it refers to and will load the record from the database only when a method is called.
The same mechanism is used for Aspect Oriented Programming and many other cases.
You can create a proxy class using only the JDK reflection API so long as the target class implements the interface you want to proxy. You can use the ByteBuddy library if there is no such interface.
|
Note
|
The cglib library is widely used and well-known in many frameworks, but it has been deprecated recently. |
When you create such classes, you do not need any name for these classes. You get the reference to the class and the reference to the instance. The framework injects the reference to the field it has to, and then the code uses them as any object. It does not need to know the name of the class. All it needs to know is that it is an instance of the target class or interface. However, some codes may discover the name. These classes have some names that reflection can discover. Some "clever" junior may discover it and play some neat trick that you may have later issues maintaining. Would it be better if there was no name at all? Probably yes, it would be cleaner. Hence: hidden classes.
|
Note
|
The proxy classes may also cause issues when you implement the |
In addition to that, there is another reason to have hidden classes. As soon as a class has a name, it is possible to discover it by the name. The class loader has to keep the class alive to keep it discoverable. The class loader has a reference to the loaded classes. It means the garbage collector will not be able to collect the class, even when it is no longer in use.
If a class has no name, the class loader does not need to keep a reference to this class. Class loaders do not keep references to hidden classes unless you explicitly instruct them to do so. When all instances of a hidden class are collected, and there is no reference to the class, the garbage collector will recognize it as garbage. The class loader will not keep the class in memory.
That way, the frameworks will not over-consume memory when a long-running code creates a lot of classes. Better frameworks that collect unused classes do not need to create separate class loaders for these ephemeral classes. There is no need to create a short-living, disposable class loader to make the class also disposable.
Support extending an access control nest with non-discoverable classes.
It is the second bullet point in the list of goals in the JEP371. JVM can load hidden classes in a way that they become a member of a nest. What is a nest?
|
Note
|
If you know what a nesting host is and are impatient, jump to the following quote. |
Well, it is a long story, but since it is Christmas time, I am on holiday, and the press will not print this article on the paper wasting tree, here it goes. (I wrote this article for weeks. It is still not published, and already next year, January.)
Once upon a time, Java version 1.0 did not have inner classes.
Now, you better stop reading it here if you ask me what inner classes are.
Then Java version 1.1 introduced inner classes but did not change the JVM structure.
The JVM did not know anything about inner classes.
The Java compiler created regular (almost) top-level classes from the inner classes.
It invented some funny names, like A$B when there was a class B inside A.
|
Note
|
You can try to define an A$B top-level class in the same package where the class A containing the class B is.
A$B is a valid name.
You will see what the compiler does.
|
There was some hacking with the visibility though. An inner class has the same visibility as the top-level class. Anything private inside one compilation unit (file) is visible. Visibility, however, is also enforced by the JVM. But the JVM sees two top-level classes. The compiler generated bridge methods in the classes wherever needed to overcome this issue. They are package level for the JVM, and when called, they pass on the call to the private method.
Then came Java 11 something like 25 years later and introduced the nest control. Since Java 11, every class has a relation to another class or to itself, which is the nest host of the class. Classes having the same nest host can see each other’s, private members. The JVM does not need the bridge methods anymore.
When you load a class hidden, you can specify it to become a member of the same nest (having the same nest host) as the class that created the lookup object.
|
Note
|
We have not yet discussed what a lookup object is and how to load a class hidden. It will come. As for now: a lookup object is something that can load a byte array as a hidden class into the JVM memory. When a lookup object is created from inside a method of a class, the lookup object will belong to that class. When a class is loaded as hidden using the lookup object, it is possible to pass an option to make the new hidden class belong to the nest in which the code created the lookup object. |
Without the hidden class functionality, I do not know any other possibility to load a class that will belong to an already existing nest. If you know of any possibility, write it in a comment.
The following bullet point reads:
Support aggressive unloading of non-discoverable classes, so that frameworks have the flexibility to define as many as they need.
It is an important point. When you create a class, it remains in the memory so long as the classloader is alive. Classloaders keep references to all the classes they loaded. These references say that some code may ask the classloader to return the loaded class object by the name. The application logic may long forget the class; nobody will ever need it. Still, the garbage collector cannot collect it because there is a reference in the class loader. A solution is to create a new class loader for every new non-hidden dynamically created class, but that is overkill.
Classloaders loading hidden classes do not keep a reference to the hidden class by default. As with the nesting host, it is possible to provide an option to differ.
I do not see any reason. There is no name, not discoverable, but keep an extra reference so the GC cannot throw it away. If you see any reasonable use case, again: comment.
Deprecate the non-standard API sun.misc.Unsafe::defineAnonymousClass, with the intent to deprecate it for removal in a future release.
Very well. Yes. Absolutely. Separate articles and many of them.
Do not change the Java programming language in any way.
Nice point. Sure.
With these, we discussed what hidden classes are. You should have a firm understanding of their nature and why they are essential. We also derailed a bit to nest hosting or host nesting, nesting hosting… whatever. I hope it was of some value.
In the following, I will discuss how we create hidden classes using the JDK API and then using SourceBuddy.
4. Creating Hidden Classes
Articles and tutorials showing how to load hidden classes use precompiled Java classes.
These are usually part of the running application.
The tutorial calculates the path to the .class file and reads the byte code.
Technically this is correct but does not demonstrate the basic need for hidden class loading: load dynamically created classes hidden. These classes are not dynamically created and could be loaded the usual way.
In this article, we will create a class from text, Java source on the fly — during run-time — and then load the resulting byte code as a hidden class.
Code Sample Disclaimer
The sample project for this article contains only unit test files.
The class is TestHiddenClassLoader.
We have the source code for the hidden class stored in a field variable.
1. private static final String CODE1 = """
2. package com.javax0.blog.hiddenclasses;
3.
4. public class MySpecialClass implements TestHiddenClassLoader.Hello {
5.
6. @Override
7. public void hello() {
8. System.out.println("Hello, from the hidden class.");
9. }
10. }
11. """;The interface is also inside the same class.
1. interface Hello {
2. void hello();
3. }
4.The following code is from one of the unit tests:
1. final var byteCode = Compiler.java().from(CODE1).compile().get();
2. final var lookup = MethodHandles.lookup();
3. final var classLookup = lookup.defineHiddenClass(byteCode, true);
4. final var helloClass = (Class<Hello>) classLookup.lookupClass();
5.
6. final var hello = helloClass.getConstructor().newInstance();
7. hello.hello();We use the SourceBuddy library in this code to compile the Java source to byte code. The first line of the sample does that. We use SourceBuddy version 2.1.0.
We need a lookup object to load the compiled byte code as a hidden class.
This object is created on the second line.
The lookup object is used on the third and fourth lines to load the class hidden.
Line 3 defines the class loading it into the JVM.
The second argument, true, initializes the class.
That is when the static{} blocks execute.
The last line invokes the interface-defined method hello().
Now the local variable hello is an instance of an object, a hidden class.
What are a hidden class’s name, simple name, and canonical name?
Let’s print it out.
1. System.out.println("1. " + hello.getClass());
2. System.out.println("2. " + hello.getClass().getClassLoader());
3. System.out.println("3. " + this.getClass().getClassLoader());
4. System.out.println("4. " + hello.getClass().getSimpleName());
5. System.out.println("5. " + hello.getClass().getName());
6. System.out.println("6. " + hello.getClass().getCanonicalName());
7. System.out.println("7. " + lookup.getClass());
8. System.out.println("8. " + lookup.getClass().getClassLoader());Output Disclaimer
Hello, from the hidden class.
1. class com.javax0.blog.hiddenclasses.MySpecialClass/0x00000008011b0c00
2. jdk.internal.loader.ClassLoaders$AppClassLoader@5b37e0d2
3. jdk.internal.loader.ClassLoaders$AppClassLoader@5b37e0d2
4. MySpecialClass/0x00000008011b0c00
5. com.javax0.blog.hiddenclasses.MySpecialClass/0x00000008011b0c00
6. null
7. class java.lang.invoke.MethodHandles$Lookup
8. nullYou can see the output from calling hello() and then the name as printed from the implicit toString() from the class object, the class loader that loaded the hidden class, the simple name, the name, and in the last line the canonical name.
This last one is interesting as it is null, showing no class name.
It is hidden.
The class, although hidden, has a reference to the class loader that loaded it. It is needed when there is anything to resolve during the execution of the code. The difference is that the class loader does not have a reference to the class. One direction from the class to the loader exists, but the other direction from the loader to the class does not.
The class loader is the same as the one that loaded the class calling MethodHandles.lookup().
You can see that since we printed out the class loader of the this object in the test.
Finally, we also print out the class of the lookup object and the class loader.
The latter is null, which means the bootstrap class loader loaded it. (For more information on class loaders, I can recommend reading the article class loaders from the Baeldung blog.)
You should also note that the interface hello is package private.
It is still visible for the dynamically created code because it is in the same package and module.
|
Note
|
Starting with Java 9, there is a module system in Java. Many developers I meet say they are not interested in JPMS; they do not need to use it. The fact is that you DO use it, whether you want it or not. It is the same as concurrent programming. Java is concurrent; at least there are three threads in a JVM, so your code runs in a concurrent environment, whether you want it or not. You may not have trouble understanding the details for a long time. However, when you start digging deeper and creating code that uses some "tricks" or does something special, you almost certainly face some weird errors. You must know and understand the underlying theory to understand the errors, handle them, mitigate the cause, and fix the bug. Loading hidden classes dynamically created is precisely such a trick. You should learn Java Modules. |
When the hidden class is loaded, it is in the same package as the one where the interface is defined. It is not enough, however, as we will see an example in the next section. It is also a requirement that the same class loader loads the interface and the hidden class. That way, the interface, and the hidden class are in the same module, in this case, the same unnamed module. The different class loaders load classes into different modules; thus, when you load a class using a different class loader, it may not see the package fields, methods, interfaces, etc., even if they are in the same package.
It is not the only requirement that the lookup object is from the same module. It is also a requirement that it is from the same package as the class to be loaded. We must stop here to clarify things, to be painfully precise, because it is easy to confuse things at this point.
The lookup object is an instance of a class in the java.lang.invoke package.
The class loader loaded this class is null as shown in the output.
It means the bootstrap class loader.
The bootstrap class loader is implemented in C/C++ and not in Java.
No corresponding Java object represents this class loader; thus, there cannot be a reference to it.
It is solved by returning null from getClassloader().
There is a module, package, and class that "belongs" to the lookup object.
The code’s module, package, and class were called the MethodHandles.lookup() method.
You cannot create a hidden class from one package for another. If you try that, like in the following sample code:
1. try {
2. final var byteCode = Compiler.java()
3. .from("package B; class A{}").compile().get();
4. MethodHandles.lookup().defineHiddenClass(byteCode, true);
5. } catch (Throwable t) {
6. System.out.println(t);
7. }still from the test class com.javax0.blog.hiddenclasses.TestHiddenClassLoader.
The class to be loaded is NOT in the same package as the caller for MethodHandles.lookup().
It will result in the printout:
java.lang.IllegalArgumentException: B.A not in same package as lookup class5. Creating hidden classes the easy way
In the previous section, we created a new class dynamically and loaded the new class hidden.
The loading was done using lookup objects we acquired from the MethodHandles class.
In this section, we will see how we can do the same by calling the fluent API of SourceBuddy.
The code creating a class saying hello is the following:
1. final var hello = Compiler.java()
2. .from(CODE1.replaceAll("\\.Hello", ".PublicHello")).hidden()
3. .compile().load().newInstance(PublicHello.class);
4. hello.hello();In this code, we replaced the interface from Hello to PublicHello, which you may guess:
1. public interface PublicHello {
2. void hello();
3. }It is essentially the same as the previous interface but is public.
The process is much more straightforward than before.
We specify the source code; we declare that it is a hidden class calling hidden(), and we compile, load, and ask for an instance cast to PublicHello.
If we want to use the package-private interface, like (not replacing Hello to PublicHello):
1. Assertions.assertThrows(IllegalAccessError.class, () ->
2. Compiler.java().from(CODE1).hidden().compile().load().newInstance(PublicHello.class));we will get an error.
java.lang.IllegalAccessError: class com.javax0.blog.hiddenclasses.MySpecialClass/0x00000008011b1c00 cannot access its superinterface com.javax0.blog.hiddenclasses.TestHiddenClassLoader$Hello (com.javax0.blog.hiddenclasses.MySpecialClass/0x00000008011b1c00 is in unnamed module of loader com.javax0.sourcebuddy.ByteClassLoader @4e5ed836; com.javax0.blog.hiddenclasses.TestHiddenClassLoader$Hello is in unnamed module of loader 'app')The reason is explained clearly in the error message.
The interface and the class implementing it are in two different modules.
Both are unnamed modules, but they are not the same.
In Java, starting with Java 9, there are modules, and when the application does not use modules, it essentially creates pseudo modules putting the classes there.
The JDK classes are still in modules, like java.base.
The hidden class creation, as created above, uses a separate class loader to load the dynamically written Java class. The separate class loader loads classes to its module. Code in different modules cannot see classes from other modules unless they are public.
Although SourceBuddy does a little trick to load a class hidden, it cannot overcome this restriction.
Loading a hidden class needs a lookup object.
The application usually provides this object.
The calls above do not specify any lookup object, but SourceBuddy still needs one.
To have one, it creates one.
The lookup object remembers the class called MethodHandles.lookup() to create one.
When loading a class hidden, it is required that the lookup object "belongs" to the class’s package.
The lookup object was created, calling for it from a class, which is in that package.
The lookup object will "belong" to that class and hence to the class’s package.
To have a lookup object that comes from a class from a specific package we need a class in that package that can give us one.
If there is none in the code, we must create one dynamically.
SourceBuddy does that exactly.
It creates the Java source code for the class, compiles it and loads it, instantiates it, and calls the Supplier<MethodHandles.Lookup> defined get() method the class implements.
It is a kind of trick that seems to violate the access control built-in to Java. We seem to get a new hidden class in a package that was not prepared for it. A package is protected from external access in Java (trivial). Only public and protected members and classes can be used from outside the package. The package can be accessed using reflection from the outside, but only in the same module, or the module has to be opened explicitly. Similarly, an object loaded using a lookup object should be in the same package and access the package’s internal members and whatnot if a class in the package provided that lookup.
As we can see from the error message, it only seems to be the package. In reality, the new hidden class is in a package with the same name but in a different module.
If you want to have a hidden class in the same package and not only a package with the same name, you need a lookup object from that package.
In our example, it is simple.
Our Hello interface is in the same package as the test code so that we can create the lookup object ourselves:
1. final var hi = Compiler.java().from(CODE1).hidden(MethodHandles.lookup()).compile()
2. .load().newInstance(Hello.class);
3. hi.hello();Access to a lookup object may be a bit more complex in real-life examples. When the code calling SourceBuddy is in a different package than the code generated, the lookup object creation cannot be in the SourceBuddy calling code.
In the following example, we will see how that will be done.
We have a class OuterClass in the package com.javax0.blog.hiddenclasses.otherpackage.
1. package com.javax0.blog.hiddenclasses.otherpackage;
2.
3. import java.lang.invoke.MethodHandles;
4.
5. public class OuterClass {
6.
14. public static MethodHandles.Lookup lookup() {
15. return MethodHandles.lookup();
16. }
17. }|
Note
|
Some lines are skipped from the class. We will use those later. |
This class has a method, lookup().
It creates a lookup object and returns it.
We will have a proper lookup object if we call this method from our code.
Note that this class is in a different package and not the same as our test code.
Our test code is in com.javax0.blog.hiddenclasses, and OuterClass is a package deeper.
Essentially in a different package.
We also have another class for the demonstration.
1. package com.javax0.blog.hiddenclasses.otherpackage;
2.
3. class MyPackagePrivateClass {
4.
5. void sayHello(){
6. System.out.println("Hello from package private.");
7. }
8.
9. }It is a package-private class with a package-private method in it. If we dynamically create a hidden class, as in the following example:
1. final var hidden = Compiler.java().from("""
2. package com.javax0.blog.hiddenclasses.otherpackage;
3.
4. public class AnyName_ItWillBeDropped_Anyway {
5. public void hi(){
6. new MyPackagePrivateClass().sayHello();
7. }
8. }""").hidden(OuterClass.lookup()).compile().load().newInstance();
9. final var hi = hidden.getClass().getDeclaredMethod("hi");
10. hi.invoke(hidden);It will work.
There is one topic that we have not touched on. It is how to create a nestmate.
When you have a binary class file, you can load it as a nestmate to a class that provides a lookup object. The JVM does not care how that class was created. When we compile Java sources, we only have one possibility. The class has to be an inner class.
When you use SourceBuddy, you have to provide your source code as an inner class to the one you want the hidden to be nest mate with. The source code and the class was already provided when you compiled your code. It is not possible to insert into THAT source code any new inner class. We have to fool the compiler.
We provide a class having the same name as the one we want to insert our inner class later. When the compilation is done, we have the outer class and the inner class as well. We tell the class loading to forget the outer and only to load the inner one, hidden.
It is what we will do. This time we display here the whole outer class that we use for demonstration including the skipped lines.
1. package com.javax0.blog.hiddenclasses.otherpackage;
2.
3. import java.lang.invoke.MethodHandles;
4.
5. public class OuterClass {
6.
7. // skip lines
8. private int z = 55;
9.
10. public int getZ() {
11. return z;
12. }
13. // end skip
14. public static MethodHandles.Lookup lookup() {
15. return MethodHandles.lookup();
16. }
17. }As you will see, it has a private field and a getter to test the changed value effectively.
It also has the before-mentioned lookup() method.
The code dynamically creating an inner class is the following:
1. final var inner = Compiler.java().from("""
2. package com.javax0.blog.hiddenclasses.otherpackage;
3.
4. public class OuterClass
5. {
6. private int z;
7.
8. public static class StaticInner {
9. public OuterClass a(){
10. final var outer = new OuterClass();
11. outer.z++;
12. return outer;
13. }
14. }
15.
16. }""").nest(MethodHandles.Lookup.ClassOption.NESTMATE).compile().load()
17. .newInstance("StaticInner");
18. final var m = inner.getClass().getDeclaredMethod("a");
19. final var outer = (OuterClass)m.invoke(inner);
20. Assertions.assertEquals(56, outer.getZ());There is an OuterClass in the source, but it is only to help the compilation and to tell SourceBuddy the name of the nesting host.
When we call the method nest() with the option NESTMATE, it knows that the class OuterClass is the nesting host.
It also marks the class not to be loaded by the class loader ever.
The inner class compiles to a different byte code, and when it is loaded, it becomes a nestmate of OuterClass.
If you pay attention to the intricate details of Java access control discussed in this article, you will notice that we do not provide a lookup object.
And the example above still works.
How is it possible?
There is no magic.
When you call nest(), SourceBuddy looks for the already loaded version of OuterClass and fetches the lookup object using reflection.
To do that the outer class has to have a static field or method of type MethodHandles.Lookup.
OuterClass has a method, so SourceBuddy calls this method to get the lookup object.
The example above creates a static inner class. You can create the same way a non-static inner class as well.
|
Note
|
The difference between static and non-static inner classes in Java is that non-static inner class instances have a reference to an outer class instance. Static inner classes do not. It is where the name comes from. Static inner class instances belong to the class. Non-static belongs to an instance of the outer class. To get the reference to the outer class instance, the inner class’s constructor is modified. When you specify a constructor for an inner class, the compiled adds an extra parameter in front of the other parameters specified in the Java source code. This extra first parameter is the reference to the outer class instance. This reference is stored in a field not available at the source level but used by the code to access the fields and methods of the outer instance. |
The creation of a non-static inner class looks very much the same as the creation of a static inner class:
1. final var outer = new OuterClass();
2. final var inner = Compiler.java().from("""
3. package com.javax0.blog.hiddenclasses.otherpackage;
4.
5. public class OuterClass {
6. private int z;
7.
8. public class Inner {
9. public void a(){
10. z++;
11. }
12. }
13.
14. }""").nest(MethodHandles.Lookup.ClassOption.NESTMATE).compile().load()
15. .newInstance("Inner", classes(OuterClass.class), args(outer));
16. final var m = inner.getClass().getDeclaredMethod("a");
17. m.invoke(inner);
18. Assertions.assertEquals(56, outer.getZ());We need an instance of the outer class to instantiate the inner class.
It is the variable outer.
We must pass this variable to the constructor through the newInstance() API of SourceBuddy.
This method call has a version that accepts a Class[] and an Object[] array specifying the constructor argument types and values.
In the case of an inner class, it is the outer class and an instance.
6. Summary
This article discussed some details of the hidden classes introduced in Java 15. We went a little deeper than the usual introductory articles. Now you understand how hidden classes work and how to use them in your projects.
How to Comment
Comments
Please leave your comments using Disqus, or just press one of the happy faces. If for any reason you do not want to leave a comment here, you can still create a Github ticket.