Spring CgLib Dynamic AOP Proxies - proper Pointcut equals method is simply essential

Dynamic proxies can be very nasty if you don't know what happening under the cover. Last week I was searching for the memory leak that caused our application to crash. Even though Tomcat had assigned 1GB memory for heap and 0,5GB for PermGenSpace it stood alive for only approximately twelve hours. It's pretty nasty situation having known that application is only in betatesting with relatively low traffic.

When analyzing generated heap dump I have found, that memory leak was caused by web application classloader, that managed thousands of CgLib dynamically generated classes. I was using Eclipse Memory Analyzer, that's probably the best tool for memory heap dump analysis I have ever seen. It's the third time it quickly identified the suspicious classes, by heuristic analysis called Leak suspect.

I could identify the offending code fairly quickly. At one place I was creating dynamic proxies around existing instaces, registering advices that took care of caching results of methods' calls on original object. Code looked like that:


public Object getProxiedVersion(Object original) {
    //will create proxy factory based on original object
    ProxyFactory proxyFactory = new ProxyFactory(original);
    //this will add our result caching advisor
    proxyFactory.addAdvisor(
       new DefaultPointcutAdvisor(
          new DataProviderPointcut(),
          new ResultCachingAdvice()
       )
    );
    //this will cause CgLib will be used to generate proxy class
    //and not JDK proxy proxying only interfaces that orignal object implements
    proxyFactory.setProxyTargetClass(true);
    //this is only optimalization thing - means we won't touch advisors
    //after proxy has been created - so Spring could optimalize calls
    proxyFactory.setFrozen(true);
    //this will create dynamic class and instance of it
    return proxyFactory.getProxy();
}

I couldn't find out where the problem lies - this code looked right according to Spring documentation.

So I dug deep to the AOP internals and there are things I have discovered:

Generated classes are never garbage collected

First I thought that dynamically generated classes could be garbage collected when there are no living instances of them. But that's not true - once class is generated and first instance of it is created it keeps living in classloader object until the classloader itself is garbage collected (in our situation it would infer stopping web application in Tomcat and starting it again). This statement I have proven to myself by this test:


public void testDynamicClassGarbageCollection() {
    //create proxy factory
    ProxyFactory proxyFactory = new ProxyFactory(new Object());
    proxyFactory.setProxyTargetClass(true);
    //we'll force proxy factory to use our own classloader
    //with defining standard classloader as a parent of it
    ClassLoader clsLdr = new ClassLoader(
            Thread.currentThread().getContextClassLoader()
    ) {};
    //create proxy class and intance and keep weak references
    //to them
    WeakReference proxyRef = new WeakReference(
            proxyFactory.getProxy(clsLdr)
    );
    WeakReference proxyClassRef = new WeakReference(
            proxyRef.get().getClass()
    );
    //this should destroy all non referenced instances
    //we have no reference either to proxy or proxy class
    //(WeakReferences don't count)
    System.gc();
    //proxy instance gets garbage collected
    assertNull(proxyRef.get());
    //but class does not!
    assertNotNull(proxyClassRef.get());
    //we have dispose class loader in order to dispose generated
    //proxy class
    clsLdr = null;
    System.gc();
    assertNull(proxyClassRef.get());
}

This means that if code of getProxiedVersion method led to the repetitive class generation, there is no way how one could keep Tomcat living for a long time. But such thing would have for sure forced authors of Spring not to recommend programmatical proxy creation - or at least put some kind of warning into the documentation. But that's not true.

It's easy to check whether your programmatic AOP code leaks PermGenSpace

I have also wrote simple test, that proved getProxiedVersion metod was flawed:


public void testGetProxiedVersion() {
    long iteration = 0;
    Lookup tested = new Lookup();
    do {
        tested.getProxiedVersion(new Object());
        iteration++;
        if (iteration % 100 == 0) {
            System.out.println("Successfuly proxied: " + iteration + " objects");
        }
    } while (true);
}

Running this test with JVM constrained to -XX:MaxPermSize=8m led to quick test fail on OutOfMemoryError (it created roughly about 400 proxied instances and finished with OOME). I played a bit with the getProxiedVersion method and found out that if I comment out following piece of tested code:


//this will add our result caching advisor
proxyFactory.addAdvisor(
   new DefaultPointcutAdvisor(
      new DataProviderPointcut(),
      new ResultCachingAdvice()
   )
);

test kept running creating thousands of proxied instances. I tried to exchange my ResultCachingAdvisor for some standard Spring advisor (for example new DefaultIntroductionAdvisor(new ConcurrencyThrottleInterceptor())) and the test was still happily running. So the problem wasn't inside getProxiedVersion method, but in the DefaultPointcutAdvisor code or one of instances passed to the constructor! I was closer to the final solution, but not yet there.

First I blamed my ResultCachingAdvice because it was much more complicated than the pointcut implementation. So I exchanged it for some standard Advice provided by Spring. When leak didn't disappeared, I did the same with Pointcut implementation and ... voila, leak was gone. So the wrong piece of the puzzle was been discovered, but where was the problem?

Pointcut implementation was quite simple - no error visible on the first sight:


public class DataProviderPointcut implements Pointcut {
    public ClassFilter getClassFilter() {
        return new RootClassFilter(DataProvider.class);
    }
    public MethodMatcher getMethodMatcher() {
        return MethodMatcher.TRUE;
    }
}

CgLib optimizes class generation and won't generate the same dynamic class again

In order to break this mystery we have to know how CgLib works internally with class generation. There is a magic flag useCache in AbstractClassGenerator class that causes CgLib not to generate class it has already created again (see protected method Object create(Object key)). This magic flag is true by default, so in our test there should be only one dynamic proxy class generated and not thousands as it was in my case.

The key to our problem is the mechanism how CgLib recognizes whether two generated classes equals. I won't pretend, that I deeply understand to this mechanism - but key part of it is hidden in the Enhancer class, especially in the method createHelper:


private Object createHelper() {
	validate();
	if (superclass != null) {
		setNamePrefix(superclass.getName());
	} else if (interfaces != null) {
		setNamePrefix(interfaces[ReflectUtils.findPackageProtected(interfaces)].getName());
	}
	return super.create(KEY_FACTORY.newInstance((superclass != null) ?
           superclass.getName() : null,
           ReflectUtils.getNames(interfaces),
           filter,
           callbackTypes,
           useFactory,
           interceptDuringConstruction,
           serialVersionUID)
       );
}

We can see in that snippet, that CgLib examines several things to distinguish two classes. For example:

  • superclass of our class
  • implemented interfaces
  • callbackTypes
  • filter
  • serialVersionUID and so on

It's not easy to understand it by just staring at the code, so the debugger might come handy. Problematic part is the filter (CallbackFilter interface implemented by Cglib2AopProxy$ProxyCallbackFilter). Let's examine its equals method (look expecially at the end where advisors are consulted):


public boolean equals(Object other) {
	if (other == this) {
		return true;
	}
	if (!(other instanceof ProxyCallbackFilter)) {
		return false;
	}
	ProxyCallbackFilter otherCallbackFilter = (ProxyCallbackFilter) other;
	AdvisedSupport otherAdvised = otherCallbackFilter.advised;
	if (this.advised == null || otherAdvised == null) {
		return false;
	}
	if (this.advised.isFrozen() != otherAdvised.isFrozen()) {
		return false;
	}
	if (this.advised.isExposeProxy() != otherAdvised.isExposeProxy()) {
		return false;
	}
	if (this.advised.getTargetSource().isStatic() != otherAdvised.getTargetSource().isStatic()) {
		return false;
	}
	if (!AopProxyUtils.equalsProxiedInterfaces(this.advised, otherAdvised)) {
		return false;
	}
	// Advice instance identity is unimportant to the proxy class:
	// All that matters is type and ordering.
	Advisor[] thisAdvisors = this.advised.getAdvisors();
	Advisor[] thatAdvisors = otherAdvised.getAdvisors();
	if (thisAdvisors.length != thatAdvisors.length) {
		return false;
	}
	for (int i = 0; i < thisAdvisors.length; i++) {
		Advisor thisAdvisor = thisAdvisors[i];
		Advisor thatAdvisor = thatAdvisors[i];
		if (!equalsAdviceClasses(thisAdvisor, thatAdvisor)) {
			return false;
		}
		if (!equalsPointcuts(thisAdvisor, thatAdvisor)) {
			return false;
		}
	}
	return true;
}
private boolean equalsAdviceClasses(Advisor a, Advisor b) {
	Advice aa = a.getAdvice();
	Advice ba = b.getAdvice();
	if (aa == null || ba == null) {
		return (aa == ba);
	}
	return aa.getClass().equals(ba.getClass());
}
private boolean equalsPointcuts(Advisor a, Advisor b) {
	// If only one of the advisor (but not both) is PointcutAdvisor, then it is a mismatch.
	// Takes care of the situations where an IntroductionAdvisor is used (see SPR-3959).
	if (a instanceof PointcutAdvisor ^ b instanceof PointcutAdvisor) {
		return false;
	}
	// If both are PointcutAdvisor, match their pointcuts.
	if (a instanceof PointcutAdvisor && b instanceof PointcutAdvisor) {
		return ObjectUtils.nullSafeEquals(((PointcutAdvisor) a).getPointcut(), ((PointcutAdvisor) b).getPointcut());
	}
	// If neither is PointcutAdvisor, then from the pointcut matching perspective, it is a match.
	return true;
}

As you can see, advices don't need to have equals method as ProxyCallbackFilter compares only their classes. On the contrary Pointcuts are compared by calling their equals methods. So the exact problem in my case was missing implementation of equals method, that got inherited from java.lang.Object and returned true only for the same object instance comparisement.

Solution

When we know all this, the solution is quite simple. Better said, we have a plenty solutions at hand:

  • use prepared Pointcut from Spring - they just work
  • write a proper equals and hashCode methods in your Pointcut implementations
  • use static or singleton references to your Pointcuts (as Spring often does - just look at Pointcut.TRUE) - then default equals implementation in java.lang.Object will work as there is only one instance of the Pointcut class

This conclusion should be more propagated in Spring documentation, I suppose. But at least this article does it. Happy coding ...