It seems that there is indeed room for third-party vendors to get involved in the mobile code security business. However, many mobile code security issues cannot be addressed in a trivial manner. In this section, we raise a number of tricky issues that are harder to deal with than they seem at first. Many of these issues are reflected in the questions to ask from the previous section. Keep these points in mind when you are shopping for mobile code solutions.
As we pointed out in 1996, identifying applets is much harder than simply scanning for the <APPLET> tag in port 80 HTTP traffic. The problem of encrypted communication using something like SSL is especially hard to deal with. (Similar problems crop up with encrypted mail that includes mobile code.) The only completely guaranteed way of identifying all mobile code before it runs is by stopping it as it enters the VM. Your browser certainly knows when it encounters a Java applet, and it is probably in the best position to do something about it.
From a management perspective, identifying mobile code at the firewall has more appeal than trying to identify it on every desktop in an organization. This is especially true of organizations in which employees do not follow security policy. However, the problem is nontrivial. Ask hard questions about systems that claim to be able to stop all mobile code. Make sure the answers deal with all ports, multiple protocols, and encrypted connections.
Some vendors will tell you that their products allow the user to kill a running applet if it is misbehaving. Although many people are surprised to hear it, there's really no effective way to kill an applet if the applet doesn't want to die. The problem is that the only way to kill a Java thread is to throw an Exception at it, and the thread can simply catch the Exception. See page 125.
You might think that this problem can be solved by creating a new kind of Exception that cannot be caught, but that won't solve the problem either. The difficulty is that the try/finally statement in the Java language allows a program to do an arbitrary amount of "cleanup" work before it finishes running. In fact, the "cleanup" code can simply carry on the work of the thread you wanted to kill.
An even stronger way to try killing an applet is simply to stop all of the applet's threads, not even letting them execute their cleanup code. This doesn't work either, in general, because the stopped threads may be holding locks, and the VM may be unable to make progress without the locks. Revoking the locks while the victim threads are in the middle of using them only makes things worse, leaving the VM in a potentially inconsistent state.
This gets pretty complicated, but the bottom line is that there's no way to safely kill a thread that doesn't want to die. And if an applet thread is trying to cause trouble, why would it willingly let you kill it?
Of course, you can stop all running applets by killing the VM or browser entirely, but that's a pretty drastic solution, and you don't need some fancy tool to let you do it.
Java has important dynamic properties. Because of Java's dynamic loading capabilities, it is not possible to statically analyze a Java class completely. Some analysis will always have to be put off until runtime. The fact that the Verifier has a runtime phase is no accident (see Chapter 2).
Java's dynamic capabilities do not bode well for systems that claim to be able to scan applets for potentially harmful behavior. After all, if the code is not all there (or if it includes dangling references to not-yet-loaded code), how can you scan it? Static analysis is not sufficient to ensure security, and short of firing up a VM, that's all you can do.
Another problem is that scanning works according to the assumption that attack code can be categorized as such. We have seen (or written) plenty of attack code in the lab. It is not the case that this code can be identified as attack code before it does its dirty work. For example, one of the attacks described in Chapter 5 can be written as an applet that carries out seemingly innocuous type-casting operations, one after another. The casting actually implements the important part of a type-confusion attack. Looking at the code, it would be impossible to tell that the applet is an attack applet.
Many of the malicious applets written as demonstrations use very obvious techniques to carry out their hostile activities. For example, early denial-of-service or monitoring applets redefined the stop() method. This does not mean that every applet that redefines stop is malicious, nor does it mean that each and every attack applet will do so.
Scanning for unknown malicious behavior is a very difficult task that is certainly impossible in theory.
Providing security-related feedback to a user is a good thing, as long as the information is correct. However, if a meter can be spoofed, you can bet that a hostile applet will do so. The spoofing problem may be severe enough that it is best not to have meters. This may in fact be why Sun, Netscape, and Microsoft do not include such feedback mechanisms in their implementations.
Ask yourself why you should trust a meter and how you can ensure that it is giving you the correct information. Improving the VM's Security Model In the end, the most bang for the buck in terms of security will come from Sun and the Java licensees who create the VMs and APIs. Compared to creating a complete VM with added security mechanisms, improving the security situation by adding on minor services is probably nowhere near as effective.
Copyright ©1999 Gary McGraw and Edward Felten.