More Open Files for Java Apps on Mac OS X

Copyright 2003-2004, by Gregory L. Guerin. All rights reserved.
Redistribute only with all original accompanying files.

Last revised: 16Mar2004 GLG

Table of Contents

You can download a ZIP file containing this HTML document along with the plain-text template for the Prelude shell script discussed below. Of necessity, they apply only to Mac OS X, although they are readable on any platform.

Required Versions

The solution outlined here requires Mac OS X 10.2 or higher. It will not work before 10.2.

Introduction

Problem

A Java application can't have more than 256 files open at once on Mac OS X.

Cause

Each Unix process has an OS-enforced limit for certain consumable or limited resources. The number of files open at once is one such limit. Enforcing limits keeps processes from monopolizing these limited resources, either intentionally or unintentionally (i.e. due to software bugs). Reducing limits makes it easier to manage certain kinds of programs.

Background

For open files, the resource actually being limited is the file descriptor, often abbreviated fd. Each open file uses one descriptor. When a file is closed, its descriptor is released and is then reusable by the OS.

Other I/O channels also use file descriptors. An open network socket uses two descriptors. A pipe also uses two descriptors, one at each end. An open I/O device uses one descriptor. Current working directories also use one descriptor.

On Mac OS X, the default limit is 256 file-descriptors per process, as of 10.2. There is also a system-imposed hard-wired limit on the total number of descriptors in use at once, but it's at least in the thousands, so isn't a practical concern here.

Rather than explain all the possible resource-limits here, and how they affect running programs, see this man page:
    man 2 getrlimit

To see the hard-wired limit for file-descriptors, type this command:
    sysctl kern.maxfiles

Synopsis of Limits

Each limitable resource has a per-process soft-limit and hard-limit. A hard-limit can only be lowered, not raised (only the superuser can raise hard-limits). A soft-limit can be lowered or raised by any user, but never raised above the corresponding hard-limit.

Some resources have a "hard-wired" limit. Neither soft nor hard limits can exceed the hard-wired limit for that resource, not even for superuser. The number of file descriptors has a system-wide hard-wired limit (it's over 10,000 on my machine; YMMV).

A limit (soft or hard) changed in a process only affects that process. This means you can't run another process and have it affect the limits in the current process. This precludes the Java Runtime.exec() method from being of any use in changing resource limits for a Java program.

A process's current limits become the default limits inherited by a child process. This means a process can set limits, then fork or exec, and the new limits (soft and hard) will be inherited. For Java, the Runtime.exec() methods encompass both fork and exec. There is no standard Java API for changing limits, so a Process returned from Runtime.exec() inherits whatever limits were given to the Java process.

From the Shell Command-Line

Viewing Current Limits

To see the current limit values, use this command line:
tcsh :
limit
sh :
ulimit -a
You can also see the hard-limits in sh, but not in tcsh:
tcsh :
# impossible
sh :
ulimit -H -a

Setting Limits

A Java app run from the command-line can simply use the shell's own builtin limit-setting commands first. Or you can write a simple shell-script that sets the limits and then runs the 'java' command.

The shells have different command-names and syntax for setting open-file limits:

tcsh :
limit descriptors N
sh :
ulimit -S -n N
In both cases, the soft-limit for the number of open files is set to the decimal number N.

The sh shell lets you set soft-limits and hard-limits independently, using the -S and -H options, respectively. If both options are omitted, then both limits are set. This may be undesirable in some situations.

The tcsh shell only sets the soft-limit. You can't set the hard-limit using tcsh. This may be undesirable in some situations.

Running a Java Application with New Limits

You can run a Java app with either raised or lowered limits. Either manually at the command-line or in a shell-file, simply invoke the limit-setting command before running the java command:
tcsh :
limit descriptors 1000
java -jar YourApp.jar

sh :
ulimit -S -n 1000
java -jar YourApp.jar
With the sh shell, you can set the soft-limit as shown, or you can set the hard-limit, or you can set both limits. To a Java app, the soft-limit and hard-limit are indistinguishable, since there's no way for the Java app to change either one. Both limits are effectively immutable.

However, if a Java app uses Runtime.exec() to run other processes from native executable files, then those processes can make the soft/hard distinction. This may be important, depending on whether your Java app runs other processes that can raise the soft-limit for themselves.

From the Finder

Bundled Java Apps

Increasing the limits for a bundled double-clickable Java app is more difficult than in the shell. The most obvious impediment is that the Finder has no capability for viewing, raising, or lowering any resource limits. If it did, the desired limits would be a property of the application, perhaps changed in the app's "Get Info" window, or perhaps in the "Info.plist" file inside the app-bundle. Until Apple adds such a feature to the Finder or to Info.plist, we'll have to seek a solution elsewhere.

One solution would be to insert a "prelude" shell script into an app-bundle. The prelude would set the desired limits using shell commands, then execute the app-bundle's existing executable.

But is this possible? Will the app still look and act the same? Will it work when the app-bundle is moved or copied to another location? Can an app-bundle be developed with a prelude? Is a prelude compatible with installers? Can we see exactly what commands go into this prelude shell script?

As you may have guessed by now, the answers are all "Yes", BUT ONLY ON 10.2 OR HIGHER. Details below.

Double-Clickable Jar Files

The outlook is not nearly so rosy for double-clickable jar files. These Java apps do not provide any opportunity for inserting a "prelude" script. As a result, you can't change any limits for Java apps deployed solely as double-clickable jars. You can repackage the double-clickable jar as a Mac OS X app-bundle, though, then use the same techniques as for other Java app-bundles.

A double-clickable jar can be turned into an app-bundle fairly easily. Apple supplies the MRJAppBuilder tool, or the Jar Bundler tool for Java 1.4.1. These tools use a graphical interface to collect all the necessary information, along with the jar-file, then create a double-clickable app-bundle.

You can also use ProjectBuilder to make an app-bundle from an existing jar. The simplest approach is to create a "Pure Java app" project, add the existing jar-file to it, and specify the main-class as being the main class from the existing jar-file. Be sure to merge the existing jar-file into the output target, or add a Copy Files build phase for it. And be sure the rest of the target's Java-related options and properties are correct. You don't even have to remove the template-provided Java source files, since they won't actually be used or executed.

Prelude in /bin/sh

Nonrequisites

You don't have to know much about shell programming.

You don't have to be an expert on obscure Unix commands.

You don't have to change your Terminal or login shell to /bin/sh.

Prerequisites

You should already understand the basic anatomy of an app-bundle. You should also understand the basic role of the "Info.plist" file.

You will need to type a few commands in Terminal. If you've never done that before, this may not be the safest time to start.

You will need a plain text editor. TextEdit is fine. Just be sure to save as plain text, or the program will not run.

You will need to edit an "Info.plist" file. This is easiest with "Property List Editor", which is part of Apple's Developer Tools for Mac OS X. If you don't have it, a plain text editor like TextEdit will work, but then you should be a little familiar with XML. Deep XML expertise is not necessary, only passing familiarity, since you'll only be changing one value.

If you are uncertain, be cautious. If something might break, be sure you can throw it away. Copy files before changing them. Leave yourself an escape.

Brief Overview

Before going through the steps in detail, here they are in brief:
  1. Look into the app-bundle and get the name of the executable.
  2. Copy and paste the prelude-script's template into a text file.
  3. Change a name in the template to that of the current executable.
  4. Change the permissions of the prelude script so it will be executed.
  5. Move the prelude script to the correct place in the app-bundle.
  6. Change "Info.plist" to execute the prelude.
None of these steps is difficult. All of them are necessary. So if you leave one out, the resulting program will not work any more, and you'll have to determine why not and then fix it.

Follow These Steps

If you are unsure of something at any point, exercise the appropriate caution. Make copies before doing anything irreversible. If something breaks, be sure you can throw it away and start over.
  1. Look into the app-bundle and get the name of the executable.
Contextual-click on the app-bundle and choose "Show Package Contents". A View as List or as Columns will be most useful.

In the Contents/MacOS/ sub-directory, note the name of the executable file located there. If there is more than one file in Contents/MacOS, you will have to open the Info.plist file located in Contents. Find the CFBundleExecutable key and use its string value as the name of the executable.

  1. Copy and paste the prelude-script's template into a text file.
Run your plain text editor of choice, and open a new window. Copy and paste the following template text into the window.
#!/bin/sh

# To understand this, read 'man sh' below "substring processing".
# This can also be accomplished with the 'dirname' command,
# but it may not be installed.
here="${0%/*}"

# This should be the name of your native Java-app's executable.
cmd='JavaApplicationExecutable'

# Set all desired soft-limits here.  See 'ulimit' in 'man sh'.
ulimit -S -n 1000

# Execute the Java-app in the current process.  See 'exec' in 'man sh'.
exec "$here/$cmd" "$@"

You can also download a ready-to-use text template, along with this HTML document, and work from those files on your local machine.

  1. Change a name in the template to that of the current executable.
In the pasted template text, find this line:
cmd='JavaApplicationExecutable'
Change the JavaApplicationExecutable between the quotes to the name of the executable determined in step 1. Leave the quotes around the name. Quotes are required if the name contains a space or special character. They are harmless if it doesn't.

OPTIONAL: If you need more than 1000 files open at once, find this line:
ulimit -S -n 1000
and change the 1000 to a higher limit.

SAVE THE FILE AS PLAIN TEXT.

Give the file a name without any suffix. The remaining steps assume you named it Prelude. If you named it something else, substitute your file-name as appropriate.

Close the text editor's file window.

  1. Change the permissions of the prelude script so it will be executed.
In Terminal, go to wherever you saved your Prelude script file. That is, use cd as appropriate.

Type this command:

 
chmod 755 Prelude

If you named your file something else, substitute that name for Prelude.

  1. Move the prelude script to the correct place in the app-bundle.
First, double-check that the Prelude file has the correct permissions, and that you have write-permission on the Contents/MacOS sub-directory within the app-bundle. Assuming that Prelude is adjacent to the app-bundle, and you have cd'ed to that location, use this command:
 
ls -ld Prelude YourAppNameHere.app/Contents/MacOS

If you don't have write-permission on the directory, you won't be able to put Prelude into place. If Prelude is writable by anyone, then any user can change Prelude, possibly compromising system security.

Use the Finder or the mv command to move Prelude into Contents/MacOS. If you don't know how to use mv, use the Finder.

  1. Change "Info.plist" to execute the prelude.
First, confirm that you can write "Info.plist" by checking its permissions with ls.

If Property List Editor is available, simply double-click on "Info.plist" to launch it. Change the CFBundleExecutable key's value to be Prelude, or the name of your prelude script file. Only give the file name, not the pathname. Hit RETURN to enter the change, then save the "Info.plist".

If you don't have Property List Editor, drag "Info.plist" onto TextEdit or your plain text editor of choice. Find the XML-tagged key CFBundleExecutable. Change the immediately following tagged string to be Prelude. Save the file and close the window.

Double-Click It

But first, now is a good time to check your work. The Prelude script file must have execute permissions, or the app will not launch on double-click and you'll get an error message on the console. The Prelude file must be plain text, or the shell won't be able to understand it, even if the file permissions allow execution.

If everything seems OK, then double-click the app you've modified. It should launch and run normally. If it was previously hitting the open-files limit, it should no longer do so, assuming you've raised the limit high enough.

Normally, there is no visible difference in how the app runs. This is intentional. There is no simple way to tell that the Prelude script file is running first. No Terminal window is opened, nor is there a shell process waiting when you list all processes.

If you want a visible indication of the new behavior, you can add an echo command to the Prelude script file, or perhaps ulimit -a to list the limit values after you've changed them. All text emitted on stdout and stderr by the script will appear on the system console log.

You can see the system console log with the application /Applications/Utilities/Console.

Understanding the Prelude Commands

I'm not going to explain the commands used in the Prelude script, because this isn't a shell programming tutorial. If you want to understand exactly what they're doing, I suggest reading the appropriate man pages, as cited in Prelude's copious comments.

Fugue on /bin/sh

You can put any shell commands, either builtin or external, in the Prelude script before the exec command runs the original executable. Some shell commands affect the Java app's execution environment, often in useful or beneficial ways.

Some commands can adversely affect affect the Java app's execution environment, compromising the integrity or coherence of the Java Virtual Machine (JVM). These should be avoided.

Setting Hard-Limits

Every limitable resource has a hard-limit, as well as a soft-limit. Setting the hard-limit prevents any subsequent raising of the soft-limit beyond the hard-limit value. This may be useful in certain situations, though for most Java apps a soft-limit is indistinguishable from a hard-limit, since neither one is changeable by any Java API.

To set a hard-limit, use this command:
     ulimit -H lim value

Here, lim and value are the limit-code and the desired value. Once a hard-limit is set, it can only be lowered, never raised.

Setting the hard-limit alone has no effect on the soft-limit. To set both limits, invoke ulimit with both -H and -S, or omit both options so both limits will be set to the same value.

Changing Created-File Permissions (umask)

The value assigned to umask (a per-process variable maintained by the OS) affects the default read, write, and execute permissions of newly created files and directory. The umask value only affects newly created files and directories. It has no effect when an existing file is truncated and rewritten.

The default umask value is 022 (octal), so files and directories are normally created with write-permission cleared for group and other, but read-permission allowed to all. There may be situations where you want either more or less restrictive default permissions. To further understand the role of umask see man 2 umask, which describes the system-call, not the shell command.

Setting the umask value in the shell means the Java app inherits it. The umask value is also inherited by other processes the Java app executes. Regardless of the default settings for creation, the owner can change the permissions later. From Java, you'd use Runtime.exec() to run the chmod command, giving an absolute pathname for each file or directory to change.

The command to set the umask value takes one parameter, an octal mask value:
     umask 000

Setting the umask to 000 means that files and directories created by the Java app will be writable by the group and by everyone else.

To restrict created files and directories so they are only accessible to the owner, use:
     umask 077

A umask value of 0777 causes all permission-bits except owner-permissions to be cleared on newly created files and directories.

Changing Process Priority (nice)

A process's scheduling favorability affects all its threads, regardless of the individual Thread priority. You can reduce this favorability by increasing the "niceness" of the process. That is, a process with a higher "nice" level takes less CPU time away from other processes. In short, it "plays nice" with other processes.

To change the Java app's nice level, replace the last line of Prelude with:
     exec nice -n NN "$here/$cmd" "$@"

Replace NN with the numeric nice-level you want. 10 is a good all-around value. Actually, NN is an increment applied to the current nice-level, but the nice-level is usually 0 by default.

Ignoring Signals (trap)

By default, Mac OS X Java 1.3.1 apps arrange for the disposition of two signals: SIGQUIT and SIGPIPE. The SIGQUIT signal generates a full thread-dump to the console. The SIGPIPE signal is ignored.

By default, Mac OS X Java 1.4.1 apps also arrange for the disposition of additional signals: SIGHUP and SIGINT. Both signals are caught and cause the process to exit, after executing all onExit() handlers. Java 1.3.1 on Mac OS X DOES NOT do this. It's a known bug.

The -Xrs JVM option is honored by both 1.3.1 and 1.4.1 Java. It prevents any signal-handlers from being installed at all, and also stops SIGPIPE from being ignored.

The simplest way to tailor the signal environment of a Java app is with the nohup command. To use it in the Prelude replace the last line with this:
     exec nohup "$here/$cmd" "$@"

Using nohup causes SIGHUP to be ignored, but all other signal handling is unchanged.

To further tailor the signal environment, shell commands can arrange to ignore any ignorable signal before the Java app is executed:
     trap "" sig

Replace sig with the symbolic signal name, or the signal number, of the signal to be ignored. The trap command accepts signal names with or without the leading SIG part of the name. Repeat the trap command shown for every signal you want ignored, or simply append the additional sig names or numbers to a single trap command.

Ignored signals are inherited, so ignoring a signal means it will also be ignored in any processes spawned by the Java app. This may be desirable or undesirable, depending on circumstance and intent.

To see a list of symbolic signal-names, type this command in either sh or tcsh:
     kill -l

That's a lower-case ell, not the digit 1.

Commands to Avoid

Using cd to change the current directory is risky. The normal Java launcher stub may expect the current directory to be what it normally is at double-click launch. If you must change the directory, use the appropriate "Info.plist" property.

Ignoring signals that represent serious software failures is risky. For example, bus-errors (SIGBUS), illegal instructions (SIGILL), and segmentation violations (SIGSEGV) are normally fatal to a process, and for good reasons. If ignored, the Java virtual machine's integrity is lost, and who knows what may happen.

Anything involving superuser privileges is risky. That means the sudo command or the setuid-root bit. If you don't fully understand all the security implications, you could end up creating a very large security loophole that just screams "exploit me".

Scherzo for ProjectBuilder

If you are developing a Java-app in ProjectBuilder (PB), you can build it directly with a prelude shell script. You simply add the Prelude-tempate as a source file, change one target setting, then add a build phase that copies the Prelude into place in the app-bundle. This approach works with any bundled Java app.

The following description is for ProjectBuilder 2.1. It will generally work for earlier PB versions, but you may have to adjust some details or terms.

I don't know if these changes will work for PB-WO or XCode, since I don't use them.

Prelude-Template as Source

ProjectBuilder prefers source files to have suffixes. Without a suffix, PB won't run the text-editor when you double-click the file. We'll use a source file with a ".sh" suffix. You can change this in your own project if you don't like it.
  1. Copy the prelude-template file into the source directory of your project. It should not be nested in any sub-directory within the source directory.
  2. Add the prelude-template file to the Files pane of your project. Move it where you like in the Files pane.
  3. Uncheck the checkbox at the left of the prelude-template file in the Files pane.
  4. Edit the prelude-template and change this line so it reflects your project's product name:
    cmd='JavaApplicationExecutable'
    Usually PB will give the native executable the base name of the app, so use that name. In PB parlance, this base name is called the "product name".

Even though you have unchecked the prelude-template's checkbox, it will still be used to build the target, so is still considered a source file for the target. If you leave the checkbox checked, then PB will naively treat the file as a Java resource file, and will copy it verbatim into the JAR-file. This is nearly useless and Mostly Harmless, though it still shouldn't happen. Unchecking the checkbox prevents it.

Target Settings

One target setting must be changed from its typical Java-app value, so the Prelude script will be executed instead of the native executable. You can change this setting using either the Simple View or the Expert View of the relevant target section. Change it in one view or the other, not both.
Target : Info.plist Entries : Simple View
Under Basic Information, change the Executable: text-field from the target-name to Prelude.

Target : Info.plist Entries : Expert View
In the list of entries, locate the CFBundleExecutable key and change its value to Prelude.

PB will still create the native executable and give it the product name, placing it in the Contents/MacOS directory in the app-bundle. So be sure you don't have a product name of "Prelude", or grave disorder will ensue.

Build Phase

You must also add a Build Phase to the target. It will copy the prelude-template "source file" into the proper place in the app-bundle, and give it executable permissions. This takes more than just a Copy Files Build Phase, it takes a Shell Script Build Phase.

This new build phase doesn't use a separate shell script. Instead, the shell commands are pasted into a text-area directly in a sub-pane of the Target window.

To add and configure the build phase, follow these steps:

  1. In the main Project window, click the Targets tab.
  2. Under the Targets item double-click the target.
       (By default, it will have the same name as the base name of the project.)
  3. The Target window will open.
       (This is the same window used above to set the Info.plist entry to execute Prelude.)
  4. In the left-hand list, locate the Build Phases sub-list, and click the last item in its sub-list. This determines where the new build phase will be added: at the end of the building process.
  5. From the Project menu, chhose the New Build Phase hierarchical menu item, then choose the New Shell Script Build Phase menu item. A build phase will be added called Shell Script Files. Both the new phase and the former last phase will be selected.
  6. Select only the new Shell Script Files phase. Two text fields and a checkbox will appear in the large pane at the right:
    • The Shell: text field contains /bin/sh. Don't change it.
    • The Script: text area below it is empty. It will expand as you add to it.
    • The Run only when installing checkbox is unchecked. Don't change it.
  7. Click in the empty text area and paste the text of this shell script into it:
    # Define source and dest filenames here
    prelude_src="Prelude-template.sh"
    prelude_dst="Prelude"
    
    x_dir="$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS"
    
    /bin/cp "$SRCROOT/$prelude_src" "$x_dir/$prelude_dst" 
    /bin/chmod +x "$x_dir/$prelude_dst" 
    
    This script is also provided in a separate text file in the download.
  8. Edit this line to match the name of your prelude source file:
    prelude_src="Prelude-template.sh"
    Only enter the filename, not the entire source pathname. The shell script will automatically use the source-directory.
  9. Edit this line to match the name of your executable Prelude script:
    prelude_dst="Prelude"
    It should be the executable name you entered in the earlier step where you changed the Info.plist entry.
  10. Close the Target window
  11. Build.
  12. After the first build, you should look inside the resulting Java app-bundle and double-check that all the changes you made have turned out correctly:
    1. The Info.plist file should designate Prelude for CFBundleExecutable. No other keys or values should be any different, unless you changed them at the same time.
    2. The Contents/MacOS/ sub-directory should contain the original executable, and a Prelude file.
    3. The Contents/MacOS/Prelude file should have all executable permissions set.
    4. The contents of the Contents/MacOS/Prelude file should be plain text, and the line starting with cmd= should designate the name of your Java app's original executable (i.e. the name of the other file in Contents/MacOS/).
    If any of the above are not as described, make the necessary changes to your project's Target window, source-file pane, etc.

Coda

You may freely distribute the provided Prelude-template shell script, or any modified version of it in your app-bundles. The same goes for any PB project that uses the prelude-copying shell script. Both shell scripts are freely released into the public domain, AS-IS AND WITHOUT WARRANTY.

You may freely redistribute this HTML file only when it is accompanied by the original unmodified Prelude-template and PB shell scripts.

YOU MAY NOT DISTRIBUTE MODIFIED VERSIONS OF THIS HTML FILE WITHOUT EXPRESS WRITTEN PERMISSION.

Revision History

2004-Mar-16

Changed to use shell's ${parameter%word} notation, rather than 'dirname' command. This should allow the use of a prelude-script on machines where the user has elected not to install the BSD sub-system.

Added "Required Versions" and "Revision History" sections.


To Greg's Home Page
To Greg's How-To Page