Repository navigation
[Java] Implement arrow-memory-ffm #163
Description
Activity
Not until Java 21+ (or really Java 24, when the new memory APIs will presumably be stabilized)
Also, this is a runtime flag, not a build flag
Ah I see. Updated to reference runtime now. Shall I close this given this won't happen any time soon?
The interesting things to me would be (1) implementing an arrow-memory-ffm that uses the experimental API so we can figure out how well it works and (2) refactoring BufferAllocator and other core APIs to be in line with the new upstream APIs so that when Java 24 comes, we're ready
Reacted by Dane PitkinBecause if something is missing for us it's kinda too late to communicate any feedback upstream
- changed the title
[-][Java] Remove dependency on JDK internals in Java 9+[/-][+][Java] Implement arrow-memory-ffm[/+]on Sep 28, 2023 - marked [Java] java.lang.reflect.InaccessibleObjectException on Java 18 #340 as a duplicate of this issue
on Mar 13, 2025 - marked [Java] Apache Arrow API fails with exceeded limit on max bytes to buffer when fetching data from snowflake #206 as a duplicate of this issue
on Mar 13, 2025 Hi there, the FFM API is final (since JDK 22, JEP 454) and
sun.misc.Unsafeis on its way out, so I prepared anarrow-memory-ffmmodule along the lines of what is suggested above. I'd like to open a PR if there's interest.
Here my branch: https://github.com/fb64/arrow-java/tree/memory-ffmOn the Unsafe side, the memory-access methods were deprecated for removal in JDK 23 (JEP 471), and since JDK 24 the JVM prints a warning on first use (JEP 498). The next phase, planned for "JDK 26 or later", makes them throw
UnsupportedOperationExceptionby default, followed by removal. You can already try that behavior with--sun-misc-unsafe-memory-access=deny. SinceMemoryUtilrelies on Unsafe whichever allocation manager is used, this will affect every Arrow Java user, not onlyarrow-memory-unsafe.It also covers the original request: no more
--add-opens=java.base/java.nio=ALL-UNNAMED.What my branch does:
- Moves the low-level operations of
MemoryUtilbehind a smallMemoryUtilAccessorinterface. The current Unsafe code becomesUnsafeMemoryAccessorand stays the default, so nothing changes for existing users. - Adds
arrow-memory-ffm(JDK 22+) with an FFM-based accessor and anFfmAllocationManager. Setting-Darrow.allocation.manager.type=FFMswitches both, so Unsafe is not used at all. - Includes a test run without
--add-opensto prove it isn't needed.
⚠️ One caveat: wrapping raw addresses goes throughMemorySegment.reinterpret, which is a restricted method, so--enable-native-access=org.apache.arrow.memory.ffm(orALL-UNNAMEDon the classpath) is needed to avoid a warning (JEP 454 says a future release may turn that warning into an error). That flag is the official opt-in for a public, final API, whereas--add-opensgives reflective access to private JDK fields that can change in any release.ℹ️ Full disclosure: I built this with the help of AI (Claude). Happy to walk through any part of it.
I can split it into two PRs if that's easier to review: the accessor refactoring first, then the new module...
- Moves the low-level operations of
Describe the enhancement requested
Add a memory module that uses the java.lang.foreign module (currently in preview, APIs updated in Java 21).
When using Java 9 or later with the current implementations, some JDK internals must be exposed by adding --add-opens=java.base/java.nio=ALL-UNNAMED to the java command[1]. Here's an article that explains the general Java issue here[2].
Can using java.lang.foreign remove this dependency so users do not have to run with custom options ?
[1]https://arrow.apache.org/docs/java/install.html#java-compatibility
[2]https://community.snowflake.com/s/article/JDBC-Driver-Compatibility-Issue-With-JDK-16-and-Later
Component(s)
Java