Skip to content

[Java][Flight] ClassCastException in FlightClient when using Epoll Domain Socket: EpollDomainSocketChannel incorrectly cast to ServerChannel #1039

Closed

Description

@clh-kubernetes

Describe the bug, including details regarding any error messages, version, and platform.

h3. Summary
A {{ClassCastException}} occurs when initializing a {{FlightClient}} with Unix Domain Socket (UDS) and Epoll transport in Apache Arrow 16.0.0, 17.0.0, and 18.0.0. This is a regression, as the same code works perfectly in versions 12.0.0 and 15.0.0.

h3. Description
When creating a {{FlightClient}} to connect via {{Location.forGrpcUnix(...)}} on Linux with Epoll native transport enabled, the application crashes during initialization. The root cause is an incorrect type cast in the internal Netty/gRPC logic bundled with {{flight-core}}: it attempts to cast the client-side class {{EpollDomainSocketChannel}} to {{ServerChannel}}.

This logic error was introduced between version 15.0.0 and 16.0.0, and persists in all subsequent releases (16, 17, 18).

h3. Affected Versions

  • ❌ 18.0.0 (Confirmed Broken)
  • ❌ 17.0.0 (Confirmed Broken)
  • ❌ 16.0.0 (Confirmed Broken)
  • ✅ 15.0.0 (Works Correctly)
  • ✅ 12.0.0 (Works Correctly)

h3. Steps to Reproduce

Create a Java project with {{arrow-flight}} dependency set to 16.0.0, 17.0.0, or 18.0.0.

Ensure {{netty-transport-native-epoll}} is on the classpath.

Execute the following code:

{code:java}
import org.apache.arrow.flight.FlightClient;
import org.apache.arrow.flight.Location;

public class TestUds {
public static void main(String[] args) {
// Try to create a client using Unix Domain Socket
Location location = Location.forGrpcUnix("/tmp/arrow-test.sock");

    // This line triggers the crash in v16/v17/v18
    FlightClient client = FlightClient.builder(location).build();
    
    System.out.println("Client created successfully!");
}

}
{code}

Observe the {{ClassCastException}}.

Verification: Change dependency to 15.0.0, re-run, and observe it works without error.

h3. Error Log / Stack Trace
{code:java}
Exception in thread "main" java.lang.ClassCastException: class io.netty.channel.epoll.EpollDomainSocketChannel cannot be cast to class io.netty.channel.ServerChannel
at java.lang.Class.asSubclass(Class.java:3406)
at org.apache.arrow.flight.grpc..(.java:)
// NOTE: Please replace , , and with actual info from your stack trace if available.
// If using shaded jar, the class name might be obfuscated or internal.
...
{code}

h3. Root Cause Analysis
By decompiling {{flight-core}} jars for versions 16.0.0, 17.0.0, and 18.0.0, the following erroneous code was found in the initialization path:
{code:java}
// BUG: Incorrectly casting Client Channel to ServerChannel
Class<?> clazz = Class.forName("io.netty.channel.epoll.EpollDomainSocketChannel");
clazz.asSubclass(ServerChannel.class); // Fails because EpollDomainSocketChannel implements Channel, not ServerChannel
{code}

In versions 12.0.0 and 15.0.0, this logic either did not exist or correctly handled the channel types without cross-casting. It appears a refactoring introduced in v16.0.0 caused this regression, which has not been fixed in later versions.

h3. Environment

  • OS: Linux (Ubuntu/CentOS/Alpine)
  • Java: OpenJDK 11/17/21
  • Netty: 4.1.x (bundled with respective Arrow versions)
  • Architecture: x86_64 / aarch64

h3. Workaround

Downgrade: Use Apache Arrow 15.0.0 (or earlier) if UDS is strictly required.

Switch Protocol: Use TCP ({{Location.forGrpcInsecure}}) instead of Unix Domain Sockets in v16/v17/v18.

h3. Impact
This breaks any production system relying on FlightClient over Unix Domain Sockets in the latest three major stable releases (16, 17, 18), forcing users to downgrade significantly or switch to TCP.

Component(s)

FlightRPC

Activity

  1. transferred this issue fromapache/arrowon Feb 26, 2026
  2. raulcd commented on Feb 26, 2026

    @raulcd
    Member

    I've transferred the issue to the corresponding repository (https://github.com/apache/arrow-java instead of https://github.com/apache/arrow)

  3. idear1203 commented on Feb 26, 2026

    @idear1203

    I met this issue too. Is there a workaround? I am using v18.0.0 @raulcd

  4. ennuite commented on Feb 26, 2026

    @ennuite
    Contributor

    Hello. I will take a look into this one later this week if no one else grabs it until then.

  5. ennuite commented on Feb 27, 2026

    @ennuite
    Contributor

    @idear1203 and @clh-kubernetes I haven't yet repro-ed this on Linux as I am on Mac (I realise that Epoll is Linux-specific). However, I have checked and the Location class has no method called Location.forGrpcUnix(). UDS based Locations are instantiated via Location.forGrpcDomainSocket(String path)

    The FlightClient.Builder API also has (as of now and as of 18.0.0) no constructor that takes a single Location as an argument.

    Are you sure you are using the correct API? Can you provide a proper repro on the current version?

  6. clh-kubernetes commented on Feb 28, 2026

    @clh-kubernetes
    Author

    @ennuite
    Thank you for the quick review. I apologize for the confusion regarding the API method names in my initial description.

    The issue is not with the API usage itself, but with the internal implementation logic triggered when FlightClient.Builder.build() is called with a Domain Socket location on Linux.

    Since you are on macOS (which uses KQueue, not Epoll), you cannot reproduce this locally. The bug is strictly specific to the Epoll native transport path on Linux.

    Root Cause Analysis

    I have decompiled flight-core-17.0.0.jar (16 or 18) and identified the exact line causing the crash inside the FlightClient.Builder.build() method:
    // Internal logic inside FlightClient.Builder.build() or related Epoll initializer


    **Code: **
    case LocationSchemes.GRPC_DOMAIN_SOCKET:
    {
    // The implementation is platform-specific, so we have to find the classes at runtime
    builder = NettyChannelBuilder.forAddress(location.toSocketAddress());
    try {
    try {
    // Linux
    builder.channelType(
    Class.forName("io.netty.channel.epoll.EpollDomainSocketChannel")
    .asSubclass(ServerChannel.class));

      }
    

    // ❌ BUG: This cast is invalid.
    // EpollDomainSocketChannel implements 'Channel' (Client), NOT 'ServerChannel'.
    Class<? extends ServerChannel> serverClazz = clazz.asSubclass(ServerChannel.class);

    Why it fails:
    EpollDomainSocketChannel is a client-side channel class.

    • ServerChannel is the base interface for server-side listeners.
    • The correct server class would be EpollServerDomainSocketChannel.
    • Attempting to cast the client class to ServerChannel throws a java.lang.ClassCastException at runtime.

    Corrected Reproduction Steps (Linux Required)

    Environment:

    • OS: Linux (Ubuntu/CentOS/Alpine) - Must be Linux to trigger Epoll
    • Java: OpenJDK 11+
    • Dependency: {{org.apache.arrow:arrow-flight:17.0.0}} (or 18.0.0)
    • Native Lib: {{io.netty:netty-transport-native-epoll}} must be on the classpath.

    Code:
    import org.apache.arrow.flight.FlightClient;
    import org.apache.arrow.flight.Location;

    public class ReproUdsBug {
    public static void main(String[] args) {
    // 1. Use the correct public API
    Location loc = Location.forGrpcDomainSocket("/tmp/arrow-test.sock");
    BufferAllocator allocator = new RootAllocator();
    try {
    // 2. The crash happens INSIDE this .build() call
    FlightClient client = FlightClient.builder(allocator,loc).build();
    System.out.println("Success: Client created.");
    } catch (Exception e) {
    System.err.println("Error occurred: " + e.getClass().getSimpleName());
    e.printStackTrace();
    // Expected Output:
    // java.lang.ClassCastException: class io.netty.channel.epoll.EpollDomainSocketChannel
    // cannot be cast to class io.netty.channel.ServerChannel
    }
    }
    }
    {code}

    Summary

    • API Usage: The user code is correct (using {{forGrpcDomainSocket}} and {{builder().build()}}).
    • Trigger: Calling {{.build()}} with a Domain Socket location on Linux.
    • Defect: Internal reflection logic incorrectly casts a Client Channel class to a Server Channel interface.
    • Regression: Works in v15.0.0, broken in v16.0.0+.

    Thanks again for your help!

  7. ennuite commented on Mar 3, 2026

    @ennuite
    Contributor

    I will try to repro on Linux using the correct API. I will probably only get to this late this week or next week.

  8. clh-kubernetes commented on Apr 3, 2026

    @clh-kubernetes
    Author

    @ennuite
    Just wanted to follow up on this issue. I understand you mentioned you’d try to repro on Linux a few weeks ago, and I was wondering if you’ve had a chance to look into it yet.
    No rush—just checking in to see if there’s any update or if I can help provide more details to move this forward.
    Thanks for your time!

  9. ennuite commented on Apr 9, 2026

    @ennuite
    Contributor

    @clh-kubernetes sorry, I haven't looked into it yet as more pressing matters appeared. If you can, go ahead and provide a PR. Otherwise, I'll try to get to it before the next version is released.

  10. ennuite commented on Apr 20, 2026

    @ennuite
    Contributor

    @clh-kubernetes I had the time to look into this again and I was able to repro on Linux. The bug is also present on Mac with KQueue. I repro-ed on ARM with

        <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-transport-native-kqueue</artifactId>
        <version>4.1.112.Final</version>
        <classifier>osx-aarch_64</classifier>
        </dependency>
    

    instead of epoll. I have a PR for a fix.

  11. clh-kubernetes commented on May 6, 2026

    @clh-kubernetes
    Author

    @ennuite Thank you for the community's help and the proposed solution.
    I will apply the PR patch to our product code as suggested and conduct thorough testing.
    I will also keep an eye on the PR's merging progress and upgrade to the official release once available.

    Thanks again for all the guidance. Please let me know if there's anything else I should be aware of.

  12. ennuite commented on May 11, 2026

    @ennuite
    Contributor

    @clh-kubernetes is the patch working for you?

  13. added this to the 20.0.0 milestone on May 13, 2026
  14. clh-kubernetes commented on Jul 6, 2026

    @clh-kubernetes
    Author

    @ennuite Thank you for following up.

    I haven't applied the PR patch to our product code yet. Currently, we are temporarily avoiding this issue by pinning to a specific version of the dependency.

    We plan to wait for this fix to be officially merged and released in a stable version, and then we will upgrade directly to that official release to resolve the issue.

    Thanks again for your work on this!

  15. added a commit that references this issue on Aug 26, 2026
  16. ennuite commented on Aug 29, 2026

    @ennuite
    Contributor

    @clh-kubernetes just a heads-up that the fix was merged and is likely to be released in 20.0.0

    If possible let me know if worked for you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type: bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions