PRISM_DEPTH_MAXIMUM assumes ~6 MB of C stack; Ruby threads (1 MB) overflow at ~2000 levels #4305

Follow-up to #4252. With #4258 and fb2fea4 every nesting I tried now stops at "nesting too deep" on the main thread, including a pure C harness at 64000 levels. Thanks for the quick fix.

The limit only holds on a main-thread-sized stack though. Each level costs roughly 500–600 bytes of C stack (nested arrays are the worst case I measured; other constructs cost 150–450 bytes per depth unit), so reaching PRISM_DEPTH_MAXIMUM (10000) needs close to 6 MB. A Ruby thread gets 1 MB, and there the check never fires:

require "prism"

src = "x = " + "[" * 2000 + "]" * 2000
Prism.parse(src).errors                 # => []
Thread.new { Prism.parse(src) }.value   # => SystemStackError

Measured on main (cde8d11, Ruby 3.4.9, arm64 macOS):

Outside of CRuby that is a segfault rather than an exception, and Rust and the JVM default to 1–2 MB thread stacks as well; RUBY_THREAD_MACHINE_STACK_SIZE only helps whoever starts the CRuby process. I think the root is that #3413 matched the limit to Bison's YYMAXDEPTH, but Bison's stack is on the heap and Prism's is C frames (parse.y parses 9000 nested [ under ulimit -s 2048), so the same number does not give the same safety.

Options, in increasing order of effort:

1. Lower the default to something that fits 1 MB. Simple, but reverts Increase value of PRISM_DEPTH_MAXIMUM to 10000 #3413 and rejects code that parses today; real view templates already went past 1000 in Support a maximum depth to protect against malicious payloads #1774.
2. Keep the default and let the host set the limit through pm_options_t. No behaviour change; CRuby could pass a smaller value on non-main threads, which would reject some code that parses today (Support a maximum depth to protect against malicious payloads #1774 shows real templates past depth 1000) but still beats a segfault.
3. Check actual stack use instead of a count, like CRuby's stack_check and CPython 3.14. Most robust, but needs a platform-specific way to find the stack bounds, with the counter as fallback.

Happy to open a PR for whichever you prefer. I have option 2 implemented (yuki-300#1, if it helps to see the shape of it); option 3 I would want to discuss the API for first.

#include <prism.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) {
    FILE *f = fopen(argv[1], "rb");
    fseek(f, 0, SEEK_END); long n = ftell(f); fseek(f, 0, SEEK_SET);
    uint8_t *buf = malloc(n); fread(buf, 1, n, f); fclose(f);

    pm_arena_t *arena = pm_arena_new();
    pm_parser_t *parser = pm_parser_new(arena, buf, n, NULL);
    pm_node_t *root = pm_parse(parser);
    printf("parsed %p\n", (void *) root);

    pm_parser_free(parser);
    pm_arena_free(arena);
    return 0;
}

$ cc -Iinclude harness.c build/libprism.a -o harness
$ ruby -e 'File.write("n.rb", "x = " + "[" * 2000 + "]" * 2000)'
$ (ulimit -s 1024; ./harness n.rb); echo $?
139